From nobody Sat Sep 26 03:17:25 2026 Received: from fout-a1-smtp.messagingengine.com (fout-a1-smtp.messagingengine.com [103.168.172.144]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4C69B4E56C8 for ; Fri, 4 Sep 2026 15:10:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.144 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534635; cv=none; b=mWde+im5ARCRPp3SHr88Q6w2BOXGVCpH/0O+bPlyjKF66dBwNUzY2gIpvpU8FKG3i38mQFHf4D8LoazfNoGxFYDMVqDNSKLvPQtwHd65kHQJTz3etA9azeBzrejW4h/u+lSpyt8ynrEiB2KwppvhLNTVnRzP3nqt2eF97ry2fyM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534635; c=relaxed/simple; bh=DV43FZGIbSPlphinx7+Q4+0ApcdEK/kWt0TA3LKgMGs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=W1skRc+F8m97vu4Lemp+vdyRDacX+R//z5mVxvCs3ATUkxM/SXn5y3PRA7tXyXgvOZWf4wPq4A7dbEsUFfM5FNlZn3ENR78fQ7/9CEObFHtZtEwPWn7Ykf+VhrAGfGzf1ICHdBXIk0GaYGQ7G9q1i5ATC3dVBdNdcPW8RuJfLkE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=QuXF294A; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=S/TZ0Oq8; arc=none smtp.client-ip=103.168.172.144 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="QuXF294A"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="S/TZ0Oq8" Received: from phl-compute-01.internal (phl-compute-01.internal [10.202.2.41]) by mailfout.phl.internal (Postfix) with ESMTP id 18492EC00D1; Fri, 4 Sep 2026 11:10:32 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-01.internal (MEProxy); Fri, 04 Sep 2026 11:10:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1788534632; x= 1788621032; bh=C3vy3fSlXRoSP/bUwTQaq1sU+R07aCpj0vZUm+2kQF8=; b=Q uXF294AYI6HgdOVTi+0SXzP7FxYI8kwK/qKnYVmzqF9onqaZLKP2h+w6iVfYEjxg mj0Iv36T4NH/woorV3EvLIPRp0+tqL8CxizFiVr0ed+8SCmtn285eowLuS7mennZ avl+UMO4wS16rxUGxkDQZYaRq8fwsPhEG8OChARpkKrNNiDIi16+qQc3tAhyp2c1 F64eqtA2S99GsbmCNGJEAyf9/GJHvJQt6FF/OUw2pf22UHjhmCkB9JCShidDNiPc mQQodiW9mO+zXuH+uyEsZlI4YKxL/rLCz+Jasi7wgO1q30L73C4SnB9Z07ddJvuc kpzJdLW5hIbgb+helDTLg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1788534632; x=1788621032; bh=C 3vy3fSlXRoSP/bUwTQaq1sU+R07aCpj0vZUm+2kQF8=; b=S/TZ0Oq8tXHI29LV9 KmdOux2c6cOZTioUS397dqp7YOKaO65dtIPmGmW6F7lI6QBOJNOwo4VlP7tq0QwX YRMrjTh1UrPR5s6UrTPU4lySmK8yFPKqaWWEzgxNlNt5MWZy4CoKyEnX8J+m/Hig ZptmSG6AX0M1QPGJXKZw1evw8lzDKy7CnjqVTT6n2IIRx/Dgq5G76Wkqs7JFeISb YF52Go48ysGkK7grkANFBWiiVbDFDhXida0buGv+4YCDT+spxf9kuCj61tsSl2VM hb9al5mtCML5QJSXAZ1TZsaeqTxRbCJjzT+JhMhOPhdXS+SQUTrSxx7L8PbzTTVk /hPJQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFdOocfRN4jOpcQf0AqfSai/YCGo+xlO94MLvzn7lqitof50EENUNEgJzLPt2H+RQ DWgrwJL5dg2BaLkWQfjz0w8mX09EzID7A24sM1R0CcrzJ0aCsxP+pgO1accdG5RnvXKoNP Qi2+ranCgsFBQlrx3RUM64Xgm9l9zr2TPWbzFNtan3mUwkVlJ/gBEGXolzLuuMZ3CJ6e8L gmzum6oDDF3av5ecJE/wINba64+XlYa/U9oL8c0Dkr16eu3wp8w8wOsHTRTsCgO4oV0meO 0fb7rtZliPO8XPlA3k/rGcTv+01JWMnSe6fG3OACTE4UctUjoP9cM2YnBbUckeF0hhckij NOPFSUiZtbWGWwIaoa72i4eX2MNwTMmQMx1JJKB0oGx212Jn/3PmqqYSFnTW53IcV3p0vH NELpSMfP2Z6zjqkZ/tEiPBrUvz4BT4nMxYpIRTGKvlH2jencFZjJb++8kiwDeJqEUfh9Yl EQE2A7nt8xhteP9AZZOdyf4kp9m6aRLFjbBdIdcb2ZBxL7yuIn2fG7q6mssAIMYyV5HiSJ k/3VzUcClTcA/1nmaBr2Uwzq82jgLagZaIDFssmgoDl8IGnQ+pwmdJPCLoCpoUmy9OBDpM 8u0pkgSoDulr8fg8y6EFVhqgeMChQB4PJLZO2pmG8gfRsiQnGFn/nhHensIg X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 4 Sep 2026 11:10:31 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn , "Kiryl Shutsemau (Meta)" Subject: [PATCH 01/12] mm/khugepaged: drop redundant mm_struct pin in madvise_collapse() Date: Fri, 4 Sep 2026 16:10:15 +0100 Message-ID: <1ff754df1d57154e01356f35d72b1cb06c0f8664.1788533997.git.kas@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" madvise_collapse() holds an mmgrab() reference across its work. It is redundant. Every caller already holds mm_users: - madvise(2) works on current->mm, which lives as long as the task is in the syscall; - process_madvise(2) reaches a remote mm through mm_access(), which takes an mm_users reference and holds it until the syscall returns; - io_uring passes current->mm; - DAMON takes one with get_task_mm() and drops it after the call. Drop the mmgrab()/mmdrop() pair. Assisted-by: Claude-Code:claude-opus-5 Signed-off-by: Kiryl Shutsemau (Meta) Reviewed-by: Baolin Wang Reviewed-by: Zi Yan --- mm/khugepaged.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index f49a6710933b..f1f8d4375e96 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -3236,7 +3236,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, cc->is_khugepaged =3D false; cc->progress =3D 0; =20 - mmgrab(mm); lru_add_drain_all(); =20 for (addr =3D hstart; addr < hend; addr +=3D HPAGE_PMD_SIZE) { @@ -3292,7 +3291,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, } out_nolock: mmap_assert_locked(mm); - mmdrop(mm); kfree(cc); =20 return thps =3D=3D ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0 --=20 2.54.0 From nobody Sat Sep 26 03:17:25 2026 Received: from fhigh-a2-smtp.messagingengine.com (fhigh-a2-smtp.messagingengine.com [103.168.172.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2AF0D4CB8B7 for ; Fri, 4 Sep 2026 15:10:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.153 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534637; cv=none; b=jo44maCs6Ynnt+tolR1LKy3HrfOjqFZOa49Kb4HVhF7P8JJ1gZ6HxhOuRUajpSCtOPwql2IgKihYE29mg1gRmpuUi1XJjJbtsnucaSx0pAnlJ7TnrnNNXRkafskD6D4KSOQguisAipzLjuG1azH7h9O3W96MB48kfGzOnSOuuQQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534637; c=relaxed/simple; bh=pNR48Fc/9IOVMlUYIZ9D1Mkt9Aw1S4x4HIvGKwL0w9w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ePd8+X9Q/aAhIplBcD3AVtz+X4JZM7mrw+oPcKwF+PzdMfTlD98I38ndunrhGC0j8eDmkgE9XeunYYJW8A5xG/uu09VP9RQjI45rTc7EmbMHCneQLKczOF2JHCZzj5HGWdc7aIiy+TALR9SBPe54l14vsZYddAnLlx30efvqVMU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=L3NweBxO; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=gyV2zD0K; arc=none smtp.client-ip=103.168.172.153 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="L3NweBxO"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="gyV2zD0K" Received: from phl-compute-12.internal (phl-compute-12.internal [10.202.2.52]) by mailfhigh.phl.internal (Postfix) with ESMTP id 262ED1400106; Fri, 4 Sep 2026 11:10:34 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-12.internal (MEProxy); Fri, 04 Sep 2026 11:10:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1788534634; x= 1788621034; bh=b+81dAj5x80FGoAjSvzsRnSg4Kzca8fLKWA2APX95ZQ=; b=L 3NweBxOYvekh7y+hOqszpAnTsy8gz0WprsWq8AldTZqaL/QPKFChdgvlQrLfc5Y6 aqXesZ2d9nEwwTqE9cR6Ivgc0b3042sKxWSgANB2qWTF5KD3Ijzuqw3ZwjfQSneQ 3UtQLvzV5hPT5zDu3WX4SlMKFiszdVoUK/pjKw3IlW6k+Gr59ovuysX6fMaaCbL/ aDoZ0aJbBG0bQFGmRs9JtnF14+SJCcnByRyzvJmBaPSY4RJfW2ObrlDgq1GXoGjO mc+9E9DURSxJ/XYL5lvd+QAw8Ar1v0PMwoxF1fsHmlKKaZt5SFOaNenQY6QZpG47 srbYSaeyQOf6m7DwbmBRQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1788534634; x=1788621034; bh=b +81dAj5x80FGoAjSvzsRnSg4Kzca8fLKWA2APX95ZQ=; b=gyV2zD0K4arKTtjK+ 89MDgRn69V26iXslSbZHsojlXX+xmoSbFVH6nABVDMWD8tO2/sZv6ko71UnxWQXE xGDSSeA0yRJq7tu85SH/ENfOCw0ImG1pFhRwD+ZWfSRbzkXbWIo5p8c1sP+aY35b iLnC1mKnostEQ9pyNkYX7BANKA00/iX9hP9AJ3KLcL1JCrmjJoqdPmO6y7iTczon AgrxR4aQ+QKDO3lwuQhm8NVkLBR7aoMLp5Pj15nGOJzVRK5nlFJGoPgfnnvOM4dp 0rc0bA32oZfval7SI2f7+egEvMh4ylVRp+EzTYIt4LzPag2gC8hyJRKROgECubqT 1ThNA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFdOocfRN4jOpcQf0AqfSai/YCGo+xlO94MLvzn7lqitof50EENUNEgJzLPt2H+RQ DWgrwJL5dg2BaLkWQfjz0w8mX09EzID7A24sM1R0CcrzJ0aCsxP+pgO1accdG5RnvXKoNP Qi2+ranCgsFBQlrx3RUM64Xgm9l9zr2TPWbzFNtan3mUwkVlJ/gBEGXolzLuuMZ3CJ6e8L gmzum6oDDF3av5ecJE/wINba64+XlYa/U9oL8c0Dkr16eu3wp8w8wOsHTRTsCgO4oV0meO 0fb7rtZliPO8XPlA3k/rGcTv+01JWMnSe6fG3OACTE4UctUjoP9cM2YnBbUckeF0hhckpt aMYL+zmXSgfpUsAfEbdq3YcIoDlCmZxuiv4vEQV80awfFv7+kV3xGOfWwk+caLcPCmKQfj TKRiZUVKN65tTPwVy12o9rg9A8OAnzwOdHKxy66fYZzQOeRzRghLrCcywsyUmjySG2SorR AI7XMp9dEEpZKqUw0c8FQEevu0Y2iZ3joWaO39rAcm9Zba9QxhTcdlDK8OlRJM7b/9GNfo rX6/rUjmWqQUUBqaQ7kTJfP+KvRdWdx9PhtIWbhMDsjMWyFqorifeMQGo8xtnO9vLKpSNG Oo97AIto7b7gxYB/alJR+3nNAoiHuOgKfpdgoWIPPdo/w6UAQeSR9SqqhTMQ X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 4 Sep 2026 11:10:33 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn , "Kiryl Shutsemau (Meta)" Subject: [PATCH 02/12] mm/khugepaged: count collapses where khugepaged makes them Date: Fri, 4 Sep 2026 16:10:16 +0100 Message-ID: X-Mailer: git-send-email 2.55.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" collapse_single_pmd() bumps khugepaged_pages_collapsed for its caller, and tests cc->is_khugepaged to know whether it should: the counter belongs to the daemon, and MADV_COLLAPSE must not touch it. The daemon sees every result of every collapse it asks for, so it can keep its own counter without the shared path testing who called. Assisted-by: Claude-Code:claude-opus-5 Signed-off-by: Kiryl Shutsemau (Meta) Reviewed-by: Baolin Wang Reviewed-by: Zi Yan --- mm/khugepaged.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index f1f8d4375e96..a6dfd8cddc50 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -2822,10 +2822,8 @@ static enum scan_result collapse_single_pmd(unsigned= long addr, =20 mmap_assert_locked(mm); =20 - if (vma_is_anonymous(vma)) { - result =3D collapse_scan_pmd(mm, vma, addr, lock_dropped, cc); - goto end; - } + if (vma_is_anonymous(vma)) + return collapse_scan_pmd(mm, vma, addr, lock_dropped, cc); =20 file =3D get_file(vma->vm_file); pgoff =3D linear_page_index(vma, addr); @@ -2861,9 +2859,6 @@ static enum scan_result collapse_single_pmd(unsigned = long addr, result =3D SCAN_SUCCEED; mmap_read_unlock(mm); } -end: - if (cc->is_khugepaged && result =3D=3D SCAN_SUCCEED) - ++khugepaged_pages_collapsed; return result; } =20 @@ -2940,6 +2935,8 @@ static void collapse_scan_mm_slot(unsigned int progre= ss_max, =20 *result =3D collapse_single_pmd(khugepaged_scan.address, vma, &lock_dropped, cc); + if (*result =3D=3D SCAN_SUCCEED) + khugepaged_pages_collapsed++; /* move to next address */ khugepaged_scan.address +=3D HPAGE_PMD_SIZE; if (lock_dropped) --=20 2.54.0 From nobody Sat Sep 26 03:17:25 2026 Received: from fhigh-a2-smtp.messagingengine.com (fhigh-a2-smtp.messagingengine.com [103.168.172.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 542AD4DB54D for ; Fri, 4 Sep 2026 15:10:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.153 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534639; cv=none; b=ZMk0mvbnn0xvbz7CPed5e7PbZZ1WXB3LOyLLLTUlrLwifPnLwFmGeaKagctKf0Vbjh02mQt4kFqtorZanDdgI82G0QWnrYTlMENtMFQvk7w/S1Jrn+6K4OY0wltHqC4pcrscFRITw5i0Ypfvvr6+DZFeuhwY4GzjpcuN8Y5gmMk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534639; c=relaxed/simple; bh=jL/f54IMG5VmysE1ZlLv1NWtDbPe2S6xW+fhY41T6gk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mwnqdB5WfNLQHVbqk0dEcTmYOQ3sWTyJbi6IL/be3CjUC92zvM0OH3rDKi1ufZBzcOG+PgM9RhCcsgfHeQsQ3UbPkL5cFUGXU+NWw3PVI1E9mjM/veJS2hJghehH6U0StxkJMZ2qzBBmfFJiHTCUq8BYgXXSom3DFj0cLPVtLRg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=UeQYIDSD; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=TbxokLAR; arc=none smtp.client-ip=103.168.172.153 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="UeQYIDSD"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="TbxokLAR" Received: from phl-compute-04.internal (phl-compute-04.internal [10.202.2.44]) by mailfhigh.phl.internal (Postfix) with ESMTP id 26F08140011C; Fri, 4 Sep 2026 11:10:36 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-04.internal (MEProxy); Fri, 04 Sep 2026 11:10:36 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1788534636; x= 1788621036; bh=YY32sTxGw5u22cQhPIzHWlsuJTNCMKfMNjS6KMgVuVA=; b=U eQYIDSDnH5OvngSpvGHvVo3gjlzcllo+eRqgriAT4qaUdR6byVr4t3cUEMw+MS92 2MXf8VXpVmVRnK3fxA8i5Ui6gOMB5bljDM3QIj2+lISJBh2GjJ7shINU0h3CI4oN j0B+MdUpZvN4cPoHW+XcKLS/43zigUAEIvWDiyNCNrzAkBjcRPjdkBRGFDqrE/00 h+ULUrlS+pjvaHTEmxlsG2PyITKgFpUxF6YzDzICmhNSh1o1res4UkjW1f7wztDj +bZgCZEmSJBmQ8UQXn3b7gTzNicBHfc/BSvTOktu33K09iCznyKkP8DLpOg6npBO Stp85hdL+5fDLM5mCR/oA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1788534636; x=1788621036; bh=Y Y32sTxGw5u22cQhPIzHWlsuJTNCMKfMNjS6KMgVuVA=; b=TbxokLAR+WGQjP10e MH2CLKul/0MB9TxdSvyFhHKSB1veiP7eJzxXBUwxm2yQ1a3bF5HtuXiuOYY3XR4o TmZi7FeDN9sBhK5gydL7H7JEhXl69+pU/OJXDHskXtD5V3h5X5vQRPoeXEWpR/dc Zrx2MXE2RoSM/wxssYawghAvBmpHF5U1xagG+ObRRISgz5hqDV6UvOc3WQBc5gfR yuY/GxgiGdrjNb+wo1iPY5pVQ6fBKc93Jd6JwOYcGVIlEmJcI5sWXygRKS2668fp 5q2WXie3TN2m8rxBe2Ozm11BGMxH6j0mTR4jPgAzJsfNaPaEa37ylleQ6y8xkFXl GAzyg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFdOocfRN4jOpcQf0AqfSai/YCGo+xlO94MLvzn7lqitof50EENUNEgJzLPt2H+RQ DWgrwJL5dg2BaLkWQfjz0w8mX09EzID7A24sM1R0CcrzJ0aCsxP+pgO1accdG5RnvXKoNP Qi2+ranCgsFBQlrx3RUM64Xgm9l9zr2TPWbzFNtan3mUwkVlJ/gBEGXolzLuuMZ3CJ6e8L gmzum6oDDF3av5ecJE/wINba64+XlYa/U9oL8c0Dkr16eu3wp8w8wOsHTRTsCgO4oV0meO 0fb7rtZliPO8XPlA3k/rGcTv+01JWMnSe6fG3OACTE4UctUjoP9cM2YnBbUckeF0hhckN2 IulBVu1pSPD4k7hx5yDYEsMaz/SErjERf9GO6dSsa5y3Y99AOOolJH9jARZNZeHimVdXjA 4YclnFCUC+U/t6wU7LYHopP4G/HV07ttDttUc/oRFksnZw752uJJ2/3C9D3bESssdeF9fW WRCGTCnLL0HxFbtT3gGrFWAUtIg/5ry9onTV5dKzanRUQXzh1NOSOI2MD6lWx6z1HjLhEF heeP30UM6YbYHN2RWXbT2QWzph3LUMiV5gc5+OU9iC5milImg4V6UerIkxU5boAlpMFhUV b7RXy/WOhJw9CkjjYyUArotiZ8q/DcBeqCEkOzEoZYQFCDauvcdoWIZf3NqA X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 4 Sep 2026 11:10:35 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn , "Kiryl Shutsemau (Meta)" Subject: [PATCH 03/12] mm/khugepaged: rename mthp_present_ptes bitmap to eligible_ptes Date: Fri, 4 Sep 2026 16:10:17 +0100 Message-ID: X-Mailer: git-send-email 2.55.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" The name says less than the bit means. A set bit means not only that the PTE is present, but also that it passed the other checks: uffd, lazyfree, anonymity, sharing. The PTE can be considered a collapse source. mthp_collapse() then reads the bitmap starting at the PMD order, so the bitmap is not specific to mTHP either. Name it for what a set bit means, and update the comments that named it. No functional change. Assisted-by: Claude-Code:claude-opus-5 Signed-off-by: Kiryl Shutsemau (Meta) Reviewed-by: Baolin Wang Reviewed-by: Zi Yan --- mm/khugepaged.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index a6dfd8cddc50..6a437d6fe016 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -115,8 +115,8 @@ struct collapse_control { /* nodemask for allocation fallback */ nodemask_t alloc_nmask; =20 - /* Each bit represents a single occupied (!none/zero) page. */ - DECLARE_BITMAP(mthp_present_ptes, MAX_PTRS_PER_PTE); + /* Each bit marks a PTE the scan accepted as a collapse source */ + DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); }; =20 /** @@ -627,7 +627,7 @@ static void collapse_control_init_scan(struct collapse_= control *cc) { memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); - bitmap_zero(cc->mthp_present_ptes, MAX_PTRS_PER_PTE); + bitmap_zero(cc->eligible_ptes, MAX_PTRS_PER_PTE); } =20 static void release_pte_folio(struct folio *folio) @@ -1482,8 +1482,8 @@ static unsigned int max_order_from_offset(unsigned in= t offset) * mthp_collapse() consumes the bitmap that is generated during * collapse_scan_pmd() to determine what regions and mTHP orders fit best. * - * Each bit in cc->mthp_present_ptes represents a single occupied (!none/z= ero) - * page. We start at the PMD order and check if it is eligible for collaps= e; + * Each bit in cc->eligible_ptes marks a PTE the scan accepted as a collap= se + * source. We start at the PMD order and check if it is eligible for colla= pse; * if not, we check the left and right halves of the PTE page table we are * examining at a lower order. * @@ -1514,12 +1514,12 @@ static enum scan_result mthp_collapse(struct mm_str= uct *mm, goto next_order; =20 max_ptes_none =3D collapse_max_ptes_none(cc, NULL, order); - nr_occupied_ptes =3D bitmap_weight_from(cc->mthp_present_ptes, offset, + nr_occupied_ptes =3D bitmap_weight_from(cc->eligible_ptes, offset, offset + nr_ptes); =20 /* * Swap PTEs accepted during the scan are counted in @unmapped, - * not in the present-PTE bitmap. Account them for the PMD-order + * not in cc->eligible_ptes. Account them for the PMD-order * candidate. */ if (is_pmd_order(order)) @@ -1731,8 +1731,8 @@ static enum scan_result collapse_scan_pmd(struct mm_s= truct *mm, } } =20 - /* Set bit for occupied pages */ - __set_bit(i, cc->mthp_present_ptes); + /* The scan accepted this PTE as a collapse source */ + __set_bit(i, cc->eligible_ptes); /* * Record which node the original page is from and save this * information to cc->node_load[]. --=20 2.54.0 From nobody Sat Sep 26 03:17:25 2026 Received: from fout-a1-smtp.messagingengine.com (fout-a1-smtp.messagingengine.com [103.168.172.144]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D93384EBAFD for ; Fri, 4 Sep 2026 15:10:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.144 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534641; cv=none; b=aux2VwJbqKRq5Zaw4iskuRia44xJ4BqjqPaGvIZM51g39UqoaQl4/hoalKkkxYXI6LU8vzGtF361NR71XkPaTe13e2KoZjYHV4s2KX/9UScTZnVxnGsq9UdTm46f1qrrrKcxsQs1aCPMtiZl0/4VXm/Ugx4em9Lg3eQFoCtI0OE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534641; c=relaxed/simple; bh=1NZNGa/Aw9M2kA9PN64vUL4MGJOi0br60Hrt3XL1fUI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KS8RZTcJGGlHckzmBoImarlrBL2UIHK35kemekbp0mFircI9E0m+IUN3HzcZ0YygzgmWxdPr6UHCqvPRO5VWtdK8ANjzQ8Z/398kKJKeru8SQboLX9NDxXbsOPCwT/8/nrB4/gPGip5dXrelLXLMwpOItxVQXd6sMut0GPnXa/E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=BwOt+/mF; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=Tcm0kQra; arc=none smtp.client-ip=103.168.172.144 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="BwOt+/mF"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Tcm0kQra" Received: from phl-compute-03.internal (phl-compute-03.internal [10.202.2.43]) by mailfout.phl.internal (Postfix) with ESMTP id EAD38EC00AA; Fri, 4 Sep 2026 11:10:37 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-03.internal (MEProxy); Fri, 04 Sep 2026 11:10:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1788534637; x= 1788621037; bh=nyjSpTkitaIBwBQZFVJW04eBDamEuOqIJHNiEPmdqUM=; b=B wOt+/mFnt/h/sU+raMZxq/x1lYjxE5boHz0Hy8uREHkZaQbiqeWSGEEFwPzXfZ9h qBOpgGexXa8fFndj4lUwdjMX/97SC1FNwJg4r3t7b8gjcLFHckbY4m5xfI9ROWZI sAljvXaKv4Mq+L3AqEiQGuML1CabxtZYzP2dek7FYyb9Mxl5y+XUw4Ajzej3NW7i zftOnVvczlm5eviZgAjOYIPIaT1wVEF3Yeb4b++ElYDHwfYS00vW6+ygv7yKPj5V dVE9ZgPZH2XQmCAeFkZziCiHJRCj5++s4Y3BMMG4My7tajZh+OJFHG3pwKu/YNfD /VdwsO6YNaeZAATW66brQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1788534637; x=1788621037; bh=n yjSpTkitaIBwBQZFVJW04eBDamEuOqIJHNiEPmdqUM=; b=Tcm0kQratdTRy0Ygm P8kL1rZ0CLg9Qv0GrMWpgMEY6iXtoSrrRVbqm5C9u9NNxdBkQrNXr2Ux6f2+PkYr DGqQNm264adjxGuRKJ+uEA43oJMBjsBvJX+zS6cdjPTi7Ec6YVImhp5Cz5Oppjki gt3NZc2PBG4nSI/tjQYOCzi4bZW/LWIyrhZ8qAnusL3ok9dYhw5H7SA8Cv0jDzK4 RnKDMiZxrlGHZQNavRrncekq+gfqZxPnmbvN6v5Qu/tNalB3JqyXPK15tATFXo9T IHtV5Nhkjw+7WncrfcXxlC3ezOD9F1Ml7jky71kwLy9YLJT90yzw1vcsU5FV3U5H VVr8A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFdOocfRN4jOpcQf0AqfSai/YCGo+xlO94MLvzn7lqitof50EENUNEgJzLPt2H+RQ DWgrwJL5dg2BaLkWQfjz0w8mX09EzID7A24sM1R0CcrzJ0aCsxP+pgO1accdG5RnvXKoNP Qi2+ranCgsFBQlrx3RUM64Xgm9l9zr2TPWbzFNtan3mUwkVlJ/gBEGXolzLuuMZ3CJ6e8L gmzum6oDDF3av5ecJE/wINba64+XlYa/U9oL8c0Dkr16eu3wp8w8wOsHTRTsCgO4oV0meO 0fb7rtZliPO8XPlA3k/rGcTv+01JWMnSe6fG3OACTE4UctUjoP9cM2YnBbUckeF0hhckbq Cvz8P0mWGH45WdOwqnu+TKhOr5lwKVKaMaZXfHg9pm/WzuEvez6nqQcMig9ee23pQT31jw lmXwJAIgzz8MoxjOko5trLcvXSnI2oKuuqaTJA8txpBkIwZoisx0vYswzoGI0pqomz0FKS m5gEJgYwGJd/KDyqT/mCOiTQE3HyM+MgLriFa3c35xwK3aPx3PR6PBmlY2DR++iRoh09E7 L7tCNNLglqlPCbc406tgC8hCPZQSP/xqDAWKT1xiyKaedO/SI/dc56VVJrkiarMU3nCC+e oKsesJWIY0XwAoZMINTbJm7TYwNxM4dPRS0e1rXNI2QuCQn0BPdZKdwmuPVQ X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 4 Sep 2026 11:10:37 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn , "Kiryl Shutsemau (Meta)" Subject: [PATCH 04/12] mm/collapse: add collapse.h for the collapse interface Date: Fri, 4 Sep 2026 16:10:18 +0100 Message-ID: X-Mailer: git-send-email 2.55.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" khugepaged.c holds both the users of collapse and the machinery that performs it. The daemon's scan loop, the sysfs tunables, MADV_COLLAPSE's entry point and the collapse itself all sit in one file and reach into each other freely. Nothing marks where a user ends and the engine begins. Start drawing that line. Add mm/collapse.h for what the two sides have to agree on: - enum scan_result - what the engine hands back; - struct collapse_control - the state a request carries. And two constants move with them: - KHUGEPAGED_MAX_PTES_LIMIT -> COLLAPSE_MAX_PTES_LIMIT; - KHUGEPAGED_MIN_MTHP_ORDER -> COLLAPSE_MIN_MTHP_ORDER. Neither is a fact about the daemon, so both lose the KHUGEPAGED_ prefix. No functional change. Assisted-by: Claude-Code:claude-opus-5 Signed-off-by: Kiryl Shutsemau (Meta) Reviewed-by: Baolin Wang Reviewed-by: Zi Yan --- MAINTAINERS | 1 + mm/collapse.h | 67 +++++++++++++++++++++++++++++++++++++++++ mm/khugepaged.c | 79 ++++++++----------------------------------------- 3 files changed, 81 insertions(+), 66 deletions(-) create mode 100644 mm/collapse.h diff --git a/MAINTAINERS b/MAINTAINERS index a9245d827ddb..4d1ff4c76496 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -17433,6 +17433,7 @@ F: Documentation/admin-guide/mm/transhuge.rst F: include/linux/huge_mm.h F: include/linux/khugepaged.h F: include/trace/events/huge_memory.h +F: mm/collapse.h F: mm/huge_memory.c F: mm/khugepaged.c F: mm/mm_slot.h diff --git a/mm/collapse.h b/mm/collapse.h new file mode 100644 index 000000000000..1c40229b9554 --- /dev/null +++ b/mm/collapse.h @@ -0,0 +1,67 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __MM_COLLAPSE_H +#define __MM_COLLAPSE_H + +#include +#include +#include +#include + +/* The most the max_ptes_* tunables accept */ +#define COLLAPSE_MAX_PTES_LIMIT (HPAGE_PMD_NR - 1) + +/* The smallest order a collapse will build */ +#define COLLAPSE_MIN_MTHP_ORDER 2 + +enum scan_result { + SCAN_FAIL, + SCAN_SUCCEED, + SCAN_NO_PTE_TABLE, + SCAN_PMD_MAPPED, + SCAN_EXCEED_NONE_PTE, + SCAN_EXCEED_SWAP_PTE, + SCAN_EXCEED_SHARED_PTE, + SCAN_PTE_NON_PRESENT, + SCAN_PTE_UFFD, + SCAN_PTE_MAPPED_HUGEPAGE, + SCAN_LACK_REFERENCED_PAGE, + SCAN_PAGE_NULL, + SCAN_SCAN_ABORT, + SCAN_PAGE_COUNT, + SCAN_PAGE_LRU, + SCAN_PAGE_LOCK, + SCAN_PAGE_ANON, + SCAN_PAGE_LAZYFREE, + SCAN_PAGE_COMPOUND, + SCAN_ANY_PROCESS, + SCAN_VMA_NULL, + SCAN_VMA_CHECK, + SCAN_ADDRESS_RANGE, + SCAN_DEL_PAGE_LRU, + SCAN_ALLOC_HUGE_PAGE_FAIL, + SCAN_CGROUP_CHARGE_FAIL, + SCAN_TRUNCATED, + SCAN_PAGE_HAS_PRIVATE, + SCAN_STORE_FAILED, + SCAN_COPY_MC, + SCAN_PAGE_FILLED, + SCAN_PAGE_DIRTY_OR_WRITEBACK, +}; + +struct collapse_control { + bool is_khugepaged; + + /* Num pages scanned per node */ + u32 node_load[MAX_NUMNODES]; + + /* Num pages scanned (see khugepaged_pages_to_scan) */ + unsigned int progress; + + /* nodemask for allocation fallback */ + nodemask_t alloc_nmask; + + /* Each bit marks a PTE the scan accepted as a collapse source */ + DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); +}; + +#endif /* __MM_COLLAPSE_H */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6a437d6fe016..972843c45250 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -26,44 +26,10 @@ #include =20 #include +#include "collapse.h" #include "internal.h" -#include "page_alloc.h" #include "mm_slot.h" - -enum scan_result { - SCAN_FAIL, - SCAN_SUCCEED, - SCAN_NO_PTE_TABLE, - SCAN_PMD_MAPPED, - SCAN_EXCEED_NONE_PTE, - SCAN_EXCEED_SWAP_PTE, - SCAN_EXCEED_SHARED_PTE, - SCAN_PTE_NON_PRESENT, - SCAN_PTE_UFFD, - SCAN_PTE_MAPPED_HUGEPAGE, - SCAN_LACK_REFERENCED_PAGE, - SCAN_PAGE_NULL, - SCAN_SCAN_ABORT, - SCAN_PAGE_COUNT, - SCAN_PAGE_LRU, - SCAN_PAGE_LOCK, - SCAN_PAGE_ANON, - SCAN_PAGE_LAZYFREE, - SCAN_PAGE_COMPOUND, - SCAN_ANY_PROCESS, - SCAN_VMA_NULL, - SCAN_VMA_CHECK, - SCAN_ADDRESS_RANGE, - SCAN_DEL_PAGE_LRU, - SCAN_ALLOC_HUGE_PAGE_FAIL, - SCAN_CGROUP_CHARGE_FAIL, - SCAN_TRUNCATED, - SCAN_PAGE_HAS_PRIVATE, - SCAN_STORE_FAILED, - SCAN_COPY_MC, - SCAN_PAGE_FILLED, - SCAN_PAGE_DIRTY_OR_WRITEBACK, -}; +#include "page_alloc.h" =20 #define CREATE_TRACE_POINTS #include @@ -91,7 +57,6 @@ static DECLARE_WAIT_QUEUE_HEAD(khugepaged_wait); * * Note that these are only respected if collapse was initiated by khugepa= ged. */ -#define KHUGEPAGED_MAX_PTES_LIMIT (HPAGE_PMD_NR - 1) unsigned int khugepaged_max_ptes_none __read_mostly; static unsigned int khugepaged_max_ptes_swap __read_mostly; static unsigned int khugepaged_max_ptes_shared __read_mostly; @@ -101,24 +66,6 @@ static DEFINE_READ_MOSTLY_HASHTABLE(mm_slots_hash, MM_S= LOTS_HASH_BITS); =20 static struct kmem_cache *mm_slot_cache __ro_after_init; =20 -#define KHUGEPAGED_MIN_MTHP_ORDER 2 - -struct collapse_control { - bool is_khugepaged; - - /* Num pages scanned per node */ - u32 node_load[MAX_NUMNODES]; - - /* Num pages scanned (see khugepaged_pages_to_scan) */ - unsigned int progress; - - /* nodemask for allocation fallback */ - nodemask_t alloc_nmask; - - /* Each bit marks a PTE the scan accepted as a collapse source */ - DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); -}; - /** * struct khugepaged_scan - cursor for scanning * @mm_head: the head of the mm list to scan @@ -267,7 +214,7 @@ static ssize_t max_ptes_none_store(struct kobject *kobj, unsigned long max_ptes_none; =20 err =3D kstrtoul(buf, 10, &max_ptes_none); - if (err || max_ptes_none > KHUGEPAGED_MAX_PTES_LIMIT) + if (err || max_ptes_none > COLLAPSE_MAX_PTES_LIMIT) return -EINVAL; =20 khugepaged_max_ptes_none =3D max_ptes_none; @@ -292,7 +239,7 @@ static ssize_t max_ptes_swap_store(struct kobject *kobj, unsigned long max_ptes_swap; =20 err =3D kstrtoul(buf, 10, &max_ptes_swap); - if (err || max_ptes_swap > KHUGEPAGED_MAX_PTES_LIMIT) + if (err || max_ptes_swap > COLLAPSE_MAX_PTES_LIMIT) return -EINVAL; =20 khugepaged_max_ptes_swap =3D max_ptes_swap; @@ -318,7 +265,7 @@ static ssize_t max_ptes_shared_store(struct kobject *ko= bj, unsigned long max_ptes_shared; =20 err =3D kstrtoul(buf, 10, &max_ptes_shared); - if (err || max_ptes_shared > KHUGEPAGED_MAX_PTES_LIMIT) + if (err || max_ptes_shared > COLLAPSE_MAX_PTES_LIMIT) return -EINVAL; =20 khugepaged_max_ptes_shared =3D max_ptes_shared; @@ -378,19 +325,19 @@ static unsigned int collapse_max_ptes_none(struct col= lapse_control *cc, if (is_pmd_order(order)) return max_ptes_none; /* - * for mTHP collapse with the sysctl value set to KHUGEPAGED_MAX_PTES_LIM= IT, + * for mTHP collapse with the sysctl value set to COLLAPSE_MAX_PTES_LIMIT, * scale the maximum number of PTEs to the order of the collapse. */ - if (max_ptes_none =3D=3D KHUGEPAGED_MAX_PTES_LIMIT) + if (max_ptes_none =3D=3D COLLAPSE_MAX_PTES_LIMIT) return (1 << order) - 1; /* - * For mTHP collapse of values other than 0 or KHUGEPAGED_MAX_PTES_LIMIT, + * For mTHP collapse of values other than 0 or COLLAPSE_MAX_PTES_LIMIT, * emit a warning and return 0. */ if (max_ptes_none) pr_warn_once("mTHP collapse does not support max_ptes_none" " values other than 0 or %u, defaulting to 0.\n", - KHUGEPAGED_MAX_PTES_LIMIT); + COLLAPSE_MAX_PTES_LIMIT); return 0; } =20 @@ -476,7 +423,7 @@ int __init khugepaged_init(void) return -ENOMEM; =20 khugepaged_pages_to_scan =3D HPAGE_PMD_NR * 8; - khugepaged_max_ptes_none =3D KHUGEPAGED_MAX_PTES_LIMIT; + khugepaged_max_ptes_none =3D COLLAPSE_MAX_PTES_LIMIT; khugepaged_max_ptes_swap =3D HPAGE_PMD_NR / 8; khugepaged_max_ptes_shared =3D HPAGE_PMD_NR / 2; =20 @@ -1571,8 +1518,8 @@ static enum scan_result mthp_collapse(struct mm_struc= t *mm, * any smaller order enabled. When at the smallest order * we must always move to the next offset. */ - if (order > KHUGEPAGED_MIN_MTHP_ORDER && - (enabled_orders & GENMASK(order - 1, 0))) { + if (order > COLLAPSE_MIN_MTHP_ORDER && + (enabled_orders & GENMASK(order - 1, 0))) { order--; continue; } @@ -1636,7 +1583,7 @@ static enum scan_result collapse_scan_pmd(struct mm_s= truct *mm, * is then checked again in mthp_collapse() for each attempted order. */ if (enabled_orders !=3D BIT(HPAGE_PMD_ORDER)) - max_ptes_none =3D KHUGEPAGED_MAX_PTES_LIMIT; + max_ptes_none =3D COLLAPSE_MAX_PTES_LIMIT; =20 pte =3D pte_offset_map_lock(mm, pmd, start_addr, &ptl); if (!pte) { --=20 2.54.0 From nobody Sat Sep 26 03:17:25 2026 Received: from fhigh-a2-smtp.messagingengine.com (fhigh-a2-smtp.messagingengine.com [103.168.172.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A182E4EC66B for ; Fri, 4 Sep 2026 15:10:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.153 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534643; cv=none; b=HI1ECZmVI+G6qwYrI5rl51vQz/vGm4XR9a8uORWHzOigeAx66vx5UZN3WOeQBlSxvhzDlAJknHaefqwRsNtN3vnqR9B+2dmesNT84HqScYkj0dm1CiHAWT/7rgGcFQUcXDkAM+AayM1JKS97QbbERHCOeDBV2uq/NAQDClpQjjo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534643; c=relaxed/simple; bh=WdBtMF4isc7/JDB6frBzaI22IARq9/7jAG7UchgoEgE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=c/e0uFlXDnTIbR35eLJYsoBRA5vGx0oTHK51SS6a+pQWsxdI0VfeegzZJOGAuDQzLaO+J94l2cBJwcgel9UzWsZpunKLWReeOpeyxbclJlI0y0oCo/Tx/LuXXqRuTRsThAW0FZKxWfHH1k+sk2lVoO8sFET/CvAFP6hm914RbTg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=hgV2KVoQ; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=KGzzt8l1; arc=none smtp.client-ip=103.168.172.153 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="hgV2KVoQ"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="KGzzt8l1" Received: from phl-compute-06.internal (phl-compute-06.internal [10.202.2.46]) by mailfhigh.phl.internal (Postfix) with ESMTP id B8CF914000EA; Fri, 4 Sep 2026 11:10:39 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-06.internal (MEProxy); Fri, 04 Sep 2026 11:10:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1788534639; x= 1788621039; bh=rLA7WGbf/bUqUAdGBfnElNM3zHqju4YnHH35h18Mb0g=; b=h gV2KVoQPO2VsLzcoHKArMbkEFSMR3/2iBz7Ox7ti/ffXDO+SSS6/b1bkIZ4Akrng OFvD1tfiT5m0A3OgAcYeUdmZzFQ9uSWp1UcWud+KTizuuxx2jh/OV2vpuK2sHjVF y9HKKigWYGhNyGxM8lTpbF2yMSqyScXCUiZW/pogdklw8giFDqtWoS4WA0MMRq2L c3Zgfg4nekTRKQN/f9Jyz7kUQYds5piJRI4as5YOV7NGEnHrwKpRnDQ8TMdyioAk pmV/kQJnOR8tbQGPCjWR8q/PajFO5ZHwjH88g12DnLH+XXuqz3GOg/ad1wIRF7tq l2ZdwGHYkA89Ilja7aJwA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1788534639; x=1788621039; bh=r LA7WGbf/bUqUAdGBfnElNM3zHqju4YnHH35h18Mb0g=; b=KGzzt8l1pL6S1EyO0 i0PNKdSw0JtMFkYulF0YfChV+1ePOoI6CiT6FtW7C11Q6bHJEG/PGIH25boZb+pe zjOVZRwfGr3DUEfqtybahHXL99g5FB2I5t4B8pg71h6FZH7G9I3ucyUeH+pk+473 9sezE/Xn+tSlSXDppUz5iU/3Pdzh8TZWRhFWWQ4TYCnTj6xgUATgr047c7TrCrrP YHOw/RAAgyVkXmI7hrUR5DrzV1zF8nXfnjma17FwDjTQZHIUYMzw7X6NX7vZL1jm gkVcUA+TwOuQyI1iCoBBkby5S/eH9vXub82bi2Bg8FAU5tn2y7hJLAdzHga7Jw0c oZJxA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFdOocfRN4jOpcQf0AqfSai/YCGo+xlO94MLvzn7lqitof50EENUNEgJzLPt2H+RQ DWgrwJL5dg2BaLkWQfjz0w8mX09EzID7A24sM1R0CcrzJ0aCsxP+pgO1accdG5RnvXKoNP Qi2+ranCgsFBQlrx3RUM64Xgm9l9zr2TPWbzFNtan3mUwkVlJ/gBEGXolzLuuMZ3CJ6e8L gmzum6oDDF3av5ecJE/wINba64+XlYa/U9oL8c0Dkr16eu3wp8w8wOsHTRTsCgO4oV0meO 0fb7rtZliPO8XPlA3k/rGcTv+01JWMnSe6fG3OACTE4UctUjoP9cM2YnBbUckeF0hhckq8 CrtWwR2uhU2I1JD0k6Cz3GcTQq/ifO5r/VydO+twK6bWOVPlUdwD/E7yEObtvrMuZYO8Qf lczd1T9bKgnPs22g/xVjpuS0fzG2XlUxsiH40/4fjjzhQHttmkrWuKguyfBOC69U2jazhH /s3PpCy0OVqY4FPMbfCrnujEPoBPv85yS/kdjm2jDKfEaki2L4L89yYh1gE/GbTjrLmg5v 50BFYNaeK25GWSLYOoPe4/3Wb/yglv7RRIhvSOAVQtAJ0Jt+XHH9Lmc6GlNjy0tHPCQwqq dgwKinq6Xf1wvdKZT9MDeaN+vDxTbIV021ed1imjviQLBE9hKuailnlUAMxA X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 4 Sep 2026 11:10:39 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn , "Kiryl Shutsemau (Meta)" Subject: [PATCH 05/12] mm/collapse: state what a collapse may do in the policy Date: Fri, 4 Sep 2026 16:10:19 +0100 Message-ID: X-Mailer: git-send-email 2.55.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" Tests scattered through the collapse path decide what a collapse is allowed to do by asking whether khugepaged started it. Between them they settle: - which VMAs are eligible, and how hard to try for a folio; - how many empty, swapped-out or shared PTEs a window may contain, and whether a sub-PMD window is held to a stricter rule than a PMD; - whether a range has to look used, and whether a MADV_FREE'd page is left alone; - whether the PMD is mapped as part of the request, and whether dirty pages are worth writing back and retrying. None of those is a fact about khugepaged. Each is something the caller decided before asking, and the collapse code should not have to look up who called to find out. Add struct collapse_policy for the caller to fill: khugepaged from its own settings, MADV_COLLAPSE from the fact that a user asked explicitly. Every test becomes a read of a field, and cc->is_khugepaged goes, having no reader left. khugepaged fills the policy once per scan pass, MADV_COLLAPSE once per call. That is the one change in behaviour. The max_ptes_* limits and the defrag setting behind the allocation mask are sampled once per pass rather than on every table. A table scanned early in a pass and one scanned late are then judged alike. collapse_file() also drops a NULL check on the collapse_control. It has one call site, reached only from collapse_single_pmd(), which dereferences cc unconditionally, so the check was already dead. Assisted-by: Claude-Code:claude-opus-5 Signed-off-by: Kiryl Shutsemau (Meta) --- mm/collapse.h | 40 ++++++++++++++++- mm/khugepaged.c | 114 ++++++++++++++++++++++++++---------------------- 2 files changed, 102 insertions(+), 52 deletions(-) diff --git a/mm/collapse.h b/mm/collapse.h index 1c40229b9554..05282eed9a35 100644 --- a/mm/collapse.h +++ b/mm/collapse.h @@ -48,8 +48,46 @@ enum scan_result { SCAN_PAGE_DIRTY_OR_WRITEBACK, }; =20 +/* What a collapse is allowed to do, decided by the caller that asks for i= t */ +struct collapse_policy { + /* Limits, stated per PMD; HPAGE_PMD_NR means "no limit" */ + unsigned int max_ptes_none; + unsigned int max_ptes_swap; + unsigned int max_ptes_shared; + + /* + * Hold a sub-PMD window to a stricter rule than a PMD: no swapped-out + * and no shared PTEs at all, and max_ptes_none as + * collapse_max_ptes_none() scales it. + */ + bool strict_sub_pmd; + + /* + * Collapse only where it looks worth doing: require some sign the + * range is in use, and leave clean lazyfree folios for reclaim rather + * than collapsing them into a folio that is not lazyfree. + */ + bool skip_lazyfree; + bool require_referenced; + + /* + * Finish the job rather than leaving it half done for a fault to pick + * up: map the PMD over a file collapse before returning, and write + * dirty pages back and retry once instead of refusing them. Both cost + * latency the caller has to be willing to pay. + */ + bool install_pmd; + bool writeback_dirty; + + /* How hard to try for a destination folio */ + gfp_t gfp; + + /* Which VMAs are eligible, as thp_vma_allowable_orders() spells it */ + enum tva_type tva_type; +}; + struct collapse_control { - bool is_khugepaged; + struct collapse_policy policy; =20 /* Num pages scanned per node */ u32 node_load[MAX_NUMNODES]; diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 972843c45250..b2ebacfcc0be 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -314,15 +314,12 @@ static bool pte_none_or_zero(pte_t pte) static unsigned int collapse_max_ptes_none(struct collapse_control *cc, struct vm_area_struct *vma, unsigned int order) { - const unsigned int max_ptes_none =3D khugepaged_max_ptes_none; + const unsigned int max_ptes_none =3D cc->policy.max_ptes_none; =20 if (vma && userfaultfd_armed(vma)) return 0; - /* for MADV_COLLAPSE, allow any empty/shared zeropage PTEs */ - if (!cc->is_khugepaged) - return HPAGE_PMD_NR; - /* for PMD collapse, respect the user defined maximum */ - if (is_pmd_order(order)) + /* The limit as given, at the PMD order and wherever it is not capped */ + if (is_pmd_order(order) || !cc->policy.strict_sub_pmd) return max_ptes_none; /* * for mTHP collapse with the sysctl value set to COLLAPSE_MAX_PTES_LIMIT, @@ -354,19 +351,12 @@ static unsigned int collapse_max_ptes_shared(struct c= ollapse_control *cc, unsigned int order) { /* - * For MADV_COLLAPSE, do not restrict the number of PTEs that map shared - * anonymous pages. + * A sub-PMD window held to the strict rule takes no shared page at all: + * an mTHP is not worth the CoW-breaking. */ - if (!cc->is_khugepaged) - return HPAGE_PMD_NR; - /* - * for mTHP collapse do not allow collapsing anonymous memory pages that - * are shared between processes. - */ - if (!is_pmd_order(order)) + if (!is_pmd_order(order) && cc->policy.strict_sub_pmd) return 0; - /* for PMD collapse, respect the user defined maximum */ - return khugepaged_max_ptes_shared; + return cc->policy.max_ptes_shared; } =20 /** @@ -382,16 +372,12 @@ static unsigned int collapse_max_ptes_swap(struct col= lapse_control *cc, unsigned int order) { /* - * For MADV_COLLAPSE, do not restrict the number PTEs entries or - * pagecache entries that are non-present. + * A sub-PMD window held to the strict rule takes nothing non-present: + * reading pages back to build an mTHP is not worth the latency. */ - if (!cc->is_khugepaged) - return HPAGE_PMD_NR; - /* for mTHP collapse do not allow any non-present PTEs or pagecache entri= es */ - if (!is_pmd_order(order)) + if (!is_pmd_order(order) && cc->policy.strict_sub_pmd) return 0; - /* for PMD collapse, respect the user defined maximum */ - return khugepaged_max_ptes_swap; + return cc->policy.max_ptes_swap; } =20 int hugepage_madvise(struct vm_area_struct *vma, @@ -678,7 +664,7 @@ static enum scan_result __collapse_huge_page_isolate(st= ruct vm_area_struct *vma, * If the vma has the VM_DROPPABLE flag, the collapse will * preserve the lazyfree property without needing to skip. */ - if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) && + if (cc->policy.skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) && folio_test_lazyfree(folio) && !pte_dirty(pteval)) { result =3D SCAN_PAGE_LAZYFREE; goto out; @@ -767,12 +753,12 @@ static enum scan_result __collapse_huge_page_isolate(= struct vm_area_struct *vma, if (folio_test_large(folio)) list_add_tail(&folio->lru, compound_pagelist); next: - if (cc->is_khugepaged && + if (cc->policy.require_referenced && folio_pte_referenced(folio, vma, addr, pteval)) referenced++; } =20 - if (unlikely(cc->is_khugepaged && !referenced)) { + if (unlikely(cc->policy.require_referenced && !referenced)) { result =3D SCAN_LACK_REFERENCED_PAGE; } else { result =3D SCAN_SUCCEED; @@ -938,9 +924,7 @@ static void khugepaged_alloc_sleep(void) remove_wait_queue(&khugepaged_wait, &wait); } =20 -static struct collapse_control khugepaged_collapse_control =3D { - .is_khugepaged =3D true, -}; +static struct collapse_control khugepaged_collapse_control; =20 static bool collapse_scan_abort(int nid, struct collapse_control *cc) { @@ -976,6 +960,36 @@ static inline gfp_t alloc_hugepage_khugepaged_gfpmask(= void) return khugepaged_defrag() ? GFP_TRANSHUGE : GFP_TRANSHUGE_LIGHT; } =20 +/* khugepaged collapses on its own initiative, so it obeys its own setting= s */ +static void collapse_policy_khugepaged(struct collapse_policy *p) +{ + p->max_ptes_none =3D READ_ONCE(khugepaged_max_ptes_none); + p->max_ptes_swap =3D READ_ONCE(khugepaged_max_ptes_swap); + p->max_ptes_shared =3D READ_ONCE(khugepaged_max_ptes_shared); + p->strict_sub_pmd =3D true; + p->skip_lazyfree =3D true; + p->require_referenced =3D true; + p->install_pmd =3D false; + p->writeback_dirty =3D false; + p->gfp =3D alloc_hugepage_khugepaged_gfpmask(); + p->tva_type =3D TVA_KHUGEPAGED; +} + +/* MADV_COLLAPSE was asked for explicitly, so it is not held to those */ +static void collapse_policy_forced(struct collapse_policy *p) +{ + p->max_ptes_none =3D HPAGE_PMD_NR; + p->max_ptes_swap =3D HPAGE_PMD_NR; + p->max_ptes_shared =3D HPAGE_PMD_NR; + p->strict_sub_pmd =3D false; + p->skip_lazyfree =3D false; + p->require_referenced =3D false; + p->install_pmd =3D true; + p->writeback_dirty =3D true; + p->gfp =3D GFP_TRANSHUGE; + p->tva_type =3D TVA_FORCED_COLLAPSE; +} + #ifdef CONFIG_NUMA static int collapse_find_target_node(struct collapse_control *cc) { @@ -1013,8 +1027,7 @@ static enum scan_result hugepage_vma_revalidate(struc= t mm_struct *mm, unsigned l struct collapse_control *cc, unsigned int order) { struct vm_area_struct *vma; - enum tva_type type =3D cc->is_khugepaged ? TVA_KHUGEPAGED : - TVA_FORCED_COLLAPSE; + enum tva_type type =3D cc->policy.tva_type; =20 if (unlikely(collapse_test_exit_or_disable(mm))) return SCAN_ANY_PROCESS; @@ -1197,8 +1210,7 @@ static enum scan_result __collapse_huge_page_swapin(s= truct mm_struct *mm, static enum scan_result alloc_charge_folio(struct folio **foliop, struct m= m_struct *mm, struct collapse_control *cc, unsigned int order) { - gfp_t gfp =3D (cc->is_khugepaged ? alloc_hugepage_khugepaged_gfpmask() : - GFP_TRANSHUGE); + gfp_t gfp =3D cc->policy.gfp; int node =3D collapse_find_target_node(cc); struct folio *folio; =20 @@ -1551,7 +1563,7 @@ static enum scan_result collapse_scan_pmd(struct mm_s= truct *mm, const unsigned int max_ptes_shared =3D collapse_max_ptes_shared(cc, HPAGE= _PMD_ORDER); const unsigned int max_ptes_swap =3D collapse_max_ptes_swap(cc, HPAGE_PMD= _ORDER); unsigned int max_ptes_none =3D collapse_max_ptes_none(cc, vma, HPAGE_PMD_= ORDER); - enum tva_type tva_flags =3D cc->is_khugepaged ? TVA_KHUGEPAGED : TVA_FORC= ED_COLLAPSE; + enum tva_type tva_flags =3D cc->policy.tva_type; pmd_t *pmd; pte_t *pte, *_pte, pteval; int i; @@ -1651,7 +1663,7 @@ static enum scan_result collapse_scan_pmd(struct mm_s= truct *mm, * If the vma has the VM_DROPPABLE flag, the collapse will * preserve the lazyfree property without needing to skip. */ - if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) && + if (cc->policy.skip_lazyfree && !(vma->vm_flags & VM_DROPPABLE) && folio_test_lazyfree(folio) && !pte_dirty(pteval)) { result =3D SCAN_PAGE_LAZYFREE; failed_pfn =3D folio_pfn(folio); @@ -1717,13 +1729,13 @@ static enum scan_result collapse_scan_pmd(struct mm= _struct *mm, goto out_unmap; } =20 - if (cc->is_khugepaged && + if (cc->policy.require_referenced && folio_pte_referenced(folio, vma, addr, pteval)) referenced++; } - if (cc->is_khugepaged && - (!referenced || - (unmapped && referenced < HPAGE_PMD_NR / 2))) { + if (cc->policy.require_referenced && + (!referenced || + (unmapped && referenced < HPAGE_PMD_NR / 2))) { result =3D SCAN_LACK_REFERENCED_PAGE; } else { result =3D SCAN_SUCCEED; @@ -2585,11 +2597,11 @@ static enum scan_result collapse_file(struct mm_str= uct *mm, unsigned long addr, xas_unlock_irq(&xas); =20 /* - * Remove pte page tables, so we can re-fault the page as huge. - * If MADV_COLLAPSE, adjust result to call try_collapse_pte_mapped_thp(). + * Remove pte page tables, so we can re-fault the page as huge. A + * caller that wants the PMD mapped now is told to go and do that. */ retract_page_tables(mapping, start); - if (cc && !cc->is_khugepaged) + if (cc->policy.install_pmd) result =3D SCAN_PTE_MAPPED_HUGEPAGE; folio_unlock(new_folio); =20 @@ -2780,11 +2792,8 @@ static enum scan_result collapse_single_pmd(unsigned= long addr, retry: result =3D collapse_scan_file(mm, addr, file, pgoff, cc); =20 - /* - * For MADV_COLLAPSE, when encountering dirty pages, try to writeback, - * then retry the collapse one time. - */ - if (!cc->is_khugepaged && result =3D=3D SCAN_PAGE_DIRTY_OR_WRITEBACK && + /* Dirty pages are worth a writeback and one more try, if asked for */ + if (cc->policy.writeback_dirty && result =3D=3D SCAN_PAGE_DIRTY_OR_WRITEB= ACK && !triggered_wb && mapping_can_writeback(file->f_mapping)) { const loff_t lstart =3D (loff_t)pgoff << PAGE_SHIFT; const loff_t lend =3D lstart + HPAGE_PMD_SIZE - 1; @@ -2801,7 +2810,7 @@ static enum scan_result collapse_single_pmd(unsigned = long addr, result =3D SCAN_ANY_PROCESS; else result =3D try_collapse_pte_mapped_thp(mm, addr, - !cc->is_khugepaged); + cc->policy.install_pmd); if (result =3D=3D SCAN_PMD_MAPPED) result =3D SCAN_SUCCEED; mmap_read_unlock(mm); @@ -2950,6 +2959,9 @@ static void khugepaged_do_scan(struct collapse_contro= l *cc) =20 lru_add_drain_all(); =20 + /* One policy for the whole pass, so every table is judged the same */ + collapse_policy_khugepaged(&cc->policy); + cc->progress =3D 0; while (true) { cond_resched(); @@ -3177,7 +3189,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, cc =3D kmalloc_obj(*cc); if (!cc) return -ENOMEM; - cc->is_khugepaged =3D false; + collapse_policy_forced(&cc->policy); cc->progress =3D 0; =20 lru_add_drain_all(); --=20 2.54.0 From nobody Sat Sep 26 03:17:25 2026 Received: from fhigh-a2-smtp.messagingengine.com (fhigh-a2-smtp.messagingengine.com [103.168.172.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 809484EC66A for ; Fri, 4 Sep 2026 15:10:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.153 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534644; cv=none; b=WmFuF5hlm/nghWnk+6HCjFSWAaix1ATyNVgVd4F/pUcao+Qx/yUTmL7jCxzKW6RecSBDiPGqF5WXr64oQTOBcp5mV2EnVk4pg7ipNMberXFRGh7HzcBFOFStLGn3T8OBH1rpaKtwb3eCG8bOTDgyVnJ7QG+PopOd3gQRrluZH8c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534644; c=relaxed/simple; bh=4SwiAWgnMxZCM7ZH+Jm7F+4dNIvxguffe8DUPkPElq0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Rn3J2h7PA7bMhdGNcnof23cfsrHGHInrkqRuYDlmeUW2hlOhL0WqmmrE45myfrefgKp5DN2GsbGpquYrXHOs4VViD1+DGDkvgtuZOT+QdfUCOSDFQL2TVycEq9gkd+pTs8VkZyF0vTj0mBswOdEbnEruFodkRbDgWe2YHfANQuk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=XwaYnG5j; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=DX+ujuqC; arc=none smtp.client-ip=103.168.172.153 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="XwaYnG5j"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="DX+ujuqC" Received: from phl-compute-02.internal (phl-compute-02.internal [10.202.2.42]) by mailfhigh.phl.internal (Postfix) with ESMTP id 8DED61400106; Fri, 4 Sep 2026 11:10:41 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-02.internal (MEProxy); Fri, 04 Sep 2026 11:10:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1788534641; x= 1788621041; bh=csaSFAKtS72MfKNk/auWa3JqV7XZUOIFQz4rsLI1NhI=; b=X waYnG5j1sSW2fwEzdyndDYCRRtS4ErnAiqu+9/6bftCwiKT6VUiCZJ1AdCV6cZqv FpAdvxcgEnOXmuigqNFSCvGhGxnk/WJ71//t2I+jw2Lhm1v1zQ4PwefHaKZhpFEv 5365GptykMY+ws5agze1/r5HJmw1RPftgMBbJ9GMI3NjjAMwP6eWjb+pmxmBn6OZ 36BqbGhu/f9ifoK0sU33hHsGlm7Wu1UksKLt8r2dU6MH++9hin4zN33/zIt4i8T7 NsNds6UrQF+LxguJcPRqdTzvAkHf5somE3LPmdV6XtqMzxuzxTgkBj44etoqjE/b ocYQFy0RPos87gZANUWuQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1788534641; x=1788621041; bh=c saSFAKtS72MfKNk/auWa3JqV7XZUOIFQz4rsLI1NhI=; b=DX+ujuqCNxKmXGTWG rrJ0ev8RZB/VwCcBAgnB2OSN1Yzjzjw9iUp3LJodqP5FJ+7fkh6eUGtyajP1x45N AxBEpGkwhHGRjREV99cTB3wFwcJesYGu/PtcX92vlNPK4EjweIjL94lULMS4v3FQ X8Shq0T2jhRnEVNwaDR+8Ri+wNkPhoUOuEnvdhDSNc6Ze0ioUA99is0TjyqknHgE KIvMor/UZdd5BZdQhVjyvAX5rxBl6VVz4F3W3qC2BKH1PvpbdvRNFsmtJPd6lrwd q7tkhc6FOTE+eNHv5cck+e6xMnNpnNXP0oGmngJmatSdSnZXhzYDO4pKfiAexanA msRWg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFdOocfRN4jOpcQf0AqfSai/YCGo+xlO94MLvzn7lqitof50EENUNEgJzLPt2H+RQ DWgrwJL5dg2BaLkWQfjz0w8mX09EzID7A24sM1R0CcrzJ0aCsxP+pgO1accdG5RnvXKoNP Qi2+ranCgsFBQlrx3RUM64Xgm9l9zr2TPWbzFNtan3mUwkVlJ/gBEGXolzLuuMZ3CJ6e8L gmzum6oDDF3av5ecJE/wINba64+XlYa/U9oL8c0Dkr16eu3wp8w8wOsHTRTsCgO4oV0meO 0fb7rtZliPO8XPlA3k/rGcTv+01JWMnSe6fG3OACTE4UctUjoP9cM2YnBbUckeF0hhckLt MMHeYhNDUPl4H+anSl0bx4O31nPweggRutuPuOSD3x0SBk0h7PGsgbLKNhVs6w0dYfOFiK uYRiJbsY1zG3wyIMbFQDDpgC56vqbkfDVNXSp3AcIV3y1+SR/EVO5E6n2mSmw5kgS1Faln KMe0a7IYL3FakTwavg3ixHmnnqMW/ll9wuw4GLGJgwQiA3X9obmCgwVf36wlnRxDap0Eki tDwPFPWlgUMJdvEnd+0mPBdshgAAjvK8sQxd7Y9dei+aygeLU8zgYgO4eYrnVK3AQ+JjML OjzR8R+srhjyw4VZMOJryhJc0YwyTEIK8RraXetOdXVeQhlZwH0f/tNpkNow X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 4 Sep 2026 11:10:40 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn , "Kiryl Shutsemau (Meta)" Subject: [PATCH 06/12] mm/collapse: drop the collapse_possible() wrapper Date: Fri, 4 Sep 2026 16:10:20 +0100 Message-ID: X-Mailer: git-send-email 2.55.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" collapse_possible() only forwards to collapse_possible_orders() and turns its mask into a bool. Its three callers can test the mask themselves. No functional change. Assisted-by: Claude-Code:claude-opus-5 Signed-off-by: Kiryl Shutsemau (Meta) Reviewed-by: Baolin Wang Reviewed-by: Zi Yan --- mm/khugepaged.c | 15 +++++---------- 1 file changed, 5 insertions(+), 10 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index b2ebacfcc0be..22272cfe97e6 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -512,17 +512,11 @@ static unsigned long collapse_possible_orders(struct = vm_area_struct *vma, return thp_vma_allowable_orders(vma, vm_flags, tva_flags, orders); } =20 -static bool collapse_possible(struct vm_area_struct *vma, - vm_flags_t vm_flags, enum tva_type tva_flags) -{ - return collapse_possible_orders(vma, vm_flags, tva_flags); -} - void khugepaged_enter_vma(struct vm_area_struct *vma, vm_flags_t vm_flags) { - if (!mm_flags_test(MMF_VM_HUGEPAGE, vma->vm_mm) && hugepage_enabled() - && collapse_possible(vma, vm_flags, TVA_KHUGEPAGED)) + if (!mm_flags_test(MMF_VM_HUGEPAGE, vma->vm_mm) && hugepage_enabled() && + collapse_possible_orders(vma, vm_flags, TVA_KHUGEPAGED)) __khugepaged_enter(vma->vm_mm); } =20 @@ -2864,7 +2858,8 @@ static void collapse_scan_mm_slot(unsigned int progre= ss_max, cc->progress++; break; } - if (!collapse_possible(vma, vma->vm_flags, TVA_KHUGEPAGED)) { + if (!collapse_possible_orders(vma, vma->vm_flags, + TVA_KHUGEPAGED)) { cc->progress++; continue; } @@ -3177,7 +3172,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, BUG_ON(vma->vm_start > start); BUG_ON(vma->vm_end < end); =20 - if (!collapse_possible(vma, vma->vm_flags, TVA_FORCED_COLLAPSE)) + if (!collapse_possible_orders(vma, vma->vm_flags, TVA_FORCED_COLLAPSE)) return -EINVAL; =20 hstart =3D ALIGN(start, HPAGE_PMD_SIZE); --=20 2.54.0 From nobody Sat Sep 26 03:17:25 2026 Received: from fhigh-a2-smtp.messagingengine.com (fhigh-a2-smtp.messagingengine.com [103.168.172.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4C3274EE876 for ; Fri, 4 Sep 2026 15:10:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.153 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534646; cv=none; b=Ig6tl5x0yYQs/0a/rawi5M5RbcDf1OC+j+P26TlaEq9bTD90dw4O6Pg+nzQ3DhfEDK/LTPyRfl2+syiHG1YUm/JyNGiR4QTcceuP4d+ByOUSmmHQE5YhUpOujQ1t3XTiUSSFrz5fne7d01CG3E9yjbxzcFPjFabCQD+i4+X9ZTU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534646; c=relaxed/simple; bh=5ZEuPx9Q3vDmn9iVMcqwHHzUpAZcyMrU3Y6gQyuzihA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ptxZJu4nHOY9pABw2DiEnYrw2mpbM2AP8XQTPk4xjEPoyGCujSPQEVnhhIOCdRIpNRi1Xy+sr4zT5HnrRnpRe/zBow29VMMDVbCOpx65hRqGFlVJcJL1hlSY6cXwSlWrlwjL7Zii1mPMHb/x9GVxyl0rAc3xESJg7f5ZOpPGmxo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=aiNRlLY0; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=iZKeYrJv; arc=none smtp.client-ip=103.168.172.153 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="aiNRlLY0"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="iZKeYrJv" Received: from phl-compute-04.internal (phl-compute-04.internal [10.202.2.44]) by mailfhigh.phl.internal (Postfix) with ESMTP id 4ED2F1400122; Fri, 4 Sep 2026 11:10:43 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-04.internal (MEProxy); Fri, 04 Sep 2026 11:10:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1788534643; x= 1788621043; bh=DqDmzPS8A0heXGsnD3SSNRWbRweyoQUnesWHCHkvF4Y=; b=a iNRlLY0yFkRuNBNoMOTbkSBg1NEQpfcBCqEzNvxZ35G8wJh8ENf5lzdLKFY5qew3 ecSkTJS8YSzk9BF9e/xqXFOSW7We+/WvuO4qYCj2AoIuR+dh+jr/czK199UXzRmc FEO+dJ2Cb1QgDKLHsT4aibC+HGfUvqjoIMz64fC9vA0rEfDvuQYHd+Jx9Q7WzYoD EqAw62HLDm0esEgxpS+t1ghUMey/kzKjRNBvgH2z2fgX5aLDt3C+iVVnARV37AbQ 4BceyOe6EyHyHWxcT+9AtiwmdiCo/C+JoaMRJX9cpBQhzZ96AKTZ4bSwKWRJQ1dX cdmaIQgPBRlhFTYEvpiEA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1788534643; x=1788621043; bh=D qDmzPS8A0heXGsnD3SSNRWbRweyoQUnesWHCHkvF4Y=; b=iZKeYrJvpGiY/Nr47 wz76ETjHnIlha350S2w8hadghwZQBPIe8uQpDqpCQiA/sGnFZjhyTvUuiXlhVhuA vEkKcRk6EvY5Q8FezNEu/h65F1nmO0IcWZOyM7ifgmeyFUHUEwH+zy9d5A2zQ2O5 EPwjHoM06+cGOrDhOe7XmbMSM/VkYZGg170wG/MVOl0UDoQepb5MDY2v196Z4K/n GQc2zHWlSquhNKXpgcMBpeJMTzvWxHCy5WTAuiZfo8MSgJgbSrS7ISzjN1uwfGva aj60SUzvEf73AsP3T2xlge9SO2eBd5xmjCIezKe1leJFqZ536kM6yIzXqjSeMSd2 Od4gw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTET6cjphBfz4H8Wg4Mwj3Q2fsO4rbzY0RZa6uwdK6ABZ7pEhWaf3DWE0Wh23XozXl YXqBYYUtwtfZ1qQ9wX2bCqZqo/yq027VrzWqESBcvph06bOS8vyiOGOCS0fh2RY7A/kSFI L9MElk4eZ3HddGFY9A2GQHzM4/P7C4mhUDQ1iVikEtpcmuF5W8uFTm0u0CjZbROqFnINk4 xO1zzX/Hj74YRRcHjJyG4ygWf0Hx5RhGZptpxeuVNBh7kD8dz0lfmQnwRvV5yZ57YM54K9 KH8vVIjAAxpTayQfOZqnAJ2Uka0rmt6zQ0IYLz8TzxO+2W2OgPHZMW2lenfH2WoNpxg3WD kFoVBGZvgOEe88v5nMnx07Cy+ZRyXffepGnIEKsGa7dZWyUYqFYxlmHpu/pujCOsKJzCP2 G18G4R9ED1zc3xEMfxQd1aA5krFgZzp0SnlDTDaiszQev8U+2LtqTwJNgLFIxDCoe4a3XF 6ju9R760FF98GTSg3YFQhFvgpzntzkCc1enEyfOroI+8l8scsQ7cXGkotwLuub8q3xzRh3 qP/KCxPHSRHdftib7Az3yS8wv2Cec5t0zf45Weo4/GQ8LlIODtgWu+Fjgo8hyfRskTHX/T YZbFGnP/OYDfQs6JjgDvfI2B+wbeptAoGCnnJluV5+xLZHUd36GfWHdI3Log X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 4 Sep 2026 11:10:42 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn , "Kiryl Shutsemau (Meta)" Subject: [PATCH 07/12] mm/collapse: name the per-table scan reset for what it resets Date: Fri, 4 Sep 2026 16:10:21 +0100 Message-ID: X-Mailer: git-send-email 2.55.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" collapse_control_init_scan() resets what one scan accumulates: the node load, the allocation nodemask and the eligible-PTE bitmap. It runs before every PTE table a scan is given, not once per control, so its name points at the wrong thing. Call it collapse_scan_reset(). Preparation for giving a control a real init and release, run once each for a whole series of scans. Two names a word apart would then stand for two different jobs. No functional change. Assisted-by: Claude-Code:claude-opus-5 Signed-off-by: Kiryl Shutsemau (Meta) Reviewed-by: Baolin Wang Reviewed-by: Zi Yan --- mm/khugepaged.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 22272cfe97e6..511ffb381fe9 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -550,7 +550,7 @@ void __khugepaged_exit(struct mm_struct *mm) } } =20 -static void collapse_control_init_scan(struct collapse_control *cc) +static void collapse_scan_reset(struct collapse_control *cc) { memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); @@ -1579,7 +1579,7 @@ static enum scan_result collapse_scan_pmd(struct mm_s= truct *mm, goto out; } =20 - collapse_control_init_scan(cc); + collapse_scan_reset(cc); =20 enabled_orders =3D collapse_possible_orders(vma, vma->vm_flags, tva_flags= ); =20 @@ -2660,7 +2660,7 @@ static enum scan_result collapse_scan_file(struct mm_= struct *mm, =20 present =3D 0; swap =3D 0; - collapse_control_init_scan(cc); + collapse_scan_reset(cc); rcu_read_lock(); xas_for_each(&xas, folio, start + HPAGE_PMD_NR - 1) { if (xas_retry(&xas, folio)) --=20 2.54.0 From nobody Sat Sep 26 03:17:25 2026 Received: from fhigh-a2-smtp.messagingengine.com (fhigh-a2-smtp.messagingengine.com [103.168.172.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A15F34F0531 for ; Fri, 4 Sep 2026 15:10:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.153 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534648; cv=none; b=EnaIyBcIam6G6nVrgJ4SMcLkcMRI91jLb+4wHpyTw3BlkLAl62jWBkFq+BxzceZ4AHp+B7j8jboQ+IDGalL77+HPEvOFNQiClX8EU03FsbNnx2W99tfDsTYAGA/AERwwvEAJWnIeo57RwYnl8K/sYwE2BYkSZhLpcjMY49mfwrM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534648; c=relaxed/simple; bh=+Cj83DNNCelBx+9Z3BZjXb5ZzXg4eTZtKEyCQOW558M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DSvMR8vfUS5DddI7xYPCaTICPt9yEhrZHRsEXv9xmNsXhBgm+s4ZwOM09ZCaH/bYLyE7mmljBT65gprEy8AgKOJVUWsr17N9zliXaX2Wb8hqGC1tg3uSPpcULNVSSBc2PD9I7ULn+fBky6MBUwi/XTMPaYbZExQt98tssMad560= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=e16hbKIp; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=SKTzj4/4; arc=none smtp.client-ip=103.168.172.153 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="e16hbKIp"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="SKTzj4/4" Received: from phl-compute-01.internal (phl-compute-01.internal [10.202.2.41]) by mailfhigh.phl.internal (Postfix) with ESMTP id EA54E14000EA; Fri, 4 Sep 2026 11:10:44 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-01.internal (MEProxy); Fri, 04 Sep 2026 11:10:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1788534644; x= 1788621044; bh=r93rUlDolzw1y/GkO8pLufE2QiyunUPjQLxePIB9XW4=; b=e 16hbKIphh+Ow3R7n0z+kJKD9ONUcVEFDCOGO2E8vrb2+imltlrO3ZhsCKXNPWqgR 7t1oGPD+d6GWvatfWUgRTQl1YvqEEtCj05McCzMhto5h8BkBnx1pJJ18OkincXRF FFHWckDj7ojS1nF6KX/cQRbsAruYHmm2jy4LutA4RJITCMu2fOAV3o/jkzdp5krm zhlQuYCsFyQD8aQ8DA5I0tDFmPnmHO8KEV/WEMoCqpZyUgk7dJ4KJSie1OA9GZIL lL9Gi4riD1jtYuU+tbFhCbUBijquLPLJCKdzre9Qw++W6tSNOb84dc+2Palhop1L mfPAwYKMpdXeavMKG9RRg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1788534644; x=1788621044; bh=r 93rUlDolzw1y/GkO8pLufE2QiyunUPjQLxePIB9XW4=; b=SKTzj4/4U68GjSc+o Hi5BEucItBtyQ69SNx773bFJp5k0BkWpJui5LB1oLLE0Puz7qfWuxAfe38rc/oAs KcHEvUq6fxJ8YXJlzH3lWtUACOaQJb0vdEDaMWAGUhvoduDu7BDVipipdsV8WJYG CT2phgyWyWLyNi+zUUJmOaQfCRgywLud1aD5K18LdJ+8JJaYZu69LGEoERyl3ElP APFGLwSaEJnIiF6pnpZYPZq9Ud3hsHc0sGPPLWiU+2iFypA5CpjA6U4KDpJaklOE 98vuCMre3jUuV4Prttan7KiAS5WIGCuOXtBcOhyDbFmMv1g7geuE2/QhMIDQOoxB +Ynrw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTET6cjphBfz4H8Wg4Mwj3Q2fsO4rbzY0RZa6uwdK6ABZ7pEhWaf3DWE0Wh23XozXl YXqBYYUtwtfZ1qQ9wX2bCqZqo/yq027VrzWqESBcvph06bOS8vyiOGOCS0fh2RY7A/kSFI L9MElk4eZ3HddGFY9A2GQHzM4/P7C4mhUDQ1iVikEtpcmuF5W8uFTm0u0CjZbROqFnINk4 xO1zzX/Hj74YRRcHjJyG4ygWf0Hx5RhGZptpxeuVNBh7kD8dz0lfmQnwRvV5yZ57YM54K9 KH8vVIjAAxpTayQfOZqnAJ2Uka0rmt6zQ0IYLz8TzxO+2W2OgPHZMW2lenfH2WoNpxg3Fu Is3rPBrDcBofCc7b6n/mkzPMHKzzFB4wzBYnbd1msgXXEWv5pcCEVAkkpXV9YTA1f+AM90 fYGiwvFVPfibntr1gJm7N6PIziBa7N46Lwg7LpYrTxie3Cplo8XWo5c5qbj8fii7o4EfVR zfSbEJevvnyN65ADRgmIr9g+rY4WZTz40ZJjjzhrz6KlTzdXqInEhXD/5UgvIKmrLiG/Ei WEbYbLJho2MkPL6L3dEmJJAHfoOP0bVno1RgvPqFSrXsgRxEx+FsUvVjz0Lo15IVXh8/9y v+lUAhRzWHZNhw1wTWJu6ajdwVgYkVDa1b2c/L4M/R8ruWXpUUg6Ab5PYxIA X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 4 Sep 2026 11:10:44 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn , "Kiryl Shutsemau (Meta)" Subject: [PATCH 08/12] mm/collapse: separate scanning a PTE table from collapsing it Date: Fri, 4 Sep 2026 16:10:22 +0100 Message-ID: <1a1bc537850bd7ef73bed5ac4985634bb8dd95e1.1788533997.git.kas@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" A collapse is two jobs. One reads a PTE table under mmap_lock and decides whether the range is worth collapsing. The other allocates, isolates, copies and flushes, and wants the lock given up first. collapse_single_pmd() did both, so the boundary between them was somewhere in the middle of a function. Give each half its own function: - collapse_scan_pmd() scans one table and only reads. The anonymous scan that used to carry that name keeps its body as collapse_scan_anon_pmd(), and collapse_scan_pmd() is now the entry that picks the anonymous or the file side. - collapse_run_pmd() does the collapse the scan asked for. SCAN_SUCCEED from the scan means there is something to run; anything else is why there is not. collapse_single_pmd() is now the two of them with the mmap_lock drop in between, so its callers see what they saw before. Scan results (beyond SCAN_SUCCEED) communicated via collapse_control structure: the orders, the referenced and swapped-out counts, and for a file the file itself and the offset in it. A file collapse works on the page cache and never sees a VMA. The scan takes the file reference while it still has VMA and the run unpins it when it is done. Tracing changes with it. mm_khugepaged_scan_pmd now fires before mm_collapse_huge_page instead of after it. Its status field already reads SCAN_SUCCEED for an accepted table, so what the collapse then made of that table is mm_collapse_huge_page's to report, per order. The two calls to that tracepoint become one. They differed in what the collapse between them changed; with the collapse no longer here, both carry the same arguments. failed_pfn is set only where a PTE was refused, so it is -1 exactly when the result is SCAN_SUCCEED. Assisted-by: Claude-Code:claude-opus-5 Signed-off-by: Kiryl Shutsemau (Meta) --- mm/collapse.h | 14 ++++++ mm/khugepaged.c | 121 ++++++++++++++++++++++++++++++++++-------------- 2 files changed, 100 insertions(+), 35 deletions(-) diff --git a/mm/collapse.h b/mm/collapse.h index 05282eed9a35..f03cad8ed40e 100644 --- a/mm/collapse.h +++ b/mm/collapse.h @@ -100,6 +100,20 @@ struct collapse_control { =20 /* Each bit marks a PTE the scan accepted as a collapse source */ DECLARE_BITMAP(eligible_ptes, MAX_PTRS_PER_PTE); + + /* + * What a scan found and the run after it needs. Live only between the + * two, and read by nobody else. + * + * The file side takes a reference while it still has the VMA, since a + * file collapse works on the page cache and never sees one; the run is + * what gives it back. + */ + unsigned long scan_orders; + int scan_referenced; + int scan_unmapped; + struct file *scan_file; + pgoff_t scan_pgoff; }; =20 #endif /* __MM_COLLAPSE_H */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 511ffb381fe9..120af57540df 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1550,14 +1550,14 @@ static enum scan_result mthp_collapse(struct mm_str= uct *mm, return last_result; } =20 -static enum scan_result collapse_scan_pmd(struct mm_struct *mm, - struct vm_area_struct *vma, unsigned long start_addr, - bool *lock_dropped, struct collapse_control *cc) +static enum scan_result collapse_scan_anon_pmd(struct vm_area_struct *vma, + unsigned long start_addr, struct collapse_control *cc) { const unsigned int max_ptes_shared =3D collapse_max_ptes_shared(cc, HPAGE= _PMD_ORDER); const unsigned int max_ptes_swap =3D collapse_max_ptes_swap(cc, HPAGE_PMD= _ORDER); unsigned int max_ptes_none =3D collapse_max_ptes_none(cc, vma, HPAGE_PMD_= ORDER); enum tva_type tva_flags =3D cc->policy.tva_type; + struct mm_struct *mm =3D vma->vm_mm; pmd_t *pmd; pte_t *pte, *_pte, pteval; int i; @@ -1737,19 +1737,17 @@ static enum scan_result collapse_scan_pmd(struct mm= _struct *mm, out_unmap: pte_unmap_unlock(pte, ptl); if (result =3D=3D SCAN_SUCCEED) { - /* collapse_huge_page() expects the lock to be dropped before calling */ - mmap_read_unlock(mm); - result =3D mthp_collapse(mm, start_addr, referenced, - unmapped, cc, enabled_orders); - /* mmap_lock was released above, set lock_dropped */ - *lock_dropped =3D true; - trace_mm_khugepaged_scan_pmd(mm, -1, referenced, none_or_zero, - SCAN_SUCCEED, unmapped); - } else { -out: - trace_mm_khugepaged_scan_pmd(mm, failed_pfn, referenced, - none_or_zero, result, unmapped); + cc->scan_orders =3D enabled_orders; + cc->scan_referenced =3D referenced; + cc->scan_unmapped =3D unmapped; } +out: + /* + * failed_pfn is only set where a PTE was refused, so it is -1 on the + * path that returns SCAN_SUCCEED. + */ + trace_mm_khugepaged_scan_pmd(mm, failed_pfn, referenced, + none_or_zero, result, unmapped); return result; } =20 @@ -2759,30 +2757,58 @@ static enum scan_result collapse_scan_file(struct m= m_struct *mm, return result; } =20 -/* - * Try to collapse a single PMD starting at a PMD aligned addr, and return - * the results. - */ -static enum scan_result collapse_single_pmd(unsigned long addr, - struct vm_area_struct *vma, bool *lock_dropped, - struct collapse_control *cc) +static void collapse_control_init(struct collapse_control *cc) { - struct mm_struct *mm =3D vma->vm_mm; - bool triggered_wb =3D false; - enum scan_result result; - struct file *file; - pgoff_t pgoff; + cc->progress =3D 0; + cc->scan_file =3D NULL; +} =20 - mmap_assert_locked(mm); +static void collapse_control_release(struct collapse_control *cc) +{ + /* A scan that took a file reference should have been run */ + if (WARN_ON_ONCE(cc->scan_file)) { + fput(cc->scan_file); + cc->scan_file =3D NULL; + } +} + +static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, + unsigned long addr, struct collapse_control *cc) +{ + mmap_assert_locked(vma->vm_mm); + /* Whatever the last scan found has to have been run by now */ + if (WARN_ON_ONCE(cc->scan_file)) { + fput(cc->scan_file); + cc->scan_file =3D NULL; + } =20 if (vma_is_anonymous(vma)) - return collapse_scan_pmd(mm, vma, addr, lock_dropped, cc); + return collapse_scan_anon_pmd(vma, addr, cc); =20 - file =3D get_file(vma->vm_file); - pgoff =3D linear_page_index(vma, addr); + /* + * A file collapse works on the page cache and never sees a VMA, so take + * what it needs from this one while it is still here. Judging the + * range needs the page cache and no lock, so it happens in the run. + */ + cc->scan_file =3D get_file(vma->vm_file); + cc->scan_pgoff =3D linear_page_index(vma, addr); + return SCAN_SUCCEED; +} =20 - mmap_read_unlock(mm); - *lock_dropped =3D true; +static enum scan_result collapse_run_pmd(struct mm_struct *mm, + unsigned long addr, struct collapse_control *cc) +{ + struct file *file =3D cc->scan_file; + bool triggered_wb =3D false; + enum scan_result result; + pgoff_t pgoff; + + if (!file) + return mthp_collapse(mm, addr, cc->scan_referenced, + cc->scan_unmapped, cc, cc->scan_orders); + + cc->scan_file =3D NULL; + pgoff =3D cc->scan_pgoff; retry: result =3D collapse_scan_file(mm, addr, file, pgoff, cc); =20 @@ -2812,6 +2838,28 @@ static enum scan_result collapse_single_pmd(unsigned= long addr, return result; } =20 +/* + * Try to collapse a single PMD starting at a PMD aligned addr, and return + * the results. + */ +static enum scan_result collapse_single_pmd(unsigned long addr, + struct vm_area_struct *vma, bool *lock_dropped, + struct collapse_control *cc) +{ + struct mm_struct *mm =3D vma->vm_mm; + enum scan_result result; + + result =3D collapse_scan_pmd(vma, addr, cc); + if (result !=3D SCAN_SUCCEED) + return result; + + /* The collapse takes its own locks, so give this up */ + mmap_read_unlock(mm); + *lock_dropped =3D true; + + return collapse_run_pmd(mm, addr, cc); +} + static void collapse_scan_mm_slot(unsigned int progress_max, enum scan_result *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock) @@ -2954,10 +3002,10 @@ static void khugepaged_do_scan(struct collapse_cont= rol *cc) =20 lru_add_drain_all(); =20 + collapse_control_init(cc); /* One policy for the whole pass, so every table is judged the same */ collapse_policy_khugepaged(&cc->policy); =20 - cc->progress =3D 0; while (true) { cond_resched(); =20 @@ -2988,6 +3036,8 @@ static void khugepaged_do_scan(struct collapse_contro= l *cc) khugepaged_alloc_sleep(); } } + + collapse_control_release(cc); } =20 static bool khugepaged_should_wakeup(void) @@ -3184,8 +3234,8 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, cc =3D kmalloc_obj(*cc); if (!cc) return -ENOMEM; + collapse_control_init(cc); collapse_policy_forced(&cc->policy); - cc->progress =3D 0; =20 lru_add_drain_all(); =20 @@ -3242,6 +3292,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, } out_nolock: mmap_assert_locked(mm); + collapse_control_release(cc); kfree(cc); =20 return thps =3D=3D ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0 --=20 2.54.0 From nobody Sat Sep 26 03:17:25 2026 Received: from fhigh-a2-smtp.messagingengine.com (fhigh-a2-smtp.messagingengine.com [103.168.172.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D511D4EBAEC for ; Fri, 4 Sep 2026 15:10:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.153 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534650; cv=none; b=JLBZ3w87MbEt3RwySgJEM/OH1e5E0mAnjbEvSHEefbFaxCKxLurHkIunIHlYPaLZP1cgkTI+peXlg4y9AOxF/0TDvUx9SvNMqb1D6ciLbtoSJY2Hy6VkCWFgV63WomCJFOGpvAZtzQgJcabkWTLn7LxpZeq04j9d1vRgkB9xNbI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534650; c=relaxed/simple; bh=V6LLwo/Ivv2nqLpKd8bqd36JsaVkaIiNjXCUF6YhOvg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GTfwGFCO7h0tDylDAcQwAmCuIN7n63NSRn9nmSwZ5HYq0VhrZ6TB8CDbA069943PFDeKnoyZ4RtQp5p2m3I+MLTC3ghm29/AzXAV+zK+0ovGcQ1im4jalskjugOnLzOkdejsPDNr6IFvTus9mByQ6CPa3c2+/xX+66Tj10T7JQQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=CdKHiZyr; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=MqMsJWDE; arc=none smtp.client-ip=103.168.172.153 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="CdKHiZyr"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="MqMsJWDE" Received: from phl-compute-07.internal (phl-compute-07.internal [10.202.2.47]) by mailfhigh.phl.internal (Postfix) with ESMTP id CAC651400112; Fri, 4 Sep 2026 11:10:46 -0400 (EDT) Received: from phl-frontend-03 ([10.202.2.162]) by phl-compute-07.internal (MEProxy); Fri, 04 Sep 2026 11:10:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1788534646; x= 1788621046; bh=/nQCmOK93ZfCpG7MK6pnmh6In4f0JqXwsy1QQOqdTZ8=; b=C dKHiZyrzPXIps9mN6Qg7ZWt9rYjOQOTeYliRUbwc/M+W+2PHBBYT3l5kxMvvUCK0 QQp03m5WWb2JbV55xWX7db34PbO1CE7MDVLC1nL6FAe2K99KRj74Xzr4U6zNJPzZ YnqaeOzmGMXzLavDe4bpcegdEJ7ai11CC1c/j/IMXLf8dJ1XDH/HSf0Iae2YiCZ/ cQnxFnqHW+5mEkfcN0x0NaXwX126pthldlLLtBaZwJe7bmWgAQ95daJKYG55PTTK fQqLArCJdRXDz39bpIBttkrDK+OUBdGuyItKplKtVxwewPAAc6AsZD831U/D/jBQ LDPlKQqNOzA6BQfRMGicQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1788534646; x=1788621046; bh=/ nQCmOK93ZfCpG7MK6pnmh6In4f0JqXwsy1QQOqdTZ8=; b=MqMsJWDE7NQ/l8FT7 3vWLoZECCXH4g7lgrfDtnIGu7KBYAgrsLaXFoo4e+fHN8JpBi+PNYy6buj/6gq1K RH0P7Z+jbr9Q4Kz3IQlchNBZh7dh6X8CVIv81Y/zglLstLJa7cLDljEXdiEITPWM TX27VGRWzm740JBj5yOPHiOeE1tt6gqQZjbxuKPZ253NYryXzQMdPJYmTe6itb63 by2x61iM2so2vdpdvBZxqCZjDQdpQFPlCvCoraUsAJSeSuV7j1h+CS/fJ8nFmpRq pKNSN2AFj41RmTyIErk8OCmX9clFnNeOAPAf1kzgXdpOl4yr6oU9povzGmf5LWrN Zz+Rw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFdOocfRN4jOpcQf0AqfSai/YCGo+xlO94MLvzn7lqitof50EENUNEgJzLPt2H+RQ DWgrwJL5dg2BaLkWQfjz0w8mX09EzID7A24sM1R0CcrzJ0aCsxP+pgO1accdG5RnvXKoNP Qi2+ranCgsFBQlrx3RUM64Xgm9l9zr2TPWbzFNtan3mUwkVlJ/gBEGXolzLuuMZ3CJ6e8L gmzum6oDDF3av5ecJE/wINba64+XlYa/U9oL8c0Dkr16eu3wp8w8wOsHTRTsCgO4oV0meO 0fb7rtZliPO8XPlA3k/rGcTv+01JWMnSe6fG3OACTE4UctUjoP9cM2YnBbUckeF0hhckhT mcMOmV1mEg5NTO+Vk1Ef1mEuAvCQJhoLl/JdUrkcZSNk4dRWNiyxq5mrUlNNzYFkA21dlA 0jFgqxhYxnVwlrsjm3rpTztYN+kKkNf4CIO61J2MJ1HoVSooKtsQ3znJA2hOepZl/kxg5Y JbljyLOXRzfOX0SZkLijl6kXPnOt637ny3/enMKBFaaTEsPYvoUAwkZPUNQs5DPbf1UcQM yndtfQTRsA+XhGFed8nRmDe7Au45EKl11FbNdjzZZo4CukdX5gtqcd11A05vO/uH6754sb Zd5k5cbMDa7FhGGCuFwsZ6NHb65p0kdvShdZHdsDtfNA39yXh1fCoVgBSxbQ X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 4 Sep 2026 11:10:45 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn , "Kiryl Shutsemau (Meta)" Subject: [PATCH 09/12] mm/collapse: open-code collapse_single_pmd() in its two callers Date: Fri, 4 Sep 2026 16:10:23 +0100 Message-ID: <94aed028401ac48759eb7d056e451dbe50c1ea32.1788533997.git.kas@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" collapse_scan_pmd() and collapse_run_pmd() each have a clear locking contract. The scan is called with mmap_lock held for reading and returns with it still held. The collapse is called without it. collapse_single_pmd() kept that boundary inside itself. It dropped the lock on some paths and not others, and reported which by way of a bool its callers had to carry along and then act on. Open-code it in the two callers. Each scans under the lock it already holds and, on SCAN_SUCCEED, gives the lock up before running the collapse. khugepaged's lock_dropped and madvise_collapse()'s mmap_unlocked both go: the code dropping the lock is now the code that wanted to know. khugepaged's walk carries on to the next table while the scan keeps refusing, and ends once a collapse has taken the lock from under it. madvise_collapse() re-finds its VMA after a collapse, which it did before, and now uses a NULL vma to say that it has to. It still reports the drop to its own caller, from the line that does it. The lock is given up and taken again at the same points as before. No functional change. Assisted-by: Claude-Code:claude-opus-5 Signed-off-by: Kiryl Shutsemau (Meta) --- mm/khugepaged.c | 102 +++++++++++++++++++++++------------------------- 1 file changed, 49 insertions(+), 53 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 120af57540df..40fcdd4f2712 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -2838,28 +2838,6 @@ static enum scan_result collapse_run_pmd(struct mm_s= truct *mm, return result; } =20 -/* - * Try to collapse a single PMD starting at a PMD aligned addr, and return - * the results. - */ -static enum scan_result collapse_single_pmd(unsigned long addr, - struct vm_area_struct *vma, bool *lock_dropped, - struct collapse_control *cc) -{ - struct mm_struct *mm =3D vma->vm_mm; - enum scan_result result; - - result =3D collapse_scan_pmd(vma, addr, cc); - if (result !=3D SCAN_SUCCEED) - return result; - - /* The collapse takes its own locks, so give this up */ - mmap_read_unlock(mm); - *lock_dropped =3D true; - - return collapse_run_pmd(mm, addr, cc); -} - static void collapse_scan_mm_slot(unsigned int progress_max, enum scan_result *result, struct collapse_control *cc) __releases(&khugepaged_mm_lock) @@ -2922,7 +2900,7 @@ static void collapse_scan_mm_slot(unsigned int progre= ss_max, VM_BUG_ON(khugepaged_scan.address & ~HPAGE_PMD_MASK); =20 while (khugepaged_scan.address < hend) { - bool lock_dropped =3D false; + unsigned long addr; =20 cond_resched(); if (unlikely(collapse_test_exit_or_disable(mm))) @@ -2932,23 +2910,29 @@ static void collapse_scan_mm_slot(unsigned int prog= ress_max, khugepaged_scan.address + HPAGE_PMD_SIZE > hend); =20 - *result =3D collapse_single_pmd(khugepaged_scan.address, - vma, &lock_dropped, cc); - if (*result =3D=3D SCAN_SUCCEED) - khugepaged_pages_collapsed++; + addr =3D khugepaged_scan.address; /* move to next address */ khugepaged_scan.address +=3D HPAGE_PMD_SIZE; - if (lock_dropped) - /* - * We released mmap_lock so break loop. Note - * that we drop mmap_lock before all hugepage - * allocations, so if allocation fails, we are - * guaranteed to break here and report the - * correct result back to caller. - */ - goto breakouterloop_mmap_lock; - if (cc->progress >=3D progress_max) - goto breakouterloop; + + *result =3D collapse_scan_pmd(vma, addr, cc); + /* Nothing to collapse here, and the lock is still ours */ + if (*result !=3D SCAN_SUCCEED) { + if (cc->progress >=3D progress_max) + goto breakouterloop; + continue; + } + + /* + * A collapse takes its own locks and is slow enough + * that a writer should not wait behind it, so give the + * lock up. That ends this walk: vma and the mm are + * whatever the collapse leaves them. + */ + mmap_read_unlock(mm); + *result =3D collapse_run_pmd(mm, addr, cc); + if (*result =3D=3D SCAN_SUCCEED) + khugepaged_pages_collapsed++; + goto breakouterloop_mmap_lock; } } breakouterloop: @@ -3217,7 +3201,6 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, unsigned long hstart, hend, addr; enum scan_result last_fail =3D SCAN_FAIL; int thps =3D 0; - bool mmap_unlocked =3D false; =20 BUG_ON(vma->vm_start > start); BUG_ON(vma->vm_end < end); @@ -3240,25 +3223,40 @@ int madvise_collapse(struct vm_area_struct *vma, un= signed long start, lru_add_drain_all(); =20 for (addr =3D hstart; addr < hend; addr +=3D HPAGE_PMD_SIZE) { - enum scan_result result =3D SCAN_FAIL; + struct vm_area_struct *found; + enum scan_result result; =20 - if (mmap_unlocked) { + /* + * A collapse gives the lock up, so the VMA has to be found + * again after one: it can shrink while nothing is held. A scan + * that finds nothing to collapse leaves the lock alone, so a + * range that is already collapsed walks on without relocking. + */ + if (!vma) { cond_resched(); mmap_read_lock(mm); - mmap_unlocked =3D false; - *lock_dropped =3D true; - result =3D hugepage_vma_revalidate(mm, addr, false, &vma, + result =3D hugepage_vma_revalidate(mm, addr, false, &found, cc, HPAGE_PMD_ORDER); if (result !=3D SCAN_SUCCEED) { last_fail =3D result; - goto out_nolock; + goto out_locked; } - + vma =3D found; hend =3D min(hend, vma->vm_end & HPAGE_PMD_MASK); } =20 - result =3D collapse_single_pmd(addr, vma, &mmap_unlocked, cc); + result =3D collapse_scan_pmd(vma, addr, cc); + /* Nothing to collapse here, and the lock is still ours */ + if (result !=3D SCAN_SUCCEED) + goto tally; =20 + /* The collapse takes its own locks, so give this up */ + mmap_read_unlock(mm); + *lock_dropped =3D true; + vma =3D NULL; + + result =3D collapse_run_pmd(mm, addr, cc); +tally: switch (result) { case SCAN_SUCCEED: case SCAN_PMD_MAPPED: @@ -3280,17 +3278,15 @@ int madvise_collapse(struct vm_area_struct *vma, un= signed long start, default: last_fail =3D result; /* Other error, exit */ - goto out_maybelock; + goto out; } } =20 -out_maybelock: +out: /* Caller expects us to hold mmap_lock on return */ - if (mmap_unlocked) { - *lock_dropped =3D true; + if (!vma) mmap_read_lock(mm); - } -out_nolock: +out_locked: mmap_assert_locked(mm); collapse_control_release(cc); kfree(cc); --=20 2.54.0 From nobody Sat Sep 26 03:17:25 2026 Received: from fout-a1-smtp.messagingengine.com (fout-a1-smtp.messagingengine.com [103.168.172.144]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E2D5F4F0548 for ; Fri, 4 Sep 2026 15:10:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.144 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534652; cv=none; b=pQaABSaUU0KD2+USgXWNTxXOAI5mgBZ5C6/1J95LhlNGAwnTVOWAMgyyvohj6jJclOEDZmC9Nxn7I7SgfTs8fQ99d7ojL36cybkAF+Nwgfi5PdiEzvaHT+zyxhjsjz7S4q4oJJfZt/S6XbZefhYE9y6g0kcKCWhmLjql4SWsYPo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534652; c=relaxed/simple; bh=EXj9gHKKDroVMAa1tybjktoiUdVkhyiBRRLuz18g5RU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TaQLBYLKL6TB/wuCNSr+EJqylPuLgFAxR2MtYoZfE+aCD/Sl87ERBJG+Q1PVqONEv6g4BhoNMm5A0je88mfIblqn5GHnKfyBrjv8tIIMgu7IkhX7Tp4RIDV2WMu3OYZ5dU7ZFg1HBza93+xXCQgMpc7YfJd6lDImEJafcambUo0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=bIU4zTkX; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=d9Cl+Dxs; arc=none smtp.client-ip=103.168.172.144 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="bIU4zTkX"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="d9Cl+Dxs" Received: from phl-compute-12.internal (phl-compute-12.internal [10.202.2.52]) by mailfout.phl.internal (Postfix) with ESMTP id 7C86DEC017D; Fri, 4 Sep 2026 11:10:48 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-12.internal (MEProxy); Fri, 04 Sep 2026 11:10:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1788534648; x= 1788621048; bh=Ma/0lPtOSoslW4rYZfqjpQhzgHNyjT+wt2R0Nf4TT/U=; b=b IU4zTkXENwyseuVnQjybnGe65B69eUWUOwK8UJXoIIsALtveXfw1vPCstx4U5KPM GQoU2CCdLR8vdGxIf7r/t3AkKFWCQma4lgGk30jtY/wLZ3zS/hDnDGw0WJUlaajf EgrIOdClltmhZxUB6D9HslXuTO470f079WRJg4coEJtnU/2aE9rBi+maKi0tFQGW G8H8oUlItqn/b/YcbC+AosHawuRdUsDkdo8vhrg08zh5bPrAfyK7VUglzvxmHiwO TmlD37/1VDx3Eobfl6f+n7VsyOobOmsgIQF7AD9FCqZ4lLsdwVhJiF+YvlQ0rn+A dN235UZfb5knGo4AiPSGg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1788534648; x=1788621048; bh=M a/0lPtOSoslW4rYZfqjpQhzgHNyjT+wt2R0Nf4TT/U=; b=d9Cl+DxsYmUg1Ee/7 4PDuV1s/RgmbFlB6XysOAJBzkDbJSvkg9qYYohZlX8RqfZ1h2z3/jMrdBQevGYVq z20uVUAnCqcB2GWw3KqxhIR9yNpiWhLlQxHrtskUGtCPJe+43+yLM8DZONRMO14u zWuYnwuYQvre/y0hblzZB8+DBtGZX35pQIvjAClFVQEXLuHRpPZ68y/6Avb72FwP OUeyo/txuMrXCZspIDRc10Z1RNQpTHgGZi8CbRxJsEzydRrgS+AkorXpKZ4qu7Q1 gr9dBbJKZaudFvLgrXicV71E9B/hc21det3soxqDFaLiZONw0lxJib3S28x5nhbZ ekF4Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTET6cjphBfz4H8Wg4Mwj3Q2fsO4rbzY0RZa6uwdK6ABZ7pEhWaf3DWE0Wh23XozXl YXqBYYUtwtfZ1qQ9wX2bCqZqo/yq027VrzWqESBcvph06bOS8vyiOGOCS0fh2RY7A/kSFI L9MElk4eZ3HddGFY9A2GQHzM4/P7C4mhUDQ1iVikEtpcmuF5W8uFTm0u0CjZbROqFnINk4 xO1zzX/Hj74YRRcHjJyG4ygWf0Hx5RhGZptpxeuVNBh7kD8dz0lfmQnwRvV5yZ57YM54K9 KH8vVIjAAxpTayQfOZqnAJ2Uka0rmt6zQ0IYLz8TzxO+2W2OgPHZMW2lenfH2WoNpxg3sE 8WblxGqVLg0Bt0AT1lOybodMq55HcpaAOi4fb66mrBF11B+c0n5/ojcCMLj8yggFMpmyba xujKgPZLnfVEWUbWBZx1XQ7s2f6iC/rOAs1I7ckFmM651lWFpY8JYD0q0ZxqV3OL/Ru7ZZ ECq6jlG1PNY6qNadUsT5HJ3ABpnq0T1BEeV4Hayg3rBdK0Rowfn9nEkDsoUnNpqY8SMj3D fC05ebHO439F/wozaYoHJF11vPLoaSBLsqYTnkgdVjYcNpY4ZvJTeifXJA3WIJfdvsvVu6 cv46LP2/YfqvlXGYPih0MAwTB3Aiarl5KveB17/HJsdLGUkRo5vmp5NbosNA X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 4 Sep 2026 11:10:47 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn , "Kiryl Shutsemau (Meta)" Subject: [PATCH 10/12] mm/collapse: work out the orders a VMA allows once per VMA Date: Fri, 4 Sep 2026 16:10:24 +0100 Message-ID: <3ce85bb56f2bc60f91bf4e9645f247460f3c3f1e.1788533997.git.kas@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" The scan asked collapse_possible_orders() for every PTE table, for an answer that is a property of the VMA. Both callers walk a VMA a table at a time, so let them work it out once and pass the mask in. It is only good while the lock that produced it is held, so madvise_collapse() takes it again after every collapse. The mask is then sampled once per VMA rather than once per table. A thp enabled knob written during a walk takes effect one VMA later, and cannot widen a collapse: hugepage_vma_revalidate() tests the order again under the lock the collapse retakes. Assisted-by: Claude-Code:claude-opus-5 Signed-off-by: Kiryl Shutsemau (Meta) --- mm/khugepaged.c | 32 ++++++++++++++++++-------------- 1 file changed, 18 insertions(+), 14 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 40fcdd4f2712..f862abb1dbbd 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1551,12 +1551,12 @@ static enum scan_result mthp_collapse(struct mm_str= uct *mm, } =20 static enum scan_result collapse_scan_anon_pmd(struct vm_area_struct *vma, - unsigned long start_addr, struct collapse_control *cc) + unsigned long start_addr, struct collapse_control *cc, + unsigned long enabled_orders) { const unsigned int max_ptes_shared =3D collapse_max_ptes_shared(cc, HPAGE= _PMD_ORDER); const unsigned int max_ptes_swap =3D collapse_max_ptes_swap(cc, HPAGE_PMD= _ORDER); unsigned int max_ptes_none =3D collapse_max_ptes_none(cc, vma, HPAGE_PMD_= ORDER); - enum tva_type tva_flags =3D cc->policy.tva_type; struct mm_struct *mm =3D vma->vm_mm; pmd_t *pmd; pte_t *pte, *_pte, pteval; @@ -1567,7 +1567,6 @@ static enum scan_result collapse_scan_anon_pmd(struct= vm_area_struct *vma, struct folio *folio =3D NULL; unsigned long failed_pfn =3D -1; unsigned long addr; - unsigned long enabled_orders; spinlock_t *ptl; int node =3D NUMA_NO_NODE, unmapped =3D 0; =20 @@ -1581,8 +1580,6 @@ static enum scan_result collapse_scan_anon_pmd(struct= vm_area_struct *vma, =20 collapse_scan_reset(cc); =20 - enabled_orders =3D collapse_possible_orders(vma, vma->vm_flags, tva_flags= ); - /* * If PMD is the only enabled order, enforce max_ptes_none, otherwise * scan all pages to populate the bitmap for mTHP collapse. The bitmap @@ -2773,7 +2770,8 @@ static void collapse_control_release(struct collapse_= control *cc) } =20 static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, - unsigned long addr, struct collapse_control *cc) + unsigned long addr, struct collapse_control *cc, + unsigned long orders) { mmap_assert_locked(vma->vm_mm); /* Whatever the last scan found has to have been run by now */ @@ -2783,7 +2781,7 @@ static enum scan_result collapse_scan_pmd(struct vm_a= rea_struct *vma, } =20 if (vma_is_anonymous(vma)) - return collapse_scan_anon_pmd(vma, addr, cc); + return collapse_scan_anon_pmd(vma, addr, cc, orders); =20 /* * A file collapse works on the page cache and never sees a VMA, so take @@ -2877,15 +2875,17 @@ static void collapse_scan_mm_slot(unsigned int prog= ress_max, =20 vma_iter_init(&vmi, mm, khugepaged_scan.address); for_each_vma(vmi, vma) { - unsigned long hstart, hend; + unsigned long hstart, hend, orders; =20 cond_resched(); if (unlikely(collapse_test_exit_or_disable(mm))) { cc->progress++; break; } - if (!collapse_possible_orders(vma, vma->vm_flags, - TVA_KHUGEPAGED)) { + /* One mask for the whole VMA */ + orders =3D collapse_possible_orders(vma, vma->vm_flags, + cc->policy.tva_type); + if (!orders) { cc->progress++; continue; } @@ -2914,7 +2914,7 @@ static void collapse_scan_mm_slot(unsigned int progre= ss_max, /* move to next address */ khugepaged_scan.address +=3D HPAGE_PMD_SIZE; =20 - *result =3D collapse_scan_pmd(vma, addr, cc); + *result =3D collapse_scan_pmd(vma, addr, cc, orders); /* Nothing to collapse here, and the lock is still ours */ if (*result !=3D SCAN_SUCCEED) { if (cc->progress >=3D progress_max) @@ -3198,14 +3198,16 @@ int madvise_collapse(struct vm_area_struct *vma, un= signed long start, { struct collapse_control *cc; struct mm_struct *mm =3D vma->vm_mm; - unsigned long hstart, hend, addr; + unsigned long hstart, hend, addr, orders; enum scan_result last_fail =3D SCAN_FAIL; int thps =3D 0; =20 BUG_ON(vma->vm_start > start); BUG_ON(vma->vm_end < end); =20 - if (!collapse_possible_orders(vma, vma->vm_flags, TVA_FORCED_COLLAPSE)) + orders =3D collapse_possible_orders(vma, vma->vm_flags, + TVA_FORCED_COLLAPSE); + if (!orders) return -EINVAL; =20 hstart =3D ALIGN(start, HPAGE_PMD_SIZE); @@ -3243,9 +3245,11 @@ int madvise_collapse(struct vm_area_struct *vma, uns= igned long start, } vma =3D found; hend =3D min(hend, vma->vm_end & HPAGE_PMD_MASK); + orders =3D collapse_possible_orders(vma, vma->vm_flags, + cc->policy.tva_type); } =20 - result =3D collapse_scan_pmd(vma, addr, cc); + result =3D collapse_scan_pmd(vma, addr, cc, orders); /* Nothing to collapse here, and the lock is still ours */ if (result !=3D SCAN_SUCCEED) goto tally; --=20 2.54.0 From nobody Sat Sep 26 03:17:25 2026 Received: from fhigh-a2-smtp.messagingengine.com (fhigh-a2-smtp.messagingengine.com [103.168.172.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 766794EC66B for ; Fri, 4 Sep 2026 15:10:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.153 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534653; cv=none; b=hOVXlgaG4CaZLxyfB4C0iKcq47wAp6lJ6a76QV1rAjaDaEu/NUP29l2HSdnqQvfSz0hUn6y6HAv7JVpQ6PEbpNAlKYxNrVJUkto2ApQxppg8q0BkFx4Mv8zcCiuZorQCu+stQWxNdIZHfudcpYRz/jBlIIaBvJl3rAiqql/+XvI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534653; c=relaxed/simple; bh=bzO8l7iY2sS3LRuVxwWLB8LUyo4/Eq3WtuDcZflS43Q=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TGlHIIlyREzIVVQuPAajwCs6grTAxLrUO4o/7sK35N48uEELYuLT8M1PN/hil+I5+kTMuUUVoVjww9ZsxuusKJIvfBDBH7PY8e8cBiNMFlkz3/oRDxudvAXb9X7VmwVxplpEEn2LeUD0ZSkkyrNenDf+xO/Y2jaChr0WbSvEoOw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=hQ6Tp+Ke; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=U5+vtqbo; arc=none smtp.client-ip=103.168.172.153 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="hQ6Tp+Ke"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="U5+vtqbo" Received: from phl-compute-05.internal (phl-compute-05.internal [10.202.2.45]) by mailfhigh.phl.internal (Postfix) with ESMTP id 605FA1400106; Fri, 4 Sep 2026 11:10:50 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-05.internal (MEProxy); Fri, 04 Sep 2026 11:10:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1788534650; x= 1788621050; bh=IyGYt/hGSRRfip2XU7o0oYq5+nS3sb9WHKUu0hUMGJs=; b=h Q6Tp+KevTAL6klGeElULZhwBqt2vBtH+Rb+By+1KsAo+tGSTxklPVxEYarT6yx+z 7MXi1TiarSEFZpUJDrM27bnCV1WJ2bnWqKzcpjFSoQoBxlCOqn98WTk6ZCUmLSvM +kkugRC82knUvSVjcidxCAZeIL3CZ+MeZZnpnxkV9RkPBh6qOZtUM62USenPj1Kh jBilwgz/krits9LLm1clhkzPHjC2GBwajiXgnBmLtDEh6r4dXB8DHJe/lghz0UYb Fd3x+r5A5kktD006Hy/knALpUdbYtmlw6z2KZu1pbp8geZy1tDQEg/tfdEAtb0hJ SCxcBrPE38ikwcX0dDdRQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1788534650; x=1788621050; bh=I yGYt/hGSRRfip2XU7o0oYq5+nS3sb9WHKUu0hUMGJs=; b=U5+vtqbocwSs8M5p4 OfvVBotzWFoSSF3/iNLmjaoVMz7vvALSsIBriEbS4bTyw+JW3BwpYtUYHLoaqA5p 4xUNhtrKUoN1SS7osx1oMWszSE7fVu6/wByk/ZLPGarmHCeBiKwrViDfnGUvZMCN JPsD5veAROMMJeM+qqLmkM6rimBKFDKyY/3dpztrNLw2i+LEQ2VRDbmmZ8+BDDjz kaT981QAgQccWfJUxSfIeIfR9xbqzSs1gIpYy6Vh2Zt/Lgft+hufZxNUnzLa+1J6 5B2UQdrAyq7QK5X9RJJzXJauI283jq6kH47Vs/tgxksjlL1eegARtlmeyu7lzTl+ cx8kA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTFdOocfRN4jOpcQf0AqfSai/YCGo+xlO94MLvzn7lqitof50EENUNEgJzLPt2H+RQ DWgrwJL5dg2BaLkWQfjz0w8mX09EzID7A24sM1R0CcrzJ0aCsxP+pgO1accdG5RnvXKoNP Qi2+ranCgsFBQlrx3RUM64Xgm9l9zr2TPWbzFNtan3mUwkVlJ/gBEGXolzLuuMZ3CJ6e8L gmzum6oDDF3av5ecJE/wINba64+XlYa/U9oL8c0Dkr16eu3wp8w8wOsHTRTsCgO4oV0meO 0fb7rtZliPO8XPlA3k/rGcTv+01JWMnSe6fG3OACTE4UctUjoP9cM2YnBbUckeF0hhckL5 D6MHrUzf167y2omQL0hymJoFC8UpYyFDv/iBC5j4SYdLZ84BhZnQdV7WTJmUex23TQmx1h gjgtmvz28+cPVK/JOJqAk+MNlvRziAEZ+CTcq0wtB0tp6LhJcm02nOa/ojCMYITHqb24A7 QIqUCFztbGQ+NHmwRoRVMAhkTPbFHhTTCa4PD3iaWyXlHN0H6ZMmAvUd2dPwR4gEgwXbGx uk8dkf2xdbLJbLQfCbfCYZe81/Gzhlg+sy8BCfAmWyKlKsqP3LRm5mQ7PMvMI7NYBG1G65 JdoexBRqSTE0TqifcgFHhWz9ADIfHMm6UETnZObymJ+CMf79bNsYA0KDTw5g X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 4 Sep 2026 11:10:49 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn , "Kiryl Shutsemau (Meta)" Subject: [PATCH 11/12] mm/collapse: declare the collapse interface in collapse.h Date: Fri, 4 Sep 2026 16:10:25 +0100 Message-ID: <3b698ec5d0d8f2fe51a298369778cb7d6a6a1f4b.1788533997.git.kas@kernel.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" A collapse takes four calls: - collapse_control_init() - set up the control a caller carries; - collapse_scan_pmd() - scan one PTE table, under mmap_lock; - collapse_run_pmd() - collapse what the scan found, no mmap_lock; - collapse_control_release() - done with the control. All four are static in khugepaged.c, as are collapse_possible_orders(), which says what a VMA allows, and the revalidate a caller needs once a collapse has given the mmap_lock up. No other file can ask for a collapse without them. Declare them in collapse.h, with a comment stating the order they are called in and who holds the lock over each step. hugepage_vma_revalidate() becomes collapse_vma_revalidate(): it is part of what a collapse offers now, not a helper of the daemon. Preparation for implementing MADV_COLLAPSE in madvise.c. No functional change. Assisted-by: Claude-Code:claude-opus-5 Signed-off-by: Kiryl Shutsemau (Meta) --- mm/collapse.h | 44 ++++++++++++++++++++++++++++++++++++++++++++ mm/khugepaged.c | 18 +++++++++--------- 2 files changed, 53 insertions(+), 9 deletions(-) diff --git a/mm/collapse.h b/mm/collapse.h index f03cad8ed40e..c5773e273e15 100644 --- a/mm/collapse.h +++ b/mm/collapse.h @@ -116,4 +116,48 @@ struct collapse_control { pgoff_t scan_pgoff; }; =20 +/* Which orders a VMA may collapse to, zero when it may not collapse at al= l */ +unsigned long collapse_possible_orders(struct vm_area_struct *vma, + vm_flags_t vm_flags, enum tva_type tva_flags); + +/* + * A caller states what it allows in cc->policy and then hands over one PTE + * table's worth of a VMA at a time: + * + * collapse_control_init(cc) once, before the first table + * collapse_scan_pmd(vma, addr, ...) per table + * collapse_run_pmd(mm, addr, cc) when a scan found work + * collapse_control_release(cc) once, when done with the control + * + * The caller holds mmap_lock for reading over the scan and passes an addr= ess + * within @vma, aligned to the PTE table the scan is to judge. + * + * The scan returns with that lock still held. It only reads, and almost = every + * table it is offered has nothing to collapse, so a caller walks a whole = VMA + * under the one lock it took to get there. SCAN_SUCCEED means there is + * something to collapse; anything else is why there is not. + * + * The run is called without the lock and returns without it, taking what = it + * needs in between: what it does -- allocate, isolate, copy, flush -- is = slow + * enough that a writer would wait behind it. The caller gives the lock up + * first, and with it @vma and anything derived under it, so a caller carr= ying + * on has to look up again with collapse_vma_revalidate(). The run revali= dates + * for itself rather than trusting what the scan saw. + * + * A scan that found something has to be run: the file side takes a refere= nce on + * the file while it still has the VMA to take it from, and the run is what + * gives it back. + */ +void collapse_control_init(struct collapse_control *cc); +void collapse_control_release(struct collapse_control *cc); +enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, + unsigned long addr, struct collapse_control *cc, + unsigned long orders); +enum scan_result collapse_run_pmd(struct mm_struct *mm, unsigned long addr, + struct collapse_control *cc); +enum scan_result collapse_vma_revalidate(struct mm_struct *mm, + unsigned long address, bool expect_anon, + struct vm_area_struct **vmap, struct collapse_control *cc, + unsigned int order); + #endif /* __MM_COLLAPSE_H */ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index f862abb1dbbd..13c4dbf04379 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -498,7 +498,7 @@ void __khugepaged_enter(struct mm_struct *mm) * Check what orders are possible based on the vma and collapse type. * This is used to determine if mTHP collapse is a viable option. */ -static unsigned long collapse_possible_orders(struct vm_area_struct *vma, +unsigned long collapse_possible_orders(struct vm_area_struct *vma, vm_flags_t vm_flags, enum tva_type tva_flags) { unsigned long orders; @@ -1016,7 +1016,7 @@ static int collapse_find_target_node(struct collapse_= control *cc) * Returns enum scan_result value. */ =20 -static enum scan_result hugepage_vma_revalidate(struct mm_struct *mm, unsi= gned long address, +enum scan_result collapse_vma_revalidate(struct mm_struct *mm, unsigned lo= ng address, bool expect_anon, struct vm_area_struct **vmap, struct collapse_control *cc, unsigned int order) { @@ -1264,7 +1264,7 @@ static enum scan_result collapse_huge_page(struct mm_= struct *mm, unsigned long s } =20 mmap_read_lock(mm); - result =3D hugepage_vma_revalidate(mm, pmd_addr, /*expect_anon=3D*/ true, + result =3D collapse_vma_revalidate(mm, pmd_addr, /*expect_anon=3D*/ true, &vma, cc, order); if (result !=3D SCAN_SUCCEED) { mmap_read_unlock(mm); @@ -1299,7 +1299,7 @@ static enum scan_result collapse_huge_page(struct mm_= struct *mm, unsigned long s * mmap_lock. */ mmap_write_lock(mm); - result =3D hugepage_vma_revalidate(mm, pmd_addr, /*expect_anon=3D*/ true, + result =3D collapse_vma_revalidate(mm, pmd_addr, /*expect_anon=3D*/ true, &vma, cc, order); if (result !=3D SCAN_SUCCEED) goto out_up_write; @@ -2754,13 +2754,13 @@ static enum scan_result collapse_scan_file(struct m= m_struct *mm, return result; } =20 -static void collapse_control_init(struct collapse_control *cc) +void collapse_control_init(struct collapse_control *cc) { cc->progress =3D 0; cc->scan_file =3D NULL; } =20 -static void collapse_control_release(struct collapse_control *cc) +void collapse_control_release(struct collapse_control *cc) { /* A scan that took a file reference should have been run */ if (WARN_ON_ONCE(cc->scan_file)) { @@ -2769,7 +2769,7 @@ static void collapse_control_release(struct collapse_= control *cc) } } =20 -static enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, +enum scan_result collapse_scan_pmd(struct vm_area_struct *vma, unsigned long addr, struct collapse_control *cc, unsigned long orders) { @@ -2793,7 +2793,7 @@ static enum scan_result collapse_scan_pmd(struct vm_a= rea_struct *vma, return SCAN_SUCCEED; } =20 -static enum scan_result collapse_run_pmd(struct mm_struct *mm, +enum scan_result collapse_run_pmd(struct mm_struct *mm, unsigned long addr, struct collapse_control *cc) { struct file *file =3D cc->scan_file; @@ -3237,7 +3237,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, if (!vma) { cond_resched(); mmap_read_lock(mm); - result =3D hugepage_vma_revalidate(mm, addr, false, &found, + result =3D collapse_vma_revalidate(mm, addr, false, &found, cc, HPAGE_PMD_ORDER); if (result !=3D SCAN_SUCCEED) { last_fail =3D result; --=20 2.54.0 From nobody Sat Sep 26 03:17:25 2026 Received: from fhigh-a2-smtp.messagingengine.com (fhigh-a2-smtp.messagingengine.com [103.168.172.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 41FE94F30E8 for ; Fri, 4 Sep 2026 15:10:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.153 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534655; cv=none; b=PykGLeQjEmPVK5T75IA2VhJBy1ykt0Zi5TFOWlEaoCVZh9mKuAz+VtMMHHC6JEHiPaHxTNGd0HWzywnjZoGfAsffwsMiB2aV3cKQs+ubzJR2+5Bha42uHLUJ/vJYEFI5jJ2cl1HmDt3qS3eu16+Ocx3buEDkpO3vV49MciKsBdQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788534655; c=relaxed/simple; bh=9/2p7fwQjwoZ0kVps89n0/5hQA8VlbpFGkjXCpBqlAQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oNlW/VFk2E6CBveb00SHOen62yfnk9n8pl9se+f5NB5jSz4N7PuUIE29/Z3Y9/j8BTpiE1MoIe41xwuZVykBuE91eXA9mVX1ekwd6lEXotzKDauZz7ZAeQArgqW01Ly1QkMqBzJI3gOx3pufusO34w7miXKVy7CdwZuInNWMSnc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name; spf=pass smtp.mailfrom=shutemov.name; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b=aMPSNVUm; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=CVLXwanr; arc=none smtp.client-ip=103.168.172.153 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=shutemov.name Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shutemov.name Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shutemov.name header.i=@shutemov.name header.b="aMPSNVUm"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="CVLXwanr" Received: from phl-compute-02.internal (phl-compute-02.internal [10.202.2.42]) by mailfhigh.phl.internal (Postfix) with ESMTP id 08DAC1400112; Fri, 4 Sep 2026 11:10:52 -0400 (EDT) Received: from phl-frontend-04 ([10.202.2.163]) by phl-compute-02.internal (MEProxy); Fri, 04 Sep 2026 11:10:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shutemov.name; h=cc:cc:content-transfer-encoding:content-type:date:date:from :from:in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to; s=fm2; t=1788534652; x= 1788621052; bh=Ccy6RVDvD/8YFeNTOJ5S1kQA1S4Ako8Y2iYhuKh8Zuk=; b=a MPSNVUmkZ4q94Dd/bhIhiZ09l+zrxVTy215hZytih6waWGbMA2v2ESz1Df+YElOY Ot5XOqO+uyZY3crH0ppV/b/ISl3APPo2YgHAhE8ODgXjgt1uzrhRXHCjzGzBaoPn DabRInv8Oclx91vbauuOag/Qc0hqXljsav+tV8PTSPJL18pRyROdiRteYQ/hh2MG V8T/+x6QFzKYNcSwiyShHasRPeAFEFJuPourrenAthi8VWnr1otiz+O2bD9uGt/X oHskaaxTPFfAcWKt8ISM3QV3qtOCGL278M9gytGQxZoK4/bcKvrgbEaiIZG4SK7s vZABkGbBukkjP4arT+xnw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:subject:subject:to:to:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1788534652; x=1788621052; bh=C cy6RVDvD/8YFeNTOJ5S1kQA1S4Ako8Y2iYhuKh8Zuk=; b=CVLXwanrO6sThVsge KJyXdziqUqvvs+AiYHEXenfLYKhwP+EO+9NWSBoSnHblOmqnGz1VdD88KGf7oDYm kncA7r3S87+p5Btz4Rpf6bSvfK5+1Z8/doCVf4KcLiKSaSxddFiMSJlcTSEz+FhT +rOS5utqwwOqESTuP5b6nSY5SyG3ctvAVk4/eooqEbILJCIx5wmYtzGhSt4NdiWy V6+JPUfGOhH7jKF5Pom6DpyUKiFEaQfjQnrC81JkLdjlM0dBf2iY4RX0Qb/Ba8oy nSeiHKhis2EEcn5eWpJ/aEqlGvemR1YvfE9GlpCXN8ZKl8XHLa21YFPZx5LXDCvs DHHYQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: dmFkZTET6cjphBfz4H8Wg4Mwj3Q2fsO4rbzY0RZa6uwdK6ABZ7pEhWaf3DWE0Wh23XozXl YXqBYYUtwtfZ1qQ9wX2bCqZqo/yq027VrzWqESBcvph06bOS8vyiOGOCS0fh2RY7A/kSFI L9MElk4eZ3HddGFY9A2GQHzM4/P7C4mhUDQ1iVikEtpcmuF5W8uFTm0u0CjZbROqFnINk4 xO1zzX/Hj74YRRcHjJyG4ygWf0Hx5RhGZptpxeuVNBh7kD8dz0lfmQnwRvV5yZ57YM54K9 KH8vVIjAAxpTayQfOZqnAJ2Uka0rmt6zQ0IYLz8TzxO+2W2OgPHZMW2lenfH2WoNpxg3YY DbliSH1tOdYs4x4aCoptHA30V8qxy/dp7UU56lpnsMwYKlJ2phGDgNx0bHcFNcOPxepHMX jl09zm5j5DUI+QBOkfeGkm58rFd36YqbNGTXV/60BMrYHPLHK9N5dtTvv94hEddeWWJTS7 p+WCzfY+L6jW/W42Qd2vbvlKjObq9OvBSNjlRaiWuC/wWZ0bxY1sCE1ULsugksgvekA8sJ Rt+UMhHMKN/v3KiYFi7p6LeOb+BlwcQqfqWxWHh8IqUCxI6XsEYGhSWfuvND6uPMnra4gY qVNN0c/MZ/BeVEv8Q3lgjNIZqeDA32eL0WSaslD3pgt3xYWz8Hvnx2LVfdKw X-ME-Proxy: Feedback-ID: ie3994620:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 4 Sep 2026 11:10:51 -0400 (EDT) From: Kiryl Shutsemau To: Andrew Morton , David Hildenbrand , Lorenzo Stoakes Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Zi Yan , Baolin Wang , "Liam R . Howlett" , Nico Pache , Ryan Roberts , Dev Jain , Barry Song , Lance Yang , Usama Arif , Vlastimil Babka , Jann Horn , "Kiryl Shutsemau (Meta)" Subject: [PATCH 12/12] mm/collapse: implement MADV_COLLAPSE in madvise.c Date: Fri, 4 Sep 2026 16:10:26 +0100 Message-ID: X-Mailer: git-send-email 2.55.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: "Kiryl Shutsemau (Meta)" MADV_COLLAPSE is a madvise operation, but its implementation sat in khugepaged.c. The daemon's file therefore also held a syscall's worth of code that has nothing to do with the daemon: the walk over the user's range, the per-PMD loop, and the errno translation. Move it to madvise.c, among the operations it belongs with, along with the errno map and the policy it states for itself. It takes a struct madvise_behavior like every one of those operations, which is where the range, the VMA and the lock-dropped flag it used to be handed separately already live. It stays a caller of the interface khugepaged uses, so nothing about the collapse changes. The !CONFIG_TRANSPARENT_HUGEPAGE stub moves in with it. Assisted-by: Claude-Code:claude-opus-5 Signed-off-by: Kiryl Shutsemau (Meta) --- include/linux/huge_mm.h | 9 --- mm/khugepaged.c | 157 +------------------------------------ mm/madvise.c | 169 +++++++++++++++++++++++++++++++++++++++- 3 files changed, 169 insertions(+), 166 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index c745f7ad2298..8ca0fa3be2ac 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -510,8 +510,6 @@ change_huge_pud(struct mmu_gather *tlb, struct vm_area_= struct *vma, =20 int hugepage_madvise(struct vm_area_struct *vma, vm_flags_t *vm_flags, int advice); -int madvise_collapse(struct vm_area_struct *vma, unsigned long start, - unsigned long end, bool *lock_dropped); void vma_adjust_trans_huge(struct vm_area_struct *vma, unsigned long start, unsigned long end, struct vm_area_struct *next); spinlock_t *__pmd_trans_huge_lock(pmd_t *pmd, struct vm_area_struct *vma); @@ -715,13 +713,6 @@ static inline int hugepage_madvise(struct vm_area_stru= ct *vma, return -EINVAL; } =20 -static inline int madvise_collapse(struct vm_area_struct *vma, - unsigned long start, - unsigned long end, bool *lock_dropped) -{ - return -EINVAL; -} - static inline void vma_adjust_trans_huge(struct vm_area_struct *vma, unsigned long start, unsigned long end, diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 13c4dbf04379..0bafdb725204 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -969,21 +969,6 @@ static void collapse_policy_khugepaged(struct collapse= _policy *p) p->tva_type =3D TVA_KHUGEPAGED; } =20 -/* MADV_COLLAPSE was asked for explicitly, so it is not held to those */ -static void collapse_policy_forced(struct collapse_policy *p) -{ - p->max_ptes_none =3D HPAGE_PMD_NR; - p->max_ptes_swap =3D HPAGE_PMD_NR; - p->max_ptes_shared =3D HPAGE_PMD_NR; - p->strict_sub_pmd =3D false; - p->skip_lazyfree =3D false; - p->require_referenced =3D false; - p->install_pmd =3D true; - p->writeback_dirty =3D true; - p->gfp =3D GFP_TRANSHUGE; - p->tva_type =3D TVA_FORCED_COLLAPSE; -} - #ifdef CONFIG_NUMA static int collapse_find_target_node(struct collapse_control *cc) { @@ -2793,8 +2778,8 @@ enum scan_result collapse_scan_pmd(struct vm_area_str= uct *vma, return SCAN_SUCCEED; } =20 -enum scan_result collapse_run_pmd(struct mm_struct *mm, - unsigned long addr, struct collapse_control *cc) +enum scan_result collapse_run_pmd(struct mm_struct *mm, unsigned long addr, + struct collapse_control *cc) { struct file *file =3D cc->scan_file; bool triggered_wb =3D false; @@ -3160,141 +3145,3 @@ bool current_is_khugepaged(void) { return kthread_func(current) =3D=3D khugepaged; } - -static int madvise_collapse_errno(enum scan_result r) -{ - /* - * MADV_COLLAPSE breaks from existing madvise(2) conventions to provide - * actionable feedback to caller, so they may take an appropriate - * fallback measure depending on the nature of the failure. - */ - switch (r) { - case SCAN_ALLOC_HUGE_PAGE_FAIL: - return -ENOMEM; - case SCAN_CGROUP_CHARGE_FAIL: - case SCAN_EXCEED_NONE_PTE: - return -EBUSY; - /* Resource temporary unavailable - trying again might succeed */ - case SCAN_PAGE_COUNT: - case SCAN_PAGE_LOCK: - case SCAN_PAGE_LRU: - case SCAN_DEL_PAGE_LRU: - case SCAN_PAGE_FILLED: - case SCAN_PAGE_HAS_PRIVATE: - case SCAN_PAGE_DIRTY_OR_WRITEBACK: - return -EAGAIN; - /* - * Other: Trying again likely not to succeed / error intrinsic to - * specified memory range. khugepaged likely won't be able to collapse - * either. - */ - default: - return -EINVAL; - } -} - -int madvise_collapse(struct vm_area_struct *vma, unsigned long start, - unsigned long end, bool *lock_dropped) -{ - struct collapse_control *cc; - struct mm_struct *mm =3D vma->vm_mm; - unsigned long hstart, hend, addr, orders; - enum scan_result last_fail =3D SCAN_FAIL; - int thps =3D 0; - - BUG_ON(vma->vm_start > start); - BUG_ON(vma->vm_end < end); - - orders =3D collapse_possible_orders(vma, vma->vm_flags, - TVA_FORCED_COLLAPSE); - if (!orders) - return -EINVAL; - - hstart =3D ALIGN(start, HPAGE_PMD_SIZE); - hend =3D ALIGN_DOWN(end, HPAGE_PMD_SIZE); - - if (hstart >=3D hend) - return 0; - - cc =3D kmalloc_obj(*cc); - if (!cc) - return -ENOMEM; - collapse_control_init(cc); - collapse_policy_forced(&cc->policy); - - lru_add_drain_all(); - - for (addr =3D hstart; addr < hend; addr +=3D HPAGE_PMD_SIZE) { - struct vm_area_struct *found; - enum scan_result result; - - /* - * A collapse gives the lock up, so the VMA has to be found - * again after one: it can shrink while nothing is held. A scan - * that finds nothing to collapse leaves the lock alone, so a - * range that is already collapsed walks on without relocking. - */ - if (!vma) { - cond_resched(); - mmap_read_lock(mm); - result =3D collapse_vma_revalidate(mm, addr, false, &found, - cc, HPAGE_PMD_ORDER); - if (result !=3D SCAN_SUCCEED) { - last_fail =3D result; - goto out_locked; - } - vma =3D found; - hend =3D min(hend, vma->vm_end & HPAGE_PMD_MASK); - orders =3D collapse_possible_orders(vma, vma->vm_flags, - cc->policy.tva_type); - } - - result =3D collapse_scan_pmd(vma, addr, cc, orders); - /* Nothing to collapse here, and the lock is still ours */ - if (result !=3D SCAN_SUCCEED) - goto tally; - - /* The collapse takes its own locks, so give this up */ - mmap_read_unlock(mm); - *lock_dropped =3D true; - vma =3D NULL; - - result =3D collapse_run_pmd(mm, addr, cc); -tally: - switch (result) { - case SCAN_SUCCEED: - case SCAN_PMD_MAPPED: - ++thps; - break; - /* Whitelisted set of results where continuing OK */ - case SCAN_NO_PTE_TABLE: - case SCAN_PTE_NON_PRESENT: - case SCAN_PTE_UFFD: - case SCAN_LACK_REFERENCED_PAGE: - case SCAN_PAGE_NULL: - case SCAN_PAGE_COUNT: - case SCAN_PAGE_LOCK: - case SCAN_PAGE_COMPOUND: - case SCAN_PAGE_LRU: - case SCAN_DEL_PAGE_LRU: - last_fail =3D result; - break; - default: - last_fail =3D result; - /* Other error, exit */ - goto out; - } - } - -out: - /* Caller expects us to hold mmap_lock on return */ - if (!vma) - mmap_read_lock(mm); -out_locked: - mmap_assert_locked(mm); - collapse_control_release(cc); - kfree(cc); - - return thps =3D=3D ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0 - : madvise_collapse_errno(last_fail); -} diff --git a/mm/madvise.c b/mm/madvise.c index 73c2901b9adb..25a58f97b4e5 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -38,6 +38,7 @@ =20 #include "internal.h" #include "swap.h" +#include "collapse.h" =20 #define __MADV_SET_ANON_VMA_NAME (-1) =20 @@ -905,6 +906,171 @@ bool madvise_dontneed_free_valid_vma(struct madvise_b= ehavior *madv_behavior) return true; } =20 +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + +/* MADV_COLLAPSE was asked for explicitly, so it is not held to those */ +static void collapse_policy_forced(struct collapse_policy *p) +{ + p->max_ptes_none =3D HPAGE_PMD_NR; + p->max_ptes_swap =3D HPAGE_PMD_NR; + p->max_ptes_shared =3D HPAGE_PMD_NR; + p->strict_sub_pmd =3D false; + p->skip_lazyfree =3D false; + p->require_referenced =3D false; + p->install_pmd =3D true; + p->writeback_dirty =3D true; + p->gfp =3D GFP_TRANSHUGE; + p->tva_type =3D TVA_FORCED_COLLAPSE; +} + +static int madvise_collapse_errno(enum scan_result r) +{ + /* + * MADV_COLLAPSE breaks from existing madvise(2) conventions to provide + * actionable feedback to caller, so they may take an appropriate + * fallback measure depending on the nature of the failure. + */ + switch (r) { + case SCAN_ALLOC_HUGE_PAGE_FAIL: + return -ENOMEM; + case SCAN_CGROUP_CHARGE_FAIL: + case SCAN_EXCEED_NONE_PTE: + return -EBUSY; + /* Resource temporary unavailable - trying again might succeed */ + case SCAN_PAGE_COUNT: + case SCAN_PAGE_LOCK: + case SCAN_PAGE_LRU: + case SCAN_DEL_PAGE_LRU: + case SCAN_PAGE_FILLED: + case SCAN_PAGE_HAS_PRIVATE: + case SCAN_PAGE_DIRTY_OR_WRITEBACK: + return -EAGAIN; + /* + * Other: Trying again likely not to succeed / error intrinsic to + * specified memory range. khugepaged likely won't be able to collapse + * either. + */ + default: + return -EINVAL; + } +} + +static int madvise_collapse(struct madvise_behavior *madv_behavior) +{ + struct madvise_behavior_range *range =3D &madv_behavior->range; + struct vm_area_struct *vma =3D madv_behavior->vma; + struct mm_struct *mm =3D madv_behavior->mm; + struct collapse_control *cc; + unsigned long hstart, hend, addr, orders; + enum scan_result last_fail =3D SCAN_FAIL; + int thps =3D 0; + + BUG_ON(vma->vm_start > range->start); + BUG_ON(vma->vm_end < range->end); + + orders =3D collapse_possible_orders(vma, vma->vm_flags, + TVA_FORCED_COLLAPSE); + if (!orders) + return -EINVAL; + + hstart =3D ALIGN(range->start, HPAGE_PMD_SIZE); + hend =3D ALIGN_DOWN(range->end, HPAGE_PMD_SIZE); + + if (hstart >=3D hend) + return 0; + + cc =3D kmalloc_obj(*cc); + if (!cc) + return -ENOMEM; + collapse_control_init(cc); + collapse_policy_forced(&cc->policy); + + lru_add_drain_all(); + + for (addr =3D hstart; addr < hend; addr +=3D HPAGE_PMD_SIZE) { + struct vm_area_struct *found; + enum scan_result result; + + /* + * A collapse gives the lock up, so the VMA has to be found + * again after one: it can shrink while nothing is held. A scan + * that finds nothing to collapse leaves the lock alone, so a + * range that is already collapsed walks on without relocking. + */ + if (!vma) { + cond_resched(); + mmap_read_lock(mm); + result =3D collapse_vma_revalidate(mm, addr, false, &found, + cc, HPAGE_PMD_ORDER); + if (result !=3D SCAN_SUCCEED) { + last_fail =3D result; + goto out_locked; + } + vma =3D found; + hend =3D min(hend, vma->vm_end & HPAGE_PMD_MASK); + orders =3D collapse_possible_orders(vma, vma->vm_flags, + cc->policy.tva_type); + } + + result =3D collapse_scan_pmd(vma, addr, cc, orders); + /* Nothing to collapse here, and the lock is still ours */ + if (result !=3D SCAN_SUCCEED) + goto tally; + + /* The collapse takes its own locks, so give this up */ + mmap_read_unlock(mm); + mark_mmap_lock_dropped(madv_behavior); + vma =3D NULL; + + result =3D collapse_run_pmd(mm, addr, cc); +tally: + switch (result) { + case SCAN_SUCCEED: + case SCAN_PMD_MAPPED: + ++thps; + break; + /* Whitelisted set of results where continuing OK */ + case SCAN_NO_PTE_TABLE: + case SCAN_PTE_NON_PRESENT: + case SCAN_PTE_UFFD: + case SCAN_LACK_REFERENCED_PAGE: + case SCAN_PAGE_NULL: + case SCAN_PAGE_COUNT: + case SCAN_PAGE_LOCK: + case SCAN_PAGE_COMPOUND: + case SCAN_PAGE_LRU: + case SCAN_DEL_PAGE_LRU: + last_fail =3D result; + break; + default: + last_fail =3D result; + /* Other error, exit */ + goto out; + } + } + +out: + /* Caller expects us to hold mmap_lock on return */ + if (!vma) + mmap_read_lock(mm); +out_locked: + mmap_assert_locked(mm); + collapse_control_release(cc); + kfree(cc); + + return thps =3D=3D ((hend - hstart) >> HPAGE_PMD_SHIFT) ? 0 + : madvise_collapse_errno(last_fail); +} + +#else /* CONFIG_TRANSPARENT_HUGEPAGE */ + +static int madvise_collapse(struct madvise_behavior *madv_behavior) +{ + return -EINVAL; +} + +#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + static long madvise_dontneed_free(struct madvise_behavior *madv_behavior) { struct mm_struct *mm =3D madv_behavior->mm; @@ -1372,8 +1538,7 @@ static int madvise_vma_behavior(struct madvise_behavi= or *madv_behavior) case MADV_DONTNEED_LOCKED: return madvise_dontneed_free(madv_behavior); case MADV_COLLAPSE: - return madvise_collapse(vma, range->start, range->end, - &madv_behavior->lock_dropped); + return madvise_collapse(madv_behavior); case MADV_GUARD_INSTALL: return madvise_guard_install(madv_behavior); case MADV_GUARD_REMOVE: --=20 2.54.0