From nobody Thu Apr 2 14:14:42 2026 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B55021ABB1 for ; Sat, 21 Feb 2026 09:39:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771666795; cv=none; b=HiPGx1h/XCt4Gdk0g8KkhdsPLF1uZI3PuydYF7+vHFfobzaerGxxEL03AbRDN0YT2Rc8gdUWp8CphXM1gnHgl96P6RgfdEnebtL8BjLNMnkYO0+MF4ytsXg+mL8PRx39Oh3IMnVwy7O+9xnuiPqg4ZoRJXmvfbuNIKGFTtRanUw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771666795; c=relaxed/simple; bh=IozlCoE3m+EM1aLfHJdZWpN7/bdk36hI9bvT4eSwteI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jxpmhB8yyLCjLHRrvXxCRPb3MrrJxhCSPcakB8Gp5ZBuf+tTdoGEiZsERl/8e+gK8GBpKKJHj/LzYIjGw7chGHIjFMMfFYYXybIPw92Ph3f5TMymhYleNZqi9JY3SNK+T8hWSNVcyRlplqMJyS9m0XQj9dO6OT3oo9iosTfj1qE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=jxRMq/bW; arc=none smtp.client-ip=209.85.210.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jxRMq/bW" Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-82318b640beso1542219b3a.0 for ; Sat, 21 Feb 2026 01:39:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1771666794; x=1772271594; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eRhfRltQOpryarjivZ7aunnsdQEKu4QRgnr74+an19Y=; b=jxRMq/bWqFGibVZkFclTE4jNhka+CXJTbYlA41/JPtiNZ/xVya25CMLddjfdHNZDaZ ggdVDnf0spNPifbofhY9QuRZFa2y4fI2KI1xNS0fK6ovWrPcyMbuBotn3vuZIKlOyGDv 3VYeTBovYmKqJpUKGTffPQprrmNR51zmSyvd4MXdE0rzOE5+hkann1bwOwXuiTvQR9yT NlYbkll0UlykS/ModoxCrqVerVCZ/Pf4iShFJsQhjOxZdtmdskNwXF75kOv6B4UBp3Vq GGAIi2cnrBijtH4wI+rwc71KPG9cnxNotp4ahy+earebP6tFoCXB12ynYMwlF0NCknz0 5ArQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771666794; x=1772271594; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=eRhfRltQOpryarjivZ7aunnsdQEKu4QRgnr74+an19Y=; b=J3fcX4x6NNBhQHiPd80Cp1WldnjWIe7oKMjsFKdU2R9vb19vIrkme7cYnXppIkjRaz gneeQAIwLzWWCS5LMongL7DcaWPsXVZ1Nzqi/xnj3Mm+GQ2ZXI2W8vFgeaOR40CRh+P8 D40OjRzQRg3gq7uqYn5Nfj/CigSDvNu78hnDMC3mOjpsLnUPiIY7PJbJflVPtLmlpbT5 n7h5nF4SNDgUrcvamf8fEw17Fuo1h+hXB3vxDkdak+ugcrWMCeosPlejIL8hpQJ3foEs cXq9SY4cIrjtjrz5gMozfHzQtOjjqqWW1ZT8Ev7C7YZ12NxPAHZfG36oseT7U0O16ial 2QzA== X-Forwarded-Encrypted: i=1; AJvYcCX47EMyf3VqH/inLe8j1a8pwf0D4iV9NKlpQ9f0ODfwfqioiPhDSoAXcbkbGR+T//hAoWl+GSdW02BtdhU=@vger.kernel.org X-Gm-Message-State: AOJu0Yw3fpqN2OIa182nd979U5EjIEN8a5JdnHkArHs0etd3HvZBckku EZP9DrcKCA8NRHlKfL/q2fTu6S0CyqDzseVT8hpuc9QLlRBg1TH2og5u X-Gm-Gg: AZuq6aIQfUcw6lbTyZxSVxPia0c4DFRbHbs7dnZH9GjbmaXBgSx1IHBVF8yntkm1DmU JE1IGNLGcO1NyCVMv6naiuInGLIo5pkpFClyFt8JKNNVjFgVbCScb02It20i4FgEeJ4eQBqYEwP u9daZiHtp+gMgEd2an+T7L8GBG3o3xfOyVyuEpfG0FY6sHU6wQTzuJd1lKbR1bvgv0rvtABP6ZW ekKjWqefLUTUWq+cQkocrqgEAZyVxaPtyta+vNHUEqSv6rPrY5XxtmWDnqeJ3jsUEpqnP8Gzs/G jlh9o1utDgHFFUJlgO+FwWDOfnnPe3doUrEs8ajBZOSCw3r5TtZ0sj1CJ1xaZM7z/Nc/9ijnF3T ksMWT0byZsY3/wr7PsCPLX4hh+UzhSQnvr1Lp7TvH1cup6Leufj0Dhud26gKz/Nf5nMGb1jSR3P e2yXJXxkTIJ7/Pjl9NokBNY15IYfzGPD+0kLKoaLFWUnhB X-Received: by 2002:a05:6a00:4189:b0:824:a8ec:ed51 with SMTP id d2e1a72fcca58-826da91f743mr1678128b3a.27.1771666793584; Sat, 21 Feb 2026 01:39:53 -0800 (PST) Received: from localhost.localdomain ([49.79.21.101]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-826dd8ba11bsm1761708b3a.50.2026.02.21.01.39.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Feb 2026 01:39:53 -0800 (PST) From: Vernon Yang To: akpm@linux-foundation.org, david@kernel.org Cc: lorenzo.stoakes@oracle.com, ziy@nvidia.com, dev.jain@arm.com, baohua@kernel.org, lance.yang@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Vernon Yang Subject: [PATCH mm-new v8 1/4] mm: khugepaged: add trace_mm_khugepaged_scan event Date: Sat, 21 Feb 2026 17:39:15 +0800 Message-ID: <20260221093918.1456187-2-vernon2gm@gmail.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260221093918.1456187-1-vernon2gm@gmail.com> References: <20260221093918.1456187-1-vernon2gm@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Vernon Yang Add mm_khugepaged_scan event to track the total time for full scan and the total number of pages scanned of khugepaged. Signed-off-by: Vernon Yang Acked-by: David Hildenbrand (Red Hat) Reviewed-by: Barry Song Reviewed-by: Lance Yang Reviewed-by: Dev Jain Reviewed-by: Lorenzo Stoakes (Oracle) --- include/trace/events/huge_memory.h | 25 +++++++++++++++++++++++++ mm/khugepaged.c | 2 ++ 2 files changed, 27 insertions(+) diff --git a/include/trace/events/huge_memory.h b/include/trace/events/huge= _memory.h index 4e41bff31888..384e29f6bef0 100644 --- a/include/trace/events/huge_memory.h +++ b/include/trace/events/huge_memory.h @@ -237,5 +237,30 @@ TRACE_EVENT(mm_khugepaged_collapse_file, __print_symbolic(__entry->result, SCAN_STATUS)) ); =20 +TRACE_EVENT(mm_khugepaged_scan, + + TP_PROTO(struct mm_struct *mm, unsigned int progress, + bool full_scan_finished), + + TP_ARGS(mm, progress, full_scan_finished), + + TP_STRUCT__entry( + __field(struct mm_struct *, mm) + __field(unsigned int, progress) + __field(bool, full_scan_finished) + ), + + TP_fast_assign( + __entry->mm =3D mm; + __entry->progress =3D progress; + __entry->full_scan_finished =3D full_scan_finished; + ), + + TP_printk("mm=3D%p, progress=3D%u, full_scan_finished=3D%d", + __entry->mm, + __entry->progress, + __entry->full_scan_finished) +); + #endif /* __HUGE_MEMORY_H */ #include diff --git a/mm/khugepaged.c b/mm/khugepaged.c index c0f893bebcff..e2f6b68a0011 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -2527,6 +2527,8 @@ static unsigned int khugepaged_scan_mm_slot(unsigned = int pages, enum scan_result collect_mm_slot(slot); } =20 + trace_mm_khugepaged_scan(mm, progress, khugepaged_scan.mm_slot =3D=3D NUL= L); + return progress; } =20 --=20 2.51.0 From nobody Thu Apr 2 14:14:42 2026 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C1782F6192 for ; Sat, 21 Feb 2026 09:39:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771666800; cv=none; b=MTBNIPX3zsNHu068aDWJZVquX9R5vNN9OjBrNuuJ1p4ZbppRkLMGRJa+a2yEgDWBO3KbnNcozFgnrXE9q/WN4C1hy15/jJjx2q7r2vGEBamwOUxFxwf0hHDKot9CIbqrObSszlZ0MIg4CMj3mNeIKYw4EMwjqLhllHFGmYHMGIo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771666800; c=relaxed/simple; bh=MuOppxHTXWhTZdWc0VDsZZC1NCGM+gxAgSHNnjZR0KI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LwU/1il+uv1U7snO+9JsmYNjnh+rCW9CS1mf122FSrOKSlPDjt/X2QFB/mfi6DI8fVbCAt2xJy1GuB89vgODzwMlMxZwtttf7veXPHfmwXQ6MRWb50fYXUGML/KyJnYYeLRxodpcEG9nCtcl7kgqFyyjlGBKcOvoTlE7ROmhin0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Y4SsUaOp; arc=none smtp.client-ip=209.85.210.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Y4SsUaOp" Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-8217f2ad01eso2887596b3a.2 for ; Sat, 21 Feb 2026 01:39:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1771666799; x=1772271599; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FhfMK34w+7c74FDll4ied9faGONb5ohZzrn5WysTLck=; b=Y4SsUaOpdcphci+Exa4yN3hYeapdosC2kRFrjPvBiUp9Hh8N8//7QIxlguM0GurpHZ GekDxCjXqwrOdG6ikwZ8WRbD+8BPu8d4BT9ONhQVazEkvJTfLzGNeVFyTS3KXjtsnbGi L7kP0aAG4+FO55MvFz4iyUqLb+D0k5Kl4oANlHL44tmSrUpLAOjkKK1/bz06C5itZtt5 Z5dPbgrljLU/fVwniIQVzJJdyhLcxWXZBBi2SPhCGMxt6REv0hA8aj3DDPhfNJa4cXdQ +WUOt6/RdPWqWU/3keE/TC7VGzcd55uTxubIFLIpgUynqk9CofEEkbk+gnP3DSr8/GEm 7/qA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771666799; x=1772271599; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=FhfMK34w+7c74FDll4ied9faGONb5ohZzrn5WysTLck=; b=eTUUqnop9FTLAoL0noEoGJOaYepNpbEWA4C0KkGroTnJfu8q/zyfs7RMqQZWaWbdRg sAF7ZHv2tQee2awWHinzgbUfoGTB9o8VKeITQMhgBcdO4CO+ih5G0p0BieXXMbN51KoQ B/nW9CtMxRTE0RyVB53ujrBK+deOHE1mbn14alX/x4SKlEmEJXamhE7mXyCAonPD9KCj qG8+gnWIjUu9hl0ycE6uiWM5F7dZpE5OIsTJ+XlqPq2BIlUh3D0vlUxYL5ILdIH7JzJR jPRVn5CNT51Pl9mfAQ9IP7lpqFjeW44mdY5WntoNJE5T1PP4z+Ehiwx/ItiZLr9TAcGK yDuQ== X-Forwarded-Encrypted: i=1; AJvYcCXLl/DkNveKTu0s4/doUADGOm1/xJWxEGho6fbdHgFEkFtANwyZSgOsob3i4VYwBZx1td7pBj9266UJKXM=@vger.kernel.org X-Gm-Message-State: AOJu0YzviABM1LMWs8+oV72LS9x4Ftoyv53hL44Ej6b5imiRIgfq89I0 aLCPpatYWbIE353KIqTQQpdZuFWyZvRtYXCz433VVPQtKN8y4M0g2gct X-Gm-Gg: AZuq6aK4qGIucxvwylq86ezcYLsiR6uaoU9REcqbJVEgLT9vhI1L6O7uLkyQxs6A7Zk DBm2818bIrOfM9lCQRMoESmaJTFyaTPZylIKTKX5HrmgWan10v2QuriMlCwhGGrcLanJhlqNuPv yaVoBb7S93G7uGhRm4TIVwJTa41l1nJcWmE3sMWXIxg45zrgwr0Ejd3a2vqdHR8ovW8YwMtgtRR M6g80PElmXEJvoiLIqfFoq1suHV0QTSiaNxDlnUljPJnogx2+NyqfXTeg86GgU+yVugXRg4Rro6 6Pqx9w/bg2r2Pa/IquD1GTLYiCA16bexHPCciXYPe9HPwr1ChNaBDp7lI+UII7JU5ziHqSmlMfE cQuQqx5tzMbjkbPZ2IHskKcxfmAqaWuNgvcrCZUVhTzvHm1/eHGusN7rzxdmDVpw3kLfXEc6y7S mthhbZLLqB2fjrugjYquAbOJPcNrZrrx6IpA== X-Received: by 2002:a05:6a00:2913:b0:7e8:4433:8fb4 with SMTP id d2e1a72fcca58-826daa82a8dmr2231143b3a.60.1771666798773; Sat, 21 Feb 2026 01:39:58 -0800 (PST) Received: from localhost.localdomain ([49.79.21.101]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-826dd8ba11bsm1761708b3a.50.2026.02.21.01.39.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Feb 2026 01:39:58 -0800 (PST) From: Vernon Yang To: akpm@linux-foundation.org, david@kernel.org Cc: lorenzo.stoakes@oracle.com, ziy@nvidia.com, dev.jain@arm.com, baohua@kernel.org, lance.yang@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Vernon Yang Subject: [PATCH mm-new v8 2/4] mm: khugepaged: refine scan progress number Date: Sat, 21 Feb 2026 17:39:16 +0800 Message-ID: <20260221093918.1456187-3-vernon2gm@gmail.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260221093918.1456187-1-vernon2gm@gmail.com> References: <20260221093918.1456187-1-vernon2gm@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Vernon Yang Currently, each scan always increases "progress" by HPAGE_PMD_NR, even if only scanning a single PTE/PMD entry. - When only scanning a sigle PTE entry, let me provide a detailed example: static int hpage_collapse_scan_pmd() { for (addr =3D start_addr, _pte =3D pte; _pte < pte + HPAGE_PMD_NR; _pte++, addr +=3D PAGE_SIZE) { pte_t pteval =3D ptep_get(_pte); ... if (pte_uffd_wp(pteval)) { <-- first scan hit result =3D SCAN_PTE_UFFD_WP; goto out_unmap; } } } During the first scan, if pte_uffd_wp(pteval) is true, the loop exits directly. In practice, only one PTE is scanned before termination. Here, "progress +=3D 1" reflects the actual number of PTEs scanned, but previously "progress +=3D HPAGE_PMD_NR" always. - When the memory has been collapsed to PMD, let me provide a detailed example: The following data is traced by bpftrace on a desktop system. After the system has been left idle for 10 minutes upon booting, a lot of SCAN_PMD_MAPPED or SCAN_NO_PTE_TABLE are observed during a full scan by khugepaged. From trace_mm_khugepaged_scan_pmd and trace_mm_khugepaged_scan_file, the following statuses were observed, with frequency mentioned next to them: SCAN_SUCCEED : 1 SCAN_EXCEED_SHARED_PTE: 2 SCAN_PMD_MAPPED : 142 SCAN_NO_PTE_TABLE : 178 total progress size : 674 MB Total time : 419 seconds, include khugepaged_scan_sleep_millisecs The khugepaged_scan list save all task that support collapse into hugepage, as long as the task is not destroyed, khugepaged will not remove it from the khugepaged_scan list. This exist a phenomenon where task has already collapsed all memory regions into hugepage, but khugepaged continues to scan it, which wastes CPU time and invalid, and due to khugepaged_scan_sleep_millisecs (default 10s) causes a long wait for scanning a large number of invalid task, so scanning really valid task is later. After applying this patch, when the memory is either SCAN_PMD_MAPPED or SCAN_NO_PTE_TABLE, just skip it, as follow: SCAN_EXCEED_SHARED_PTE: 2 SCAN_PMD_MAPPED : 147 SCAN_NO_PTE_TABLE : 173 total progress size : 45 MB Total time : 20 seconds SCAN_PTE_MAPPED_HUGEPAGE is the same, for detailed data, refer to https://lore.kernel.org/linux-mm/4qdu7owpmxfh3ugsue775fxarw5g2gcggbxdf5psj7= 5nnu7z2u@cv2uu2yocaxq Signed-off-by: Vernon Yang Reviewed-by: Dev Jain --- mm/khugepaged.c | 42 ++++++++++++++++++++++++++++++++---------- 1 file changed, 32 insertions(+), 10 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index e2f6b68a0011..61e25cf5424b 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -68,7 +68,10 @@ enum scan_result { static struct task_struct *khugepaged_thread __read_mostly; static DEFINE_MUTEX(khugepaged_mutex); =20 -/* default scan 8*HPAGE_PMD_NR ptes (or vmas) every 10 second */ +/* + * default scan 8*HPAGE_PMD_NR ptes, pmd_mapped, no_pte_table or vmas + * every 10 second. + */ static unsigned int khugepaged_pages_to_scan __read_mostly; static unsigned int khugepaged_pages_collapsed; static unsigned int khugepaged_full_scans; @@ -1231,7 +1234,8 @@ static enum scan_result collapse_huge_page(struct mm_= struct *mm, unsigned long a } =20 static enum scan_result hpage_collapse_scan_pmd(struct mm_struct *mm, - struct vm_area_struct *vma, unsigned long start_addr, bool *mmap_locked, + struct vm_area_struct *vma, unsigned long start_addr, + bool *mmap_locked, unsigned int *cur_progress, struct collapse_control *cc) { pmd_t *pmd; @@ -1247,19 +1251,27 @@ static enum scan_result hpage_collapse_scan_pmd(str= uct mm_struct *mm, VM_BUG_ON(start_addr & ~HPAGE_PMD_MASK); =20 result =3D find_pmd_or_thp_or_none(mm, start_addr, &pmd); - if (result !=3D SCAN_SUCCEED) + if (result !=3D SCAN_SUCCEED) { + if (cur_progress) + *cur_progress =3D 1; goto out; + } =20 memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); pte =3D pte_offset_map_lock(mm, pmd, start_addr, &ptl); if (!pte) { + if (cur_progress) + *cur_progress =3D 1; result =3D SCAN_NO_PTE_TABLE; goto out; } =20 for (addr =3D start_addr, _pte =3D pte; _pte < pte + HPAGE_PMD_NR; _pte++, addr +=3D PAGE_SIZE) { + if (cur_progress) + *cur_progress +=3D 1; + pte_t pteval =3D ptep_get(_pte); if (pte_none_or_zero(pteval)) { ++none_or_zero; @@ -2279,8 +2291,9 @@ static enum scan_result collapse_file(struct mm_struc= t *mm, unsigned long addr, return result; } =20 -static enum scan_result hpage_collapse_scan_file(struct mm_struct *mm, uns= igned long addr, - struct file *file, pgoff_t start, struct collapse_control *cc) +static enum scan_result hpage_collapse_scan_file(struct mm_struct *mm, + unsigned long addr, struct file *file, pgoff_t start, + unsigned int *cur_progress, struct collapse_control *cc) { struct folio *folio =3D NULL; struct address_space *mapping =3D file->f_mapping; @@ -2370,6 +2383,12 @@ static enum scan_result hpage_collapse_scan_file(str= uct mm_struct *mm, unsigned } } rcu_read_unlock(); + if (cur_progress) { + if (result =3D=3D SCAN_PTE_MAPPED_HUGEPAGE) + *cur_progress =3D 1; + else + *cur_progress =3D HPAGE_PMD_NR; + } =20 if (result =3D=3D SCAN_SUCCEED) { if (cc->is_khugepaged && @@ -2448,6 +2467,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned = int pages, enum scan_result =20 while (khugepaged_scan.address < hend) { bool mmap_locked =3D true; + unsigned int cur_progress =3D 0; =20 cond_resched(); if (unlikely(hpage_collapse_test_exit_or_disable(mm))) @@ -2464,7 +2484,8 @@ static unsigned int khugepaged_scan_mm_slot(unsigned = int pages, enum scan_result mmap_read_unlock(mm); mmap_locked =3D false; *result =3D hpage_collapse_scan_file(mm, - khugepaged_scan.address, file, pgoff, cc); + khugepaged_scan.address, file, pgoff, + &cur_progress, cc); fput(file); if (*result =3D=3D SCAN_PTE_MAPPED_HUGEPAGE) { mmap_read_lock(mm); @@ -2478,7 +2499,8 @@ static unsigned int khugepaged_scan_mm_slot(unsigned = int pages, enum scan_result } } else { *result =3D hpage_collapse_scan_pmd(mm, vma, - khugepaged_scan.address, &mmap_locked, cc); + khugepaged_scan.address, &mmap_locked, + &cur_progress, cc); } =20 if (*result =3D=3D SCAN_SUCCEED) @@ -2486,7 +2508,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned = int pages, enum scan_result =20 /* move to next address */ khugepaged_scan.address +=3D HPAGE_PMD_SIZE; - progress +=3D HPAGE_PMD_NR; + progress +=3D cur_progress; if (!mmap_locked) /* * We released mmap_lock so break loop. Note @@ -2809,7 +2831,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, mmap_locked =3D false; *lock_dropped =3D true; result =3D hpage_collapse_scan_file(mm, addr, file, pgoff, - cc); + NULL, cc); =20 if (result =3D=3D SCAN_PAGE_DIRTY_OR_WRITEBACK && !triggered_wb && mapping_can_writeback(file->f_mapping)) { @@ -2824,7 +2846,7 @@ int madvise_collapse(struct vm_area_struct *vma, unsi= gned long start, fput(file); } else { result =3D hpage_collapse_scan_pmd(mm, vma, addr, - &mmap_locked, cc); + &mmap_locked, NULL, cc); } if (!mmap_locked) *lock_dropped =3D true; --=20 2.51.0 From nobody Thu Apr 2 14:14:42 2026 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9EA44310763 for ; Sat, 21 Feb 2026 09:40:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771666805; cv=none; b=kdLlfWFu4GlLn8iYlr1ag9l7s5yJk93Xwlr9HoZyle2ynsOLQ3OSQb9x2Vcp/PbgwB3ALz7Z71oGyGpI3XjNot2lCIw6J5/aNRq/tlgouj5zPOHS581VA0XErjGVjqoaCELie8hsrpVNqfHt6bXFmcGQ3gUU2iR1jFoJQEP/V5c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771666805; c=relaxed/simple; bh=1RVzURF1BxTpjwmeN4p40P0q6ytdcDqq3VQ8950kS3A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZVYUzBSoMDJnDSAoEPO3uVygQrg3+WAo+99ku7Nye9IFwne52ycFaSxTLkjAZCQyt04+yugy0g5CVlFaPDXrq9txH6YMrVDWfN3tYUCibqwdXkNzsTSMUkwuDjRsK+Ozsldd1f/WswmEzemKJ4v9nR15fk5+0bnwZfqpp+52f4Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=IjQBZ3Zm; arc=none smtp.client-ip=209.85.210.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="IjQBZ3Zm" Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-823081bb15fso1679075b3a.3 for ; Sat, 21 Feb 2026 01:40:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1771666803; x=1772271603; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Lr0amJtGkckOgt1zwhQW2YcmsdAALF/BjkLZrgfyvsQ=; b=IjQBZ3Zmf0XLgual0HBaB/u2yDMmOrZ2mO7L6vAN18QhfKBGy2OOYeOBVOFpjvIUxo AsDhg2Kg1XK3BURQoS0UPytVlNE6Z6iny7cxH/zAsHr8FEtba//YD+uAkWFung6dHGvf VDug9aD4Fnc4Gyk+dqjwCsZowKo+kzY3JHy4lneZ60NUB6b+CrJFAfq/6KhvnxBGcal+ 9juIBUoQFPeY7/ez2GwaudPzNPqEmc+RJ2e5GrDCEtUO1p8S1kEA0A/0MNqoftuoYU4P 9Z9KYQ2TSpC4XYCoycu58jalOybHFpacSPDqwZLc90x1qNniXpfXlxM57aPCvtaqCbmG tK5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771666803; x=1772271603; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Lr0amJtGkckOgt1zwhQW2YcmsdAALF/BjkLZrgfyvsQ=; b=UC5R6/ooC2yyyjjVZpHsXJhuMcXpU4e5i3jPCoVBxjOFuvzpZA/d6Cyx32ig1IlyFa 3lCafXZQZpiDOjcio87Jr0hTHTu+wyjxeZ3pV0IHHZa5S91L4/kRYYU9wCDvolhr6dkg Nwka9a6E/1sZ7PcuwJRS0PwC+1ze4BafKZbVzKV5wfZIAHJ761+VPBlOfpsHD7TxsGNG PsHL8OoTDtSaAjF6tPDbf2+Ezllk3x508kBQhxmV4egosPowJ9cJ0T4mXcGRSwjgLWyd oct3CUrWQmlPdwWToUA5mzgkOUz7ecvroYLbzjeEj8YgyLiFZvmULVtnCaDKAkUMS3O4 gueQ== X-Forwarded-Encrypted: i=1; AJvYcCUmPhYbrjH7QilAeTrad/QZzkFC5L9+DjpYduaV0HizT2faJNeB8QXGKavVsVxJq/aQExDQNKtHTO1EkGs=@vger.kernel.org X-Gm-Message-State: AOJu0Yxxp2a3jvyV1dQgFzPiV+y/qjZd8ibHPcBhe8P8JkD7TICVM8DV cAs22lLf6wDQodfvHNKOYw/mzh7pHwu7RjEghZm84qfe8q2N8iVL2aC+ X-Gm-Gg: AZuq6aK90lP2w8s4AVvkgotLX32MCNtav97kGW7tLq0a8JG3KYFSIu11+fc60PUr3cJ fbxAitUyxGyEnd8DqJyMIoDYUViSW6n6UibaXRSb/gfIVe98K3zq2sCqfAJ0dPxPPAha+JuUSyM DYbwAvQhdHsgctPrPLPgWPOg4nW4fKv+0hMw3jSydI7xzW4jP2FFn/RF3BZmhSwx3IQZ11qpPA3 sb7tOX+karDvmI20+llMTL/YbUoVZRNFrlv1UzmpCcDDPTXXFG2rmATg/NeMnqzYFX1I3CuY7Uz /W9A4FyclO9D30GEVn4xH2vfkK2F13ZCO7yBeiKvkBihvxuYYHkF455AhQiE4Dk9Ej11rBhcu4q GU9COigi7Xxd2xhp/TtqBY9YrxJhXJLCk7qlnqD4BmgOgcarnbyeQ8JLj/ZpfkqkqU0nVzmG2c8 bZC6OIbWOmcb4NM3P5ygU6acGcbCaDSeP0Jg== X-Received: by 2002:aa7:88c9:0:b0:81e:f623:ba04 with SMTP id d2e1a72fcca58-826da8e2f04mr2478100b3a.13.1771666802917; Sat, 21 Feb 2026 01:40:02 -0800 (PST) Received: from localhost.localdomain ([49.79.21.101]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-826dd8ba11bsm1761708b3a.50.2026.02.21.01.39.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Feb 2026 01:40:02 -0800 (PST) From: Vernon Yang To: akpm@linux-foundation.org, david@kernel.org Cc: lorenzo.stoakes@oracle.com, ziy@nvidia.com, dev.jain@arm.com, baohua@kernel.org, lance.yang@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Vernon Yang Subject: [PATCH mm-new v8 3/4] mm: add folio_test_lazyfree helper Date: Sat, 21 Feb 2026 17:39:17 +0800 Message-ID: <20260221093918.1456187-4-vernon2gm@gmail.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260221093918.1456187-1-vernon2gm@gmail.com> References: <20260221093918.1456187-1-vernon2gm@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Vernon Yang Add folio_test_lazyfree() function to identify lazy-free folios to improve code readability. Signed-off-by: Vernon Yang Acked-by: David Hildenbrand (Red Hat) Reviewed-by: Lance Yang Reviewed-by: Dev Jain Reviewed-by: Barry Song --- include/linux/page-flags.h | 5 +++++ mm/rmap.c | 2 +- mm/vmscan.c | 5 ++--- 3 files changed, 8 insertions(+), 4 deletions(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index fb6a83fe88b0..0426cac91c0b 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -724,6 +724,11 @@ static __always_inline bool folio_test_anon(const stru= ct folio *folio) return ((unsigned long)folio->mapping & FOLIO_MAPPING_ANON) !=3D 0; } =20 +static __always_inline bool folio_test_lazyfree(const struct folio *folio) +{ + return folio_test_anon(folio) && !folio_test_swapbacked(folio); +} + static __always_inline bool PageAnonNotKsm(const struct page *page) { unsigned long flags =3D (unsigned long)page_folio(page)->mapping; diff --git a/mm/rmap.c b/mm/rmap.c index 0f00570d1b9e..bff8f222004e 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2046,7 +2046,7 @@ static bool try_to_unmap_one(struct folio *folio, str= uct vm_area_struct *vma, } =20 if (!pvmw.pte) { - if (folio_test_anon(folio) && !folio_test_swapbacked(folio)) { + if (folio_test_lazyfree(folio)) { if (unmap_huge_pmd_locked(vma, pvmw.address, pvmw.pmd, folio)) goto walk_done; /* diff --git a/mm/vmscan.c b/mm/vmscan.c index 6a87ac7be43c..9ce3f54f43b8 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -963,8 +963,7 @@ static void folio_check_dirty_writeback(struct folio *f= olio, * They could be mistakenly treated as file lru. So further anon * test is needed. */ - if (!folio_is_file_lru(folio) || - (folio_test_anon(folio) && !folio_test_swapbacked(folio))) { + if (!folio_is_file_lru(folio) || folio_test_lazyfree(folio)) { *dirty =3D false; *writeback =3D false; return; @@ -1508,7 +1507,7 @@ static unsigned int shrink_folio_list(struct list_hea= d *folio_list, } } =20 - if (folio_test_anon(folio) && !folio_test_swapbacked(folio)) { + if (folio_test_lazyfree(folio)) { /* follow __remove_mapping for reference */ if (!folio_ref_freeze(folio, 1)) goto keep_locked; --=20 2.51.0 From nobody Thu Apr 2 14:14:42 2026 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 44DF0342521 for ; Sat, 21 Feb 2026 09:40:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771666809; cv=none; b=Hfj+yQJ5k8pDE1uN+xhwxByT0Zgrn540pIX8gSt2mn3KYkh1SBHD4GNWVtbmSZT3Aa+uOH0lQQvGzhWclFmiptgGmWxffzsIvEOYbEeDd4IZbe2rIr0oGzQp8wpTHybDmqh6zZzwOLGas0eGVvDnflVKMDbvXNOz7dYixClFjeo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771666809; c=relaxed/simple; bh=g2pZ9aWWz5k5+nKkY2aZc9wtDJl6I/IrJ4T6R9qWr8g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LYh6DB4zzHFH0gSz1WBcrl6AgbG3peYsM5n+HdKZUQy0ly4chL7nBHtaqkFMLGwQL/9Ne2KthvXa8JEhe38tQ9+P5gfPTDuf8fd9BbR3+/zypmJjpIAL6BwZXKBN9CY1X4iQ77lTSxhDbFpfNmPWJMddUJHOIZEChP/1oSPJ3h4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LHjPrW6b; arc=none smtp.client-ip=209.85.210.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LHjPrW6b" Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-8217f2ad01eso2887692b3a.2 for ; Sat, 21 Feb 2026 01:40:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1771666807; x=1772271607; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=edzhGk8tZ/b1bwNazhyh0vnVmdmocc6fgdNviuUpG8U=; b=LHjPrW6bcHt2LG0mBUU23Z+sG2h7KB9cfl2cpoo9t8+u+19uY9T+UF8ERnmCX5beOD kzORVtvyzRiHBa7MoN8IMfsCryxYtCKFhBYYDFkNROPcHkbuz+DR7FxhW7/3cJ+PQ09/ CJZFdFJmFmdkH814uUxkK24T3AMXpRtYWNOGBg5ZeTGSp9q2VHfOQx/yIoHXf9pM94jR 2lL9IaAaIWg2LrAqnHV48m0XPd8vp33D+krBNQrM4GVzyUnVdZR7Fu6H9NXpdWZCva65 0E2JuJl9aVKmUyvk1umiY2gSmI3hsp5RjBMqMBfaH7FjyJjO5rVSwqrvj5OTU+k2rOSl abPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771666807; x=1772271607; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=edzhGk8tZ/b1bwNazhyh0vnVmdmocc6fgdNviuUpG8U=; b=re1tsVAK5P3iy34Ae+74A9BxtIqjKjN+hkNBichME7FxPjLo8GivnmcA+a06AmtVLq tdrZZQp6TmIRUIpCIiKPBtm5TBsE2O7U+AvHTrRNYAIpdbTA2AET0tDRIBxopCtPXbTu B3yyRtguuJEnivgLAXH+yRbL80Ht/iHaWXVtkNY/sak+07uuQthEGV1MgyjSkMzWO+Wi N5Ur76lJLS6EjcD2FfPFz0vRHUyUCg9YyZftiOaKGC7FsH8fWq8Fxv5roOBJf0yj9s/3 Z2Fp9X2H3IxE+TYs4fsHV5a6CcqwNmoaUQp+6VOlQvCGeZQxvO6ZXxtLGBvx9sUH6Equ cpAQ== X-Forwarded-Encrypted: i=1; AJvYcCXSXyM38p5LssfNAV2pJXdllp9CdLa1UMJt2FIDKaOoO0YDurO9l5LPDI5B51Ec6VzyitnMzLWnRHic7vo=@vger.kernel.org X-Gm-Message-State: AOJu0YzhqUv+KyG5N5rh4D+bQIRZMrtsdWAbRt09d6+wl1wRZ5KrANxC 8ynxVJMnnYHzVGdIyEqV11fngBbnOEaXeuWUCbaO3zDMWmeCvbtBHRR5 X-Gm-Gg: AZuq6aJhZBXBsCMnMpHc1VZfUadKrZRVEUcBtdmm/o5sgeDnTb65wVmuBhoVb+lElfA iwOShvzunL/AGwEktGZDC3rJ7ZCE2ZAXge2JBfyMaKtMiccdwe4Wi/TxGMLHbLM/G9222g0qtUf okW+p2PuvtcNPM4+2INvu1QL2ebYuugRcLNXhzIx0MjCdZo6ewx6sJKUNC7JdEZPHuAgQJIQP1D Pqdh296VkDXbAi8zj4S+6WGneszPQmzmkIHXpAA8x9lCx7H2l0FSLe6BwZ8EBNAKet/tWX2nqvU L0+91Lo6obPW1IT4XmH1qJsiivWe+u2mJdDAsGIrQqkI7SbYQ9gw7hsYdQ15SsIZA8I8QYFzDak gbd/kpnnpzaQ6aFynpopKhOcdM4b3g3sD7Z6BRhUnDeswLIZnlT/rFQGWvsakRXzYqb8vdFdapu 4zjaI6Lipyc5K/sDF+TesVi/R0KkZvwcHaw/mYtHvcoSHO X-Received: by 2002:a05:6a00:450a:b0:81f:3d13:e07e with SMTP id d2e1a72fcca58-826da9ee90dmr2286218b3a.41.1771666807525; Sat, 21 Feb 2026 01:40:07 -0800 (PST) Received: from localhost.localdomain ([49.79.21.101]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-826dd8ba11bsm1761708b3a.50.2026.02.21.01.40.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 21 Feb 2026 01:40:07 -0800 (PST) From: Vernon Yang To: akpm@linux-foundation.org, david@kernel.org Cc: lorenzo.stoakes@oracle.com, ziy@nvidia.com, dev.jain@arm.com, baohua@kernel.org, lance.yang@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Vernon Yang Subject: [PATCH mm-new v8 4/4] mm: khugepaged: skip lazy-free folios Date: Sat, 21 Feb 2026 17:39:18 +0800 Message-ID: <20260221093918.1456187-5-vernon2gm@gmail.com> X-Mailer: git-send-email 2.51.0 In-Reply-To: <20260221093918.1456187-1-vernon2gm@gmail.com> References: <20260221093918.1456187-1-vernon2gm@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Vernon Yang For example, create three task: hot1 -> cold -> hot2. After all three task are created, each allocate memory 128MB. the hot1/hot2 task continuously access 128 MB memory, while the cold task only accesses its memory briefly and then call madvise(MADV_FREE). However, khugepaged still prioritizes scanning the cold task and only scans the hot2 task after completing the scan of the cold task. And if all folios in VM_DROPPABLE are lazyfree, Collapsing maintains that property, so we can just collapse and memory pressure in the future will free it up. In contrast, collapsing in !VM_DROPPABLE does not maintain that property, the collapsed folio will not be lazyfree and memory pressure in the future will not be able to free it up. So if the user has explicitly informed us via MADV_FREE that this memory will be freed, and this vma does not have VM_DROPPABLE flags, it is appropriate for khugepaged to skip it only, thereby avoiding unnecessary scan and collapse operations to reducing CPU wastage. Here are the performance test results: (Throughput bigger is better, other smaller is better) Testing on x86_64 machine: | task hot2 | without patch | with patch | delta | |---------------------|---------------|---------------|---------| | total accesses time | 3.14 sec | 2.93 sec | -6.69% | | cycles per access | 4.96 | 2.21 | -55.44% | | Throughput | 104.38 M/sec | 111.89 M/sec | +7.19% | | dTLB-load-misses | 284814532 | 69597236 | -75.56% | Testing on qemu-system-x86_64 -enable-kvm: | task hot2 | without patch | with patch | delta | |---------------------|---------------|---------------|---------| | total accesses time | 3.35 sec | 2.96 sec | -11.64% | | cycles per access | 7.29 | 2.07 | -71.60% | | Throughput | 97.67 M/sec | 110.77 M/sec | +13.41% | | dTLB-load-misses | 241600871 | 3216108 | -98.67% | Signed-off-by: Vernon Yang Acked-by: David Hildenbrand (arm) Reviewed-by: Lance Yang Reviewed-by: Barry Song --- include/trace/events/huge_memory.h | 1 + mm/khugepaged.c | 13 +++++++++++++ 2 files changed, 14 insertions(+) diff --git a/include/trace/events/huge_memory.h b/include/trace/events/huge= _memory.h index 384e29f6bef0..bcdc57eea270 100644 --- a/include/trace/events/huge_memory.h +++ b/include/trace/events/huge_memory.h @@ -25,6 +25,7 @@ EM( SCAN_PAGE_LRU, "page_not_in_lru") \ EM( SCAN_PAGE_LOCK, "page_locked") \ EM( SCAN_PAGE_ANON, "page_not_anon") \ + EM( SCAN_PAGE_LAZYFREE, "page_lazyfree") \ EM( SCAN_PAGE_COMPOUND, "page_compound") \ EM( SCAN_ANY_PROCESS, "no_process_for_page") \ EM( SCAN_VMA_NULL, "vma_null") \ diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 61e25cf5424b..e792e9074b48 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -46,6 +46,7 @@ enum scan_result { SCAN_PAGE_LRU, SCAN_PAGE_LOCK, SCAN_PAGE_ANON, + SCAN_PAGE_LAZYFREE, SCAN_PAGE_COMPOUND, SCAN_ANY_PROCESS, SCAN_VMA_NULL, @@ -574,6 +575,12 @@ static enum scan_result __collapse_huge_page_isolate(s= truct vm_area_struct *vma, folio =3D page_folio(page); VM_BUG_ON_FOLIO(!folio_test_anon(folio), folio); =20 + if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) && + folio_test_lazyfree(folio) && !pte_dirty(pteval)) { + result =3D SCAN_PAGE_LAZYFREE; + goto out; + } + /* See hpage_collapse_scan_pmd(). */ if (folio_maybe_mapped_shared(folio)) { ++shared; @@ -1326,6 +1333,12 @@ static enum scan_result hpage_collapse_scan_pmd(stru= ct mm_struct *mm, } folio =3D page_folio(page); =20 + if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) && + folio_test_lazyfree(folio) && !pte_dirty(pteval)) { + result =3D SCAN_PAGE_LAZYFREE; + goto out_unmap; + } + if (!folio_test_anon(folio)) { result =3D SCAN_PAGE_ANON; goto out_unmap; --=20 2.51.0