From nobody Sat Feb 7 19:45:34 2026 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2EC847E777 for ; Tue, 9 Apr 2024 08:26:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712651218; cv=none; b=Fbw1OV2tfRpOnGCU3u5MRH3dtDYBSdobT9W1W+mCAgywEUILDqC4KWMMopCPoptBRLtBeu7qZEctEQDf9m9Fxcp84Y8ZJRTdQBvUQbim8vrVNmECC9TV/90PQq9eJW77yAmaE1rwkinr7/khV3w1BKVaA/1a5wdRoe0J4ecGPKA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712651218; c=relaxed/simple; bh=ZVVZQhufHZgVYRPoViiMgUgHDRKoZRsicNrqLop/teo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=grgF4NKNskq4tBgDzafFAvkdvchC4rifvJGjMu87DbGW406Xiei3R5xJBuDUPD0nvZtzdD/cmnhdTammLbkhTsUtHQ326cqu9QuCDhc/ODYAXd3DryHZ20mhbjXXhnt/aTEzSg9kTh3+oiln67Y6awhgpsQA4MoPWY+u2pjEX3Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=J2Ghk5W2; arc=none smtp.client-ip=209.85.210.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="J2Ghk5W2" Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-6ecfeefe94cso2614593b3a.0 for ; Tue, 09 Apr 2024 01:26:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712651216; x=1713256016; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ei1Crh3xKYkUssDLy2Y3ak8IjHOYPwPhmj2M8HC/g7Q=; b=J2Ghk5W2gCEB0aYkxGY2E322Cjo94GTTUh0IJMz9+MLDbJb7m40yi3aAdRdajuqDD/ mrlcCOps9dNdzV4Qikq1ouFrVvsQUStKNWaCvuwWXkSnBUuJI7uS+xtmTrm7bi/hLv1E BshgUb90wMhKiML0Q3BZ63jdNZqFXMB+IsuFpZXP9ekabxWdGd9NLawnyQR0IX5sv8Ry eipjO12gFqXzOi+tCTABUKexhnpcnkaZdqYjIL4CBVXYIwT8yzgI76eNnT/QPDhYScND 2JMXz4sUwgj8OIoz/51CZJBw3XYRTC8KLGn16cqFICuQzJ5kbZgOjV2HdDCHFC+47mAu Wgcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712651216; x=1713256016; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ei1Crh3xKYkUssDLy2Y3ak8IjHOYPwPhmj2M8HC/g7Q=; b=ksnQ8nm3eFyosLmWbeCx+ZNZgJm3MULNEEJHmx4icbRl65FQFzDaJ6tvSLo5oh6UyR tyNq1sz+4irgijBm5/LIZ0H3lO/rvmAi7G2W+6elfqtERjr40bhlpkbBz4xEcFE38UUf dAjE8GeDzUu87yCqUIR/baT4BMAQku/hNR9NFRtahVoqMohBBdyCG64nAkwguheoZwIY Y0i2Eh6PO4Y5wRKxkiOocoz0/Wct769N53gj3MmIwHu5lCti5ZCwu7NDJkxNcVba5umf 3AHWeGkuN8GpcUAUfRvwV6CWWiGN0VcexJoa+5mUqDQXyjRMo6p3pHMpgOgiDaIM5ME1 T3Gg== X-Forwarded-Encrypted: i=1; AJvYcCVk3GnYitxYAUnLZyiWEWUD1B3fRzbVnuLfEXWOSdGnrikN7Kyz5u13ScdHPyFhkmjVbtf2dbHXte3FbWHMKHDG7+5FYfQunuQhtkEu X-Gm-Message-State: AOJu0YwTqQiv3pqvYonUUaVtTL2ucqlZOgP4MFWpL6SECrIjRdIQPoWo 4Ma3gHRyWzLvrfzxrZxebiEio4MJRRRMQQEEnMAfQHpjhM+S5WNRs9SdR7m9GTM= X-Google-Smtp-Source: AGHT+IHZGO/8Oq5G4AI8d7uAa0Xfd/UfJnN/mN1F1Zgkmt5jIyqoC0gKP++Z8vrGbui309CoYrBrbQ== X-Received: by 2002:a05:6a20:430c:b0:1a7:5413:1e6d with SMTP id h12-20020a056a20430c00b001a754131e6dmr3319733pzk.4.1712651216427; Tue, 09 Apr 2024 01:26:56 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id r16-20020a632050000000b005f05c88c149sm7594238pgm.71.2024.04.09.01.26.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Apr 2024 01:26:56 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 1/5] mm: swap: introduce swap_free_nr() for batched swap_free() Date: Tue, 9 Apr 2024 20:26:27 +1200 Message-Id: <20240409082631.187483-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240409082631.187483-1-21cnbao@gmail.com> References: <20240409082631.187483-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Chuanhua Han While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Reviewed-by: Ryan Roberts --- include/linux/swap.h | 5 +++++ mm/swapfile.c | 51 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) diff --git a/include/linux/swap.h b/include/linux/swap.h index 11c53692f65f..b7a107e983b8 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -483,6 +483,7 @@ extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); extern void swap_free(swp_entry_t); +extern void swap_free_nr(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); int swap_type_of(dev_t device, sector_t offset); @@ -564,6 +565,10 @@ static inline void swap_free(swp_entry_t swp) { } =20 +void swap_free_nr(swp_entry_t entry, int nr_pages) +{ +} + static inline void put_swap_folio(struct folio *folio, swp_entry_t swp) { } diff --git a/mm/swapfile.c b/mm/swapfile.c index 28642c188c93..f4c65aeb088d 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1356,6 +1356,57 @@ void swap_free(swp_entry_t entry) __swap_entry_free(p, entry); } =20 +/* + * Free up the maximum number of swap entries at once to limit the + * maximum kernel stack usage. + */ +#define SWAP_BATCH_NR (SWAPFILE_CLUSTER > 512 ? 512 : SWAPFILE_CLUSTER) + +/* + * Called after swapping in a large folio, batched free swap entries + * for this large folio, entry should be for the first subpage and + * its offset is aligned with nr_pages + */ +void swap_free_nr(swp_entry_t entry, int nr_pages) +{ + int i, j; + struct swap_cluster_info *ci; + struct swap_info_struct *p; + unsigned int type =3D swp_type(entry); + unsigned long offset =3D swp_offset(entry); + int batch_nr, remain_nr; + DECLARE_BITMAP(usage, SWAP_BATCH_NR) =3D { 0 }; + + /* all swap entries are within a cluster for mTHP */ + VM_BUG_ON(offset % SWAPFILE_CLUSTER + nr_pages > SWAPFILE_CLUSTER); + + if (nr_pages =3D=3D 1) { + swap_free(entry); + return; + } + + remain_nr =3D nr_pages; + p =3D _swap_info_get(entry); + if (p) { + for (i =3D 0; i < nr_pages; i +=3D batch_nr) { + batch_nr =3D min_t(int, SWAP_BATCH_NR, remain_nr); + + ci =3D lock_cluster_or_swap_info(p, offset); + for (j =3D 0; j < batch_nr; j++) { + if (__swap_entry_free_locked(p, offset + i * SWAP_BATCH_NR + j, 1)) + __bitmap_set(usage, j, 1); + } + unlock_cluster_or_swap_info(p, ci); + + for_each_clear_bit(j, usage, batch_nr) + free_swap_slot(swp_entry(type, offset + i * SWAP_BATCH_NR + j)); + + bitmap_clear(usage, 0, SWAP_BATCH_NR); + remain_nr -=3D batch_nr; + } + } +} + /* * Called after dropping swapcache to decrease refcnt to swap entries. */ --=20 2.34.1 From nobody Sat Feb 7 19:45:34 2026 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3FE277E777 for ; Tue, 9 Apr 2024 08:27:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712651225; cv=none; b=LZR95tm7RrqdVfmah885tOXptSjW4HYznXSNqCDIrSMbImqdsh/ePmMdZOKd/EmdTtgei51J4ztLowmr+lW1nIoy5U70Va9xZtvWF4vtrKvmPzDSSW9w0zDmsjp1/hep2pBI4vd5PTjK97aDlHa/LqNqnyNCHO0NJy/kVfNy5Nc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712651225; c=relaxed/simple; bh=ETs94iqJVrHJRgz2q3MMrYy7MrAw5n0uJJNve7xN6Vc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=mVW3jNxaYSjOgtLsEk0hIzBwrVQSSjl5aJZBNwzoXAZpXSGpQ3aQCDIe3SDm8X9/9FqBFBXGPhf3t5EOZ7PLXRdWNoKfPCoqlv3WGwbikNIQ6AUR3ax4LFAjwzFrA3CY5szTyjmsJcI3+PstF1ChEvAwhpKBG7+xUoNxVMpXiDw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ag3lGwdW; arc=none smtp.client-ip=209.85.210.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ag3lGwdW" Received: by mail-pf1-f172.google.com with SMTP id d2e1a72fcca58-6ed267f2936so1561892b3a.3 for ; Tue, 09 Apr 2024 01:27:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712651223; x=1713256023; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GpjEhAQtgqO+DLm71KCeo2y5Y8KyQFcwcH3a5IWajwE=; b=ag3lGwdWc6YYsw/CUW1Xsv/oHfwC0QSSFCwZ0fsAp7v4g2TnbtaNOhnAjYCtILTtzm sZ+5c6ysXbZEJhHX1aq+npXHk+UqBGFiUT/v3b+KZ+OvHQGXx/4dCzhMAuLs5NVNtgRM YyV+Jogoof1E4Xa98XrHehxaGOeY+58OzcPIVr5vm9o3+NikxHU4Qdpjng4A8ow3sAdW mxsH61jkY3sYTxPflfJ1BTrrfcuOnGOoeEZsWKiqmpIPtCeef2IBSHb1ugK9ik0X4Qb4 TMjcu23agxAs1Cz4YPysXsWOVL58epyurVduXWLFf6ZLrL2CHdb+vbRn7VugobyuUz26 uKZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712651223; x=1713256023; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GpjEhAQtgqO+DLm71KCeo2y5Y8KyQFcwcH3a5IWajwE=; b=BNBiT1PZQmeQ7yHikUBMyUiDVEEyRQ6nAVdGtfBOVHfRlpxCMEsA7AJi3JiHqrtHuL JVEkHZwwULZiZHaoFQNjyedu9apAMEOZQ41XMq8kHALcmS3T3OuTNvMFQZhLsmrwqpiy uaZYyuNPiGyLhsBIJgkjWL00C+DIJ/WqRR80fWHcBjXPWY8D34okda1VPa6VhOcQf/6G 650bdn3T7woNk1clUtHOAygWBKD7EpfHhV/nMmo0gDYYa1yCNWQNTSVQVNK3u3hLkj9g bHUtXpHLGfqxHQlab6LC7YeEV+tvRQ4s3/XnHXbybqIi0qphBqmNXGDEwO1F523pBaLH BsrA== X-Forwarded-Encrypted: i=1; AJvYcCWZ+ls24MdE6j0h2yZhNbamZciPc3pr3E2bpBXjeYs+jr4jX/h4DEn39LpviDnSLfE2jtC46xZInEC4rhDb6E7HWvVDgQsmkv33jTwK X-Gm-Message-State: AOJu0YxED2dEeL0Ee5+j1RheI3F9FaZlZZmAf0dltUJfjE/VgxY8ww9/ DXUn/FBvaY8mKKFhgKOT2aKBXJ5k2s+KYUEnGt0Qb+u1jDdjzQ/u X-Google-Smtp-Source: AGHT+IGLXAZ+gunF7hyCXzEaTPUaO18uUbecCeYC/lMFr9OrqEyLkgxoorBwDY+pMLEp+CyQ1vQJpw== X-Received: by 2002:a05:6a00:4f84:b0:6ea:afd1:90e6 with SMTP id ld4-20020a056a004f8400b006eaafd190e6mr11705280pfb.6.1712651223528; Tue, 09 Apr 2024 01:27:03 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id r16-20020a632050000000b005f05c88c149sm7594238pgm.71.2024.04.09.01.26.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Apr 2024 01:27:02 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 2/5] mm: swap: make should_try_to_free_swap() support large-folio Date: Tue, 9 Apr 2024 20:26:28 +1200 Message-Id: <20240409082631.187483-3-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240409082631.187483-1-21cnbao@gmail.com> References: <20240409082631.187483-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Chuanhua Han The function should_try_to_free_swap() operates under the assumption that swap-in always occurs at the normal page granularity, i.e., folio_nr_pages =3D 1. However, in reality, for large folios, add_to_swap_cache() will invoke folio_ref_add(folio, nr). To accommodate large folio swap-in, this patch eliminates this assumption. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Acked-by: Chris Li Reviewed-by: Ryan Roberts Reviewed-by: "Huang, Ying" --- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 78422d1c7381..2702d449880e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3856,7 +3856,7 @@ static inline bool should_try_to_free_swap(struct fol= io *folio, * reference only in case it's likely that we'll be the exlusive user. */ return (fault_flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) && - folio_ref_count(folio) =3D=3D 2; + folio_ref_count(folio) =3D=3D (1 + folio_nr_pages(folio)); } =20 static vm_fault_t pte_marker_clear(struct vm_fault *vmf) --=20 2.34.1 From nobody Sat Feb 7 19:45:34 2026 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1F3B97F46A for ; Tue, 9 Apr 2024 08:27:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712651232; cv=none; b=q41fG2vl5eFDQzZV/hrIG4PxpwAJlSFpRXNHEuveXmnDjFHZvExFFNJUEG7blB2J75W5WqLF1iPgznt6TueP+BKxfor6YUS2x03lELOi9FLk3zbTngyDQTzgmdx2YpBNE5yVLPir5sJLGyug3E2wxSoymwSaMhk97QToOPzFt3g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712651232; c=relaxed/simple; bh=D+Ww0cHxH7lNQifArSTX5yyjcFVuKM5MIm3KMR2gH30=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=j/cPn+gwvAVtri9nix8HjO1Cu42sKb3N10vShz9tV8Tu5ArtTNZbBEbVOqnwiGUK7M4hCnIDk3ALxDANM9uOyyhZGKc5krW4phozvVPahen0dDo5tv6wVz07T/vxaKJ/GSPdT+iwL2D3YAzRPNUaALCPHV5Z75k6zlPU/hkDc0M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Uq0NgBcp; arc=none smtp.client-ip=209.85.216.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Uq0NgBcp" Received: by mail-pj1-f45.google.com with SMTP id 98e67ed59e1d1-2a52c544077so1216802a91.1 for ; Tue, 09 Apr 2024 01:27:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712651230; x=1713256030; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9ZXg/AhadeS891XoBBBMkIgTpnMXwbdxyyz0Yc4z5x8=; b=Uq0NgBcpYXI3lH/FTTF0NJI/iuwRHrRENxNLK0nDkygKeYShFDRAmii5YqmrjDigom 2AUhJTBei8SCUjxhtTgZElYaO2XrEDxKXFrG0JrQYp/41U6BYkd0BQBm1FN6VB5uBuil BCYwnensFDXNmPR39WF2PGkySE7H/GsoPpUTjj13fhEFLV0lbYUK6QE0XUQcODSm9RLI YUgzPrQMBxkwV849TSRqO/S8eqPsEpGFs81MlruSnntDyiJ4dti5b/+cbxDkINRPLymf 8L9JOktRBANxoKdhwbZcI2L2fNmCA/FBLx1rE/1aR7NYPZNxXbatm0GruFZBg5bwnYoq IX7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712651230; x=1713256030; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9ZXg/AhadeS891XoBBBMkIgTpnMXwbdxyyz0Yc4z5x8=; b=LvPpauAdyf1PbCOO1BTvNb+4n8XCBBWDpiaVkZXVVyR4HcseSHdPzPsMdWVOJeDE+F /dAV2hsdKZI9CGQZEqaeCMLwjSHQGtiP8wOgkgvJ5JIPthO/H21DsW6RPo1YnwbTqwqW Q2CfgcveNW8ivhIzrESmZytLiUrMBaaH2T7WjbvPlCynnKMbhgw9HhopiNqn5/CZrfht LQxAPB2N1J8HwU7IFry0GSmDmtVJM7zRw6/YtmMCTBdStasK52T5fy4ZaFm9Lu08ggjS TVfHiJ7ZrFHBhprNzVnVO54lGF65jf7N7Zej2NoXN0KL0q4faXXRxed4QAAW14L+btJg vFpQ== X-Forwarded-Encrypted: i=1; AJvYcCXpxRWeuiBmx4a3PwyNplFWYbbu957q7v/wv7Btdrd7wubzTpEqv8Uap9jdlG5uIZH1+wTU55izUUM+kxNkmql0abnMWf0BvikIteag X-Gm-Message-State: AOJu0YzYaZc7P9CtVJV3c4I5jRBcwETyXTfKI9PrihYNyGKimDMunEGf 02TOP6UDHTJwkeVy9eO9zXXbLGr26YgbSzIk+GKf9TTdr7AJtbEL X-Google-Smtp-Source: AGHT+IFg5AbIygO1G/4Sf/xhmF54Nd1zJbM7cGgBTgS8FOC9fn0UWuAxj2wjNfCbUfjBKgj68N+xMw== X-Received: by 2002:a17:90b:3587:b0:2a4:e962:b62c with SMTP id mm7-20020a17090b358700b002a4e962b62cmr4228631pjb.49.1712651230311; Tue, 09 Apr 2024 01:27:10 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id r16-20020a632050000000b005f05c88c149sm7594238pgm.71.2024.04.09.01.27.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Apr 2024 01:27:09 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 3/5] mm: swap_pte_batch: add an output argument to reture if all swap entries are exclusive Date: Tue, 9 Apr 2024 20:26:29 +1200 Message-Id: <20240409082631.187483-4-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240409082631.187483-1-21cnbao@gmail.com> References: <20240409082631.187483-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song Add a boolean argument named any_shared. If any of the swap entries are non-exclusive, set any_shared to true. The function do_swap_page() can then utilize this information to determine whether the entire large folio can be reused. Signed-off-by: Barry Song --- mm/internal.h | 9 ++++++++- mm/madvise.c | 2 +- mm/memory.c | 2 +- 3 files changed, 10 insertions(+), 3 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 9d3250b4a08a..cae39c372bfc 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -238,7 +238,8 @@ static inline pte_t pte_next_swp_offset(pte_t pte) * * Return: the number of table entries in the batch. */ -static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) +static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte, + bool *any_shared) { pte_t expected_pte =3D pte_next_swp_offset(pte); const pte_t *end_ptep =3D start_ptep + max_nr; @@ -248,12 +249,18 @@ static inline int swap_pte_batch(pte_t *start_ptep, i= nt max_nr, pte_t pte) VM_WARN_ON(!is_swap_pte(pte)); VM_WARN_ON(non_swap_entry(pte_to_swp_entry(pte))); =20 + if (any_shared) + *any_shared |=3D !pte_swp_exclusive(pte); + while (ptep < end_ptep) { pte =3D ptep_get(ptep); =20 if (!pte_same(pte, expected_pte)) break; =20 + if (any_shared) + *any_shared |=3D !pte_swp_exclusive(pte); + expected_pte =3D pte_next_swp_offset(expected_pte); ptep++; } diff --git a/mm/madvise.c b/mm/madvise.c index f59169888b8e..d34ca6983227 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -671,7 +671,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned = long addr, entry =3D pte_to_swp_entry(ptent); if (!non_swap_entry(entry)) { max_nr =3D (end - addr) / PAGE_SIZE; - nr =3D swap_pte_batch(pte, max_nr, ptent); + nr =3D swap_pte_batch(pte, max_nr, ptent, NULL); nr_swap -=3D nr; free_swap_and_cache_nr(entry, nr); clear_not_present_full_ptes(mm, addr, pte, nr, tlb->fullmm); diff --git a/mm/memory.c b/mm/memory.c index 2702d449880e..c4a52e8d740a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1638,7 +1638,7 @@ static unsigned long zap_pte_range(struct mmu_gather = *tlb, folio_put(folio); } else if (!non_swap_entry(entry)) { max_nr =3D (end - addr) / PAGE_SIZE; - nr =3D swap_pte_batch(pte, max_nr, ptent); + nr =3D swap_pte_batch(pte, max_nr, ptent, NULL); /* Genuine swap entries, hence a private anon pages */ if (!should_zap_cows(details)) continue; --=20 2.34.1 From nobody Sat Feb 7 19:45:34 2026 Received: from mail-oo1-f50.google.com (mail-oo1-f50.google.com [209.85.161.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3FEEF12880F for ; Tue, 9 Apr 2024 08:27:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712651240; cv=none; b=CTUNhhs0bp0uNTvcgB0pk4122J0IJBbtZ/RQdVjg1doe8RFr+80XxhOTQzzZte+zGmNo/N5+Uv/pyLyPMUX7AwIqVXL1+CjetZWDXYlQJv+pGEAy+Do3UQAxFeZU+AmquJMI05XR+jpGwwE073MZJN8C5++MAUrItYnR3tPCOq4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712651240; c=relaxed/simple; bh=itUkWLRcyPpOMmWElcAjrAh9Q4g+IKjYc56ZFGUD5Bw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=bw0EPSGLsIXhMZQV+bg4Q84S7dtqUduHjS3b/InEb5qjmjaEznPkQTGajO/kxdyZGWVInP9+Jz0riI0UacesXoPdFtNvIO7Sj0hxvsfXAJ2+R+PUBfXxtLdf8eHSemN4NfgpvD9lNyW5CaH9A88cZEu4oynawEwaw1/sti6UuZ0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=CUsNWrnH; arc=none smtp.client-ip=209.85.161.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CUsNWrnH" Received: by mail-oo1-f50.google.com with SMTP id 006d021491bc7-5aa1b7a37b5so1725091eaf.2 for ; Tue, 09 Apr 2024 01:27:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712651237; x=1713256037; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pRaJtzo5cg/VrfJljXw5TEaEgpb4vopERkweBTeE6OA=; b=CUsNWrnHx/Gh7NkMXM3Jt3H4B1Ckegy5CJA0YLlpI0CneWWQpRg8i+Q3gFkIVs2HLm 5n1WLukCzUSiNAGWbb4I9eCS9Aj9gvjOGUEoOWcjpmatQM8bFnQT7EaAuRAuiZ3EQgOd mMOdntJ4KGa2LhG45NNtbYW8fkl94d+LVDRZrTKVCDViKzyiapTxAQmVOw7pqPnynhPw 0PsxcsKFrWVUpAVCgeYeQPhwoCyxXTcvG0SvMvp83w5qnVRfkPZ0BqK6hrz3MFZ4Sb7R DvhIPB5WgMrw3VFkt8noULy1JzSuMon026hSrDf5BnhriN+Zp3fPD1njS0plCg8apJHA 156w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712651237; x=1713256037; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pRaJtzo5cg/VrfJljXw5TEaEgpb4vopERkweBTeE6OA=; b=G5A0kHAkgyo1LETZE/rEw9ADSOr4rmKpWz7wMZago+cWSiguDj8zpUD4drSNxK/vB+ 33YWD7arZ7DfPJzx8irfomO3X6dTHs801AkiZw1qaiAt1WKLkH1r/YhlwZTWjSdV1a5r jfJtbqeEAfcNr2D1lPs1kuWIv6mP8rNFZRIkjvfHyKqsBSyaYOopMp0RFQ5tYRB1mKIC shSgiQlY5Hhy7Lg2svZBFuU4nHs0QqvqcggklTjCYlkf+4HX6RDRfBuxmCakSnjcj56I qRLBJ3y93L/wTcrl1EYRT8PJYymg5zAUVKPcd7nXmqs11XqHpzeOYgZ2eL4SINMP+2RC Dg/A== X-Forwarded-Encrypted: i=1; AJvYcCVyup4Kb3+OV0oFp/mIIn8/pjnpvm92tDgKbtIkutOj6smXyDUZvcTGzmn5ILO6Ix7n0oRbMm6HaPWt4noamVSDa2csckdtf99tuT55 X-Gm-Message-State: AOJu0Yz7J1lNIS67orYcbFo3RPmjWaLPnyMfvI/0ndFTpXHniJjDC242 nW0A9rgRXtddtWv+WiPMyzI+1aW9CBZK1UzRbcN3pXY5jr0S+9j3 X-Google-Smtp-Source: AGHT+IFvN5ohj5Cxouk2/0fAxnzfDvT6AqBlF178427aMryNUnuyIkM8ZO3KggJQq1sFBYMpf6zx2g== X-Received: by 2002:a05:6358:d59b:b0:186:45c:d6de with SMTP id ms27-20020a056358d59b00b00186045cd6demr6322611rwb.26.1712651237158; Tue, 09 Apr 2024 01:27:17 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id r16-20020a632050000000b005f05c88c149sm7594238pgm.71.2024.04.09.01.27.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Apr 2024 01:27:16 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 4/5] mm: swap: entirely map large folios found in swapcache Date: Tue, 9 Apr 2024 20:26:30 +1200 Message-Id: <20240409082631.187483-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240409082631.187483-1-21cnbao@gmail.com> References: <20240409082631.187483-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Chuanhua Han When a large folio is found in the swapcache, the current implementation requires calling do_swap_page() nr_pages times, resulting in nr_pages page faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song --- mm/memory.c | 64 +++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 52 insertions(+), 12 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index c4a52e8d740a..9818dc1893c8 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3947,6 +3947,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte_t pte; vm_fault_t ret =3D 0; void *shadow =3D NULL; + int nr_pages =3D 1; + unsigned long start_address =3D vmf->address; + pte_t *start_pte =3D vmf->pte; + bool any_swap_shared =3D false; =20 if (!pte_unmap_same(vmf)) goto out; @@ -4137,6 +4141,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ vmf->pte =3D pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + + /* We hit large folios in swapcache */ + if (start_pte && folio_test_large(folio) && folio_test_swapcache(folio)) { + int nr =3D folio_nr_pages(folio); + int idx =3D folio_page_idx(folio, page); + unsigned long folio_start =3D vmf->address - idx * PAGE_SIZE; + unsigned long folio_end =3D folio_start + nr * PAGE_SIZE; + pte_t *folio_ptep; + pte_t folio_pte; + + if (unlikely(folio_start < max(vmf->address & PMD_MASK, vma->vm_start))) + goto check_pte; + if (unlikely(folio_end > pmd_addr_end(vmf->address, vma->vm_end))) + goto check_pte; + + folio_ptep =3D vmf->pte - idx; + folio_pte =3D ptep_get(folio_ptep); + if (!is_swap_pte(folio_pte) || non_swap_entry(pte_to_swp_entry(folio_pte= )) || + swap_pte_batch(folio_ptep, nr, folio_pte, &any_swap_shared) !=3D nr) + goto check_pte; + + start_address =3D folio_start; + start_pte =3D folio_ptep; + nr_pages =3D nr; + entry =3D folio->swap; + page =3D &folio->page; + } + +check_pte: if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte))) goto out_nomap; =20 @@ -4190,6 +4223,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ exclusive =3D false; } + + /* Reuse the whole large folio iff all entries are exclusive */ + if (nr_pages > 1 && any_swap_shared) + exclusive =3D false; } =20 /* @@ -4204,12 +4241,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * We're already holding a reference on the page but haven't mapped it * yet. */ - swap_free(entry); + swap_free_nr(entry, nr_pages); if (should_try_to_free_swap(folio, vma, vmf->flags)) folio_free_swap(folio); =20 - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - dec_mm_counter(vma->vm_mm, MM_SWAPENTS); + folio_ref_add(folio, nr_pages - 1); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); + add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages); + pte =3D mk_pte(page, vma->vm_page_prot); =20 /* @@ -4219,33 +4258,34 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * exclusivity. */ if (!folio_test_ksm(folio) && - (exclusive || folio_ref_count(folio) =3D=3D 1)) { + (exclusive || (folio_ref_count(folio) =3D=3D nr_pages && + folio_nr_pages(folio) =3D=3D nr_pages))) { if (vmf->flags & FAULT_FLAG_WRITE) { pte =3D maybe_mkwrite(pte_mkdirty(pte), vma); vmf->flags &=3D ~FAULT_FLAG_WRITE; } rmap_flags |=3D RMAP_EXCLUSIVE; } - flush_icache_page(vma, page); + flush_icache_pages(vma, page, nr_pages); if (pte_swp_soft_dirty(vmf->orig_pte)) pte =3D pte_mksoft_dirty(pte); if (pte_swp_uffd_wp(vmf->orig_pte)) pte =3D pte_mkuffd_wp(pte); - vmf->orig_pte =3D pte; =20 /* ksm created a completely new copy */ if (unlikely(folio !=3D swapcache && swapcache)) { - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_add_new_anon_rmap(folio, vma, start_address); folio_add_lru_vma(folio, vma); } else { - folio_add_anon_rmap_pte(folio, page, vma, vmf->address, - rmap_flags); + folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, start_address, + rmap_flags); } =20 VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); + set_ptes(vma->vm_mm, start_address, start_pte, pte, nr_pages); + vmf->orig_pte =3D ptep_get(vmf->pte); + arch_do_swap_page(vma->vm_mm, vma, start_address, pte, pte); =20 folio_unlock(folio); if (folio !=3D swapcache && swapcache) { @@ -4269,7 +4309,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } =20 /* No need to invalidate - it was non-present before */ - update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + update_mmu_cache_range(vmf, vma, start_address, start_pte, nr_pages); unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl); --=20 2.34.1 From nobody Sat Feb 7 19:45:34 2026 Received: from mail-oo1-f47.google.com (mail-oo1-f47.google.com [209.85.161.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 19C3A12F383 for ; Tue, 9 Apr 2024 08:27:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712651246; cv=none; b=trxGZ76nWFQJknYo4qa2aH8/+Zt2K+LWy+RMDrFg0TK/sixjaZ4EZVaLze9O1hSvQ/ovcvxu+SwrhozFyjT0EavMHmEVj3EhwpNNGBQP6xX6oKHh9lETYPQjpg6ZWIm226wgLfXZAMuO80TPp92ovK2/lHizjWIKG/4oeEiKddg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712651246; c=relaxed/simple; bh=NInqBO3mqli5kAaFy5yCoXoiS3x2iM4UBcvy+GAsVlo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=GrlFx6u2VuhOJy6Jm4QRWVmoqG3wRfgHQY21LNJn6RvPPMT0y4xsEo4C2JiZTS9aq9yKoXQn11r6OmQZcT+7RRJi0KNlSX5k5ZaPKhaLP3z4XbgOHvtsesjzEXPFKK6f2VmkhZwgkSheABJa6Rqm2AZkwRHog7GSYufziNgpCcw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hPz7Hmr+; arc=none smtp.client-ip=209.85.161.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hPz7Hmr+" Received: by mail-oo1-f47.google.com with SMTP id 006d021491bc7-5aa362cc2ccso1750910eaf.3 for ; Tue, 09 Apr 2024 01:27:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712651244; x=1713256044; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VvJHnREnFZ1cH5hEIu5QcKzQyYhaFholreUs1EUriRY=; b=hPz7Hmr+4DxDSD6V4Ax4T3+darDvLJjsik/x8oswVLk2+hxUQxcfYwRDK4pwgv67gs Ch3QAm+f4gr3DrtPBa9F+081nIokKqClGBQB0NXdO7KdUP+AB6yTLRAZru1pgjnoNBtW R9ohOkpd/ncn901W9pQENVOWkLrpXsCRAYDLWMkRGhQz232LUzQPAmZGkFZxo4T2C3pf ZRaOgTBhWvATd1X9zmM8q8jNF/5hgV7YaRswsTkFHyOpXLQC1sVVgTkFT9C3r5CBLVpj zTn8Y0SDTsqXmklGwNAYhUOP649W9teqEV3Ge7R5uZQqN37P1rrMb4TAzLa5fuYqa6tK +77g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712651244; x=1713256044; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VvJHnREnFZ1cH5hEIu5QcKzQyYhaFholreUs1EUriRY=; b=iAY6bCNJaXsJ0bL4mJW0r+0cBrXVl6XUXiQ0qHPqC/O20F3HhtTYk8v+jt5xDSp8u5 c4x8+O0rRwoJ8mMp6r304n9xFhve96PYI4ZRW8xhKRGq/o1V8T3HlmoImvH2v+TEdxG9 HMq1AmOa+lXVJjhX6nFOAGNLy1qF9aKhJUbiQ0INRNvbAqMD9WH4w/iTCl5ZiiYZrq6a QK91CYzguVph5NhxwUprOQss/itRfknHsSqVeavHCF1UNVj9NvqaIFrzdDmT8V6N2ZlM xBYXDkKcDdhVM4iQNCzl3UTmjeed5Yg6WssjNLOcWHxVv3lj1TNLPQnK5JvfnJIAJkOo urKQ== X-Forwarded-Encrypted: i=1; AJvYcCVo0RS/DQ2lnJb8xNZgyZQ68hgR88o4CDZMnlq+4JWg4xXpQYb6SJl3A2tzSLOhr5z6H2Vr9qs4JEUAIkENK3wpTPqsfuzZndO2ZLCU X-Gm-Message-State: AOJu0YyS52DeHbTzR+gP47r3q9cgKQWUBQXrKfeMHSNPrZk5rEoaRaxs IQjOUFj2ORqGY59rqSgSyccp4Z7FWVYgFKdHPQBlcKtTrJwPAUNf X-Google-Smtp-Source: AGHT+IFBs0lnKgGergw9nSynKWYDcWV9NAHxsayy25kkPHq1gK1xpsfE+xyR/l1fPI85a7fcICPnHg== X-Received: by 2002:a05:6358:d39f:b0:186:102b:777 with SMTP id mp31-20020a056358d39f00b00186102b0777mr9881254rwb.10.1712651243980; Tue, 09 Apr 2024 01:27:23 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id r16-20020a632050000000b005f05c88c149sm7594238pgm.71.2024.04.09.01.27.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Apr 2024 01:27:23 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 5/5] mm: add per-order mTHP swpin_refault counter Date: Tue, 9 Apr 2024 20:26:31 +1200 Message-Id: <20240409082631.187483-6-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240409082631.187483-1-21cnbao@gmail.com> References: <20240409082631.187483-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song Currently, we are handling the scenario where we've hit a large folio in the swapcache, and the reclaiming process for this large folio is still ongoing. Signed-off-by: Barry Song Tested-by: SeongJae Park --- include/linux/huge_mm.h | 1 + mm/huge_memory.c | 2 ++ mm/memory.c | 1 + 3 files changed, 4 insertions(+) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index c8256af83e33..b67294d5814f 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -269,6 +269,7 @@ enum mthp_stat_item { MTHP_STAT_ANON_ALLOC_FALLBACK, MTHP_STAT_ANON_SWPOUT, MTHP_STAT_ANON_SWPOUT_FALLBACK, + MTHP_STAT_ANON_SWPIN_REFAULT, __MTHP_STAT_COUNT }; =20 diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d8d2ed80b0bf..fb95345b0bde 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -556,12 +556,14 @@ DEFINE_MTHP_STAT_ATTR(anon_alloc, MTHP_STAT_ANON_ALLO= C); DEFINE_MTHP_STAT_ATTR(anon_alloc_fallback, MTHP_STAT_ANON_ALLOC_FALLBACK); DEFINE_MTHP_STAT_ATTR(anon_swpout, MTHP_STAT_ANON_SWPOUT); DEFINE_MTHP_STAT_ATTR(anon_swpout_fallback, MTHP_STAT_ANON_SWPOUT_FALLBACK= ); +DEFINE_MTHP_STAT_ATTR(anon_swpin_refault, MTHP_STAT_ANON_SWPIN_REFAULT); =20 static struct attribute *stats_attrs[] =3D { &anon_alloc_attr.attr, &anon_alloc_fallback_attr.attr, &anon_swpout_attr.attr, &anon_swpout_fallback_attr.attr, + &anon_swpin_refault_attr.attr, NULL, }; =20 diff --git a/mm/memory.c b/mm/memory.c index 9818dc1893c8..acc023795a4d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4167,6 +4167,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) nr_pages =3D nr; entry =3D folio->swap; page =3D &folio->page; + count_mthp_stat(folio_order(folio), MTHP_STAT_ANON_SWPIN_REFAULT); } =20 check_pte: --=20 2.34.1