From nobody Wed Dec 17 03:28:12 2025 Received: from mail-oa1-f54.google.com (mail-oa1-f54.google.com [209.85.160.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E721344C7B for ; Wed, 29 May 2024 08:28:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716971333; cv=none; b=jfHdnAWTP9ktNf77j/Gnqz5+KUWcE043CrhvwXGqFigyK7h+NE76vv7RpgnnZ/uAwxg5bGYybo5QIsosC2SmSLWJBPYXK0kacFVkWal7AtGJ+2Abmp8E2tCT313DfNRVJVhXcI2i9qr56y2rZLj/4DX9EBsyMGIp3bvfVVsWUfQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716971333; c=relaxed/simple; bh=08oobRwRz3gaF/temZ05IpZbrn5Dik268m1bTOTym00=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=sp+ViFYbDXy4pRbD4R7Cr8jHIqvNWGFkKfLiFcukOffqXstiv0WW3OKvS12eOnP6Mhjo6DP9EvhJagYapF/6TT3ULeW2eOUkswXqJH3+5MA2DXTcvdemai1O+60nwoxIUqRgScKsQ6wiPXoYCNnaqqIPRUV/d5/uueOBWyOgUyQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=ewyicS+C; arc=none smtp.client-ip=209.85.160.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ewyicS+C" Received: by mail-oa1-f54.google.com with SMTP id 586e51a60fabf-24fca68b554so727476fac.1 for ; Wed, 29 May 2024 01:28:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716971331; x=1717576131; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7k7udukJq/5lufC/QvLOUP0N512BAUWebXH46Q8HtFk=; b=ewyicS+CzMUpq+qTA46ap/qls+3Ujh28I1AS3Y+m42gAw8VCGd5W0zPs93yqo/qBMp yVudo/IJ6f4ScwAn305sa8RGtgG8oJPicMjiKVti2BB1td41PEdGs/CC/KqMabkKHpDu 0bHbNxoI6n+SsHDpiULlItiMk4yqm21kHeNjtaQybL5u2NFBG6IkDHwoSEpvWXEUwl2U KgYz20sIL8s/MbOJtuNxri9xVs82lcT9lkhcAP/ULCrukVqSuDz0skqgXwjRSDtv6Ses i+Zz4xliYeYa37CBKH95QZISOLDdq7bEpGR7Nlv3KMXSkcFsgJD0cuAV2ylbDFRW24nQ 5CxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716971331; x=1717576131; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7k7udukJq/5lufC/QvLOUP0N512BAUWebXH46Q8HtFk=; b=d17DbSM5Yx8l0VtH7d77VzqDPJKkaTGTBmR3SHfPe8pxI42mgW4D3QmHYt/iFYqmIO daenHd3k6J84XMZPj8aLrnhccKg/j2RdYX7s42FiOVjXGMVLRzImjPyQrwTnX8efcyGm Ra/ppFlwl93Eplb98GzLorm7F1AmTGNkrVlXolsNmAbxIyN/YrWfAQiAX1uqzOgVvq5e IvvyHoZWcS5v+dtGnxJb1IOmSUBc7ao0YMOTixw+JjlFo82Kt6tpAHkuj57K7uMR3/Yt baMoo9ddL0RX2VnELNpjIwYrcx5H4qchd18Ikvt5SMp6ktBoraii5TiHBVXnxQCYyAOU KsTA== X-Forwarded-Encrypted: i=1; AJvYcCXTnasSD++hlIALFJ/oNx6zd4R7qKrrh3558FoCoyrKyATb7826jl7eKrX1tRTVe/qeg38hQYrobxVGDf5u/TwfU2P/J6EsYqYFiJzi X-Gm-Message-State: AOJu0YxLOchNq4ociJ7UsEOExVy6wTmWffrjZzar0UmxSTMc+dBevXGD mZEqg9OaQ/wMLY/809rJYvGnyMT22XCZImOFlt8hEz0O3Vvy45Uh X-Google-Smtp-Source: AGHT+IFwms1cBvU+f9ojhWDg3nZKfLTfOVGOcCh0o9BJomwXX1OgvRueLY1XlvYRSI7zo2uCYFeifw== X-Received: by 2002:a05:6870:96a2:b0:24c:a414:3eab with SMTP id 586e51a60fabf-24ca41467f3mr15324961fac.32.1716971330911; Wed, 29 May 2024 01:28:50 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-702220d948bsm174916b3a.131.2024.05.29.01.28.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 01:28:50 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v5 1/6] mm: swap: introduce swap_free_nr() for batched swap_free() Date: Wed, 29 May 2024 20:28:19 +1200 Message-Id: <20240529082824.150954-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240529082824.150954-1-21cnbao@gmail.com> References: <20240529082824.150954-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Chuanhua Han While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free. Furthermore, this new function, swap_free_nr(), is designed to efficiently handle various scenarios for releasing a specified number, nr, of swap entries. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Acked-by: Chris Li Reviewed-by: "Huang, Ying" --- include/linux/swap.h | 5 +++++ mm/swapfile.c | 47 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+) diff --git a/include/linux/swap.h b/include/linux/swap.h index a11c75e897ec..45f76dfe29b1 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -481,6 +481,7 @@ extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); extern void swap_free(swp_entry_t); +extern void swap_free_nr(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); int swap_type_of(dev_t device, sector_t offset); @@ -562,6 +563,10 @@ static inline void swap_free(swp_entry_t swp) { } =20 +static inline void swap_free_nr(swp_entry_t entry, int nr_pages) +{ +} + static inline void put_swap_folio(struct folio *folio, swp_entry_t swp) { } diff --git a/mm/swapfile.c b/mm/swapfile.c index f1e559e216bd..92a045d34a97 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1356,6 +1356,53 @@ void swap_free(swp_entry_t entry) __swap_entry_free(p, entry); } =20 +static void cluster_swap_free_nr(struct swap_info_struct *sis, + unsigned long offset, int nr_pages) +{ + struct swap_cluster_info *ci; + DECLARE_BITMAP(to_free, BITS_PER_LONG) =3D { 0 }; + int i, nr; + + ci =3D lock_cluster_or_swap_info(sis, offset); + while (nr_pages) { + nr =3D min(BITS_PER_LONG, nr_pages); + for (i =3D 0; i < nr; i++) { + if (!__swap_entry_free_locked(sis, offset + i, 1)) + bitmap_set(to_free, i, 1); + } + if (!bitmap_empty(to_free, BITS_PER_LONG)) { + unlock_cluster_or_swap_info(sis, ci); + for_each_set_bit(i, to_free, BITS_PER_LONG) + free_swap_slot(swp_entry(sis->type, offset + i)); + if (nr =3D=3D nr_pages) + return; + bitmap_clear(to_free, 0, BITS_PER_LONG); + ci =3D lock_cluster_or_swap_info(sis, offset); + } + offset +=3D nr; + nr_pages -=3D nr; + } + unlock_cluster_or_swap_info(sis, ci); +} + +void swap_free_nr(swp_entry_t entry, int nr_pages) +{ + int nr; + struct swap_info_struct *sis; + unsigned long offset =3D swp_offset(entry); + + sis =3D _swap_info_get(entry); + if (!sis) + return; + + while (nr_pages) { + nr =3D min_t(int, nr_pages, SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER= ); + cluster_swap_free_nr(sis, offset, nr); + offset +=3D nr; + nr_pages -=3D nr; + } +} + /* * Called after dropping swapcache to decrease refcnt to swap entries. */ --=20 2.34.1 From nobody Wed Dec 17 03:28:12 2025 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7D2E44C7B for ; Wed, 29 May 2024 08:28:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716971341; cv=none; b=VZ4UVkiDsmMyRPe65VXYBOgSO9OwpqIvN0QwEVDp/etrNHm3PtGkynoot8H7+6uB4km9omXTsrTLJxca66xnOKtB5nfzW2si4IX+7VIvBnIyRm9zsUU5lToxxXNZX88zsvCImXReZUZjgMK136/sbQltnoGlL7Z5Q0AMluQZE3k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716971341; c=relaxed/simple; bh=u6KL2RcBFKVFA2AMXZl6KLq4qv71vIG0xV+kBBsY/EI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qFmF7PYSPly/YUUEVzsmReszqI9g9OQ5ZXDPhiiSV4Nh9leuu7hFoWHuX0wVc7xyFDNlBIK6MjcrmEeTLCxZfDx5inBaY9pmwX3WDk36IrxJEyjrG+P7K8UYSyIszQoxa8uIRp3kg4FSw3EwAzeIHo86R/kk+HDEr77gIbhAvUE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=UnikDJFI; arc=none smtp.client-ip=209.85.210.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UnikDJFI" Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-6f8e9555eabso1451119b3a.1 for ; Wed, 29 May 2024 01:28:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716971339; x=1717576139; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=o8PkNiQW/mkvjIvpLEp+EBywhuNk7TaiPl+XTuB214I=; b=UnikDJFIAHtdFNbDczbB3ft3RhmfXYiHIGP5d3q7NNUV7aibEb7VF+p11b6uo1gt29 c6rGSSJfm5Dl56mdtgVccGV1Ksi+03oimjWCWFElNgADGi+Bg8Z1rmADi1dmgq7nHd/D 8h/PxBEMeCZWRVY/QKHEGx2fgmK6y/A6jFPr4gX6XFn4E7fVE0bM+e5S2l+B8r+if54a 7fYCHyUWQFHWKQkI3q2R7qbRBxBdJDdCh6lFl5Ota+CS2HYu2VioFjrk6QSXYlxjmZTB dg4F6cnxX65DyM19BVJ+g2/JalHbGE7QtPJToVZLcppL/NLwGGax0fHfgkJoPcHABVX1 vEFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716971339; x=1717576139; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=o8PkNiQW/mkvjIvpLEp+EBywhuNk7TaiPl+XTuB214I=; b=OLjs6iWdb8bd0GP8RaBMD+8skNKK7m4eHsZTUVykelWlFOKSHruqjJwARDq74B6QSr b0J1RQ2DOnZuNhNNFc2qsAig8EB2ivSQqntFHV8xPZFQCNX1k2wG6kJfdgVdLzslVERp pw/ycUsC4iOez5peDpmHrsEdmi4Qdkpse0eBwWOeYaKYVUdmODAdU7AUlipsnu5XAGWE 0UOX8CaaaZsLzwygSbOO6z5HLVyX+fZ2V8Htu0+ivv+Y++4+RgTxx+s8nZyBSDYoPhaR q6ZplfcppHeekJ6lJQ4FPPPyVyzuWJuI1zMqkpX/jP94ncaNr+6F2gFvUTqt2kZ3Lnkh iCJg== X-Forwarded-Encrypted: i=1; AJvYcCW0Nk7Ezoq2Z+uQSmhXWu+6EhZZOJKuBWW3+SkcKPUu7OaTLddXoFxTVVdu3/qOhdNH2u1gpGW/RwOt4uVWDJWGD0DjXuNBxHYJ5owz X-Gm-Message-State: AOJu0YzTIY6KGmn0o+3Xfra75CfqXgLesxHu9iPJxsqCG4RRiHTKWLEW bnslKzgk2IQWiR4k2LQ4J87Z7HnMVeJEq7cvOE83xN/asKjOOThF X-Google-Smtp-Source: AGHT+IFIIMwl+a1BFU1LURBAWUDnpnOzYB+DE372n3RJK0yLX+C7PJpG+VjhO1aGqsG0jKGc/xlpAg== X-Received: by 2002:a05:6a00:4405:b0:6f8:caf2:8f4f with SMTP id d2e1a72fcca58-6f8f45d2624mr16218543b3a.33.1716971338932; Wed, 29 May 2024 01:28:58 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-702220d948bsm174916b3a.131.2024.05.29.01.28.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 01:28:58 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, "Rafael J. Wysocki" , Pavel Machek , Len Brown , Christoph Hellwig Subject: [PATCH v5 2/6] mm: remove the implementation of swap_free() and always use swap_free_nr() Date: Wed, 29 May 2024 20:28:20 +1200 Message-Id: <20240529082824.150954-3-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240529082824.150954-1-21cnbao@gmail.com> References: <20240529082824.150954-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song To streamline maintenance efforts, we propose removing the implementation of swap_free(). Instead, we can simply invoke swap_free_nr() with nr set to 1. swap_free_nr() is designed with a bitmap consisting of only one long, resulting in overhead that can be ignored for cases where nr equals 1. A prime candidate for leveraging swap_free_nr() lies within kernel/power/swap.c. Implementing this change facilitates the adoption of batch processing for hibernation. Suggested-by: "Huang, Ying" Signed-off-by: Barry Song Reviewed-by: "Huang, Ying" Acked-by: Chris Li Reviewed-by: Ryan Roberts Cc: "Rafael J. Wysocki" Cc: Pavel Machek Cc: Len Brown Cc: Hugh Dickins Cc: Christoph Hellwig --- include/linux/swap.h | 10 +++++----- kernel/power/swap.c | 5 ++--- mm/swapfile.c | 17 ++++------------- 3 files changed, 11 insertions(+), 21 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 45f76dfe29b1..3df75d62a835 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -480,7 +480,6 @@ extern int add_swap_count_continuation(swp_entry_t, gfp= _t); extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); -extern void swap_free(swp_entry_t); extern void swap_free_nr(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); @@ -559,10 +558,6 @@ static inline int swapcache_prepare(swp_entry_t swp) return 0; } =20 -static inline void swap_free(swp_entry_t swp) -{ -} - static inline void swap_free_nr(swp_entry_t entry, int nr_pages) { } @@ -611,6 +606,11 @@ static inline void free_swap_and_cache(swp_entry_t ent= ry) free_swap_and_cache_nr(entry, 1); } =20 +static inline void swap_free(swp_entry_t entry) +{ + swap_free_nr(entry, 1); +} + #ifdef CONFIG_MEMCG static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) { diff --git a/kernel/power/swap.c b/kernel/power/swap.c index d9abb7ab031d..85a8b5f4a081 100644 --- a/kernel/power/swap.c +++ b/kernel/power/swap.c @@ -200,12 +200,11 @@ void free_all_swap_pages(int swap) =20 while ((node =3D swsusp_extents.rb_node)) { struct swsusp_extent *ext; - unsigned long offset; =20 ext =3D rb_entry(node, struct swsusp_extent, node); rb_erase(node, &swsusp_extents); - for (offset =3D ext->start; offset <=3D ext->end; offset++) - swap_free(swp_entry(swap, offset)); + swap_free_nr(swp_entry(swap, ext->start), + ext->end - ext->start + 1); =20 kfree(ext); } diff --git a/mm/swapfile.c b/mm/swapfile.c index 92a045d34a97..9c6d8e557c0f 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1343,19 +1343,6 @@ static void swap_entry_free(struct swap_info_struct = *p, swp_entry_t entry) swap_range_free(p, offset, 1); } =20 -/* - * Caller has made sure that the swap device corresponding to entry - * is still around or has not been recycled. - */ -void swap_free(swp_entry_t entry) -{ - struct swap_info_struct *p; - - p =3D _swap_info_get(entry); - if (p) - __swap_entry_free(p, entry); -} - static void cluster_swap_free_nr(struct swap_info_struct *sis, unsigned long offset, int nr_pages) { @@ -1385,6 +1372,10 @@ static void cluster_swap_free_nr(struct swap_info_st= ruct *sis, unlock_cluster_or_swap_info(sis, ci); } =20 +/* + * Caller has made sure that the swap device corresponding to entry + * is still around or has not been recycled. + */ void swap_free_nr(swp_entry_t entry, int nr_pages) { int nr; --=20 2.34.1 From nobody Wed Dec 17 03:28:12 2025 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3BE0C169AC5 for ; Wed, 29 May 2024 08:29:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716971348; cv=none; b=WimsUjsfhrs3rCx78I0C1UkdvPDZceVeN/z5hfxaqJcxVCXDIQsukZdDro0/2P1sCGPH1czIf7Kf2cOG3WXnWEgRzwPeGmW08oG19OTjzI/vwtKstnRRTbRhNnLx23uToqa0vUbgz3U2ZhbfYxWJc/QMznzS92c7U95e8PmM10c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716971348; c=relaxed/simple; bh=W/pul8blKRhwRpm/56/qpb8smBFPTGG2O3j7CEDrGuc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=sxOs+MlBLWI6Hj8VdZRkqT4UPqEoV1Vk2agwr7xuZH5pU1SXXNbhTUtEwc3k1x7UAPw0V7rDP2CAAssrjPtyvgSEE8oEy3YONUS4zwuv1JwIdpuvoIinWdlDEabj+fvevu/c+5/uYqyoYado0k8YE550Z56A1H7eREOyLWhHOjM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=LxhtbDgs; arc=none smtp.client-ip=209.85.210.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LxhtbDgs" Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-6f8e9878514so1349274b3a.1 for ; Wed, 29 May 2024 01:29:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716971346; x=1717576146; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=83K8bgM725IDCOF3GoIeML6r90yg5uTNSjeWCxUl1nI=; b=LxhtbDgs1z6g5170ukJj+5en/otb5UQRPtjt4GaAGlsj4xETJzvySnpRfHPcIs0a3v 62Nr7ltm/Ikc2YhL9m6+mup715JqpqN3q0zb/prSj0voil9bjmhRBVtMRvye25JecmBu KABQmoORuGGspHzFSei+EvxUvNedA0bteh/zsThiu06FJQqks/VCLcpbkSc+HNiXYxLG 8ocm4GBMcnj2WnMc3Qm4Mz8SLlwzyxbg4JazM2ajBythT6JCWvbB+Ejw2JgCEzZAodI2 XXcvtfkP+oRJuf48/r3q+VLdRxBG66IB+j8KBmIxK/serwt9uLhw9F/uNVAsINCJO5lk IGXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716971346; x=1717576146; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=83K8bgM725IDCOF3GoIeML6r90yg5uTNSjeWCxUl1nI=; b=qbCkrpERsVJYaaj8u7aMY2CBNLOOn7FQnuANv4Ui28KIWxSbvWa6CYoaUO9hTqFw0Y DkSfoR4eKAPGLQUE6e8uR05ovetPPS2c+6/a8M3fYncRA0q1wiQZm4h8vgpYeRJ74NNg O9odwhiygmK9RdeNoXb4N3s3KBNaI502iUAKs0C5RDnd7TMqT1JrJA3P/xIHvmWzFIDz eVVIBRffKb6LsuqUp4drUtjS5LT8XjZMivO73aZ53+lf9bhehzLuCbd1n4cFSdbUni/D CG1yajCl+Exd4IWfnnEXWrXSROUMxxqwcklmUv8pXRHCBQhcrH4IjL+8Zi17znOzRmCe nXig== X-Forwarded-Encrypted: i=1; AJvYcCXJc2GWKsYENzrOn74bmVxppqOIyYyO5dfv5xR0OGXgr3jnYLzhRbT77sCFn+rIbEesU/JDwpMEngbPzWvhrGZ1aKrYpbdzdIkEE7EC X-Gm-Message-State: AOJu0Yy2n9NE3RgrBs6lTQ8HfU+O3IJKSx3nRW+j2EB5e7DUcgUGkvsy KeWClMSEDUOUUiJ9HepBb8/qkPuXsJ3zXTgy93SSqrmaTXPkJpn3 X-Google-Smtp-Source: AGHT+IGUylP38UecAp4ow7R28ofXyse8WlZ1y+8tflbwIT3kfaBZJA8cTz9JYhcDnvYjYa19Kqiapg== X-Received: by 2002:a05:6a00:4390:b0:6ea:d794:ccee with SMTP id d2e1a72fcca58-6f8f38010e9mr17376300b3a.17.1716971346524; Wed, 29 May 2024 01:29:06 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-702220d948bsm174916b3a.131.2024.05.29.01.28.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 01:29:05 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v5 3/6] mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally Date: Wed, 29 May 2024 20:28:21 +1200 Message-Id: <20240529082824.150954-4-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240529082824.150954-1-21cnbao@gmail.com> References: <20240529082824.150954-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song There could arise a necessity to obtain the first pte_t from a swap pte_t located in the middle. For instance, this may occur within the context of do_swap_page(), where a page fault can potentially occur in any PTE of a large folio. To address this, the following patch introduces pte_move_swp_offset(), a function capable of bidirectional movement by a specified delta argument. Consequently, pte_next_swp_offset() will directly invoke it with delta =3D 1. Suggested-by: "Huang, Ying" Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Reviewed-by: "Huang, Ying" --- mm/internal.h | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index bbec99cc9d9d..3419c329b3bc 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio= , unsigned long addr, } =20 /** - * pte_next_swp_offset - Increment the swap entry offset field of a swap p= te. + * pte_move_swp_offset - Move the swap entry offset field of a swap pte + * forward or backward by delta * @pte: The initial pte state; is_swap_pte(pte) must be true and * non_swap_entry() must be false. + * @delta: The direction and the offset we are moving; forward if delta + * is positive; backward if delta is negative * - * Increments the swap offset, while maintaining all other fields, includi= ng + * Moves the swap offset, while maintaining all other fields, including * swap type, and any swp pte bits. The resulting pte is returned. */ -static inline pte_t pte_next_swp_offset(pte_t pte) +static inline pte_t pte_move_swp_offset(pte_t pte, long delta) { swp_entry_t entry =3D pte_to_swp_entry(pte); pte_t new =3D __swp_entry_to_pte(__swp_entry(swp_type(entry), - (swp_offset(entry) + 1))); + (swp_offset(entry) + delta))); =20 if (pte_swp_soft_dirty(pte)) new =3D pte_swp_mksoft_dirty(new); @@ -234,6 +237,20 @@ static inline pte_t pte_next_swp_offset(pte_t pte) return new; } =20 + +/** + * pte_next_swp_offset - Increment the swap entry offset field of a swap p= te. + * @pte: The initial pte state; is_swap_pte(pte) must be true and + * non_swap_entry() must be false. + * + * Increments the swap offset, while maintaining all other fields, includi= ng + * swap type, and any swp pte bits. The resulting pte is returned. + */ +static inline pte_t pte_next_swp_offset(pte_t pte) +{ + return pte_move_swp_offset(pte, 1); +} + /** * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries * @start_ptep: Page table pointer for the first entry. --=20 2.34.1 From nobody Wed Dec 17 03:28:12 2025 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 79234168C08 for ; Wed, 29 May 2024 08:29:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716971356; cv=none; b=nvGfQpBddnELdBXHdx7NGFnsGnLDOMR/FNZD8l4J4+E5ihjUFprIuSG9Ung4EXrKVH699EJi+N7IyC+p08kpLOuBhDce8/zn+n+oqGNJYNHIQdQM5HnM3ti/atRZaFVPzNmaq4cXM1BzFuK4MajfpUD73obaPZQLTBe9Vafi0bE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716971356; c=relaxed/simple; bh=jFlA7G1DVQAoAOQlVDDsBtYyxpiTxi+T4dKpoOXZh5c=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=PC0qPdZfKtiqU1DRj1Cd2eJEK5Fw1x3rYK4Pqo7eRxAB4jToiu0TI590YMZsVnX0q7fS6g975kV8TidGIJHZ8ndPi2im9uExxdAcy7PzWcSKOBurp6lzpmx7/3AkGNTW1a+aLg5BcKVIx5DSp1l0KPJ4iav/bXgfreMg7FjgDEo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Cfhl9sZe; arc=none smtp.client-ip=209.85.210.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Cfhl9sZe" Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-6f8ea2df4b3so1391834b3a.2 for ; Wed, 29 May 2024 01:29:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716971355; x=1717576155; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SEiK5CEmIDFaJlnWMqRQRlmiXqqRnfHwoYyNloeZy6o=; b=Cfhl9sZeCrZoX5nIl+ZEYDoPKdouCajLIH04ugcNcaDmrgEXAgmMSTQaKzO0BzISDG OC+rqrG5q1gQftcIndFsaJ+bTxNOND5N3FFTHTdApWkfijwSuQWcqF3O4iEvdTpOV5/R swtYESFmAb7ROJENugZ0gWOxHgV4saOPiJTZEfrbK224sYKID1ksQK9jb7DZGAJ7QLpo ErPwiIFb3vlHGHm+HEu6YqLeHIBOKwOggXU2dHP/4+/6nS3CP3A5JDlrLAQJ/54zStGN p8NMABoaUIVVtz0meOb0IQ4yd1H75JSn79rioDeW1GlGSMbB9ZqWK7W8yz7zwmwwXf0x 45VQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716971355; x=1717576155; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SEiK5CEmIDFaJlnWMqRQRlmiXqqRnfHwoYyNloeZy6o=; b=YTBBBrbXT+YeH/qKV/VJCq9bNtxvkHeVBmnyGofTzS9OkwjAZ7etuRQEA5DrqrDnU+ 0+PJBN8JrLDt+9eIrqzgC/LNrgFSyNDUh/jbDKhLwWZlgPv5keQm4RLV5H8kQ3k79Kr3 xOX0ID3WEvtOgDM6srTfSZ2ZV4RLTvq1/m86rtnlh63mh97loGD0KG0yQY10XU1pkzwG 53KsZnzwCKFrTGHlf+pULLBL1KNgLMu+Oysv0RF2Rer47sTEVToFbHgIgYzKqMeUNsy8 R8RfPN1Y6pYodqaVsNmGYaa/lV+FYIN1kYUduuJI//9/2i9qsJqjBb49pUc6ujHb9O4x qiog== X-Forwarded-Encrypted: i=1; AJvYcCUZTELfh+gtVwi1AARFyNEiMhlg1CnB0Wi8kOBusBclj5RcTovCBWvHK1w4KFuNe3Ke6hkH9tAG/Hwta9RxRmqwbsuJYW26UiiFGvH5 X-Gm-Message-State: AOJu0Yyuwv3fc5wV6XRwhBBR5mu1ENgq1iIJPrUz5k8A7Cnd+gVyFJLG 2QHbWjnVAF6xj5PcZsEoda5zsPkdQrGN/fEKo0DmjSIeOq1cKsJj X-Google-Smtp-Source: AGHT+IHC+4YVrv2f2RKD1f5/1XnNvm559LObbEuRJtkuQpqFiFl9jlcgtsUPga8jgyaa5WCSFkT/bQ== X-Received: by 2002:a05:6a00:908f:b0:6f3:ebc4:4407 with SMTP id d2e1a72fcca58-6f8f2b49544mr16440786b3a.4.1716971354683; Wed, 29 May 2024 01:29:14 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-702220d948bsm174916b3a.131.2024.05.29.01.29.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 01:29:13 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, Khalid Aziz , "David S. Miller" , Andreas Larsson Subject: [PATCH v5 4/6] mm: introduce arch_do_swap_page_nr() which allows restore metadata for nr pages Date: Wed, 29 May 2024 20:28:22 +1200 Message-Id: <20240529082824.150954-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240529082824.150954-1-21cnbao@gmail.com> References: <20240529082824.150954-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song Should do_swap_page() have the capability to directly map a large folio, metadata restoration becomes necessary for a specified number of pages denoted as nr. It's important to highlight that metadata restoration is solely required by the SPARC platform, which, however, does not enable THP_SWAP. Consequently, in the present kernel configuration, there exists no practical scenario where users necessitate the restoration of nr metadata. Platforms implementing THP_SWAP might invoke this function with nr values exceeding 1, subsequent to do_swap_page() successfully mapping an entire large folio. Nonetheless, their arch_do_swap_page_nr() functions remain empty. Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Reviewed-by: Khalid Aziz Cc: "David S. Miller" Cc: Andreas Larsson --- include/linux/pgtable.h | 26 ++++++++++++++++++++------ mm/memory.c | 3 ++- 2 files changed, 22 insertions(+), 7 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 117b807e3f89..2f32eaccf0b9 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1089,6 +1089,15 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) }) =20 #ifndef __HAVE_ARCH_DO_SWAP_PAGE +static inline void arch_do_swap_page_nr(struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long addr, + pte_t pte, pte_t oldpte, + int nr) +{ + +} +#else /* * Some architectures support metadata associated with a page. When a * page is being swapped out, this metadata must be saved so it can be @@ -1097,12 +1106,17 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) * page as metadata for the page. arch_do_swap_page() can restore this * metadata when a page is swapped back in. */ -static inline void arch_do_swap_page(struct mm_struct *mm, - struct vm_area_struct *vma, - unsigned long addr, - pte_t pte, pte_t oldpte) -{ - +static inline void arch_do_swap_page_nr(struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long addr, + pte_t pte, pte_t oldpte, + int nr) +{ + for (int i =3D 0; i < nr; i++) { + arch_do_swap_page(vma->vm_mm, vma, addr + i * PAGE_SIZE, + pte_advance_pfn(pte, i), + pte_advance_pfn(oldpte, i)); + } } #endif =20 diff --git a/mm/memory.c b/mm/memory.c index 100f54fc9e6c..c0c6de31a313 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4314,7 +4314,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); + arch_do_swap_page_nr(vma->vm_mm, vma, vmf->address, + pte, vmf->orig_pte, 1); =20 folio_unlock(folio); if (folio !=3D swapcache && swapcache) { --=20 2.34.1 From nobody Wed Dec 17 03:28:12 2025 Received: from mail-ot1-f51.google.com (mail-ot1-f51.google.com [209.85.210.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE810169AF7 for ; Wed, 29 May 2024 08:29:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716971364; cv=none; b=klCPRMlHUPHOI3LK8Q9q1K8eeWbcGotfee9zVdXHoebQrqNrRmqfN0X7seq6Pt2gnkhxoYKvReF1840FsaCIuZrRue2FrcufqNyDLQN7C8EUf6Bxe5QA8HAUlW4XPqtSU7Eum0X4SsUGiqtPQ/VxTt5lCMuLLOsEqHqqchFML8M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716971364; c=relaxed/simple; bh=j9hjI2tH8Wq5eZsp/WXoSuOZGb8/J9q79Mq4cff2Ewc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WGM6jOaO0Cs7RzIMPJISbEfH8B/vi+r3S6H9OFLHzMaIoPYVetCwy27jXc6e5ZE3geXXapxYslaCLSKdmA4MC/14TIIr0CjtZ2cfDZTEly8do0w5pzycvPmChHEO953qp7EOot1arTC4DDuvof6I5QTOj2MChlzA900bpjSDfUM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=XPspe4G2; arc=none smtp.client-ip=209.85.210.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XPspe4G2" Received: by mail-ot1-f51.google.com with SMTP id 46e09a7af769-6f8e9522bfaso952157a34.0 for ; Wed, 29 May 2024 01:29:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716971362; x=1717576162; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6+4WZx2jk6Uy8Ser1SPRLQ9xpkk5S7WupCS1+ZFnTlU=; b=XPspe4G20Iq9otcVmNw1z1XHwfUktIcjjgXVeR50UL1f+a5d72vl6RHg+vaiUXxXXa AF17+VRdKm+WFdrOXSkr8UMzBNZWTGNQz6fS4FV/ZROBkD5oh+kbABJ9EZxl1OgNKjvD 6QZqTMG52zQQYK5DYfWsCXXQ9k8nONUpC0a6yXWejXOWSU2T81Uio7aQkywa5Gxl0Vv+ fK1AH1ZnXHpRTmFfepZG42473ntbu5CtK8jL6tHtO/3TBeE7/r8ci/ltm70gNLNgZ6sm npF8UzIrIAqdDyEgq9Zxxs9q9siQllxPYEx4tFzQ5ugbdfIY6oUE9pwygkqKA+jo6is4 hYXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716971362; x=1717576162; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6+4WZx2jk6Uy8Ser1SPRLQ9xpkk5S7WupCS1+ZFnTlU=; b=VClt+4Empgl84FirBCd2mD/genKP9GO7+b/5HyexsruagMYmUdbCwTK6wkC/4CG/93 m1x6HM12VO+poydST5xs5KcZt1IGesHrEda7JjelzYXsbA9VHpaZd5qZxiw4jgjT5iLO N875Vby3IliRr4vK4rowhlO0bt74e9m1HbQ74f/FEJPZKAFXeObLuP39/ryvMDUDHxC7 vgrVYwpkQQnJ2qK0wFuP9iz/3IRxAbjPdYiZL8SAYucxJiEstTfb4izZvVCMnnDpijUv OWvYsGSJBCC416xNAUgeoaS96xKdnyh2aj0KC9S2N3DOyaFutV5t1jKXcXYSgJDTs5J0 OOgQ== X-Forwarded-Encrypted: i=1; AJvYcCV2pfzFhX3d4dCoOqeUcvZ4EFf1+qw+quzqXH+e/tk4kPQz71zsLHkfGsPc8PA2HtiVtZMJG5SLgMS/W5PTEFFiRbBe0l28wGHEcA7P X-Gm-Message-State: AOJu0Yz1gaS81uVLUp09jmZyTotZ83uidHcevDokzViHNWurNdE0fvGm M0nnm65I5zcIoptTvkANlN6DxFyj1RVI1QjE6ueSce1xADpa7zhb X-Google-Smtp-Source: AGHT+IHGb+sXIn68+MgGWVaSAFwEZ1WhY2usjFaXYWxY+vdoWKkWciGEogVMz08VfHXPwQpQdXzQOQ== X-Received: by 2002:a05:6870:d6a2:b0:24f:e000:a6f8 with SMTP id 586e51a60fabf-24fe000ac34mr13014665fac.41.1716971361783; Wed, 29 May 2024 01:29:21 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-702220d948bsm174916b3a.131.2024.05.29.01.29.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 01:29:21 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v5 5/6] mm: swap: make should_try_to_free_swap() support large-folio Date: Wed, 29 May 2024 20:28:23 +1200 Message-Id: <20240529082824.150954-6-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240529082824.150954-1-21cnbao@gmail.com> References: <20240529082824.150954-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Chuanhua Han The function should_try_to_free_swap() operates under the assumption that swap-in always occurs at the normal page granularity, i.e., folio_nr_pages() =3D 1. However, in reality, for large folios, add_to_swap_cache() will invoke folio_ref_add(folio, nr). To accommodate large folio swap-in, this patch eliminates this assumption. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Acked-by: Chris Li Reviewed-by: Ryan Roberts Reviewed-by: "Huang, Ying" Reviewed-by: David Hildenbrand --- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index c0c6de31a313..10719e5afecb 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3925,7 +3925,7 @@ static inline bool should_try_to_free_swap(struct fol= io *folio, * reference only in case it's likely that we'll be the exlusive user. */ return (fault_flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) && - folio_ref_count(folio) =3D=3D 2; + folio_ref_count(folio) =3D=3D (1 + folio_nr_pages(folio)); } =20 static vm_fault_t pte_marker_clear(struct vm_fault *vmf) --=20 2.34.1 From nobody Wed Dec 17 03:28:12 2025 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A018516A36D for ; Wed, 29 May 2024 08:29:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716971371; cv=none; b=AoehRg/TNaHzEPJc5Nz3C4sGMooF/fGncoHwgCnVyiTQl9dbbf5dD5DIiJmICST2B3NFrSzAyvg2ghWffE+MLFKVHPh/wpR5kkVx76i14bqiSHe9cJYv0t5CMGF0JTeBOwHrsg9Hk2aput2VosBSmYOLuc4hpAyzw1qpyh39Z9U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716971371; c=relaxed/simple; bh=XCAlxfa4dcd7ZXSZyKWnqYITEwMpdGHxcWPUNoGZmLY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=htna/LYZMblwehCpVZgauMd0mIuMBfWvqdYDXf8lYG9wxD3ka3s973X+9FpXzMnJXkcTsvkDZSp5uoLCWRPui6DWd75oPfvBkyZq+GtQm3dgung1z7LDNlfjGO8PyFZ+4isTYDywnu5g1vtYGwc5HaDHDk0YQmXN8UOyN36ph84= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=BMyE4q/2; arc=none smtp.client-ip=209.85.210.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BMyE4q/2" Received: by mail-pf1-f182.google.com with SMTP id d2e1a72fcca58-6f4ed9dc7beso482543b3a.1 for ; Wed, 29 May 2024 01:29:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716971369; x=1717576169; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kxE2mAKSIadz3iJIa+4kx5y9ZNqImGpGMb9QcvFrG9A=; b=BMyE4q/2Rx12IicJL0GHmfnuvgS3MljR1+LDx3lHyRRrEfbtveSSE3sl+q2hXSoyLa iwULnpLvj6eY/oUDFvefvHgD6plYLrdbsLbLdzX55lEj8hipWZ3QSOLVeTxja85H3s+c D4gQPW5xrE+EvxK8g+bAvJsiS14M2XtIrkKjmUk0oWA5WOfuBXO9Y/oPsFuUlpEH73BY lx/k9ao7xT4+gR9+QclNpj+XTN/0v+kfdS2m/wU72azqh02GkkEZiPKDEiWO5ewgBpxM 7dx9Xyc/z+lCY8GS5gxFBr4pJlM0s3H9OxKtff4hYRAHjba80g2GkMD1X4sNsUruUhOd 0l9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716971369; x=1717576169; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kxE2mAKSIadz3iJIa+4kx5y9ZNqImGpGMb9QcvFrG9A=; b=Yncn1jzCQDpfjLqi21JXPf84kt/agKsHFojelstrokk4Asa3TQROBxoQXfEvzN2EZP V8gQm2LaW51/PTKxOu92d0IMu6Yu/ZSgiaHcMIqBvFNU3UWg85K6CvIQkDKbSXUiTIWt 6KyhtrLrBk/QqeoWybl/8mwhWOE/YExCH/cjC9HhsYFKyoMwa182ck+Fhu9HwXnuY0OO yIT8yWWcivCkZg6ICdV7zovSMESIcwpeeYlbbcNQPbCl+YdOZwMrYiersN/0JTk2i9XX tvuNo3lkgO9TDO+c9ELul9uXLM67T98pJfGpeuuMsDBHWYtNaICMrUaAibJ+DoMuyQQy XcRg== X-Forwarded-Encrypted: i=1; AJvYcCXjn2cDbhX3WYniJ5ottIBVGvwHkSJoFTZAUiMYbuL6XY7UQsEMyRONBVTmeyfRO9/6HQYkQawr8eux7H9hS+sbPeyrRx57FzWEO5a7 X-Gm-Message-State: AOJu0YzRCyo7YzqfVyTCi2eFQYg//VDo2+ZKWTSKT69rOLkEYn5oHIq7 aA5cJaj2N+qViMUzVGDRmfOiBTLSatHgumwuk+DcHwE0y53loJYr X-Google-Smtp-Source: AGHT+IEbSGge22ETQ01VYggngJlR3UpTZl4VLPzOebNpH6caOUJ2L4BysvKHtwvqOmVKwJG58Oyn5g== X-Received: by 2002:a05:6a00:2fd1:b0:702:1d51:98f7 with SMTP id d2e1a72fcca58-7021d519c44mr490666b3a.2.1716971368774; Wed, 29 May 2024 01:29:28 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-702220d948bsm174916b3a.131.2024.05.29.01.29.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 01:29:28 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v5 6/6] mm: swap: entirely map large folios found in swapcache Date: Wed, 29 May 2024 20:28:24 +1200 Message-Id: <20240529082824.150954-7-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240529082824.150954-1-21cnbao@gmail.com> References: <20240529082824.150954-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Chuanhua Han When a large folio is found in the swapcache, the current implementation requires calling do_swap_page() nr_pages times, resulting in nr_pages page faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Reviewed-by: "Huang, Ying" --- mm/memory.c | 59 +++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 48 insertions(+), 11 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 10719e5afecb..eef4e482c0c2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4016,6 +4016,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte_t pte; vm_fault_t ret =3D 0; void *shadow =3D NULL; + int nr_pages; + unsigned long page_idx; + unsigned long address; + pte_t *ptep; =20 if (!pte_unmap_same(vmf)) goto out; @@ -4214,6 +4218,38 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } =20 + nr_pages =3D 1; + page_idx =3D 0; + address =3D vmf->address; + ptep =3D vmf->pte; + if (folio_test_large(folio) && folio_test_swapcache(folio)) { + int nr =3D folio_nr_pages(folio); + unsigned long idx =3D folio_page_idx(folio, page); + unsigned long folio_start =3D address - idx * PAGE_SIZE; + unsigned long folio_end =3D folio_start + nr * PAGE_SIZE; + pte_t *folio_ptep; + pte_t folio_pte; + + if (unlikely(folio_start < max(address & PMD_MASK, vma->vm_start))) + goto check_folio; + if (unlikely(folio_end > pmd_addr_end(address, vma->vm_end))) + goto check_folio; + + folio_ptep =3D vmf->pte - idx; + folio_pte =3D ptep_get(folio_ptep); + if (!pte_same(folio_pte, pte_move_swp_offset(vmf->orig_pte, -idx)) || + swap_pte_batch(folio_ptep, nr, folio_pte) !=3D nr) + goto check_folio; + + page_idx =3D idx; + address =3D folio_start; + ptep =3D folio_ptep; + nr_pages =3D nr; + entry =3D folio->swap; + page =3D &folio->page; + } + +check_folio: /* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4273,12 +4309,12 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * We're already holding a reference on the page but haven't mapped it * yet. */ - swap_free(entry); + swap_free_nr(entry, nr_pages); if (should_try_to_free_swap(folio, vma, vmf->flags)) folio_free_swap(folio); =20 - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - dec_mm_counter(vma->vm_mm, MM_SWAPENTS); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); + add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages); pte =3D mk_pte(page, vma->vm_page_prot); =20 /* @@ -4295,27 +4331,28 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } rmap_flags |=3D RMAP_EXCLUSIVE; } - flush_icache_page(vma, page); + folio_ref_add(folio, nr_pages - 1); + flush_icache_pages(vma, page, nr_pages); if (pte_swp_soft_dirty(vmf->orig_pte)) pte =3D pte_mksoft_dirty(pte); if (pte_swp_uffd_wp(vmf->orig_pte)) pte =3D pte_mkuffd_wp(pte); - vmf->orig_pte =3D pte; + vmf->orig_pte =3D pte_advance_pfn(pte, page_idx); =20 /* ksm created a completely new copy */ if (unlikely(folio !=3D swapcache && swapcache)) { - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_add_new_anon_rmap(folio, vma, address); folio_add_lru_vma(folio, vma); } else { - folio_add_anon_rmap_pte(folio, page, vma, vmf->address, + folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, address, rmap_flags); } =20 VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page_nr(vma->vm_mm, vma, vmf->address, - pte, vmf->orig_pte, 1); + set_ptes(vma->vm_mm, address, ptep, pte, nr_pages); + arch_do_swap_page_nr(vma->vm_mm, vma, address, + pte, pte, nr_pages); =20 folio_unlock(folio); if (folio !=3D swapcache && swapcache) { @@ -4339,7 +4376,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } =20 /* No need to invalidate - it was non-present before */ - update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + update_mmu_cache_range(vmf, vma, address, ptep, nr_pages); unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl); --=20 2.34.1