From nobody Sat Feb 7 04:16:37 2026 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BAB6BBA53 for ; Fri, 3 May 2024 00:50:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714697453; cv=none; b=JzAx7bxdK/veDNMdBY3B2bP6u32EIFBweqXuSNyzlx2Q4iD2LmVlTgBamfzf7zW7eL3WeQVoypwgqTg3ndgziSN1VnHp88IUQBCsciY89WW0XVkX/McFk4k2Dl2S/sVV1XK3GhbrRxV1JXrjSxWWoA7rk6t4qB8eYN6Kf0IWdKk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714697453; c=relaxed/simple; bh=Rwkln1NM+NdA1PcVGtpQVbTXcTpZ9hnMHyu91T/s6t0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=MQ4efcyFVrqzbPbsNH7zEnKQ9aqDfEPTXl/dFd36ayLWQjTgF4nk9MZVV92QkORHj9GfxVLmkUyv5KFWCz0RkDpjPKLiDgc3DS8HQ2nfAdn3e1GKrQUkaAmDGT0wQNo3UiJAqFg3849dM4MpkMb6PjkGjaCmltYoO2ZEPGyC4Dc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=eu0rJlFw; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eu0rJlFw" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-1e3f17c6491so72045125ad.2 for ; Thu, 02 May 2024 17:50:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714697451; x=1715302251; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9ZtaFVcTSDB4ul8nNoaO/PcEtn9Rrjc3ITa59z6anjw=; b=eu0rJlFwphXNCNrUoxktH69Vn3+GsdN/ISlSxQDvPLA0uFKa34U9ePJSDiyQjmb6z4 Gf+5jzMx/fx/Hd+NeXDpnV1LMSLYFj6L8hLzgSyk6APxK8cSOEViZcs7gv4vQ7Q0QklO R2wsPnoEF3OBvJD6w7MZsY93bxNEoPMVMIUa95M/nIE45ETD/8B6/RmVX4NW3psJbkBO eS1MSamuSB1NDNfAY+VH4+BxzQaLT8o6Xe2e330RaAoH9e1zGrvUl6jJ3HMleon7qtl4 p2jeEyx3geSTv0CA6MQEzLwx420eXkHl4PwpWtREcmSV+g80FtbXLgCYnKCnOK9uuNXo 3iUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714697451; x=1715302251; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9ZtaFVcTSDB4ul8nNoaO/PcEtn9Rrjc3ITa59z6anjw=; b=Pb7irlH0131/SZBv5lljNAS8IZeOieAdS3ySOSft6d1V9t94hw9WFUYdhyRL76GicY hV8jHNQnj5qUmX+mB9bfPbOiaP4VUtw2XfdOnIdof0DWSx4LhH05Mn2HZ9W3GQa0p+qn wMfTvotfVtbCarelS2qVRgzdtRFfVQl9Nu/NmExQhYsqw6ljatWHXiR+cn4Yiz5fuY+o 21+skUI2dbkJ381LVHno8cfoPWWLVtGtEvgh9fvy7RDqjeq16FyA5ilLC7LDFhdF6b1x nDy2oyqo5VzQ2nGOrZ8cHB1oBLtaC7ZK26J5Xa0EIdVHGy98CCailSu43RCvZ52bNU6I B2jA== X-Forwarded-Encrypted: i=1; AJvYcCU6qCd5NqNW8r7FhwxqX4nY/ey5ybIPWylvN/oBRXRDXSyxvugmHUymltHMIz8LcP8Yckc8wSt2kFjeORtABEfEBF+L71zVT397mmXy X-Gm-Message-State: AOJu0YxpArfGbQW4FGpkiYlwBA51DWEHM6eWuGw7tzMY9bsJVBfwh8oB hKpIfFOc9i+eEXU5tOD9y3OZR2oq538Gn54BvYdXuxBgs/1oecpE X-Google-Smtp-Source: AGHT+IEefQkadUJjt20GqF9jK1qsxgF1ScWbbqdgQAWnvfCb3u+RiScs5Eeaq3V8oWvBSMJbyG0VfA== X-Received: by 2002:a17:902:bb8b:b0:1e8:b768:2c25 with SMTP id m11-20020a170902bb8b00b001e8b7682c25mr1211101pls.15.1714697450969; Thu, 02 May 2024 17:50:50 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d14-20020a170903230e00b001ec48e41d2esm1969175plh.212.2024.05.02.17.50.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 May 2024 17:50:50 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v3 1/6] mm: swap: introduce swap_free_nr() for batched swap_free() Date: Fri, 3 May 2024 12:50:18 +1200 Message-Id: <20240503005023.174597-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240503005023.174597-1-21cnbao@gmail.com> References: <20240503005023.174597-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Chuanhua Han While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free. Furthermore, this new function, swap_free_nr(), is designed to efficiently handle various scenarios for releasing a specified number, nr, of swap entries. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Acked-by: Chris Li Reviewed-by: "Huang, Ying" Reviewed-by: Ryan Roberts --- include/linux/swap.h | 5 +++++ mm/swapfile.c | 47 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+) diff --git a/include/linux/swap.h b/include/linux/swap.h index 11c53692f65f..d1d35e92d7e9 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -483,6 +483,7 @@ extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); extern void swap_free(swp_entry_t); +extern void swap_free_nr(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); int swap_type_of(dev_t device, sector_t offset); @@ -564,6 +565,10 @@ static inline void swap_free(swp_entry_t swp) { } =20 +static inline void swap_free_nr(swp_entry_t entry, int nr_pages) +{ +} + static inline void put_swap_folio(struct folio *folio, swp_entry_t swp) { } diff --git a/mm/swapfile.c b/mm/swapfile.c index f6ca215fb92f..ec12f2b9d229 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1356,6 +1356,53 @@ void swap_free(swp_entry_t entry) __swap_entry_free(p, entry); } =20 +static void cluster_swap_free_nr(struct swap_info_struct *sis, + unsigned long offset, int nr_pages) +{ + struct swap_cluster_info *ci; + DECLARE_BITMAP(to_free, BITS_PER_LONG) =3D { 0 }; + int i, nr; + + ci =3D lock_cluster_or_swap_info(sis, offset); + while (nr_pages) { + nr =3D min(BITS_PER_LONG, nr_pages); + for (i =3D 0; i < nr; i++) { + if (!__swap_entry_free_locked(sis, offset + i, 1)) + bitmap_set(to_free, i, 1); + } + if (!bitmap_empty(to_free, BITS_PER_LONG)) { + unlock_cluster_or_swap_info(sis, ci); + for_each_set_bit(i, to_free, BITS_PER_LONG) + free_swap_slot(swp_entry(sis->type, offset + i)); + if (nr =3D=3D nr_pages) + return; + bitmap_clear(to_free, 0, BITS_PER_LONG); + ci =3D lock_cluster_or_swap_info(sis, offset); + } + offset +=3D nr; + nr_pages -=3D nr; + } + unlock_cluster_or_swap_info(sis, ci); +} + +void swap_free_nr(swp_entry_t entry, int nr_pages) +{ + int nr; + struct swap_info_struct *sis; + unsigned long offset =3D swp_offset(entry); + + sis =3D _swap_info_get(entry); + if (!sis) + return; + + while (nr_pages) { + nr =3D min_t(int, nr_pages, SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER= ); + cluster_swap_free_nr(sis, offset, nr); + offset +=3D nr; + nr_pages -=3D nr; + } +} + /* * Called after dropping swapcache to decrease refcnt to swap entries. */ --=20 2.34.1 From nobody Sat Feb 7 04:16:37 2026 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 35FFCCA7D for ; Fri, 3 May 2024 00:50:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714697460; cv=none; b=Fdh+vo3MFI+qBm8ySjfSAoOmV01LZ+RqyafSeKJ/SvEQMvwdfmdGiRYAORpW3YB9LAe11JfSgRq3wkz2JYfK2ZE7uqrIJpNZPkD1rQ0xyiPV6nMlknVfby/gZnSjV4vBBi7jZfoIlheqW77YvvriMEROIDe6ZywNZ072m7KvXus= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714697460; c=relaxed/simple; bh=tcDnZKQv8l8iz93Yo4hfUJB3P9kNF5o4attE/UqX+GQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hIA0YBLZuwh4xPmEjLxTmrmkF847i3LE3eQoWhTLpMSSrffICpwT/3HXjWEALea65zgi+L4qOT+9RLrFSwBGyEIHvcmVDVp1oEKPcauMglVTJM3Mmhsz+ofJcYVvI2D6qVhw8PDR0yNU5n99VrhIwEGFNh+K56BEyCjMJ1ReB3A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=deFtn49Y; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="deFtn49Y" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-1ec5387aed9so31516225ad.3 for ; Thu, 02 May 2024 17:50:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714697458; x=1715302258; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wFl5l9cUG6TUpWzvnR4XiCKTQilGE1cDMcBQh7vvAdE=; b=deFtn49Yv6nvFEJHdXm2g/OaZvkH5u9JxvTUE6ZoI3G8O5PR9sNnbY4faQgyVN7qun z0nfineGlK+FzxVDCfjXsYCAkXQdSdHtDmYnt5EOlcxM+8zkBwmx6IVqjH9f1rLD7+QI iJF7vNZ1/n3fB6bp2MMA9nXEDrF7E9GIT6kv73KKFfw2sRcIAZ7DTubTO31ZGIEyRQw4 JtsFizlv0eY/a14FZ84PJGDYRy5X0G16gz55sekR7BIMKQ/KMN9HZwadam6m8G32We9u dh6gyJiyVemorpKEUVbFSSCNaiSPTh+9bWXGvg74xvckmgXJnd6LcPRmZjkHBMFkyOZL aQXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714697458; x=1715302258; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wFl5l9cUG6TUpWzvnR4XiCKTQilGE1cDMcBQh7vvAdE=; b=JVb9tqHUj1fZbvJdpGZZXosGxZ5Id6QrWtGbYqZgdNwJ8Ew0WMmuTcizIm+qNtoTVu KZUkew6QoNRBugtuzMS1ejiuOYGMAHK5bPwvWgSkGCWtBYTDFSDUzJIwnJ8Mcn5RXJrx jjMnpqSquAZIUboLx/gjiCpFJwao28GE/UqsYsQeK2EKDCclYytLbCluavEmGjBzVzRj CXuwMKuEnQuuvJSRFBBqjceHs10aDfPqAYDV6NoYWpO6qokqJ7valOMR2ZKg6yW30lva w8hkzRtVXn0L1JzrlQDYZVILzllMBQWGMDAEtRrwl0CIHD4shp4cPIErxCdvTW/ERY0m Y4Gg== X-Forwarded-Encrypted: i=1; AJvYcCUfClw4t3yvmApQWGdy/zNdPY+Yc1VJmVS4bW+HeB14pxXqRw5XSJxfOnBep74xcJ5KywXSFSaZ8OSLB2F4VVNYvtum34iEUoHLrLYy X-Gm-Message-State: AOJu0YyO8vE1tYdYImnR9kd531uuj7yKhl/94Z1i5uib6yl6xFa4CvKb Q8+8Sz3dJwSdYrDoVLQkqJIqSY+fGo2l6rww8uJFS6Nwio4JInGz X-Google-Smtp-Source: AGHT+IHgGPs6Tx/aHvD+V7N1MdRzTr0fR1RogpKIZp6wEsXbZrG/1C29hAfRf2MxmZtp2ra0jMkL5g== X-Received: by 2002:a17:902:b495:b0:1ea:f7d4:cf2b with SMTP id y21-20020a170902b49500b001eaf7d4cf2bmr1203978plr.17.1714697458583; Thu, 02 May 2024 17:50:58 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d14-20020a170903230e00b001ec48e41d2esm1969175plh.212.2024.05.02.17.50.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 May 2024 17:50:58 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, "Rafael J. Wysocki" , Pavel Machek , Len Brown Subject: [PATCH v3 2/6] mm: remove swap_free() and always use swap_free_nr() Date: Fri, 3 May 2024 12:50:19 +1200 Message-Id: <20240503005023.174597-3-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240503005023.174597-1-21cnbao@gmail.com> References: <20240503005023.174597-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song To streamline maintenance efforts, we propose discontinuing the use of swap_free(). Instead, we can simply invoke swap_free_nr() with nr set to 1. This adjustment offers the advantage of enabling batch processing within kernel/power/swap.c. Furthermore, swap_free_nr() is designed with a bitmap consisting of only one long, resulting in overhead that can be ignored for cases where nr equals 1. Suggested-by: "Huang, Ying" Signed-off-by: Barry Song Cc: "Rafael J. Wysocki" Cc: Pavel Machek Cc: Len Brown Cc: Hugh Dickins Acked-by: Chris Li Reviewed-by: "Huang, Ying" --- include/linux/swap.h | 5 ----- kernel/power/swap.c | 7 +++---- mm/memory.c | 2 +- mm/rmap.c | 4 ++-- mm/shmem.c | 4 ++-- mm/swapfile.c | 19 +++++-------------- 6 files changed, 13 insertions(+), 28 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index d1d35e92d7e9..f03cb446124e 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -482,7 +482,6 @@ extern int add_swap_count_continuation(swp_entry_t, gfp= _t); extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); -extern void swap_free(swp_entry_t); extern void swap_free_nr(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); @@ -561,10 +560,6 @@ static inline int swapcache_prepare(swp_entry_t swp) return 0; } =20 -static inline void swap_free(swp_entry_t swp) -{ -} - static inline void swap_free_nr(swp_entry_t entry, int nr_pages) { } diff --git a/kernel/power/swap.c b/kernel/power/swap.c index 5bc04bfe2db1..6befaa88a342 100644 --- a/kernel/power/swap.c +++ b/kernel/power/swap.c @@ -181,7 +181,7 @@ sector_t alloc_swapdev_block(int swap) offset =3D swp_offset(get_swap_page_of_type(swap)); if (offset) { if (swsusp_extents_insert(offset)) - swap_free(swp_entry(swap, offset)); + swap_free_nr(swp_entry(swap, offset), 1); else return swapdev_block(swap, offset); } @@ -200,12 +200,11 @@ void free_all_swap_pages(int swap) =20 while ((node =3D swsusp_extents.rb_node)) { struct swsusp_extent *ext; - unsigned long offset; =20 ext =3D rb_entry(node, struct swsusp_extent, node); rb_erase(node, &swsusp_extents); - for (offset =3D ext->start; offset <=3D ext->end; offset++) - swap_free(swp_entry(swap, offset)); + swap_free_nr(swp_entry(swap, ext->start), + ext->end - ext->start + 1); =20 kfree(ext); } diff --git a/mm/memory.c b/mm/memory.c index eea6e4984eae..f033eb3528ba 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4225,7 +4225,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * We're already holding a reference on the page but haven't mapped it * yet. */ - swap_free(entry); + swap_free_nr(entry, 1); if (should_try_to_free_swap(folio, vma, vmf->flags)) folio_free_swap(folio); =20 diff --git a/mm/rmap.c b/mm/rmap.c index 087a79f1f611..39ec7742acec 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1865,7 +1865,7 @@ static bool try_to_unmap_one(struct folio *folio, str= uct vm_area_struct *vma, goto walk_done_err; } if (arch_unmap_one(mm, vma, address, pteval) < 0) { - swap_free(entry); + swap_free_nr(entry, 1); set_pte_at(mm, address, pvmw.pte, pteval); goto walk_done_err; } @@ -1873,7 +1873,7 @@ static bool try_to_unmap_one(struct folio *folio, str= uct vm_area_struct *vma, /* See folio_try_share_anon_rmap(): clear PTE first. */ if (anon_exclusive && folio_try_share_anon_rmap_pte(folio, subpage)) { - swap_free(entry); + swap_free_nr(entry, 1); set_pte_at(mm, address, pvmw.pte, pteval); goto walk_done_err; } diff --git a/mm/shmem.c b/mm/shmem.c index fa2a0ed97507..bfc8a2beb24f 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1836,7 +1836,7 @@ static void shmem_set_folio_swapin_error(struct inode= *inode, pgoff_t index, * in shmem_evict_inode(). */ shmem_recalc_inode(inode, -1, -1); - swap_free(swap); + swap_free_nr(swap, 1); } =20 /* @@ -1927,7 +1927,7 @@ static int shmem_swapin_folio(struct inode *inode, pg= off_t index, =20 delete_from_swap_cache(folio); folio_mark_dirty(folio); - swap_free(swap); + swap_free_nr(swap, 1); put_swap_device(si); =20 *foliop =3D folio; diff --git a/mm/swapfile.c b/mm/swapfile.c index ec12f2b9d229..ddcd0f24b9a1 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1343,19 +1343,6 @@ static void swap_entry_free(struct swap_info_struct = *p, swp_entry_t entry) swap_range_free(p, offset, 1); } =20 -/* - * Caller has made sure that the swap device corresponding to entry - * is still around or has not been recycled. - */ -void swap_free(swp_entry_t entry) -{ - struct swap_info_struct *p; - - p =3D _swap_info_get(entry); - if (p) - __swap_entry_free(p, entry); -} - static void cluster_swap_free_nr(struct swap_info_struct *sis, unsigned long offset, int nr_pages) { @@ -1385,6 +1372,10 @@ static void cluster_swap_free_nr(struct swap_info_st= ruct *sis, unlock_cluster_or_swap_info(sis, ci); } =20 +/* + * Caller has made sure that the swap device corresponding to entry + * is still around or has not been recycled. + */ void swap_free_nr(swp_entry_t entry, int nr_pages) { int nr; @@ -1930,7 +1921,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_= t *pmd, new_pte =3D pte_mkuffd_wp(new_pte); setpte: set_pte_at(vma->vm_mm, addr, pte, new_pte); - swap_free(entry); + swap_free_nr(entry, 1); out: if (pte) pte_unmap_unlock(pte, ptl); --=20 2.34.1 From nobody Sat Feb 7 04:16:37 2026 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C675BE58 for ; Fri, 3 May 2024 00:51:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714697467; cv=none; b=ZD9osuseL6dxbUQWDStGeaBvdTxsWr0b1wUB6zfl/HTneBTU9F5jNAXkUT1xbslUwu/1m2UnXYuMLyoo7gNRWemj6rz6MFi6W3jHlJBQ8e4zIE9c2Xr2CYxAhaLGuZE9GMs/9fFw7+A+hGj3yG4W/nSJBjWWcSvav3A6goFNaTI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714697467; c=relaxed/simple; bh=UvcQuwNEhAycNFg7PwGBytTqZKIoZm9LHS0WW5GsLno=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=AjBVU6kxWcEpLbOLxWxE6FupsZnMu28XVyflO+vD+K8UdeKPy5kFk9lYwNJVTbm85avFjnmm2wncQav5WkTj4BkWeG0dRD5bQWdFuxDFfet3vAZ0av6a0fDbzBHqVE545FxI+lBFMtUGkwJHdHKFu2nhgH/MjuBCXYtdpffHVG4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=PtCsYF+k; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PtCsYF+k" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-1ecc23e6c9dso14875445ad.2 for ; Thu, 02 May 2024 17:51:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714697465; x=1715302265; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=V8/xoQXqz11CQNwI09Ird7+bH6VcV03qRwDFMsnqJvs=; b=PtCsYF+kfqsh0WA1F+WMj9uRX2OUC87d31j8gU4VmKIJE40+ZoAhHX3NDaKn0WaWC6 DODmW6rRlxNG/oIfNB+cWPCbTXDLA91wksIWA98JbbJGdKsYyi2Qpw1LK/CZei0pxL4d aHN2otiyBaFyyZvqd5zNwn/l/Q666BB2dOqG9l1IlsstcoFTshG+AXm4SnSxRnLddWqe MXDFWo1+nVxTQH4cAvePAIFObE7jbfb/ifErhz2oDvFqFsatrWTtMccWwuDuTiZUvAZ2 a/iYjl9Q67bgdFMNL07kOVim/pCEvdBROXzhcQnwBVabgZJVlGULb5149Y+OdoFtshrb qz8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714697465; x=1715302265; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=V8/xoQXqz11CQNwI09Ird7+bH6VcV03qRwDFMsnqJvs=; b=LlUxmAlV1yIflLgyo421/5I4SyS+hQscqlhB7wiSd2xOv8kj5BzJA/TGJBXxF2InFB vBsaF5GxcWPXQ8cDRxLjk1AiVrxmRCcq638KoO8OXHRe2aysRHi7RwxGcdWyuRZLUBhg ehz/OqSG36kTg1mdrpnr7GIhmkaZWXJ37I5VmvC/eqBOGRlVrVu+AyWDhgLDo9Ene69t /ky2UgUuFJWIpLIE3+i/tNeExoUJuARFIido5I96U8DNtx+5cUjWpD7oAYLkaxZaNoyi bKrZYOw69zAcm3UzMaITOlOVZ27RSX5RfFjjgUhqNi4+UOtd6xZNgLU1+XE19VFwZxHA BYeg== X-Forwarded-Encrypted: i=1; AJvYcCVdOCxF3/nskE0pCNRXk2N9ZCDR2jTsPKWIGY4fS5qa6snjvDda1EYuwYLV2pXm1fWqrgUT93cm7XxHYOa7iTOwBLywANX6c9qmc+zh X-Gm-Message-State: AOJu0YyydHeevTCQzXEtjqZpy2xMvyxUKDtsOQQqnzRKwCJcPs2MsK2M GlBNa5BOSLOoHZqhuJM9rmw6AogJxxDDPYFUnvgX7oZ/uVruiqln X-Google-Smtp-Source: AGHT+IGFQTJEucm2ac55Okxx+HqgTWR7zEJF87WiBhCpMaLEsrJyrBBOIayzx6pibugr3l3cuGjLhA== X-Received: by 2002:a17:903:2384:b0:1ed:36dc:a570 with SMTP id v4-20020a170903238400b001ed36dca570mr620343plh.49.1714697465334; Thu, 02 May 2024 17:51:05 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d14-20020a170903230e00b001ec48e41d2esm1969175plh.212.2024.05.02.17.50.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 May 2024 17:51:04 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v3 3/6] mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally Date: Fri, 3 May 2024 12:50:20 +1200 Message-Id: <20240503005023.174597-4-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240503005023.174597-1-21cnbao@gmail.com> References: <20240503005023.174597-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song There could arise a necessity to obtain the first pte_t from a swap pte_t located in the middle. For instance, this may occur within the context of do_swap_page(), where a page fault can potentially occur in any PTE of a large folio. To address this, the following patch introduces pte_move_swp_offset(), a function capable of bidirectional movement by a specified delta argument. Consequently, pte_increment_swp_offset() will directly invoke it with delta =3D 1. Suggested-by: "Huang, Ying" Signed-off-by: Barry Song Reviewed-by: "Huang, Ying" Reviewed-by: Ryan Roberts --- mm/internal.h | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index c5552d35d995..cfe4aed66a5c 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio= , unsigned long addr, } =20 /** - * pte_next_swp_offset - Increment the swap entry offset field of a swap p= te. + * pte_move_swp_offset - Move the swap entry offset field of a swap pte + * forward or backward by delta * @pte: The initial pte state; is_swap_pte(pte) must be true and * non_swap_entry() must be false. + * @delta: The direction and the offset we are moving; forward if delta + * is positive; backward if delta is negative * - * Increments the swap offset, while maintaining all other fields, includi= ng + * Moves the swap offset, while maintaining all other fields, including * swap type, and any swp pte bits. The resulting pte is returned. */ -static inline pte_t pte_next_swp_offset(pte_t pte) +static inline pte_t pte_move_swp_offset(pte_t pte, long delta) { swp_entry_t entry =3D pte_to_swp_entry(pte); pte_t new =3D __swp_entry_to_pte(__swp_entry(swp_type(entry), - (swp_offset(entry) + 1))); + (swp_offset(entry) + delta))); =20 if (pte_swp_soft_dirty(pte)) new =3D pte_swp_mksoft_dirty(new); @@ -234,6 +237,20 @@ static inline pte_t pte_next_swp_offset(pte_t pte) return new; } =20 + +/** + * pte_next_swp_offset - Increment the swap entry offset field of a swap p= te. + * @pte: The initial pte state; is_swap_pte(pte) must be true and + * non_swap_entry() must be false. + * + * Increments the swap offset, while maintaining all other fields, includi= ng + * swap type, and any swp pte bits. The resulting pte is returned. + */ +static inline pte_t pte_next_swp_offset(pte_t pte) +{ + return pte_move_swp_offset(pte, 1); +} + /** * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries * @start_ptep: Page table pointer for the first entry. --=20 2.34.1 From nobody Sat Feb 7 04:16:37 2026 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C3E3BE58 for ; Fri, 3 May 2024 00:51:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714697474; cv=none; b=daGxm3wuBAd1FpyIo0QtgS+2NSbJsQcUCcZOdO/Dli7FT+E6hFcpjHhFXQae6uAXAFc2umjPuakD1eQDca8vLmIjl+FoltgzLNyPIAG7l9VOFSm9Sd04FKwEmxCoVNBsA2DENdzQr3p79TtUiPDbC97EHb8MSIgm727j+M1gbZE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714697474; c=relaxed/simple; bh=ZbPZGuPdHECEmvacz8mhsy2+SQbopck4uFKIvnD1TxE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=n87jm3vUznv0ADVtOwn+p1ptEh4CT6A3sDqRacrfokytJn/QlsziSZX5ImGCHDv75flsUy7m5vTy9ng2RRpx9/1HHdDr2yIdF/5xZxrchKxmHjTMWJWafSXngtJgc4NsVAKvyVsR27aunY7+7sWjk7ARoACiu800p0iJGtZmZT4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=bcU1fJOd; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bcU1fJOd" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-1e4c4fb6af3so17831065ad.0 for ; Thu, 02 May 2024 17:51:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714697473; x=1715302273; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=iTZWC4mSVwFMl/e34a7NVZCzCVLqGdUKErQlq6SYz3Y=; b=bcU1fJOd3Yo0cWsVowsNLFjY/9423bHgJUWDdKVh0UeKs5L3NBNG79MTisoR1nTQrU TZDVU4UEyYdfzXDo3gQFaAxCGMV4/uRvwUW6d58oUBp3NUJHs7iZmDu5kQ297kGe+xHw 5cBx8yG2lGoxnpIxFNE8eZfdp4FDsfVp37v8SFY4gbJmGIVD7BmKKmoPcjzhPj/4HbxT VgFpYkmHx4MRmPUhBZ9sSQolD27+1GgDVgpNC5cFqJhhq19rwHnajoZ/xHxRaD6xUXbg oEw2+HycVlyT2PjIGQoHsC98scKeODMG2ySYrw79qOk0t+rHrSALBQX1evxgRXOqNbMR 3WtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714697473; x=1715302273; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iTZWC4mSVwFMl/e34a7NVZCzCVLqGdUKErQlq6SYz3Y=; b=XxDyQpX9xQ2TVrTomVxbOun4kTFZwm/gQN0SziOpxPmQPG1kAun01v1OXuwJENcimF eBIKL6+/+RkG1iy92+gTEXD3bEgK4bdpgqK22Ok/oz8z6oz7kOW55iQAHrPXh7asxkhx nLY83roLP3KdW9YL8ZmFSrCD4F4YHiwRNjsE5GJ2HrHwm6zyQx/uWclGAd6gFZLsVZva bHPWJuoMn4ofJRJHhZjIpW6X5WxQaVlgfapBUW5ajSLtInviaO+IlOhJpKEU5ApBBKD9 oO2ZkVBIDCRgV3kyW7GxmQ8GPyX36av2G1IgKgOIcnjRV4gUKc7Z24ojiQGX3T4GoRor jXfw== X-Forwarded-Encrypted: i=1; AJvYcCWn0KvoZwK9HsL3TSl2/CMHeRzDfIK/i8K7XRn2odnc3g1XwJ+H1piJIUYo/BZCPgZBNVFfUORSD9wn4LRiV5xuakZ0Zl5eMQWDBNMu X-Gm-Message-State: AOJu0YxvZ9YPGTBP7sJex3425459D8WU8j6MZ7KRt8NotjKVlnU6SXdV pJS1meOUWMVc2B3Zkrrq5xLhqTlLBSmzJ+GuFzJ4W/vHb963USw7 X-Google-Smtp-Source: AGHT+IF6KPbkDG4oPcXF3nnGu2vkHcu3vUbx1MmrFOaaiRUAVvVI/eyC0SeYLYYd7YsLphvtZmA6tw== X-Received: by 2002:a17:902:ecc4:b0:1e4:59a2:d7c1 with SMTP id a4-20020a170902ecc400b001e459a2d7c1mr6562669plh.33.1714697472877; Thu, 02 May 2024 17:51:12 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d14-20020a170903230e00b001ec48e41d2esm1969175plh.212.2024.05.02.17.51.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 May 2024 17:51:12 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, Khalid Aziz , "David S. Miller" , Andreas Larsson Subject: [PATCH v3 4/6] mm: introduce arch_do_swap_page_nr() which allows restore metadata for nr pages Date: Fri, 3 May 2024 12:50:21 +1200 Message-Id: <20240503005023.174597-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240503005023.174597-1-21cnbao@gmail.com> References: <20240503005023.174597-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Barry Song Should do_swap_page() have the capability to directly map a large folio, metadata restoration becomes necessary for a specified number of pages denoted as nr. It's important to highlight that metadata restoration is solely required by the SPARC platform, which, however, does not enable THP_SWAP. Consequently, in the present kernel configuration, there exists no practical scenario where users necessitate the restoration of nr metadata. Platforms implementing THP_SWAP might invoke this function with nr values exceeding 1, subsequent to do_swap_page() successfully mapping an entire large folio. Nonetheless, their arch_do_swap_page_nr() functions remain empty. Cc: Khalid Aziz Cc: "David S. Miller" Cc: Andreas Larsson Signed-off-by: Barry Song Reviewed-by: Ryan Roberts --- include/linux/pgtable.h | 26 ++++++++++++++++++++------ mm/memory.c | 3 ++- 2 files changed, 22 insertions(+), 7 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 18019f037bae..463e84c3de26 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1084,6 +1084,15 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) }) =20 #ifndef __HAVE_ARCH_DO_SWAP_PAGE +static inline void arch_do_swap_page_nr(struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long addr, + pte_t pte, pte_t oldpte, + int nr) +{ + +} +#else /* * Some architectures support metadata associated with a page. When a * page is being swapped out, this metadata must be saved so it can be @@ -1092,12 +1101,17 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) * page as metadata for the page. arch_do_swap_page() can restore this * metadata when a page is swapped back in. */ -static inline void arch_do_swap_page(struct mm_struct *mm, - struct vm_area_struct *vma, - unsigned long addr, - pte_t pte, pte_t oldpte) -{ - +static inline void arch_do_swap_page_nr(struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long addr, + pte_t pte, pte_t oldpte, + int nr) +{ + for (int i =3D 0; i < nr; i++) { + arch_do_swap_page(vma->vm_mm, vma, addr + i * PAGE_SIZE, + pte_advance_pfn(pte, i), + pte_advance_pfn(oldpte, i)); + } } #endif =20 diff --git a/mm/memory.c b/mm/memory.c index f033eb3528ba..74cdefd58f5f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4266,7 +4266,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); + arch_do_swap_page_nr(vma->vm_mm, vma, vmf->address, + pte, vmf->orig_pte, 1); =20 folio_unlock(folio); if (folio !=3D swapcache && swapcache) { --=20 2.34.1 From nobody Sat Feb 7 04:16:37 2026 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 573D4BE58 for ; Fri, 3 May 2024 00:51:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714697481; cv=none; b=mgXRl74wZZlPcVZjEOX7A4f/xvEvDUZApOQs/JdaqNyEVnXZHUfyqMeensyplcl165T6sH2bSJXmxjJvif8hm5XRq4yoE04WD9j38QojW4CBGzUBSlsunsNne6oYgIiHfw3rf2W3Aj005RfWq+vFqZ5G0rf4xj4Quxbl144K5zY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714697481; c=relaxed/simple; bh=gJXAWzFv78CWKiMkFGxOs0FBDTmpFMkGrRWrHMYNEvg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dmNtQGyrmsU/i5srvx4Kdlg9Rw+IwKftu7IprYK1MIOf2zAb2EmCgUFTKx1JXDg+6Qpl/cZmIQbwIVD+uFTiHZfj7AvyVrjSVr0r+EWnt8BKk1yMoUNEyAUOBsx+AaDQC4XaNeLnDjn/JiMplGua9QF76R4TtGrIVNmdQns0eXs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=MYuvJsq0; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MYuvJsq0" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-1e3c3aa8938so57656395ad.1 for ; Thu, 02 May 2024 17:51:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714697480; x=1715302280; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Y50dQl+upnub8ly1QPHs/qiP8XcDN3yKo0FUp2Qgag8=; b=MYuvJsq08lktuMOh041jlzvejjCgzdlJ3H4+EWA+3PuXfjRgnALz7y6PKWnP4+W2U/ u8wRAXLSpEOV2livtIF285csmgnyRFP6bZm8NRG2mp/CatmDijhZojNFgH887ljPteSy Aw6aM5mOxnSq2PGPecd0w4t26S6ywNj1omgFPxTnu7e1FHJh2ouzRZ0SoML5/wIZTh3s sCE0+UUD7bSSj2orz0rQZN7ys0FjwWqRak6zFk3pCgc93N6tLT9XytMOpk5tUDJOxicg AUzlnJNskosgbEMSRJDqlv66tMSPWCvrcYp7Jqro6s2FSDUGaMCOAu7ruQooILSzeQGp 7kpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714697480; x=1715302280; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Y50dQl+upnub8ly1QPHs/qiP8XcDN3yKo0FUp2Qgag8=; b=WeQFLwifBENjzHNL/ileqNEHbrVzicww4NNg5so2l3ontq3qboxBFn6i4MaC/BOUx8 UZioVrK22IeIcbcVBeUA8FRfYB9IvD46ibY7HyqaQAxVswPuKgTjJOOvZTbLYXtmjQBc jUbZjv+K4fJzF5rKJDu9sMlgBUSqSFth2nEc/JDMl891lnAXNXSJ3uEDIhSXeEwYtzzm 5nLO53jEuRbrJtfw+1bnX1s0u2FkDNmEfDT+UphMeFoSFmxGLAX3VpOi2QyLqkE6v3MN 6LQ5a+cEjcYp20i5AO98Nxsm6wfv8k27PlJdg3XmBFIVpuAzYt4Ii/hVQT3l4AfQ1W+X LRMA== X-Forwarded-Encrypted: i=1; AJvYcCWbL70W3Gg0XNP2VYLwKpvubYU+rLqwOKMSoFU4Mbm2eeGOfJiORenke5QJ1LJ3lUJODN+X6jq6iakRacBfYJRPjfjiR3MmqXnlA016 X-Gm-Message-State: AOJu0YxrWOVNisD6th+xNbHyvXKP159m590CqpNIjcoWGIj9VGGQSsrx 2y6SpfgmUs+bREBPASwMus1gOOHiA19D2ACovmZF0yQF0N7K2Bn8 X-Google-Smtp-Source: AGHT+IF/YGckt6lCQLRiwu1+5wNzKcMLcZACnvpyDGbcY6XjGmdPdZXMo8MwxWmuAEpW19MZtkC8pA== X-Received: by 2002:a17:902:ce8b:b0:1e4:362b:17d5 with SMTP id f11-20020a170902ce8b00b001e4362b17d5mr1757897plg.4.1714697479694; Thu, 02 May 2024 17:51:19 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d14-20020a170903230e00b001ec48e41d2esm1969175plh.212.2024.05.02.17.51.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 May 2024 17:51:19 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v3 5/6] mm: swap: make should_try_to_free_swap() support large-folio Date: Fri, 3 May 2024 12:50:22 +1200 Message-Id: <20240503005023.174597-6-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240503005023.174597-1-21cnbao@gmail.com> References: <20240503005023.174597-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Chuanhua Han The function should_try_to_free_swap() operates under the assumption that swap-in always occurs at the normal page granularity, i.e., folio_nr_pages() =3D 1. However, in reality, for large folios, add_to_swap_cache() will invoke folio_ref_add(folio, nr). To accommodate large folio swap-in, this patch eliminates this assumption. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Acked-by: Chris Li Reviewed-by: Ryan Roberts Reviewed-by: "Huang, Ying" Reviewed-by: David Hildenbrand --- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 74cdefd58f5f..22e7c33cc747 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3877,7 +3877,7 @@ static inline bool should_try_to_free_swap(struct fol= io *folio, * reference only in case it's likely that we'll be the exlusive user. */ return (fault_flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) && - folio_ref_count(folio) =3D=3D 2; + folio_ref_count(folio) =3D=3D (1 + folio_nr_pages(folio)); } =20 static vm_fault_t pte_marker_clear(struct vm_fault *vmf) --=20 2.34.1 From nobody Sat Feb 7 04:16:37 2026 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 289C015E9B for ; Fri, 3 May 2024 00:51:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714697488; cv=none; b=sTDOhvIe6k4PCAJ93IottPt4Z0vjo18gkGwXtjyPtoVbtO5DcF8t6FbJbl5RLISjesWJ+2Q/KeEoxpAjk8VVzv1FBOibZ3oqlDw+7HH7Ie180edMUddPKmvpaai/f/OxRzP3KieL18mEac+VUdjjzXFsYb6sLj3T1dreWeOPww0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714697488; c=relaxed/simple; bh=YzHhYMj3uQPxfTb6VikRZEwdmbE7GFcDdgDgNuXcpRk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=SXjNW4f5ts10s2wf/7Bb2bdVOpfsHCznOJ/RkyFcmy4hAU9mSSlWblsC9BFxtHXCkZ2X0JZtPiLn74Sb0wqREeOzhKZw9HYUHffNAdOIeas0INI/FtF52AjP1BcvjIbiz0dMIp9lhg3bLLpWbNADSA0ahbA3UXDyF+UA48q0fFI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=i+jvNSbB; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="i+jvNSbB" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-1ed012c1afbso9547405ad.1 for ; Thu, 02 May 2024 17:51:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714697486; x=1715302286; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pCPlcQiDc4aEgZRUGIFwOS4XTVcepx+utYhjOWkqIL0=; b=i+jvNSbBO0RFzCaTpVfmycbfcuty+Qt26T6FJufmncje1bqOB5uuUv4xVxxfLGRsKh sQYVLOgoOAAzHEH6d7vdSS9YQJYo/jg0Z/kbrrVWq5ixVJbV089nBmuvvLDLqGp3epMD DQd2YHAVBnLeLnH1PnHnY3hTBzBG3+Kh0/Q5gwXL+3enQkt65ECRNK0uvRsa0U3QEx7v xEAaW4Csi3ovx+VZWu1x+xAnE5xGWCyulUzGWND3f3x+DWHfsIBzR/yRIfnUSGrnuPtN g1x4zbV8v1NzuvZAMDBrYzWtKVcLZds7j0Szat1Jg7BBwm+6eKeRPOlAYJ/jWCJDGfbV xuhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714697486; x=1715302286; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pCPlcQiDc4aEgZRUGIFwOS4XTVcepx+utYhjOWkqIL0=; b=dTd9giwMEuBQk0pvX3pmLz+E1KR+VVyZLfyGS3xJgzYvvuqglq4SD8eR1bNp09bQLR rmRE9TdGBtFh/iHLNac+Te4CLZ+6Sy0PrhA2jF3s1qbsN8XcLbJAu5+aVo+YjzqDIotd k6SepZ9AeuQXJ8VwzaIbDzzdEGPDQQwQY4AoeH8M1zp7cX861kRb4nH8wbrv+uGy7gHs tHhgN7NqB+L9BVVKZko1JeDmXQJo0kPJuVioXEutsv74ISrhlFSOoAccDKzJk5Vruci9 Tu2cPNAbGRjlRWQa7er6w2pHUSGjEy2Iwd4zA5qJX5+JFoviwU665XMUwOF+U/jk7bi8 xTtg== X-Forwarded-Encrypted: i=1; AJvYcCVrAsF0YQcWbrnfHhoQbS1S0NZ2iJI1ujStgOS1qN08aZh5yv5YsUiuYZEOYz/G+EmSNVAuhrkr0WqYaKec36vQCjmncQWbrF1Wq7xm X-Gm-Message-State: AOJu0Yz+Qb50NpHa7UuSc2dH1zOO16Mc0oGnUIoCEJp+G1XsScso/m6U sdAbkK+bItHuAx26cno2ZT8R48G8Zg5d+kgsxv5Ei4KpCnD9fKxc X-Google-Smtp-Source: AGHT+IHl/5rfXtgmTPFxtFxlvOgwDmCzyiDZXrRPvhqN/z1hpeuCBJnRPssnBcPqlIhHPeY6Yn+CcQ== X-Received: by 2002:a17:902:c407:b0:1e5:e676:4b09 with SMTP id k7-20020a170902c40700b001e5e6764b09mr1726427plk.25.1714697486404; Thu, 02 May 2024 17:51:26 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d14-20020a170903230e00b001ec48e41d2esm1969175plh.212.2024.05.02.17.51.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 May 2024 17:51:26 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v3 6/6] mm: swap: entirely map large folios found in swapcache Date: Fri, 3 May 2024 12:50:23 +1200 Message-Id: <20240503005023.174597-7-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240503005023.174597-1-21cnbao@gmail.com> References: <20240503005023.174597-1-21cnbao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Chuanhua Han When a large folio is found in the swapcache, the current implementation requires calling do_swap_page() nr_pages times, resulting in nr_pages page faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Reviewed-by: Ryan Roberts --- mm/memory.c | 60 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 48 insertions(+), 12 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 22e7c33cc747..940fdbe69fa1 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3968,6 +3968,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte_t pte; vm_fault_t ret =3D 0; void *shadow =3D NULL; + int nr_pages =3D 1; + unsigned long page_idx =3D 0; + unsigned long address =3D vmf->address; + pte_t *ptep; =20 if (!pte_unmap_same(vmf)) goto out; @@ -4166,6 +4170,36 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } =20 + ptep =3D vmf->pte; + if (folio_test_large(folio) && folio_test_swapcache(folio)) { + int nr =3D folio_nr_pages(folio); + unsigned long idx =3D folio_page_idx(folio, page); + unsigned long folio_start =3D vmf->address - idx * PAGE_SIZE; + unsigned long folio_end =3D folio_start + nr * PAGE_SIZE; + pte_t *folio_ptep; + pte_t folio_pte; + + if (unlikely(folio_start < max(vmf->address & PMD_MASK, vma->vm_start))) + goto check_folio; + if (unlikely(folio_end > pmd_addr_end(vmf->address, vma->vm_end))) + goto check_folio; + + folio_ptep =3D vmf->pte - idx; + folio_pte =3D ptep_get(folio_ptep); + if (!pte_same(folio_pte, pte_move_swp_offset(vmf->orig_pte, -idx)) || + swap_pte_batch(folio_ptep, nr, folio_pte) !=3D nr) + goto check_folio; + + page_idx =3D idx; + address =3D folio_start; + ptep =3D folio_ptep; + nr_pages =3D nr; + entry =3D folio->swap; + page =3D &folio->page; + } + +check_folio: + /* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4225,12 +4259,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * We're already holding a reference on the page but haven't mapped it * yet. */ - swap_free_nr(entry, 1); + swap_free_nr(entry, nr_pages); if (should_try_to_free_swap(folio, vma, vmf->flags)) folio_free_swap(folio); =20 - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - dec_mm_counter(vma->vm_mm, MM_SWAPENTS); + folio_ref_add(folio, nr_pages - 1); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); + add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages); pte =3D mk_pte(page, vma->vm_page_prot); =20 /* @@ -4240,34 +4275,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * exclusivity. */ if (!folio_test_ksm(folio) && - (exclusive || folio_ref_count(folio) =3D=3D 1)) { + (exclusive || (folio_ref_count(folio) =3D=3D nr_pages && + folio_nr_pages(folio) =3D=3D nr_pages))) { if (vmf->flags & FAULT_FLAG_WRITE) { pte =3D maybe_mkwrite(pte_mkdirty(pte), vma); vmf->flags &=3D ~FAULT_FLAG_WRITE; } rmap_flags |=3D RMAP_EXCLUSIVE; } - flush_icache_page(vma, page); + flush_icache_pages(vma, page, nr_pages); if (pte_swp_soft_dirty(vmf->orig_pte)) pte =3D pte_mksoft_dirty(pte); if (pte_swp_uffd_wp(vmf->orig_pte)) pte =3D pte_mkuffd_wp(pte); - vmf->orig_pte =3D pte; + vmf->orig_pte =3D pte_advance_pfn(pte, page_idx); =20 /* ksm created a completely new copy */ if (unlikely(folio !=3D swapcache && swapcache)) { - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_add_new_anon_rmap(folio, vma, address); folio_add_lru_vma(folio, vma); } else { - folio_add_anon_rmap_pte(folio, page, vma, vmf->address, + folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, address, rmap_flags); } =20 VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page_nr(vma->vm_mm, vma, vmf->address, - pte, vmf->orig_pte, 1); + set_ptes(vma->vm_mm, address, ptep, pte, nr_pages); + arch_do_swap_page_nr(vma->vm_mm, vma, address, + pte, pte, nr_pages); =20 folio_unlock(folio); if (folio !=3D swapcache && swapcache) { @@ -4291,7 +4327,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } =20 /* No need to invalidate - it was non-present before */ - update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + update_mmu_cache_range(vmf, vma, address, ptep, nr_pages); unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl); --=20 2.34.1