From nobody Tue Jun 16 17:59:27 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EF4BB349AE6; Thu, 30 Apr 2026 04:04:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777521899; cv=none; b=EtHsea6m9wQ+bLj4BD2x1UXfNplnqGkg+Xe5+VXOOB1jLve5spzLOBWpA2JFhEBDxRRGTGeB7eInR6cXk0CJvNFbzE8tJJgmWS52bxKvY96yXHyaJZfE5kDYKbY3E85R5nnznJ9MV8KgLRe0nGpiNvCWe7FYxTmd1kTx2Wlhs7w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777521899; c=relaxed/simple; bh=Bph0u8wp8/lT6x+zIK7zTGylHyQ++q1X8M3Sbba/pLg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=m4MA80Mg7cZKDpmVEwUEtAQT8sLwL4DC1HEN9BYsNOYcTNf7YooCxZowMVCfROsjoNPduMpjk16IDk0DsVga70tDtJODbHY2fTSUa7HtNReUiSccTV3EtFf/Mi0ag/mAPfxmDxoWP30RcDOj7g6AZLfDA27oB+TCqoPIhZuV5nM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=tDZQx6oM; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="tDZQx6oM" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BA3EFC2BCC7; Thu, 30 Apr 2026 04:04:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1777521898; bh=Bph0u8wp8/lT6x+zIK7zTGylHyQ++q1X8M3Sbba/pLg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tDZQx6oMyj/Pbp4oBZWlhtLtNlz20xltWaQQB7lkuzfNAsCqOFPj1LDkODNunAa/l 2By5McbFvCHbx4vd1YOH1CgGYWq47II+aDe8CMmYB3kuvppesXkGksqverSZ3qGPIU VKF7cIFaSqdwzevlCnjgHtL/gURYNW12fE4YtXp4gq3rPI4AQcGoZQHGnEYTMVrI99 T88uENlzRWQhIDjN/pZcjvAfUvgg81aDjgHE6Gwn7VPmdmS8mvmNv0wWK/a+u+e/6G Saw+bXv1YLhhn5fwyqZhztWdHAozVqlrG2Vsrs64/3e0lRUNv1XjbrN+Qi0mKHxbyF M0Zmb6n7L2IMw== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org, willy@infradead.org Cc: david@kernel.org, ljs@kernel.org, liam@infradead.org, vbabka@kernel.org, rppt@kernel.org, surenb@google.com, mhocko@suse.com, jack@suse.cz, pfalcato@suse.de, wanglian@kylinos.cn, chentao@kylinos.cn, lianux.mm@gmail.com, kunwu.chan@gmail.com, liyangouwen1@oppo.com, chrisl@kernel.org, kasong@tencent.com, shikemeng@huaweicloud.com, nphamcs@gmail.com, bhe@redhat.com, youngjun.park@lge.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, Barry Song Subject: [PATCH v2 1/5] mm/filemap: Retry fault by VMA lock if the lock was released for I/O Date: Thu, 30 Apr 2026 12:04:23 +0800 Message-Id: <20260430040427.4672-2-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260430040427.4672-1-baohua@kernel.org> References: <20260430040427.4672-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Oven Liyang If the current page fault is using the per-VMA lock, and we only released the lock to wait for I/O completion (e.g., using folio_lock()), then when the fault is retried after the I/O completes, it should still qualify for the per-VMA-lock path. Acked-by: Pedro Falcato Tested-by: Wang Lian Tested-by: Kunwu Chan Reviewed-by: Wang Lian Reviewed-by: Kunwu Chan Signed-off-by: Oven Liyang Co-developed-by: Barry Song Signed-off-by: Barry Song --- arch/arm/mm/fault.c | 5 +++++ arch/arm64/mm/fault.c | 5 +++++ arch/loongarch/mm/fault.c | 4 ++++ arch/powerpc/mm/fault.c | 5 ++++- arch/riscv/mm/fault.c | 4 ++++ arch/s390/mm/fault.c | 4 ++++ arch/x86/mm/fault.c | 4 ++++ include/linux/mm_types.h | 9 +++++---- mm/filemap.c | 5 ++++- 9 files changed, 39 insertions(+), 6 deletions(-) diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c index e62cc4be5adf..5971e02845f7 100644 --- a/arch/arm/mm/fault.c +++ b/arch/arm/mm/fault.c @@ -391,6 +391,7 @@ do_page_fault(unsigned long addr, unsigned int fsr, str= uct pt_regs *regs) if (!(flags & FAULT_FLAG_USER)) goto lock_mmap; =20 +retry_vma: vma =3D lock_vma_under_rcu(mm, addr); if (!vma) goto lock_mmap; @@ -420,6 +421,10 @@ do_page_fault(unsigned long addr, unsigned int fsr, st= ruct pt_regs *regs) goto no_context; return 0; } + + /* If the first try is only about waiting for the I/O to complete */ + if (fault & VM_FAULT_RETRY_VMA) + goto retry_vma; lock_mmap: =20 retry: diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 739800835920..d0362a3e11b7 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -673,6 +673,7 @@ static int __kprobes do_page_fault(unsigned long far, u= nsigned long esr, if (!(mm_flags & FAULT_FLAG_USER)) goto lock_mmap; =20 +retry_vma: vma =3D lock_vma_under_rcu(mm, addr); if (!vma) goto lock_mmap; @@ -719,6 +720,10 @@ static int __kprobes do_page_fault(unsigned long far, = unsigned long esr, goto no_context; return 0; } + + /* If the first try is only about waiting for the I/O to complete */ + if (fault & VM_FAULT_RETRY_VMA) + goto retry_vma; lock_mmap: =20 retry: diff --git a/arch/loongarch/mm/fault.c b/arch/loongarch/mm/fault.c index 2c93d33356e5..738f495560c0 100644 --- a/arch/loongarch/mm/fault.c +++ b/arch/loongarch/mm/fault.c @@ -219,6 +219,7 @@ static void __kprobes __do_page_fault(struct pt_regs *r= egs, if (!(flags & FAULT_FLAG_USER)) goto lock_mmap; =20 +retry_vma: vma =3D lock_vma_under_rcu(mm, address); if (!vma) goto lock_mmap; @@ -265,6 +266,9 @@ static void __kprobes __do_page_fault(struct pt_regs *r= egs, no_context(regs, write, address); return; } + /* If the first try is only about waiting for the I/O to complete */ + if (fault & VM_FAULT_RETRY_VMA) + goto retry_vma; lock_mmap: =20 retry: diff --git a/arch/powerpc/mm/fault.c b/arch/powerpc/mm/fault.c index 806c74e0d5ab..cb7ffc20c760 100644 --- a/arch/powerpc/mm/fault.c +++ b/arch/powerpc/mm/fault.c @@ -487,6 +487,7 @@ static int ___do_page_fault(struct pt_regs *regs, unsig= ned long address, if (!(flags & FAULT_FLAG_USER)) goto lock_mmap; =20 +retry_vma: vma =3D lock_vma_under_rcu(mm, address); if (!vma) goto lock_mmap; @@ -516,7 +517,9 @@ static int ___do_page_fault(struct pt_regs *regs, unsig= ned long address, =20 if (fault_signal_pending(fault, regs)) return user_mode(regs) ? 0 : SIGBUS; - + /* If the first try is only about waiting for the I/O to complete */ + if (fault & VM_FAULT_RETRY_VMA) + goto retry_vma; lock_mmap: =20 /* When running in the kernel we expect faults to occur only to diff --git a/arch/riscv/mm/fault.c b/arch/riscv/mm/fault.c index 04ed6f8acae4..b94cf57c2b9a 100644 --- a/arch/riscv/mm/fault.c +++ b/arch/riscv/mm/fault.c @@ -347,6 +347,7 @@ void handle_page_fault(struct pt_regs *regs) if (!(flags & FAULT_FLAG_USER)) goto lock_mmap; =20 +retry_vma: vma =3D lock_vma_under_rcu(mm, addr); if (!vma) goto lock_mmap; @@ -376,6 +377,9 @@ void handle_page_fault(struct pt_regs *regs) no_context(regs, addr); return; } + /* If the first try is only about waiting for the I/O to complete */ + if (fault & VM_FAULT_RETRY_VMA) + goto retry_vma; lock_mmap: =20 retry: diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c index 191cc53caead..e0576e629f65 100644 --- a/arch/s390/mm/fault.c +++ b/arch/s390/mm/fault.c @@ -294,6 +294,7 @@ static void do_exception(struct pt_regs *regs, int acce= ss) flags |=3D FAULT_FLAG_WRITE; if (!(flags & FAULT_FLAG_USER)) goto lock_mmap; +retry_vma: vma =3D lock_vma_under_rcu(mm, address); if (!vma) goto lock_mmap; @@ -318,6 +319,9 @@ static void do_exception(struct pt_regs *regs, int acce= ss) handle_fault_error_nolock(regs, 0); return; } + /* If the first try is only about waiting for the I/O to complete */ + if (fault & VM_FAULT_RETRY_VMA) + goto retry_vma; lock_mmap: retry: vma =3D lock_mm_and_find_vma(mm, address, regs); diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index f0e77e084482..0589fc693eea 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -1322,6 +1322,7 @@ void do_user_addr_fault(struct pt_regs *regs, if (!(flags & FAULT_FLAG_USER)) goto lock_mmap; =20 +retry_vma: vma =3D lock_vma_under_rcu(mm, address); if (!vma) goto lock_mmap; @@ -1351,6 +1352,9 @@ void do_user_addr_fault(struct pt_regs *regs, ARCH_DEFAULT_PKEY); return; } + /* If the first try is only about waiting for the I/O to complete */ + if (fault & VM_FAULT_RETRY_VMA) + goto retry_vma; lock_mmap: =20 retry: diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index a308e2c23b82..5907200ea587 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1678,10 +1678,11 @@ enum vm_fault_reason { VM_FAULT_NOPAGE =3D (__force vm_fault_t)0x000100, VM_FAULT_LOCKED =3D (__force vm_fault_t)0x000200, VM_FAULT_RETRY =3D (__force vm_fault_t)0x000400, - VM_FAULT_FALLBACK =3D (__force vm_fault_t)0x000800, - VM_FAULT_DONE_COW =3D (__force vm_fault_t)0x001000, - VM_FAULT_NEEDDSYNC =3D (__force vm_fault_t)0x002000, - VM_FAULT_COMPLETED =3D (__force vm_fault_t)0x004000, + VM_FAULT_RETRY_VMA =3D (__force vm_fault_t)0x000800, + VM_FAULT_FALLBACK =3D (__force vm_fault_t)0x001000, + VM_FAULT_DONE_COW =3D (__force vm_fault_t)0x002000, + VM_FAULT_NEEDDSYNC =3D (__force vm_fault_t)0x004000, + VM_FAULT_COMPLETED =3D (__force vm_fault_t)0x008000, VM_FAULT_HINDEX_MASK =3D (__force vm_fault_t)0x0f0000, }; =20 diff --git a/mm/filemap.c b/mm/filemap.c index ab34cab2416a..a045b771e8de 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3525,6 +3525,7 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) struct folio *folio; vm_fault_t ret =3D 0; bool mapping_locked =3D false; + bool retry_by_vma_lock =3D false; =20 max_idx =3D DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); if (unlikely(index >=3D max_idx)) @@ -3621,6 +3622,8 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) */ if (fpin) { folio_unlock(folio); + if (vmf->flags & FAULT_FLAG_VMA_LOCK) + retry_by_vma_lock =3D true; goto out_retry; } if (mapping_locked) @@ -3671,7 +3674,7 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) filemap_invalidate_unlock_shared(mapping); if (fpin) fput(fpin); - return ret | VM_FAULT_RETRY; + return ret | VM_FAULT_RETRY | (retry_by_vma_lock ? VM_FAULT_RETRY_VMA : 0= ); } EXPORT_SYMBOL(filemap_fault); =20 --=20 2.39.3 (Apple Git-146) From nobody Tue Jun 16 17:59:27 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 589B23559E1; Thu, 30 Apr 2026 04:05:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777521904; cv=none; b=nJtCdTanY4zjc2/yTa1w8vAX9pDTQk/BZlGD/yru6rJQm8l5w5QIC7/aMDUg0iZYY3/ZpS4VFCEfwi8mExsu5+OVknhKy7EvJa0MZ3GcnCfxvSwVeorKA4fPBIQK2gzIKlVMrlYMCvUfPYx4W/gEbn9EmNeE1YOm+mvd0Qx5ctM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777521904; c=relaxed/simple; bh=3UzddeMl13pC5cJUrT4HHkjYBFF8Z0grqJoknqLkHzk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=SnOShz8To6mIT6H4owX1k8DQRlDqDvbWxVPtLQPN3hnuPrMdGDcslKU1bTKtIZIzVKTKtNg1y0D6Aka4xGdyrpcx4W8fuIF/wjvVAnve859gE0C/oqXPUFflnlweraPvY82UiVgWNiIVj9VKvPX1AxgRzOLyJuY+NeHfdqNdHpw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mKwVglCF; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mKwVglCF" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0C627C2BCC4; Thu, 30 Apr 2026 04:04:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1777521903; bh=3UzddeMl13pC5cJUrT4HHkjYBFF8Z0grqJoknqLkHzk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mKwVglCF7XEsFftOn0BAtLwGM5IZ72wXKgPaG3s9rdnfQtx+pmDdlMhp0BTc5EYNf FJboHwOiMXC33Onbze1GSoNvzLlFPIK9rhRrqSsueRkYACFW51iYCE+t5LiSSUtrDZ 6Zgo/egoBF46WoaKKLJpL+hNtjheakCbbtpFjT8y7TDrK7Ows6M9EqK3rdErLUzivK 4Ay85oZFt3iFrC53ukdn7sqXcw6md9eq3jFLK0YjDszHvyQDD9gHGtkAjd47bH+XH1 stGeQfEGcaAiSsSh1IY9gXD9Yt5DxgBQ/zzQ6kcBG30A7WMR3XO465Eebcb6ZNIjyv BawWjDw+St+Lw== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org, willy@infradead.org Cc: david@kernel.org, ljs@kernel.org, liam@infradead.org, vbabka@kernel.org, rppt@kernel.org, surenb@google.com, mhocko@suse.com, jack@suse.cz, pfalcato@suse.de, wanglian@kylinos.cn, chentao@kylinos.cn, lianux.mm@gmail.com, kunwu.chan@gmail.com, liyangouwen1@oppo.com, chrisl@kernel.org, kasong@tencent.com, shikemeng@huaweicloud.com, nphamcs@gmail.com, bhe@redhat.com, youngjun.park@lge.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, "Barry Song (Xiaomi)" Subject: [PATCH v2 2/5] mm/swapin: Retry swapin by VMA lock if the lock was released for I/O Date: Thu, 30 Apr 2026 12:04:24 +0800 Message-Id: <20260430040427.4672-3-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260430040427.4672-1-baohua@kernel.org> References: <20260430040427.4672-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If the current do_swap_page() took the per-VMA lock and we dropped it only to wait for I/O completion (e.g., use folio_wait_locked()), then when do_swap_page() is retried after the I/O completes, it should still qualify for the per-VMA-lock path. Tested-by: Wang Lian Tested-by: Kunwu Chan Reviewed-by: Wang Lian Reviewed-by: Kunwu Chan Signed-off-by: Barry Song (Xiaomi) --- mm/memory.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 199214f8de08..00ee1599d637 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4791,6 +4791,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) unsigned long page_idx; unsigned long address; pte_t *ptep; + bool retry_by_vma_lock =3D false; =20 if (!pte_unmap_same(vmf)) goto out; @@ -4896,8 +4897,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) =20 swapcache =3D folio; ret |=3D folio_lock_or_retry(folio, vmf); - if (ret & VM_FAULT_RETRY) + if (ret & VM_FAULT_RETRY) { + if (fault_flag_allow_retry_first(vmf->flags) && + !(vmf->flags & FAULT_FLAG_RETRY_NOWAIT) && + (vmf->flags & FAULT_FLAG_VMA_LOCK)) + retry_by_vma_lock =3D true; goto out_release; + } =20 page =3D folio_file_page(folio, swp_offset(entry)); /* @@ -5182,7 +5188,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } if (si) put_swap_device(si); - return ret; + return ret | (retry_by_vma_lock ? VM_FAULT_RETRY_VMA : 0); } =20 static bool pte_range_none(pte_t *pte, int nr_pages) --=20 2.39.3 (Apple Git-146) From nobody Tue Jun 16 17:59:27 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A6214279917; Thu, 30 Apr 2026 04:05:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777521909; cv=none; b=FGZEA+iQw61H6X4XKDxfZzzWuAWPcDXEg34gBhfj0kerjKjQjwxX9LRWpskOiQalHS163+X/SER7JEjxN0OFgnO8EhMXaaFFJcKTiE+Rrrc2DWL8okXs9oR2wzM+qOvPxK80AXRhppA/XQQXIFn0j72edrNb9Yg+ymSsMDEmpdg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777521909; c=relaxed/simple; bh=IOVfryRIB//G7TP35s/rdTIpoHZE33+XU/HgNFu89vQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=QPQNHLyeM7HauSyMbr7nGMe0qW0GVDRmzk78WqDyzHmv574+yJV5//B5jCZA1qJ9MPsev5Z2uFSunHOdpH4cs0LwZQx5kqjx+ezKKKZKyfvzJ8+nYrd+oTY5TpVzQE5tV/lTaiQ8i5VU6P3D4g8NA3nSTZ4GCRxnQ6ug/ZagqS4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=HVHmp1Z+; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="HVHmp1Z+" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 52BA2C2BCB8; Thu, 30 Apr 2026 04:05:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1777521909; bh=IOVfryRIB//G7TP35s/rdTIpoHZE33+XU/HgNFu89vQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HVHmp1Z+siadLb4Q4UHbDBBFXRgvKznPzVZgfJaZuDYbe4T7jnjjPpsDtJInwZIeE 4+I5FPsV3n+vKdFvmZlaVSvEPy/79jVItKW9FKiSkpofZcGT8gtPXTdJbnuWCRBQ7C IotyNjhp+vK1JXwZXT1524wwOq3YNWA8MbefdUygDl892tC+sKqCeBWIognUzoFWBz 1ZfPBOz5ggtDlvNtoxkZ97/ykVcs4f1SLEbGN/iXyxKX9lzWM6c+fEG+I0aotBh4/p pL2UOnf780mijkJBKnrAw7yfy8qSdrHqgtuQ3Q+6qcVgHDWif223rrlC6SFHrfHNEh uuxK+cvO1oYsg== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org, willy@infradead.org Cc: david@kernel.org, ljs@kernel.org, liam@infradead.org, vbabka@kernel.org, rppt@kernel.org, surenb@google.com, mhocko@suse.com, jack@suse.cz, pfalcato@suse.de, wanglian@kylinos.cn, chentao@kylinos.cn, lianux.mm@gmail.com, kunwu.chan@gmail.com, liyangouwen1@oppo.com, chrisl@kernel.org, kasong@tencent.com, shikemeng@huaweicloud.com, nphamcs@gmail.com, bhe@redhat.com, youngjun.park@lge.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, "Barry Song (Xiaomi)" Subject: [PATCH v2 3/5] mm: Move folio_lock_or_retry() and drop __folio_lock_or_retry() Date: Thu, 30 Apr 2026 12:04:25 +0800 Message-Id: <20260430040427.4672-4-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260430040427.4672-1-baohua@kernel.org> References: <20260430040427.4672-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" folio_lock_or_retry() is effectively only used in mm/memory.c, not in the filemap code. Move it there and make it static. The helper __folio_lock_or_retry() can be folded into folio_lock_or_retry(), allowing it to be removed. Signed-off-by: Barry Song (Xiaomi) --- include/linux/pagemap.h | 17 ------------- mm/filemap.c | 45 ---------------------------------- mm/memory.c | 53 +++++++++++++++++++++++++++++++++++++++++ 3 files changed, 53 insertions(+), 62 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 1f50991b43e3..500ab783bf70 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -1101,7 +1101,6 @@ static inline bool wake_page_match(struct wait_page_q= ueue *wait_page, =20 void __folio_lock(struct folio *folio); int __folio_lock_killable(struct folio *folio); -vm_fault_t __folio_lock_or_retry(struct folio *folio, struct vm_fault *vmf= ); void unlock_page(struct page *page); void folio_unlock(struct folio *folio); =20 @@ -1198,22 +1197,6 @@ static inline int folio_lock_killable(struct folio *= folio) return 0; } =20 -/* - * folio_lock_or_retry - Lock the folio, unless this would block and the - * caller indicated that it can handle a retry. - * - * Return value and mmap_lock implications depend on flags; see - * __folio_lock_or_retry(). - */ -static inline vm_fault_t folio_lock_or_retry(struct folio *folio, - struct vm_fault *vmf) -{ - might_sleep(); - if (!folio_trylock(folio)) - return __folio_lock_or_retry(folio, vmf); - return 0; -} - /* * This is exported only for folio_wait_locked/folio_wait_writeback, etc., * and should not be used directly. diff --git a/mm/filemap.c b/mm/filemap.c index a045b771e8de..b532d6cbafc8 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1740,51 +1740,6 @@ static int __folio_lock_async(struct folio *folio, s= truct wait_page_queue *wait) return ret; } =20 -/* - * Return values: - * 0 - folio is locked. - * non-zero - folio is not locked. - * mmap_lock or per-VMA lock has been released (mmap_read_unlock() or - * vma_end_read()), unless flags had both FAULT_FLAG_ALLOW_RETRY and - * FAULT_FLAG_RETRY_NOWAIT set, in which case the lock is still held. - * - * If neither ALLOW_RETRY nor KILLABLE are set, will always return 0 - * with the folio locked and the mmap_lock/per-VMA lock is left unperturbe= d. - */ -vm_fault_t __folio_lock_or_retry(struct folio *folio, struct vm_fault *vmf) -{ - unsigned int flags =3D vmf->flags; - - if (fault_flag_allow_retry_first(flags)) { - /* - * CAUTION! In this case, mmap_lock/per-VMA lock is not - * released even though returning VM_FAULT_RETRY. - */ - if (flags & FAULT_FLAG_RETRY_NOWAIT) - return VM_FAULT_RETRY; - - release_fault_lock(vmf); - if (flags & FAULT_FLAG_KILLABLE) - folio_wait_locked_killable(folio); - else - folio_wait_locked(folio); - return VM_FAULT_RETRY; - } - if (flags & FAULT_FLAG_KILLABLE) { - bool ret; - - ret =3D __folio_lock_killable(folio); - if (ret) { - release_fault_lock(vmf); - return VM_FAULT_RETRY; - } - } else { - __folio_lock(folio); - } - - return 0; -} - /** * page_cache_next_miss() - Find the next gap in the page cache. * @mapping: Mapping. diff --git a/mm/memory.c b/mm/memory.c index 00ee1599d637..0c740ca363cc 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4442,6 +4442,59 @@ void unmap_mapping_range(struct address_space *mappi= ng, } EXPORT_SYMBOL(unmap_mapping_range); =20 +/* + * folio_lock_or_retry - Lock the folio, unless this would block and the + * caller indicated that it can handle a retry. + * + * Return values: + * 0 - folio is locked. + * non-zero - folio is not locked. + * mmap_lock or per-VMA lock has been released (mmap_read_unlock() or + * vma_end_read()), unless flags had both FAULT_FLAG_ALLOW_RETRY and + * FAULT_FLAG_RETRY_NOWAIT set, in which case the lock is still held. + * + * If neither ALLOW_RETRY nor KILLABLE are set, will always return 0 + * with the folio locked and the mmap_lock/per-VMA lock is left unperturbe= d. + */ +static inline vm_fault_t folio_lock_or_retry(struct folio *folio, + struct vm_fault *vmf) +{ + unsigned int flags =3D vmf->flags; + + might_sleep(); + if (folio_trylock(folio)) + return 0; + + if (fault_flag_allow_retry_first(flags)) { + /* + * CAUTION! In this case, mmap_lock/per-VMA lock is not + * released even though returning VM_FAULT_RETRY. + */ + if (flags & FAULT_FLAG_RETRY_NOWAIT) + return VM_FAULT_RETRY; + + release_fault_lock(vmf); + if (flags & FAULT_FLAG_KILLABLE) + folio_wait_locked_killable(folio); + else + folio_wait_locked(folio); + return VM_FAULT_RETRY; + } + if (flags & FAULT_FLAG_KILLABLE) { + bool ret; + + ret =3D __folio_lock_killable(folio); + if (ret) { + release_fault_lock(vmf); + return VM_FAULT_RETRY; + } + } else { + __folio_lock(folio); + } + + return 0; +} + /* * Restore a potential device exclusive pte to a working pte entry */ --=20 2.39.3 (Apple Git-146) From nobody Tue Jun 16 17:59:27 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B4D6279917; Thu, 30 Apr 2026 04:05:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777521914; cv=none; b=NaGt5xzQrYigSVXQQu/tSGuyi48o9OuxUUwx5L/oYJUebq01OjQplDHtCTMx/vZWkEzA4d1K+6UyA9+UzB4M7NfkLJk4i+t7JMIkVvdwCszgTiGkFxbgfwRD+YuAGQ7ws4keXgpf2QBKqfjM2s8ki7WaE6Vda3oQizvBqdOKytE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777521914; c=relaxed/simple; bh=6MPYWPf1QbIXw7VCTkYtFV9yBQTJYYcQnuchAn2u66s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=sSptMJJ8mYU9cM0fEULW5ESudviOS40A5qCVDjKguKsqPunbI0CUiDTCFonWCRu8oBsJqfG5nhtw8Tr2YD9LVTWpco6051uzMnZPj//dj+fk43+kR0F/GJvHfDDsh9NSQJrm78xN86pCngoOH3gnNR7AsNIAEKcyNj7+71+BIx4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=uJyCN/cP; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="uJyCN/cP" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9B00CC2BCC6; Thu, 30 Apr 2026 04:05:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1777521914; bh=6MPYWPf1QbIXw7VCTkYtFV9yBQTJYYcQnuchAn2u66s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uJyCN/cPD0/zuU5TMG/FWh17eVTVcTCG22wyND+Ln22vdCzvScs+KeOLVG9RmOssZ jDw1d8PhCAGGbMI/IFgx4Z5EBNcmkVO03z+9/CJtVz3K4wxUxjDLHuhXQrygfZY0vj Mz9EPYWMtvXMUkv1GvrXcIXqYy7f2Nbz3f61nlophvJmqpD4ikN6UOlFyXJonZi3kU apK9iwjy49sl7fMM6lNjIBPwaa1bmY1pG/z4xtRWqVUh4kt9XKZhnZLwyhRWE61vDF fxzmTVdxergJndpikZIoTxLjBRYHAUamicyvLwT/yqUnIVLaZSAeOqCic0yYJ36c4O EWCGitvEei8LA== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org, willy@infradead.org Cc: david@kernel.org, ljs@kernel.org, liam@infradead.org, vbabka@kernel.org, rppt@kernel.org, surenb@google.com, mhocko@suse.com, jack@suse.cz, pfalcato@suse.de, wanglian@kylinos.cn, chentao@kylinos.cn, lianux.mm@gmail.com, kunwu.chan@gmail.com, liyangouwen1@oppo.com, chrisl@kernel.org, kasong@tencent.com, shikemeng@huaweicloud.com, nphamcs@gmail.com, bhe@redhat.com, youngjun.park@lge.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, "Barry Song (Xiaomi)" Subject: [PATCH v2 4/5] mm: Don't retry page fault if folio is uptodate during swap-in Date: Thu, 30 Apr 2026 12:04:26 +0800 Message-Id: <20260430040427.4672-5-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260430040427.4672-1-baohua@kernel.org> References: <20260430040427.4672-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" If we are waiting for long I/O to complete, it makes sense to avoid holding locks for too long. However, if the folio is uptodate, we are likely only waiting for a concurrent PTE update to finish. Retrying the entire page fault seems excessive. Signed-off-by: Barry Song (Xiaomi) --- mm/memory.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 0c740ca363cc..a2e4f2d87ec8 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4949,6 +4949,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } =20 swapcache =3D folio; + /* + * If the folio is uptodate, we are likely only waiting for + * another concurrent PTE mapping to complete, which should + * be brief. No need to drop the lock and retry the fault. + */ + if (folio_test_uptodate(folio)) + vmf->flags &=3D ~FAULT_FLAG_ALLOW_RETRY; ret |=3D folio_lock_or_retry(folio, vmf); if (ret & VM_FAULT_RETRY) { if (fault_flag_allow_retry_first(vmf->flags) && --=20 2.39.3 (Apple Git-146) From nobody Tue Jun 16 17:59:27 2026 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 287A319E839; Thu, 30 Apr 2026 04:05:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777521920; cv=none; b=DiXndxxemP/308EwKpkY4C2jN6TfqdD6AYjCc/IE/eATtwdhFqUKASbpGDgdUROq66V5+zIV+aJuVBMasf9xyoPqCTXUORqapWoXn0kCDYOH46zsPdU4ZZkjk4y6EPIBxJQQpJBSux8d2thgX9pNte6neK7ysVfPEsuDxSvxuRE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777521920; c=relaxed/simple; bh=pbXcAk16+rZv3mixELQ0Ipha1GpoRaVFtegxsWu8Zqk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=g5K+6NjbuMbAAL9PPzmC8U+pksa3pl3diVTqlf5UKoEcVbwJ9hiszi6HDFeQrQNewX84XYtx6h48VlbQ/U4LweKTRY0EpgCiTirc5sNulMCpGbAcFPyMKkT1wLH4e2uK1pzXYnAKd1jcHCCrQssBhLmb8q0iw6iBvBtg9YuDIbU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ocBfK8O4; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ocBfK8O4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DC6E6C2BCB8; Thu, 30 Apr 2026 04:05:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1777521919; bh=pbXcAk16+rZv3mixELQ0Ipha1GpoRaVFtegxsWu8Zqk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ocBfK8O4Ywas1DnfVulkPf8JAygFLLLSXfSSIhFab5aPydgPGyQRMZl4VC0fcDxzh ZSF8ZjBo9bGMKXhwl/M9RckUdNTcuoEDs1sQYC8HxwbacO47214ueIB5PuN5R9qPyl 7/vXQCjK6DhEeLKCFGPA8mBl5pi8Xir7/uhc151Nze4/YpDG2oez+rIhbMYqevbaCB eLjkznrbZrf9uMIIMqF0/apYtZGMqURcRmb3l4AFBitMcYEDgyMeHrj8n9yVB0vf4O no7KHvYOh/5gIxhaiyHe3dpMaQbh59rmzxY6J3CuNafpQ/IRafc5BIkfCVPTmXqOYt bTDK/08BVuABA== From: "Barry Song (Xiaomi)" To: akpm@linux-foundation.org, linux-mm@kvack.org, willy@infradead.org Cc: david@kernel.org, ljs@kernel.org, liam@infradead.org, vbabka@kernel.org, rppt@kernel.org, surenb@google.com, mhocko@suse.com, jack@suse.cz, pfalcato@suse.de, wanglian@kylinos.cn, chentao@kylinos.cn, lianux.mm@gmail.com, kunwu.chan@gmail.com, liyangouwen1@oppo.com, chrisl@kernel.org, kasong@tencent.com, shikemeng@huaweicloud.com, nphamcs@gmail.com, bhe@redhat.com, youngjun.park@lge.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, loongarch@lists.linux.dev, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, "Barry Song (Xiaomi)" Subject: [PATCH v2 5/5] mm/filemap: Avoid retrying page faults on uptodate folios in filemap faults Date: Thu, 30 Apr 2026 12:04:27 +0800 Message-Id: <20260430040427.4672-6-baohua@kernel.org> X-Mailer: git-send-email 2.39.3 (Apple Git-146) In-Reply-To: <20260430040427.4672-1-baohua@kernel.org> References: <20260430040427.4672-1-baohua@kernel.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For uptodate folios, we are not waiting on I/O. We should be able to acquire the folio lock shortly, so there is no need to drop per-vma locks and perform a full PF retry. Signed-off-by: Barry Song (Xiaomi) --- mm/filemap.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/mm/filemap.c b/mm/filemap.c index b532d6cbafc8..0d2f6af5d0fe 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3533,6 +3533,13 @@ vm_fault_t filemap_fault(struct vm_fault *vmf) } } =20 + /* + * If the folio is uptodate, we are likely only waiting for + * another concurrent PTE mapping to complete, which should + * be brief. No need to drop the lock and retry the fault. + */ + if (folio_test_uptodate(folio)) + vmf->flags &=3D ~FAULT_FLAG_ALLOW_RETRY; if (!lock_folio_maybe_drop_mmap(vmf, folio, &fpin)) goto out_retry; =20 --=20 2.39.3 (Apple Git-146)