From nobody Sat Feb 7 19:41:39 2026 Received: from mail-oi1-f180.google.com (mail-oi1-f180.google.com [209.85.167.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 725B117C7D5 for ; Thu, 22 Aug 2024 07:13:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310837; cv=none; b=ZEDwNkFgr6r4pDrrJWobmm1Bs/HnXaVHqOxBG7dMxnK9Epm6ZWSDP4ePs9eOky0z1vjw9CUR3H6RbbAo/+CedKQXPRw/Yuf1ZBHYUEGyJ8ZH+/rMkEmMOXzJi0UeHHZtbFvYZQjUEcl6THdcjNdRKHGHPtK1Ne2PXKNjYFwDNNs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310837; c=relaxed/simple; bh=x09LfGHiLcReHBhPIDfpc7YYhL+zohEqRTd3hoyo6yA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=uvGoDZmi1XlqUsqwCUTYY2KOh2cMjHHUsddw1DbSsY9WkOCmH2HN/a4si0HZNsetRwuXCdOeF8xJviy/KS1MyU2TEujYMN2XZWG2EIegI5+vdKgQOOdV5W/u/PPQxyOIKuJ69Qx0n4E40nQGT0hC5+823cQnsWB0dXxKIHBLgLs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=I3pID1MM; arc=none smtp.client-ip=209.85.167.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="I3pID1MM" Received: by mail-oi1-f180.google.com with SMTP id 5614622812f47-3db1d4dab7fso291815b6e.1 for ; Thu, 22 Aug 2024 00:13:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310834; x=1724915634; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CO0xfYDzsVihapVFXw+qMWwHYfpsvq03XEdZOsUU02k=; b=I3pID1MM3YuM7bQfkQ7gjkTXTaS71xo4ssYD+guF1lQRQsM/4wD+rts50U/2D7SjPq Bf9vzimBJA+E1XnVBnMLPSlIeDJFiyOenr69D8KEbDEuvZnThmKkW4x+vNBYDmox50Yw MvSkKctPXpKK5bG07OXqaqUBoHptDRAKn5UjxUXv6kcbO8S8u/vazqTL66bAt+QhINvb k+PN/HCFFS82oRJ72ij7VqS0nUP2E+7NfHR71BeM7eULFSFwhi2K/Ewz812yt1dj7QVf M8+RT5Fyd7091hOoz60ppaT36Jjudx0QmcU9TCuiFuAaSoFK5MkvmAJNduObiEHL2wy0 gvlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310834; x=1724915634; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CO0xfYDzsVihapVFXw+qMWwHYfpsvq03XEdZOsUU02k=; b=Y5NQE2DLiI0/YeWMJzcd5bHCWgxRhz1vuIs1qFZVt559qAmte/iOdlZXfudZ3a+9v8 zDeDCQSZaq9J2s1Ad6pRhjcJf+ykrlCo7eFlEv2jze0BFTY9mGex8uk8atPmCUfyMqwK ckAM4x5EmvTMzGawtgOq6hk/UG0kC2mLKRzdnitq5lDjruITElm+cM62+gVa0R49Y4yK 8wHmWoQlbKh9ogfdLbrh9CVF+6NR1HoAC9Liq4Tr8th+b6anPAXaTCWxEUAGQJNjDT2V pAXgWnE1UuHYiYn47+c1sp7TbpSzoAfZvh1mKkfLRpqq145C12XoQULGFgH6EKBbejid EsDQ== X-Gm-Message-State: AOJu0YyCpPe4nT51icMSmXRtZ0F5pL2XG7k8YI+DguttdttXVxCVQo9l ESIlEbEVzsd7o1NSsb2ohUsWhWJ/HhxyWGlYiRbPyaJPLO8lmtFSBtp4Q+ktDcA= X-Google-Smtp-Source: AGHT+IHZvtclVw9AATG67WMtfqsJZD4bpWoHNpEdN87kwQejpgQ5CLrsmOGdaNcwygIvYs6P1O1W6A== X-Received: by 2002:a05:6808:148a:b0:3db:15ed:2a24 with SMTP id 5614622812f47-3de1950bb78mr6135912b6e.24.1724310834483; Thu, 22 Aug 2024 00:13:54 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.13.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:13:54 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 01/14] mm: pgtable: introduce pte_offset_map_{ro|rw}_nolock() Date: Thu, 22 Aug 2024 15:13:16 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, the usage of pte_offset_map_nolock() can be divided into the following two cases: 1) After acquiring PTL, only read-only operations are performed on the PTE page. In this case, the RCU lock in pte_offset_map_nolock() will ensure that the PTE page will not be freed, and there is no need to worry about whether the pmd entry is modified. 2) After acquiring PTL, the pte or pmd entries may be modified. At this time, we need to ensure that the pmd entry has not been modified concurrently. To more clearing distinguish between these two cases, this commit introduces two new helper functions to replace pte_offset_map_nolock(). For 1), just rename it to pte_offset_map_ro_nolock(). For 2), in addition to changing the name to pte_offset_map_rw_nolock(), it also outputs the pmdval when successful. This can help the caller recheck *pmd once the PTL is taken. In some cases, that is, either the mmap_lock for write, or pte_same() check on contents, is also enough to ensure that the pmd entry is stable. But in order to prevent the interface from being abused, we choose to pass in a dummy local variable instead of NULL. Subsequent commits will convert pte_offset_map_nolock() into the above two functions one by one, and finally completely delete it. Signed-off-by: Qi Zheng --- Documentation/mm/split_page_table_lock.rst | 7 ++++ include/linux/mm.h | 5 +++ mm/pgtable-generic.c | 43 ++++++++++++++++++++++ 3 files changed, 55 insertions(+) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/= split_page_table_lock.rst index e4f6972eb6c04..08d0e706a32db 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -19,6 +19,13 @@ There are helpers to lock/unlock a table and other acces= sor functions: - pte_offset_map_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_ro_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_rw_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken) and the value of its pmd entry, or returns NULL + if no PTE table; - pte_offset_map() maps PTE, returns pointer to PTE, or returns NULL if no PTE table; - pte_unmap() diff --git a/include/linux/mm.h b/include/linux/mm.h index da29b066495d6..a00cb35ce065f 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2954,6 +2954,11 @@ static inline pte_t *pte_offset_map_lock(struct mm_s= truct *mm, pmd_t *pmd, =20 pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp); =20 #define pte_unmap_unlock(pte, ptl) do { \ spin_unlock(ptl); \ diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index a78a4adf711ac..9a1666574c959 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -317,6 +317,33 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd= _t *pmd, return pte; } =20 +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + pte =3D __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp =3D pte_lockptr(mm, &pmdval); + return pte; +} + +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + BUG_ON(!pmdvalp); + pte =3D __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp =3D pte_lockptr(mm, &pmdval); + *pmdvalp =3D pmdval; + return pte; +} + /* * pte_offset_map_lock(mm, pmd, addr, ptlp), and its internal implementati= on * __pte_offset_map_lock() below, is usually called with the pmd pointer f= or @@ -356,6 +383,22 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd= _t *pmd, * recheck *pmd once the lock is taken; in practice, no callsite needs tha= t - * either the mmap_lock for write, or pte_same() check on contents, is eno= ugh. * + * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like + * pte_offset_map(); but when successful, it also outputs a pointer to the + * spinlock in ptlp - as pte_offset_map_lock() does, but in this case with= out + * locking it. This helps the caller to avoid a later pte_lockptr(mm, *pm= d), + * which might by that time act on a changed *pmd: pte_offset_map_ro_noloc= k() + * provides the correct spinlock pointer for the page table that it return= s. + * For readonly case, the caller does not need to recheck *pmd after the l= ock is + * taken, because the RCU lock will ensure that the PTE page will not be f= reed. + * + * pte_offset_map_rw_nolock(mm, pmd, addr, pmdvalp, ptlp), above, is like + * pte_offset_map_ro_nolock(); but when successful, it also outputs the + * pdmval. For cases where pte or pmd entries may be modified, that is, ma= ywrite + * case, this can help the caller recheck *pmd once the lock is taken. In = some + * cases, that is, either the mmap_lock for write, or pte_same() check on + * contents, is also enough to ensure that the pmd entry is stable. + * * Note that free_pgtables(), used after unmapping detached vmas, or when * exiting the whole mm, does not take page table lock before freeing a pa= ge * table, and may not use RCU at all: "outsiders" like khugepaged should a= void --=20 2.20.1 From nobody Sat Feb 7 19:41:39 2026 Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F29AE17E017 for ; Thu, 22 Aug 2024 07:14:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310842; cv=none; b=b4DaRN5XBe/TqYUU78YD6SYhmfwNzo3EPYvQ7jCdBCNDGOJLJz+v712Xn1C0EeTXYlNL3XUnanTLYBPG3dqNjvOelz9N7SrwDj9G5zeja0s/gYEaBQC4HQvk1Z99inQVUM8jO54DrXdbDOKg+mNzhhqWhsLmApX8fO27eHMFvXE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310842; c=relaxed/simple; bh=teCrZ+KkZo6t4Or+udcvaGbXMI71z8Jup5FtH5y7iEs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=F3u34HMo8a3jnsTkQswrTWtS3qWgLgI7Zb3nC3du5IB+Er15TiZz7ylWxW/TeA9qklSX9BRdxQg/G5dnHNFMTxD/VClnI94b/1yK1tWu6E7MyGedwwc+OxHoU7jaEkQVkIyblhd7OJahrQ0cBvQ3xsmK6Y9rDfD8i1Do26k9JGA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=jgVV2gB5; arc=none smtp.client-ip=209.85.215.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="jgVV2gB5" Received: by mail-pg1-f179.google.com with SMTP id 41be03b00d2f7-7cd830e0711so482772a12.0 for ; Thu, 22 Aug 2024 00:14:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310840; x=1724915640; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=d/m1IXPfk7xvDU+Qn41r4NiqBFnWc/gEWMU/opSilGI=; b=jgVV2gB5HUOcTcQdvv/+80zLDKJW01SWXJSVjMSFhFIu4hIloKYuC4+dSWaY2Vxrgr 6m+UDN8wZ5gdfGZ3Z61GSEE98Qwiho5wfSiVxJp5VgxQfz0bHZVD5N9zNO2wEqMxFs91 aIzh2YTPejzQ02S78aX5SoaE01lqhRJAN6Rg0vX3IYqVXMu7z0PPNgxvcwdEY/T7LTlw 7h6Sv03dUppIvVQQybaxIsWyYafvyATJDPyTlvVOf9afKP1b0wIJFbN19rlafBOasUUF Yo43ZzSHh5UEf+RnMZ+ysu7VbxjnZAZ/Mo+FLdhIVcLMD80cjUjwNyE5eOPsrt5MxpmG oY4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310840; x=1724915640; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=d/m1IXPfk7xvDU+Qn41r4NiqBFnWc/gEWMU/opSilGI=; b=WmjefoIX7Yc4mNsoFkcUzRK5666b4eJSJjHnDjX/TFGZykdSLl7xkyIO/ncdcIjFpr woJrgwRUnxuLKGuG4x+cbcVkVHhLqN8B04wdJ3amajRg0JGJYnDwQi2x2Rd3muS6N2zY 3nFoH1uPqQ7ZUIMEzePRfPpMMJFUI5uXOQvPKMwBFG32ctxZ2ffPkRy14BYN/sePMwhU ZejPjLQkdyVmy8qIl1rku3+B6ZzQncH9fei4O1YQ8e+G9fVbtjRGOFKhB3RPRILyrSWK VI7+pvBWV60oBdbeUDwB+ZKppYbJX1vjXIeKOoMqqyIMMB4KTnPc/y36OtxQ/sfbS6qE TYQQ== X-Gm-Message-State: AOJu0Yy3pp4DEpxvzfzqbYt/D2PYj3un2M3yT3FPWuxVfHTnNLWVekgn xNCDGtEOmmgTlVHcvP0tXIyo2hpdv0fqqjf8DVTHLaFtVw4ovlUg3OcjkBYj7Go= X-Google-Smtp-Source: AGHT+IEhsa9RMy8FC6eeC7Dc/qzW9nWoiqPGY6DtNfgP/dDTOo2T0oKgHsMPC17UGp2n78kSJG5uNg== X-Received: by 2002:a05:6a20:6a2a:b0:1c4:9c08:5df3 with SMTP id adf61e73a8af0-1cae51cf01cmr3205085637.7.1724310840022; Thu, 22 Aug 2024 00:14:00 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.13.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:13:59 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 02/14] arm: adjust_pte() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:17 +0800 Message-Id: <7915acf5887e7bf0c5cc71ff30ad2fe8447d005d.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In do_adjust_pte(), we may modify the pte entry. At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the PTL held. The corresponding pmd entry may have been modified concurrently. Therefore, in order to ensure the stability if pmd entry, use pte_offset_map_rw_nolock() to replace pte_offset_map_nolock(), and do pmd_same() check after holding the PTL. Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- arch/arm/mm/fault-armv.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c index 831793cd6ff94..de6c7d8a2ddfc 100644 --- a/arch/arm/mm/fault-armv.c +++ b/arch/arm/mm/fault-armv.c @@ -94,6 +94,7 @@ static int adjust_pte(struct vm_area_struct *vma, unsigne= d long address, pud_t *pud; pmd_t *pmd; pte_t *pte; + pmd_t pmdval; int ret; =20 pgd =3D pgd_offset(vma->vm_mm, address); @@ -112,16 +113,22 @@ static int adjust_pte(struct vm_area_struct *vma, uns= igned long address, if (pmd_none_or_clear_bad(pmd)) return 0; =20 +again: /* * This is called while another page table is mapped, so we * must use the nested version. This also means we need to * open-code the spin-locking. */ - pte =3D pte_offset_map_nolock(vma->vm_mm, pmd, address, &ptl); + pte =3D pte_offset_map_rw_nolock(vma->vm_mm, pmd, address, &pmdval, &ptl); if (!pte) return 0; =20 do_pte_lock(ptl); + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + do_pte_unlock(ptl); + pte_unmap(pte); + goto again; + } =20 ret =3D do_adjust_pte(vma, address, pfn, pte); =20 --=20 2.20.1 From nobody Sat Feb 7 19:41:39 2026 Received: from mail-oi1-f179.google.com (mail-oi1-f179.google.com [209.85.167.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D1DDF1836D5 for ; Thu, 22 Aug 2024 07:14:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310848; cv=none; b=tVfs4tvHRI2d/a4AMQaymBRFI4xwn5/YmmOPHDZOVko7qQBjODek1KGJBc/bFvAETA4q4/2FMX5tAVNP5hCyKMr+aqgcMCg6KrVSW0cVvrv5xQSusZxQHeyKINIO7okyfqZTpyIERiLen7FIXGRyW+ihnxcMJrkIMYe3P2QoZ8A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310848; c=relaxed/simple; bh=cK3nAL5Tb7kKM1XsbrxCIrMB0FYSIjXtYlj4OZV8d1o=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=S01R++ZHWbd95amm5twoXBeI5S2xVClfpfEU1uZdFrWILrOir87nh0gCBi2hUpT6Hrh//e+wV27rGMifEnCt5PRhsgNlGwaTqMTqAULYmgGtE5q9kQU0Z4LAWz0Avj7uRqtBqXR3Iye9n0cxsqVOeERu3KumqKUw81ve0FikapQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=JkkonRDO; arc=none smtp.client-ip=209.85.167.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="JkkonRDO" Received: by mail-oi1-f179.google.com with SMTP id 5614622812f47-3db1d4dab7fso291887b6e.1 for ; Thu, 22 Aug 2024 00:14:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310846; x=1724915646; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=O1mpKMHwwZ1CFkcRRT4IxjhKZ5nh5A3TknIoqewquLI=; b=JkkonRDOJRPG1hnBRvVEB17gladcW4Xb2VYD73s050s4okefohY1qXqvul1tQ94/zT 1qjzDhquVlN4M4AW4gd0o9AMHHslxwCNpw2un2OSdaeBzzYq77Cc1jxCV0XHws1rj29/ E2DEON9xo92KdenJtMZ2KWvHvXu/6Qq724YT0f8KZ9KYvBb4q+QxtzAcMSncXoPPfw40 FLl9+UrUgWa7IkJ7XALI/E2QL5LSqq5FFzr7rmbXAXUohe95exeOR0aUwhWneSWI2CSc SKVC+LIYvnmHU6bMqnnNifD3yWEHL0DHQRPd7T6xakneGLwJ9Y3C16KhVAnwiwIKxPMI oTSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310846; x=1724915646; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O1mpKMHwwZ1CFkcRRT4IxjhKZ5nh5A3TknIoqewquLI=; b=uB6/jCqREFnj72zdYtW1HV9ia7Y9dnLlZf5jty2L7wj2p8nv4a5pCjjJ36MNFTi5YS jQRA1CMyOzq0I4u5wfUcJmloQJCNPyPRh01nlUD6GCdgDvRXD+6PQFLjBJRRJaysS8/K livgAMrdH3pLrd43ozAonGbcmwBCdaR2T9sNXw1kBtZnsGnGb6xhDoROMR7qX6QZvU3E LncEy6HVASxbv2/rgvMuovYqXkvzOMApnATJGU2/6ofdUk0TorZcMwDITUzD/JJOxGn8 vPZPdpEm913i2KmhRIcTdmQeyNfwEhKeokJ5hRuVQ8Rytk443PB1lY1dSIxMemYAnJia 0H1g== X-Gm-Message-State: AOJu0YzNPPCTgJ1lhI51YXZd5fSfkgGcbDURwSOJVO3i57y/LA6t3SQJ YfdaezCjj6IvUcRrOnByaWHCDzxIIvoXPPfFzsIghCZ+xf5n8SV+cT5w5bLIqGM= X-Google-Smtp-Source: AGHT+IFDX2PqWoIoI6wCA8pFxmw/BmBf/PDUvtd9hijDBvTgLhn0KCKCIBnod1yVuIa/+/M4c6qbig== X-Received: by 2002:a05:6808:1a08:b0:3da:a16e:1764 with SMTP id 5614622812f47-3de194e0e97mr6243309b6e.4.1724310845744; Thu, 22 Aug 2024 00:14:05 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:05 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 03/14] powerpc: assert_pte_locked() use pte_offset_map_ro_nolock() Date: Thu, 22 Aug 2024 15:13:18 +0800 Message-Id: <89ae94f722996b4ee7ef20d44cd7f5e2f0495c47.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In assert_pte_locked(), we just get the ptl and assert if it was already held, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- arch/powerpc/mm/pgtable.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 7316396e452d8..61df5aed79894 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -398,7 +398,7 @@ void assert_pte_locked(struct mm_struct *mm, unsigned l= ong addr) */ if (pmd_none(*pmd)) return; - pte =3D pte_offset_map_nolock(mm, pmd, addr, &ptl); + pte =3D pte_offset_map_ro_nolock(mm, pmd, addr, &ptl); BUG_ON(!pte); assert_spin_locked(ptl); pte_unmap(pte); --=20 2.20.1 From nobody Sat Feb 7 19:41:39 2026 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D73018453C for ; Thu, 22 Aug 2024 07:14:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310853; cv=none; b=KP3lHigjUUhDo3kCxcygeTlo9mI21JMvLNAPO7MO2TumpuJg6idocW7v60o4jxhITHVgM1S/YXGlTP82yVXmvkkuFa1xNbKn6yXZgqeONjhWgWD7eVWyr0Ua4ZVrvX6IKVVSmpJTPCV0nFADuA4jPj4wCVrU11hDkwhNxezXg4o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310853; c=relaxed/simple; bh=0V1622YwpCIkpKRqBh1KtWCMlB3fK7JnnpQBbFUsUac=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=cH+QHGDlfiRaQubrcQ+RVvKbcLSQz4cw6iBMf397alEbbo3p+ajaVDgLi2swCgJwnrDWAb8vPFM8czAzOsLTQrbaMDrr93FszyNUYrEiLR/QfDU6+f1P72/cXyLF44U2PIZd9CdY6JS+jI63VwrFrCE011JiKRzVCuM4UEHbvkk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=aFpAFeMD; arc=none smtp.client-ip=209.85.210.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="aFpAFeMD" Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-714186ce2f2so440656b3a.0 for ; Thu, 22 Aug 2024 00:14:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310851; x=1724915651; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jnbwyZ8nm7OA7xAj7wrWrabgK1t5zgYyH3DelQTt/t4=; b=aFpAFeMDhecsz/zCeeHYgrRIVIr5hDA3oYXwIXTYMGsugD5Sf7VVB7vV3iuyndPkq9 jdI6qKWAi6Rc+ocw13a7a8Kj0WLKZRVuvlLQxHOEoIh2Tav16omKYBbtfRuSG2L/Tmfb iBMzYxWG5jgg6XZjybsk1S51/+oLJq2AxZSTjgQEB7+njd7FoHqnFheOP0JmG47tkR8T Ygk8eqgTgUgEe3a/G6tQql8o3JAmEMna5Ut/thZmjt0XlUZL5jidKJX02mGZRNN+Zt99 gtqv+QVZ6ybuO2ZKtTzpldBl0q1H0QDQXmcM/Gtp64ZwBeiLU9+kUQODP/+OlqyLML61 OmwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310851; x=1724915651; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jnbwyZ8nm7OA7xAj7wrWrabgK1t5zgYyH3DelQTt/t4=; b=VOeukT758vGb3cRXJhiuhMqjReeZc4+Gzfnfn99qLGNoEpLGjAafvO4k8lGeLN6YLD rQ8VoNcD88Yq5k6EoVM2fB238U0TQe3zCzrkZKGYblyUHU9TYGgato1xwLB8tiAUAebZ OpfJtT0Mhnw6AW+eCYMbMro+qvfiK9+PuArRO5hn6V5TuAJ2tSMdVgr+NlqmiZENYZdQ GDZzbRTCX+yH1/FmJ6AMxK5K/BLQMN95CBl/FcMGsSgXwuOPkLryEGjPS8QlD5QcUVI+ VG13WtNC51xTqiRw9sjppHlSkqCZb/htB0gbX6O3U4knWUlplPLVR8PhrhSqZQ/fNshS Oeyw== X-Gm-Message-State: AOJu0YwEFclybC3YJEUqty/mkytN+k3fK5Dco7dAAQs5agMCTs0TNkcy 5ouDY3yFp2qcS4ckrryGKUEZuk4o5mnmAJdX8crO5/UsJyEsLB+RNxFZla/NLWY= X-Google-Smtp-Source: AGHT+IGXmNRuv+mQEijx6lEwkmYjHk6XirTogWRG+N5Gk1h3xzTz9eMzqUkZv9fzF0cNQyeHFRUhTA== X-Received: by 2002:a05:6a00:c93:b0:70d:3777:da8b with SMTP id d2e1a72fcca58-714235a5404mr6267050b3a.25.1724310851578; Thu, 22 Aug 2024 00:14:11 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:10 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 04/14] mm: filemap: filemap_fault_recheck_pte_none() use pte_offset_map_ro_nolock() Date: Thu, 22 Aug 2024 15:13:19 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In filemap_fault_recheck_pte_none(), we just do pte_none() check, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 0f13126b43b08..c98da9af6b9bd 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3234,8 +3234,8 @@ static vm_fault_t filemap_fault_recheck_pte_none(stru= ct vm_fault *vmf) if (!(vmf->flags & FAULT_FLAG_ORIG_PTE_VALID)) return 0; =20 - ptep =3D pte_offset_map_nolock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + ptep =3D pte_offset_map_ro_nolock(vma->vm_mm, vmf->pmd, vmf->address, + &vmf->ptl); if (unlikely(!ptep)) return VM_FAULT_NOPAGE; =20 --=20 2.20.1 From nobody Sat Feb 7 19:41:39 2026 Received: from mail-oi1-f172.google.com (mail-oi1-f172.google.com [209.85.167.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 03D33165F10 for ; Thu, 22 Aug 2024 07:14:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310859; cv=none; b=Q4phGpmnTJ9WbpQU2ing//m2AZUimrvgBSUYR2rIJ9O/udft8EfdNugpCWO36LUKCQ+Bi1UwcXkiTn4b1axo0tSEQaY0PC4DbSqX8ukZSpzxp1XybYWDEozG9xtBDMiol163L6SPsBlMsYjZEavNlP5evcQyP9rKI7s0ednddHA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310859; c=relaxed/simple; bh=UCHg2qODrZao5tr4OqATG8KswlV5SkxQ77Nurnm1UwA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=CF4k0yPAvLJ7d6hMdPruPmeV0ChUlDDLSXuSRZv14D8lfohIR3hArB7S6iW/7HLYOOaUY8NedLfetw2KXdRI8M3eZPWOE6j2tTQadvYe76kPJrRmozptr+KjmLIXjOmYV0SiGL7R5oXxwW8nz2YPGHvYyWNEaoIVF0E6v4GJZU8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=hxWbjQo0; arc=none smtp.client-ip=209.85.167.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="hxWbjQo0" Received: by mail-oi1-f172.google.com with SMTP id 5614622812f47-3dd25b2c3f2so186188b6e.0 for ; Thu, 22 Aug 2024 00:14:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310857; x=1724915657; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=toohyZrQNwk2kNUrhQFlaPWHYnggLi3YB3M65Yq8QzA=; b=hxWbjQo0YE2V9j1a7iwlLnpHrWGQMTut9q2OYqwMWMlgxX1Mqr33j/9F3E2UOXNEA8 7fzuC90m77MNgBn7tE1YuwQng47pCktAovnNq6ragMZ8dVQoxutj7EaI7k+OQW4XovM+ naQ0WcQDr+zImNXGGza6TixfaG80f2mEfy2WQtlbbGVhbJE6GO9mC/vNPATq8pogkkE8 IW8GbADbqwEknWzXLvDisIcX6XDXoOZf61pGGy7nHej/zk5r4JdI1TC1CRlI/kOr8vMy cBW2WC5/2NUUyKeiD130pOxj1r5Sl+X5pKJQhspEDYq8B6HKI1UsINS1WfID02LjaPKJ 3Vdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310857; x=1724915657; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=toohyZrQNwk2kNUrhQFlaPWHYnggLi3YB3M65Yq8QzA=; b=AHkxkIwoe9oYZ8uqNWHhu1k5eu/GmeUSFnvSjQhniVRBHF2T40p2YuXcp2CUK7odmO tTF7ae/6h/uFVmY1oNqdOyHtSjgBIr1hoUr01FUiYoonB9NkECF9t7DxY1wvnVAoRudU MZ3nHCM/SRKvt927x/xGFH6/l6By/0UsI7CUvGmkhSLrbyzwFzUU/b5sSihsEEgz14hG NDA1bJvMOtFdBdk/ewfYiGwzCgh53S7gXzu9iBUt+f2eCmBJiU6Xt+T6k6ZuISScA9kC BhxlRC4ZiU7FmwKmI37XjethY+Q72fHmtRPnM4o0W4QkbNNPJqHWGL8sxmnzzjip/Zrp SqGg== X-Gm-Message-State: AOJu0YzpmPOj/2UolEmdQkNR/AT3fuSu2fvYJTsp7EH5sL4d+dxjABDi 6iSggZ78SGWhRDZOwzBzOIWjnyw87DdRvbqpB1XPO7Nk4KJYjQN33dmpSJc80MU= X-Google-Smtp-Source: AGHT+IGgjighdWtbrKc2Se12Iutexgq8AwJLXy8Rdygo/b6qt6ZS1aEL6K5S14xFU0XlAJpbImifsA== X-Received: by 2002:a05:6808:179b:b0:3d9:38e2:5392 with SMTP id 5614622812f47-3de195c380emr6158328b6e.36.1724310857179; Thu, 22 Aug 2024 00:14:17 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:16 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 05/14] mm: khugepaged: __collapse_huge_page_swapin() use pte_offset_map_ro_nolock() Date: Thu, 22 Aug 2024 15:13:20 +0800 Message-Id: <2b7dceaa12da273c6decf92e3bf2ebc4f4e85ec5.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In __collapse_huge_page_swapin(), we just use the ptl for pte_same() check in do_swap_page(). In other places, we directly use pte_offset_map_lock(), so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/khugepaged.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 4a83c40d90538..53bfa7f4b7f82 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1011,7 +1011,11 @@ static int __collapse_huge_page_swapin(struct mm_str= uct *mm, }; =20 if (!pte++) { - pte =3D pte_offset_map_nolock(mm, pmd, address, &ptl); + /* + * Here the ptl is only used to check pte_same() in + * do_swap_page(), so readonly version is enough. + */ + pte =3D pte_offset_map_ro_nolock(mm, pmd, address, &ptl); if (!pte) { mmap_read_unlock(mm); result =3D SCAN_PMD_NULL; --=20 2.20.1 From nobody Sat Feb 7 19:41:39 2026 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B8F3E149DFD for ; Thu, 22 Aug 2024 07:14:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310865; cv=none; b=HzJHw5UfSGOBasmfkJH5wDdgWoTKNTq8SEkIoAQtDVsv0oy0+idumCWdUvkTYgfapMI9NE4AQLrCAMJzMgLJdX0fRBiO/wyeD/yTslD29tmOCBqSGYtwkcxLPeBmlPjWhYQEf+Su4vRt2DF1AqnbctpBiu3hxfd1Q5RA7dfE01U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310865; c=relaxed/simple; bh=zGFC12VfR2Lu98Id6HwQmHSrNI/SYJwpzP6u5eIGDC8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=fFwottnYIOPXPQCk/ZwmFewLbvVOFrkDEAX/nSyuvmwUJcZi0WT30CfmsVtKqwfQqCt3Yd7RP0oATBfTjRX9m2BAXJdK5zMcFkXqcARZ1HsaGcUuxvi8xiEmMXXuJQ+gpnrAII5u6jU6uyDmCNYvBinkXfFbc+HJ7xQosJXr+ZA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=MUbnGPUw; arc=none smtp.client-ip=209.85.210.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="MUbnGPUw" Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-7142a93ea9cso394349b3a.3 for ; Thu, 22 Aug 2024 00:14:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310863; x=1724915663; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YTJMrM+GLhYLeZwUuiK6dImcBAbTGjqYt196yQPATUA=; b=MUbnGPUwopBF0lbcogAfE1boCRWCwqj/TWznuIZBqg2NBazuHglqRR2JVgKXVS5ZpY 32w7S9cMkQTvyLfprxeMLC23wItX6kNzz+gxVddOu/6vnkqNcJ5Y9efbZLavwCVMPJsC OcuvTHPMbiiAtatCHdJE0NCVe/8aFQ0C4I12dZxJwTSNJmhVCyGqzVqY60Qft1rYUuwz 10exixL6x/rjS/X4mlyzefKDVDjiE3lJsWA/8oTp5+zVSz/2YGix/zEO8/XEUhBGHva9 yNWP6WBHF79+5c2xVMuYt130tmaHj5dT42D5qXYyfoOteDo0Rf961v3slFJm7NYyxxYp noCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310863; x=1724915663; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YTJMrM+GLhYLeZwUuiK6dImcBAbTGjqYt196yQPATUA=; b=mfstBvW5rnkmOEJAVu6Rwm+FZCJFm4og6BibEHii3QbqnzdT1saaE+W9Hfj5qbmIld jp2F5i+iW8CdI/Qy1BOqHH3isVYJVuxZcHPy+vmis2FacnfIuSvdavG0aSr2sOno59IY UgWhYXvKk2hz1WG2oqi64Pml2nOwFwafNbWjYdM8+7vwPM+xIZuSoZ50yMLfZVYTV/bi bayLfbR4nBQzvf7YzUV9H4afwHXHr4QZuZXJjTp25lYg15RsZh+LGnmBIdFeBXiZWHgM uma6ijfoap/aR1zPWXgwLz6Ns/JBkmKSJ/7cnH0YMnPCoTHqiGDec02OMKxjYAUr+n/F 7hEA== X-Gm-Message-State: AOJu0YxSjOii6sp8wjWHrJg7un6ecur5eHnnW4krc7RRSzYSnfEO/Rsh ahLjZS17oVFugm2OgMgtUvcE8HFDrFwhGeojtHU5e3jrab6xa55gxOTpdkh1R4Y= X-Google-Smtp-Source: AGHT+IEJaUpX6CQIEutPHebqIOZXfyMaKxdLW1UElMR3gb10gY3PgzFG7UEaK+UwdtskMN8KUpfIGw== X-Received: by 2002:a05:6a00:cd4:b0:714:1bce:913a with SMTP id d2e1a72fcca58-7142356e960mr5852557b3a.21.1724310862916; Thu, 22 Aug 2024 00:14:22 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:22 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 06/14] mm: handle_pte_fault() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:21 +0800 Message-Id: <5acabedfae7ded01b075960b4a91f2e15b4d76b5.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In handle_pte_fault(), we may modify the vmf->pte after acquiring the vmf->ptl, so convert it to using pte_offset_map_rw_nolock(). But since we will do the pte_same() check, so there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/memory.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 93c0c25433d02..7b6071a0e21e2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5499,14 +5499,22 @@ static vm_fault_t handle_pte_fault(struct vm_fault = *vmf) vmf->pte =3D NULL; vmf->flags &=3D ~FAULT_FLAG_ORIG_PTE_VALID; } else { + pmd_t dummy_pmdval; + /* * A regular pmd is established and it can't morph into a huge * pmd by anon khugepaged, since that takes mmap_lock in write * mode; but shmem or file collapse to THP could still morph * it into a huge pmd: just retry later if so. + * + * Use the maywrite version to indicate that vmf->pte will be + * modified, but since we will use pte_same() to detect the + * change of the pte entry, there is no need to get pmdval, so + * just pass a dummy variable to it. */ - vmf->pte =3D pte_offset_map_nolock(vmf->vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + vmf->pte =3D pte_offset_map_rw_nolock(vmf->vma->vm_mm, vmf->pmd, + vmf->address, &dummy_pmdval, + &vmf->ptl); if (unlikely(!vmf->pte)) return 0; vmf->orig_pte =3D ptep_get_lockless(vmf->pte); --=20 2.20.1 From nobody Sat Feb 7 19:41:39 2026 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 36E61185E50 for ; Thu, 22 Aug 2024 07:14:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310870; cv=none; b=TUt0tAPjEAMVrWcx1vIthGw1IlSDskoulPcF0tDuoK8DKZtaB/uMhbAK8VHMd6kEwPX01KkDUguYctDXjDYwcRk7STfd9ufA5aT6oc+p+UVYpkODn1OkJLkAUM4XEOKywzA1WckisBtuMusCaaAeZCLbjpEaOx7BPp8tiC3yiK0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310870; c=relaxed/simple; bh=qgClebIsXEeUL9yPTJGCxcrfISQbfQCIihdRM2yOoPk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=d7i6ZF1kRCtKevWxyKu2fkwPYl2IiQzQrpLCS5BAciQcpwDrmJCzC1I2hnpdGkLRCJLrkuQxne5hnQqjI58XYsj4aqBOtcmcbyDDMcKEL9lg9bxDLxYNpQH9Mmi/PY+aaz6UL/PtSH0qsOFp0egdEqt3PTX27kkmQW3LYv/jEQA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=E09U+T4f; arc=none smtp.client-ip=209.85.210.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="E09U+T4f" Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-714302e7285so400158b3a.2 for ; Thu, 22 Aug 2024 00:14:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310868; x=1724915668; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jy8AvBGPXewIN0f06tSdN0J5TZbHLAwXxEIbEfEZm70=; b=E09U+T4fY5aEPg/kz2jMeahCYGQOUuHYZi7H0ofH2tWyKQOsLheAeZYvSlcrrYBBve of7993rzaWvg/x0CTHaN7sMHlwUQSpwYiydp4falUyZDJdyBLhgBxYuK8FeKZlzhbjii haBy3hS0ZeHzGXthkXh+z7B94B4/tzqQNYumlcuEtptAvsIy3m40i7MmupAeAkEqgsXO IAw+EKE0aEzv9zZo3mMDCfM8r0z8C3hiTJ5KEpF+Rbvf1eDgwufih4pKtPyOya04pRkR bekPOiSSEvWyjteotPedSEa/pUVDoksCrC/yNPwod9tOClDXs81EUIpnyYhw9yhVFWKO z9Uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310868; x=1724915668; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jy8AvBGPXewIN0f06tSdN0J5TZbHLAwXxEIbEfEZm70=; b=wj8StzBD70GncDzdpRslekaKRkcLy7cONc5OkJAckNk5pInt9lneVryZnU+XsnZ/Wx dbV6uJJVaS80+cLw11b3nEOGYHWwkGhIimLBmwAHJnRLfYucujV0wdH96Q9X1IrxHzgp 78+0euUrNoO+sCWeskEzjLvLCCQ4r5Qup063WjtrAxUK6rHXx0Qwl4NFtNjQGfXvkMKi DbreuYj9xDQAkdwGLgkkP4XdiGeUD5AVV1PN01C0bEwYUpEjoG+sDjQCLHYXMQ0LSLS4 WR9OfG4f/0k1ao2KxQjMs+657/8VHzCkIRm1T8+kbIoLXZYFN2jA0tF83m57DTOmycMg MbKw== X-Gm-Message-State: AOJu0Yzt/SjDaP9yd+Rl//SxIlyfQlZHEMIETNKiNqu6ESkGsIB80SF0 FtMr/PQ/v1/z9UBQe50vAdXyo6+XN6hkPd1oJ9Z0/hD9/b7YmgerszwzAWq4UD8= X-Google-Smtp-Source: AGHT+IEV/3obR+4TCbDGR6rVI1h9FT7v1/u/+fxVcK1o1ztnph09CHD4OJqmjxMXnPNwi8/vaSiIPQ== X-Received: by 2002:a05:6a21:6b0a:b0:1c0:e728:a99e with SMTP id adf61e73a8af0-1cada078d94mr5247385637.26.1724310868515; Thu, 22 Aug 2024 00:14:28 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:28 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 07/14] mm: khugepaged: collapse_pte_mapped_thp() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:22 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In collapse_pte_mapped_thp(), we may modify the pte and pmd entry after acquring the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the PTL held. So we should get pgt_pmd and do pmd_same() check after the ptl held. For the case where the ptl is released first and then the pml is acquired, the PTE page may have been freed, so we must do pmd_same() check before reacquiring the ptl. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 53bfa7f4b7f82..15d3f7f3c65f2 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1604,7 +1604,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) pml =3D pmd_lock(mm, pmd); =20 - start_pte =3D pte_offset_map_nolock(mm, pmd, haddr, &ptl); + start_pte =3D pte_offset_map_rw_nolock(mm, pmd, haddr, &pgt_pmd, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ goto abort; if (!pml) @@ -1612,6 +1612,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, else if (ptl !=3D pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); =20 + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) + goto abort; + /* step 2: clear page table and adjust rmap */ for (i =3D 0, addr =3D haddr, pte =3D start_pte; i < HPAGE_PMD_NR; i++, addr +=3D PAGE_SIZE, pte++) { @@ -1657,6 +1660,16 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, un= signed long addr, /* step 4: remove empty page table */ if (!pml) { pml =3D pmd_lock(mm, pmd); + /* + * We called pte_unmap() and release the ptl before acquiring + * the pml, which means we left the RCU critical section, so the + * PTE page may have been freed, so we must do pmd_same() check + * before reacquiring the ptl. + */ + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + spin_unlock(pml); + goto pmd_change; + } if (ptl !=3D pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); } @@ -1688,6 +1701,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, pte_unmap_unlock(start_pte, ptl); if (pml && pml !=3D ptl) spin_unlock(pml); +pmd_change: if (notified) mmu_notifier_invalidate_range_end(&range); drop_folio: --=20 2.20.1 From nobody Sat Feb 7 19:41:39 2026 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BC91E186E59 for ; Thu, 22 Aug 2024 07:14:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310876; cv=none; b=jAT1ZznNQRpkEP0wliENfiRlQBdVmaKHqH1M6iSdLhX3PgIaOVuq6WuW5pkl+bwbNAlpmrsmujoSb+bY6RLS/2CkgRf2F5YP9/ZUlc0f8g/TxTQf5Sau4+g51vuCHS67gFP8aeKfRDQPbPBf961mv9AYgiq4Q2kdPN/auUNFZjA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310876; c=relaxed/simple; bh=WdDTWPZUPkSR+kcJuyPx2uR8BaTylSuO9xQOyQmoS7Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=csZGugFPKsi6yBnoSCxzn6AUjum74XJMU4z15BOK8B3kaiVZue70RFptJcsLmeOMejOym//fPtjP3fClfeOk8uxLa9tyP5Oz3I30goOdzePXWs5ih4joTEWHuKLmFZ70abMz5RJ2Cc5rVsGshBwPjSOpEHlulHYGKHIlifWkjOc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=ErxJaY8c; arc=none smtp.client-ip=209.85.210.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="ErxJaY8c" Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-713edc53429so393939b3a.2 for ; Thu, 22 Aug 2024 00:14:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310874; x=1724915674; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lG9D96+xW+VlDYZHTIMi8bcfezh9NEOiBopQjA6w29o=; b=ErxJaY8cMvWIqMENGPWTrpgSx/6nOKfPjChRLppkUwiYLndY80aOsf81E2sES/l5mn nx/CYnT2s+0rilg1MW/zC0GU16FotNvM445/SoN1PqeinqUuyaPWcclR6sajvoszNlDP l741y4q5vi1aOqDYq6JFzKjnqQxcBxnJyn/BX8XNaSv4uhsghzX/7UG4vXPF5ah9aP4v RTYvIO0df6CEPGvb9+67vJuM4JEPMTZMw+e5fwOD1zIlK0Rf/ffD93XFkpCul5yF5G4H WoseaOcPWwtu8ZJNsqaQ18h4Lk6d1KtMK5GkRy49T8NGSmXHT6K4xNcbgL00O+qGXfkT 9wSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310874; x=1724915674; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lG9D96+xW+VlDYZHTIMi8bcfezh9NEOiBopQjA6w29o=; b=cvejM8v1jF22FpgBd2quc+lllFfyndL8xyKeQJABg3apPU6cnhJvDr9JkCSoPpMREn s3Yued6RraTzV+puhcGLCBOVmdg4dRGYDnIGJgwfcdH1B/Lej49yN6Lk+IDTstv0FIFB 6EaHdfylcyJOG+0g7c9vTIh0cAVFv/tNocKYkXc8s3hDVDFfGEDSaFpFwbI4gCMHhQcp wgXcJf165zEw716N5QqpUnj3wJC7sgDgbuqN7ayYruDFiYf6ZytEU3orYTKwJT1/uA+V yayqdCcpf0xPOnT3E++bgG4O19P6+nSMZ2iX9hfad/qhG8odlJ4kwE8Ch0LmoQPlT0Ux dHug== X-Gm-Message-State: AOJu0YwyvgLSQ7zHvglrByENZWYTB6LNbiekCrcYUHEg2NLnKEOPo+Eb dE3gwELPlMQI22b04OMS7quwxwcO6wUFX5NIwYreTOI+7F080dRI0getT9hYfbQ= X-Google-Smtp-Source: AGHT+IE1Azw7tBspdYFUO9+EydXCiNKc3KqrNzafyoF+69WNl4yC4JRDYA6cJOAAfX+V+yHVonAuzQ== X-Received: by 2002:a05:6a20:9d91:b0:1c4:f209:f1ea with SMTP id adf61e73a8af0-1cad80f657emr5853305637.31.1724310873985; Thu, 22 Aug 2024 00:14:33 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:33 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 08/14] mm: copy_pte_range() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:23 +0800 Message-Id: <71100c3867c4cf6f5f429ce9f2db8432066d0e99.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In copy_pte_range(), we may modify the src_pte entry after holding the src_ptl, so convert it to using pte_offset_map_rw_nolock(). But since we already hold the write lock of mmap_lock, there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/memory.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 7b6071a0e21e2..30d98025b2a40 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1083,6 +1083,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct= vm_area_struct *src_vma, struct mm_struct *src_mm =3D src_vma->vm_mm; pte_t *orig_src_pte, *orig_dst_pte; pte_t *src_pte, *dst_pte; + pmd_t dummy_pmdval; pte_t ptent; spinlock_t *src_ptl, *dst_ptl; int progress, max_nr, ret =3D 0; @@ -1108,7 +1109,15 @@ copy_pte_range(struct vm_area_struct *dst_vma, struc= t vm_area_struct *src_vma, ret =3D -ENOMEM; goto out; } - src_pte =3D pte_offset_map_nolock(src_mm, src_pmd, addr, &src_ptl); + + /* + * Use the maywrite version to indicate that dst_pte will be modified, + * but since we already hold the write lock of mmap_lock, there is no + * need to get pmdval to do pmd_same() check, just pass a dummy variable + * to it. + */ + src_pte =3D pte_offset_map_rw_nolock(src_mm, src_pmd, addr, &dummy_pmdval, + &src_ptl); if (!src_pte) { pte_unmap_unlock(dst_pte, dst_ptl); /* ret =3D=3D 0 */ --=20 2.20.1 From nobody Sat Feb 7 19:41:39 2026 Received: from mail-oi1-f181.google.com (mail-oi1-f181.google.com [209.85.167.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8C3A6187877 for ; Thu, 22 Aug 2024 07:14:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310882; cv=none; b=Jmcu2YxaMMvMKQhtOj8vsEJcV1CKhFYx+dMr2bdumCqLVi5+xyt02HedvhUr8R3snrVW/KnZPa1Bj2dqRhrVoHF5CHQWs21NV+eqVGjOZmGjwBFucfJhA8I9mY9qPtskGddG8tOBGe/0ZByIxV9erini48PCYTtvGksG7Y26aUU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310882; c=relaxed/simple; bh=g7vfX863SYbCR0NBqllFlwtVsFEJGXyQFbet06hRe4k=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=i+bDg40Yh+BYvlbo+xKeTs7OcDrAZPAKHj/AFS1taV2QOk3hm/yR5iW4gp2YNKfJzXbzgnTkHHa5+CrOIFgw8wpb7W1xy0bwj4Gw8nanNXa5NyUXGaUiwifc+88nefcgv8hCyw5ZP0VHUBxuj/mBdk9H4WAAxQy22gB6Y9MEtEo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Gay50H9m; arc=none smtp.client-ip=209.85.167.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Gay50H9m" Received: by mail-oi1-f181.google.com with SMTP id 5614622812f47-3db1eb76702so323966b6e.0 for ; Thu, 22 Aug 2024 00:14:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310879; x=1724915679; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kCwbwV/YmKb9rcRCZPyi2vEyGZwxl6QKnoBllHH347k=; b=Gay50H9mT8GjjGyVsbWP8oRTPvazyrfrkHQNLggD4tWJAvcUx8EpIEHbIZi+lsqMly uS2nxE0KEDluiuTMnycVI8DdwCcdSRT9UgKN/kr7lSOMldmvFlFMziR82qNIl6Z+3mV8 7jEkE/PG8OPuv9bmcmDvGsufFxPGO6R5WYUfLUglLyKPOpb8+X97KW+GiP5p9RYBqYJK ipWWXyxJuRLvGuMMb2V2YY3xOH48bCSVnCIsistevtzzCYpfspf2ZW5LLy6conJu8dMU k8p7SHIMbZpiYhYMiJ8QQtUPU70/Uyaf/Ui+qRkb3GPbkYLDfbndVGfYBBf1l/CgbMPV yaQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310879; x=1724915679; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kCwbwV/YmKb9rcRCZPyi2vEyGZwxl6QKnoBllHH347k=; b=AM1loo33rEDHEebWwQXrChFx+PB9NaCHhME1rL8Ni9sxBWgMJ2IiUK6Yc3TD61pSd4 r0ExUvBxUOVY+NngBzKq/iy8eZ04ILnnU4FKg6OIjLk+LOetgYjcPAbVlaZ2XDl6BkGm /gNx0n6GxYTKOFMrU/cL09QKv8scPUHkneRHR6v9RCDfifKzc0u+53toISGJ0bSjOO1L yH9c/TMoYGxy+AMG24kLCNhLEre+HOFKjwGC3C54D5mGvViZxl/zmO2APrbpxoTMRb0X ugvq47W1yIUAEofkdY2na4TZvvSHXbyWSdqQ/5eUHTKU7pOvJvkDDTi2kn4ryC49/CFR HJaQ== X-Gm-Message-State: AOJu0Ywc/IJbda/u09DPgzPQ9/A01Esan0zih1bVHhKaTT0M2knBy5Cy NDiZdQ2yhPjljCD2SA9TEa0AWSsYNs24aTGub3qw4xxLKiQuyhXssZTbfWkYPfL2ZUEwI/tWa4i FQuk= X-Google-Smtp-Source: AGHT+IFsTn+Q5A1WOhbyPTAjHclYiOvt3nhLZV/Hi5KOu9NOtbxQa/3JykhMvHE3PrFxCVBhO2k7lw== X-Received: by 2002:a05:6871:149:b0:268:afc3:648e with SMTP id 586e51a60fabf-273cffcb557mr1073020fac.42.1724310879532; Thu, 22 Aug 2024 00:14:39 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:39 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 09/14] mm: mremap: move_ptes() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:24 +0800 Message-Id: <12097a187fee5f21860d4f87e7aa103f822e298e.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In move_ptes(), we may modify the new_pte after acquiring the new_ptl, so convert it to using pte_offset_map_rw_nolock(). But since we already hold the exclusive mmap_lock, there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng --- mm/mremap.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/mremap.c b/mm/mremap.c index 24712f8dbb6b5..f96b025c09079 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -143,6 +143,7 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t = *old_pmd, spinlock_t *old_ptl, *new_ptl; bool force_flush =3D false; unsigned long len =3D old_end - old_addr; + pmd_t dummy_pmdval; int err =3D 0; =20 /* @@ -175,7 +176,13 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t= *old_pmd, err =3D -EAGAIN; goto out; } - new_pte =3D pte_offset_map_nolock(mm, new_pmd, new_addr, &new_ptl); + /* + * Use the maywrite version to indicate that new_pte will be modified, + * but since we hold the exclusive mmap_lock, there is no need to + * recheck pmd_same() after acquiring the new_ptl. + */ + new_pte =3D pte_offset_map_rw_nolock(mm, new_pmd, new_addr, &dummy_pmdval, + &new_ptl); if (!new_pte) { pte_unmap_unlock(old_pte, old_ptl); err =3D -EAGAIN; --=20 2.20.1 From nobody Sat Feb 7 19:41:39 2026 Received: from mail-oi1-f178.google.com (mail-oi1-f178.google.com [209.85.167.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C1D11885A2 for ; Thu, 22 Aug 2024 07:14:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310887; cv=none; b=pkdhGgvLTOJoxRL/h9C67sOVuVVtWPucaOoNcz+jNgIacJBQc3069IBbUvz5PyesBclkHPRkbuieQsxQzx467KG1Bsxy9Ih3ZWA6J81Qj86BWHxg0dB6fKpVvw5UjazK26M0G3lMgpfy17gLfAOySjiSbzEqyY+NU5idtxAk8Ps= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310887; c=relaxed/simple; bh=NDVt8UFk+zbfrRMfMiJ0eEKdBEzxM75Y5QRnhcSFNmY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=c4nrUdLEq1WxLgbdwLDAv3RSh6mgp74xIvo1T9D+ScuEC0iXLFy0pprX2dhQHNno+9MUN1+Q98xhYJA59OTOa7BnazwcUQAJJGb4vQTRWnxC+YZBwKU2zjka+/dyWcOZaOHqcAMKLSFvnpdnYb1fSVyDRMXc10GBO8Al0258ESE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=BSUNtw57; arc=none smtp.client-ip=209.85.167.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="BSUNtw57" Received: by mail-oi1-f178.google.com with SMTP id 5614622812f47-3db2315d7ceso298595b6e.1 for ; Thu, 22 Aug 2024 00:14:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310885; x=1724915685; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Crbz/R3rJcbpbcovfp9fh4ozsAEMZSaAxrx7zvZfRRw=; b=BSUNtw576jvEjIJoJ9xgAAwGsuSieh8ylg8a5IgnAubAVyCc2jyMnkLhSE+C4uBAow TegzwyVMOSAWIwytdourj9Qu8i4ywKkofc+PPJKX6G89ErjdsjXwlxP1UIO/oBQc0+30 WaOGPr4B9MkyehQJ0A2Zf82WELGkIjjL2rGNQ3VPTa7nTzNdQa9D53poRsiQKn7eEk3R np3iOayYoPW6CJ4KUz2R6EkqQtHCuhn22jIvS9bUt2lALzUxHFu+Kp/HU2o1Wm9VcDE0 Ytx7+jQLbq7T9SfpIvOUHi296FCCM+vsTPdolPwZVJIKAqezQkSF+mDFYHqt4jCpyieL KdiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310885; x=1724915685; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Crbz/R3rJcbpbcovfp9fh4ozsAEMZSaAxrx7zvZfRRw=; b=T84h6pXSkv1gNHTxMT87VtbYkkS3R+DixNp60tDLYZ0NPsOiOfmdtx1j9JVMGCrQs5 2Wm9WQkfFD3Cx5w+fi2gPfFLqf8230YXVVlE9+gebxo8bjbpzC2cWZ0exPF7YXW/aCA2 DYL6c9ZCDWInYQxqjWpuJkHzsp2iX5lmny4ju+qop8P3cm+vg8ByuLsb/P9jiE5BjM2B j5aWb61Cwlar4+5ORuhWOqq1R7Jw8Tl3sd6Gvl9RejTrddl/5XZGZ+20m8min68LjeDe sg02Rz4NyQ/JCShB9fVo1rgIt8SW3Rbfrmog8WLZbtjcf94OK0T/FjVIYkWnJRNNzN8D he2A== X-Gm-Message-State: AOJu0YxNMWaMhkBfSB595SF6XEiWF0uVBncwIN4gR3edo8Spx699bPAE MexhO3t9B3YgGphxY51LFbSzGDOI0HW4MOMQbvijahyKe+aVSbB/kGtaNf5V0vo= X-Google-Smtp-Source: AGHT+IF73IAzwqsx0nomJCVJryAV7cndKc0EgZ0+2TYoDy25bfQlq+9JSwXFUBQmZZcxXSkt111lxw== X-Received: by 2002:a05:6808:d46:b0:3d9:29c1:be5c with SMTP id 5614622812f47-3de194fbf2bmr5329807b6e.12.1724310885053; Thu, 22 Aug 2024 00:14:45 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:44 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 10/14] mm: page_vma_mapped_walk: map_pte() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:25 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In the caller of map_pte(), we may modify the pvmw->pte after acquiring the pvmw->ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the pvmw->ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pvmw->pmd. Signed-off-by: Qi Zheng --- mm/page_vma_mapped.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index ae5cc42aa2087..f1d73fd448708 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -13,9 +13,11 @@ static inline bool not_found(struct page_vma_mapped_walk= *pvmw) return false; } =20 -static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) +static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp, + spinlock_t **ptlp) { pte_t ptent; + pmd_t pmdval; =20 if (pvmw->flags & PVMW_SYNC) { /* Use the stricter lookup */ @@ -25,6 +27,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, sp= inlock_t **ptlp) return !!pvmw->pte; } =20 +again: /* * It is important to return the ptl corresponding to pte, * in case *pvmw->pmd changes underneath us; so we need to @@ -32,10 +35,11 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, = spinlock_t **ptlp) * proceeds to loop over next ptes, and finds a match later. * Though, in most cases, page lock already protects this. */ - pvmw->pte =3D pte_offset_map_nolock(pvmw->vma->vm_mm, pvmw->pmd, - pvmw->address, ptlp); + pvmw->pte =3D pte_offset_map_rw_nolock(pvmw->vma->vm_mm, pvmw->pmd, + pvmw->address, &pmdval, ptlp); if (!pvmw->pte) return false; + *pmdvalp =3D pmdval; =20 ptent =3D ptep_get(pvmw->pte); =20 @@ -69,6 +73,12 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, s= pinlock_t **ptlp) } pvmw->ptl =3D *ptlp; spin_lock(pvmw->ptl); + + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pvmw->pmd)))) { + spin_unlock(pvmw->ptl); + goto again; + } + return true; } =20 @@ -278,7 +288,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *= pvmw) step_forward(pvmw, PMD_SIZE); continue; } - if (!map_pte(pvmw, &ptl)) { + if (!map_pte(pvmw, &pmde, &ptl)) { if (!pvmw->pte) goto restart; goto next_pte; @@ -307,6 +317,12 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk = *pvmw) if (!pvmw->ptl) { pvmw->ptl =3D ptl; spin_lock(pvmw->ptl); + if (unlikely(!pmd_same(pmde, pmdp_get_lockless(pvmw->pmd)))) { + pte_unmap_unlock(pvmw->pte, pvmw->ptl); + pvmw->ptl =3D NULL; + pvmw->pte =3D NULL; + goto restart; + } } goto this_pte; } while (pvmw->address < end); --=20 2.20.1 From nobody Sat Feb 7 19:41:39 2026 Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B8DA71885AC for ; Thu, 22 Aug 2024 07:14:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310893; cv=none; b=tSU7yakpW8CRVZLY9QHYjqfhHfiex2w+8KZiY2LsxqyHk0pQojwCdVk6XeizA4bb4almBRVn5STc7g8eVgzYCrHOEBjpfJKpBYCSqC1xBNngos+KkA0xjGE7nlVa7k1E7ilUzMM8+SKZSuUtbgydtjDT76/V+5jiFuzp4v9WXLY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310893; c=relaxed/simple; bh=iOolcBEuZVXthEhfPky/D4p9xkDBHkn1HC+6Bh1rqVI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ntCMz+EIYj7gZoEsxpBlPxE5kxBNv7pBBHD92FSssNmX+lS7RVJzxrno2XELp8bCmCtSx9yl96+OGZKKoL3OYkgXosd3gSDpCO1lCPrMXa+zhhqLZOBdKsYEbIYL8Gbs4LPWHA6yNNXGtswZchFMK3a4uzPHJ/MQF03kx0EVBIg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=IkyPdiXK; arc=none smtp.client-ip=209.85.167.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="IkyPdiXK" Received: by mail-oi1-f177.google.com with SMTP id 5614622812f47-3db14930a1aso293535b6e.3 for ; Thu, 22 Aug 2024 00:14:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310891; x=1724915691; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7wvuIpVK8M95P7HYfgMO/FZeMoPfoEIc6BtRbN48KOY=; b=IkyPdiXKlhn4kLb2IcnRxmAgjXJ3fkw2EzfdSRp3+e6IeGbEYguSlmi2QS9viGo4S5 r1ZPd9gF0Dy5xYOhGrI4C2O/ySXiwS7cuJRRRV1Pla7cFJZn5Oomj1WmtDaujRGuaJ5P DsXyvEAhlfEDOU7eXApBiK2oEi5WM3xrS9S5y/H2dtAZgazrzRnBiShE+iQyjFt2BvQP kTmAOZitgJPAhz15r//1c/z7woJRA6ZMJeqT3CWo9/ackcfHAeBqYH2eAlcF6YqkFSZi 3sTDYBISHAomxfBGS+c0kDiPBgHoj2tr9Ze/pSw6Blg3bEsFflT6lWXF0J7dBXPclxZ3 0FHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310891; x=1724915691; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7wvuIpVK8M95P7HYfgMO/FZeMoPfoEIc6BtRbN48KOY=; b=N5x4ywR2O58vEvKzZKvWcBPymDvmgOfkWgrWWvZKJ9Br2vQZmwcvqR+Mm2Z+7J2JW9 2aIID3kH7FCGAsc/M5ekb2nAPNKLJa+1tE28T99TG6Y0d44s7QFQSYCmc9TF13+TwcE+ md83wz6bq9ZfOwiQ/dCYejXyVesdM6LRth3Nn5CNzhJIXShvuLzlxQDaZUXuJ0nX+XOs v5yUW559HYVEiPdiikT9Cc9pOnVtVWfuSmE+BtHkoK5lwtSldXzw74Lh1WAViLCl5JIJ iPhUk4dacTA5lpykq7PXIU8clulSRu4f6gIya3r2HNGnN/84DKWiVHGpDG3BS4bcWlKN 8VHQ== X-Gm-Message-State: AOJu0YxRF2yGkUXgkocozNSAU3iLeISPIzdqbgm0HnWbwJsUXTn9XGwk xXIklsP0pPS7lcdDUnB0LkFm+b0gOcR/B4Zc1kHZOcMeJrTOmaqJcYHDLGjMy+o= X-Google-Smtp-Source: AGHT+IH3dTXEnimLAtKAfnTveP36n8fQueYjLUMvugh9RpWMjyLI+ldrjyQ+VF2Ub3qToXnGQhbMbg== X-Received: by 2002:a05:6808:3847:b0:3d6:2b12:7dc0 with SMTP id 5614622812f47-3de22ff6ae0mr1325714b6e.20.1724310890896; Thu, 22 Aug 2024 00:14:50 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:50 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 11/14] mm: userfaultfd: move_pages_pte() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:26 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In move_pages_pte(), we may modify the dst_pte and src_pte after acquiring the ptl, so convert it to using pte_offset_map_rw_nolock(). But since we already do the pte_same() check, there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng --- mm/userfaultfd.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 6ef42d9cd482e..414ee744257b7 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -1135,7 +1135,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t= *dst_pmd, pmd_t *src_pmd, spinlock_t *src_ptl, *dst_ptl; pte_t *src_pte =3D NULL; pte_t *dst_pte =3D NULL; - + pmd_t dummy_pmdval; struct folio *src_folio =3D NULL; struct anon_vma *src_anon_vma =3D NULL; struct mmu_notifier_range range; @@ -1146,7 +1146,14 @@ static int move_pages_pte(struct mm_struct *mm, pmd_= t *dst_pmd, pmd_t *src_pmd, src_addr, src_addr + PAGE_SIZE); mmu_notifier_invalidate_range_start(&range); retry: - dst_pte =3D pte_offset_map_nolock(mm, dst_pmd, dst_addr, &dst_ptl); + /* + * Use the maywrite version to indicate that dst_pte will be modified, + * but since we will use pte_same() to detect the change of the pte + * entry, there is no need to get pmdval, so just pass a dummy variable + * to it. + */ + dst_pte =3D pte_offset_map_rw_nolock(mm, dst_pmd, dst_addr, &dummy_pmdval, + &dst_ptl); =20 /* Retry if a huge pmd materialized from under us */ if (unlikely(!dst_pte)) { @@ -1154,7 +1161,9 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t= *dst_pmd, pmd_t *src_pmd, goto out; } =20 - src_pte =3D pte_offset_map_nolock(mm, src_pmd, src_addr, &src_ptl); + /* same as dst_pte */ + src_pte =3D pte_offset_map_rw_nolock(mm, src_pmd, src_addr, &dummy_pmdval, + &src_ptl); =20 /* * We held the mmap_lock for reading so MADV_DONTNEED --=20 2.20.1 From nobody Sat Feb 7 19:41:39 2026 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B3FE2170A0F for ; Thu, 22 Aug 2024 07:14:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310901; cv=none; b=C8Z8Iq8BWQcNnDTwgDy5aXgWjdhH3Blz4WSINjOmnv6r0SE4WcOMeNx/1jlL7Reff2FoBSUeNVZAIaC3FEiqb3PO2tenRYYE3vFSJYXDrBRG5QIyOclCam8kl25Qz58yuHhliPmWv3ZBFBNaeg8RJ6Q3xesN0CyA29Sahb1CpoY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310901; c=relaxed/simple; bh=CEF/qocup8yjgXxjODGjGheQI4te/SVpQzyunVQ4H4s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nR9SHBHEIMlGEKsAOYa8mFhPm16/BCJDHfeve7/9ja9OR+GqbX0/DQjsB8aJA5ph6XReRFUJGK1IjIZm1y7zenz8YvLAvG1BvbhJnMnHSe8n3kbdCwamAnqbDZrfiLlzm1/h/mSoOmzjdQcs7puXx6NoTms/fo02rm+cTaZ2zUo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=IHvPHmCH; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="IHvPHmCH" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-20202df1c2fso10661905ad.1 for ; Thu, 22 Aug 2024 00:14:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310897; x=1724915697; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TKuz+eT/BjM4oBQXjKjArJ5u0hGAcRe4llal02NX1d4=; b=IHvPHmCHofjvo1cTkHEKx6d+vtxEQdhRAsvXkDXSx7fEJWRE35xHshqnndWDXLB9BP hawJiw1ragErqss2EM1P94ZJqo+ZsnSgE2b+9Ym1eWDft0+YdegE+oeDbYi58TzMK0cl PPv00Izn4u+9mRNyEmB6MzwJxUV8dGUIu6pw4vXTB5QDRrGS3n1yKkSOmBz5la9mcrxG 3NIgXnC4n9PwmPVrVNuTCoyj2RNUOcj0dwtYl7+3a6wtXdd2I1qrWtnpe8uV+2/eJtiW PIPyl7JrSjL/rDiWgTX9flwCN7ZOqjXmrnTi4JKBzhrvybSf9PImHwmdiHAJ3TNCPyHO /g+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310897; x=1724915697; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TKuz+eT/BjM4oBQXjKjArJ5u0hGAcRe4llal02NX1d4=; b=A0wPNAQnJz8IfXZGudiwJiJ63pRtMqZ7A9WhYPaSYcg38RID8RhaKB3t9aTD/Hpo34 b14Dswdz8UVIHv8rWeacEFNREFQOKpqpJV6JqXuHcE5OKJMTfxqE6J3a78AZqNo80tMV 7I2+b0kf2CWFXpVWf/FclUYFLsV6/5jL87HDYV5tS+Sjk6j8vxpERmddAU3yRI4tmWDv OsZkTk++SvQ6ritlZa87IzFhRAT9my1bqXI1hWoSou75p9Fbbur2A58V0u/Vw8b5iQpP lc+geDfvcoDTsUuXmTunlP097lAPkYphQZTCtQ7qy3QIji5/NBRfrO87f59RWEcaA4nc Ki6w== X-Gm-Message-State: AOJu0Yymlum2v4NthbIL4X5c70J7Plol2qbgsRMxNT5PMXC+qEgvtagp MZidHIjni3UeLGDIgJ8GZqYEsCodwWlwAlhGcO1pxUvi60PX/tsbqnEHo9vTo+g= X-Google-Smtp-Source: AGHT+IF6H0tDfgQqKcPaVFXqnNnvtvkdaP6bepcTCK17Vph5rD33YztktUCyS9gQ0weOULsun7jJQw== X-Received: by 2002:a17:902:e88d:b0:1ff:43a8:22f2 with SMTP id d9443c01a7336-2037fe188b8mr31159405ad.24.1724310896888; Thu, 22 Aug 2024 00:14:56 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:14:56 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 12/14] mm: multi-gen LRU: walk_pte_range() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:27 +0800 Message-Id: <0e2980e9a31d3503a5dfb350c04f4ede4d05220e.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In walk_pte_range(), we may modify the pte entry after holding the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pmd entry. Signed-off-by: Qi Zheng --- mm/vmscan.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 5dc96a843466a..89ef1ac8eb1a6 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3396,8 +3396,10 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long= start, unsigned long end, struct pglist_data *pgdat =3D lruvec_pgdat(walk->lruvec); DEFINE_MAX_SEQ(walk->lruvec); int old_gen, new_gen =3D lru_gen_from_seq(max_seq); + pmd_t pmdval; =20 - pte =3D pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK, &ptl); + pte =3D pte_offset_map_rw_nolock(args->mm, pmd, start & PMD_MASK, &pmdval, + &ptl); if (!pte) return false; if (!spin_trylock(ptl)) { @@ -3405,6 +3407,11 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long= start, unsigned long end, return false; } =20 + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + return false; + } + arch_enter_lazy_mmu_mode(); restart: for (i =3D pte_index(start), addr =3D start; addr !=3D end; i++, addr += =3D PAGE_SIZE) { --=20 2.20.1 From nobody Sat Feb 7 19:41:39 2026 Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7FC6818892B for ; Thu, 22 Aug 2024 07:15:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310904; cv=none; b=KqaVuzyUhOFWpHk5OjtSosRLl1JZ91O9NttX5cJPFazRtMK3pUlpPZlWwkPjlgIojLGko81QxvhnOVtkyQmswiHMvUvzQ8mbr8i7w3Gk+qYCQQOGwgraQhu+E1fyLpiWoI+j9q4ZCgRGl8Ei+QfkbOuQyqz5tXwGevJevrWLfjs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310904; c=relaxed/simple; bh=guFP/UvtdmTvqla5F80lTymUue3B5FcUPfzU+3Ti5Ko=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=BoI1rIjykyBJ7sj726kaQiCnOxpbK+TdX2Id9kVsaUBd0dxxHRr/WZ9WR4VaLPfSFTZssSeE+zMigpjEOd1N3gkn1Ma+nj4iPQRzxl/AQZOsmtcPAvQcgLBk3utiuz1aWfKWGiRFssaNRYXCqPBxLdHc7NBHdSWpLD8xBwDpEYA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=J+kCUQRc; arc=none smtp.client-ip=209.85.210.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="J+kCUQRc" Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-71430e7eaf8so392202b3a.1 for ; Thu, 22 Aug 2024 00:15:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310903; x=1724915703; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gFdM7jnff0z8zF8oYknH78hRu/cQudWIIbslP9zGsAs=; b=J+kCUQRcx8ze1ZyKU3x9SsBB0Pkwsob02rFo3T021VU0R3hkW9rsenIH3hD1rlKq+a FtI+84imgQpBOTbGmmXfXyWmYGzAeOBBV/VFtUNBqJuvsdeBlTR8QwItzgdskxOg7VIR aA9/VybgStagTTqIUSG1uHOuo/cSeJEcjCDu5Dl79K/077WPogpTb0x5xQ7I+Gj0BzXQ CzZGHNdK71GtMnwJiTx+Hr5Mt5tD1k8KMHdBjc6Sb3BbruqswpajWs9QcC77PySpGx9Y y4v9Z/t8zOsmv9Ykp2glMqEbPfn7flj/UlDn2Pvn5bF7FuC9axvwKNuXPdM/vUKghCLo k5RQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310903; x=1724915703; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gFdM7jnff0z8zF8oYknH78hRu/cQudWIIbslP9zGsAs=; b=OIqxtPhNrV6NyQhf87FcVw0tSyIaxUcz5/caUUVcxxDDJ/D2V64QFLMkmVRGiEQ5fM dC9kV5qQ3RU6SWu9xoKAHSFiVGHf4vp1KIT2zU9HskyAUaRl9GJnBVLfqz43Ak2xols+ E0PdjiI1nzs68KG9nXoJhfQMpDrEgpztM0TSC6CPVvbjU7U2K5hX5hmtgq2rtTEHGz2S cg9S3aiXABj28K7TCDZ9rA2bem0slcRQGxsA9aPFLR1n4GFjntFDuEzxTLyejeoEjPOs NjdaQm/WOcnD8tiBtCvsOLkN5yiY8/rFdl4xOdRp3eXPL0+xgTt3yiRqPdN5zuGImbVC ZS0g== X-Gm-Message-State: AOJu0YyHGKrfDAyxuy3SXp9AxAga/zY7PGXRSewIjWWbamiR1LRbeYYw AGcaCUQpJMSAd025JWMxo30sTAHROpkUjvel4lGvfTIIU0KuHtZbthUFu/pwoZ1TS+ZPoawTalX pDXA= X-Google-Smtp-Source: AGHT+IElSFOoF4KMPSsF0Gy5tbhJtfEKgqkCJ+pjJm8bK93AAf5vkcbpbvWiZRIPotD+TEYG0fABhg== X-Received: by 2002:a05:6a00:b53:b0:70d:2fb5:f996 with SMTP id d2e1a72fcca58-714234922a8mr5431869b3a.11.1724310902611; Thu, 22 Aug 2024 00:15:02 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.14.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:15:02 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 13/14] mm: pgtable: remove pte_offset_map_nolock() Date: Thu, 22 Aug 2024 15:13:28 +0800 Message-Id: <061a61c5871948680488ad02fd1e03547948a500.1724310149.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now no users are using the pte_offset_map_nolock(), remove it. Signed-off-by: Qi Zheng --- Documentation/mm/split_page_table_lock.rst | 3 --- include/linux/mm.h | 2 -- mm/pgtable-generic.c | 21 --------------------- 3 files changed, 26 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/= split_page_table_lock.rst index 08d0e706a32db..581446d4a4eba 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -16,9 +16,6 @@ There are helpers to lock/unlock a table and other access= or functions: - pte_offset_map_lock() maps PTE and takes PTE table lock, returns pointer to PTE with pointer to its PTE table lock, or returns NULL if no PTE table; - - pte_offset_map_nolock() - maps PTE, returns pointer to PTE with pointer to its PTE table - lock (not taken), or returns NULL if no PTE table; - pte_offset_map_ro_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; diff --git a/include/linux/mm.h b/include/linux/mm.h index a00cb35ce065f..1d3e8882f6ba1 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2952,8 +2952,6 @@ static inline pte_t *pte_offset_map_lock(struct mm_st= ruct *mm, pmd_t *pmd, return pte; } =20 -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index 9a1666574c959..3b6dd8bb5b134 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -305,18 +305,6 @@ pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr= , pmd_t *pmdvalp) return NULL; } =20 -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp) -{ - pmd_t pmdval; - pte_t *pte; - - pte =3D __pte_offset_map(pmd, addr, &pmdval); - if (likely(pte)) - *ptlp =3D pte_lockptr(mm, &pmdval); - return pte; -} - pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp) { @@ -374,15 +362,6 @@ pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, = pmd_t *pmd, * and disconnected table. Until pte_unmap(pte) unmaps and rcu_read_unloc= k()s * afterwards. * - * pte_offset_map_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_m= ap(); - * but when successful, it also outputs a pointer to the spinlock in ptlp = - as - * pte_offset_map_lock() does, but in this case without locking it. This = helps - * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that = time - * act on a changed *pmd: pte_offset_map_nolock() provides the correct spi= nlock - * pointer for the page table that it returns. In principle, the caller s= hould - * recheck *pmd once the lock is taken; in practice, no callsite needs tha= t - - * either the mmap_lock for write, or pte_same() check on contents, is eno= ugh. - * * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like * pte_offset_map(); but when successful, it also outputs a pointer to the * spinlock in ptlp - as pte_offset_map_lock() does, but in this case with= out --=20 2.20.1 From nobody Sat Feb 7 19:41:39 2026 Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F133C188A06 for ; Thu, 22 Aug 2024 07:15:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310910; cv=none; b=glRXcNpGcOPpQ+FdAufovJz41FonDI8Th+Xp5Qb8xBgBBZZRJaTDgh1HLTHuTI9AtNFS2GWkvLmZ0AX8plxQATtwJpUJ4qUoR7F5RCcajCeFZ/4ryf935SsEPo1cvaKUgrTtMocQyCNSvXJR9tKPhyW4qGB33iGpSPgddnbudC4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724310910; c=relaxed/simple; bh=Inuyknc0CoWbp4f/hNsXEVlWUN283GegYS4zgy+gX3E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=KJe1mWz3pKBFybD9HHoCDYDxOiDD6JYlP7fLhUcXemugvVfM52fSrliCeZI8lpUPlq0ETzCuNlddmYyhd48/nTaHZ6xqFJNQOudX6UWqp9GkdRZLycQHZrWz1Zk/zRFG954NCq5Ed7/aYs1ooMWqJal64xQ7/Md4rppuHHu78cw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Nj8KKCHk; arc=none smtp.client-ip=209.85.210.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Nj8KKCHk" Received: by mail-pf1-f173.google.com with SMTP id d2e1a72fcca58-714261089c1so332089b3a.0 for ; Thu, 22 Aug 2024 00:15:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724310908; x=1724915708; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3XLeH1tU+payUby5GohREupClIvRGZdbeLsp/8eA5FY=; b=Nj8KKCHk/bC4TSv1CV6IhulMO1Iak0Utmn3Kve7+g3rdo6xGFFLyoAkOjZlKmwRRi6 aPW9AkWRcigEIjpOGqE/fx7K68ZZQpM0zVJN0nLRQLtbbJd7F4XGq94S9L+x0YIPmYdc TXkNAVJpYxYuyAXG5XkOzOQIIFaefCJl2KGT2eZgEKqs4iVgJU/2PcBOTyfy3JL+cS3Y B7MF7mzfZzkG22GvIOzjgued2uU2YhDVWz7uR5wSDyOFSL7AZK8OpKLOG10PlF5/kWKA vRG3tOE1yoQTiPKfgJcin4FPzfnzbUGCtgXQpUO6n+2k5PetaMYSq5zEdWlerk8eMz1I kISA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724310908; x=1724915708; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3XLeH1tU+payUby5GohREupClIvRGZdbeLsp/8eA5FY=; b=pmH5YuOL8H7voZoL6JTBWQuy21a5FuZb71JpotzQfYzNww1vEb8rY2yKvrpjK+ThDS IpM8UANkN12VffCCUwgJDjcE4H0p1tiUhNYT68bUaBYbTArJwMP/sR0fWY6KgZf6J5eu QjMw+JmozdUhAC/KSOPss894Efob0puOL5FcO5uQ+9c9yHMORsvemb+J2FQibY61NE9X ZOZJiCcIFoBBM3UfI1G9bSXZJ6TfN238KijXc+EwQeMNneXWlhI8lsX789lepqCccFl8 M5/udVtVzHHZ3JCaS0Sa4bfBiOOwpA06VCV8O9qxxZ66qBfQF5lYi3QWjdESrC3MEv9y O+vA== X-Gm-Message-State: AOJu0YwK2UTHwhVKqtwkdl2EsVV/CMPDcVm5+ldO+g4RgnVrX8//5LyR Rk0bS1la3YMgIx9CGbF0QX8g1tDYpF8U+n7oyQHkxjYVahjebNc/57zerzwM2G4= X-Google-Smtp-Source: AGHT+IGjO58kdb6JjGpOfA0UnXFn2ByeOkp/uIZzxQt7YtlV+9PVIdBNBOEj4fOrvZDCpXHiWUn49Q== X-Received: by 2002:aa7:8244:0:b0:710:d294:a2b with SMTP id d2e1a72fcca58-7143176773amr3032765b3a.8.1724310908168; Thu, 22 Aug 2024 00:15:08 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([240e:473:c90:f96:d029:ea8a:4e6d:d272]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7cd9ac994a3sm695095a12.16.2024.08.22.00.15.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Aug 2024 00:15:07 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v2 14/14] mm: khugepaged: retract_page_tables() use pte_offset_map_rw_nolock() Date: Thu, 22 Aug 2024 15:13:29 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In retract_page_tables(), we may modify the pmd entry after acquiring the pml and ptl, so we should also check whether the pmd entry is stable. Using pte_offset_map_rw_nolock() + pmd_same() to do it. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 15d3f7f3c65f2..799412041d262 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1723,6 +1723,7 @@ static void retract_page_tables(struct address_space = *mapping, pgoff_t pgoff) spinlock_t *pml; spinlock_t *ptl; bool skipped_uffd =3D false; + pte_t *pte; =20 /* * Check vma->anon_vma to exclude MAP_PRIVATE mappings that @@ -1758,11 +1759,25 @@ static void retract_page_tables(struct address_spac= e *mapping, pgoff_t pgoff) addr, addr + HPAGE_PMD_SIZE); mmu_notifier_invalidate_range_start(&range); =20 + pte =3D pte_offset_map_rw_nolock(mm, pmd, addr, &pgt_pmd, &ptl); + if (!pte) { + mmu_notifier_invalidate_range_end(&range); + continue; + } + pml =3D pmd_lock(mm, pmd); - ptl =3D pte_lockptr(mm, pmd); if (ptl !=3D pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); =20 + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + if (ptl !=3D pml) + spin_unlock(pml); + mmu_notifier_invalidate_range_end(&range); + continue; + } + pte_unmap(pte); + /* * Huge page lock is still held, so normally the page table * must remain empty; and we have already skipped anon_vma --=20 2.20.1