From nobody Mon Jun 22 23:02:56 2026 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A24A01898E1 for ; Wed, 21 Aug 2024 08:19:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228375; cv=none; b=fe46k2n/EPyQAb7G4dCvMp3LTrZDam6+wXfGM/K4hxZMbYbJFhCcc3yVe6K9MoHfB0EMH/5Jbzbv36OkO2vwzfizWZDtQfgtw1O4jYUlr7K/8KJ+4wghlERvgv947CQRttPOJ25SpsvrzNA/FR3gWY7m78yJfIfCnh1CyOEMKgU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228375; c=relaxed/simple; bh=wK8isjDAuZDQnKwQW3LujcJ9UW8AZ4wxQ3aOMNHx6L4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Z1AxDLds0z7JNAq8R3iMTRHh1yrXbnem0ah3Ne4RWgzUOX+N+KUxMqL0zxbD5N7gHIiPlPrEwmzMMir6iOn/3Jniqzy7yRdd60MgpKyjbTrUiWgJjvRq0aDMZd/HKgq8I+vTBe6I402GECN7mWZ4x6t2GtxHnWbMGK8e38gt2pg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=V9d7gzZf; arc=none smtp.client-ip=209.85.210.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="V9d7gzZf" Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-7141feed424so768679b3a.2 for ; Wed, 21 Aug 2024 01:19:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228373; x=1724833173; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VgE/WG98IUTB5tF1fVgrlRXpW1wPHqZ1M9xizEuPG+g=; b=V9d7gzZfi6A/NiEM/iqvqeFJeb3U7LVXAmyCZ+rsra5sVbWlKETATeUGA2j5Qg3AL3 LdnuOlthkQZZeC9sA1C4Oxj34YdASM98y8aurT9N132imv0ZrPFJuXGGryjCqdLgAijv 5p5sPuDZqa+sfm/LdJcFQiN1bxq+NqcAR+NqIsszFxiqmhsnjK0mhy0x1neijpQMTQl3 I3LOHd66ViJvy9XQuB0kUjfsSk5kuDr31CU8LGlp23citaV4Zw5E0SWRQV8HIlwfKg+D EVI8mSoMtub0wRiRGm1WN3wn+RlsUGJSS4kfm3LJfwR5EKrDDG9SU9X5EUeZky1NbgiM bfog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228373; x=1724833173; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VgE/WG98IUTB5tF1fVgrlRXpW1wPHqZ1M9xizEuPG+g=; b=E6nDyYgawVZWn7/z1uHWQZMNMmMU8BSeTuaRGzLh5FLVYEmoMUadX0nMPT85mAKR1s HhRBic05k85VRsOoxnB7R4DDSjr5mvcM7d8bgkQmQnQ6dOoW2TyvRyNLCZKMrHcrlsBT /blINbuY6CVsHjm6YnOY0CoaAoZNsShhNaqGLz0juO/sLo40e7gCsbLEBesconp3pTjj UBPqUZOhOwVuO8RjQbfk+5whCWwbQtFIQ0XN9Z1eA2PlmhKtHYZFWwGtKxWEQk7FQIz9 nRHF8x1BvlgvvW8xO67uO3kBWcI9xH8I+iwMyW9v3+H/JcaYGk+fwYu28gxLt0cpJPhf BZfA== X-Gm-Message-State: AOJu0YwX6D0KnRrhRRGJ5oAFp6ZOinDxCtrAqk6ogJxRr1eRbwVa3Ojq tdBtxrP6S9cHn7ke7Xs1qFY2HmA7c5ZLzbCX8hodT4IlStxXBSiD8r5JdhBuJDA= X-Google-Smtp-Source: AGHT+IF11KqU9EwWDBxqOuV+x8Or0DtjlDyZMn8HSaNUPAUvyVetfGjOkbkYFHx43yYUuJqRZJ73CA== X-Received: by 2002:a05:6a21:2986:b0:1c0:e49a:6900 with SMTP id adf61e73a8af0-1cad80ef085mr1849540637.7.1724228372801; Wed, 21 Aug 2024 01:19:32 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.19.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:19:32 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 01/14] mm: pgtable: introduce pte_offset_map_{readonly|maywrite}_nolock() Date: Wed, 21 Aug 2024 16:18:44 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, the usage of pte_offset_map_nolock() can be divided into the following two cases: 1) After acquiring PTL, only read-only operations are performed on the PTE page. In this case, the RCU lock in pte_offset_map_nolock() will ensure that the PTE page will not be freed, and there is no need to worry about whether the pmd entry is modified. 2) After acquiring PTL, the pte or pmd entries may be modified. At this time, we need to ensure that the pmd entry has not been modified concurrently. To more clearing distinguish between these two cases, this commit introduces two new helper functions to replace pte_offset_map_nolock(). For 1), just rename it to pte_offset_map_readonly_nolock(). For 2), in addition to changing the name to pte_offset_map_maywrite_nolock(), it also outputs the pmdval when successful. This can help the caller recheck *pmd once the PTL is taken. In some cases we can pass NULL to pmdvalp: either the mmap_lock for write, or pte_same() check on contents, is also enough to ensure that the pmd entry is stable. Subsequent commits will convert pte_offset_map_nolock() into the above two functions one by one, and finally completely delete it. Signed-off-by: Qi Zheng --- Documentation/mm/split_page_table_lock.rst | 7 ++++ include/linux/mm.h | 5 +++ mm/pgtable-generic.c | 43 ++++++++++++++++++++++ 3 files changed, 55 insertions(+) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/= split_page_table_lock.rst index e4f6972eb6c04..f54f717ae8bdf 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -19,6 +19,13 @@ There are helpers to lock/unlock a table and other acces= sor functions: - pte_offset_map_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_readonly_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_maywrite_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken) and the value of its pmd entry, or returns NULL + if no PTE table; - pte_offset_map() maps PTE, returns pointer to PTE, or returns NULL if no PTE table; - pte_unmap() diff --git a/include/linux/mm.h b/include/linux/mm.h index 00501f85f45f0..1fe0ceabcaf39 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2954,6 +2954,11 @@ static inline pte_t *pte_offset_map_lock(struct mm_s= truct *mm, pmd_t *pmd, =20 pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_readonly_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_maywrite_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp); =20 #define pte_unmap_unlock(pte, ptl) do { \ spin_unlock(ptl); \ diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index a78a4adf711ac..29d1fd6fd2963 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -317,6 +317,33 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd= _t *pmd, return pte; } =20 +pte_t *pte_offset_map_readonly_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + pte =3D __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp =3D pte_lockptr(mm, &pmdval); + return pte; +} + +pte_t *pte_offset_map_maywrite_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + pte =3D __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp =3D pte_lockptr(mm, &pmdval); + if (pmdvalp) + *pmdvalp =3D pmdval; + return pte; +} + /* * pte_offset_map_lock(mm, pmd, addr, ptlp), and its internal implementati= on * __pte_offset_map_lock() below, is usually called with the pmd pointer f= or @@ -356,6 +383,22 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd= _t *pmd, * recheck *pmd once the lock is taken; in practice, no callsite needs tha= t - * either the mmap_lock for write, or pte_same() check on contents, is eno= ugh. * + * pte_offset_map_readonly_nolock(mm, pmd, addr, ptlp), above, is like + * pte_offset_map(); but when successful, it also outputs a pointer to the + * spinlock in ptlp - as pte_offset_map_lock() does, but in this case with= out + * locking it. This helps the caller to avoid a later pte_lockptr(mm, *pm= d), + * which might by that time act on a changed *pmd: pte_offset_map_readonly= _nolock() + * provides the correct spinlock pointer for the page table that it return= s. + * For readonly case, the caller does not need to recheck *pmd after the l= ock is + * taken, because the RCU lock will ensure that the PTE page will not be f= reed. + * + * pte_offset_map_maywrite_nolock(mm, pmd, addr, pmdvalp, ptlp), above, is= like + * pte_offset_map_readonly_nolock(); but when successful, it also outputs = the + * pdmval. For cases where pte or pmd entries may be modified, that is, ma= ywrite + * case, this can help the caller recheck *pmd once the lock is taken. In = some + * cases we can pass NULL to pmdvalp: either the mmap_lock for write, or + * pte_same() check on contents, is also enough. + * * Note that free_pgtables(), used after unmapping detached vmas, or when * exiting the whole mm, does not take page table lock before freeing a pa= ge * table, and may not use RCU at all: "outsiders" like khugepaged should a= void --=20 2.20.1 From nobody Mon Jun 22 23:02:56 2026 Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 164111898E1 for ; Wed, 21 Aug 2024 08:19:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228381; cv=none; b=na7lH7s95JmBKnnV4gpUDikBiV6ygF013CW7rbKRbpysqvizZQMgrp6XX9rky8Oyh32Es1wJSDSxhiHihyIkqi7Nmq7lCYxvFwvRDcftFamAwCQsYEyxkUQ0NWvpb5mSU0p20O0kO2k/NMShnPNqOKvU4pSXBoQ80zhwXck19Ts= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228381; c=relaxed/simple; bh=uGjiG/CFl2j6ES0y5dOhuNObOgpK8L+F0ILwLT9qNWE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=XwNchM8gmoaqwhWvAteL7Zr5J181MAyti3r3AkTigkuoiUy2OYZv1HqdZfPdQrB14dVRydt51ZxjMHVRVZsu99XQ6AJjOHCJyF8c6QKj4tJqKzJGLB8FUqF9YSQ/5dGW5LO1FTvyooD5NH1dO13OZdNHNZ2ObyEJky9mhMnw0Ac= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=UxLjfJ9P; arc=none smtp.client-ip=209.85.215.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="UxLjfJ9P" Received: by mail-pg1-f174.google.com with SMTP id 41be03b00d2f7-7cb3db0932cso2899935a12.1 for ; Wed, 21 Aug 2024 01:19:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228379; x=1724833179; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=arLLKSkNLaMIjiTVBc0cMKPJV4f+y4Msx8WJt/66k5c=; b=UxLjfJ9PUaE4i6AMxfTPrcPaHALSW3BtNK8Cafv9iUqPeBSpr0cMnh15EefXLe1M2+ AtK1QcR2+uoJLW3jA1OrWMfs+OMZ8MjBojUfeOCExuUO39jfU1UCrLAXQjhO9wqk2KRA SaA08LrViOSfR+ZaTjHG+BbnezHS4JTffQGblIUl0nDkA32y78J3oOCYeu8vc8LMMcnK /aUT0Fsn5nJ9xj4V7wIcGiZASzt5r0s98E+TNjnqgnaLPzc7CHLxHPpasNiIfatzWOSN /ALWiMF5EnfuQ+h3Me7ENprdvt5GfPG7JnXzHLEWvJN58ZeBci69gFnh6ZT0O+rlVCSL ceTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228379; x=1724833179; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=arLLKSkNLaMIjiTVBc0cMKPJV4f+y4Msx8WJt/66k5c=; b=AwWVh75wqjBidRmv+61GrR7P/xb6LHShcrqsL0jK3aLFR7+L4wiYoY7T9knFwKXPQZ S+8AiI0ywhN2dwWZh1gXMdZW/Xq+Pd1an1jTKj694cWrd7GmoyUczMKNTf1cUvHDNI/a Vd4wuqo5AOOkjZ4v+GuBtPIilJWXExMgnRugVQigrrBpgsJDUlhVbyh3BNRruCJA1l2S cPJL95aaoF0+8K0Y+Va9K7ZI4gwvw9GE6MXUOfcGC0wQt/DpafxPFmeFMHWUHT6i95ee mbxiKIFPP+rciXcTD3uqrD6b5SDmAZpIHGiADehzE8CpJGbxhUavWIDESIb+zRS2vGju Lr9g== X-Gm-Message-State: AOJu0YxsJHm7nuk22A/i3tjVkNbok0Y3a0MLeu8Uv5/9/G4OknVVDIeX j6ijFWN+y77WoMo3tJkFTJPeCyUypRr+qvaG6illNf2GZGaeUYoRjGqkenUuzp4= X-Google-Smtp-Source: AGHT+IEvHWRxH/KV5glvGK0Xiv2dJ13pnLs9scNXFJwmOSFepkjBPO6rW4EjBZpYY7GPQ4yq94i7jA== X-Received: by 2002:a05:6a21:168e:b0:1c4:d05c:a967 with SMTP id adf61e73a8af0-1cada179499mr1734402637.51.1724228379371; Wed, 21 Aug 2024 01:19:39 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.19.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:19:38 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 02/14] arm: adjust_pte() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:45 +0800 Message-Id: <3d151aaf876b93a97fd3f070780345a3bf572123.1724226076.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In do_adjust_pte(), we may modify the pte entry. At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the PTL held. The corresponding pmd entry may have been modified concurrently. Therefore, in order to ensure the stability if pmd entry, use pte_offset_map_maywrite_nolock() to replace pte_offset_map_nolock(), and do pmd_same() check after holding the PTL. Signed-off-by: Qi Zheng --- arch/arm/mm/fault-armv.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c index 831793cd6ff94..5371920ec0550 100644 --- a/arch/arm/mm/fault-armv.c +++ b/arch/arm/mm/fault-armv.c @@ -94,6 +94,7 @@ static int adjust_pte(struct vm_area_struct *vma, unsigne= d long address, pud_t *pud; pmd_t *pmd; pte_t *pte; + pmd_t pmdval; int ret; =20 pgd =3D pgd_offset(vma->vm_mm, address); @@ -112,16 +113,22 @@ static int adjust_pte(struct vm_area_struct *vma, uns= igned long address, if (pmd_none_or_clear_bad(pmd)) return 0; =20 +again: /* * This is called while another page table is mapped, so we * must use the nested version. This also means we need to * open-code the spin-locking. */ - pte =3D pte_offset_map_nolock(vma->vm_mm, pmd, address, &ptl); + pte =3D pte_offset_map_maywrite_nolock(vma->vm_mm, pmd, address, &pmdval,= &ptl); if (!pte) return 0; =20 do_pte_lock(ptl); + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + do_pte_unlock(ptl); + pte_unmap(pte); + goto again; + } =20 ret =3D do_adjust_pte(vma, address, pfn, pte); =20 --=20 2.20.1 From nobody Mon Jun 22 23:02:56 2026 Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D29E199FA3 for ; Wed, 21 Aug 2024 08:19:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228387; cv=none; b=rTPgztVQSlytlTFts4oWrjXYq/YaKpvvbRotINKxD6G2uASiTWCtTuJn8J3fz4aL9jEAc40sYEJquyhL8IgCLhC1/eylzsE9wPftvKc97ioT0PF36CHgkFfTtVaUMSIs6SUu2h5D5mIL/6kOPcqTvoYwWV29j6bv9B/ggNC+OdA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228387; c=relaxed/simple; bh=yDALv7oJp6DYj4uXpgdGiO5A+F1mvOQOyud1tIP6NFQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=rNzAp2a6+uDrGBEjq7rLAfh5590AhEX2UKnLP8rxiRL9GDB3TE+nXc8NzBUmKKAV86sACnaTg0hnGxdPDV9NG45+NKQ7jEzRFJAANIEZls56Vclc7h6u5LQOqtapzUgnCw7v6fvKAHFaIMZrJzuhJpHhvBtQvLbz00gMK6BTB8s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=YgJu+bvt; arc=none smtp.client-ip=209.85.215.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="YgJu+bvt" Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-7c6aee8e8daso4420274a12.0 for ; Wed, 21 Aug 2024 01:19:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228386; x=1724833186; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WkLNXpSDw35lj4Q8QTsPG+BhSNUK5Ar50Dqo0ISPkHo=; b=YgJu+bvtUp2DJmrfv+ISknWMWaBG8ZQRcyd8K1vcZ9IMkHB+x6LkzGyX5Ra1bGbNK4 vpRp0T5M3BbjVmhTZTm6EnF8hlshYUTWtg6dlAAwKxGCQkJ7OOEYHZS/RbW29Jghxlj+ 5yzuq4+ZLXn4a6yL1hZxAYCeTCB0dvMwWgwu0cS9Ki441kldNRdp7p8+7tIKix2/0kxe FMfnquWdHAorX/PHTEPpHQlSKVhugsfDlrSE17N5Hs+2dTZwrPxXztgvjxcYJZMBJ6HA YLMpl2Tx79BBV0KtCcq5w84EFN2jZ7PbX3jnGjSDPm7oxXSSCwXxm/b/YT3PnvvYhFO+ 7P0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228386; x=1724833186; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WkLNXpSDw35lj4Q8QTsPG+BhSNUK5Ar50Dqo0ISPkHo=; b=uouxjLkcLIokuW2TCRkJ/K8teTZdmr0Ks0DqdaYroR6tPcjTCEXgi+vRrkzaQEWFFl +zRj3iR216TsBvEURza7yHo0X2vNjSabzYcU1aS4v7AvXk+DssPtztq0v7oHANY/ooCU hpY5oazUuUYK5lGk4ZIKNYVHLrqi9bLxRaMKPyBpMQynbZiQQMsMPmaAgFeuaXoOdP4q c1/qywQXKJRfbztSuMUOtoPRu9DtHiVlpSYin7wkPXwrwwZh16u0b3BoVYhn6mGlMVcj mnitZGCS6ylz5fXpgtYGiobptAFA6OXY5nmxl4f015PQi5IDhz9tk7nVrRDwKmdot86l d7Fw== X-Gm-Message-State: AOJu0YxVYeZN3fJHTVIoGwk6VneZ4KhnTwl8TAcnKmnUTzrSdSxGFr09 a5C+UXkiFKrzC+y3fvhSuUde5RcbqZG2mM2b9sxbUwZh1i66DnK6xKO4SovBUXY= X-Google-Smtp-Source: AGHT+IFyY3NlNyztu6E70clqH4irJgHSXlPUHA4AtOW6RrWWSbwAKaoUYDBEK6KZMX7nlR++kviHPA== X-Received: by 2002:a17:90a:e7c6:b0:2cf:cc0d:96cc with SMTP id 98e67ed59e1d1-2d5e99c8b75mr1585510a91.9.1724228385781; Wed, 21 Aug 2024 01:19:45 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.19.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:19:45 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 03/14] powerpc: assert_pte_locked() use pte_offset_map_readonly_nolock() Date: Wed, 21 Aug 2024 16:18:46 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In assert_pte_locked(), we just get the ptl and assert if it was already held, so convert it to using pte_offset_map_readonly_nolock(). Signed-off-by: Qi Zheng --- arch/powerpc/mm/pgtable.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 7316396e452d8..ae4d236b5cef7 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -398,7 +398,7 @@ void assert_pte_locked(struct mm_struct *mm, unsigned l= ong addr) */ if (pmd_none(*pmd)) return; - pte =3D pte_offset_map_nolock(mm, pmd, addr, &ptl); + pte =3D pte_offset_map_readonly_nolock(mm, pmd, addr, &ptl); BUG_ON(!pte); assert_spin_locked(ptl); pte_unmap(pte); --=20 2.20.1 From nobody Mon Jun 22 23:02:56 2026 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5623E199FA3 for ; Wed, 21 Aug 2024 08:19:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228394; cv=none; b=k3TGwqsbPBNuv1yK21/O1+jizQUA0K3exUBcuVGBfA3IYAEyrFonyLnUhiUMG3urNxne2rZsUXVTzlf5tMUECtxz5QzaJIbWCS3FX+GjX0u7vZ27C37dUqBpbb+TVEkHDd9vWaTydFu8p1vzUZd4PjZM9cu6wEu/jEHdazXFFj4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228394; c=relaxed/simple; bh=e1tBcpkPCPYCjJVE5fYukBveVHFGvsGMgvqtXeHa2/s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=o2NGZKhfJuWr2ZY9G8qpYmZtyWHuYlRcigQlqHspEKa3cSeFXBdCMbK5lpXr6WHKDCUOwQNpv8lpKv3bo7TRxD/6QGv1CRVkQHRWH1yJz/Dl304Xz53JzfSG3rsvTxpGa3Sovync3I2lN0p0+tTnSm6oRl0cK8SCHcppVFJf0JY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=GLUuoRwc; arc=none smtp.client-ip=209.85.216.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="GLUuoRwc" Received: by mail-pj1-f44.google.com with SMTP id 98e67ed59e1d1-2d41b082ab8so2511403a91.3 for ; Wed, 21 Aug 2024 01:19:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228392; x=1724833192; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GHfDnFSOmlgsN5ypR9SiqdvWOZgzK74sPfUFdgGElDM=; b=GLUuoRwcR8/rzdEMCh8SCEAyaQaHQbCttJqbwM4neB3yQKsX6XVasXn5WQGXV8yEbl gFdhC66Ac5sG5hutzdKySwUtGzpxiAEbnHv/HgLFvIcT29jCobQ8tVZ3MeEvMMV548nY sh9KwbxkPDmNCXqkIc9EQow3skmuPHW9142r53eBBT7+L7G96YUNcEbQCsSuAudH18q/ PqN2Sqskvq2ctCeEQTlYi17bU/8W4atHRCbh92iLkDqOh1cLY3INGrpgHDp0Lf01NHyS m9NYDPhomB6h3suTE/XuiO2hhe7/KABYT/jdnmeTEzTbxQqPCXaNcGNepdA5wxXKOiRm tpHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228392; x=1724833192; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GHfDnFSOmlgsN5ypR9SiqdvWOZgzK74sPfUFdgGElDM=; b=Ya+gvSmWV8jvSDtgDzmT3+oGj+ocZwrBr9e3cB5lCHiqj4+067BmGR5IqqMEiq1Mja MZ2AR92X9I6AteDDyFXzcfxN1wik2fbVnOjTsyu9GT5ZKPZBDgbApc/T6I2w9wcM9fL1 WpqkWu7Ir5f3NGrAo8elNhmWE2+TOspeXM0rvHLcBCKNIrLCVRk3NM2qw+Q+CqI5bCDQ F0uA6SY7PWnV7lI+8akeNnEjJBfJ72LqRfDISRD0p9bwjRXQSpj0w7cPks/STrlmpysH Btz3cLpGUnNxK7n3hPbAFwSe0H0nsdmijeP/X11zFQGtY7T/0rAWLBsfi5I+ZFN4ysAl Rc/w== X-Gm-Message-State: AOJu0YyKl5KJ/tydAuXXlZa2lqb2zGlgyZdOKEt1PfnQnTKCezuSfVJt 5oQgVEiUO/A0EPFCM1ihzlVuW7IqgG6aqo/BlrJflso9OSUOFWMTUA5TTKxXx/c= X-Google-Smtp-Source: AGHT+IEDKdhDrkAt8Xw9HjhxvAn16yLBj0ykk9gzeBgjUCoqqbqdejlnt+3gntiVrRuxkT/2XR+s6w== X-Received: by 2002:a17:90a:d98e:b0:2c2:c149:ca4 with SMTP id 98e67ed59e1d1-2d5eaa9d141mr1321075a91.43.1724228392525; Wed, 21 Aug 2024 01:19:52 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.19.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:19:52 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 04/14] mm: filemap: filemap_fault_recheck_pte_none() use pte_offset_map_readonly_nolock() Date: Wed, 21 Aug 2024 16:18:47 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In filemap_fault_recheck_pte_none(), we just do pte_none() check, so convert it to using pte_offset_map_readonly_nolock(). Signed-off-by: Qi Zheng --- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index d87c858465962..491eb92d6db1f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3228,8 +3228,8 @@ static vm_fault_t filemap_fault_recheck_pte_none(stru= ct vm_fault *vmf) if (!(vmf->flags & FAULT_FLAG_ORIG_PTE_VALID)) return 0; =20 - ptep =3D pte_offset_map_nolock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + ptep =3D pte_offset_map_readonly_nolock(vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); if (unlikely(!ptep)) return VM_FAULT_NOPAGE; =20 --=20 2.20.1 From nobody Mon Jun 22 23:02:56 2026 Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F199B199FA3 for ; Wed, 21 Aug 2024 08:20:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228402; cv=none; b=kuG1no6VgQl8CNzuvf2eKhHef3C3Jsg+e6pOPaKH/lrbqaRrb9rq7K/5wWn8nNbNTOcjIhIoyslohnzozHRklIYX22VFECA8w4sSfDoCjaZyIp2NnTZsVdagNcmhkZiay/IIktpd79BWax7Pidu4DHPlYAkIUWXItNFUhVgDdJ4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228402; c=relaxed/simple; bh=bPoIcZSmQoKSl1mt0GTa8HT2fvR5BUPjDUZk6CLI5r8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=kSCNNy9RmNuDn0wizpW8iBowUiyK/uWyX/sBDO0FElm36fVGps4mRw8JuX/iuMAJ47244mDKibrc6/T0vdC5ydWqo0bAzQrKRM8jNOEgyn9ghPiy6cSn3PRWkO+bRIlEKen4uBaKBZjtaIR+eEC9LiGC/lZdRZqRLsoLUoNQgkM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=JguoRKLH; arc=none smtp.client-ip=209.85.216.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="JguoRKLH" Received: by mail-pj1-f47.google.com with SMTP id 98e67ed59e1d1-2d5f5d8cc01so117815a91.0 for ; Wed, 21 Aug 2024 01:20:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228400; x=1724833200; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AWBymtmrK19rfh+o6mKAIf3qo5pwJHde1CNDuWd4778=; b=JguoRKLHodNuG6yyL3Vi4vpX/EsgpPP6gbPWkHCLf0leT9WJzHCmMD/Yn74Lp0/nW1 6xITO+e6JtSwGNWdPomcaDOaOWJxyTejR50Mrbjzreh0KBOg/Tnzu+E6Jyhtoqc3J2wY rPM4FNauDsw3w3k6jJ1TD4jXM+2hhgL+uPMjVbYxsjnCks01GHQt+nliQScLLf1doJbg DrrQNZOSqBsvmQPuY3aquE0txgCCecWwgUJNGc2VgU3EAY/o5BAo48Wd6+Ukm3bWKfFF mVfOmYvh9hHmSJGPlBVMdjMahDHRzdVacCtCMhdSx7DwBgAgVH/4AwbJpDa9n59PrZ9/ rZ5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228400; x=1724833200; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AWBymtmrK19rfh+o6mKAIf3qo5pwJHde1CNDuWd4778=; b=pPfVUcmnzOM/gfHx2ZloTqPaHSig4DNzhwEVjZDuaaw38eMLOSTFWjitNYDFWVg8/V uiyN3rBoakBejX/WK7vDsEku+sbj5WhIyVstljuhLiEDWWB05gZCkKz6lgH9zD7Bk52Y Ie+titnlKV94Qz0c4XHb8Q98PzEqSNhWavhZuK93n23jL79YauVUr0FJYmf9F3Qdeomu eNj+Q+YbNHLZcqskZUdz+hcmJg0T4nmtzyV6CKl0Ll1wsqgFvoUcC9UmhvYbKa0Uz74v LbPpg60B2sbjxtQRNXIM6JOyDCywnMCNSeqz4FmXMl4oXgkOoaC3Nsvv/rNOtb3JQQes XrVg== X-Gm-Message-State: AOJu0YwRZjBzKE/xcdL/MWaGEv84WUSyvteEV6wpIR73PUjIxBzskgP/ W4SZifz699S6QPZTFUcmsJe3QVFSrjQTomMZgmnDPKOIFNXKJWNqy8PgZrGc154= X-Google-Smtp-Source: AGHT+IETLNjbMtdResL0gCQrMShCwCpeGL/L1zL6e1zJRM+ER4woSJMvk4wtR2Cz+Z4m4+aIJL98tA== X-Received: by 2002:a17:90b:4c8d:b0:2cb:50b8:e59d with SMTP id 98e67ed59e1d1-2d5e99a5e05mr2805975a91.12.1724228400217; Wed, 21 Aug 2024 01:20:00 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.19.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:19:59 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 05/14] mm: khugepaged: __collapse_huge_page_swapin() use pte_offset_map_readonly_nolock() Date: Wed, 21 Aug 2024 16:18:48 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In __collapse_huge_page_swapin(), we just use the ptl for pte_same() check in do_swap_page(). In other places, we directly use pte_offset_map_lock(), so convert it to using pte_offset_map_readonly_nolock(). Signed-off-by: Qi Zheng --- mm/khugepaged.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index cdd1d8655a76b..26c083c59f03f 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1009,7 +1009,11 @@ static int __collapse_huge_page_swapin(struct mm_str= uct *mm, }; =20 if (!pte++) { - pte =3D pte_offset_map_nolock(mm, pmd, address, &ptl); + /* + * Here the ptl is only used to check pte_same() in + * do_swap_page(), so readonly version is enough. + */ + pte =3D pte_offset_map_readonly_nolock(mm, pmd, address, &ptl); if (!pte) { mmap_read_unlock(mm); result =3D SCAN_PMD_NULL; --=20 2.20.1 From nobody Mon Jun 22 23:02:56 2026 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B133A1B1D60 for ; Wed, 21 Aug 2024 08:20:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228411; cv=none; b=Mr1wqONnxuMNK3IiaaCq3b7RGkv68dJLxMJWVJGCcCz8PVywDVwz+28AizBc/cEAAQS8uvuuMyehUa5p5l6ik/yzrVRyXy/F8NuqNhiH0BXAXN8iarqsJHjHqtYMDTTLDszwxsaf55PGTpAt1dY6LDNrVQg0O6U2dANHhX0Kx3U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228411; c=relaxed/simple; bh=6VVYech/7eI8919A6pOE+PmFQ5cOQ+vb1NF6FJ9YCKU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FF+3eEb/kLlLtw+gBEy8DiEeO0B91xJM4OR6KXEe/1NTPTdJx76Ajq4sejKPShysN/8wsUVrpx6rhSorGE7cnlyvgRuq5BkYiat6wlQwJwg8JSwcU+Dtn/coPtLPIWd17QogP7lYnBmwbMOnS8dXjYa7s20IWjkIbKchuXSqNT8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=ZaK4qU+6; arc=none smtp.client-ip=209.85.216.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="ZaK4qU+6" Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-2d439572aeaso379668a91.1 for ; Wed, 21 Aug 2024 01:20:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228409; x=1724833209; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xk5nZGQoU/UHicxrt5T+UXYIL6+4L5PcGNmpQKRqaJs=; b=ZaK4qU+6lg5h6QpKcBH1Ve0mKjYKu7M2HDK5rXcUWrqI+LxfuJCBmcZ2VxtHhLXrqr 15sZU6UFHWfRM0EgbwM3uUk6S8icIu31qK8e9TOGl0tpZIYAzr7RDwU1wvxQa+cs50Sj GPnR6ZccSRzOzdyBb2uRqqqnVzTP+PLxW6WcMYBsD3Aky6D0hUK8wuEC3AzwE3dynTQK 49aDYRBVafRFlo8EUJkOfEudZ96jUC8evd0TOc9//M7cj0GITkl2efTFsZ7oubAOEnEw GLd7oY2BKU/yWQv7hqW3YgvuKhpgWrc50Y66AXh49G6BdwRapJmNsNPQGKBnsP4F76Dj x/IA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228409; x=1724833209; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xk5nZGQoU/UHicxrt5T+UXYIL6+4L5PcGNmpQKRqaJs=; b=CnocdjaU2QspOsORERyQYFno54JB7PEYBEA924qEc1ZGh6jA0GAVnytXCx/I4PUsQK dnr5ze1HhEFTRS+viQ0N2PW7Pb7cAQTkzdiCAT30kwXB9vxiHKSfvCopWkMVYgiE0p1c W99XWr4GOR5Y9Uli0kGwAXLoLEMW2J+Wc+TqYjDjlr/Y+KTFxEeZ/HKmbhfk3N2Zet4v tUxCTKsrB+hLHxNDIvIiDNCNNoX9XsAoX4HDfaSlEWv/CT8mShWuteXEYb1z6S4Loy3q WAb3CgKGm5GtXMHvfr9Nw1hqhrZgpfLn8sL1TGNFqidbCJII6XeM8UPcsIwHSwdNVvJD FGdA== X-Gm-Message-State: AOJu0YxRLEreIBpk76EHUphqe2RyeMNOlYDwyHNDUfCx6B9cjLXqNbP+ fLtdHU+NUtPq3AHgxY7rsBz3Or0U9oVAPy7WiqqFQu2gj4rrIrFJFZbD41hLkbaDlj+3P2EFwtm h X-Google-Smtp-Source: AGHT+IEz6hEUMU7QDWWn6q3mNXap3/YJ5XtpoR3UgLNTk8qimJ6ZhRH05YhkApxF5Pk84JIDmuV6pg== X-Received: by 2002:a17:90b:4a10:b0:2cb:5dbb:d394 with SMTP id 98e67ed59e1d1-2d5e995c932mr2717849a91.4.1724228407428; Wed, 21 Aug 2024 01:20:07 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:07 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 06/14] mm: handle_pte_fault() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:49 +0800 Message-Id: <239432a0bc56464e58a6baf3622fdc72526c8d57.1724226076.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In handle_pte_fault(), we may modify the vmf->pte after acquiring the vmf->ptl, so convert it to using pte_offset_map_maywrite_nolock(). But since we already do the pte_same() check, so there is no need to get pmdval to do pmd_same() check, just pass NULL to pmdvalp parameter. Signed-off-by: Qi Zheng --- mm/memory.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 93c0c25433d02..d3378e98faf13 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5504,9 +5504,14 @@ static vm_fault_t handle_pte_fault(struct vm_fault *= vmf) * pmd by anon khugepaged, since that takes mmap_lock in write * mode; but shmem or file collapse to THP could still morph * it into a huge pmd: just retry later if so. + * + * Use the maywrite version to indicate that vmf->pte will be + * modified, but since we will use pte_same() to detect the + * change of the pte entry, there is no need to get pmdval. */ - vmf->pte =3D pte_offset_map_nolock(vmf->vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + vmf->pte =3D pte_offset_map_maywrite_nolock(vmf->vma->vm_mm, + vmf->pmd, vmf->address, + NULL, &vmf->ptl); if (unlikely(!vmf->pte)) return 0; vmf->orig_pte =3D ptep_get_lockless(vmf->pte); --=20 2.20.1 From nobody Mon Jun 22 23:02:56 2026 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4C6551BAEFD for ; Wed, 21 Aug 2024 08:20:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228416; cv=none; b=akg6qWG0BweRjE2T9qPjPqJWTsB5gd5DNNVwVz/wp9ha6yKuoH7XbcdEDfU8Sp1rYfO2I5tEKBDpjpj43eOnloEHF7Hk5CcL9yxYx7DPPn0Hmlmff++rheX9GrRG4wMmp7VHXbX8EPV68HnxU2Q4xSj+O4H/9DXxAZJNVBNYrOo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228416; c=relaxed/simple; bh=whO6kmMVZ3486illXrVIP3CwwypQL42Bi9p6yT8iY8s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=eS9KuKh++CXiarZIbPaHYnO0KE22CLIeZ1pegub1Y+WodhV1AqYNlgczycarDR1dtgThdmfJ630M0m1OxwBO83gK8ZNd9PfxUUMNgc4K4iGbG8YIyxenc0d+Fj5UF4zHUaO6fHb/PC2CIiRq4lYL8sdhBmRMVgJpgTkkey7QsoE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Spf+cYDM; arc=none smtp.client-ip=209.85.216.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Spf+cYDM" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-2d3c0d587e4so4493302a91.2 for ; Wed, 21 Aug 2024 01:20:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228414; x=1724833214; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ATsR6rCpxZFOFORcauHss5xESVmegH/7Abibhpgyw90=; b=Spf+cYDM7v5YuObDC/7Wvl7J+Y6efhPsBIodCd6xlN7kDe7LUtMFYK/FZ3S0KvIjpf D6UZSjjbNIWvz6d766vR0fpiGoUuU7EwwyhdHks6zXIwWOxF81h9yMa8w3LdAcmYPqZa ciC4atuTiu1GgKYnc7kZvXBCYa+38CxQxp5bWwB51/s9M4iFqluKAlvkIlUMQsX1DwRp //L9QTP4dTCV0RPmWZTAw7fN6oVAqyKMAj7Z0OnvdvQVlA/wbyxjxukMpnKLtzmE4O9l 9DxNm6HSiYSM9jjG9TO04mQKcTV0JiJWQmHtZGjWzI8hLlF+BWgyuk4jFIXOnjVYbLlf obWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228414; x=1724833214; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ATsR6rCpxZFOFORcauHss5xESVmegH/7Abibhpgyw90=; b=ZEzuUHgGxfgtoQ1i61GoTrGUHFN/HF0v7Iw8viEm2opfzFGPOPjHrwB6TuG+Cnnpd+ el+24IXL9r41drm/zeH2caDVdfE6Sc8VnOAxB1SgKtScVIwKOzSZGImnRRkzliouRov2 s5ubW4ZBauS7NXeh1hzYm4Sbg2qXFhjufksKxBiloC88AAQZdG3BUeP2g01gZP5s275Z r7cwxZxPRvBdWamVHemsL0E3KJlrihROFAasMZuBI6s4ClbUZ7Bse2vOKwPvVJaeQ2v9 hvJC4sg6X3XEpWgFN1O0csVahhC3JGtqjI7s6LaR2wtmAL4+t+EjDYTK1rMFf+p6E24G vi5Q== X-Gm-Message-State: AOJu0YzP8RNDQ4c7H0kScMjqxhHFCd/MdRLg5h6s5bsA6XBXRADLhV20 A49ME38uX8GNuk1fkEXZZ2XgWzkTvbm1bH1GCESs1GQQhhrFlSYvFVDS/qmG+yE= X-Google-Smtp-Source: AGHT+IEASO/tYfbmKEqODBO5ptoLkIYqajvyb9bjTzXqrTBGFe58odiOJRUvoUKiSb3oI+1Cc8Ov6A== X-Received: by 2002:a17:90b:1c01:b0:2c9:9658:d704 with SMTP id 98e67ed59e1d1-2d5ea4c9ab9mr1366119a91.40.1724228414402; Wed, 21 Aug 2024 01:20:14 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:14 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 07/14] mm: khugepaged: collapse_pte_mapped_thp() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:50 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In collapse_pte_mapped_thp(), we may modify the pte and pmd entry after acquring the ptl, so convert it to using pte_offset_map_maywrite_nolock(). At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the PTL held. So we should get pgt_pmd and do pmd_same() check after the ptl held. For the case where the ptl is released first and then the pml is acquired, the PTE page may have been freed, so we must do pmd_same() check before reacquiring the ptl. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 26c083c59f03f..8fcad0b368a08 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1602,7 +1602,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) pml =3D pmd_lock(mm, pmd); =20 - start_pte =3D pte_offset_map_nolock(mm, pmd, haddr, &ptl); + start_pte =3D pte_offset_map_maywrite_nolock(mm, pmd, haddr, &pgt_pmd, &p= tl); if (!start_pte) /* mmap_lock + page lock should prevent this */ goto abort; if (!pml) @@ -1610,6 +1610,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, else if (ptl !=3D pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); =20 + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) + goto abort; + /* step 2: clear page table and adjust rmap */ for (i =3D 0, addr =3D haddr, pte =3D start_pte; i < HPAGE_PMD_NR; i++, addr +=3D PAGE_SIZE, pte++) { @@ -1655,6 +1658,16 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, un= signed long addr, /* step 4: remove empty page table */ if (!pml) { pml =3D pmd_lock(mm, pmd); + /* + * We called pte_unmap() and release the ptl before acquiring + * the pml, which means we left the RCU critical section, so the + * PTE page may have been freed, so we must do pmd_same() check + * before reacquiring the ptl. + */ + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + spin_unlock(pml); + goto pmd_change; + } if (ptl !=3D pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); } @@ -1686,6 +1699,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, pte_unmap_unlock(start_pte, ptl); if (pml && pml !=3D ptl) spin_unlock(pml); +pmd_change: if (notified) mmu_notifier_invalidate_range_end(&range); drop_folio: --=20 2.20.1 From nobody Mon Jun 22 23:02:56 2026 Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BEEB61C86F2 for ; Wed, 21 Aug 2024 08:20:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228423; cv=none; b=I6X3QYIsaJub/17vW7EafN463VwWpHh4cmqa+7e9nOFF4/JxMlwqv0EUAtEeKWb42G3a1xbaTLWfq5G20JxQPUNVzvFeT8FqROrrPzGVeLyyGciEACcJTZ8XY58W/ocI1HPE0PMOz1IdQRbfcc8fx2N9vDQKqYMFeb8EuvVy7VM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228423; c=relaxed/simple; bh=xZxfKK2WH1d1pr19c4Cwsg5EAO66G+hvyLkF40Xhpj0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Mx0cDFz+mR5kwYUfJIXE+DTAiJYVorZUSiwIlLz+WEhgooZMsOXqLC0Hqa2heltRZsfN7jX1pKhL872HAZ19ABe4lssJjkdvwQaY+DxzLNQNaatpK096iGw6qkXVk0YmwTCLlsxbaGu8GjZ14kSvEQaI82un/fTXk4P09ouDZow= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=c+asUPRD; arc=none smtp.client-ip=209.85.215.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="c+asUPRD" Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-7c1324be8easo376477a12.1 for ; Wed, 21 Aug 2024 01:20:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228421; x=1724833221; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ib2peXsn8Xzv08d3et99PPledeQt2suHPjEB9mw45Nc=; b=c+asUPRD4R4IaZErVcVCapOTusoSxw2z9GaG5BsWC7WPQOhC2LRQyRdCcB9yXp5woX NMj1mEK1tzrOo4zxyTpHVYnoTkmjUII1lcNVncxwhulouLuvLMvxI8UCZpCMDw64AxJC wDPsm2/wC5FJMwoEsjiCespNwvDliHUYbCx0Eg2WOJTRsDJsH7h6dGeHzubxgyWbt5Ae YWC1kqDB0fOVJ/QPmf44RfHrL7kLh1hp6FAELx8cXgHKtQ9Fb3FFE/L7ubsWDjjRpJTa WIOqYzmdVMdVHrN7FN0WJ0HoNVcjJh6ZUVmzPUlyMiqIo4SUqiSqK4nYaxBRWsr/xabX q44A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228421; x=1724833221; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ib2peXsn8Xzv08d3et99PPledeQt2suHPjEB9mw45Nc=; b=bmvQ2jwePaGpYAINFNJLtWsxR+ooS6IOzR/rHsOeEsT39WzSCsA2F5bT4Z3RUTFsrr zYdkDdpM/1r1jJJL1zIjfiL5+fNR+COIW9YGCOpUcPA2NusBE99DKPCE5hu6MMK8suP9 HpgMUCo0wu+YUI7jAFrgM004GThdU0Sbox2jXuiVRRucXGscRXuqh8XvgzMdPU8oMJP8 V+Mn/2q/FRczL8164DKIF4L0gHfWXFx+hTzUo8c+zPYBBtluU3VVJuQmqUzqhFs808lM g3KIU3+LQIaauLUgFhZlpBmij/AYKvbnqmAulP/kR8LHQIU9wPrIahVNtzNYdbQpUyi+ kCjg== X-Gm-Message-State: AOJu0YzYCAajupPAWdoDE9Am73WuRHEkiOYKLdrjtYDDD1eHP9kuQijz wJzpvm6u3Z+AXpsC3N/r3SdKi2EHAv2MCXyT0g1PHR1Husu6uRHAZZ8vhtmkTO8= X-Google-Smtp-Source: AGHT+IE93wrdhvj1jPWy5zvxTPwSbhlcdosDYnf+NOlDda6ytSjhdKrpXWKVADSrOUal4AF7Aqrd8g== X-Received: by 2002:a17:90b:350a:b0:2d3:c084:9d93 with SMTP id 98e67ed59e1d1-2d5e9a2f0damr2610948a91.14.1724228421082; Wed, 21 Aug 2024 01:20:21 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:20 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 08/14] mm: copy_pte_range() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:51 +0800 Message-Id: <05c311498fc8e7e9b2143c7b5fef6dc624cfc49f.1724226076.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In copy_pte_range(), we may modify the src_pte entry after holding the src_ptl, so convert it to using pte_offset_map_maywrite_nolock(). But since we already hold the write lock of mmap_lock, there is no need to get pmdval to do pmd_same() check, just pass NULL to pmdvalp parameter. Signed-off-by: Qi Zheng --- mm/memory.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index d3378e98faf13..3016b3bf0c3b0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1083,6 +1083,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct= vm_area_struct *src_vma, struct mm_struct *src_mm =3D src_vma->vm_mm; pte_t *orig_src_pte, *orig_dst_pte; pte_t *src_pte, *dst_pte; + pmd_t pmdval; pte_t ptent; spinlock_t *src_ptl, *dst_ptl; int progress, max_nr, ret =3D 0; @@ -1108,7 +1109,8 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct= vm_area_struct *src_vma, ret =3D -ENOMEM; goto out; } - src_pte =3D pte_offset_map_nolock(src_mm, src_pmd, addr, &src_ptl); + src_pte =3D pte_offset_map_maywrite_nolock(src_mm, src_pmd, addr, NULL, + &src_ptl); if (!src_pte) { pte_unmap_unlock(dst_pte, dst_ptl); /* ret =3D=3D 0 */ --=20 2.20.1 From nobody Mon Jun 22 23:02:56 2026 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C6B291C86F2 for ; Wed, 21 Aug 2024 08:20:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228430; cv=none; b=oZFUW9qKMzY2hLUs/XJin94wA+07vE6m9JPWougIMjwPTk2u9FyHDT8ZGUq5GmUbH0O6nO95y5yND4XQCgYVHKA3zuFSHisjVXhNaJ4CBoJ2wRAsIEx8SBLkHV9EvAXxaVAj1nGasiXBdBhs/w/LV0cXpM4udVzpfRobekjo2QE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228430; c=relaxed/simple; bh=jJszyVjKZJbuadp7xF9NHpuk6YVJlOW7Oz98iizgmSw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=KPYjsTL6a4cewM5kSixvwGWKe7h4BE5UWZnjSsYMdck60Kaoot1wC7wv+M8ZhifiHuHQnmY6WPZ7ruaTc80zvocUNIctA8JcLHChaNzxtKduQaAQRDftAqJa7hJ4MMp5qSHiPEKGSfrSOCu5M7VpiQEsVZRjN9AkP+r+FU+bdgg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Od+7Pb05; arc=none smtp.client-ip=209.85.216.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Od+7Pb05" Received: by mail-pj1-f44.google.com with SMTP id 98e67ed59e1d1-2d3c5f769d6so4458448a91.3 for ; Wed, 21 Aug 2024 01:20:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228428; x=1724833228; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6wf5h3KMWWmEe6/3TeJpf55VgCdFHsYKNAkaT296AWg=; b=Od+7Pb05nM6n3TsGPEM1lc7AiZhNj/DX+/24kh4670eLmLMqiondESy2iOtbzjvcaH Wu16UTtLr0DWcPL66tslsN8fb1g5+svA19UGOfjeVGDjI5zIa1bfSTKjgEW4goEzxzZS SxgA07Q2TZjbtYjs5rFuzJw1M0gmYK7o5UwFwPNxdr9etCQnKyIyYwS5QqiL81BM+yj9 jZqO/osAG3bmOQ4GxtSXA2+JT05YupQ50G3LAzdKMyysnQPkeOJzTS6X32HHHQznxAwh kZbD9fH15jUODV1cToXbeEsG3GFObqqshh8jqPCMczw0yOIlKRVZL85mun6f5Hiq/9lE Z3RA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228428; x=1724833228; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6wf5h3KMWWmEe6/3TeJpf55VgCdFHsYKNAkaT296AWg=; b=lqpzzwKvaiVjlSdJaIo9lmk2qnRv7aev4CfRuZLA3boW27700xoq7rtNOsCHOSWNuh E2ALnxsaJrOSEKeDSigfAmrJvdrdz9o0/bfO0niBIgsBFAdFSHtAXmfojICzxSqjTTsQ y/R4ZsM6DG01h+xhTrD1cZ0K3a1Xb2PKdsQj/TvLFq9IDelaVXqLKRI+CCoSgD6d/DT4 Dk0VZYFzycxJ/EwFOdRdOdycMTW81aFj9bm8Kz2aYw0vXxIiN2W1Kq8LJtcMEbByQgND ppxQmS//oejg6VG9sqktIwTt4p9cbEEwQQly+JK3wna2iMMoHNpe+UzinURN3oJ1OEZW /44w== X-Gm-Message-State: AOJu0YxIGo4SN5dxoTx1B3b+r/2K/zWI+daRWS4PjcTvFNEpJsgxEHM5 Ae5VCHIwwVn/loXKuqjTg12a0wAE9F4bSMlSB+pHa4ZYTTq7xjksXB+xUZ/d2as= X-Google-Smtp-Source: AGHT+IGct1lfg7s+Av92JTuFI7wGr/k4q1ybIYiVEVcgmdImld0nRYlRbV5Vh9eNiEYSyV0nSbS/hA== X-Received: by 2002:a17:90a:7893:b0:2c9:635b:7271 with SMTP id 98e67ed59e1d1-2d5e9a24ea2mr1616047a91.21.1724228428100; Wed, 21 Aug 2024 01:20:28 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:27 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 09/14] mm: mremap: move_ptes() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:52 +0800 Message-Id: <89bd1d48db1986ec83dcae228defa748f38238ff.1724226076.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In move_ptes(), we may modify the new_pte after acquiring the new_ptl, so convert it to using pte_offset_map_maywrite_nolock(). But since we already hold the exclusive mmap_lock, there is no need to get pmdval to do pmd_same() check, just pass NULL to pmdvalp parameter. Signed-off-by: Qi Zheng --- mm/mremap.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/mm/mremap.c b/mm/mremap.c index e7ae140fc6409..33a0ccf79c32d 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -175,7 +175,12 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t= *old_pmd, err =3D -EAGAIN; goto out; } - new_pte =3D pte_offset_map_nolock(mm, new_pmd, new_addr, &new_ptl); + /* + * Use the maywrite version to indicate that new_pte will be modified, + * but since we hold the exclusive mmap_lock, there is no need to + * recheck pmd_same() after acquiring the new_ptl. + */ + new_pte =3D pte_offset_map_maywrite_nolock(mm, new_pmd, new_addr, NULL, &= new_ptl); if (!new_pte) { pte_unmap_unlock(old_pte, old_ptl); err =3D -EAGAIN; --=20 2.20.1 From nobody Mon Jun 22 23:02:56 2026 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8D35C1C86F2 for ; Wed, 21 Aug 2024 08:20:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228436; cv=none; b=hrMUe0yE7kuPYVJUnACRu86yyt/G1Z4R6uom4W/DWQT7Q/U5sq3ZxT78NSqkpJnTQCNZjxnam6KG9jAA/Zu1X6JjLT4VuhifXUziN5URlrL64Nzmwl2MekEva9qQ7ByFTNIlIuZMeXI02UHDCujkdzOAW9viFTZ1agDkY8xxN/g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228436; c=relaxed/simple; bh=NiiNJF6H5kOWV296o4Bpn1eahVPpz1/Giwkf5jrcY5g=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=swjUIdAJay03P5Nq8DtCMa0DXGr2/u5cW1t59/QxtZZJSj6B0M7c4Ur0MEwOaJUwHVlsJuV1aSLDuBELK0RBTAqTjXRa8rn/hpldTp8+JUkmw1iMnBuNXpmVGgYIzhpqlS6LibmGD1ADhGTzF+2LmVERmhznn+bXiakUFuukCa4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=ezwfIaLk; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="ezwfIaLk" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-1fee6435a34so42386415ad.0 for ; Wed, 21 Aug 2024 01:20:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228435; x=1724833235; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MvwQXaavnjACpbuRjCPShfvJkJjZwFPa7EtFYmq5Q9U=; b=ezwfIaLkRJoJ/s6lSflT9wI3iDSeYzfVGmfxHof6gERJLoTLwQUDuywqWP+uwvZAe2 tIJrrJ6zOCUSwa43fhTx4mU7kBOI4U9FhR4iZbhBpHeeEeqyej6jj1ZrhawOIBkMfnQm QfDptpENUpUTBFei+zm04bACbja84c7UfvxwpD9tgKmEblkmsJ9gHaoOfxw4u90QLTny 7qK0KO/9SKGCQEvXnWBzXFyM5AdQd1K1udYy/VRxY23kAY1V77+UlVFi0XXWuNAnez2y j0jLYNzhzt2mT5if/NHH3tW/5d4xF2kbtafB6Ezp8ToneSC7XLe0JKLY+SCYnzgCuDjf 1EvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228435; x=1724833235; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MvwQXaavnjACpbuRjCPShfvJkJjZwFPa7EtFYmq5Q9U=; b=CPRYFPFzF3143UuC+XRbcpP6fs8aYhFRtp1slWD9oh86rfS9AVpfuLuvYfxhrnQsD6 RS94EHqBsHiWR+qh6q3vqeWLHJd/rJroBJCoFN5Zwl2UQt9kEuVeTgsDnpL/Bvuzs5sY 4fI/TNViQ+RTOdH3zS8hG7JZD2AAaglIlu6QvITr/Nbe5e7MTLNNDrkdXAkZLKUd2IDE xIxyo7cb5YFIqoyAD9REY2vJWsCXHoFYLNKNENhJMgyWjqOCOPZt+jt6Yr2SMIhJd54+ PHo4EzBZJ2An557WoegUP8/BJJHQogbeTJrH9HdA+kIyA/PsiO75zHM2AqZSYhpPIEhD DxEQ== X-Gm-Message-State: AOJu0Ywl/inRo4thToyraC0cQ8izrZ1xZnS39LzWAyDwlYn/pWZtiT5/ mXEXc94YliImBptzAmkgT2pdNHmjXSVsk2ZkTowGSrrAqaCtG7N4AT222iVEjrI= X-Google-Smtp-Source: AGHT+IF3P/4155uMvmGYpoowuXrhy2v6aHwudYM9PJFMCLkX9j2rCBHO6+Z7X1wkaLdhxTlExnQxpQ== X-Received: by 2002:a17:90b:3b41:b0:2d3:b5ca:dedf with SMTP id 98e67ed59e1d1-2d5e9a0db5cmr1626393a91.17.1724228434854; Wed, 21 Aug 2024 01:20:34 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:34 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 10/14] mm: page_vma_mapped_walk: map_pte() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:53 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In the caller of map_pte(), we may modify the pvmw->pte after acquiring the pvmw->ptl, so convert it to using pte_offset_map_maywrite_nolock(). At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the pvmw->ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pvmw->pmd. Signed-off-by: Qi Zheng --- mm/page_vma_mapped.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index ae5cc42aa2087..da806f3a5047d 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -13,9 +13,11 @@ static inline bool not_found(struct page_vma_mapped_walk= *pvmw) return false; } =20 -static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) +static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp, + spinlock_t **ptlp) { pte_t ptent; + pmd_t pmdval; =20 if (pvmw->flags & PVMW_SYNC) { /* Use the stricter lookup */ @@ -25,6 +27,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, sp= inlock_t **ptlp) return !!pvmw->pte; } =20 +again: /* * It is important to return the ptl corresponding to pte, * in case *pvmw->pmd changes underneath us; so we need to @@ -32,10 +35,11 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, = spinlock_t **ptlp) * proceeds to loop over next ptes, and finds a match later. * Though, in most cases, page lock already protects this. */ - pvmw->pte =3D pte_offset_map_nolock(pvmw->vma->vm_mm, pvmw->pmd, - pvmw->address, ptlp); + pvmw->pte =3D pte_offset_map_maywrite_nolock(pvmw->vma->vm_mm, pvmw->pmd, + pvmw->address, &pmdval, ptlp); if (!pvmw->pte) return false; + *pmdvalp =3D pmdval; =20 ptent =3D ptep_get(pvmw->pte); =20 @@ -69,6 +73,12 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, s= pinlock_t **ptlp) } pvmw->ptl =3D *ptlp; spin_lock(pvmw->ptl); + + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pvmw->pmd)))) { + spin_unlock(pvmw->ptl); + goto again; + } + return true; } =20 @@ -278,7 +288,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *= pvmw) step_forward(pvmw, PMD_SIZE); continue; } - if (!map_pte(pvmw, &ptl)) { + if (!map_pte(pvmw, &pmde, &ptl)) { if (!pvmw->pte) goto restart; goto next_pte; @@ -307,6 +317,12 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk = *pvmw) if (!pvmw->ptl) { pvmw->ptl =3D ptl; spin_lock(pvmw->ptl); + if (unlikely(!pmd_same(pmde, pmdp_get_lockless(pvmw->pmd)))) { + pte_unmap_unlock(pvmw->pte, pvmw->ptl); + pvmw->ptl =3D NULL; + pvmw->pte =3D NULL; + goto restart; + } } goto this_pte; } while (pvmw->address < end); --=20 2.20.1 From nobody Mon Jun 22 23:02:56 2026 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 50B631C2DA1 for ; Wed, 21 Aug 2024 08:20:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228443; cv=none; b=r/JvRCnAKvKw+/Aot6Dho8YbLJznThXrEZ2mhM6gKfVWaVH2FCEI6VC821aTnmCSGcRn0G08o0h2YYHuib67WNPymfh5+TQjcLdW7MqZxUZlWjNWL79oDYLSqNOrS8HL95+LY0yxwBQtjmPxwnaizZ+TQI/ltwPkSOOD4OH08EU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228443; c=relaxed/simple; bh=jipI0HnrMJtWTaSzKSKu3sudEFw8XtUqiZMmeV6Lz9s=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=IEOZkXaoAzzaKjdu1vmo64rZ9l7k0OLDRoOQCwTpS+ZuFe5yqxT1WwvlN07sP1b1x6GMFIftnYKaiJAuuvyWZMsFd018Jsuwyj3f3e1j+HVa1xrXsOjbI3lzny0/Hz2cJZNDoHZTsilJx14Vi4cHIy4RnClzeG6YQARheipla28= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=bpKlZ24o; arc=none smtp.client-ip=209.85.210.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="bpKlZ24o" Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-70cec4aa1e4so4316349b3a.1 for ; Wed, 21 Aug 2024 01:20:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228442; x=1724833242; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jOO3xScatAPPPb8tw2FfV0S/NtjYDESd+MYV0EHgGGs=; b=bpKlZ24oh+WPmHDChiWIFK1I46hMjN0WLjvPBs34Rmfr+eojdhxx7Eg78iyYfE+jDk ZCHcE24Ws7HAvAbEUl5SCJL4m78BJ0DuXF5Ya62DNg0YwIVhsze7O7+n+rjWgv4xCw2i +icApaxLoIb/LzWTMA/aLrhf+MPeVF6O1j5RqEqwgFFDY9wvvDds646eyKUJDP/1KUbk lqUDNq8R5nIEOiIuivBpUe4Hw4P+6e7tLG41erQ1TNEwmCB+wtp1R6Xgh6Gi0cHPC9pE 0xTX3MZlxXs0prm/r6Aab6qrbPsdubCF/XiY6D7RK8QX3H0RnWn+ynK2WWWecu4MPBUZ zIcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228442; x=1724833242; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jOO3xScatAPPPb8tw2FfV0S/NtjYDESd+MYV0EHgGGs=; b=JVr6euu+cY08qdVr7XNslkSpUb5d9Y+OL2v+CTd0bxNyJCZCDBo/LpfJkFJDGkKrj8 0fXxXy6/ECBtXSFaT7oj2pqLHqzAuf+kmn+88To0Orawn6As9LI6szF19R4JFSo+mezd cMdb4p5nMNNu4SNXhTCzh2kS1yilsX5YB1a2ObXaqC3X+W6dblmHrfZ8BkLGRhKsXUoy A9jM8+dPBp6slJE+5ZczsR6cECv+UAUIKCY8jUAmbLCRXLu5AgUg7LvaLsDw/NoI9GCX Y5WbOREr/qPfShP/AndfavWoUHfTfpYdT9Y81JRpEnxjmSyxjj3rJu4jvNmsAZMQYWkf N2sw== X-Gm-Message-State: AOJu0YwKYZJzU14U2cUefPJvD+4vVBgdLH97J8VtTCJOo/F2FEdB9kkw 6oKefovJohkJCwFwCBroTcOZ/SwzIuUzfPDJFH6w4esXH7hmnFvsiwb6THJeYYU= X-Google-Smtp-Source: AGHT+IFM+x1tDjPFZswDxs94u/u9nyN6BfWsdj/MLtJd+fBRDvg5mEN2O6omshjTkNyUCHM1AQzwKQ== X-Received: by 2002:a05:6a20:e605:b0:1c3:ce0f:bfb7 with SMTP id adf61e73a8af0-1cada055937mr1798278637.21.1724228441599; Wed, 21 Aug 2024 01:20:41 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:41 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 11/14] mm: userfaultfd: move_pages_pte() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:54 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In move_pages_pte(), we may modify the dst_pte and src_pte after acquiring the ptl, so convert it to using pte_offset_map_maywrite_nolock(). But since we already do the pte_same() check, there is no need to get pmdval to do pmd_same() check, just pass NULL to pmdvalp parameter. Signed-off-by: Qi Zheng --- mm/userfaultfd.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 6ef42d9cd482e..310289fad2b32 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -1146,7 +1146,13 @@ static int move_pages_pte(struct mm_struct *mm, pmd_= t *dst_pmd, pmd_t *src_pmd, src_addr, src_addr + PAGE_SIZE); mmu_notifier_invalidate_range_start(&range); retry: - dst_pte =3D pte_offset_map_nolock(mm, dst_pmd, dst_addr, &dst_ptl); + /* + * Use the maywrite version to indicate that dst_pte will be modified, + * but since we will use pte_same() to detect the change of the pte + * entry, there is no need to get pmdval. + */ + dst_pte =3D pte_offset_map_maywrite_nolock(mm, dst_pmd, dst_addr, NULL, + &dst_ptl); =20 /* Retry if a huge pmd materialized from under us */ if (unlikely(!dst_pte)) { @@ -1154,7 +1160,9 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t= *dst_pmd, pmd_t *src_pmd, goto out; } =20 - src_pte =3D pte_offset_map_nolock(mm, src_pmd, src_addr, &src_ptl); + /* same as dst_pte */ + src_pte =3D pte_offset_map_maywrite_nolock(mm, src_pmd, src_addr, NULL, + &src_ptl); =20 /* * We held the mmap_lock for reading so MADV_DONTNEED --=20 2.20.1 From nobody Mon Jun 22 23:02:56 2026 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B65FD1CB131 for ; Wed, 21 Aug 2024 08:20:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228450; cv=none; b=N5eBFuYxzGqA82Rh6l8a7TSontZUXC7c0o6xy6qk8cDPUfa9pWSeQROeOKludu9ko1YqmYM3dqKrGl1Ei/a1i7K9iPpyVv/4K3Hkjba6JWAc1St1hREmmZu2H39EZTl1HM//PsWG2HvGXoGGn5oXyi4FdwutSgX2WS6/xBZLyts= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228450; c=relaxed/simple; bh=0/2LlQVYxaSpsSprlESW/M/nT1cUvsLBdrUAt0LwVUc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=avcKx/jHKDzQmrxnm6VNGm0qeovuvr8gfgwU/iKaTkIVni8uRD9urQYNBY8xPZXn9wBiX2CsaG+jEmO1Ha1pyaoe8Wpl62mQw4N1CJU1IJ96kxPpqh1XxNP8VLSPGZZi38q9+BGY7WodmQ0GnVBSJJEANhim7QTur5zCKjxClv8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=gL1bZxkt; arc=none smtp.client-ip=209.85.216.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="gL1bZxkt" Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2d3b595c18dso390334a91.0 for ; Wed, 21 Aug 2024 01:20:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228448; x=1724833248; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=y8nsSUZs0Rgdiv42AW7Io4vKbgKEW01jAKVQTF6mEcw=; b=gL1bZxktTGgfcGmanWbgIkqxRlEVivxE7ssa9jAo7Oyd4vKLPaTtgSDzPsm60REEhE PuFUNzy3da++Ty/EPo7H/uY+0BAvNWdrzj5q9sQnbgb+fizwb/qn6pGDndIWhrJTD7Zn BlGsAzIc4Qz4NN7AxOdJkiZ3bEczTIlzvWdU9fBqRAlhIjsFnOJ2QRY1daBO6lCtmQgV IqRaAL9kvQUeFnhDgTcHRT9Q1p4gLCH8BbIMWEqkIPhbfAKwWo3MLFcUxGD8sFHMaaYY UOG6ba8avnSqeZdvu3w/iQatQQagUFU3iYbwS++OgwZKyuVRpepAVO77dOsSum2Tm8PE imBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228448; x=1724833248; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=y8nsSUZs0Rgdiv42AW7Io4vKbgKEW01jAKVQTF6mEcw=; b=ZiBsUrHuatucsrlbsMIZmTTBHWU4WQQABlwHODOj7xFMTpLklFkxOTiE2Vv7KYp4NC 577dIQSEIDcgfKTb0TRdzOP2B0y0xS6sRljpBdk5NINinTjZdpMA920i1KTjsPbGTbx3 8aP7/ClcImn4Zbzv9aL9jf/kF2bUwwhTQzIK6qwi1937ZowztKAl4f0+29WfFJB5/rOC XzGWSQXXjDHwdJjKb/njQhczReUtipwJYbLhXwkmo4geVHmLZUFqEHyZtiY5pZd4B/8+ r8I4RIif1iaMcEg1o1PK+6XW3HCkr9y2tI2RAXWYoR5JHdRUYMLf/OQWUPiD32Kge7Kr YODg== X-Gm-Message-State: AOJu0YylnJN71pz6o0a83+pKhFZZjVv4APM7EGldjOF6dt3jkFH9LhO6 3Ah9SJATvwp6kdjsUQkJdjCZUTy2DdoATnw2Uo0fE04sNKWDn2IfHAA1YvTseEE= X-Google-Smtp-Source: AGHT+IFBw9G+gRGiVw886HN5nxLIybNKSDx7re7KNY7nJCh1bBtNFP6GTlUB4MiVeFJVy0IWyCdYMw== X-Received: by 2002:a17:90a:d987:b0:2cf:fd50:a2d8 with SMTP id 98e67ed59e1d1-2d5e997c3demr2593063a91.7.1724228447873; Wed, 21 Aug 2024 01:20:47 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:47 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 12/14] mm: multi-gen LRU: walk_pte_range() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:55 +0800 Message-Id: <1bdb1ec4a30acf6b3a376f746c96236328904768.1724226076.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In walk_pte_range(), we may modify the pte entry after holding the ptl, so convert it to using pte_offset_map_maywrite_nolock(). At this time, the write lock of mmap_lock is not held, and the pte_same() check is not performed after the ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pmd entry. Signed-off-by: Qi Zheng --- mm/vmscan.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 5dc96a843466a..a6620211f138a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3396,8 +3396,10 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long= start, unsigned long end, struct pglist_data *pgdat =3D lruvec_pgdat(walk->lruvec); DEFINE_MAX_SEQ(walk->lruvec); int old_gen, new_gen =3D lru_gen_from_seq(max_seq); + pmd_t pmdval; =20 - pte =3D pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK, &ptl); + pte =3D pte_offset_map_maywrite_nolock(args->mm, pmd, start & PMD_MASK, + &pmdval, &ptl); if (!pte) return false; if (!spin_trylock(ptl)) { @@ -3405,6 +3407,11 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long= start, unsigned long end, return false; } =20 + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + return false; + } + arch_enter_lazy_mmu_mode(); restart: for (i =3D pte_index(start), addr =3D start; addr !=3D end; i++, addr += =3D PAGE_SIZE) { --=20 2.20.1 From nobody Mon Jun 22 23:02:56 2026 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8414E1CB14C for ; Wed, 21 Aug 2024 08:20:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228456; cv=none; b=AfDMi9lbhxR3ftKHhHS4w8jj6uV7pTPHj8CUS1/yxoslBLla40CQb9kFGDt46hBe6Ri1z8VXDdkdH9ASJv8CYxScK10gY1zIhk83p5Z3tkdBFtY8mMgAo7H4O3p+DtpXdig464eBHHJanypxz57mwKsNokyBKJdqsOgUxBtYQ9M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228456; c=relaxed/simple; bh=CfjhG+zBfgBTHbDX5rndwLBFWLKwheXgYDb9hI3tnlg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=eYCED+gyNkTrwuqDLf/3fUZOosYly0fwFUWOpf8cLanD+COngiAQB+pYyAoH4LCRTMW2hmLBobVD9ANdrRm7tt/WRVFbCjKNxQnrTkNd/e3rm8zzSc3AvMhQVT+an/QvS8ESt8JBwYL1pVg6egcg3Tk0iOs0UoFxw18LzAv/22Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=WLYzURG+; arc=none smtp.client-ip=209.85.215.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="WLYzURG+" Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-7c6b4222fe3so3875136a12.3 for ; Wed, 21 Aug 2024 01:20:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228455; x=1724833255; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FpkIwk88smkdA0CmKwnRaaffLUYg3/8r+GLLHrp0y28=; b=WLYzURG+sGHUmP+vlsBG8pccYIjUPOfMq3Vj++fCmYrlb7drXFOZ6L3qQsE0nQ5Fbq xW5SWudCVc2AacP2ndziK9FWhbJ7MYOlSd1zMXQIEFRfmTKw/RSP3TlJxl4ULrDZLMf/ GGChNvKcF4MVLh4QFAAZbnmm1LgWbS7hN0TBRKmGWiBf5Cq7LWXMy8ce8cPtKuuLbMFJ GclYyqMveE0+CKrUS83nskEpCYoQB+ZeqGg5kqqu8GfKm0jRerxBot2gSEF3enLR8iPD FhlIy82ALDRntOUFxS4MifL5obVOVPmkOeNMoThndE97/FqQxUqub1zPZIDruo79/Mje 7eWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228455; x=1724833255; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FpkIwk88smkdA0CmKwnRaaffLUYg3/8r+GLLHrp0y28=; b=q7ZY59z1GM14wPAKB4vxIiH04fZRGW7zbJ6jIgf3gpNG8QSLghfRo9rCkJFOykNh4o 3uY/oUdgxZm40vgksfCkayv7/YuR4IaO+VbQ0E7vkMMHHnn2kFgXrjnPM9kL8ZZQoZD+ 2xC9shC3d9Ew8DW9RpnwkI8KwzvM92d8/6VD0kdO6PtwoEJy+coGmMqS7CPwBlBAS1tN Ghd6KlPInfsob7lZ14KLwttp0kIDc0wK7aBcvHq1khnN7zW5SpdrpaUFaEAboGXxqI7a QY9k/NW5QzqP4Jw3XlkQ4rGf1KKFRm8vhDyVhctwxcnsUbkAVqqE2cRPh3zvdGZsEuuX Gs2Q== X-Gm-Message-State: AOJu0YxGPRWs/wX2poZ2xlMT8riYV6rRyIQEHVs+1994B6c2c6WLTdvU AooASh0TNSIlT3ymZEI4kovntEJBJhYOMwwgavZHaHkZGs+NpIgbvwMEM8GiMck= X-Google-Smtp-Source: AGHT+IGrVkekNFESU70hIpUuBMPasbPeBaSvK0LFz7NLCJlpTpTqHUrWHKc0kKTsT//aSMihe2hbaA== X-Received: by 2002:a05:6a20:ce47:b0:1c4:a49b:403 with SMTP id adf61e73a8af0-1cad81a4248mr2186429637.46.1724228454750; Wed, 21 Aug 2024 01:20:54 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:20:54 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 13/14] mm: pgtable: remove pte_offset_map_nolock() Date: Wed, 21 Aug 2024 16:18:56 +0800 Message-Id: <129f11e3434c5af898096c1cf3fe378f50bc94b3.1724226076.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now no users are using the pte_offset_map_nolock(), remove it. Signed-off-by: Qi Zheng --- Documentation/mm/split_page_table_lock.rst | 3 --- include/linux/mm.h | 2 -- mm/pgtable-generic.c | 21 --------------------- 3 files changed, 26 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/= split_page_table_lock.rst index f54f717ae8bdf..596b425fb28e6 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -16,9 +16,6 @@ There are helpers to lock/unlock a table and other access= or functions: - pte_offset_map_lock() maps PTE and takes PTE table lock, returns pointer to PTE with pointer to its PTE table lock, or returns NULL if no PTE table; - - pte_offset_map_nolock() - maps PTE, returns pointer to PTE with pointer to its PTE table - lock (not taken), or returns NULL if no PTE table; - pte_offset_map_readonly_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; diff --git a/include/linux/mm.h b/include/linux/mm.h index 1fe0ceabcaf39..f7c207c3ab701 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2952,8 +2952,6 @@ static inline pte_t *pte_offset_map_lock(struct mm_st= ruct *mm, pmd_t *pmd, return pte; } =20 -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_readonly_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_maywrite_nolock(struct mm_struct *mm, pmd_t *pmd, diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index 29d1fd6fd2963..9ba2e423bdb41 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -305,18 +305,6 @@ pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr= , pmd_t *pmdvalp) return NULL; } =20 -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp) -{ - pmd_t pmdval; - pte_t *pte; - - pte =3D __pte_offset_map(pmd, addr, &pmdval); - if (likely(pte)) - *ptlp =3D pte_lockptr(mm, &pmdval); - return pte; -} - pte_t *pte_offset_map_readonly_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp) { @@ -374,15 +362,6 @@ pte_t *pte_offset_map_maywrite_nolock(struct mm_struct= *mm, pmd_t *pmd, * and disconnected table. Until pte_unmap(pte) unmaps and rcu_read_unloc= k()s * afterwards. * - * pte_offset_map_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_m= ap(); - * but when successful, it also outputs a pointer to the spinlock in ptlp = - as - * pte_offset_map_lock() does, but in this case without locking it. This = helps - * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that = time - * act on a changed *pmd: pte_offset_map_nolock() provides the correct spi= nlock - * pointer for the page table that it returns. In principle, the caller s= hould - * recheck *pmd once the lock is taken; in practice, no callsite needs tha= t - - * either the mmap_lock for write, or pte_same() check on contents, is eno= ugh. - * * pte_offset_map_readonly_nolock(mm, pmd, addr, ptlp), above, is like * pte_offset_map(); but when successful, it also outputs a pointer to the * spinlock in ptlp - as pte_offset_map_lock() does, but in this case with= out --=20 2.20.1 From nobody Mon Jun 22 23:02:56 2026 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C5A031B1D6E for ; Wed, 21 Aug 2024 08:21:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228463; cv=none; b=XvJV9SKqbkOZlgcFkk6Qwsq1shHNZcxhUE7B+lWh//ndKWNodUqNK4YVfaw97PwRp4b9oLt/WDdycnJaLvMb+I2un1Z1/ywFqEKX7wZMEoV/iy1Wl4+mVswoFwGjyf3FNNfKwB1A8j9+s5VonQTgDtjznofmLbbgZ3Hc6OlnLU8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724228463; c=relaxed/simple; bh=rHqDHFrF26/hCS+KcrGpklG8aSJcd2dafPbIgZSe4as=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=lAmLVVbcVg3U1WPhXQ1x79LPRGk1SC2EJMNW6tr9NG8VijNAQCjXdMy7w8e6c+Ut7rB4HC9HrJkxyk5nBFvl5a7qzIrOUE8knbVfQMietLTq1s1dFvkKwgC/8cicHA/nSJ1ndBoZ7qzideCBHm0SGumDH4wo2WzPZyWJ+N5pcZw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=FFtR0xWj; arc=none smtp.client-ip=209.85.216.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="FFtR0xWj" Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-2d1daa2577bso4879454a91.2 for ; Wed, 21 Aug 2024 01:21:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1724228461; x=1724833261; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9nMMKQBovMbpHaGn8Lvg/undANxxFw84odzUdkp5IAc=; b=FFtR0xWjW0Ff9PeIQ1vFAx8Xjor5qGWkGdw8ur89bD5vRjYHB/jHnTHHWO8GPG5TLn rX6X5Konib859FnurKYzsteqFM3hbkCJIXzq9ElNMzsCBMNhpXu/LDZcu9zCGOHsizT5 X//YL+mEs1patQZIz+/njGORUSLTe23tlqOzzLtFapYUqQ/hT2vJXnd6bydUPzpIDyTb yb/XceEvIRRaXrrBTAumavxQ8PGROiwMUCqfOLa+S75G//fXR9h8U5xZl3pouJh6jeMg jAFgDHNLxFwXe5NkwUc4W3B1/XRUDCHWmKPZ5ZwMJCzaXkBS2O9uDcV385WhoDVXxGvT 3dpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724228461; x=1724833261; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9nMMKQBovMbpHaGn8Lvg/undANxxFw84odzUdkp5IAc=; b=FUa8FN3en7r3LpHuRW8D27/e+NLTINNeZoVGt7Phl7FF1tcVq67jVO0nB02yUUqXlb Le6Pfdnd/IcymXQnoS+w+G1QLWS8/yCsM7PGU49DAT2YsZ/kJkR6dhCxpKPprt36UGak m87VPN887vcgZP8iWs2i13e6qwd224BQsFnpSVJRkMnpyExQqwV77OucIFiUO9vTo+l5 mEG776N7qyP58qb1bavozpacJnHDyo1GJg/OppdG3LD1vpYkQUNqB7Xqu+HueX5YzCEK ZT0Tq0n4K/1SPhAfyeUtdOKfrRqA6MwJgBd6RZaEVW8f9ihUczenMlmDDRHKvUM1tep3 pw0A== X-Gm-Message-State: AOJu0YyhgNzd0u6YQnF1wLie1iVt/gYykz8Thu5tvf0URR/E9AbavzH7 6UjiIBosPgVEFjPayF2xznuwiMyyx/I+ld5AvgZ8PRQeOKicCq34YVQO8/TEBdYG9f2fVaLwO0o W X-Google-Smtp-Source: AGHT+IGVCW7LuSOfGhGGSqonUfZfQEf1Y51LpMxCXHDPBrQX3ltYUdmnoycYhvy3ErzvacrcFVYd1w== X-Received: by 2002:a17:90b:3851:b0:2d3:d239:24c2 with SMTP id 98e67ed59e1d1-2d5e9b99f57mr1469884a91.19.1724228461120; Wed, 21 Aug 2024 01:21:01 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2d5eb9049b0sm1091453a91.17.2024.08.21.01.20.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Aug 2024 01:21:00 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH 14/14] mm: khugepaged: retract_page_tables() use pte_offset_map_maywrite_nolock() Date: Wed, 21 Aug 2024 16:18:57 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In retract_page_tables(), we may modify the pmd entry after acquiring the pml and ptl, so we should also check whether the pmd entry is stable. Using pte_offset_map_maywrite_nolock() + pmd_same() to do it. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 8fcad0b368a08..821c840b5b593 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1721,6 +1721,7 @@ static void retract_page_tables(struct address_space = *mapping, pgoff_t pgoff) spinlock_t *pml; spinlock_t *ptl; bool skipped_uffd =3D false; + pte_t *pte; =20 /* * Check vma->anon_vma to exclude MAP_PRIVATE mappings that @@ -1756,11 +1757,25 @@ static void retract_page_tables(struct address_spac= e *mapping, pgoff_t pgoff) addr, addr + HPAGE_PMD_SIZE); mmu_notifier_invalidate_range_start(&range); =20 + pte =3D pte_offset_map_maywrite_nolock(mm, pmd, addr, &pgt_pmd, &ptl); + if (!pte) { + mmu_notifier_invalidate_range_end(&range); + continue; + } + pml =3D pmd_lock(mm, pmd); - ptl =3D pte_lockptr(mm, pmd); if (ptl !=3D pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); =20 + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + if (ptl !=3D pml) + spin_unlock(pml); + mmu_notifier_invalidate_range_end(&range); + continue; + } + pte_unmap(pte); + /* * Huge page lock is still held, so normally the page table * must remain empty; and we have already skipped anon_vma --=20 2.20.1