From nobody Fri Dec 19 15:19:36 2025 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9E941194C82 for ; Wed, 4 Sep 2024 08:40:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439252; cv=none; b=dihVyPXiMjtvWfrJGLiH9FsrzSdtSs23OmyDGzJWmzpymD+H/XODtEHxC/CwcFg6ulzyXSMJpY9yhXqORF3v8LkPTGB8Me6T56xUZZGoDRhKi5SMy9A3/kTGfZwYjyeb7AwKrpOEtbboCMN9o72U6n5BuipMuQugJU/9Jdq5MLE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439252; c=relaxed/simple; bh=YbZWhpQA/x/lVzUxYznnJcxx68ktxTGhLzD3CkihiAo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=khx/uaSejdZFe8o1WdrKwIUXhSaLe2apOhSHwzCA9TipLv0WWsf0i5IiCslG8g69MtIg9XFZPRbbNJ3qEYO4v3IwDgD9QEpb+JoSblCmGrR3R03yAA+sY6o24do/4+WdTQTL6GM8FSxzo5Xtpcm4V9KdH/t5HFI6OZcPO1DMisM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Vyq+khq/; arc=none smtp.client-ip=209.85.214.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Vyq+khq/" Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-2055136b612so36608785ad.0 for ; Wed, 04 Sep 2024 01:40:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439250; x=1726044050; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3ZoTvF0msswRhisxHHwhwQPLPrn7vj0AmYqzYo7APLw=; b=Vyq+khq/MO/TofvbEDRsrARtstiBN7AQwZzX+u6yWdtBz3hnPvPf38o/GBEWHlri3W KTzi6fS/xSHGYXkW54RKVPrMDJOslC3N1cAM4utSg5jMhypcr/DVy96rfguIiIehwL7O OSoMXZs68b89lEeCzrCpZU6PbT6b9Eu3oDioENbri8opwH+kqhwBx9Vh7G3Ve1V/7qIq XBo3hcqD7DjN1NgOjYrSY/K/uILkz49wBlXRbCk0YSSKpHIQxKj0NlKFqJ6tA5aHg2hN pJ0a1YI7y04mW/5O5AYCIFt2TKvtq+DzIOQWFTW2qGkW7WeHikH61iDE+1p3G0OK94nb xu1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439250; x=1726044050; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3ZoTvF0msswRhisxHHwhwQPLPrn7vj0AmYqzYo7APLw=; b=o65ZITafn/sRK0QE4DqshFuCV0L7PchCO1k/LNlZvGE/TfE2xmiW+YnN4MqA+wAqNK ryyTcfk27CD6k2uT1IELorV29aQdQYp8e4JNk/BoFIZzgEK+wIO3qqIwaEEFoAwQp1iz hRRYPJFhkG3iTp2hmMsgB16RWlet8iBcfICpD6/vJgw/qejnJjPvlSuXS1VAlDdobhUw sUPnKnSOTH6cNgVIaurTI7NFCr6Se/AlHpkiPMeLM8CSS1ecT3k3WzqYcMSS5TCqlbVn ePcaT7U9b57PIk4xp7esG4v30CxmoGyTzs4+vbwQDPaq8eXNxrdIuAk5v7Ed1O+7kswo WN5Q== X-Gm-Message-State: AOJu0YwoNYH8SJZwwSAncTjS0eXfqMhpRojSW7cQsxkQUv1Fnpc2Knsp LfGrC87nmlkoUnRmYeFcIQ0EWq9FJzCQUh7FrTQ4EPg6XvUJ7LLaxRVnk/g0PRU= X-Google-Smtp-Source: AGHT+IHLu4af/0RdmwIJrtep3I3PIUPy/4FB7EG8zsCDexMsa/uyvNrQYTXwy/3KiLtDhw4LWBklXg== X-Received: by 2002:a17:902:ce0b:b0:202:3dcf:8c38 with SMTP id d9443c01a7336-20546b35d3fmr190359795ad.44.1725439249775; Wed, 04 Sep 2024 01:40:49 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.40.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:40:49 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 01/14] mm: pgtable: introduce pte_offset_map_{ro|rw}_nolock() Date: Wed, 4 Sep 2024 16:40:09 +0800 Message-Id: <20240904084022.32728-2-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, the usage of pte_offset_map_nolock() can be divided into the following two cases: 1) After acquiring PTL, only read-only operations are performed on the PTE page. In this case, the RCU lock in pte_offset_map_nolock() will ensure that the PTE page will not be freed, and there is no need to worry about whether the pmd entry is modified. 2) After acquiring PTL, the pte or pmd entries may be modified. At this time, we need to ensure that the pmd entry has not been modified concurrently. To more clearing distinguish between these two cases, this commit introduces two new helper functions to replace pte_offset_map_nolock(). For 1), just rename it to pte_offset_map_ro_nolock(). For 2), in addition to changing the name to pte_offset_map_rw_nolock(), it also outputs the pmdval when successful. It is applicable for may-write cases where any modification operations to the page table may happen after the corresponding spinlock is held afterwards. But the users should make sure the page table is stable like checking pte_same() or checking pmd_same() by using the output pmdval before performing the write operations. Note: "RO" / "RW" expresses the intended semantics, not that the *kmap* will be read-only/read-write protected. Subsequent commits will convert pte_offset_map_nolock() into the above two functions one by one, and finally completely delete it. Signed-off-by: Qi Zheng --- Documentation/mm/split_page_table_lock.rst | 7 +++ include/linux/mm.h | 5 +++ mm/pgtable-generic.c | 50 ++++++++++++++++++++++ 3 files changed, 62 insertions(+) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/= split_page_table_lock.rst index e4f6972eb6c04..08d0e706a32db 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -19,6 +19,13 @@ There are helpers to lock/unlock a table and other acces= sor functions: - pte_offset_map_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_ro_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_rw_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken) and the value of its pmd entry, or returns NULL + if no PTE table; - pte_offset_map() maps PTE, returns pointer to PTE, or returns NULL if no PTE table; - pte_unmap() diff --git a/include/linux/mm.h b/include/linux/mm.h index a7c74a840249a..1fde9242231c9 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3006,6 +3006,11 @@ static inline pte_t *pte_offset_map_lock(struct mm_s= truct *mm, pmd_t *pmd, =20 pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp); =20 #define pte_unmap_unlock(pte, ptl) do { \ spin_unlock(ptl); \ diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index a78a4adf711ac..262b7065a5a2e 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -317,6 +317,33 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd= _t *pmd, return pte; } =20 +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + pte =3D __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp =3D pte_lockptr(mm, &pmdval); + return pte; +} + +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + VM_WARN_ON_ONCE(!pmdvalp); + pte =3D __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp =3D pte_lockptr(mm, &pmdval); + *pmdvalp =3D pmdval; + return pte; +} + /* * pte_offset_map_lock(mm, pmd, addr, ptlp), and its internal implementati= on * __pte_offset_map_lock() below, is usually called with the pmd pointer f= or @@ -356,6 +383,29 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd= _t *pmd, * recheck *pmd once the lock is taken; in practice, no callsite needs tha= t - * either the mmap_lock for write, or pte_same() check on contents, is eno= ugh. * + * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like pte_offse= t_map(); + * but when successful, it also outputs a pointer to the spinlock in ptlp = - as + * pte_offset_map_lock() does, but in this case without locking it. This = helps + * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that = time + * act on a changed *pmd: pte_offset_map_ro_nolock() provides the correct = spinlock + * pointer for the page table that it returns. Even after grabbing the spi= nlock, + * we might be looking either at a page table that is still mapped or one = that + * was unmapped and is about to get freed. But for R/O access this is suff= icient. + * So it is only applicable for read-only cases where any modification ope= rations + * to the page table are not allowed even if the corresponding spinlock is= held + * afterwards. + * + * pte_offset_map_rw_nolock(mm, pmd, addr, pmdvalp, ptlp), above, is like + * pte_offset_map_ro_nolock(); but when successful, it also outputs the pd= mval. + * It is applicable for may-write cases where any modification operations = to the + * page table may happen after the corresponding spinlock is held afterwar= ds. + * But the users should make sure the page table is stable like checking p= te_same() + * or checking pmd_same() by using the output pmdval before performing the= write + * operations. + * + * Note: "RO" / "RW" expresses the intended semantics, not that the *kmap*= will + * be read-only/read-write protected. + * * Note that free_pgtables(), used after unmapping detached vmas, or when * exiting the whole mm, does not take page table lock before freeing a pa= ge * table, and may not use RCU at all: "outsiders" like khugepaged should a= void --=20 2.20.1 From nobody Fri Dec 19 15:19:36 2025 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 42574194C82 for ; Wed, 4 Sep 2024 08:40:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439258; cv=none; b=TV6Zu6iPfilqSHVMOtngpr20XZWlR8sMzS1FX7w0mHWXcQ0Fy2Diig+LX/AVnXIsFZo2YRPJBBrtHDHU3k+pyEwECimqnqIQeisCAH/9K76n5I3NUb9BTSskKMaC0ZxxSUWZGMBemB+iq7Tw4uh5Hu6IV644LTKdaCgxw3pRzUQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439258; c=relaxed/simple; bh=c/yYCNxp5OWZX71e2ENivdxcb3XkxtbTNr4k4b77EWA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=U7KUCvg0HqwL47oyhbn4HcTO3R2oQhmoKupZWE1UmVwyxo/8s/xkIlmsb+Y3bFCgCuXfj0NHR/4QqFhX6V33oTj4fSuyUlJktBt4RDr77Eux/a1Tn8AoFyYUXQFykqwKfl+dyk8dDShUN23karmZv2dehA7K2XQcwPO3mh+99BU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=TU75VRcD; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="TU75VRcD" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-206aee4073cso6770375ad.1 for ; Wed, 04 Sep 2024 01:40:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439256; x=1726044056; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H/kyF9mn9oESktxWYdr3s8d2VJ5iyYEZZY3E5BXrvv8=; b=TU75VRcD3UjtW+11f2MplaOTrUw+snGqTPk2JncPd1FVYH6k4hKkf5aKmr8XCA9PVv Mn8/Xnaft27VKfWMfDM4ndGiYRElvjm06/ctbqsz/SNJ26u2G8wK420Xenr8nWVZUi0d 0sg0ACmOYbI9O4Wn9BD/4lxJmkfGxl7I9cVl9ZixfVQiiRfjNi+LWNaEmnDQYuKWh0kg HjzN08p1u7hMpbY5KUZ6JIJRe4J6MLXKpIXAVFeQSygewMB01MAl7GlPRnXjRoOvCl7N wKhqmOrK0H5Q13c1uXqMyzadrR0Sy2aLNBH7Kij7bMmu0/rZPeA4HKJaGz4yCo1k7fGI 60Tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439256; x=1726044056; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=H/kyF9mn9oESktxWYdr3s8d2VJ5iyYEZZY3E5BXrvv8=; b=C4kXCFmH70smSCO92nmbbo1ncSAtPNZOLOmq8A8ZV78hsWkmZ2Qlqjquj47e7LgGHx DV4dbaaLkecgBvWnqqJkibAY/bVQYGxK73OEF+TJN05GKKUvgr9gu5Rp2OHG+an/aikp Jps4kEDtyIfyLZa/xpNv7klKvS9GKm0mgTXrhC7MiXAcguj4sVAY3nEW0pz25RCqVx6w LSU1PkLZ8smpScWGoffqdJAvterD0k4UV/EAtq5anfDYTx+eOzlye2igus62v7hMEFrd gS+M0eIfaeI/njyDIF9a7XyCcGrLEIt6DWypT8Py44G03rHeEdOTtcLb/ybIeCH/ERuD cwdA== X-Gm-Message-State: AOJu0YywLp7O1gVLj59Nm3xYFPnClD8sykq+qgRXmsdAW1Q9jpn6j6Il BGPKuIENp43jDa6tY6ayr+M0qUGW8q5vi3V0Ia82K/mLFf0o3jBQf3zm6UJyzyQ= X-Google-Smtp-Source: AGHT+IEEdhBdCGQBBk6sF5V8bi+PhpxKTNotIZgwSj7kMgQxLZV/04loActQGvyTL5xDqZZl1Kbj6A== X-Received: by 2002:a17:902:f547:b0:206:b618:1d7f with SMTP id d9443c01a7336-206b61824c9mr14802495ad.37.1725439256381; Wed, 04 Sep 2024 01:40:56 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.40.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:40:56 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 02/14] arm: adjust_pte() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:10 +0800 Message-Id: <20240904084022.32728-3-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In do_adjust_pte(), we may modify the pte entry. The corresponding pmd entry may have been modified concurrently. Therefore, in order to ensure the stability if pmd entry, use pte_offset_map_rw_nolock() to replace pte_offset_map_nolock(), and do pmd_same() check after holding the PTL. Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- arch/arm/mm/fault-armv.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c index 831793cd6ff94..de6c7d8a2ddfc 100644 --- a/arch/arm/mm/fault-armv.c +++ b/arch/arm/mm/fault-armv.c @@ -94,6 +94,7 @@ static int adjust_pte(struct vm_area_struct *vma, unsigne= d long address, pud_t *pud; pmd_t *pmd; pte_t *pte; + pmd_t pmdval; int ret; =20 pgd =3D pgd_offset(vma->vm_mm, address); @@ -112,16 +113,22 @@ static int adjust_pte(struct vm_area_struct *vma, uns= igned long address, if (pmd_none_or_clear_bad(pmd)) return 0; =20 +again: /* * This is called while another page table is mapped, so we * must use the nested version. This also means we need to * open-code the spin-locking. */ - pte =3D pte_offset_map_nolock(vma->vm_mm, pmd, address, &ptl); + pte =3D pte_offset_map_rw_nolock(vma->vm_mm, pmd, address, &pmdval, &ptl); if (!pte) return 0; =20 do_pte_lock(ptl); + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + do_pte_unlock(ptl); + pte_unmap(pte); + goto again; + } =20 ret =3D do_adjust_pte(vma, address, pfn, pte); =20 --=20 2.20.1 From nobody Fri Dec 19 15:19:36 2025 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 09F381CF5C3 for ; Wed, 4 Sep 2024 08:41:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439265; cv=none; b=sKxypGBIDJlg7co0VUjZnRIkEnwwocK5uJ900tH73rf7oa7jTTWRx5vQtahtJHBLTlazMiV+OyL7gwXA3+YKjk08iMHSpWP7UANyws5X4r2DiulAp+5Jlq180qJyCGVSPmDAHE3RfUo24wUy8Ws7qogz3/IZKWCpvYF+G+y5gyI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439265; c=relaxed/simple; bh=Ks+ctZjoe6oexIKNJorWHsj/FHxBIY1SArqgBxdFKSU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=FDM76jLAm0IAsHtijjo/ocpXjuglqqNXLkCCVE9e6kn8ZJq8qw+qiFclQCGvtxzeBr+MRMmpq8Lux94yjTjY4S93WAH3ulfechSsl59wFl1q+Y/EWGirQ4vZVAXP3ToOpHx+JWpHiypglut+l14kwgGU3oAzCn3Ah1zjYZk4L0w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=V2hc6NaB; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="V2hc6NaB" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-20573eb852aso3452365ad.1 for ; Wed, 04 Sep 2024 01:41:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439263; x=1726044063; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7WBwKEY9MWquOlo01BPdpzLz4TZdRIsQgPATlcDwMyk=; b=V2hc6NaBrZkWeQ0WQyvpl2gJ76mcNH2gR+15KXmAYJb8WXMaw3FCdaJL45jSV5xX/I No/yREwGCzNI6qRIFgnANtmslsaBBNq4jx2COw2gC+edRn2c56mdZuZZmI2qAsrXeN20 vtJfCNRAePt9G9d992zvuzNXdGyfW5DkTGRt56ARDuAUValARVZtDtGQ/CvnvK1Oanmk mTEHKML0CkmM2/y3PB/PARPHU9POC2D0mLuANdx3L4WwhzkxQbBNhbIyo9YnB0NgkKgA DTEu+L9+UHmeog9gUAGQYtbM0iXukANUj4Z3ThjqmjpUXZwDeUMFOCOstaHVW2xpFcU+ GQVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439263; x=1726044063; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7WBwKEY9MWquOlo01BPdpzLz4TZdRIsQgPATlcDwMyk=; b=F1QClwJIXCMFpf7YdroD29mOEmQV3u/qIR4QxBaSX6wUQPXKw7OtCBbD4oIJBOCOPl GCzOS2kyR1p6e4t+Ra/S9fQkFG2qNDKcunXoay9S+AFgc0psHDg9SF1ZmCgkgOoJG+VJ 7MpHkGBd8fH1OndsblGM58PzZh8VNDbbTs85R+jaL7E6i+0ed3Y7q6l2KreC+o7WsLBd 86dmYygY3JhNxDlaLmbBQhLzGnlau4k3xzDfwzwmTFNa0FFZpPDb3DcI3p0/S0lt+hoC TiL/6FLvfKmg3tJMIkNXeWLofGRLgIuxZsr/99QzzUGE1Br9+TGqm0eiJws4QPFj735M GI9Q== X-Gm-Message-State: AOJu0YzdzB4rCppTHm804PbAY/V+9u96nhydL5V0xN5ml4uFRi5urNnG pLczLdPWbsCHN29LZWpw0KrMK83i9tSJQs/xeX6D1LbnPql4yQAvekdlMoMJVaw= X-Google-Smtp-Source: AGHT+IFosxGKKraD23O7QyExDWz973MicB0gfFUEF3j/RrDnY2sLd35727GFsppJnEl62/xkyI7W0A== X-Received: by 2002:a17:902:c412:b0:204:e471:8f06 with SMTP id d9443c01a7336-206b83566dcmr20294065ad.17.1725439263160; Wed, 04 Sep 2024 01:41:03 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.40.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:02 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 03/14] powerpc: assert_pte_locked() use pte_offset_map_ro_nolock() Date: Wed, 4 Sep 2024 16:40:11 +0800 Message-Id: <20240904084022.32728-4-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In assert_pte_locked(), we just get the ptl and assert if it was already held, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- arch/powerpc/mm/pgtable.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 7316396e452d8..61df5aed79894 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -398,7 +398,7 @@ void assert_pte_locked(struct mm_struct *mm, unsigned l= ong addr) */ if (pmd_none(*pmd)) return; - pte =3D pte_offset_map_nolock(mm, pmd, addr, &ptl); + pte =3D pte_offset_map_ro_nolock(mm, pmd, addr, &ptl); BUG_ON(!pte); assert_spin_locked(ptl); pte_unmap(pte); --=20 2.20.1 From nobody Fri Dec 19 15:19:36 2025 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3DE4A1CF5C3 for ; Wed, 4 Sep 2024 08:41:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439271; cv=none; b=nBCBcspPmiakuZqJYYdwnLUK2MqQ2LTB3SBFZf6hoJfRqDO13n//Sh7Cmh+Ft4bSRLhcuH3m0Y4ltOBfIt+jC//CwGntXbal4IMG9PxPmEebashLzjmu2vnZ1ehdT/DMawEAL1VikWsuwWwpT+T8HdM5F0mEBEklv1Cd1fAqsnY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439271; c=relaxed/simple; bh=ztiWE7lPSf5Qz3/xb/OElUVpR656wNTzrQbQRyJfRhU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=oUGmWDSAlj1DzCp+X7B9CDRDWd7ckDHusLHvmuxZA01vbVp2G71GFgT2PLTihd/sl1CHUU2l4100uJUd28BEcV2O/OVtIx1d6X00oQXFuK01HidUh7Ex7Ue+PP6Rcjj2aTR1iZkxmvfzVZO3CJG9OpKPEc3hNChK35q5VwJKDuc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=S5IPLCIZ; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="S5IPLCIZ" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-20573eb852aso3452845ad.1 for ; Wed, 04 Sep 2024 01:41:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439269; x=1726044069; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Mwcb7WVpeL59BmxGU8FtBVb/FvaDY4mjrVacU9l97yw=; b=S5IPLCIZu0tOxgny9stAU34AGmZRLAdlHqzvwAul8Z6n7RXS5hC6v9DmtbH4HEbRX3 ttLgX7p/oh20N1Z1PZFgJhvt3ysLzRMAhaDdD7zmg1Xwt60YryH4K9gSPAC70DMzBUEY j6ApuKfgntVAIJxEvOEzWVMjGpKwdJYnUM0/yyGsZVcXm9tyNBIYCqXGQG69DBottaiN UP5eRuCplGs5bX89dmQRs2+E3Tac3rSHJvn/7Juv2I1yLQzkOMNZgHX2B/vGxLAAuAW+ YGrchhfbdSzAI0v/ClwQ8Bm+4+CF/KHNKHSBoh6bNVErR+d6jdHTwFyZA3mxGk7pgk5R aK8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439269; x=1726044069; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Mwcb7WVpeL59BmxGU8FtBVb/FvaDY4mjrVacU9l97yw=; b=p34AzCmKYTxJoDGXz71gEGoPAtw+3HCfXTC0PYD0tb2B99RtU2pYGmiGPOkcU/oxOO Fn08kbe2+b7+hs1tm/pCiT8FG2XfnjKWUEtdP2UJTkn6ndVvVp7w6BCPydxbXbguQTvp /7bp3znOXUIu+lG8DnkINhzhOGRrP2JgvyHJIL33iZWX18s6ro8qaE1nf+Q1zuBbUDNk lpKls8sWg9jmS0WEWGX5CIz5mdVdbl+vv4zkzMbndCNB/riL2RzcqADFVJa/Lw4FCdKt PDyXqURc7VCKCmIj5kmYHM/tEk7W+kXBqOaGHuk87XYdEGSFsijfDTb9OkGVvrES7/ON btiQ== X-Gm-Message-State: AOJu0YwncBqBC7ZCG9f479vTLiEHa/pIvzXGDZCySSHUHAtlAaH/hif8 OPmAWDMCwv8MTQAsHatj64YB9wKnWfHE8fpr5UjIMmXkAFbWmzfYfobjEK9aS+k= X-Google-Smtp-Source: AGHT+IHVi0h+YpQWstt2F+C2N++FuREpmzHqlVlqHX+adh/zaJT2MbahvdDpXAnk4gZu2G3FJSfhgA== X-Received: by 2002:a17:902:f60b:b0:206:9dfb:3e9e with SMTP id d9443c01a7336-206b832146bmr25150715ad.10.1725439269514; Wed, 04 Sep 2024 01:41:09 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:09 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 04/14] mm: filemap: filemap_fault_recheck_pte_none() use pte_offset_map_ro_nolock() Date: Wed, 4 Sep 2024 16:40:12 +0800 Message-Id: <20240904084022.32728-5-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In filemap_fault_recheck_pte_none(), we just do pte_none() check, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 20346df53df3b..216405ba497ea 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3251,8 +3251,8 @@ static vm_fault_t filemap_fault_recheck_pte_none(stru= ct vm_fault *vmf) if (!(vmf->flags & FAULT_FLAG_ORIG_PTE_VALID)) return 0; =20 - ptep =3D pte_offset_map_nolock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + ptep =3D pte_offset_map_ro_nolock(vma->vm_mm, vmf->pmd, vmf->address, + &vmf->ptl); if (unlikely(!ptep)) return VM_FAULT_NOPAGE; =20 --=20 2.20.1 From nobody Fri Dec 19 15:19:36 2025 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 123811CF5C2 for ; Wed, 4 Sep 2024 08:41:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439278; cv=none; b=b+qOfsSj2j713fFoTc/ghWsgZIuniswnVcA6C0FfLC+1zewdiFyzNlpZn1rmK0uGc71QVG6JTNlvAhnEwNefGahykJFqYqY5Rc6n7weK0T4mAMQOYdOpkqEdlh9BcixnLzHkzaSlSco1Afpf0htYkMUKlESmYzGU8rGxp+jMa7k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439278; c=relaxed/simple; bh=v+8eBabVrYBW01Qi/P2m8jF8QqAnjVEwJS4a+M2dkLU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jS16LFPb1pZMwwmstUNueq0D0nzzTvhV1xrPwxj/O2NbmBOjfo2F3PD6e/wp4Ni/XHQ1b2KSo6KBcGQEe46e46e7z9A612M4rWmvpVSBVKZSHFviXkqi/moSWSLdscaddwc6ESWAsQWAQ3hgkCH3s0xci+JU6aED8fO8lNc7ChE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=eW/P+OYT; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="eW/P+OYT" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-2057c6c57b5so2891285ad.1 for ; Wed, 04 Sep 2024 01:41:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439276; x=1726044076; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=C6e2rRMn75kFV6Fv258+yl1EB6KpOV48BOzHpMuHQ/Y=; b=eW/P+OYTQf9GPSJlWWaaNSIgGyIfTjGNLGTerkP5Kn7aNOYotrzGBxApTJKVCxijP2 OwUPSf9rKJeDAPUyO/C/5t8iKZ5/vaDJIvQbxB95cBiu/gCrUx8FMIDv4/njP1jLNDbH y53o1ZYiNx1Xn84GP7BK8AIR4em5CBuq46ZWABSgsW4Z02eWrFseiRNsXJ6q6yG26EKu VqD7sJYRohFbSXcYyO6xJSKDTqZPPoLdFKjxapENqWhdb5Xhne4e5+3wMnWNAjVkh+kg GiUXHoCcjMQhq22F0/aWt/tQG8kQiBuqEsUDCm8j/uiKyyrQEzPsElDeM2OrBLTs+fMp H6eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439276; x=1726044076; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C6e2rRMn75kFV6Fv258+yl1EB6KpOV48BOzHpMuHQ/Y=; b=Sz/ZH3//IlGaCQkv3nDP2nyUAhdn3GA1zWQDtaYlFVXLwU54F3QUsXJ9LLWNu+PAld tQ1TwkgPmPx9DqBP5jJE46kwkKpKa3Fa1ijkQTWB72ZqrWhN15VLERdTdAIG/TJwG5vl p0rEi8Y3zhsEcJb+T771I1X7bQkPuMSfeT4QhY9Ix2VlWGFAf4wOZZ355Dl72j5NAvph ZxMIg1y+gK45VfTt5tLfWTD6vGmRmA9sWjd7DZ1rTZTGlU9OxHUL2L+D9Jf+lPI3NoXV OU5f2pZn5TzsarySzkwkOs78sm8iU02lPoqlDdPAd21fCcE0zUOVSZr6nUSu6PdQdiBg lI9g== X-Gm-Message-State: AOJu0YwbWZFrhCrpxpSgGIfaujd4kPxrnfMIX/78Q06ZgYVr93xq5VkX qM4mz3cBuUAmWxxV38j/l5eQl/Xd9QzJQ7j4HlJiWEHQSCsejoekFN1Bp9SBjeI= X-Google-Smtp-Source: AGHT+IEy+qEnEqpj3sReiKH3tPFUkwjqR7spuVkWXeQJodyHWsfCM0cenE/z+pCdCppofQTZA+ekKQ== X-Received: by 2002:a17:903:283:b0:205:76c1:3742 with SMTP id d9443c01a7336-206b7d00d3amr22901025ad.3.1725439276249; Wed, 04 Sep 2024 01:41:16 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:15 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 05/14] mm: khugepaged: __collapse_huge_page_swapin() use pte_offset_map_ro_nolock() Date: Wed, 4 Sep 2024 16:40:13 +0800 Message-Id: <20240904084022.32728-6-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In __collapse_huge_page_swapin(), we just use the ptl for pte_same() check in do_swap_page(). In other places, we directly use pte_offset_map_lock(), so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/khugepaged.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index f9c39898eaff6..6498721d4783a 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1011,7 +1011,11 @@ static int __collapse_huge_page_swapin(struct mm_str= uct *mm, }; =20 if (!pte++) { - pte =3D pte_offset_map_nolock(mm, pmd, address, &ptl); + /* + * Here the ptl is only used to check pte_same() in + * do_swap_page(), so readonly version is enough. + */ + pte =3D pte_offset_map_ro_nolock(mm, pmd, address, &ptl); if (!pte) { mmap_read_unlock(mm); result =3D SCAN_PMD_NULL; --=20 2.20.1 From nobody Fri Dec 19 15:19:36 2025 Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 34C7B1CF7AF for ; Wed, 4 Sep 2024 08:41:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439284; cv=none; b=liQxwAiJa3VOyRlGvUZGZbnOR1oXyOyEsTOCguO7wlYXZXEwdStmne3MzTpuLGkWgQgurCVWUgS6xyIav3hYykty6NmZbsxy+ddvG8F6M6EQBSyAxo6rsMsG7vVpBDATcV309CwOVjDEGLF+8YJljHn2Zrta0GyCT7VPEAhWUro= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439284; c=relaxed/simple; bh=hlsEpWmf1qa6LsjjHd7NQfjzneKSvrpzaH+0KKra15I=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=lOK0xI29+Q21XjVjCXNNAHGqMgC22KxjVg2L2KUwTmZ0DSA8AGdUoHPpwMKK6qlRXU2wMKDLdxFEUbQ2izEqAz8xGt0CjKV49j2SlCZ+VjtBlVlwwRtV94D+DKMmk9gnIz2GHsbBOliNgONr4wexwKzmu1gFpbU1a/t9t8H/Mbw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=TzmGGeR4; arc=none smtp.client-ip=209.85.215.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="TzmGGeR4" Received: by mail-pg1-f180.google.com with SMTP id 41be03b00d2f7-6e7b121be30so3911056a12.1 for ; Wed, 04 Sep 2024 01:41:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439282; x=1726044082; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pGIEfQHzsThgJvUA20OC+gjd0SFgF9+h9gc5ip8dlng=; b=TzmGGeR4OjyX+vdh9g+O+p7U0yyJ/FyMALADy2OCaQb2zk/lzTGCVXr+EFq37k2pJ4 S7n4uaEhKptnQF8CKhAD2cBULWvseCQ944xVQIQUtMpYVeUOX7IUuMPpSHolemlv1KUw CaT/4mH/WNBuLR6Qjyojxxr5UdpbKPN67bEiJucMBkarx2uUD3i0aAQz6crTJA1GWf2Z e7hY/QAInIwnDppxsA/wl9kardvOy49E3dT2wlJ4kXn646/PRpeIXxYju6fgNaK+BenU kT8VGTcYQwo/YpJ83pLJwcqHlxaqutvj964tPqfTzkTs/6NtKmgG3LHX8ZpDp01EQ1Vk Icbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439282; x=1726044082; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pGIEfQHzsThgJvUA20OC+gjd0SFgF9+h9gc5ip8dlng=; b=Z0kRY93/bO6Vi65fsGYEno603f10aeRaWdsopG52/VvoDE7FB47hyAIK50Us9SeUoJ D6LwqkOnjGZJmr0XHL18Xw/32GaSSCCwPiiUPP+AhXILQARExkK9SmI1809u1WtFqd6j RHy7G/1eBwH4LWYf75tVE+JvfLM2Qk2vECq5LmipVp/oxpSb0GNyZ1BT8mDTK5sC9363 IQcDobfctvf5WYM9F1vJLiHuFqtL7OYYkuPLvVyYTz7We89deobDQwBV5Q5XR09D2HAx bVLNGQPSFB3hYNm/BAEo0wBSCp6JWdOux9fs4d0j3zAiAs1r4xNc7T6op87RT5coEyoj mwNg== X-Gm-Message-State: AOJu0YzLVG22pV0wsfE20gaz+0OHvH9ZnlDY95AXS0aWyFZgCqQvBpyJ MbRfX73Vjwfsr+wxrYB1H++LLpM9Oe9gfxnM1Rt7wkb4+DTE5w7hsH15dA6e7SU= X-Google-Smtp-Source: AGHT+IGQLG6wPNgDOgfzFUGIly89eF+GLm9izjxQGbRYP1g8ZSKlFJK5pYdbKQOnPMpn9+eqCP/7EQ== X-Received: by 2002:a05:6a21:2d86:b0:1c8:b65b:3db8 with SMTP id adf61e73a8af0-1ced607378emr11775108637.9.1725439282360; Wed, 04 Sep 2024 01:41:22 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:22 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 06/14] mm: handle_pte_fault() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:14 +0800 Message-Id: <20240904084022.32728-7-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In handle_pte_fault(), we may modify the vmf->pte after acquiring the vmf->ptl, so convert it to using pte_offset_map_rw_nolock(). But since we will do the pte_same() check, so there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/memory.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 42674c0748cba..06674f94b7a4e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5498,14 +5498,24 @@ static vm_fault_t handle_pte_fault(struct vm_fault = *vmf) vmf->pte =3D NULL; vmf->flags &=3D ~FAULT_FLAG_ORIG_PTE_VALID; } else { + pmd_t dummy_pmdval; + /* * A regular pmd is established and it can't morph into a huge * pmd by anon khugepaged, since that takes mmap_lock in write * mode; but shmem or file collapse to THP could still morph * it into a huge pmd: just retry later if so. + * + * Use the maywrite version to indicate that vmf->pte may be + * modified, but since we will use pte_same() to detect the + * change of the !pte_none() entry, there is no need to recheck + * the pmdval. Here we chooes to pass a dummy variable instead + * of NULL, which helps new user think about why this place is + * special. */ - vmf->pte =3D pte_offset_map_nolock(vmf->vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + vmf->pte =3D pte_offset_map_rw_nolock(vmf->vma->vm_mm, vmf->pmd, + vmf->address, &dummy_pmdval, + &vmf->ptl); if (unlikely(!vmf->pte)) return 0; vmf->orig_pte =3D ptep_get_lockless(vmf->pte); --=20 2.20.1 From nobody Fri Dec 19 15:19:36 2025 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8A99B1CF7AF for ; Wed, 4 Sep 2024 08:41:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439291; cv=none; b=SegGs+Jm7LKDsxLQJ2GUAmB0dNO3KH7G0EliYDv5iViH5GhE4zEDFqHbVxYXrloVn+1rTY90d2yf1ngj9uM09WEzvGLB9aa5o74h5k3Kid0QkTPo5x0Rv/qEdSuvxdkLvsVP1G2IOzx/UHsG2vPVY+jduN62g+NQDzxGgx6uY+w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439291; c=relaxed/simple; bh=dswOOZEK3YN0rQ2t/qfVl5+dn5LvxI1SZRUT+v+P+Dk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Foc3lhNMxxo7oGbQN1uhxtm7zAS9aThJoJh1gXvQvzBn/w3Ffm4LXGNFfDKZeCJ65+Z6mUuxl+e266+xLKDtop7nKXbRZBPyhCDWSM4m2fHQYeUscaWYmdz70PmrJhJM9qwpX2M3WmbLhAFK8tXNaYquh0trXIY4aU/QC9loI9k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=KELdJdon; arc=none smtp.client-ip=209.85.210.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="KELdJdon" Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-71434174201so5159336b3a.0 for ; Wed, 04 Sep 2024 01:41:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439289; x=1726044089; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jJ720IHYb4PoZSjS34HAH7fD9gF/j7WA/1t49m6Kiwg=; b=KELdJdon9tFSGVXdZP+O9X4ZSqFOdWO+QhNtQXvC2QS/YsGhe4XkOeC79AC9XR4rxD ILh8M01isntUfmuC79ogKJEkIQveQrBnAv2LD6ImUmcellIuFa5yS0nxN0YMarHyLTOj hFiRXyqnnoeFplkb483kvfEbwhh25Io3OcKWdQeWrr7Fgl3LvRH5G8+gxiV3vOCcoMtB J+5j4vMUVtcZoxG5J+hmY2qR6sDTN1VVX6SESuM38umH6k+4LwA+9X3gzFib2ZN2x0Bp xJS53xMqRn53gjGx+hszOP3f66zxC390fCaWEPkt43EJVhEfwIx6WFVbVOBScFarz/QZ 4mJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439289; x=1726044089; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jJ720IHYb4PoZSjS34HAH7fD9gF/j7WA/1t49m6Kiwg=; b=Wy3xthzp0fpm8RSilxy8okysFdcR3qhdhxycKmRlkOk0wOwKFl/uy5iBJnkTdgQu54 I11jTDiy2b5Nk6ClKIFa5ZozUTMpF/WSUu5gSLWscW1Rd/lNIqhes24MeM84udrbzXuf robyDRSeeBFtRaevD38L7CXERDTsF5F47uiGKCoUP5mdMW4d7TxWV45uNSSaOJGq6SJw 93Ictq+QCBfPDXYh8lbyIja2yxo9ypHrf6vhWAIyCCvT87D0d/ms3vAVMSWBNywwkolM ocIZJPFh2HHMIDz6fDn0Rii27mPio26fZq4gTUfzTIdQeiVVzkQHJv4FqF5iX+bCXvCJ FqYQ== X-Gm-Message-State: AOJu0Yyu4s8sS6HgqwLvohxJBcD4PffubTaRRGBCjZ09UCgit6k0DL+A Iv4vypDHgiZxSYUWLwMyamncxjA4q7PUylQ/IDs8nStCCdGX7vPQODmKXMzzONc= X-Google-Smtp-Source: AGHT+IHVhSodGEWLJbqfntGARXqFTBke8IrjtWuvc6P4tKsUnW/m6YB1esVTiG0+MVp8pwa1UVDERA== X-Received: by 2002:a05:6a21:330b:b0:1c6:edfb:431f with SMTP id adf61e73a8af0-1cecf788518mr14875554637.44.1725439288977; Wed, 04 Sep 2024 01:41:28 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:28 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 07/14] mm: khugepaged: collapse_pte_mapped_thp() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:15 +0800 Message-Id: <20240904084022.32728-8-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In collapse_pte_mapped_thp(), we may modify the pte and pmd entry after acquring the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the PTL held. So we should get pgt_pmd and do pmd_same() check after the ptl held. For the case where the ptl is released first and then the pml is acquired, the PTE page may have been freed, so we must do pmd_same() check before reacquiring the ptl. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6498721d4783a..a117d35f33aee 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1605,7 +1605,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) pml =3D pmd_lock(mm, pmd); =20 - start_pte =3D pte_offset_map_nolock(mm, pmd, haddr, &ptl); + start_pte =3D pte_offset_map_rw_nolock(mm, pmd, haddr, &pgt_pmd, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ goto abort; if (!pml) @@ -1613,6 +1613,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, else if (ptl !=3D pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); =20 + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) + goto abort; + /* step 2: clear page table and adjust rmap */ for (i =3D 0, addr =3D haddr, pte =3D start_pte; i < HPAGE_PMD_NR; i++, addr +=3D PAGE_SIZE, pte++) { @@ -1658,6 +1661,16 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, un= signed long addr, /* step 4: remove empty page table */ if (!pml) { pml =3D pmd_lock(mm, pmd); + /* + * We called pte_unmap() and release the ptl before acquiring + * the pml, which means we left the RCU critical section, so the + * PTE page may have been freed, so we must do pmd_same() check + * before reacquiring the ptl. + */ + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + spin_unlock(pml); + goto pmd_change; + } if (ptl !=3D pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); } @@ -1689,6 +1702,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, pte_unmap_unlock(start_pte, ptl); if (pml && pml !=3D ptl) spin_unlock(pml); +pmd_change: if (notified) mmu_notifier_invalidate_range_end(&range); drop_folio: --=20 2.20.1 From nobody Fri Dec 19 15:19:36 2025 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 72BF91CF7D7 for ; Wed, 4 Sep 2024 08:41:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439297; cv=none; b=SF1le3clKWdFvF8pUrDj6C/GVzIsZCV4U57D8bHRTsq0IOuJ1kC4PrgMwmKZjhUdgVwvAS3VFe+60xRBJUlao7k3O29A+KLCkMejLtLls7DzT0OAjUcRwOxGTw0KlMjPk3AJY4H1uVT2TZ6JRteoyHeFBR5M2Il2WHdLS3HDOzQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439297; c=relaxed/simple; bh=3TVv2wnCxLZA49MglLWh5UzIriJ5sUjCe3cfQyCkNmM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=in7xKuTK5+Vh0Qx5m9R0b6/0lRSubgt5rg34D88AKn4a4gywCimcyAvQerH3rPawEIRZmpnpdbGTqQIQ93sjIN2e3dwi0lankOyLaE01Rt7lhdERmnaVsDctQWBU4HaK/bs+9UaAjF+9Q8tnSl5i8+vaYEqH/RK00kX4H2rARXw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=VYtgc1sL; arc=none smtp.client-ip=209.85.215.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="VYtgc1sL" Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-7d4ed6158bcso1440730a12.1 for ; Wed, 04 Sep 2024 01:41:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439296; x=1726044096; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ffoF6/h1+uprLqHi9escfobEZTHc+3j5UpPJif6C52k=; b=VYtgc1sLM4mBjtI0kR77pOz7TTVuOK8zDz2S5PQNbxdJD0/jZ7tsEbeHh78tDlk4pA UJiH5VzbL9PuOPTMliKvOTz26Uun9rQnWhJULf9lr51vi0/IWAft1VGKcvLIWY46c0c0 mzF+QqBqhU8PJqyEzFKRdLYj6sxYDcejN1YblMO8vZIuMtGntOri/y+J0M0bBapUC7hr I8wR8OCmg99fxBscvMMnCIpR6dQA5s6mlFO2OhNo1OkaJFK10UFXrQBs2Inc1cTyJNVC hjrPomhX9y6OdeZ+6NuPu3sUHYlwOQI4wK1cQuTDZwIddL35TxTLg0s97TqICyhrS9oy ilQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439296; x=1726044096; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ffoF6/h1+uprLqHi9escfobEZTHc+3j5UpPJif6C52k=; b=b03cf/Fk7UYrcZuChYIjCrIHRVB+Sl3plltv2F3Zt58laZsdT3as3L6sdl0hcgzE67 Is8T0i65VCPNNFoSynWbqYRzQ+cJR7KtqdWPdsW9ef4/z1N78bQI4Krrd5Q0HRqZh9Cr cWGKBVMQxSBRnnepuXuDrKXptF5dyDWn1Gd/eSRSJs+0B56i6aWmUILIhTczZc5TeyrQ 3VGST2GL1Uv/grN7ZiAV6mODHwOekOlPcPIspIaSXMws3AIl5r/prpI7ZKfxI5o0oyz5 Lfv4Zvq9NK3tfS8yA1iu7SySOGzKWO1s869IA2YCD847vTFB1X4nutIOFU3mj1c3GHYZ wfww== X-Gm-Message-State: AOJu0Yy9fcZiZpoVaNN16B3ZhvMU38HYKxeN2nPeYFxx680bACXE26pK mosCKl9wQdTTyI1tYZlc4cLZmhznkEbtcmc78FpWKks6VGyZiJubCKhWatr4rFY= X-Google-Smtp-Source: AGHT+IHnX4UPdOrG4UbMFan2fo99NWoUyUAqXexLL4zg80mEgnx1zTpF+DDeMcB5gD3OIXAOIo10hg== X-Received: by 2002:a05:6a20:b40a:b0:1ce:f77a:67d4 with SMTP id adf61e73a8af0-1cef77a69e6mr6594806637.6.1725439295802; Wed, 04 Sep 2024 01:41:35 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:35 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 08/14] mm: copy_pte_range() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:16 +0800 Message-Id: <20240904084022.32728-9-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In copy_pte_range(), we may modify the src_pte entry after holding the src_ptl, so convert it to using pte_offset_map_rw_nolock(). Since we may free the PTE page in retract_page_tables() without holding the read lock of mmap_lock, so we still need to get pmdval and do pmd_same() check after the ptl is held. Signed-off-by: Qi Zheng --- Hi Muchun, since the code has changed, I dropped your Reviewed-by tag here. mm/memory.c | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 06674f94b7a4e..47974cc4bd7f2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1082,6 +1082,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct= vm_area_struct *src_vma, struct mm_struct *src_mm =3D src_vma->vm_mm; pte_t *orig_src_pte, *orig_dst_pte; pte_t *src_pte, *dst_pte; + pmd_t pmdval; pte_t ptent; spinlock_t *src_ptl, *dst_ptl; int progress, max_nr, ret =3D 0; @@ -1107,13 +1108,28 @@ copy_pte_range(struct vm_area_struct *dst_vma, stru= ct vm_area_struct *src_vma, ret =3D -ENOMEM; goto out; } - src_pte =3D pte_offset_map_nolock(src_mm, src_pmd, addr, &src_ptl); + + /* + * Since we may free the PTE page in retract_page_tables() without + * holding the read lock of mmap_lock, so we still need to do a + * pmd_same() check after holding the PTL. + */ + src_pte =3D pte_offset_map_rw_nolock(src_mm, src_pmd, addr, &pmdval, + &src_ptl); if (!src_pte) { pte_unmap_unlock(dst_pte, dst_ptl); /* ret =3D=3D 0 */ goto out; } spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); + + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(src_pmd)))) { + pte_unmap_unlock(src_pte, src_ptl); + pte_unmap_unlock(dst_pte, dst_ptl); + /* ret =3D=3D 0 */ + goto out; + } + orig_src_pte =3D src_pte; orig_dst_pte =3D dst_pte; arch_enter_lazy_mmu_mode(); --=20 2.20.1 From nobody Fri Dec 19 15:19:36 2025 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 35A571CDA20 for ; Wed, 4 Sep 2024 08:41:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439304; cv=none; b=UZwlrfpp+h1RvwkCMGNd2MfJOSW7ACZM29WJELHdXm7kDn/bpIR5or4CRlwWeJaw5KorALUC0KGYHFjhScpI/Md8bMdRM4z0xfdRHV9sOCviIx2ML7M0h+Pe+pwcljyoeJkpmphRR+P817+7gxucq205hX1QrWA/mlxnX89z0yg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439304; c=relaxed/simple; bh=hDpOwgBOPPSAVs5qcIYRUt4zcDmO9dDuotmNaP9YlXc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=VlVJMy+/Z9DahPVqv4ydmHKN8bclAlOf8oo5Qi3mGsGBgMJ1rrOqJn0NPdOeEZ6g1x++rulFQTudoIGi/CqyUhZbv0orTp1smour+I9FEE7mWJgCsRaUVQnKWv5ynihUKEXOcCg30cpeUCa3Oi/AcqgH/KYBZ6Q6KtH3GmCtLPM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=hub1K39V; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="hub1K39V" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-2068acc8a4fso21204755ad.1 for ; Wed, 04 Sep 2024 01:41:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439302; x=1726044102; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vHfOyy55HFvG/XxGZW1ra6nqYg449VLN9cHJI9/IEBs=; b=hub1K39VoAw3gViDaM9WmbKAbGh92ejNsayldik9Ype+Cx4Jp9xXs/o3eQGlhn8FuL sm58pkVMVyBBY8gBM3YWbpP/MSf7ZLDVJ8WSQZcWikFkvPWjczx+hJedc5YSqKUcdDp5 w8vlVC99nv2TqZGVa5CEBTgK8xn0F+GqNJWZK34Z81ZZNcGzO9Jg+iRnw8Z54ozqGsc+ xkMVzCP77jzpW1OKnbnXUfhmM/t6mhTgrtu75+1gYjY3gcYF5zSQH7oE7STQRJkwP9TN azaD9CiTkviyv2nxFHE1rAk1ek1dLG/Sh0PG3zsKYCaR95B3fpf7VFQ/aEJ7On3bGwCQ D2Rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439302; x=1726044102; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vHfOyy55HFvG/XxGZW1ra6nqYg449VLN9cHJI9/IEBs=; b=XamsjWduUbTYOfxdFKo2uDn7G9Xtmikmie8Hv1iOnjVUcPMXHU72UQkhpS/rjRm5se Ew7qoCJm80PqE27wkiKoihkTCAG28hwv2ONzJ8hOX5ieuLqS+EM+QR1N2C8uVpmyV+mZ T+qXM5j/BH8mPvahCcCknIgf2r0Bv4qugydXq7nL687+jxpY/L4HIXzcElkuMahDzmuS S0wKnDnaKVPggtIbKRCns+B9fiCa9BQhWjHKXxY8QoPoIjBQ7oTyoTnejx0R5tNqGRN3 qgNG5lbbFUMYnlaLhqSTvrSPBLZvcCRNoFTgHQNz47496FmV1DRGkvnJlvp/YIrFm9NI fTSA== X-Gm-Message-State: AOJu0YxcljuXIedvpauvKbMQ7qpYORgQPUuUoL95AtL6Zw8YVeFmPV21 GLBkwh0M0qqpNPzTnRZ1CndsFtC+eFST2OJWo+ae6PXPQpA1zjBOXjWP4lwHRo0= X-Google-Smtp-Source: AGHT+IGlDE+bZkJMe3LtnR3dMefm4RFnOsn/qpiRzdld3O/wCPCkO6r/LjjSl6iGWtDIKmNPthI/eg== X-Received: by 2002:a17:902:b68b:b0:202:4a24:ee with SMTP id d9443c01a7336-2050c45e710mr147539585ad.55.1725439302519; Wed, 04 Sep 2024 01:41:42 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:42 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 09/14] mm: mremap: move_ptes() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:17 +0800 Message-Id: <20240904084022.32728-10-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In move_ptes(), we may modify the new_pte after acquiring the new_ptl, so convert it to using pte_offset_map_rw_nolock(). Since we may free the PTE page in retract_page_tables() without holding the read lock of mmap_lock, so we still need to do a pmd_same() check after holding the PTL. Signed-off-by: Qi Zheng --- mm/mremap.c | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/mm/mremap.c b/mm/mremap.c index 24712f8dbb6b5..16e54151395ad 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -143,6 +143,7 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t = *old_pmd, spinlock_t *old_ptl, *new_ptl; bool force_flush =3D false; unsigned long len =3D old_end - old_addr; + pmd_t pmdval; int err =3D 0; =20 /* @@ -175,14 +176,29 @@ static int move_ptes(struct vm_area_struct *vma, pmd_= t *old_pmd, err =3D -EAGAIN; goto out; } - new_pte =3D pte_offset_map_nolock(mm, new_pmd, new_addr, &new_ptl); + /* + * Since we may free the PTE page in retract_page_tables() without + * holding the read lock of mmap_lock, so we still need to do a + * pmd_same() check after holding the PTL. + */ + new_pte =3D pte_offset_map_rw_nolock(mm, new_pmd, new_addr, &pmdval, + &new_ptl); if (!new_pte) { pte_unmap_unlock(old_pte, old_ptl); err =3D -EAGAIN; goto out; } - if (new_ptl !=3D old_ptl) + if (new_ptl !=3D old_ptl) { spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); + + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(new_pmd)))) { + pte_unmap_unlock(new_pte, new_ptl); + pte_unmap_unlock(old_pte, old_ptl); + err =3D -EAGAIN; + goto out; + } + } + flush_tlb_batched_pending(vma->vm_mm); arch_enter_lazy_mmu_mode(); =20 --=20 2.20.1 From nobody Fri Dec 19 15:19:36 2025 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 291421CDFA3 for ; Wed, 4 Sep 2024 08:41:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439311; cv=none; b=bcq3MGD9rGfCEMgGKgjjLKJKV9TnWf/oo6PaKnqCmgtDJHhY3hUUJJ8WwpwyMqRbGFzDjDZG+n1+ccuqA/ewAwE4wen1MtjdcQ6ny4YlgoGGfPHjqd9viXkh2OIwyGCqEzD3ZqAwVqrNIEQcWgLQWMHBUKnRdtKkZKqvilhgpTs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439311; c=relaxed/simple; bh=zVU5JZ4/ifmYtzZKes/SHFL4sJ0rbrE0x8fSa2fyEJU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=oBpnqu+IQM3Z2z1eopTAQpLtAsDvWn2dhSMSHeilL0j8vxOxwE/icZtdhawD+t8kUVMnvQxeGLT0QvvBia6i0beAWS03ibyGTjuVwmUMX8k+sJjIkmkc/u6Jrxhta+isJ7K2FOlZVAri6RTx/21F0+A0gnof6Cg/mZhO0o9eYcI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=hAV1l7pp; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="hAV1l7pp" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-20696938f86so13468105ad.3 for ; Wed, 04 Sep 2024 01:41:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439309; x=1726044109; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DklgHNcyYpWTok/ex913aKZBAOiPUnESXBkqa0TTMVk=; b=hAV1l7pp5zvXkfO+L/hBkHrbueUVLJOQpusLGWoAtbzlEqSR/zrTtPuEK/1BV0QBWU hc7TfzebAV2FVLT9PmfClLab4s2lYLNsw9O85CSFuBDele81ggwmUqtf/4ma1of+fB8l pEG2COE8DmgYFFXpnWkDIF5JaxLwoidCLQYDKdjd6edQlAS6/RbkfwQ5S6CBBF9T9oOs EF4Gd0tlTTsneGaClQtj4rWILgZm0fPwWbtIx4znj3FUMx4Xr/otVi0qqqqwRz95Cg+A Rdezyxa0uzqlHSdFIaydGwF0OfXGFiytV8Pmowk6R0if7w+JI+Wzi8/l5ppGk4KCA10i rJvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439309; x=1726044109; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DklgHNcyYpWTok/ex913aKZBAOiPUnESXBkqa0TTMVk=; b=GeECGDPzsf3ZyANKL4eLkhE607rHUVRf2/wTv8aqph6aasFWt52bMWM+dlQYYkyzez bR3SdkafHiiu2Pr99DcynADKtscBuCCsRyDvxrH8fixeObX5DKmBvZv5N2GK/Psugrfw v3hgz/ed0tKhfBuZhIrxjT5TrPl9ng6wm+wZfPtJ9O+NK3lG7GhcuSgZ5H8SjaFLIUvl wPkGCCUEalGUNCzV5jkGTBKo5Bgz8CRv6mde51IOO1P1rkF28K7FIAAZEVGmD9pqpUpz T/1lyVLUehuty8SbZYJ8p4wr9FxZBksRyCCAfpiMmhPcBOFCSTL1mg59z1L3TdhpIjE/ ptAA== X-Gm-Message-State: AOJu0Yzd6RJj0Ry4FAUV++2rT7tAd4YMJvNsJXq28Nne7eHJq5YuSYCH UzvWu4zCmvE7vSS4hD0Ts7CPeKUlvVq5HeR+Te60qMNgNKfqKCwzLqA2oXA0W2Y= X-Google-Smtp-Source: AGHT+IErn+gfx0jNqN8oYReDzi5ALFGdG/LuObb3FXvUpU+trUkKEzs/YdM2MbdzLHq+FDgFhXw6+A== X-Received: by 2002:a17:902:d50a:b0:205:6c25:fe12 with SMTP id d9443c01a7336-2056c25ffb5mr96453155ad.34.1725439309345; Wed, 04 Sep 2024 01:41:49 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:49 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 10/14] mm: page_vma_mapped_walk: map_pte() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:18 +0800 Message-Id: <20240904084022.32728-11-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In the caller of map_pte(), we may modify the pvmw->pte after acquiring the pvmw->ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the pvmw->ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pvmw->pmd. Signed-off-by: Qi Zheng --- mm/page_vma_mapped.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index ae5cc42aa2087..f1d73fd448708 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -13,9 +13,11 @@ static inline bool not_found(struct page_vma_mapped_walk= *pvmw) return false; } =20 -static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) +static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp, + spinlock_t **ptlp) { pte_t ptent; + pmd_t pmdval; =20 if (pvmw->flags & PVMW_SYNC) { /* Use the stricter lookup */ @@ -25,6 +27,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, sp= inlock_t **ptlp) return !!pvmw->pte; } =20 +again: /* * It is important to return the ptl corresponding to pte, * in case *pvmw->pmd changes underneath us; so we need to @@ -32,10 +35,11 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, = spinlock_t **ptlp) * proceeds to loop over next ptes, and finds a match later. * Though, in most cases, page lock already protects this. */ - pvmw->pte =3D pte_offset_map_nolock(pvmw->vma->vm_mm, pvmw->pmd, - pvmw->address, ptlp); + pvmw->pte =3D pte_offset_map_rw_nolock(pvmw->vma->vm_mm, pvmw->pmd, + pvmw->address, &pmdval, ptlp); if (!pvmw->pte) return false; + *pmdvalp =3D pmdval; =20 ptent =3D ptep_get(pvmw->pte); =20 @@ -69,6 +73,12 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, s= pinlock_t **ptlp) } pvmw->ptl =3D *ptlp; spin_lock(pvmw->ptl); + + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pvmw->pmd)))) { + spin_unlock(pvmw->ptl); + goto again; + } + return true; } =20 @@ -278,7 +288,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *= pvmw) step_forward(pvmw, PMD_SIZE); continue; } - if (!map_pte(pvmw, &ptl)) { + if (!map_pte(pvmw, &pmde, &ptl)) { if (!pvmw->pte) goto restart; goto next_pte; @@ -307,6 +317,12 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk = *pvmw) if (!pvmw->ptl) { pvmw->ptl =3D ptl; spin_lock(pvmw->ptl); + if (unlikely(!pmd_same(pmde, pmdp_get_lockless(pvmw->pmd)))) { + pte_unmap_unlock(pvmw->pte, pvmw->ptl); + pvmw->ptl =3D NULL; + pvmw->pte =3D NULL; + goto restart; + } } goto this_pte; } while (pvmw->address < end); --=20 2.20.1 From nobody Fri Dec 19 15:19:36 2025 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B19091CFED1 for ; Wed, 4 Sep 2024 08:41:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439318; cv=none; b=V3ES/gDSCyACGadfmUVFiqi1zfZxKAgTU+hxRYwGcvO5/ah6lzBxWSOlYX9ee+rcchLVYLPKvoh2woUk81zkx5MiIsx5T8QlihXCT+7H9yPS2Ke9c8RHfhkZ1jsPdSQAqsFTPFCRJ8ElJHNyqsJsWW1Z38qofEAYoOU4MeKL+1M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439318; c=relaxed/simple; bh=uIbfcvTo6Td8qMQXROw79N/GDlcQcfN90m4HD/LjEwI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Ax+sllgyGNPvlC1ImGEGltvIb2nfdvfnFydUM7WD/iDh2uP4FAkwNpqxXgCke1UdcGLa6n2RDr6qQX0dG4VxKcjW4BUvRNBUVD4A5TzoJHBl4dDHftxVKlGeGKF4WCmtVauxRkjUTa3UKLeghyKEeixV8nC5UhkJ517Z7xBGhS4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=aCe8WgS9; arc=none smtp.client-ip=209.85.214.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="aCe8WgS9" Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-206aee4073cso6775605ad.1 for ; Wed, 04 Sep 2024 01:41:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439316; x=1726044116; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4PxEMdLIb7w0CeH690VB/ed9+hRnYGoSsqaGDg4GBd4=; b=aCe8WgS9qI6c4pq2U4xhuZdXsibXD4jY8RkrzjyLaKJhekXS7RlTsChuuY0l73HFkp cCTFGNQchnpKv52V1Hq3VdFXNzzfMBjc2aycQPaw2X+5b90F5+JAwmX2itg3Pdk9oF2g flAEBwtnWyr0ysR44IjA2s78ouBEkPnXSJXfahGGl6lA2CJj9Tdn1AEQQElo1EyaTO4v aclqSglz6itFzsVmAXvjjeWy0of1/feotus75LypHIOAfEienFKtmCm7bcNSDcK1gB/I zOZeIHDbtIPiAsIOWcSClmd+Tl00x60YvVwwllk+EwlCPxwnfoM//g5sesOrLGHuQ4Un Plew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439316; x=1726044116; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4PxEMdLIb7w0CeH690VB/ed9+hRnYGoSsqaGDg4GBd4=; b=mtu/3X0HpnG88FV/qWmrmSjxRFbBLZdHuJ5aU2mtiDfBKGqOl2mflIZzYeV/D6My8n SexBjlxuC4zvAeQcxzaMAUZczNpFX4i66n1FyUUdkjDzZXk1NaocgTA0SbmL0PgUTypw 0r8JltL4adzW8fEoAQYGG+UJnt9HLEZFWFpy0N3MCxsh1LdrcmTkxgAOOkjsdskmgfZi em6MAeVYRSJhAFlABLaob/N1x3R/l+osFR94QpBonXivY1HvFEhL18uJJBdfJ6yUCZqV HVMZO9Mlu6j5BTzjsAXrxLvmDyDrRb6rbuiWIHMqcv85dCkcgF9Eh+kEQylGuM0sJBid hEOw== X-Gm-Message-State: AOJu0Ywszu/sMU9eq0eUcDN/++ZiGIplrHPpEHq4usy2fbi4yYMqWNSY rgJunUSv0uORfpumI0LKWsQuNuAgTwbaGalHNGKIRkigWMDRtbVbV9Jg7OLkKmY= X-Google-Smtp-Source: AGHT+IHUb0AH81Bf+t9sl3x8o3nWMR3fxDcIYnthyKOt2Ee/pQCOOrlcmsn4X1RsHo/HQ+Mvmo6mhg== X-Received: by 2002:a17:902:f94c:b0:205:4273:7d65 with SMTP id d9443c01a7336-20546131866mr119665875ad.21.1725439316150; Wed, 04 Sep 2024 01:41:56 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:41:55 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 11/14] mm: userfaultfd: move_pages_pte() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:19 +0800 Message-Id: <20240904084022.32728-12-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In move_pages_pte(), we may modify the dst_pte and src_pte after acquiring the ptl, so convert it to using pte_offset_map_rw_nolock(). But since we already do the pte_same() check, there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/userfaultfd.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index ce13c40626472..48b87c62fc3dd 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -1135,7 +1135,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t= *dst_pmd, pmd_t *src_pmd, spinlock_t *src_ptl, *dst_ptl; pte_t *src_pte =3D NULL; pte_t *dst_pte =3D NULL; - + pmd_t dummy_pmdval; struct folio *src_folio =3D NULL; struct anon_vma *src_anon_vma =3D NULL; struct mmu_notifier_range range; @@ -1146,7 +1146,14 @@ static int move_pages_pte(struct mm_struct *mm, pmd_= t *dst_pmd, pmd_t *src_pmd, src_addr, src_addr + PAGE_SIZE); mmu_notifier_invalidate_range_start(&range); retry: - dst_pte =3D pte_offset_map_nolock(mm, dst_pmd, dst_addr, &dst_ptl); + /* + * Use the maywrite version to indicate that dst_pte will be modified, + * but since we will use pte_same() to detect the change of the pte + * entry, there is no need to get pmdval, so just pass a dummy variable + * to it. + */ + dst_pte =3D pte_offset_map_rw_nolock(mm, dst_pmd, dst_addr, &dummy_pmdval, + &dst_ptl); =20 /* Retry if a huge pmd materialized from under us */ if (unlikely(!dst_pte)) { @@ -1154,7 +1161,9 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t= *dst_pmd, pmd_t *src_pmd, goto out; } =20 - src_pte =3D pte_offset_map_nolock(mm, src_pmd, src_addr, &src_ptl); + /* same as dst_pte */ + src_pte =3D pte_offset_map_rw_nolock(mm, src_pmd, src_addr, &dummy_pmdval, + &src_ptl); =20 /* * We held the mmap_lock for reading so MADV_DONTNEED --=20 2.20.1 From nobody Fri Dec 19 15:19:36 2025 Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C31F91CE710 for ; Wed, 4 Sep 2024 08:42:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439325; cv=none; b=eGsHFPSvqQbdixKRzCJz5cu50H+va9aVdJSnJUo922yFcxZaZ76mUca+MlaFe2ZD8Q6mYb1qu5J9CdKEnzKTum+ENgC64GLD5VXzZD439O/py/0REwlv5d2GFJIl7PR0n0IQ545MdbVvwm10IfIdxzNkRbS0+iGFIk8PuQvdhaI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439325; c=relaxed/simple; bh=My0ULxrplZDhPoFxWfT7wAiiLVrqrf3pw9vNGA5BsnY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=acrqVUWYEuvFTqjw0zGVk+No5BxlPzaTN0wfypY4cFhcsFURh/D4PTOLsapthSlMPurRVthQP8uAMo1ouhSEzigQHszXPimbc0G1b7ZNcR9O2dhsKBarSvgLar1lT99HNWEsuW+in6zp5UijqD/h6b9CyFmWSoM+Zqvj1giB59Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=PxaWHlwc; arc=none smtp.client-ip=209.85.215.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="PxaWHlwc" Received: by mail-pg1-f179.google.com with SMTP id 41be03b00d2f7-7cd830e0711so369084a12.0 for ; Wed, 04 Sep 2024 01:42:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439323; x=1726044123; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Y4jsyz9eZnVjuIeNrv9jm5llMD04VewCinfUm/XXhZs=; b=PxaWHlwcQMfnGVtS0De19fRNHa77B+kN35uTq9FgFz51Ml5DgmmfN6g2Sv2eTwOdU/ aVeIHP6JaulxMTFkeby8oxSRtMWwQhn4rt7oLZVPla877uYE+lVMaEqO3P1l9CjosthR DnKwTbJ8YDuDkIgtyWbWeGNdwWqlSZ/wrMENCOamHQ1mAdAysg/+no1VmgyrrcTXmycu qrd1EQRll7rspN0fmaAReA0qJXk2FbqJ9MZAqsiKjsaBv5uVFmgrVKnTNUZrCydUh2Lv qY399+DopwYaoEkY0B38oILYqVhvUub9vn5OOX6rDxsXVdeQKQGSNJab/H21Mh0NaPiv OTgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439323; x=1726044123; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Y4jsyz9eZnVjuIeNrv9jm5llMD04VewCinfUm/XXhZs=; b=oQP+qMiYJYPc+3o/IYtMucb1U5kG6DG4HcAC+Oxyftcq8wUSQlhm7fP8cfjkicoaAH PkD9ik5qmXm0fHRC+Q3MT4kqczk/rcacuV3sirV+F0WEsBU10eZgMhM009zEe2TswBnr P73cb4+gbzp+ex9a/++d4TQ2Z0ifnCofLtF/CpHwmOOS3p3x+DCykaVi8y8fjtV8iJlL GtyhHyquQldIoGNrjmiZbogkIyNgLxlrC56qbKTBYBmEf8EQctK+oqsso3Cul9/C0ZQ4 NS4mqM80zeUeWySlSV3eaeJwX6vA3Dx3ilrS0Iy+kTmcyY4nwSXOqZOCdL0ZHZfZwWXC SPnA== X-Gm-Message-State: AOJu0YxbZRkMMlEn+WJxTyFpzBlrur8bW1lpiR5YbhyVTOFo/Lg4ifet AuknFX1rsLsqi4Fj65ztn1J+H2xasZl+A1tSn6qP7qJENkHm8Wck8lx2w2BnTME= X-Google-Smtp-Source: AGHT+IGXU/qy5PPe2L4ehUmGkYFpO+LVdbh/09mdfTo5RpVJKjaH3T6TEVRYBI0feUcEJkjAodGgeA== X-Received: by 2002:a17:902:e545:b0:1fd:6ca4:f987 with SMTP id d9443c01a7336-206b8341fdfmr24676655ad.15.1725439323059; Wed, 04 Sep 2024 01:42:03 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.41.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:42:02 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 12/14] mm: multi-gen LRU: walk_pte_range() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:20 +0800 Message-Id: <20240904084022.32728-13-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In walk_pte_range(), we may modify the pte entry after holding the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pmd entry. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/vmscan.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index a9b6a8196f958..36b84e46cd7b5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3375,8 +3375,10 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long= start, unsigned long end, struct pglist_data *pgdat =3D lruvec_pgdat(walk->lruvec); DEFINE_MAX_SEQ(walk->lruvec); int old_gen, new_gen =3D lru_gen_from_seq(max_seq); + pmd_t pmdval; =20 - pte =3D pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK, &ptl); + pte =3D pte_offset_map_rw_nolock(args->mm, pmd, start & PMD_MASK, &pmdval, + &ptl); if (!pte) return false; if (!spin_trylock(ptl)) { @@ -3384,6 +3386,11 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long= start, unsigned long end, return false; } =20 + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + return false; + } + arch_enter_lazy_mmu_mode(); restart: for (i =3D pte_index(start), addr =3D start; addr !=3D end; i++, addr += =3D PAGE_SIZE) { --=20 2.20.1 From nobody Fri Dec 19 15:19:36 2025 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6316E1CE710 for ; Wed, 4 Sep 2024 08:42:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439331; cv=none; b=pHTWkTbhoQ6dNT5KphyMBw4xzo68ZML/Zm+DJhQU/jTuKaS2iRLrFjpaJdQWgdk/YrTvOLAvkM0MGKFYYY5UaCeMot6Ku09balzlfqevoxl/G+l9SDVcjR0YQRtm52dHGn0aYXwtme3cTKoWD2BIvNd7JVkJqrN5r6sB7hrPFPk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439331; c=relaxed/simple; bh=pGvXpoizu5EfkA5gqHIohJBgIdkJzD8TMccCB9Mq6Lc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=fJaj5hlG6bN3Z/sV4PSIwGIZ8eX5nqCYPv7/dVIM/tYPrIkxoMsvqXzfR90SzNTMgPDlEfG8WE7LpX3w43vSojwyEoLOiSfFzzOcTHbMBwjVtKwQ4eQD+0pVr33esSBnWJLudSNCDOoZj4mk0xmeX/bO1Pa4Mudxsb0Nk2b+mSg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=U1BlvZVS; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="U1BlvZVS" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-20573eb852aso3457815ad.1 for ; Wed, 04 Sep 2024 01:42:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439330; x=1726044130; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TUS6V5hps9BhpBteFi/blbp/e+0aHtnhPyo5Zqj4Bag=; b=U1BlvZVS/ML764yO6TEsEJUdiRVInumsAABGCI7c9jw6TsXni8U5Kx/g5Y6tHRlFj/ 9FI8V4rEQdXZ0QBQ2tNNMtww9wJtRIlZQUhZfiG1eHHcAvnLWTNG1c6VMbxVv45Xj3cL aacZ0OKi6uG8gXq0TDdaTDGqv73/15ULx9w8h9xHx0VvT3S3bt5fTAt5cVbI5zALy9Tg aw2ihQ7CfHkujw5qv/k5Xw2/sd2/cmmA7+9bk18kqIlIq2CfXJ9wLl3+ZMdoDy3s3Ob1 lXO30JutAs4Bu4CFg0mLhvjGTQEXeh3EFq79F9lwzg9lUIOAWr8M5oq2fUszaC3wXfTK UZAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439330; x=1726044130; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TUS6V5hps9BhpBteFi/blbp/e+0aHtnhPyo5Zqj4Bag=; b=oBls/dIjhBGBP8H3wAJkvhBPQudnZtwvFTsoflJJH8gVnX5WieY2vvQNTZatHbHqFC R7A4smuhNLLIckgjkNv61aEydX/cjBWdIoNVfJX2QYh1H+zvfPHziMTrw7i9LVLxr+wf L2IuKCu0F3tLUZbL+UI4G8qkX2MkpHOTYJWslZYGdGe3muL7xlZKU28DpJIfd5tBcpIX aWP67yURs2kstdAGYb6bsifEh+h2PmHaAdZB4WLYD9FnbXaaYy6zCQCCxHa6kPw6r+Sg WCECnGlVIS3o7Pwj3GjMtYKwcumxpcmex51oCHbexTGS9o3/3v1iAo/F4uDEu6RAyizF RsiA== X-Gm-Message-State: AOJu0YwYKykeRGILe7sKJSCfJsUpt1xVjn019EwAe4J2xNq+wngkd9qY IMqtPjwm/S4dlpCQhyhsXkFMGZHpi3NPkyCJDfhF3M+xQl1eIl/2XHTLEgo1Cyw= X-Google-Smtp-Source: AGHT+IHV13v46mfi8eJv03VSK3v4h4kaXbg2iDfLcHBwpuweJiwY6TIZiDRC9qw17xCxMuqJ20qctg== X-Received: by 2002:a17:902:ecc9:b0:1fb:7654:4a40 with SMTP id d9443c01a7336-206b833e006mr24427095ad.14.1725439329649; Wed, 04 Sep 2024 01:42:09 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.42.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:42:09 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 13/14] mm: pgtable: remove pte_offset_map_nolock() Date: Wed, 4 Sep 2024 16:40:21 +0800 Message-Id: <20240904084022.32728-14-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now no users are using the pte_offset_map_nolock(), remove it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- Documentation/mm/split_page_table_lock.rst | 3 --- include/linux/mm.h | 2 -- mm/pgtable-generic.c | 21 --------------------- 3 files changed, 26 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/= split_page_table_lock.rst index 08d0e706a32db..581446d4a4eba 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -16,9 +16,6 @@ There are helpers to lock/unlock a table and other access= or functions: - pte_offset_map_lock() maps PTE and takes PTE table lock, returns pointer to PTE with pointer to its PTE table lock, or returns NULL if no PTE table; - - pte_offset_map_nolock() - maps PTE, returns pointer to PTE with pointer to its PTE table - lock (not taken), or returns NULL if no PTE table; - pte_offset_map_ro_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; diff --git a/include/linux/mm.h b/include/linux/mm.h index 1fde9242231c9..5b5a774902bd6 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3004,8 +3004,6 @@ static inline pte_t *pte_offset_map_lock(struct mm_st= ruct *mm, pmd_t *pmd, return pte; } =20 -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index 262b7065a5a2e..c68aa655b7872 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -305,18 +305,6 @@ pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr= , pmd_t *pmdvalp) return NULL; } =20 -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp) -{ - pmd_t pmdval; - pte_t *pte; - - pte =3D __pte_offset_map(pmd, addr, &pmdval); - if (likely(pte)) - *ptlp =3D pte_lockptr(mm, &pmdval); - return pte; -} - pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp) { @@ -374,15 +362,6 @@ pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, = pmd_t *pmd, * and disconnected table. Until pte_unmap(pte) unmaps and rcu_read_unloc= k()s * afterwards. * - * pte_offset_map_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_m= ap(); - * but when successful, it also outputs a pointer to the spinlock in ptlp = - as - * pte_offset_map_lock() does, but in this case without locking it. This = helps - * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that = time - * act on a changed *pmd: pte_offset_map_nolock() provides the correct spi= nlock - * pointer for the page table that it returns. In principle, the caller s= hould - * recheck *pmd once the lock is taken; in practice, no callsite needs tha= t - - * either the mmap_lock for write, or pte_same() check on contents, is eno= ugh. - * * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like pte_offse= t_map(); * but when successful, it also outputs a pointer to the spinlock in ptlp = - as * pte_offset_map_lock() does, but in this case without locking it. This = helps --=20 2.20.1 From nobody Fri Dec 19 15:19:36 2025 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E3F971D0DE1 for ; Wed, 4 Sep 2024 08:42:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439338; cv=none; b=l8GeGsSF5OZLSRyPcPUf6YLaG2LYpGHT+Ukxk7/gXVU3RYTTI3Rb+GcF80aAPayotluutCu93djYziok5vjplM1G8tCw3aKn5/7pYlCRNd4/1KNpg4E/aGVaovQks4yl5Ksgcam7iNNJqEJzoOvMHlZudGirMmEbXCj/8fkRhBs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1725439338; c=relaxed/simple; bh=8AzoJE8l7/0rgAUJHjuIMB0gJiKomtVvJS2dRHBDsH8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=f21m5StnWCWl/QKS8IX05YqkcTXjko+nMIiTVbh8hHaRlr12VSxWHS+lagIUfgOpQcakwcMeJ1zxT/KHXIiPu0nHxD8ZX00TrjD7c1yjOlhrwQKmCQ3JylykGQKvj68Hk3oPrNURGH+FOdTrJE2OYKySXM0+Kg23bOONnTYsEDQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=StYn44fm; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="StYn44fm" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-20551eeba95so27654975ad.2 for ; Wed, 04 Sep 2024 01:42:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1725439336; x=1726044136; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ylbiZHnndQyBxRrEPbsYoqCd5Ue19pmhRhZr0okk+70=; b=StYn44fmgOEupfxYXUiRKwXo7Mnkef9gc/TMa31YkVOq0QqwcEDAcqRAgxfN3Z5Aw6 QKOZJ6eK4MQIUqWMmiDPrbMHDsdYFUlDLwJKt5fJ4GT3u/xWlUHm6xdSxcurS7UpQnWH 8UaBZa7764uoXRkD/uz8hc0ld+maAbYICvWWyswRJ5Hgq7lJRAFOJcTT6ftaHl+XOJKG czK8XJPdlBXRYttavgvnSmfKb14jHp/BfDjgeAZbDgVfUVu6sHI61gap9XkuIFW26BsA orvpTSifteOrw57BbcRL+2Xp9gxdgp3nSHGY8ztoPqbw+swZUSCVNjTDXueF6R7vnkKN eZ5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1725439336; x=1726044136; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ylbiZHnndQyBxRrEPbsYoqCd5Ue19pmhRhZr0okk+70=; b=ncvzdEopC1drGGxx67mcaOcBR6T5R5slXzCvp6sJX1tFoGbWPbCXs259zI3vel4Fm9 m7asLnow1CehbXWuSMWekzPhY1iSLW1xUPq5ISM//n4Na2jmKv04NTsWHRkLJ2eDtzHW VcEE+csfxqt/D09AapSmki0nnv6D9tN1rBM5VunxonrB49xTNyQC/+n5oCuHet8xyuFv XIOKdIoQT+lnZvaS5OKDhnFLW6OW6grSYu1Y+JPcSe05f1MOR146FrOQ4Vz+mL8wl/pV Yg9gnBQeqdjST2DDToWpE3vKbGQUM2mB6YOOVIbYk+jn9OMRMbrWK/EDvC0uPv3LxvOX DNNQ== X-Gm-Message-State: AOJu0YxqPC3tl9lFBEf5DAxHroAr6URhgZ44zN0S4aZqXxBzAD90XjkB vVG4/Q94jgxlGUA+Oup99rC6roahlOg/gCIZr0WaJOHQsiZ0UAlAqUEK2PySNEw= X-Google-Smtp-Source: AGHT+IGRRbNTDVHBE5FEDByUohEYiQ4MupDiF5X1fYOKyMaX8J1f74Xy7lHxJ4SbBUityp3jftUwsg== X-Received: by 2002:a17:902:e848:b0:206:c43f:7896 with SMTP id d9443c01a7336-206c43f8dd5mr2410565ad.21.1725439336115; Wed, 04 Sep 2024 01:42:16 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.242]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-206ae95a51csm9414045ad.117.2024.09.04.01.42.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Sep 2024 01:42:15 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v3 14/14] mm: khugepaged: retract_page_tables() use pte_offset_map_rw_nolock() Date: Wed, 4 Sep 2024 16:40:22 +0800 Message-Id: <20240904084022.32728-15-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: <20240904084022.32728-1-zhengqi.arch@bytedance.com> References: <20240904084022.32728-1-zhengqi.arch@bytedance.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In retract_page_tables(), we may modify the pmd entry after acquiring the pml and ptl, so we should also check whether the pmd entry is stable. Using pte_offset_map_rw_nolock() + pmd_same() to do it. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 17 ++++++++++++++++- 1 file changed, 16 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index a117d35f33aee..318cc3eefb040 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1724,6 +1724,7 @@ static void retract_page_tables(struct address_space = *mapping, pgoff_t pgoff) spinlock_t *pml; spinlock_t *ptl; bool skipped_uffd =3D false; + pte_t *pte; =20 /* * Check vma->anon_vma to exclude MAP_PRIVATE mappings that @@ -1759,11 +1760,25 @@ static void retract_page_tables(struct address_spac= e *mapping, pgoff_t pgoff) addr, addr + HPAGE_PMD_SIZE); mmu_notifier_invalidate_range_start(&range); =20 + pte =3D pte_offset_map_rw_nolock(mm, pmd, addr, &pgt_pmd, &ptl); + if (!pte) { + mmu_notifier_invalidate_range_end(&range); + continue; + } + pml =3D pmd_lock(mm, pmd); - ptl =3D pte_lockptr(mm, pmd); if (ptl !=3D pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); =20 + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + if (ptl !=3D pml) + spin_unlock(pml); + mmu_notifier_invalidate_range_end(&range); + continue; + } + pte_unmap(pte); + /* * Huge page lock is still held, so normally the page table * must remain empty; and we have already skipped anon_vma --=20 2.20.1