From nobody Fri Nov 29 05:50:55 2024 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 392A142A8F for ; Tue, 24 Sep 2024 06:10:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158237; cv=none; b=sJYbN4Cde2eWor4suFhC39pOoWOSPg1lRW3P864c5Rgt2BnOYfcZzFxIpKX/xWNN+DCMvPkyCVFXDe6juuOCTQxAgPnqLn0+6Ha9ABTnTuaA5bO6fQuvsDWOa1vMRJ5Q7KY4IPlx/FRdcmqiyzryzSSyrMxdSSmuKYXh3jwunTE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158237; c=relaxed/simple; bh=ZbaeWw3Cvpk7pq6FuZm+ealD8Sv3/4DHQMcnViX1FcU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=i/2lJoF9RERihrcoHdXv6EV0Cv6IWc2VWAiuLt4CAPlV7cFjaAisfgHdS4k/8FsS4Qw4rmnKyqy7EcbqAm1+O9R2sMwRj33I2xSKXIkib2RBfNX/+WEGMx3QY7icTtIk7IsHiPBxUWSN7yUJCExK1lzyb5Y2VhwyYuQHs4tfXkw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=gn+5kId8; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="gn+5kId8" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-2057c6c57b5so30983215ad.1 for ; Mon, 23 Sep 2024 23:10:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158234; x=1727763034; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wFV2Gg1HPn9r0uTPIEIMJioMCNlgr9zWJ7S9CBl14ro=; b=gn+5kId81ccBpHvWdF97g6P4FqzaOivDPrrQTDybkP8N12qzRJxR8ldChPpOVXLT0Z d+GGLXYnknKm3K0QG5jPY6M9PkYVLJj45LW7WVlKnfnME0AhqZocvDwRVn0rOfE2LjP6 FjIkvrVoN7T8NvUdGRgTM5fRuHSS+UHcIC+e7Crh1gut3VIzF+nHNreSvZZixBd++cSX KRFR2HEpXuQ56UnT5Vx7slr7JVYHi6iUQemD1lzMGUcQAxUaIs85s+GmBDJ+v/W93tJs 8NgtK13umGb7M8v1f9vKzsEpZQ8uqUlc9NEtkqUpsc4bRUOtrrZ2EDKgDfLwvhzaoU2B Gq7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158234; x=1727763034; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wFV2Gg1HPn9r0uTPIEIMJioMCNlgr9zWJ7S9CBl14ro=; b=imXHp+dfdEz8b+VBJH1HFgm7UmTSGJ9BmQhtpgKqeZsgZu9tSYUjOlIWUt/DXEPs1B 0HSCL8W+QpLLxZAchj30mvbk1VlXzymeHwb3KHDsag3XHCs5lJDmxHB2NPtr0JHL0gg3 +rf27unlc/ZnhGwQdJWm5ixCDxAvUNaH+pmoHuvbPjjlB39ti0Cfd9KPZ6ff0hOPQk3X 0XA67wBLgB0mEFj/GfafqKpiANl7okfIjk7dZ77zpDNQ0kYNYe+uDLQs4ilwuHtl5A8T jV0cOOuXnycjTmc6FmgBGBpaxk7Wa2HuzV6Jzo8B7MSww5LT8Ppo/zLJtmCHWdvKe6uJ 9/lA== X-Gm-Message-State: AOJu0Yyb9M2077hjA0OyqvQRExBIPg7Q7+hZBpfWASUkAPBY200fEirb zix9PLo1K+0pokX9jrmc0hR0gVFZm0KNcNjVVPppnSYcDXQvxKk9k3fv8nicz9I= X-Google-Smtp-Source: AGHT+IFurxEbSfxWnkgUvSmQpfQ7iOb3of4fLdIGxM8iu8OEamvgIzhTlXEYzcUx/o7HLvoMu2wctg== X-Received: by 2002:a17:902:ce8c:b0:207:6d2:1aa5 with SMTP id d9443c01a7336-20aed0b299cmr27872445ad.13.1727158234439; Mon, 23 Sep 2024 23:10:34 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.10.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:10:33 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 01/13] mm: pgtable: introduce pte_offset_map_{ro|rw}_nolock() Date: Tue, 24 Sep 2024 14:09:53 +0800 Message-Id: <1a4fea06f8cada72553a8d8992a92e9c09f2c9d4.1727148662.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, the usage of pte_offset_map_nolock() can be divided into the following two cases: 1) After acquiring PTL, only read-only operations are performed on the PTE page. In this case, the RCU lock in pte_offset_map_nolock() will ensure that the PTE page will not be freed, and there is no need to worry about whether the pmd entry is modified. 2) After acquiring PTL, the pte or pmd entries may be modified. At this time, we need to ensure that the pmd entry has not been modified concurrently. To more clearing distinguish between these two cases, this commit introduces two new helper functions to replace pte_offset_map_nolock(). For 1), just rename it to pte_offset_map_ro_nolock(). For 2), in addition to changing the name to pte_offset_map_rw_nolock(), it also outputs the pmdval when successful. It is applicable for may-write cases where any modification operations to the page table may happen after the corresponding spinlock is held afterwards. But the users should make sure the page table is stable like checking pte_same() or checking pmd_same() by using the output pmdval before performing the write operations. Note: "RO" / "RW" expresses the intended semantics, not that the *kmap* will be read-only/read-write protected. Subsequent commits will convert pte_offset_map_nolock() into the above two functions one by one, and finally completely delete it. Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- Documentation/mm/split_page_table_lock.rst | 7 +++ include/linux/mm.h | 5 +++ mm/pgtable-generic.c | 50 ++++++++++++++++++++++ 3 files changed, 62 insertions(+) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/= split_page_table_lock.rst index e4f6972eb6c04..08d0e706a32db 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -19,6 +19,13 @@ There are helpers to lock/unlock a table and other acces= sor functions: - pte_offset_map_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_ro_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_rw_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken) and the value of its pmd entry, or returns NULL + if no PTE table; - pte_offset_map() maps PTE, returns pointer to PTE, or returns NULL if no PTE table; - pte_unmap() diff --git a/include/linux/mm.h b/include/linux/mm.h index 546a9406859ad..9a4550cd830c9 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3017,6 +3017,11 @@ static inline pte_t *pte_offset_map_lock(struct mm_s= truct *mm, pmd_t *pmd, =20 pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp); =20 #define pte_unmap_unlock(pte, ptl) do { \ spin_unlock(ptl); \ diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index a78a4adf711ac..262b7065a5a2e 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -317,6 +317,33 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd= _t *pmd, return pte; } =20 +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + pte =3D __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp =3D pte_lockptr(mm, &pmdval); + return pte; +} + +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + VM_WARN_ON_ONCE(!pmdvalp); + pte =3D __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp =3D pte_lockptr(mm, &pmdval); + *pmdvalp =3D pmdval; + return pte; +} + /* * pte_offset_map_lock(mm, pmd, addr, ptlp), and its internal implementati= on * __pte_offset_map_lock() below, is usually called with the pmd pointer f= or @@ -356,6 +383,29 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd= _t *pmd, * recheck *pmd once the lock is taken; in practice, no callsite needs tha= t - * either the mmap_lock for write, or pte_same() check on contents, is eno= ugh. * + * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like pte_offse= t_map(); + * but when successful, it also outputs a pointer to the spinlock in ptlp = - as + * pte_offset_map_lock() does, but in this case without locking it. This = helps + * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that = time + * act on a changed *pmd: pte_offset_map_ro_nolock() provides the correct = spinlock + * pointer for the page table that it returns. Even after grabbing the spi= nlock, + * we might be looking either at a page table that is still mapped or one = that + * was unmapped and is about to get freed. But for R/O access this is suff= icient. + * So it is only applicable for read-only cases where any modification ope= rations + * to the page table are not allowed even if the corresponding spinlock is= held + * afterwards. + * + * pte_offset_map_rw_nolock(mm, pmd, addr, pmdvalp, ptlp), above, is like + * pte_offset_map_ro_nolock(); but when successful, it also outputs the pd= mval. + * It is applicable for may-write cases where any modification operations = to the + * page table may happen after the corresponding spinlock is held afterwar= ds. + * But the users should make sure the page table is stable like checking p= te_same() + * or checking pmd_same() by using the output pmdval before performing the= write + * operations. + * + * Note: "RO" / "RW" expresses the intended semantics, not that the *kmap*= will + * be read-only/read-write protected. + * * Note that free_pgtables(), used after unmapping detached vmas, or when * exiting the whole mm, does not take page table lock before freeing a pa= ge * table, and may not use RCU at all: "outsiders" like khugepaged should a= void --=20 2.20.1 From nobody Fri Nov 29 05:50:55 2024 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 282C078297 for ; Tue, 24 Sep 2024 06:10:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158242; cv=none; b=qpv7eFXkJQzDl2FG8uMhviQw4pewpOFBzSpG0WFggrxNm83HNVveUr2nwtyqvuACKnZRgnt6vBvjNmtxAlwfspGjv7kYHd2pQkH8HWxIaE4UOqlDhSV/4SXmPplEwscJaL5wmxqpqkmydEeUsvqDpTBNbr2xSAi88hjwPNjsP8o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158242; c=relaxed/simple; bh=Ks+ctZjoe6oexIKNJorWHsj/FHxBIY1SArqgBxdFKSU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=fpd6/lMR5fVVy+f33yvT+J++t5Aw4SrvbMbkAj5yZGCfOM8cU4xPNIDHHaOFOj//FiI6iUBevlHRIfwACusw5ZrHGMmaV5bXVHrUph1AtVNf+phP9y0eMG+PPWsXUPFJu6+gjKqx3nTMMt3TRUj7pboVh7WsY2Z0HpW57dQb1Ts= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=KrGxiqDx; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="KrGxiqDx" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-20536dcc6e9so32521965ad.2 for ; Mon, 23 Sep 2024 23:10:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158240; x=1727763040; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7WBwKEY9MWquOlo01BPdpzLz4TZdRIsQgPATlcDwMyk=; b=KrGxiqDxpcYfLBHeknxzWyC5sZBD4Q911Ghn+67/XTWosP5ofOWOIwTuMUUrLwHkAr ypp50VXraHx8Jp0spCPhvhvrAmX3Z38iw7AgJ3qtqb3XlW7y2AdleoVH9XzlLjCRAXfk y2iqtt2fuHaRn3+mBZWgPvftzaUfvNcJnk2mSSrAJ0wpy9ILSrbnkVoW0chKQ9lCW9TM dMS9tcL9NgtVL/c7gtnkaeUHL+VgFGYsRJRxRaQQg4CFN27NbgMUe+A/jxeTmouPoFhr K9R1xHFMOMAhuhz1oTaIEeX7UD0GxmLFdrdJLrdMSlWEvCTtFp9Cd5pF+wKXlHItNty9 WC4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158240; x=1727763040; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7WBwKEY9MWquOlo01BPdpzLz4TZdRIsQgPATlcDwMyk=; b=MVxDCZjFNAdnlsmO3m12Po/6dYgAoyMZTKEkCCtWuLYlcRV1/g9MmNz5KiK/TZdDM3 4MVBciQ8smbxyCrpaRPGwb4DMxQU7+efNamHu6FdVWkJ1JYHFDVnxFWA2fwEeZfd9MVq 0b/TStGJBDMNN7QxUQxbnBQUzGcqAGwjnjDCl/3vfLc151sPA9GBjQvntwrthkbjqpaL gxFzMcPJ7waosU/s9rRlaZU5FBNpqGec1v8hVZsIKF9Oqu7sYaiAwWnl3M9TrO5Fikgl w1+ZBaR6rgwDjvFdpOQ21rkinuRl5hC0B+l9tPsd6uf7VF+hsbLJU+USAcX7CTgjNhJJ 0uMw== X-Gm-Message-State: AOJu0YwBexzQ6Ipzrl5Rt99WAEojww2jOBO6H/aKjk5vaTJJ9DxyDahD sJYBRSx0YHLRwpPUHRwFZi74n3wLx5UFGelcnMqidAELbvepJcbwHHMlgMOFaV/zwH12qKRx4uk Ewkg= X-Google-Smtp-Source: AGHT+IElcy6/dcdtPcpdDMoiIIDu5/E2AQfQDwpdj/dfRLSSVL2tlA+KvKWV7BmaFFYnOD/6ZCb7Sg== X-Received: by 2002:a17:903:2311:b0:206:a935:2f8 with SMTP id d9443c01a7336-208d832a910mr203885105ad.2.1727158240521; Mon, 23 Sep 2024 23:10:40 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.10.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:10:40 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 02/13] powerpc: assert_pte_locked() use pte_offset_map_ro_nolock() Date: Tue, 24 Sep 2024 14:09:54 +0800 Message-Id: <597a93841525dbd58fbebcfdc547f2080694835b.1727148662.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In assert_pte_locked(), we just get the ptl and assert if it was already held, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- arch/powerpc/mm/pgtable.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 7316396e452d8..61df5aed79894 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -398,7 +398,7 @@ void assert_pte_locked(struct mm_struct *mm, unsigned l= ong addr) */ if (pmd_none(*pmd)) return; - pte =3D pte_offset_map_nolock(mm, pmd, addr, &ptl); + pte =3D pte_offset_map_ro_nolock(mm, pmd, addr, &ptl); BUG_ON(!pte); assert_spin_locked(ptl); pte_unmap(pte); --=20 2.20.1 From nobody Fri Nov 29 05:50:55 2024 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3040F74BF5 for ; Tue, 24 Sep 2024 06:10:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158249; cv=none; b=ht8g1WatZGgEg4vVoyHppBOMAZQ+TPyg6QaoOAy3R1ZX11UzEKqCpYZanIisR2O+XVLjguwRd1BluO6vW+RXZjjJey+oOZpdccN/mEeVUt6VUkNSP2OYAsmCx6JZjWyopp2yyYMk4UlD1hXK58VQKxxyCVoUCf0ZV0rFSLYl4Q0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158249; c=relaxed/simple; bh=CYi+EZMylLlMSCqw/uJB5X9IiL/Hx4QgVrtaCgCZ208=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qGTMlibivGixI6SIdmi3JoIz9cdxZ51Z7yz4G74/TfMdwPNrPNx5J59i7S5vW5ZFRHUXrvFAEJJoHIziL0PbVgHzNjO556oB46xfWt/2lao6/dvk9cNlVzqVsebiysO6OnMVHN6YoY4T3mCQP6Bav+HBAQGCv9WN7/4Vf7AtT8Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Oec2x2KV; arc=none smtp.client-ip=209.85.214.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Oec2x2KV" Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-2068acc8b98so47504115ad.3 for ; Mon, 23 Sep 2024 23:10:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158247; x=1727763047; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BmENsM1LVZMrc0W501TyzYIERT8c75Mhan1AJ9vs73Q=; b=Oec2x2KVXW3dfQVoamAzAS9d3PTvzSypv30M057Vd9KG5coCvh5bn/F+f8LqriNumk sibw4pjPGETHMb9WAn4Je73OJRORhF23XghApfuJC9RhiffzjbTQz3sepwKGhOM3U3BM jfHkrg032omPxzvf3k/vtqBDmf8lYsYKedkjscyKXZAyaH31hkGdKEpq3WSWWLqvTcrD orVzs2H33VVFsxrEHHGPP1SM3j6E9FD4QCf/NMksZmQl6X4DhYLAiGsq1JiMqatLLDKI y7RJk9vrPdjYX+sKaaPeXwxU7Y9aseUq5V2sqcgK5Q5Xhz63ZqSu/+TzoVI89EN3rdwp da3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158247; x=1727763047; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BmENsM1LVZMrc0W501TyzYIERT8c75Mhan1AJ9vs73Q=; b=f3Ok1pn26Z1CNbrb54zSBW9yXoymywnPj0J4KM6Xiu+FAtTePnhIMmR800EJNafMkk r8LQHL1mwSm83F2eiZUsVXtCsHDYssI9ee9G7KM0HtcuJsviV+ge8MAvCs2LJR5DcxVz MxllE9hcioKj/9koinqGAGEWDi5n3ZHg2ef7U89xT4eKA4NksnsabgDCdlgca743jWbt PzWPs828vHWl++4+GqKxFQsU2iLFNR4gV8ke278MxwwoqVLSlw+D6FbOfQZT8D6YJVAT MC0BWpXilQT4fxtj1VBQTJTaeWk01TpMK3IUBFUErj5PtLb5Mve1HY3+mU8h9Eo/fMyu OuQw== X-Gm-Message-State: AOJu0YzU3MjTDrHbIyyepVKsJpkHW1SfhzpPzNklMqsCKH5lnyminhVX C2XUOzuE4e/dFj7FbgUeStESLTRy1+9M7zR/nRCDaK4ke5WhtZ9v74p0oTp/gF4= X-Google-Smtp-Source: AGHT+IGm87aqttqV2Aef0OLuq60VY3Mp4w41J7dgMXq2Y+1L52786e2U6HUOab2ZHlW+VZaRsZgruQ== X-Received: by 2002:a17:903:22c5:b0:205:6f2d:adf7 with SMTP id d9443c01a7336-208d83a8387mr177369015ad.21.1727158246651; Mon, 23 Sep 2024 23:10:46 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.10.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:10:46 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 03/13] mm: filemap: filemap_fault_recheck_pte_none() use pte_offset_map_ro_nolock() Date: Tue, 24 Sep 2024 14:09:55 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In filemap_fault_recheck_pte_none(), we just do pte_none() check, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 3e46ca45e13dc..6c6ff8722550f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3271,8 +3271,8 @@ static vm_fault_t filemap_fault_recheck_pte_none(stru= ct vm_fault *vmf) if (!(vmf->flags & FAULT_FLAG_ORIG_PTE_VALID)) return 0; =20 - ptep =3D pte_offset_map_nolock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + ptep =3D pte_offset_map_ro_nolock(vma->vm_mm, vmf->pmd, vmf->address, + &vmf->ptl); if (unlikely(!ptep)) return VM_FAULT_NOPAGE; =20 --=20 2.20.1 From nobody Fri Nov 29 05:50:55 2024 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C00681745 for ; Tue, 24 Sep 2024 06:10:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158255; cv=none; b=QCMCMpdh6fxH50iyjwivlUJT7t6rZfOCfvnBxQMUW/tOyN/mYOG4AKMGrwOSY82R8RA4oCrHeGEWaDCVOQyH0wr/UE/xp5IZI5J6+qwXgH4YJpOKgCec8xVtsjH8d7Bulky5tYGBwrxESMokC67CKPlWYhBD+OWo9Q8CVX7XlFY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158255; c=relaxed/simple; bh=v+8eBabVrYBW01Qi/P2m8jF8QqAnjVEwJS4a+M2dkLU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=A917jAdFX8am8lKedrIIguj73Xd41TFz16bYGq3LsUqKo3LFSjKexAm9jgB8D2Ojy2nypaSPdXoVgYW+CGUkPBIVY0YUGH0y3+xsLCeWZY/66KGCdVVXgSmEcSAwJIRDIKmRFoyIzhMqdu6T3DFLfvUgrSZ9m0UZfoAB52qdh40= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=dh9Awln+; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="dh9Awln+" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-2059112f0a7so45869405ad.3 for ; Mon, 23 Sep 2024 23:10:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158253; x=1727763053; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=C6e2rRMn75kFV6Fv258+yl1EB6KpOV48BOzHpMuHQ/Y=; b=dh9Awln+ck7frvMR9eFgW6mXcsdG/pPZPRaOL7rTlLPqtZ+Oxf7Yh4fS9/O67h0TVS ve5nKopHKACzqzQwyPk4hoRfrwdM1pYmkbKuurEmmWvGA5iDWYkoEq2rYt3uMHMAGkPL BWMWPGrw55eGssug2FsoXPkRkfY6LWNyYox+vCVChbE9U0YEQuSR44CTMVqdUTjOjm1N +wyr30YIErAGczPKzzuJl8pYPMLBkoxE4IC6LHezqKD7yAuGTIA1/NWvWOxd7HCygaNo u1oZivXxSH0sUZ9n+zHgD3OYIQFJQf5ltx/jELDs25ifJXwN+kPomCyJQ9fefWq30AOw bfgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158253; x=1727763053; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C6e2rRMn75kFV6Fv258+yl1EB6KpOV48BOzHpMuHQ/Y=; b=aTV9+QgVGW3ssdiaDHHsh9BEvrjAhXnNMAhrtw/FSMTuNZTKxJNp1i3b91L0zap9fg i41xIsUbi+oMuIvEGeJNoMIsiDJjBsbiicWw9uuMIhLegb1ECnmqTfD1C0/y66fgIG+5 s1iUkELAXsZ9AjQe/gn0Y28VVSUXwv/VNH0jZdlgFueW3D8KwqFkN6b4HvxKYpcPfa89 nsCVDODaa4sQZhKFpXHLP4sCAqH5lQHCirh76GmxBxKiKbXpxmjAUvOLmqqXDW0X8eTM cb0DFWet/GwZBofbrYZkcR+NNLhkyY2b5frphyK7puu3AdM8Pv3W2SmcWgPc8TtRDWqI 7URA== X-Gm-Message-State: AOJu0Yy3+PllhZKqWr/bEVsNcN25G8bUERrwEHS0CP3eMiGreVorjcIE KsQqeccgG2Cg7WwfRuKYBRzKuntyL1EgFSuu/TGBcD4/+PeSY/H5ig+JA5pn4zs= X-Google-Smtp-Source: AGHT+IEXIad82sT4qTFz/mgMdJh1108B7x8i2K/1dZJvW9IHButimBDIW5JZoOsyX1eaQQUsC7JSYg== X-Received: by 2002:a17:902:c94c:b0:205:4a37:b2ac with SMTP id d9443c01a7336-208d83c8698mr211905735ad.34.1727158253643; Mon, 23 Sep 2024 23:10:53 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.10.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:10:53 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 04/13] mm: khugepaged: __collapse_huge_page_swapin() use pte_offset_map_ro_nolock() Date: Tue, 24 Sep 2024 14:09:56 +0800 Message-Id: <69e1c4483c2d679d3e2c8446264e5878e5b52a97.1727148662.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In __collapse_huge_page_swapin(), we just use the ptl for pte_same() check in do_swap_page(). In other places, we directly use pte_offset_map_lock(), so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/khugepaged.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index f9c39898eaff6..6498721d4783a 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1011,7 +1011,11 @@ static int __collapse_huge_page_swapin(struct mm_str= uct *mm, }; =20 if (!pte++) { - pte =3D pte_offset_map_nolock(mm, pmd, address, &ptl); + /* + * Here the ptl is only used to check pte_same() in + * do_swap_page(), so readonly version is enough. + */ + pte =3D pte_offset_map_ro_nolock(mm, pmd, address, &ptl); if (!pte) { mmap_read_unlock(mm); result =3D SCAN_PMD_NULL; --=20 2.20.1 From nobody Fri Nov 29 05:50:55 2024 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2CE91126BF6 for ; Tue, 24 Sep 2024 06:11:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158266; cv=none; b=CmAUszmozPHhHmuF963kWs5xlwbWpCQdM9g4Nd6pGph411gTC7B/1D9WNiCbnss/4QwBq4I511if2JrISo0sS4Hlmz720EBmj6zNOfVRcXr1CH0CoLZZBKo4QbnJoV0bZLYYgA3B3Bf9PEsfSS1sFAEJA/qzNZuHUcSAsdTDD3g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158266; c=relaxed/simple; bh=cUqvPzkChjavQi5ll+6QC49FMZw8M0EOQ/mq6UklpII=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=E2d9u5xctNVD7Iv2CEUvwIKQqISnCW1VcvQsHALNedA5jSKLKxQOSBrr3iI22OaF/hV2Ysriw7AUv0okOUJncB15P7CXvHAbiN47foht5DW/QZpYYAsP0edpVZnjoTk5LBnW3UMSvCDZ5oTLNpnIymWQzElzeiQ/4L8dQNwz4cM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=B5VUyCKx; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="B5VUyCKx" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-206aee40676so41322505ad.0 for ; Mon, 23 Sep 2024 23:11:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158264; x=1727763064; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WqTOjUnPHL3XRjrt8I2PlSfGhzwzG7n3dPRatvbgECo=; b=B5VUyCKxnQSCrR0T912xOurp7HUfyVmfG9IuLLH/NuDVTAhd/uZl71VbJx6HOotOjt uwe5q96lAHCfH0R+KwNxxA2/AEWXmda8CKQt472QJWkG0VGbz0ssci06dSw8ZBJFv9nK BqWtfh8p5uKalTpo6gV+zprKjb41nilqdzHiF1/kwh4pHm5TVCMtIZMUdzzlsPMlXmiZ qats42sBtixJn4FXE9+SwXutKTqWWn87Wu8hCB6Dml3YhKKkkadHL+qQGXZFJ0s9V6Yt Ccz8I6M1sS46cRaOb+U0MCcx3IDmRbsxp4uZuim+vj1xfNalWsbAxcC5udn2sPSzwV6r 5B2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158264; x=1727763064; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WqTOjUnPHL3XRjrt8I2PlSfGhzwzG7n3dPRatvbgECo=; b=V9tZtqqpMQ7UvQ4Q8v6xld0Htx0YO93mZhK3coONEUJSwthn5OB3hvhxl99db+7jtu gOiKa40D50hH3sJ4vlnXOWpdGZoVjubaumrcyI9IAa2+XqqHo9NvbiArir2u4P9Vz547 0zjFRyep7jW2gb+zJ5up9XiPgxNPkP7pcr1rTvg2SV4KFfeFW1+MO1mDC55+JwzvW10F ppStymExcWS7AJBHIO8g1DLt+zyqsvfOXf0nMpsCUfK3Phbz1JsD9z6CkYKirI17fStY cIrPENO6+PW+yPhkUqnj/DmbEUL45H/ZpSeGIDeyzAV2p+eWN9EKU9FkcNB3M2AGJUij fDNQ== X-Gm-Message-State: AOJu0YyM4XcjxmWQlvnJ7qk/Z5aF2Xyxip9mGV4OyC42gLVIJ9yH6sqQ ca21IcuF9wLTCZKePB/kiLESEO92kjwc3OgfDhMkQ1Y3JP9XCqAKY6LG4oYKtRM= X-Google-Smtp-Source: AGHT+IGIKI1reF9Jfo+dWimmN1cyjgiGD8qn8mcbfIJc2tOglwlNFzTXQQXYEG1VSSXByHdLorgo7Q== X-Received: by 2002:a17:903:944:b0:205:4e15:54c8 with SMTP id d9443c01a7336-208d8440f7amr165611365ad.61.1727158259613; Mon, 23 Sep 2024 23:10:59 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.10.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:10:59 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 05/13] arm: adjust_pte() use pte_offset_map_rw_nolock() Date: Tue, 24 Sep 2024 14:09:57 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In do_adjust_pte(), we may modify the pte entry. The corresponding pmd entry may have been modified concurrently. Therefore, in order to ensure the stability if pmd entry, use pte_offset_map_rw_nolock() to replace pte_offset_map_nolock(), and do pmd_same() check after holding the PTL. All callers of update_mmu_cache_range() hold the vmf->ptl, so we can determined whether split PTE locks is being used by doing the following, just as we do elsewhere in the kernel. ptl !=3D vmf->ptl And then we can delete the do_pte_lock() and do_pte_unlock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- Hi David and Muchun, I did not remove your Acked-by and Reviewed-by tags si= nce there is no functional change. arch/arm/mm/fault-armv.c | 53 +++++++++++++++++----------------------- 1 file changed, 22 insertions(+), 31 deletions(-) diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c index 831793cd6ff94..2bec87c3327d2 100644 --- a/arch/arm/mm/fault-armv.c +++ b/arch/arm/mm/fault-armv.c @@ -61,32 +61,8 @@ static int do_adjust_pte(struct vm_area_struct *vma, uns= igned long address, return ret; } =20 -#if defined(CONFIG_SPLIT_PTE_PTLOCKS) -/* - * If we are using split PTE locks, then we need to take the page - * lock here. Otherwise we are using shared mm->page_table_lock - * which is already locked, thus cannot take it. - */ -static inline void do_pte_lock(spinlock_t *ptl) -{ - /* - * Use nested version here to indicate that we are already - * holding one similar spinlock. - */ - spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); -} - -static inline void do_pte_unlock(spinlock_t *ptl) -{ - spin_unlock(ptl); -} -#else /* !defined(CONFIG_SPLIT_PTE_PTLOCKS) */ -static inline void do_pte_lock(spinlock_t *ptl) {} -static inline void do_pte_unlock(spinlock_t *ptl) {} -#endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ - static int adjust_pte(struct vm_area_struct *vma, unsigned long address, - unsigned long pfn) + unsigned long pfn, struct vm_fault *vmf) { spinlock_t *ptl; pgd_t *pgd; @@ -94,6 +70,7 @@ static int adjust_pte(struct vm_area_struct *vma, unsigne= d long address, pud_t *pud; pmd_t *pmd; pte_t *pte; + pmd_t pmdval; int ret; =20 pgd =3D pgd_offset(vma->vm_mm, address); @@ -112,20 +89,33 @@ static int adjust_pte(struct vm_area_struct *vma, unsi= gned long address, if (pmd_none_or_clear_bad(pmd)) return 0; =20 +again: /* * This is called while another page table is mapped, so we * must use the nested version. This also means we need to * open-code the spin-locking. */ - pte =3D pte_offset_map_nolock(vma->vm_mm, pmd, address, &ptl); + pte =3D pte_offset_map_rw_nolock(vma->vm_mm, pmd, address, &pmdval, &ptl); if (!pte) return 0; =20 - do_pte_lock(ptl); + /* + * If we are using split PTE locks, then we need to take the page + * lock here. Otherwise we are using shared mm->page_table_lock + * which is already locked, thus cannot take it. + */ + if (ptl !=3D vmf->ptl) { + spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + goto again; + } + } =20 ret =3D do_adjust_pte(vma, address, pfn, pte); =20 - do_pte_unlock(ptl); + if (ptl !=3D vmf->ptl) + spin_unlock(ptl); pte_unmap(pte); =20 return ret; @@ -133,7 +123,8 @@ static int adjust_pte(struct vm_area_struct *vma, unsig= ned long address, =20 static void make_coherent(struct address_space *mapping, struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, unsigned long pfn) + unsigned long addr, pte_t *ptep, unsigned long pfn, + struct vm_fault *vmf) { struct mm_struct *mm =3D vma->vm_mm; struct vm_area_struct *mpnt; @@ -160,7 +151,7 @@ make_coherent(struct address_space *mapping, struct vm_= area_struct *vma, if (!(mpnt->vm_flags & VM_MAYSHARE)) continue; offset =3D (pgoff - mpnt->vm_pgoff) << PAGE_SHIFT; - aliases +=3D adjust_pte(mpnt, mpnt->vm_start + offset, pfn); + aliases +=3D adjust_pte(mpnt, mpnt->vm_start + offset, pfn, vmf); } flush_dcache_mmap_unlock(mapping); if (aliases) @@ -203,7 +194,7 @@ void update_mmu_cache_range(struct vm_fault *vmf, struc= t vm_area_struct *vma, __flush_dcache_folio(mapping, folio); if (mapping) { if (cache_is_vivt()) - make_coherent(mapping, vma, addr, ptep, pfn); + make_coherent(mapping, vma, addr, ptep, pfn, vmf); else if (vma->vm_flags & VM_EXEC) __flush_icache_all(); } --=20 2.20.1 From nobody Fri Nov 29 05:50:55 2024 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7857D126BE8 for ; Tue, 24 Sep 2024 06:11:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158267; cv=none; b=VBq7GGoL6qQH/FwVHlLYvtt+XoYDKDP4zLBx4IoGzT3yURudZK0XYMtyaHutn6ZoP7w4+g6trA4H87vzgSsvMYJF+Lc+QBY8wHla2n18A+pBRjuEfD6TfIK3jmw5BHy9WWLJFoVpADzJ9vGqis5q82hWVcj98g1zXzznR++H3PE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158267; c=relaxed/simple; bh=AW2pi97601MGuZiCOXDhP3jZoiCUTCuHUPQzcZePTCg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=LBouQcU0axdfJBXXp6PeSx0tp+CwQtmawDiTcS4DLU6DOAusk3PKs3/l9s5SzZEGO0O4xi/hkqkth1gDTWWWBY/Nvnw2KXNi3/P3rp5gUjlVkww/yyJV2ICWu2mZWw2s1wHm3v8itpQ9NZ5ePJ1pp72n7J8vJfF9YhgmNzE4kQc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=cBCKvAGr; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="cBCKvAGr" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-2059204f448so43655425ad.0 for ; Mon, 23 Sep 2024 23:11:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158266; x=1727763066; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=88oX8R3xcxvOkzUvzbp2hva7xyyimZKCI3msRz7gwSQ=; b=cBCKvAGrulo/XUVYfJ0pCxmgoJp/TdbOYHnZqWqrjJ71jq4IFreTO9MKiEBEviH+vW +ObFOv06qZ1g/wMlnw1aEsU+YhvZXAbvuq1uxD8yaMg3/zfrLd/bcVK8wb5y03EM7BZE glVY+a7o1E0alcjCGozDdE6AHo9dJ5bT5+CI2AR5LHOT72svHCmk2kZm1Kg/BrvlXq20 1lpbAcZe/V6vl7B8RIWJQ4J+H/z6/jpOvZNk1QPzbPowW7idjh3x46Gqw+gEm1CTMg1N JP7ZmLBwPfm7ZX8VqzeOdY/FaE4T4i5DOkCJC5BlZnEnX84kkZFqisdgwuKuuVcdPkda eYLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158266; x=1727763066; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=88oX8R3xcxvOkzUvzbp2hva7xyyimZKCI3msRz7gwSQ=; b=SGjnFtEodfChzzm6qGQ3akD0L6sJ7fc3/1JvmgZNXAvtc3T8sLlKir35ZJ7paHhI2O /tz6LD3pdy2FLBzXfJSOyIanpm9UvoehTh7DahHIx8o7YS2wR9pmzbpY5gV4LJUbGfgg 9LqAbejDCLRfjVamukVLIFaeTUvZNL7c510NKCORhZi/m7bCDKsvKkWEnn9tNzQYDWPQ DtfGrcIjapGTKcQK4xxu6fTP6mHzQf2Tzcwb4mhlEtZaK4KFuTWTm0gjLInSKx901ssG 66L1h8qi3tcllRjjMc2LFWradlScdq7BIA4K+u5DAj+JjOU9m/kUoGx3Dkb59hGssU72 ae2g== X-Gm-Message-State: AOJu0YxyC05OBBTibsUsbpaQjVkcy0M8y5I9BbzKztL51Kc5PqmTqhCK GMOV/y5MtwTbzmGem1kADl+jRrzJafl4AZWdXcu770jhFhRHC7TvUoFOjIpXc04= X-Google-Smtp-Source: AGHT+IE0gKFIx3/KW4QPIJwQt51ZMLGla2X2YU2KRRy7vcX0w9t/15+gAAip1FE70+J7SP+04M4CVg== X-Received: by 2002:a17:902:d511:b0:205:7574:3b87 with SMTP id d9443c01a7336-208d980bb34mr182109485ad.15.1727158265764; Mon, 23 Sep 2024 23:11:05 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.11.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:11:05 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 06/13] mm: handle_pte_fault() use pte_offset_map_rw_nolock() Date: Tue, 24 Sep 2024 14:09:58 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In handle_pte_fault(), we may modify the vmf->pte after acquiring the vmf->ptl, so convert it to using pte_offset_map_rw_nolock(). But since we will do the pte_same() check, so there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/memory.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index a245d28787034..6432b636d1ba7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5750,14 +5750,24 @@ static vm_fault_t handle_pte_fault(struct vm_fault = *vmf) vmf->pte =3D NULL; vmf->flags &=3D ~FAULT_FLAG_ORIG_PTE_VALID; } else { + pmd_t dummy_pmdval; + /* * A regular pmd is established and it can't morph into a huge * pmd by anon khugepaged, since that takes mmap_lock in write * mode; but shmem or file collapse to THP could still morph * it into a huge pmd: just retry later if so. + * + * Use the maywrite version to indicate that vmf->pte may be + * modified, but since we will use pte_same() to detect the + * change of the !pte_none() entry, there is no need to recheck + * the pmdval. Here we chooes to pass a dummy variable instead + * of NULL, which helps new user think about why this place is + * special. */ - vmf->pte =3D pte_offset_map_nolock(vmf->vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + vmf->pte =3D pte_offset_map_rw_nolock(vmf->vma->vm_mm, vmf->pmd, + vmf->address, &dummy_pmdval, + &vmf->ptl); if (unlikely(!vmf->pte)) return 0; vmf->orig_pte =3D ptep_get_lockless(vmf->pte); --=20 2.20.1 From nobody Fri Nov 29 05:50:55 2024 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F0B6F12C544 for ; Tue, 24 Sep 2024 06:11:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158274; cv=none; b=dHdMW29ImxjKppX8mtBdevc4AcJDn9seSilcVkYxqBhfV7w6a/sfg+qgZNwCHXXuKcN2gKHgulqECH5eUgq01jhoMKzuhT+KeN11n6/kGIpjLDenXzQJeZXBEz1+1BdpFrzYZqwZl7V4m8aTezByvHTRnBd8GejTo3uBx6ynhwk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158274; c=relaxed/simple; bh=jM90FoLs4n/oQxSzhPYYLIrDBR33Qih88e32uMrgh9Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=GIvsXz64/b4eWQ4dmaaz0pa37TgONMazTtd8/Fez7YZ6hcUdLmcTlfpKPpLJsQtVny8yVi5DqGQKbBqZZarMx4uDRgZhq3b8C6hw+dcpkmcH8j2fd3gS0jK4C4iXSBERVJCwNiTl/uQRXMr+Y1aCndJP9qMStCg/7bfcyocuWa4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=GANG1nte; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="GANG1nte" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-2068bee21d8so51320605ad.2 for ; Mon, 23 Sep 2024 23:11:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158272; x=1727763072; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dtqkBA88kwvMRDkdpXIuTPjK06Yw44S9bnA3B3TRVKU=; b=GANG1nteSJP8So2t/n8yrGVqu8KElvDoag7y0NLYCWubPbvnvqudLdy5hmh3RcAyTF Cm5TXG+9mRBD890veQoQhmnWVHmISvIN2NV29v1YdE2WUK7MC4cf8tGka3poM49qSjNh WwUJwRrRih066OFBEXt/G3YpS9UsJ0mUro+1DkmhhCzGMRfxCSpsOeCvsWAcxbcIx5tm X2Rwkzbdl4XGJNa1MHll9Cbx8zqaDsZCszGCjl+zabmRUmI1nMKJVf6c3ter7F1awmqE ctx1XVQoUQS6NElfEC/uubTbK0jGHZFxFMvFETNny/RxBtqngSaV065H7xIa9OHhg6yM y1rg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158272; x=1727763072; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dtqkBA88kwvMRDkdpXIuTPjK06Yw44S9bnA3B3TRVKU=; b=BGnKXSYTq/jk7WyXWMnzRi+MY0+cjdAM1D2L/+LkBOSF1jZ4DVgdbu8/OWrQuFTQMg +lHvlSszjDEx/UXIp4SufWnM6LqG0PF0mTh2NkCnJVdl4dK401i4XwBhregOelvGQUe7 Sbu5IN1QqCNokH7QdsA+TxxYy/cOwmMlOaZBUXwtGLoDcJwZHGCLf15tW+yPtqfC/yUg l2+1Ro5MZ0WqnuAIKjW0T2z0NFpO7v/T5D7kWsP0S4RfZA0mwHK8gv3T2ipqwVQWns9H HmMsBDzWcdFmaWY9eM7OcS4KN+Q0ZFLCdktQBvVvqukWDJTe24QbSF0HbOAs/0h6V4S0 pCsw== X-Gm-Message-State: AOJu0Yzd1DQEXJ66CLN19DiA1UyRVTaQGoYZYcvdbhR3hdodG8g7kDA6 7+PCvIW3GMubF3Lb2jj+Al7Y+pp/oxQXfyV3uP+WdZ8ASTn86C+BtrDD25TK6Ek= X-Google-Smtp-Source: AGHT+IGXhl7eW49ih0T9YK4dpESD/v76/2vHb7m+lE9WqCGFaYPlARnoD/c7DbHxV22Za7qr6kwfpA== X-Received: by 2002:a17:902:f54f:b0:205:755c:dde6 with SMTP id d9443c01a7336-208d83cc8b6mr212103045ad.30.1727158272107; Mon, 23 Sep 2024 23:11:12 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.11.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:11:11 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 07/13] mm: khugepaged: collapse_pte_mapped_thp() use pte_offset_map_rw_nolock() Date: Tue, 24 Sep 2024 14:09:59 +0800 Message-Id: <07d975c50fe09c246e087303b39998430b1a66bd.1727148662.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In collapse_pte_mapped_thp(), we may modify the pte and pmd entry after acquring the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the PTL held. So we should get pgt_pmd and do pmd_same() check after the ptl held. Signed-off-by: Qi Zheng --- mm/khugepaged.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6498721d4783a..8ab79c13d077f 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1605,7 +1605,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) pml =3D pmd_lock(mm, pmd); =20 - start_pte =3D pte_offset_map_nolock(mm, pmd, haddr, &ptl); + start_pte =3D pte_offset_map_rw_nolock(mm, pmd, haddr, &pgt_pmd, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ goto abort; if (!pml) @@ -1613,6 +1613,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, else if (ptl !=3D pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); =20 + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) + goto abort; + /* step 2: clear page table and adjust rmap */ for (i =3D 0, addr =3D haddr, pte =3D start_pte; i < HPAGE_PMD_NR; i++, addr +=3D PAGE_SIZE, pte++) { @@ -1645,7 +1648,6 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, nr_ptes++; } =20 - pte_unmap(start_pte); if (!pml) spin_unlock(ptl); =20 @@ -1658,13 +1660,19 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, u= nsigned long addr, /* step 4: remove empty page table */ if (!pml) { pml =3D pmd_lock(mm, pmd); - if (ptl !=3D pml) + if (ptl !=3D pml) { spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + spin_unlock(pml); + goto abort; + } + } } pgt_pmd =3D pmdp_collapse_flush(vma, haddr, pmd); pmdp_get_lockless_sync(); if (ptl !=3D pml) spin_unlock(ptl); + pte_unmap(start_pte); spin_unlock(pml); =20 mmu_notifier_invalidate_range_end(&range); --=20 2.20.1 From nobody Fri Nov 29 05:50:55 2024 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3889549652 for ; Tue, 24 Sep 2024 06:11:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158280; cv=none; b=eQW/5PIicCuiX58zbCz5Cua38lTmFzBO6PflnsZ36pArWtR0WtIiwFRLCG3oFMLdlSncNcm1shcp11cTEOzpnEj1pwcEpjd1Zyll+93w0j+wUEVrIPepN2pPmWIgN2PGByZteKyXuBNam/0zVfJwWzpo9VMqJgF8U5dS4eiRqAM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158280; c=relaxed/simple; bh=VqnJhrj6tVNjxXhKEF9Of8+QKuUBH9ccpJxvoP5+WEw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nrQv977a5XGP5s/nOBTpUm/4032HXo52toYb9eRYhFPaa5y+teFeeamCMLZqkzW0zvC806MZcMdT97+RLgFdcu4g4nKHXaQ6M9utu34HgwqKXtfOKm3PAdZurzGN+Jvd9BlQjSDYuxCmeAS/stZvqFnAJBaZPhmN+ZM1nxYwT/I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=RnszBx6s; arc=none smtp.client-ip=209.85.214.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="RnszBx6s" Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-2053a0bd0a6so50987995ad.3 for ; Mon, 23 Sep 2024 23:11:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158278; x=1727763078; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=a5exoCzehntR4bd+ahks+h4WFwEAjFaMLu5nKDHss+4=; b=RnszBx6sZzzPUvqhlkpV/5mW/lrmgkrfeRHnma5bxErfR4uXGEkAFenx5f7u8xg3Jb Lhh5of8gIRl9JmX9FEQ3t5wYgWcRs6q2DvfF8D3ZqS3LqWYc6AnhOzaiNbATnSqT0sdF zIKbaB7oqOZ8Q3DdQwX/ZtyELYEh0xhy1zzciXIvH/0sIu2x9mCjJ75qWAlXYSp+Se+l sPB3JvhVzlZMYhsLtEBjFjhuTgaOemTrOogGltITsrD5U5RKkYFWJxiiF/rqRDXO3PM1 Haee8534RAEYCD9H1+PXm9s5Tv90R+Vwm0Z6YkfFdgjp7wKxIHCukD7Yb3ExiWwcfDKp ENIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158278; x=1727763078; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=a5exoCzehntR4bd+ahks+h4WFwEAjFaMLu5nKDHss+4=; b=YOYEf4v4H38rBAegO+kZa45SQOeW0Lo4/w+5jDiR2DMnAN0Hb51dPGWV8r7GsN3itl WQpmXz2BLCEXxMjwgtkam9uP099KiiI88exj7R3Gh2f0PxFRi9kSru27Vn6Wsc5sebA/ fkk301R1Cxcwhth1QM/P5YC80R/ypEodIVfzcxzi/L17sHgeR77DWvYCCLoLBHtb0UW6 elc/kf6m2upT1WQ0MyTMTn8GLyfq8ZfpHRzexaQdSxl9zODSO4hey3ZOIO9EKSu7OwST yeiWWPwJjHvUVepeZf+zGh1ExwUHlPWA9PM3xwDojLW2Hx03F++3SOJSmNxZlH/rAwDZ Ox5g== X-Gm-Message-State: AOJu0YzWM17Ot+8pz7zcbzT6ZyufNQYdBEITi0LC2aXppKXFqTH7tXc1 vDZwJTYfOZ8B8wvixiiFbHP4UQ9csyhSsloJY4WCsAOAkZe7MFYfQmPu/T6por4= X-Google-Smtp-Source: AGHT+IHw+OqtuQ5LCBY4YmqC9EJn7stUTlaTSc5oItaKMOqNO6mqJktvnwlzFLYt5xYGl9LymCl5gw== X-Received: by 2002:a17:902:d50f:b0:207:1709:380 with SMTP id d9443c01a7336-208d838ede3mr189786005ad.27.1727158278506; Mon, 23 Sep 2024 23:11:18 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.11.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:11:17 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 08/13] mm: copy_pte_range() use pte_offset_map_rw_nolock() Date: Tue, 24 Sep 2024 14:10:00 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In copy_pte_range(), we may modify the src_pte entry after holding the src_ptl, so convert it to using pte_offset_map_rw_nolock(). Since we already hold the exclusive mmap_lock, and the copy_pte_range() and retract_page_tables() are using vma->anon_vma to be exclusive, so the PTE page is stable, there is no need to get pmdval and do pmd_same() check. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/memory.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 6432b636d1ba7..c19cf14e1c565 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1086,6 +1086,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct= vm_area_struct *src_vma, struct mm_struct *src_mm =3D src_vma->vm_mm; pte_t *orig_src_pte, *orig_dst_pte; pte_t *src_pte, *dst_pte; + pmd_t dummy_pmdval; pte_t ptent; spinlock_t *src_ptl, *dst_ptl; int progress, max_nr, ret =3D 0; @@ -1111,7 +1112,15 @@ copy_pte_range(struct vm_area_struct *dst_vma, struc= t vm_area_struct *src_vma, ret =3D -ENOMEM; goto out; } - src_pte =3D pte_offset_map_nolock(src_mm, src_pmd, addr, &src_ptl); + + /* + * We already hold the exclusive mmap_lock, the copy_pte_range() and + * retract_page_tables() are using vma->anon_vma to be exclusive, so + * the PTE page is stable, and there is no need to get pmdval and do + * pmd_same() check. + */ + src_pte =3D pte_offset_map_rw_nolock(src_mm, src_pmd, addr, &dummy_pmdval, + &src_ptl); if (!src_pte) { pte_unmap_unlock(dst_pte, dst_ptl); /* ret =3D=3D 0 */ --=20 2.20.1 From nobody Fri Nov 29 05:50:55 2024 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1C63181AC6 for ; Tue, 24 Sep 2024 06:11:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158286; cv=none; b=VdopRHibZCNezPkR/7wcI3hkxj6+fvjGw/norCvDtxPYdCNoHX73Vw3lEKLxsE8UAZ5okumo/UCETwYRZnTq1ygb8b71n3Z83VFQT4M7esUkQF2cgseYvL1b14vDKfFmmclxDQGkD8U1/SkaC0+a+y37w8Hz2rUZQtZQA0/YZN0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158286; c=relaxed/simple; bh=fNpWWLmkLOIWq8WIIHHiPpHL0jJedeNIHk+yGdnzoao=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=trgtmDKVtvzePQdxvE+rmEPxryIAfGYaMl70/VVnyFkWVw92MncvURkEFX0mmNnB4syZCU5nKSzx1nx6Z9a/VihK2z3oGJ0Aiit96bukH7Z1SkHeYfJc8ZGd4iCwrzGmPbQQ+I6sG9nSrXBZvrccOsS/3IR90/02h7qN1gjyj2s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=kZ/VmX+K; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="kZ/VmX+K" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-208cf673b8dso50308815ad.3 for ; Mon, 23 Sep 2024 23:11:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158284; x=1727763084; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JYDx6olGBZytK5syFvtWNqesQWJmZ/VcuSiPjSyD9NM=; b=kZ/VmX+KaKMVOV2PxdWAxRQbKvOv9uU39GLuEwfcWA3AbXpNSdtkrnrmZpTTY4b+Pq tQ5kv9Em3kb8wgf8YfIglWNqHKD4yxTedlW9zeSB50KITvydf+9pYaErIYTmCxkzf6O6 TqMSQJja/R9TAdjA3rlMSekoXrbibG6KeWGR5KExGxgCpbElwd367FUvLpnckmxGpZFP n8/k4L0lV/E19U56OVnVim6j4munZ4nT2KCAK2vcciLAo0/J/k+HI90iD6paXcc25K6g z+N7kVdhk+CSUzvdlMpkNaNCA6UcOod7aHg8WNdxgE9wbCNEVXxHCHy+K2uZm9xjkD0s D1nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158284; x=1727763084; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JYDx6olGBZytK5syFvtWNqesQWJmZ/VcuSiPjSyD9NM=; b=HoLNGa6xAaNAGfF01R4YOqpXLmS3bx6fpQojUzQ8001yKSZyqui7i30zB+NJUWAvt1 t+SJjmFTT7y6QeMwqcN/rKNchdzGed1FyBj4FW25j8cHMD5nZD6em+xY0Cunk3RtM2Jo IVXto4ipk5GqSLfYySIKkHoMCwxtnj/bvpxX+AFvDL87H7rhLsp1kH6SbwZYpmorjDMi BFpRAdS0HU9hujCxiX3wQZRQVvlkyNzJ5tEhlUd6cUO6xg5LA7JEfF1zjDOq0bkPfp8Q 4oYIv7uGfXeXPSUHXmmAnc1neBWN2DcVHa9hb18UB+ExDFlbwY0qVYZk7l+aGEUGxkx2 dB5Q== X-Gm-Message-State: AOJu0YwiWJomXKG1TawNiRIpotEtrRzfLePSci+M5SkndL7psnsvhvvQ I4Zgjb/4a0ln4Ny6lbT9C6oZDxFKAApJ1H76ZGrDjv0aHj5Rp+akN5cJx+MAUzE= X-Google-Smtp-Source: AGHT+IEhn+iGj1vkzgTAalYoZbG5psMaFd6djM60OU+3J7JbD8pdYTlvGGhg/wgUQ8cNcQFEVSSWnw== X-Received: by 2002:a17:902:d483:b0:206:aac4:b844 with SMTP id d9443c01a7336-208d97e3f78mr215574535ad.6.1727158284398; Mon, 23 Sep 2024 23:11:24 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.11.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:11:24 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 09/13] mm: mremap: move_ptes() use pte_offset_map_rw_nolock() Date: Tue, 24 Sep 2024 14:10:01 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In move_ptes(), we may modify the new_pte after acquiring the new_ptl, so convert it to using pte_offset_map_rw_nolock(). Now new_pte is none, so hpage_collapse_scan_file() path can not find this by traversing file->f_mapping, so there is no concurrency with retract_page_tables(). In addition, we already hold the exclusive mmap_lock, so this new_pte page is stable, so there is no need to get pmdval and do pmd_same() check. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/mremap.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/mremap.c b/mm/mremap.c index 24712f8dbb6b5..9dffd4a5b4d18 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -143,6 +143,7 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t = *old_pmd, spinlock_t *old_ptl, *new_ptl; bool force_flush =3D false; unsigned long len =3D old_end - old_addr; + pmd_t dummy_pmdval; int err =3D 0; =20 /* @@ -175,7 +176,15 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t= *old_pmd, err =3D -EAGAIN; goto out; } - new_pte =3D pte_offset_map_nolock(mm, new_pmd, new_addr, &new_ptl); + /* + * Now new_pte is none, so hpage_collapse_scan_file() path can not find + * this by traversing file->f_mapping, so there is no concurrency with + * retract_page_tables(). In addition, we already hold the exclusive + * mmap_lock, so this new_pte page is stable, so there is no need to get + * pmdval and do pmd_same() check. + */ + new_pte =3D pte_offset_map_rw_nolock(mm, new_pmd, new_addr, &dummy_pmdval, + &new_ptl); if (!new_pte) { pte_unmap_unlock(old_pte, old_ptl); err =3D -EAGAIN; --=20 2.20.1 From nobody Fri Nov 29 05:50:55 2024 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3AB8484A22 for ; Tue, 24 Sep 2024 06:11:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158292; cv=none; b=WtgWAjj9fB0ZvkygYnL/GX6lr4WGxDdQDiU6qrKoJO7dhXIvwJAGFPRPObzDTD3hUEmAIQ0pX3Xy1yTbcWBoayW4zIsj5+TqB30vgLOS7t9QtjsBNedtrqMRMW55Rb4823IgvgF3HJwbVb0DjiOlAtuksNEr/cpAp6jmrWF/0pU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158292; c=relaxed/simple; bh=i+oN1kWXkG7Mo/Suv04Lea2nSB9WperDd3TZtQVOhCk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=YMU0o14gfTwLkwhWRmhiTlheVXimcJJr+xooBckzm1pXo8iNA0h3eUJJYko5FSxfUB625fjYxxLxLvQn1bDIDcoB8nrgMql3Z9eSiT1NNDtXABGJ2oF1C8GE/McfKniJfgR4XzFEUD/6z6NW+7N+TY8PwKU6XlplqcpQHAOG1F8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=elf8R8Ou; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="elf8R8Ou" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-2055136b612so61987355ad.0 for ; Mon, 23 Sep 2024 23:11:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158290; x=1727763090; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FxMRr8GFLSu76sONoxW9925XRbgCGcQv9BKiL/bcjLY=; b=elf8R8Ou887fRIUdHkxvQ2jFLR2BeH/wUg8Wt6Gfrf3C4XaKjxIRm68nQtgpiyZP83 A0iddivabe6fU5WWE/WcNv37WS9FWlt2SnG85bJV91iUq7YRlOhiuawCSoUUrKiC1yOs ycAQqxjgqpF1vydw8Vje/YbxAFJoPwomc/HkMF9rgdNVIV+d3iNdpGVqF9Tfrq1oMJOl QQaqx2AvbVSjhyDWymePgfW+9gj7uifsTAbXV1ENjBOuKfBFZpro0sOJp/ivB2Q4iMTE VyE+m8IzW3f2XH11DNKouzX9qF+lXG1RhA7q3Ll9ulReGvC86z2x5Qp48YTyyyjEvMl3 mjgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158290; x=1727763090; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FxMRr8GFLSu76sONoxW9925XRbgCGcQv9BKiL/bcjLY=; b=jrUXXrr2Em65liQYY7X2zFxfMIAnSKaKdX3L4pMkuZtF7YefWLmQk+etphvQ8oPdFs Bb9H83IAJRiGm7bc4DG2WuFaKa3Zkb54Ef/UCKpiZvM+88Q19lAQmosr9uIxow5BJa30 tS6dyZRsl0dnDSzsWfIxn+EEq2MOb5Loei+js49C1j+R1E4AQGg/ZNDJJkGSqcGMEVH7 jg5b6YpxLOPC2mt26AuX+eeMXzulrOoG9L3H9G5i3cy0jpCATU34ZJYCD/eJZ0Mjfjno /X8u2FS9Qy/WWWkqoSe9lTUPIKLjry9R2vg5evQD/YgEE8DwLiqCrisze8empFIfJvdh 1giw== X-Gm-Message-State: AOJu0Ywv3uji43++jga8aNGD2ofOwFHdJ/2vFqBo2SI3B3fpZ3mdXuIL biUOVstf0gV3r1oSQ3UEV1XkkOtZWB9NHMwK9UjqfTgu2LflREWfygZj6TY7Vxg= X-Google-Smtp-Source: AGHT+IGuU8VFNPNElLsd/e1qu4NWf2AelC2dd7GFKvCMDCyRlgITelf6+XB9J1CmvT70UykrnIRqmQ== X-Received: by 2002:a17:902:f542:b0:201:f70a:7492 with SMTP id d9443c01a7336-208d98603b2mr202721885ad.53.1727158290456; Mon, 23 Sep 2024 23:11:30 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.11.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:11:30 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 10/13] mm: page_vma_mapped_walk: map_pte() use pte_offset_map_rw_nolock() Date: Tue, 24 Sep 2024 14:10:02 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In the caller of map_pte(), we may modify the pvmw->pte after acquiring the pvmw->ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the pvmw->ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pvmw->pmd. Signed-off-by: Qi Zheng --- mm/page_vma_mapped.c | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index ae5cc42aa2087..6410f29b37c1b 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -13,9 +13,11 @@ static inline bool not_found(struct page_vma_mapped_walk= *pvmw) return false; } =20 -static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) +static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp, + spinlock_t **ptlp) { pte_t ptent; + pmd_t pmdval; =20 if (pvmw->flags & PVMW_SYNC) { /* Use the stricter lookup */ @@ -25,6 +27,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, sp= inlock_t **ptlp) return !!pvmw->pte; } =20 +again: /* * It is important to return the ptl corresponding to pte, * in case *pvmw->pmd changes underneath us; so we need to @@ -32,10 +35,11 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, = spinlock_t **ptlp) * proceeds to loop over next ptes, and finds a match later. * Though, in most cases, page lock already protects this. */ - pvmw->pte =3D pte_offset_map_nolock(pvmw->vma->vm_mm, pvmw->pmd, - pvmw->address, ptlp); + pvmw->pte =3D pte_offset_map_rw_nolock(pvmw->vma->vm_mm, pvmw->pmd, + pvmw->address, &pmdval, ptlp); if (!pvmw->pte) return false; + *pmdvalp =3D pmdval; =20 ptent =3D ptep_get(pvmw->pte); =20 @@ -67,8 +71,13 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, s= pinlock_t **ptlp) } else if (!pte_present(ptent)) { return false; } + spin_lock(*ptlp); + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pvmw->pmd)))) { + pte_unmap_unlock(pvmw->pte, *ptlp); + goto again; + } pvmw->ptl =3D *ptlp; - spin_lock(pvmw->ptl); + return true; } =20 @@ -278,7 +287,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *= pvmw) step_forward(pvmw, PMD_SIZE); continue; } - if (!map_pte(pvmw, &ptl)) { + if (!map_pte(pvmw, &pmde, &ptl)) { if (!pvmw->pte) goto restart; goto next_pte; @@ -307,6 +316,12 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk = *pvmw) if (!pvmw->ptl) { pvmw->ptl =3D ptl; spin_lock(pvmw->ptl); + if (unlikely(!pmd_same(pmde, pmdp_get_lockless(pvmw->pmd)))) { + pte_unmap_unlock(pvmw->pte, pvmw->ptl); + pvmw->ptl =3D NULL; + pvmw->pte =3D NULL; + goto restart; + } } goto this_pte; } while (pvmw->address < end); --=20 2.20.1 From nobody Fri Nov 29 05:50:55 2024 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 47FD784D0F for ; Tue, 24 Sep 2024 06:11:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158298; cv=none; b=a785mdmKS1G5p9OtV0ZqNbjZmZKdcWgAtnn7RQjvchYg7f9wwo8VbuQIyTHLBwOOu0TD3jpSCBCvyhMcrI4jeMCuc2iN58VTd3G8Hw2CyZqw3TMQG8JN1daFa6PznBQlwPilD6i2xF6asKRzWatl4/eHUGGsJJqP7KOhYbzRTZo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158298; c=relaxed/simple; bh=M/64Ci4YVvyqTSzbm0kd6G+2YPtWZDsuLrmsgyWUzVc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Qjrt8wyI/goVFZcZwLF5DCkgdIpQ+IZ0k62Q1gpHqjw9FYZTc+16/B/NSXWf0VwGpLduj8kdc+cmWWXV0mfDVph5Z95ky4MCDC4VR5/VV3fcQ6QFrGh0z2FzcdKMef8iQLjbllTxjU5X56kC/DSYexIcJmG3wPLQJ7qrridiwRY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=BfTQjvTh; arc=none smtp.client-ip=209.85.214.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="BfTQjvTh" Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-2053525bd90so47136865ad.0 for ; Mon, 23 Sep 2024 23:11:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158297; x=1727763097; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RnMCKwBqnYl3LJbu0M9HQTu7wcymlNTkAYqiZlfWQ7I=; b=BfTQjvThNxtE0DZX/8NgsgaEL9Hnxin+ktQXh5DZHE8TILZ83UD5tTmm8PPOGZzkuz HNYVEUeMjNYbemaqyCDyuC6+LRxIh5T06vTd3LiiqQp0mIY+5CnuXr2m7jfFR3w4ljcC UnzEKKZXW0LV4Rik4ctd5TGWfxNvYRaM/VDZ2LFk2dpbKuSUBGmPT0Y+hvnCdGpO9rE0 QPPRDH4VlNOIpBS+3Y/AvXBrfGy+s50jCOzgN6Ef74vn9ngMPROoTaQzvrfV+O5SL8+U xHH50aEEi9L+SpTX6kZtOrrzJx9+kDIbDwofkc9F7V3aMFCIgUMOydqNNORFFjfn921j qivg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158297; x=1727763097; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RnMCKwBqnYl3LJbu0M9HQTu7wcymlNTkAYqiZlfWQ7I=; b=VhnBvpTcwP1y9DMZ+fRg7vUt5UAzDxNq3i43DvPwnwZpxkL5zPkyRgBjwVtmE8AcHI w8dGDE3WPjJIb0Y1ZvMV+tBc4w9PhBbsJzzQ1h/njeNrHNG1CmlwZrgB716jAJvI1f25 Ktz+ktcZ7SCl0E+32OPH0xDdKwGStnmMtKvtcHvWduAORZqUGaJ0Z283JxcinsaKx2ic IuHZgPABC3v4xSzIPxT2hOKFMnfQr6H1R98Y0tf7g/I9+/iWG3Lh2vaFjJHXDZpN0+us IvlNdbwF8qXSdW9lShTJ1NH9pQlMi0RLvO+MZadmbQhUI7gQJXyS/r5mnUWC05GrVUjl xH/A== X-Gm-Message-State: AOJu0Yxb+6Rxk9ow+swcQsIBXa0DQueED+51HuWlxXr+CmtKFJMQCXpD FMidt8VXVYznQWqGj1gHv+hQIvKJEt8FP31rbsWB3c2haUoZNlRkpM9vaqRLyt4= X-Google-Smtp-Source: AGHT+IFBI5Ned/vbTuuRRzdnv79MFASKoi0/m2fLpf4am0H6y8GNq4qW/RgfzVT9no9zcYiBkg/Xzw== X-Received: by 2002:a17:902:ea07:b0:20a:f013:ddb5 with SMTP id d9443c01a7336-20af013e29dmr19152265ad.59.1727158296610; Mon, 23 Sep 2024 23:11:36 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.11.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:11:36 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 11/13] mm: userfaultfd: move_pages_pte() use pte_offset_map_rw_nolock() Date: Tue, 24 Sep 2024 14:10:03 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In move_pages_pte(), we may modify the dst_pte and src_pte after acquiring the ptl, so convert it to using pte_offset_map_rw_nolock(). But since we will use pte_same() to detect the change of the pte entry, there is no need to get pmdval, so just pass a dummy variable to it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/userfaultfd.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index ce13c40626472..48b87c62fc3dd 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -1135,7 +1135,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t= *dst_pmd, pmd_t *src_pmd, spinlock_t *src_ptl, *dst_ptl; pte_t *src_pte =3D NULL; pte_t *dst_pte =3D NULL; - + pmd_t dummy_pmdval; struct folio *src_folio =3D NULL; struct anon_vma *src_anon_vma =3D NULL; struct mmu_notifier_range range; @@ -1146,7 +1146,14 @@ static int move_pages_pte(struct mm_struct *mm, pmd_= t *dst_pmd, pmd_t *src_pmd, src_addr, src_addr + PAGE_SIZE); mmu_notifier_invalidate_range_start(&range); retry: - dst_pte =3D pte_offset_map_nolock(mm, dst_pmd, dst_addr, &dst_ptl); + /* + * Use the maywrite version to indicate that dst_pte will be modified, + * but since we will use pte_same() to detect the change of the pte + * entry, there is no need to get pmdval, so just pass a dummy variable + * to it. + */ + dst_pte =3D pte_offset_map_rw_nolock(mm, dst_pmd, dst_addr, &dummy_pmdval, + &dst_ptl); =20 /* Retry if a huge pmd materialized from under us */ if (unlikely(!dst_pte)) { @@ -1154,7 +1161,9 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t= *dst_pmd, pmd_t *src_pmd, goto out; } =20 - src_pte =3D pte_offset_map_nolock(mm, src_pmd, src_addr, &src_ptl); + /* same as dst_pte */ + src_pte =3D pte_offset_map_rw_nolock(mm, src_pmd, src_addr, &dummy_pmdval, + &src_ptl); =20 /* * We held the mmap_lock for reading so MADV_DONTNEED --=20 2.20.1 From nobody Fri Nov 29 05:50:55 2024 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 671A857CAC for ; Tue, 24 Sep 2024 06:11:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158304; cv=none; b=pmTooZXMceRU/H/fxCEDXy2+PrMPfRvCeAdgWaLdbnr39GEX4+lTtS+YJ3G097h5nBcrZzNUW03OEDHR7FsnSU1t7r+Mn9aCDaHAihk6iLiPS4iYlNLpArMLSTja5sPiwVgsAD7R8OINtGtb6fGjdrVgWsYx9F1bHzH6W5UhEC4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158304; c=relaxed/simple; bh=JRuw2Pk6LLh+NGlcp79p62JmtEiArxU8q3vnjCYzwiM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=auOSgUZ9v0zDVtDeDTHa94FrY2dbZIzpn1Jvy5WTzBMJOM4cHM2DS4txA5oGt0i/4xhU2FPBoEg37jrv4+5Jhhgql5sTXwlRESV6z8Z7lVUPxiBr9kRnHxpjaRfFCgZKFW+ly0vmEfYwOrEIZfcqRtNHszX9S5H4mJ9xox4Zc+w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=g0aycdaP; arc=none smtp.client-ip=209.85.214.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="g0aycdaP" Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-2053525bd90so47137605ad.0 for ; Mon, 23 Sep 2024 23:11:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158303; x=1727763103; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fwq8DT6JeNdavhckGZI91GgKWXC+kbG6stmMQ1HgBAg=; b=g0aycdaPasdgAj12pnyuSOQ9kmY0v9M+VqyrXfM61dFND85zQJ7z1usITUOVNfMO6H dphpLXR9Qa7wzqu+d2PcAk3jxDRFX+UPV1iKuPhL06B2LXCQr+1YTCuSMBVVPYoqarXp FjWCcyBiyF7Q3gWvwGxTGjjaWiWVyVb5tfiKk7oYY32l9AOhWHBPkQKFIzGwsjfco5bp ZDiU6V9d6N+iF1Odc2Ib0zhA8PZ/pGZme4S83rPx5JixJ8ff7t2m81lNY0jNuXrgRrbe WmU5k5A2MJPu2YUYoawTtZGa33hPdio6dzi9gFHuL7LfSjmE44tloAwAHyPya/bU6O9R eBHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158303; x=1727763103; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fwq8DT6JeNdavhckGZI91GgKWXC+kbG6stmMQ1HgBAg=; b=oFsBOl+psjLCqOouRxzqAZhcS9mg2gLDPl9JL0J6X2vriY2Bu+Z7Hzk+E4X3XzEm5x iYnPOjaGNRkqKU7TVhSqOVFhPoYwoJJ1AjeRpl3qKydxVt1M2yCZVhZ7E+BpFupuaiMA FQ+5w2jvKiZD+IPJ6afuQL2u64amNYgvzUTQ01INLZuwNBN2F2UqCZE4Jp94f/kvyPPI 46bpkk888CXIxRNFl/GF6TbKMIoH1y1Mf/xnYnnKBgRertHw6h+tN9ONaGPlZvze2qSq 7L5MR5zdZW+talCo/U5jZ3ORgB0ziB+IIXxVJ+sTBHYhvoCWzPg6R90wDDvszCLz5fyF 7uiA== X-Gm-Message-State: AOJu0Ywg3LySRYIEKihOjE0Ic+3NVSspQTWY9106G72h/RP/Z7ZHaAcf CmlSWHJT1p+zMmepkEB78K14Q7g5Wqm+lekMyqq/gOkkpazhCmunMRABh7XUYj8= X-Google-Smtp-Source: AGHT+IFl5AsK8a8BNnydzmhf2ct6wAj0Fi3O6gM7MY+2JS5/M4Off4jZQYnq6Yk2iDK5ETk+W1TbBw== X-Received: by 2002:a17:902:ec87:b0:206:b915:58e with SMTP id d9443c01a7336-208d83af7b0mr191498925ad.22.1727158302727; Mon, 23 Sep 2024 23:11:42 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.11.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:11:42 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 12/13] mm: multi-gen LRU: walk_pte_range() use pte_offset_map_rw_nolock() Date: Tue, 24 Sep 2024 14:10:04 +0800 Message-Id: <6efef5316d7468d13ed5f344452a0b872481972b.1727148662.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In walk_pte_range(), we may modify the pte entry after holding the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pmd entry. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song Acked-by: David Hildenbrand --- mm/vmscan.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 749cdc110c745..bdca94e663bc5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3375,8 +3375,10 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long= start, unsigned long end, struct pglist_data *pgdat =3D lruvec_pgdat(walk->lruvec); DEFINE_MAX_SEQ(walk->lruvec); int old_gen, new_gen =3D lru_gen_from_seq(max_seq); + pmd_t pmdval; =20 - pte =3D pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK, &ptl); + pte =3D pte_offset_map_rw_nolock(args->mm, pmd, start & PMD_MASK, &pmdval, + &ptl); if (!pte) return false; if (!spin_trylock(ptl)) { @@ -3384,6 +3386,11 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long= start, unsigned long end, return false; } =20 + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + return false; + } + arch_enter_lazy_mmu_mode(); restart: for (i =3D pte_index(start), addr =3D start; addr !=3D end; i++, addr += =3D PAGE_SIZE) { --=20 2.20.1 From nobody Fri Nov 29 05:50:55 2024 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8132D85283 for ; Tue, 24 Sep 2024 06:11:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158311; cv=none; b=JxaUgEBAuVJPgbNI/ysRKRceD7fR707Vm+8TnG8vFzOGPPff2iMmKlEdAIUMtMrWzVq5AzZUtKb2DroCniok8d0szUoaSqm3IrG0HdH+oYTaV04LvkR/A2hwIhwPc8aFiUsghFhIDdhth9ZuNUjESg78P9IpYEEZkIROgfVHe2g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727158311; c=relaxed/simple; bh=v9vdYPzOdAZ9IeVDUmMtwCMvWeEwkmvD5HbQnix6zVg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dPyS4Fa4EFPFA0fNe9r45coSSlrWHqMBDMV51wpAO3EgMwmnQhxYB7tH+PZ6stY7341j5AplPiuDxitelmUTx0k/VkTXFOCDi2GcmsVte3Mt2pL6sRl+9aig98SKt91thHkxGJe50+9mGYmVo8xFOAaTHBlykyumC7z/B4XR/0k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=YQZSlllJ; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="YQZSlllJ" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-20543fdb7acso40327665ad.1 for ; Mon, 23 Sep 2024 23:11:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727158309; x=1727763109; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=e0T9lvI1pcCPfr6v8oRbivQAnqkZ5MJutHJ+/bNj+/g=; b=YQZSlllJC6ojJjuXM4mrfdnYxfnGNccExsJd5xEdGqEEkLnASdok7vzGmkVaQrT6s+ cOyMjcUHW1O8HT+P7M3g4Us6K2OSA1KW2CeJuEK1g84fc4fHi6FRz371IZZT4EXg22Pr wmsG76cBBCf2VRGjNlfBEYElXYGuCV9u8e2ZLGnGrQSG5VJMIyXF1xCR9z5Jq6j2lnvW ofVkJev1Q4EuPpu81YBW8hTMr0XEw0DuLKZCXJC9cACOiKVtjRzRlQwIdUdKuQnec1Kx QfjcHeWQr+Wsf6hT32xP7wN64XPmXypq909jC1eWrhq8y18EI4ilQfjLYE9pHf2nVqKQ szLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727158309; x=1727763109; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=e0T9lvI1pcCPfr6v8oRbivQAnqkZ5MJutHJ+/bNj+/g=; b=K1l42sO7/YS2ieVsQQ7VKcYbOubhgYL7xBjHhXATdrs9I7/amjLVpf7NxEhiyg3ue0 dsHd3UpBjZjo27xn85DSbwXLVj62N95RUuwKv6vDXrObprElCc7+hs2pa5dvj98IYhhx a0C3XkC/8/UQ3eKY7Qi970aaNB47NLbbUtM7FnxOT1z2m37dD0vHI1+pZprZ9MHaHt7V veIz2crzuZ4wjYe7NsUIMkTz1moDEjoEAJRLoON+DFojDKZq3CPhWgvS8IzSq/SSP60z nPixZKDfZxncFqNEec/ZabGeR/BO+a7rwF+CeqGeUIMlZ+a5BmtRq7GrKhWy5+1nodEv W4cA== X-Gm-Message-State: AOJu0YyIFC3iEKKOo268aMo6Sh7DkS23aSHpi43UVTP3juyyNQ7O+s7N vQbLizPp/Qxi25LjdnfSGjUYRZ4H24ul2tV0KSLpX8pSFKFCURYzoEUnZ8pWq4k= X-Google-Smtp-Source: AGHT+IGkJQOlzg3g752NhI9RquMqodCfvTZ7obYXRPzPmN4sDO5P6Za1vWoZJJ9nTCdoHXRNF+rvtA== X-Received: by 2002:a17:902:ec8a:b0:202:311c:1a59 with SMTP id d9443c01a7336-208d83b6ce8mr195135205ad.27.1727158308785; Mon, 23 Sep 2024 23:11:48 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20af17229c9sm4344885ad.85.2024.09.23.23.11.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 Sep 2024 23:11:48 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v4 13/13] mm: pgtable: remove pte_offset_map_nolock() Date: Tue, 24 Sep 2024 14:10:05 +0800 Message-Id: <8eb7fcecf9ed8268980d0bd040c0a4f349cbca8f.1727148662.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now no users are using the pte_offset_map_nolock(), remove it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song Acked-by: David Hildenbrand --- Documentation/mm/split_page_table_lock.rst | 3 --- include/linux/mm.h | 2 -- mm/pgtable-generic.c | 21 --------------------- 3 files changed, 26 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/= split_page_table_lock.rst index 08d0e706a32db..581446d4a4eba 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -16,9 +16,6 @@ There are helpers to lock/unlock a table and other access= or functions: - pte_offset_map_lock() maps PTE and takes PTE table lock, returns pointer to PTE with pointer to its PTE table lock, or returns NULL if no PTE table; - - pte_offset_map_nolock() - maps PTE, returns pointer to PTE with pointer to its PTE table - lock (not taken), or returns NULL if no PTE table; - pte_offset_map_ro_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; diff --git a/include/linux/mm.h b/include/linux/mm.h index 9a4550cd830c9..e2a4502ab019b 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3015,8 +3015,6 @@ static inline pte_t *pte_offset_map_lock(struct mm_st= ruct *mm, pmd_t *pmd, return pte; } =20 -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index 262b7065a5a2e..c68aa655b7872 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -305,18 +305,6 @@ pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr= , pmd_t *pmdvalp) return NULL; } =20 -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp) -{ - pmd_t pmdval; - pte_t *pte; - - pte =3D __pte_offset_map(pmd, addr, &pmdval); - if (likely(pte)) - *ptlp =3D pte_lockptr(mm, &pmdval); - return pte; -} - pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp) { @@ -374,15 +362,6 @@ pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, = pmd_t *pmd, * and disconnected table. Until pte_unmap(pte) unmaps and rcu_read_unloc= k()s * afterwards. * - * pte_offset_map_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_m= ap(); - * but when successful, it also outputs a pointer to the spinlock in ptlp = - as - * pte_offset_map_lock() does, but in this case without locking it. This = helps - * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that = time - * act on a changed *pmd: pte_offset_map_nolock() provides the correct spi= nlock - * pointer for the page table that it returns. In principle, the caller s= hould - * recheck *pmd once the lock is taken; in practice, no callsite needs tha= t - - * either the mmap_lock for write, or pte_same() check on contents, is eno= ugh. - * * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like pte_offse= t_map(); * but when successful, it also outputs a pointer to the spinlock in ptlp = - as * pte_offset_map_lock() does, but in this case without locking it. This = helps --=20 2.20.1