From nobody Fri Nov 29 00:48:46 2024 Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 28B76224D6 for ; Thu, 26 Sep 2024 06:46:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333218; cv=none; b=fyAUX9Ltu5ZCjEZK1SAuvfJfb4O93eXr01O9558ilMpoyBFjW58Je5o8z38tqW4zOE0sfbohJ/cTnDd+tnNXSi+8ElDoC5T7Du98XNYxyMARXyIwDSEOQt56p7uce98FklUHOS9DFSzUNhIJ6Bek1cqCisVu4S/hdCUg9jx7+08= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333218; c=relaxed/simple; bh=BaPXwB7UqH53ZIAmolBim0wq4X5L/EYzHYfb/hyU8GE=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=t6mCsrYl5pdhhIEwNbdxBMiibdX5HD+wXjSUc6CZeX5m3GTFfIiGop1c9YAg/z/fMFdGOSnyexyv1zpnm1NEhwFWKgMgiHY+PzmXnN1hiiuMxBfqVQ+hUsJVly4fNjPkZTnttA9+CKc7fkI7+HeRAT0CBQ4zNLnMUuERIYQ49rE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=NNkKpxYo; arc=none smtp.client-ip=209.85.222.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="NNkKpxYo" Received: by mail-qk1-f179.google.com with SMTP id af79cd13be357-7a9ac0092d9so66625685a.1 for ; Wed, 25 Sep 2024 23:46:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333216; x=1727938016; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VEMasL/thV9Tj1Tfta4uIol3BYRuzPr3NckO1OF+52k=; b=NNkKpxYo7407DlqqocTygZh1oqeiubN/3l0lv4nc647FeYnv3taUT6ev0nmd2ANbC4 HQWxYDIspYWUgwI8ktcR9HlsCROV/joheXNQohkrUtPR4acUEVhMpKvOzVNRdVN4BA9a sWNmdb1z2lnu7gS8nhGQSb7NNCsT/8TjHvoEVtJnYjmCBCPp6/JX726cDgAYmUfKR0xQ 7RQwHpRvq8K70OGnyQRjuyi0vlHgmrcyZjZcuPN1nAkMCR0ymqL9o3pWvqhsqVzlxapi GJOAX+jbBwRAcdctQHzOAaLTevVp6n/OqKjQlkInmYw/qNcS9pfA81jp5dJ3dKlChCw8 EX7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333216; x=1727938016; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VEMasL/thV9Tj1Tfta4uIol3BYRuzPr3NckO1OF+52k=; b=vKMRv6llCCEuwvGeXNMKJFc0LdNsbvHcDerwsgqCVloo5FJB3hnsjwUAO7+UGbmX0f H88fgAxmtrkw99FTaqmMMPySWr7CWCm0hn5mS6/fp6ASJRl3/g98L4GkVXXsgUzJf4zy IaFLFqkh9P52pfZqGGHTTj1URghUXqw9D2uYztf+iT7BPX0GvSYp7lrik8ZyOqPVjg1/ L64efhj8v9vuPHVQnVl8ekhvuD1qsOHSwSCRgNPqNr93GFzbHOUWtQkRC5jgPzYw32R8 bV5gszY1l7W6A9yaDS1vebL3NcomdgMz+Bl+OLdsxp8/hml0u6P4DJehZQXFhI3376Bp P4hg== X-Gm-Message-State: AOJu0YzXbcncLFjO1TemyuqcnZMkyIBK3R3C19Uypoo/VowslGZVq9CC +FQpU8JMhK80+gjkC5EwfmZ8JCx3BJCfhuo8TosyM7Sf2+Q9wjRBykD82W6C1Ic= X-Google-Smtp-Source: AGHT+IFdBHDg/pcI8fYkiDBKrKz5/bcsNyjF/Vh4ia+kT/ICQCu69/2acL+Eh3ZQbIhYSZdsv978Jw== X-Received: by 2002:a05:622a:20b:b0:458:4b6b:ff16 with SMTP id d75a77b69052e-45b5def4888mr86143491cf.32.1727333215740; Wed, 25 Sep 2024 23:46:55 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.46.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:46:55 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 01/13] mm: pgtable: introduce pte_offset_map_{ro|rw}_nolock() Date: Thu, 26 Sep 2024 14:46:14 +0800 Message-Id: <5aeecfa131600a454b1f3a038a1a54282ca3b856.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, the usage of pte_offset_map_nolock() can be divided into the following two cases: 1) After acquiring PTL, only read-only operations are performed on the PTE page. In this case, the RCU lock in pte_offset_map_nolock() will ensure that the PTE page will not be freed, and there is no need to worry about whether the pmd entry is modified. 2) After acquiring PTL, the pte or pmd entries may be modified. At this time, we need to ensure that the pmd entry has not been modified concurrently. To more clearing distinguish between these two cases, this commit introduces two new helper functions to replace pte_offset_map_nolock(). For 1), just rename it to pte_offset_map_ro_nolock(). For 2), in addition to changing the name to pte_offset_map_rw_nolock(), it also outputs the pmdval when successful. It is applicable for may-write cases where any modification operations to the page table may happen after the corresponding spinlock is held afterwards. But the users should make sure the page table is stable like checking pte_same() or checking pmd_same() by using the output pmdval before performing the write operations. Note: "RO" / "RW" expresses the intended semantics, not that the *kmap* will be read-only/read-write protected. Subsequent commits will convert pte_offset_map_nolock() into the above two functions one by one, and finally completely delete it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song Acked-by: David Hildenbrand --- Documentation/mm/split_page_table_lock.rst | 7 ++++ include/linux/mm.h | 5 +++ mm/pgtable-generic.c | 48 ++++++++++++++++++++++ 3 files changed, 60 insertions(+) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/= split_page_table_lock.rst index e4f6972eb6c04..08d0e706a32db 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -19,6 +19,13 @@ There are helpers to lock/unlock a table and other acces= sor functions: - pte_offset_map_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_ro_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken), or returns NULL if no PTE table; + - pte_offset_map_rw_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken) and the value of its pmd entry, or returns NULL + if no PTE table; - pte_offset_map() maps PTE, returns pointer to PTE, or returns NULL if no PTE table; - pte_unmap() diff --git a/include/linux/mm.h b/include/linux/mm.h index e9077ab169723..46828b9a74f2c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3017,6 +3017,11 @@ static inline pte_t *pte_offset_map_lock(struct mm_s= truct *mm, pmd_t *pmd, =20 pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp); +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp); =20 #define pte_unmap_unlock(pte, ptl) do { \ spin_unlock(ptl); \ diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index a78a4adf711ac..daa08b91ab6b2 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -317,6 +317,31 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd= _t *pmd, return pte; } =20 +pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + pte =3D __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp =3D pte_lockptr(mm, &pmdval); + return pte; +} + +pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, pmd_t *pmdvalp, + spinlock_t **ptlp) +{ + pte_t *pte; + + VM_WARN_ON_ONCE(!pmdvalp); + pte =3D __pte_offset_map(pmd, addr, pmdvalp); + if (likely(pte)) + *ptlp =3D pte_lockptr(mm, pmdvalp); + return pte; +} + /* * pte_offset_map_lock(mm, pmd, addr, ptlp), and its internal implementati= on * __pte_offset_map_lock() below, is usually called with the pmd pointer f= or @@ -356,6 +381,29 @@ pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd= _t *pmd, * recheck *pmd once the lock is taken; in practice, no callsite needs tha= t - * either the mmap_lock for write, or pte_same() check on contents, is eno= ugh. * + * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like pte_offse= t_map(); + * but when successful, it also outputs a pointer to the spinlock in ptlp = - as + * pte_offset_map_lock() does, but in this case without locking it. This = helps + * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that = time + * act on a changed *pmd: pte_offset_map_ro_nolock() provides the correct = spinlock + * pointer for the page table that it returns. Even after grabbing the spi= nlock, + * we might be looking either at a page table that is still mapped or one = that + * was unmapped and is about to get freed. But for R/O access this is suff= icient. + * So it is only applicable for read-only cases where any modification ope= rations + * to the page table are not allowed even if the corresponding spinlock is= held + * afterwards. + * + * pte_offset_map_rw_nolock(mm, pmd, addr, pmdvalp, ptlp), above, is like + * pte_offset_map_ro_nolock(); but when successful, it also outputs the pd= mval. + * It is applicable for may-write cases where any modification operations = to the + * page table may happen after the corresponding spinlock is held afterwar= ds. + * But the users should make sure the page table is stable like checking p= te_same() + * or checking pmd_same() by using the output pmdval before performing the= write + * operations. + * + * Note: "RO" / "RW" expresses the intended semantics, not that the *kmap*= will + * be read-only/read-write protected. + * * Note that free_pgtables(), used after unmapping detached vmas, or when * exiting the whole mm, does not take page table lock before freeing a pa= ge * table, and may not use RCU at all: "outsiders" like khugepaged should a= void --=20 2.20.1 From nobody Fri Nov 29 00:48:46 2024 Received: from mail-qt1-f181.google.com (mail-qt1-f181.google.com [209.85.160.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A6E8718BC11 for ; Thu, 26 Sep 2024 06:47:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333226; cv=none; b=YIvejsFn2IsSOlMlxhmMGVojmc9m9pLwdBhoIJlkwjH4bFKiO+dteLL0p2EsRHxYC2DxEb5YattSOiGI7HFw56beP7IK3X+CdN3XBnkxHIAX46Or7i6y4PqYqIngkWFSIrXXWkuriKhyIpTByzh2aRAJMrZb79fNTOKr7WhzZSo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333226; c=relaxed/simple; bh=Ks+ctZjoe6oexIKNJorWHsj/FHxBIY1SArqgBxdFKSU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=aQ9yI6cBaszecsBxzBU8DLx1sm4rm4kvF5HpGP3sVmZA0mi12QXJLsurk3VDvWsscPazc/zG0Rfs4uXOWDT8i2e4i/HdQ8uqJh5aRqKfuyYkHzzd8KdOId2L+K2TAfP9lbzZ8SBRD3y44AHMTmrYk2eAOuo8WGSaFmpCZADtPiA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Tr/fvUKc; arc=none smtp.client-ip=209.85.160.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Tr/fvUKc" Received: by mail-qt1-f181.google.com with SMTP id d75a77b69052e-4581e7f31eeso4520651cf.0 for ; Wed, 25 Sep 2024 23:47:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333223; x=1727938023; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7WBwKEY9MWquOlo01BPdpzLz4TZdRIsQgPATlcDwMyk=; b=Tr/fvUKcc58vOt9T5WNPQo5DDKH4D5fwF3PLuyXco5Bry1fpyt9uMM6WYlM3gLSOJC qXhRmuOvoEsnwppLbU7qOcDV9fFrBix55WOrZj/vFJHUuAFNzYW6SuDTym1FuXM/p0dc ZfLrK7lnUAMECNqDg8gDoy/Add4CPKaIeA7Y+0f8fOWAhH0sV8CfTwjFcZZ4/f/rLYo9 pg3phgzpvGEs9ZqF+WaX6Z8vD6EznVSExwi7314QlF1hiq+G4n7FdsURkx9JRtPIQB3Z UbPtpF5D98IPg+aSFPTj1iRxObDFfWL+yUTXeln+87HoBblQYhNzLxFqSHkX4wUZvF1H NN6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333223; x=1727938023; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7WBwKEY9MWquOlo01BPdpzLz4TZdRIsQgPATlcDwMyk=; b=SusIVyAVJunkJYSBUHpqG0Go2hLYK5dFQAeTZpwddb09ZhK7o1fsTxKzVWiBMZodoc 7ngjEn5KnKohEg4kmfsLZ6lB8OrE8g+TIBLp3JYnzp3bT7dGYuQ6DNXSqC/gal5HO35Y lWYtlTxifFtu0aqxDkt78S8f3UzG04YnQa8vLCUp66eYXXLR89kVJBmvHpf07QBGXVpe I+mQpeH2njZB4MjTPqJoT2FIxMXZjO/pMMiSOFnCAdGYGx6RMtrHxYiscIFUe+ziIu2Z L0RQtHVx/Y/ZfvL8iJqnkWpTfYLMUXlF1HyvPr+357st7iDG4mvFAoqIXI8QGdzYAzkl u6qg== X-Gm-Message-State: AOJu0YzChHFfwtulKfEzswOmm6RJM3buxi+R2OJHIGH6eMMWOVlecgC9 5AvHkHuiXGFooE5IqXzMsvNOk2quDrGQFQLxQ3iOOyC+pRIBrlcwMvE7iiepqfo= X-Google-Smtp-Source: AGHT+IHKcdRFC9q2GD6595fUPe4NgxH7f7bGHM/1jVFUOYjtanMZU2hJPsIx01XdKDMweQg9NQ43+w== X-Received: by 2002:a05:622a:105:b0:458:4c86:4564 with SMTP id d75a77b69052e-45b5e0952b0mr82475131cf.37.1727333223424; Wed, 25 Sep 2024 23:47:03 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.46.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:47:03 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 02/13] powerpc: assert_pte_locked() use pte_offset_map_ro_nolock() Date: Thu, 26 Sep 2024 14:46:15 +0800 Message-Id: <42559e042eb6fc3129a40f710d671712030646b4.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In assert_pte_locked(), we just get the ptl and assert if it was already held, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- arch/powerpc/mm/pgtable.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 7316396e452d8..61df5aed79894 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -398,7 +398,7 @@ void assert_pte_locked(struct mm_struct *mm, unsigned l= ong addr) */ if (pmd_none(*pmd)) return; - pte =3D pte_offset_map_nolock(mm, pmd, addr, &ptl); + pte =3D pte_offset_map_ro_nolock(mm, pmd, addr, &ptl); BUG_ON(!pte); assert_spin_locked(ptl); pte_unmap(pte); --=20 2.20.1 From nobody Fri Nov 29 00:48:46 2024 Received: from mail-qk1-f169.google.com (mail-qk1-f169.google.com [209.85.222.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2A02718786B for ; Thu, 26 Sep 2024 06:47:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333233; cv=none; b=lHbX/6NgX4MlAG2ynZWOADgQ2eXC6nOJw1nSjdENHkE5BGvYo3oLmaKVZ+FZVBtwP+Rovx8r2Ehvt6ri5p6wc6Ee3Anv4mC8xfZ5SNPW+I490VbbgD6+Czped4di6tVh0YAgaaLxzOoEDl1zO0w1ZSR5YwfcbjNRaqgmf81+trA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333233; c=relaxed/simple; bh=CYi+EZMylLlMSCqw/uJB5X9IiL/Hx4QgVrtaCgCZ208=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=W3pTBEKNfgMJVVaV5NSOy9shIzb6iHSmnvWwzwLTThH6eWhnjPmW3ZZZ9wdH/MJ18LeXKdIrqqFPxO9Ve20XklSGesHqkJGq6cXtEyY4cxfBpJm+SN92zOXf4UrY8ofXW5u0FtflObwbOMlltqtsYeQxv61q5pMTZheAv+7cpps= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=LBOn9E4E; arc=none smtp.client-ip=209.85.222.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="LBOn9E4E" Received: by mail-qk1-f169.google.com with SMTP id af79cd13be357-7a99eee4a5bso52392585a.0 for ; Wed, 25 Sep 2024 23:47:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333231; x=1727938031; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BmENsM1LVZMrc0W501TyzYIERT8c75Mhan1AJ9vs73Q=; b=LBOn9E4EJdvphOHdtKRRdaxL3J8x3ZkXBrQT9wK0oPsWM2/bBHwWpBgGemTXH+iE+u 2EYftdyOgDODyl8YZS5t3Nlr45b7W+mbAcBZyry5B+1NmULZSKV2EbNTRXCQTTxw9IQ3 2q5bWuX1DNGv3x0XIrlI1yizowsjfl7Mso95qjnZakdtv/MSanKHjij718OjfP3f/P7y 2Q9jI+VeOABoEosklEjAxcvPQizzfIayhk99ighYb7qfb+Th+8DTcNwMHLOnYR2GYH2M qpmq3bf4fHc/aKibsAR2DkZfnlTa5n91N4TcXHmqxll+fEfW4P4ST/pIYArZBaytfo9q YR6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333231; x=1727938031; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BmENsM1LVZMrc0W501TyzYIERT8c75Mhan1AJ9vs73Q=; b=aYlrqfQ3XRsEMzb600YmK3RDLrioQzptC2Mu4z9K0Vng5leB4ZCthSEPZfZpbffNbO 1HK4X1YVoMTKk65gKkUkdpiw9JLYhZIGDfsHe0m+J+mNfl2bK7q2DBQH6bKV4STF7bAd /RLllLHzXAJcZzMrsr8bZ+lnxJkgd7VgCwBxEyoez6yCscoxHylnGFDdjqEL0+daTovb 0ccPBnkB2odYKvTJNlPoUGD/opooEp6VoGvm4bbs8fkIkhBuRlMGJ98/Ys9SgZbbRxwH AKisSUj2eFEYFBWahmmx63o5MEGYE62Qx7NuXQ+SPdWoD8OWtA6/krhpYMpCZRn1Ihwd DQaA== X-Gm-Message-State: AOJu0YwNjwsqOvvf8Q1ykkEYJfYjvEY4Hh/pn0V9mekcZ7g4cO4GoE6v oWF4/uK8mx6LSQ85WcSPSaIh0XHYrIG9v5oOi/69LrJlEYLdoufCF2tyjQ1iWEQ= X-Google-Smtp-Source: AGHT+IF3jq8XAx4d8MvdG//ydygdKBfb8RG21eWCHGRQrOC65QRa0E/gKmvca15+75cy4x5ki2/uwA== X-Received: by 2002:ac8:5914:0:b0:458:fb8:9dc3 with SMTP id d75a77b69052e-45b5def1dadmr61649591cf.25.1727333230912; Wed, 25 Sep 2024 23:47:10 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.47.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:47:10 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 03/13] mm: filemap: filemap_fault_recheck_pte_none() use pte_offset_map_ro_nolock() Date: Thu, 26 Sep 2024 14:46:16 +0800 Message-Id: <9f7cbbaa772385ced1b8931b67a8b9d246c9b82d.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In filemap_fault_recheck_pte_none(), we just do pte_none() check, so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/filemap.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 3e46ca45e13dc..6c6ff8722550f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3271,8 +3271,8 @@ static vm_fault_t filemap_fault_recheck_pte_none(stru= ct vm_fault *vmf) if (!(vmf->flags & FAULT_FLAG_ORIG_PTE_VALID)) return 0; =20 - ptep =3D pte_offset_map_nolock(vma->vm_mm, vmf->pmd, vmf->address, - &vmf->ptl); + ptep =3D pte_offset_map_ro_nolock(vma->vm_mm, vmf->pmd, vmf->address, + &vmf->ptl); if (unlikely(!ptep)) return VM_FAULT_NOPAGE; =20 --=20 2.20.1 From nobody Fri Nov 29 00:48:46 2024 Received: from mail-vs1-f44.google.com (mail-vs1-f44.google.com [209.85.217.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1C3A313D276 for ; Thu, 26 Sep 2024 06:47:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.217.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333241; cv=none; b=jwksG5CreD5vxNsZNvbhH3QxaV74V5cQiTzF6BlKvpIqYbpwT8K0dLB7Lg0w9ZTefodoGgaQm3FpII0n5J4rSemYfxTeg+AZv/66xrlcrDxV0bXNHfNG0TPYu+gczL+AOjmka2nlih6S50EWtmbHLtIqqfZ1+zlkyGNv8qqpYPg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333241; c=relaxed/simple; bh=v+8eBabVrYBW01Qi/P2m8jF8QqAnjVEwJS4a+M2dkLU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=IWcTu8KIpEoL2o4Dob/DWN/Hr5mUp/Rt6KfZ2MGJBnHHBO0SjcMkrb/dWgEJ/NLAP2lByXStHqI00y1r5e64bwAt/7qQMpzLJH8phqISLbjtZlIQhOWgGpPoecyNJBgC+QF3PPxN9X25giFQ6AKHmXrhuXaITJJEHTUcBV0m7wE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=MmNDotZc; arc=none smtp.client-ip=209.85.217.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="MmNDotZc" Received: by mail-vs1-f44.google.com with SMTP id ada2fe7eead31-49bdc6e2e2cso272660137.0 for ; Wed, 25 Sep 2024 23:47:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333239; x=1727938039; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=C6e2rRMn75kFV6Fv258+yl1EB6KpOV48BOzHpMuHQ/Y=; b=MmNDotZc1ZT6guSaIWNvWLcaXIj3q/dnhy1ANKdzZAush96XiAT7TMZ5dmjKmU502X IEjDWp1qOM41LnR2cdVIcCbqAULtqVIEOCDrlplHoDLZSqUn6TKTG2mndatJ9Uyrecmj rsepOxaUb+8ZBB6w7McDDS2CYdtDAexiGO7Phbz5KwmBz0vnTBaiIu1hBSUUrYGyHp3j ws6qO85CcMnUSzYRp16jN2kmLdvLqd8452Ji7Hqwr4ZTrEta63BzDx/xsjVt1UHMaGUH O9afu/tYLjvVcUr4IXAeCQxT0+mMHOzDGUPidoNBFkcSgNyrTSO4TxXBpDuD0Z3aNlXN VS1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333239; x=1727938039; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C6e2rRMn75kFV6Fv258+yl1EB6KpOV48BOzHpMuHQ/Y=; b=u040J1lD8gzd1/x3wGiM2kWYmPs0K5DBUg7pKEQVBVeJr+6KxrzHSxLlQHRjxMTqg5 kBm2Lmnq/lAMl8op1M8MgdvkesP6xsNZtsOZv4oJQLsRXUda51nKt5tWd9PeNAWL1NKK 4epb3YWTZw08V7rYVawg09/7SbTp2zZnnfPn3t3FUG4EXih8WCMGuWQGq2nUUh72z8ra u2TgF6rCyrB2Zudkc3DQ66kMy0g/dfF0or2TIrFHg/FfnlQaqx29KEr8Bj+lHdJFzXus Chkhg9Ln6qXYbdgex5AvQt/zut4bCDpB5ixMCn8Ri63aS1v1Lp9EDxwoL+gmEY7+SNIc rmlA== X-Gm-Message-State: AOJu0YwcY4iPxgOd1Z9lGQrHdQD5hhhfBEGR+IP45DrQfRPkSSr7Rv+y zCHLV0UJmfRgp5RtmH7uCOG0WJbF5cxi0v+5dCIL4cpbVSWwmOXM636FC7Ji0No= X-Google-Smtp-Source: AGHT+IHnBlRBHwKCxjYpHBiGYKwBIoiWGnfAohABSGODP2aU4XU9V+T3qIbiD3kwEf9HDK6sMKBjlg== X-Received: by 2002:a05:6102:6c3:b0:4a0:1cdb:3eb4 with SMTP id ada2fe7eead31-4a15dd1a11amr7011565137.20.1727333238630; Wed, 25 Sep 2024 23:47:18 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.47.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:47:18 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 04/13] mm: khugepaged: __collapse_huge_page_swapin() use pte_offset_map_ro_nolock() Date: Thu, 26 Sep 2024 14:46:17 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In __collapse_huge_page_swapin(), we just use the ptl for pte_same() check in do_swap_page(). In other places, we directly use pte_offset_map_lock(), so convert it to using pte_offset_map_ro_nolock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/khugepaged.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index f9c39898eaff6..6498721d4783a 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1011,7 +1011,11 @@ static int __collapse_huge_page_swapin(struct mm_str= uct *mm, }; =20 if (!pte++) { - pte =3D pte_offset_map_nolock(mm, pmd, address, &ptl); + /* + * Here the ptl is only used to check pte_same() in + * do_swap_page(), so readonly version is enough. + */ + pte =3D pte_offset_map_ro_nolock(mm, pmd, address, &ptl); if (!pte) { mmap_read_unlock(mm); result =3D SCAN_PMD_NULL; --=20 2.20.1 From nobody Fri Nov 29 00:48:46 2024 Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C06818D621 for ; Thu, 26 Sep 2024 06:47:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333249; cv=none; b=lqUm+C0yTUel9O77hPeS1L72RKRHqX3BRrOr/PFHrBUZ/EEtWltBOu3oQjYzz9ZumLDsJ404PUEeY3V3ONrYlD+f7Vj0kfUGD5ibzxb8D+dsnPacRzvdzMQYvXS4pf2EcdPERnSUCkVsLeUDNB973Ft4ad6MVP7koacHGyGDRtc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333249; c=relaxed/simple; bh=29OR+TVmri3kqIvPShCpc5yTrHLSKlzLO/ZlFINbxMI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=hI8wHOyb1H7TM/gCUb7FJwjTwhqlgnD+uvyMOk412hgEshP5b7++alNLMT10n2/+zVwqf2o4LJEyP0ETrMWWTxVJdwdC9dVhu5LihatiMKy0vGNnN8aspxHsjN9Pb3ZWkwpuW44RWLHv89+C3FBJcur6fGfHIEo2WamO7dZVtQs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=H2L8G213; arc=none smtp.client-ip=209.85.160.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="H2L8G213" Received: by mail-qt1-f175.google.com with SMTP id d75a77b69052e-45832b2784bso4621261cf.3 for ; Wed, 25 Sep 2024 23:47:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333246; x=1727938046; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VXy51M36oXFNN3kWVEF6T1v76BlvE2cEswNH4MTPbjU=; b=H2L8G213bOzJ2yhf/l0PdUzYarrxzmDIqkPlqQfUjc8yUH4DPdn0c1opY/nkQobSqO bNsNs/LTpsUOEUSEzi7/ekqAVQylI04nUez7qotyiOi1kd+92grdjTfdgmAosjusanko 1wuggjcnqQRiIkGL+Qy6Dd0lbSd3LPOcPOkmZ0tQS3KRLmsxi2+A/O834XD2iy8B9Iot 0gl1P6obaIiCs383+vrtC3K1337de2ZfN3KVXNrcHlySlQYp4eAy3lxRoA1gaBarF/Vj KKYBJ0QOSbTqKoOe/wtSg8ngX+N4BZU+mu5L0mFDtWKk9cdZ6f/MSZT6Edh4MAyVIpHy DDqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333246; x=1727938046; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VXy51M36oXFNN3kWVEF6T1v76BlvE2cEswNH4MTPbjU=; b=LkQxv1hWXQunsHWdChjyIS9+8VweZSyxBTVCI1kOYMuUoGMc6XCrJ/InMbx0qSvn2P JTl+uAfrFyfIC3UnElADQblfrfATvqW77m4vyj91nr6V/5ayJG77L/lO/X5MJzlG+XyQ UvYdN/n42kYvg0gvcC+QKo/w9FMxVNzPcuEOR8l8wcnnXcKFE0grqeE7DU1BlDqd34CU J2T5h2o8En4OvTARRRMYIcTgpKkXG2ujNnC9Wr4qqw1tJ6z2ILSZjNWXFNFB1de+U6Bj WhRkOK78GizZuZENNCflORtu5QiBxr3c+kiS034z2ZhODhwuFj0E9FJGram9z4QEYI0P bPhw== X-Gm-Message-State: AOJu0YwWqpQrjlOjSHr9fEJRQcPQrmTIQ2rHNVzC+ChhzfTQgSSP1F39 OneINZAx63N6qzmgmdMY6n1OTQFjXihl567gF0y1IpBKBJmCkoayUgEwU5l7WBI= X-Google-Smtp-Source: AGHT+IGInGARMP6O0lOtN8I02J26ctBRW9WAU/Ifxw0RvFOYg2EOnEie2juwEDAOYLUuf+d4v6O7qQ== X-Received: by 2002:ac8:5885:0:b0:456:94fe:dcdf with SMTP id d75a77b69052e-45b5def4674mr66807991cf.35.1727333246109; Wed, 25 Sep 2024 23:47:26 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.47.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:47:25 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 05/13] arm: adjust_pte() use pte_offset_map_rw_nolock() Date: Thu, 26 Sep 2024 14:46:18 +0800 Message-Id: <0eaf6b69aeb2fe35092a633fed12537efe645303.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In do_adjust_pte(), we may modify the pte entry. The corresponding pmd entry may have been modified concurrently. Therefore, in order to ensure the stability if pmd entry, use pte_offset_map_rw_nolock() to replace pte_offset_map_nolock(), and do pmd_same() check after holding the PTL. All callers of update_mmu_cache_range() hold the vmf->ptl, so we can determined whether split PTE locks is being used by doing the following, just as we do elsewhere in the kernel. ptl !=3D vmf->ptl And then we can delete the do_pte_lock() and do_pte_unlock(). Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- arch/arm/mm/fault-armv.c | 53 +++++++++++++++++----------------------- 1 file changed, 22 insertions(+), 31 deletions(-) diff --git a/arch/arm/mm/fault-armv.c b/arch/arm/mm/fault-armv.c index 831793cd6ff94..2bec87c3327d2 100644 --- a/arch/arm/mm/fault-armv.c +++ b/arch/arm/mm/fault-armv.c @@ -61,32 +61,8 @@ static int do_adjust_pte(struct vm_area_struct *vma, uns= igned long address, return ret; } =20 -#if defined(CONFIG_SPLIT_PTE_PTLOCKS) -/* - * If we are using split PTE locks, then we need to take the page - * lock here. Otherwise we are using shared mm->page_table_lock - * which is already locked, thus cannot take it. - */ -static inline void do_pte_lock(spinlock_t *ptl) -{ - /* - * Use nested version here to indicate that we are already - * holding one similar spinlock. - */ - spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); -} - -static inline void do_pte_unlock(spinlock_t *ptl) -{ - spin_unlock(ptl); -} -#else /* !defined(CONFIG_SPLIT_PTE_PTLOCKS) */ -static inline void do_pte_lock(spinlock_t *ptl) {} -static inline void do_pte_unlock(spinlock_t *ptl) {} -#endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ - static int adjust_pte(struct vm_area_struct *vma, unsigned long address, - unsigned long pfn) + unsigned long pfn, struct vm_fault *vmf) { spinlock_t *ptl; pgd_t *pgd; @@ -94,6 +70,7 @@ static int adjust_pte(struct vm_area_struct *vma, unsigne= d long address, pud_t *pud; pmd_t *pmd; pte_t *pte; + pmd_t pmdval; int ret; =20 pgd =3D pgd_offset(vma->vm_mm, address); @@ -112,20 +89,33 @@ static int adjust_pte(struct vm_area_struct *vma, unsi= gned long address, if (pmd_none_or_clear_bad(pmd)) return 0; =20 +again: /* * This is called while another page table is mapped, so we * must use the nested version. This also means we need to * open-code the spin-locking. */ - pte =3D pte_offset_map_nolock(vma->vm_mm, pmd, address, &ptl); + pte =3D pte_offset_map_rw_nolock(vma->vm_mm, pmd, address, &pmdval, &ptl); if (!pte) return 0; =20 - do_pte_lock(ptl); + /* + * If we are using split PTE locks, then we need to take the page + * lock here. Otherwise we are using shared mm->page_table_lock + * which is already locked, thus cannot take it. + */ + if (ptl !=3D vmf->ptl) { + spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + goto again; + } + } =20 ret =3D do_adjust_pte(vma, address, pfn, pte); =20 - do_pte_unlock(ptl); + if (ptl !=3D vmf->ptl) + spin_unlock(ptl); pte_unmap(pte); =20 return ret; @@ -133,7 +123,8 @@ static int adjust_pte(struct vm_area_struct *vma, unsig= ned long address, =20 static void make_coherent(struct address_space *mapping, struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, unsigned long pfn) + unsigned long addr, pte_t *ptep, unsigned long pfn, + struct vm_fault *vmf) { struct mm_struct *mm =3D vma->vm_mm; struct vm_area_struct *mpnt; @@ -160,7 +151,7 @@ make_coherent(struct address_space *mapping, struct vm_= area_struct *vma, if (!(mpnt->vm_flags & VM_MAYSHARE)) continue; offset =3D (pgoff - mpnt->vm_pgoff) << PAGE_SHIFT; - aliases +=3D adjust_pte(mpnt, mpnt->vm_start + offset, pfn); + aliases +=3D adjust_pte(mpnt, mpnt->vm_start + offset, pfn, vmf); } flush_dcache_mmap_unlock(mapping); if (aliases) @@ -203,7 +194,7 @@ void update_mmu_cache_range(struct vm_fault *vmf, struc= t vm_area_struct *vma, __flush_dcache_folio(mapping, folio); if (mapping) { if (cache_is_vivt()) - make_coherent(mapping, vma, addr, ptep, pfn); + make_coherent(mapping, vma, addr, ptep, pfn, vmf); else if (vma->vm_flags & VM_EXEC) __flush_icache_all(); } --=20 2.20.1 From nobody Fri Nov 29 00:48:46 2024 Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9596917554A for ; Thu, 26 Sep 2024 06:47:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333257; cv=none; b=bVRcMR1U4t8y4oa81BUWdnzjRxRcl0qnMi3hmKtgh/kq7/nsxfK27GCU4c60zIwEuDHDHbaytVy1OB148JSgSlfxdppmLafT7eVtYqoZ0/fQTtauC/cuGQDnr3VMspMuIK2y+AEze0yZazGRhsuB9IVljKahhyrErgnUIBjgxQQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333257; c=relaxed/simple; bh=AW2pi97601MGuZiCOXDhP3jZoiCUTCuHUPQzcZePTCg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JlQ9r7lTPFh8OsqZyP+JhrhTA1GUQwbDM2GQVFz/Vi8FOLkl/7kDyqjLEICAhLGFvAmwEkz/DmEzCX1mLAjqRIXrmqmmRigMqsi+8OoN+opoWUql1p2/QTDIhCIv6fhlHjx2dq6TpJwKY8VO78qQmiYLbKEkqJPqH/BFDAbZkDI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=itaI4G/q; arc=none smtp.client-ip=209.85.222.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="itaI4G/q" Received: by mail-qk1-f179.google.com with SMTP id af79cd13be357-7a9b049251eso47707685a.2 for ; Wed, 25 Sep 2024 23:47:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333254; x=1727938054; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=88oX8R3xcxvOkzUvzbp2hva7xyyimZKCI3msRz7gwSQ=; b=itaI4G/qzn2DcT8Rl3gcyXgeMKtQlTP2zbR9vQvlX993Z5EWneMssOdIQNlxp1tCb9 A14kWE6D4Wpgupn0ML3oSLAYNrJQ2nUtiTV9azZXBuJqmn9xsPddePzumdthL3AVRSgT 0ojFQguGdE/RCSeOjnjdQ6XtE+tUa8B+Tyb+VHnRKUL82uItVt4Mm3oJTMSGsN96oOvg GgIRbIflmJ7/ihaGstZrU7dFwmT8f3AGcJCn7FS8umhTT6gapZSnCHDdFB6Vbmeg+sd4 TskwLUolsweR5epzQsP5GGTiGr+fcUg5Z4Yg4SwD5M7DocKwePquRZ65+GRnVspkkcza 111w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333254; x=1727938054; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=88oX8R3xcxvOkzUvzbp2hva7xyyimZKCI3msRz7gwSQ=; b=lmJUmADnkV3P2xmdN7SDf7nF+33WPCXC07rlACwKJ0qvfnx+AmdXVpqVi5eF6nKV0g U2sFH4prKgjj+Wt14a3UzTenl1eddCJ+fmvX8DE2h2Bwif9JzXSnOzZ8MALOKJl/Es53 mJcojqKlFBn50TSn5oXXpJ803aR6hGG0Tl3QYYJbCph+DkC5hEdrrWCTIzTswGBXpa+f 4L28q1c9CfzbsrvfxT2dIHn3Gvo+vuZ6L5MjifmaRsxLIC0lKlRzPfRa+dqnGhsBVo1a 9R5WNSHjGMJ426YqwRuXFyKgzTrvEdaNeVDJU/2eQps8EynHaAzbt5e1gD2W7qmx+JmU 7nOw== X-Gm-Message-State: AOJu0YyckxdhMdbyVOMC6Kujh86o2OTFfba//uldJBPGqVsijnVL0Wue jfUU4wA7G/li4uG7BMWnKSV6dOzbwdHFxKkOb1L/mXZlzTnZ33rHYxh2AkvBiag= X-Google-Smtp-Source: AGHT+IEkNfjkqcOSOf8WFqyh7ycpXwLGwDHOuzbRwWfPgE4JYY85w8LbZC4JktZjOBGqNKfXWcA2mg== X-Received: by 2002:ac8:5a43:0:b0:458:3824:a083 with SMTP id d75a77b69052e-45b5def43dcmr69885101cf.32.1727333254117; Wed, 25 Sep 2024 23:47:34 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.47.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:47:33 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 06/13] mm: handle_pte_fault() use pte_offset_map_rw_nolock() Date: Thu, 26 Sep 2024 14:46:19 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In handle_pte_fault(), we may modify the vmf->pte after acquiring the vmf->ptl, so convert it to using pte_offset_map_rw_nolock(). But since we will do the pte_same() check, so there is no need to get pmdval to do pmd_same() check, just pass a dummy variable to it. Signed-off-by: Qi Zheng Acked-by: David Hildenbrand Reviewed-by: Muchun Song --- mm/memory.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index a245d28787034..6432b636d1ba7 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5750,14 +5750,24 @@ static vm_fault_t handle_pte_fault(struct vm_fault = *vmf) vmf->pte =3D NULL; vmf->flags &=3D ~FAULT_FLAG_ORIG_PTE_VALID; } else { + pmd_t dummy_pmdval; + /* * A regular pmd is established and it can't morph into a huge * pmd by anon khugepaged, since that takes mmap_lock in write * mode; but shmem or file collapse to THP could still morph * it into a huge pmd: just retry later if so. + * + * Use the maywrite version to indicate that vmf->pte may be + * modified, but since we will use pte_same() to detect the + * change of the !pte_none() entry, there is no need to recheck + * the pmdval. Here we chooes to pass a dummy variable instead + * of NULL, which helps new user think about why this place is + * special. */ - vmf->pte =3D pte_offset_map_nolock(vmf->vma->vm_mm, vmf->pmd, - vmf->address, &vmf->ptl); + vmf->pte =3D pte_offset_map_rw_nolock(vmf->vma->vm_mm, vmf->pmd, + vmf->address, &dummy_pmdval, + &vmf->ptl); if (unlikely(!vmf->pte)) return 0; vmf->orig_pte =3D ptep_get_lockless(vmf->pte); --=20 2.20.1 From nobody Fri Nov 29 00:48:46 2024 Received: from mail-qt1-f170.google.com (mail-qt1-f170.google.com [209.85.160.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0A15917554A for ; Thu, 26 Sep 2024 06:47:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333264; cv=none; b=NQCzoOJYXAxS48MHDKIIc3Rp32jMY5vVLjBdKmAa6KZJ0lWY+z1olosUoxRE5pKw6qS5AU/w2AnkOv2CH3s2QgVNdk11Br9RfpNVN54OprOVyDGUCr4y0C853shhuwzvC/Da0nlQSlan5Uv98MPkEUAiCtikHpu/Lkb+x2/jXjY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333264; c=relaxed/simple; bh=J8iz6TRBjz8Nxahcna8Ou8absZteAFKLGxxh6adklaQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Amgr0tQr/TlJVH1IHAEUw7xSdPYJ+x9ZU63/STyTSUnW0i6RMKibWFipVSwQnALTxKRSvEOBIhz+An3oJdO0ygaQ5u1V4dJ0e1RRnHAJHPhZ7O7Bje/8PvzWqTSdtSbETI0Bva+Ht+QMFSzZICvV+pbTBXjF1ZEZAMudvfcgWRk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=dMJ3pJs+; arc=none smtp.client-ip=209.85.160.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="dMJ3pJs+" Received: by mail-qt1-f170.google.com with SMTP id d75a77b69052e-4582f9abb43so4181771cf.2 for ; Wed, 25 Sep 2024 23:47:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333262; x=1727938062; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ffdIJrLHJ9HLo1zmy1fROVnOG/sDLfcmik2lSMoQorU=; b=dMJ3pJs+6KbqqyksF5ltQRBDIHtzWOBPCQRPM90zJCJac+AjZ1QNtnoambk0F2PMXG 3oYPDXVicIQPbXBnN712lJnW4VWSxGByE73+unXg7lxBPrL0heVxdUul90XZ0AVyKOv5 wypvcEZd9rWYiUi4R03Lw+dRMysxsid5V97huYBruUXzlyI2l0lRHixDA8JJz56CzjIB KmT8ETdn4h/B124tHPvnCGZpMi/tVbAyjVbGDkYFy3VLchjy019qVjvfk7g6CGORFhFV Y3Hx+K9x88AFP788wAlt25HmdYoHp/C9UdbeO6d0i5VHdjVtYh7pA3nM0Ctbc0BufcEX gnUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333262; x=1727938062; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ffdIJrLHJ9HLo1zmy1fROVnOG/sDLfcmik2lSMoQorU=; b=qDs921lOES3fuOZaw8qiDYOpciQN5Ibo2h+dYD1pd/OPiw7fITQX76tyocB8FtmN1U K6XxqFJniSjL6CLMoI59XmWpe65j1juw8KNlwAEql2SonHhdoj8tWaoN65dVaovBArW+ ETSt/aOIxclLYb3eUWqeGnNozUZqreXwE0051TvOFIEdQAXeAICngDTezZRmYUgRjpbt C3usSSWe/c+UYQPfF5EFEBXyYP/MUmFVZYK7HAE2rQkf17v0hH+6NFpxJdt40iJAAy9W 4014QHBQvnEN06iFQ1KeoEucTr69jx+kBQkEzF9BCP5x34b4QUx+iTkuCyxBxMVSbvP7 402A== X-Gm-Message-State: AOJu0YwH7Tw1LPId1QsnXP6U7wsoGMHpXJxXJRtxprvdT6NHbZ4RcLWG sGMtPTWFqN7Ky8lgfaN5DIEKCWX6WO9hSgPIZ8MjN5JPQPXxZ+7t5yAtAMNUbLk= X-Google-Smtp-Source: AGHT+IFPPE2WMfZ0ye6LGa3/R8zXQ/6KjVtS14uw6772ox/YEy7cH2DCZDAwEci91jHXPd4jVDRFUg== X-Received: by 2002:a05:622a:1493:b0:457:c747:cf3c with SMTP id d75a77b69052e-45b5e0534ebmr99370671cf.57.1727333261763; Wed, 25 Sep 2024 23:47:41 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.47.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:47:41 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 07/13] mm: khugepaged: collapse_pte_mapped_thp() use pte_offset_map_rw_nolock() Date: Thu, 26 Sep 2024 14:46:20 +0800 Message-Id: <055e42db68da00ac8ecab94bd2633c7cd965eb1c.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In collapse_pte_mapped_thp(), we may modify the pte and pmd entry after acquring the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the PTL held. So we should get pgt_pmd and do pmd_same() check after the ptl held. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/khugepaged.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6498721d4783a..854577f39957d 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1605,7 +1605,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, if (userfaultfd_armed(vma) && !(vma->vm_flags & VM_SHARED)) pml =3D pmd_lock(mm, pmd); =20 - start_pte =3D pte_offset_map_nolock(mm, pmd, haddr, &ptl); + start_pte =3D pte_offset_map_rw_nolock(mm, pmd, haddr, &pgt_pmd, &ptl); if (!start_pte) /* mmap_lock + page lock should prevent this */ goto abort; if (!pml) @@ -1613,6 +1613,9 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, else if (ptl !=3D pml) spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); =20 + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) + goto abort; + /* step 2: clear page table and adjust rmap */ for (i =3D 0, addr =3D haddr, pte =3D start_pte; i < HPAGE_PMD_NR; i++, addr +=3D PAGE_SIZE, pte++) { @@ -1645,7 +1648,6 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, nr_ptes++; } =20 - pte_unmap(start_pte); if (!pml) spin_unlock(ptl); =20 @@ -1658,14 +1660,19 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, u= nsigned long addr, /* step 4: remove empty page table */ if (!pml) { pml =3D pmd_lock(mm, pmd); - if (ptl !=3D pml) + if (ptl !=3D pml) { spin_lock_nested(ptl, SINGLE_DEPTH_NESTING); + if (unlikely(!pmd_same(pgt_pmd, pmdp_get_lockless(pmd)))) { + flush_tlb_mm(mm); + goto unlock; + } + } } pgt_pmd =3D pmdp_collapse_flush(vma, haddr, pmd); pmdp_get_lockless_sync(); + pte_unmap_unlock(start_pte, ptl); if (ptl !=3D pml) - spin_unlock(ptl); - spin_unlock(pml); + spin_unlock(pml); =20 mmu_notifier_invalidate_range_end(&range); =20 @@ -1685,6 +1692,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, uns= igned long addr, folio_ref_sub(folio, nr_ptes); add_mm_counter(mm, mm_counter_file(folio), -nr_ptes); } +unlock: if (start_pte) pte_unmap_unlock(start_pte, ptl); if (pml && pml !=3D ptl) --=20 2.20.1 From nobody Fri Nov 29 00:48:46 2024 Received: from mail-qk1-f170.google.com (mail-qk1-f170.google.com [209.85.222.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7158918C33D for ; Thu, 26 Sep 2024 06:47:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333272; cv=none; b=jRty+yewr6cXOB43HCrsmC/XgbzYXRacHwz2aPB3akyGH14E26NNlqFYO2CQ1p3PQSr8uiho4o8JaLiO4XcBlB/ozPncgfYe0vMhLGtRy8NT9czEFex4ZMt5inMK0zb16GDSEk9yFD8HdbsK4wXy79bjGvWAJg/0IWmRhX17SJ4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333272; c=relaxed/simple; bh=gOCug7h2oDN1d5Lx1fQiD5OJfBDKrJ9h7gKP2OJkqUI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dBTnkao1WLGNpLsU2rnCL+4UmBTw+4oI6eBeisCvJ91iB+TcOnTQyv/wOCsVoPV/h5Ps3ADGPfbcIwj/pxzmOjbl7srTZgs5W7+Cy3laN8p4HgSp02vJ0E7Q3C19qfRZpntQcEKQvbFnlVGdP/tmTAm4NCAY+ZnMuMjUd1ooN+M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=YlJjuAjY; arc=none smtp.client-ip=209.85.222.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="YlJjuAjY" Received: by mail-qk1-f170.google.com with SMTP id af79cd13be357-7a99de9beb2so44969285a.3 for ; Wed, 25 Sep 2024 23:47:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333269; x=1727938069; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3DwHdqVJkGu7HChE1YzlDZaDZvpQCVqvAB2HVgACkHs=; b=YlJjuAjYMJVIT5dbIKYDfOlqa4u6upPLFSmICiTN9cstDahG0+l3iO3WfvrHQ+WwCP 2v7duiNH3eh70JWky89s4RRcoYgCVR19saOb52YI686EJhUCGobzrzfCDxVWoSe3wEyQ gNgXoBp4jUxUkANbiDfMXlShct2/jG+gYVfRjl4qPthIOWt1K05lQ6elgCr36WV95+IV uHHP7O10U5a7t/DzJl9PieM+9Xy3OC77ZZ8u5BsKp1FxN1HZvFNDyjWGs6wO9FGQJ8m9 zT0f85ct4tDL3GL5HxsjmGUOuTHdPrH4X/Mlnl+EdMTetQz4Lgypqeip0QTtshYMzdpi S5dw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333269; x=1727938069; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3DwHdqVJkGu7HChE1YzlDZaDZvpQCVqvAB2HVgACkHs=; b=HpUWUfbJOgBuLYfCpQ5LBEUuKNSnsqOT2Vnd799FbWTDcJhVaq8DYCjmcrMOWZ94HL k6uxE30e4bAKkRXhYMCHP+K87pqnucS1hHfeqwCxcXjoQMYXPzsw0ULmCy2T13Gj+KLN MCGrITawky2JSgPSWQkGQLGfRTnAGKESefl91D1KkJFVOIQkklebatk6/J054lbk0drW 891jnV/38b6p8IZRpqaUUqHufx/wmrp1xA4BmBsHpdaSQecArvhenI0jhOH5GJ/hBSvK f0rOu8Lc2qPA1IYt5+EXKCT8OIEm/rZyCemjp3o2fDufVfZIfb3U3ARq87QeZo1nuHM+ jfYg== X-Gm-Message-State: AOJu0YzcciWjKDr28ieLuwFyEA80sfWFoKCsY9pAxNhkKcqnk91agaGG ln9o2/CDcdaLe05aNeAfCVYyM9WuQtn3Y83ecoEbYqnb60/2LAScRfovl5s1RdGQLu7BIkKQn1a w X-Google-Smtp-Source: AGHT+IGf6yhFgM7V9dSkmNMD310kmtT4NpPr6tml7YSLsMbfjuBNhYOZWImI0+hdwHwJZd31DH/Ogg== X-Received: by 2002:a05:620a:4009:b0:7a7:f18a:e46f with SMTP id af79cd13be357-7ace744e486mr712325485a.43.1727333269291; Wed, 25 Sep 2024 23:47:49 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.47.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:47:48 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 08/13] mm: copy_pte_range() use pte_offset_map_rw_nolock() Date: Thu, 26 Sep 2024 14:46:21 +0800 Message-Id: <9166f6fad806efbca72e318ab6f0f8af458056a9.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In copy_pte_range(), we may modify the src_pte entry after holding the src_ptl, so convert it to using pte_offset_map_rw_nolock(). Since we already hold the exclusive mmap_lock, and the copy_pte_range() and retract_page_tables() are using vma->anon_vma to be exclusive, so the PTE page is stable, there is no need to get pmdval and do pmd_same() check. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/memory.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 6432b636d1ba7..c19cf14e1c565 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1086,6 +1086,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct= vm_area_struct *src_vma, struct mm_struct *src_mm =3D src_vma->vm_mm; pte_t *orig_src_pte, *orig_dst_pte; pte_t *src_pte, *dst_pte; + pmd_t dummy_pmdval; pte_t ptent; spinlock_t *src_ptl, *dst_ptl; int progress, max_nr, ret =3D 0; @@ -1111,7 +1112,15 @@ copy_pte_range(struct vm_area_struct *dst_vma, struc= t vm_area_struct *src_vma, ret =3D -ENOMEM; goto out; } - src_pte =3D pte_offset_map_nolock(src_mm, src_pmd, addr, &src_ptl); + + /* + * We already hold the exclusive mmap_lock, the copy_pte_range() and + * retract_page_tables() are using vma->anon_vma to be exclusive, so + * the PTE page is stable, and there is no need to get pmdval and do + * pmd_same() check. + */ + src_pte =3D pte_offset_map_rw_nolock(src_mm, src_pmd, addr, &dummy_pmdval, + &src_ptl); if (!src_pte) { pte_unmap_unlock(dst_pte, dst_ptl); /* ret =3D=3D 0 */ --=20 2.20.1 From nobody Fri Nov 29 00:48:46 2024 Received: from mail-qt1-f178.google.com (mail-qt1-f178.google.com [209.85.160.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 096A618786B for ; Thu, 26 Sep 2024 06:47:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333279; cv=none; b=hXN6hd4Rn2wgLgKH+AHnyC8QVoNeF1G9sPwywOd1GKC/5XgcLypZfms8+Ev4WwtNzb/Plqj0peZP3JRpiDJqvcWvcoSQEQqPsPDwN9Z4XynlNEkU6VDCN1mF+4i7JXCVZY+70jcTAqFq1nP1KGjzfZteARIGnjrA1+udg6ZMBvU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333279; c=relaxed/simple; bh=pj4lw9RRA4PWKbqJxPwbDgZhVSrxJd4D2Kzv+dDcxPk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=PxL3b4XfRRr+UggRxIRhAaaKztFszG44CErZSb+hv4okAvvrPpVfS5XeTg0POSUdMgLlUPmJ6JXfxML2EZYmWj62wyhnCeYlE6hnJ0ssK1DCRAVTd961preIhCEnl2cnxWSAyCNxA7CFAJjzq8k608WtsOUcBqoYi7e9pMjFhSw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=EvJ6YCzX; arc=none smtp.client-ip=209.85.160.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="EvJ6YCzX" Received: by mail-qt1-f178.google.com with SMTP id d75a77b69052e-457ce5fda1aso4651191cf.1 for ; Wed, 25 Sep 2024 23:47:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333277; x=1727938077; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xHtLGGTaDvd+j7nlO+D4nlmKtDmH5wZnqdyAY1Z6B8s=; b=EvJ6YCzXlTBpt8XPQCDUn3Egcf2SJo0SDZwKcnM82A60qMd88bZpOzJ4oC9gXVjOP1 FWvCKoZVPV/hsIjq2NI6IohY/vQ22Mg/iCDxV9AH5M9J870cMBAD4jF3t/URxgbNrmDL yfA2LYlCYTOebr1Vut04/KkEYwZZzYO0GbHZY0ROsjS1AYPLGSY52cfyYDP4eC+Xe3pj 6DsLD/Xy/K7HLPf6IHH5pthfSbiJjefwyHl5b428PNnc8aE8TvxAi4ZGINJHAByfaKti +i8Nrg0lvJcFlMsLTBtaJCzIpVD1Zt0578yyQc4VIu926Ix+dDTmKJaDPqiYxoNFHjhU 78JA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333277; x=1727938077; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xHtLGGTaDvd+j7nlO+D4nlmKtDmH5wZnqdyAY1Z6B8s=; b=D77GqIIk2W8D0/tQ7Z+bgDmcS8uV3ZyfDHr4qf4jgEAAxNFSpYmEebBockzeiKtyj0 LIJWt/fiA1CynM9Z3glwXbBLHe+98zG4m+dVam1mhPCFvSEfOT6O+lfz7f/RoOun3uhW P8ZN2Y3sl4SRHnp2i8i/GXWZlQkc/vwA6Gqzu74nuJQ8CnwFKUDitvUBfnjMY9oCAhIf yl3Rqxel45wFoUjFhpSfLXTaJfFQErfAV31XfH9qhEuhEuMgOcr76eMS+NW/SPXVK1rX koe3OpLVBwUqSc7qJeE8NyjidYFh80/CAeO6zDoWu76R6v+EdvSdtxB9jZD0d8IDXxKS yrRw== X-Gm-Message-State: AOJu0YyZo+5NH/kX9Zf51bsxjpyN8GcagQNLKOwLHjgX72ZXbMruRB4k 8VR91Wk46MjtfEzYUhtpUc1eBAKL4ESMhn+Bj8znj01ujMnUtYDbT5fWZLHINmw= X-Google-Smtp-Source: AGHT+IGbo2MSjNAvLC3c2ApJAOpwKD09S+YUwBzMfKcY13i9YN/E+PXjNs0XRR1eOHqqnOUdrqUWyg== X-Received: by 2002:ac8:7d42:0:b0:458:2ce1:26f8 with SMTP id d75a77b69052e-45b5e045ea9mr77074431cf.60.1727333276776; Wed, 25 Sep 2024 23:47:56 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.47.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:47:56 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 09/13] mm: mremap: move_ptes() use pte_offset_map_rw_nolock() Date: Thu, 26 Sep 2024 14:46:22 +0800 Message-Id: <9d582a09dbcf12e562ac5fe0ba05e9248a58f5e0.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In move_ptes(), we may modify the new_pte after acquiring the new_ptl, so convert it to using pte_offset_map_rw_nolock(). Now new_pte is none, so hpage_collapse_scan_file() path can not find this by traversing file->f_mapping, so there is no concurrency with retract_page_tables(). In addition, we already hold the exclusive mmap_lock, so this new_pte page is stable, so there is no need to get pmdval and do pmd_same() check. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/mremap.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/mremap.c b/mm/mremap.c index 24712f8dbb6b5..9dffd4a5b4d18 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -143,6 +143,7 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t = *old_pmd, spinlock_t *old_ptl, *new_ptl; bool force_flush =3D false; unsigned long len =3D old_end - old_addr; + pmd_t dummy_pmdval; int err =3D 0; =20 /* @@ -175,7 +176,15 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t= *old_pmd, err =3D -EAGAIN; goto out; } - new_pte =3D pte_offset_map_nolock(mm, new_pmd, new_addr, &new_ptl); + /* + * Now new_pte is none, so hpage_collapse_scan_file() path can not find + * this by traversing file->f_mapping, so there is no concurrency with + * retract_page_tables(). In addition, we already hold the exclusive + * mmap_lock, so this new_pte page is stable, so there is no need to get + * pmdval and do pmd_same() check. + */ + new_pte =3D pte_offset_map_rw_nolock(mm, new_pmd, new_addr, &dummy_pmdval, + &new_ptl); if (!new_pte) { pte_unmap_unlock(old_pte, old_ptl); err =3D -EAGAIN; --=20 2.20.1 From nobody Fri Nov 29 00:48:46 2024 Received: from mail-qt1-f182.google.com (mail-qt1-f182.google.com [209.85.160.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6092B18D64C for ; Thu, 26 Sep 2024 06:48:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333287; cv=none; b=fcKVEwRzUI5hYeUB4v+8wRpgzpAu3NWwASovPTOPwpIGzU/UoHYNE1bNjlN5CPVxqU+VzADUD8k+VTmtxj+zvHtZAyCgvg9wmj7FOcU02zVXNWDobWwAbPeEUTYe9rYpv9MCvAizuHJF+hIr8x5+4XNCgS5j9fEjmKxzELzoS3g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333287; c=relaxed/simple; bh=ksedbhOfsf/xVArskq0f4ybneNoWGq+FSHDnXQ6TWpU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=r9IZecOz+LX2hMpCtZHjE9U9xAqgT1ZhLq8LkR3wutKbpUvqK8QmXi+HI0aY7nuzoAdujW9CYOxXtBC/cyWdj3wfAqze5LT1vfeYrU9fzbmu4sTbGNxXB4ovFtcV3MPVx4Rn+f2b+wIfNawDhwJRP8cJO0kYJNUQLk1ifDRI/ao= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Kn7Qa/C7; arc=none smtp.client-ip=209.85.160.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Kn7Qa/C7" Received: by mail-qt1-f182.google.com with SMTP id d75a77b69052e-457e153cbdcso5178971cf.2 for ; Wed, 25 Sep 2024 23:48:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333284; x=1727938084; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RsEgu/iD0T6JQDFamAgFHiL1Lm0nKVk+wAfYQhZRThE=; b=Kn7Qa/C744SfmKnSmPsS9K6Wak7ANO0QfTVuMqvVMHokuC/jRw2xoGRAu97KHygYx0 bFQHjDgrXGxNSwwVK0PHIFMVWL44OeBT1VvWa/zIB1E59n0XIq+aweUJE5pEDhUtutTo xXHqx0uYgRr61zK0QQhOC1WWT+CUjwGpuxuGaE5n1HzHm4h8XaL/gzqq91eFDJlBdaAE +ebvy9yBrMPlWgPBKDAJWqC13+Z5l9ITknWqM8PK5Pyz6ljmHaW5UjScPRgiVypySI09 Jahi/Ws/hYCqk0/lY5aOdxCEp/XisdZLHdWA6Rzlx8LlBaSSZoMgSQ82TU9tUoNTdX+/ gs2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333284; x=1727938084; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RsEgu/iD0T6JQDFamAgFHiL1Lm0nKVk+wAfYQhZRThE=; b=k4WTRVDmHkUX7saspDJzcRlN1OfEbGMBCOWmzQ6RSTXp4L0b+QLFAh8puxflIBUWDV jGki0/4pfTvZXa/C8NjfbCy8UuuqaLeL8+XKsw0yJDx+z9QowtNyggqxCaGuGr/SVd8Q reg0hTvpm46BU57vdlq7DBB5Y47EmY0Am3SXNV25fXBRDHZJNPue1+Raj5Xty5fXdX0o C77XsvK1fBBhzggeSn4bmRTs4JkN3UsofcKtdZVw8L6dMxfuOgHtO/4OPYfOExBJjQyx oYpr5lSIhij1l8PJHRyCMeQrmepDzKSlcc9g0i5RVLP66fQNGsYJ2Aq83jMTThwwoivt 9R8g== X-Gm-Message-State: AOJu0YxR7nwf0J7v9OFABrSXq5vT5IgrRQM1yiKyFhDxOyOcEfgl7mR4 Rfab/4oBtwbx5qhbstYVS3wQ4LgS/Y8cxIButcjpMf5aU1OOqZxw6TtugquG6dc= X-Google-Smtp-Source: AGHT+IGAlBKd7aKCt6XxIcAXM/Sjs0v2+e+7/droKiEQYGe4LhoKPN6bWAepHInU8bkNuwlluGjQYA== X-Received: by 2002:ac8:7d4c:0:b0:458:4aec:2749 with SMTP id d75a77b69052e-45b5e045e56mr87889061cf.57.1727333284339; Wed, 25 Sep 2024 23:48:04 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.47.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:48:03 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 10/13] mm: page_vma_mapped_walk: map_pte() use pte_offset_map_rw_nolock() Date: Thu, 26 Sep 2024 14:46:23 +0800 Message-Id: <2620a48f34c9f19864ab0169cdbf253d31a8fcaa.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In the caller of map_pte(), we may modify the pvmw->pte after acquiring the pvmw->ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the pvmw->ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pvmw->pmd. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/page_vma_mapped.c | 24 ++++++++++++++++++------ 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index ae5cc42aa2087..ab1671e71cb2d 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -13,7 +13,8 @@ static inline bool not_found(struct page_vma_mapped_walk = *pvmw) return false; } =20 -static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) +static bool map_pte(struct page_vma_mapped_walk *pvmw, pmd_t *pmdvalp, + spinlock_t **ptlp) { pte_t ptent; =20 @@ -25,6 +26,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, sp= inlock_t **ptlp) return !!pvmw->pte; } =20 +again: /* * It is important to return the ptl corresponding to pte, * in case *pvmw->pmd changes underneath us; so we need to @@ -32,8 +34,8 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, sp= inlock_t **ptlp) * proceeds to loop over next ptes, and finds a match later. * Though, in most cases, page lock already protects this. */ - pvmw->pte =3D pte_offset_map_nolock(pvmw->vma->vm_mm, pvmw->pmd, - pvmw->address, ptlp); + pvmw->pte =3D pte_offset_map_rw_nolock(pvmw->vma->vm_mm, pvmw->pmd, + pvmw->address, pmdvalp, ptlp); if (!pvmw->pte) return false; =20 @@ -67,8 +69,13 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw, s= pinlock_t **ptlp) } else if (!pte_present(ptent)) { return false; } + spin_lock(*ptlp); + if (unlikely(!pmd_same(*pmdvalp, pmdp_get_lockless(pvmw->pmd)))) { + pte_unmap_unlock(pvmw->pte, *ptlp); + goto again; + } pvmw->ptl =3D *ptlp; - spin_lock(pvmw->ptl); + return true; } =20 @@ -278,7 +285,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *= pvmw) step_forward(pvmw, PMD_SIZE); continue; } - if (!map_pte(pvmw, &ptl)) { + if (!map_pte(pvmw, &pmde, &ptl)) { if (!pvmw->pte) goto restart; goto next_pte; @@ -305,8 +312,13 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk = *pvmw) } while (pte_none(ptep_get(pvmw->pte))); =20 if (!pvmw->ptl) { + spin_lock(ptl); + if (unlikely(!pmd_same(pmde, pmdp_get_lockless(pvmw->pmd)))) { + pte_unmap_unlock(pvmw->pte, ptl); + pvmw->pte =3D NULL; + goto restart; + } pvmw->ptl =3D ptl; - spin_lock(pvmw->ptl); } goto this_pte; } while (pvmw->address < end); --=20 2.20.1 From nobody Fri Nov 29 00:48:46 2024 Received: from mail-vs1-f46.google.com (mail-vs1-f46.google.com [209.85.217.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 14EBA13D276 for ; Thu, 26 Sep 2024 06:48:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.217.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333294; cv=none; b=UFYJmsTebQdsHxXldJvcuf8yZETNZQcfSk4724wJ8Ia/Q1wOlVfm6GDRtELHYt1U0z8ut+6BcT624KjjjalYt3PPNBAOqrunog9KGA8598VOpi+Uw5fvWQTs5a4hYbxxojiB9uGPwkN6tCPMpO6ICHuwqD6ZNFNu+goV6fjo9xw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333294; c=relaxed/simple; bh=M/64Ci4YVvyqTSzbm0kd6G+2YPtWZDsuLrmsgyWUzVc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Xfk8ol5VWGIzl5tsTZ4baTrK71wZtzVQ8WZPZnTbzlGDnrrmHiO3CEr4KwtmfL08nlZR2KyQMGMxSsFHwUdhpbbzd9t8+ZFZ0bO/XD2l9dE6/HHUELCYb4z49KmRuKw0WCCv/PWXJMS38iiUb/dhyfre81ceclBXMTyEK8pBeH4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=U5nj8WXJ; arc=none smtp.client-ip=209.85.217.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="U5nj8WXJ" Received: by mail-vs1-f46.google.com with SMTP id ada2fe7eead31-49bdc6e2e2cso272884137.0 for ; Wed, 25 Sep 2024 23:48:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333292; x=1727938092; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RnMCKwBqnYl3LJbu0M9HQTu7wcymlNTkAYqiZlfWQ7I=; b=U5nj8WXJQBqjaIORF30SdurQjye0eWgmnaBreXxzmDm54zseAgADaJ1ujlZ4bbxWh6 3Usde+7EMRfiz3RrpJTTdgH9vg9UPvAtKkmQR9fvtH9gNFeZX6Xx+YpV0ilVYcLuQ1Hs ow85NZkmjT82Xd3pabu4dUAwexZ2xjjMmOalxKfo7tqxEfT2yKzi+lXyujjcwGr0NJMs nl98JqJFZGP1i0msZLck0VlUJvACLW6A4KKkvtu8iSIeEIi6RkOJTyGTHqix2J45v8tH Twi4gF9NeQKzNooghsPorLjQaGaSYNqMt0rybaYj0VOuXOH6U4Rx/zIFxJCXPVfJwqwX dUjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333292; x=1727938092; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RnMCKwBqnYl3LJbu0M9HQTu7wcymlNTkAYqiZlfWQ7I=; b=ZSZXBJ/XTnBC9ndx0wqpXxmSNCfpisDB5conPaMcHhTYtyeVuSR6ERlKkZ7xcC4E96 XYlmGzgpO3HWc9dGcCi7Bpy/ca7/OxM9+zJTUsNlt1WAE687V69X2l53Q+pCRge4cXWt J1cgeDUifoUEgT51VP9BXqx34OfCL6e9XCffrpgZP5UJw3yddF2TmlpPX3cidoiFHCgb y1UQz4kIC1B1k3mTo8eEpsELLWaAb08nE7Xic273cRVLEEiSOsWFc8X3vpxYXB5w7a2H 4bXO8MWrHwZ4srzX0m5AAx7pIlJvB6ldzPwl6zAWzxz4IPMsvfJcUXzetr1E+HS6rGuX oFHw== X-Gm-Message-State: AOJu0YyrVcnNaeCHbVfCP/mwmEu5GZaTJiR0QmXKTEoQY0rGx3NX4XnN Qayit3cWiA74CUrNV7366EsS12gu1jdYDkXmLceTL3737cNtTH+TICBDLF/M8QM= X-Google-Smtp-Source: AGHT+IEjiufoSG+9f7J9DudEHL9YuIq0UaJx7YZ52pOk+qHYx73DXmea6YZQnzcRnEEhuuY+i+vKHw== X-Received: by 2002:a05:6102:e06:b0:493:b055:bf56 with SMTP id ada2fe7eead31-4a15dc39965mr6802341137.2.1727333291940; Wed, 25 Sep 2024 23:48:11 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.48.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:48:11 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 11/13] mm: userfaultfd: move_pages_pte() use pte_offset_map_rw_nolock() Date: Thu, 26 Sep 2024 14:46:24 +0800 Message-Id: <1530e8fdbfc72eacf3b095babe139ce3d715600a.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In move_pages_pte(), we may modify the dst_pte and src_pte after acquiring the ptl, so convert it to using pte_offset_map_rw_nolock(). But since we will use pte_same() to detect the change of the pte entry, there is no need to get pmdval, so just pass a dummy variable to it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song --- mm/userfaultfd.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index ce13c40626472..48b87c62fc3dd 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -1135,7 +1135,7 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t= *dst_pmd, pmd_t *src_pmd, spinlock_t *src_ptl, *dst_ptl; pte_t *src_pte =3D NULL; pte_t *dst_pte =3D NULL; - + pmd_t dummy_pmdval; struct folio *src_folio =3D NULL; struct anon_vma *src_anon_vma =3D NULL; struct mmu_notifier_range range; @@ -1146,7 +1146,14 @@ static int move_pages_pte(struct mm_struct *mm, pmd_= t *dst_pmd, pmd_t *src_pmd, src_addr, src_addr + PAGE_SIZE); mmu_notifier_invalidate_range_start(&range); retry: - dst_pte =3D pte_offset_map_nolock(mm, dst_pmd, dst_addr, &dst_ptl); + /* + * Use the maywrite version to indicate that dst_pte will be modified, + * but since we will use pte_same() to detect the change of the pte + * entry, there is no need to get pmdval, so just pass a dummy variable + * to it. + */ + dst_pte =3D pte_offset_map_rw_nolock(mm, dst_pmd, dst_addr, &dummy_pmdval, + &dst_ptl); =20 /* Retry if a huge pmd materialized from under us */ if (unlikely(!dst_pte)) { @@ -1154,7 +1161,9 @@ static int move_pages_pte(struct mm_struct *mm, pmd_t= *dst_pmd, pmd_t *src_pmd, goto out; } =20 - src_pte =3D pte_offset_map_nolock(mm, src_pmd, src_addr, &src_ptl); + /* same as dst_pte */ + src_pte =3D pte_offset_map_rw_nolock(mm, src_pmd, src_addr, &dummy_pmdval, + &src_ptl); =20 /* * We held the mmap_lock for reading so MADV_DONTNEED --=20 2.20.1 From nobody Fri Nov 29 00:48:46 2024 Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BF62C13D276 for ; Thu, 26 Sep 2024 06:48:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333302; cv=none; b=rXbrOceCbr8kdtTCiuANxCw8E2/NRS0UgXFgG9XXgOqlPLlp5GSEeOH4EdjbtL0EodIxRgPGvEj9On4xzAxc+4WxceCodCt1U2LM59npyh92YBh0rTTdEmvrSUXgQjSkdojC+8B116Kg0w0wl2QBoJp1C44Jvj17p/jcUy0DO8Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333302; c=relaxed/simple; bh=KpepB6xnCL6q51Rr8wjzoDqqKsBPVm70U61A9qPEoYc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jUB61BhouImyNJ0jydJv1lLR9MqFFKEutnYX2C0Vk9R1EeMJqLwa4/Z6yP+1zGq7i1UnEeIj2TENbPTjqHPo/Wzzvopuj/ZQICUzq7fYPEUuIHR9OLIuMng6gUA2qdYtY8uc7RCZVeO+A7KKGSez1Hq8G/ZXYGshg7uqLXFY3Bs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=Jc1DqtAp; arc=none smtp.client-ip=209.85.160.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="Jc1DqtAp" Received: by mail-qt1-f176.google.com with SMTP id d75a77b69052e-458311b338eso4504891cf.2 for ; Wed, 25 Sep 2024 23:48:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333300; x=1727938100; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=wKWf8ZYo9tlbP3964sEqp7HhdDy32FNbDdGL8iSt+Ok=; b=Jc1DqtApy/SbXyTgs0vmdhDWkpAG72ca1vOOMrZHwwaVMjvLCl62d1MS+lUBqV/7wd b9staiftbiSvo61J+bDD7RMymty/bZHE55ZuT8stCiht/3BIO8NU/HyZEEHdnY24ewVD Rc/p1hHIz0HVG2wIrLZ8jfSbfIRMtwfdcTWS+NJkDIJb8KspENIeNEy7WChbaTMybOnA as7FJ1cqnmxdIbtCK+Q1o0aO/efy2jWHrhWpWVnkoefQ2GwWyhcmvyCsb8kXdIlOYMWf Jizcnj+OMmALaNCshsrYw36RcY+3ko1AZ15s/cql6hSePm+HlBY30rY4jHo2ab1ZFwhE zcKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333300; x=1727938100; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wKWf8ZYo9tlbP3964sEqp7HhdDy32FNbDdGL8iSt+Ok=; b=Rdrsc8FVcnH4wxTFAz1dB4YuPTcLKqgZaavTmoTEiyINisrKuOnHun+FYuFuywJpzy ZEvgV3UHfveBrhsXqv/LBkOR4E6iyA9X7TyRmp33kcv7/AtJTg/pmhlcgI8opdFW+59n PEqNLuhuQ69kYv96VHZgD3BAbFlJy5rgeAgyDV/9UCSPzi7x00MzABJz2SYYZhHa2NS6 G50U6RB4OrQq+XOiZLjLoBjZZ/xyRv0EXHB0lFnaTSaGRF7R8c8cvCFx3jyC2nMHhioc H/iZu2UPSd3h/5VUn+IlLNRRjhNHFyCs5cddjOTlj8AAK9noHdBrP9KDhpmldtFnhPKS Jg1w== X-Gm-Message-State: AOJu0YzC3SvLMA3klyNi4EuMiqcAtIcGbULL+nmhXk7OAQM7kgNRaLla dPvqCPzrQT/fP4CiZ1YcCVcC1pWVdgwjrPGdimsxJcVEX+UbgK6ZV/F/bwVSkhU= X-Google-Smtp-Source: AGHT+IH0PykjARYOlhRdYD8Ei00ntSXkZvlp+w/DvL/wK29HFMzX/PK1qWz6iq3HMQhndU3PBLr5iA== X-Received: by 2002:a05:622a:51:b0:458:3e20:65d3 with SMTP id d75a77b69052e-45b5e06627amr81705351cf.60.1727333299714; Wed, 25 Sep 2024 23:48:19 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.48.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:48:19 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 12/13] mm: multi-gen LRU: walk_pte_range() use pte_offset_map_rw_nolock() Date: Thu, 26 Sep 2024 14:46:25 +0800 Message-Id: <7e9c194a5efacc9609cfd31abb9c7df88b53b530.1727332572.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" In walk_pte_range(), we may modify the pte entry after holding the ptl, so convert it to using pte_offset_map_rw_nolock(). At this time, the pte_same() check is not performed after the ptl held, so we should get pmdval and do pmd_same() check to ensure the stability of pmd entry. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song Acked-by: David Hildenbrand --- mm/vmscan.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 749cdc110c745..bdca94e663bc5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3375,8 +3375,10 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long= start, unsigned long end, struct pglist_data *pgdat =3D lruvec_pgdat(walk->lruvec); DEFINE_MAX_SEQ(walk->lruvec); int old_gen, new_gen =3D lru_gen_from_seq(max_seq); + pmd_t pmdval; =20 - pte =3D pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK, &ptl); + pte =3D pte_offset_map_rw_nolock(args->mm, pmd, start & PMD_MASK, &pmdval, + &ptl); if (!pte) return false; if (!spin_trylock(ptl)) { @@ -3384,6 +3386,11 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long= start, unsigned long end, return false; } =20 + if (unlikely(!pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + pte_unmap_unlock(pte, ptl); + return false; + } + arch_enter_lazy_mmu_mode(); restart: for (i =3D pte_index(start), addr =3D start; addr !=3D end; i++, addr += =3D PAGE_SIZE) { --=20 2.20.1 From nobody Fri Nov 29 00:48:46 2024 Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7659418C013 for ; Thu, 26 Sep 2024 06:48:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333310; cv=none; b=NSruVibWd4q+skO7R0oPu9tc9HzRI8PrxC99teD+XLAbFCgy9x/1OCpY4NPcQLnrLvzOXPihd0bn06gLTb8awzYX7i1ElsM/bwiHmnCFf03nTL2M3mZgPi8TUjBKF7NzEtixVD/R7un3lYqU1gAqS9Xb29XQEey6//xmOJJfnZE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1727333310; c=relaxed/simple; bh=extMSNBrZETWpUQUJhKJu/3q7Zw4ufuaAk1jyXFXQTg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=aUjiJlOL3U9aHmLMf63MuUrsWWkNUDmj/UtQdzBUStTzraftFcbEGjt1g+uGEU61gmVNc0WZO15Z+ZnS8uqiv0VpzF2QklTvMm2diyMWBaMolpvYUByc1CFORVQPaq1htLD4S2NrLvrMJfCp010BuE6MfyvQBzt5+7Wr4w5dQNA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com; spf=pass smtp.mailfrom=bytedance.com; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b=DdvRyMbm; arc=none smtp.client-ip=209.85.222.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=bytedance.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=bytedance.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=bytedance.com header.i=@bytedance.com header.b="DdvRyMbm" Received: by mail-qk1-f178.google.com with SMTP id af79cd13be357-7a99fdf2e1aso75144685a.2 for ; Wed, 25 Sep 2024 23:48:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1727333307; x=1727938107; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XK9xLAd1aeWWLl9nHV9gcamA18VQvE5iTgorpb3Ql3Q=; b=DdvRyMbmuRpYlAWdqR7eSOgytLQTCWEIRd1lSnf2R8XLRjt8o7vD5p3k28/65dzuGi HcsuBLrv5fS+Huw+440SfjyBxi+durdgb0OSdVQYR7m8hVM8IKedaTYOYF95hcBHcNjC nu1NinOapX55OHGRbBZyR6j3mybcZn6LL4RmEAbI9qww1T4T+R4DhKpuTQlPplPqmu+t tQzzrAI8mtw71U6BXnOi/BkDCc0D+izQ5sEFdTALYEKO2tFYiRmHPfib6BoaG78ghbAW POw5duGX9JmT6SlM1Eb51uEjv/MlLjz+I2QLx6ogJmty8pUkUrASZJO9iT52PNXZ5LTZ cZkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727333307; x=1727938107; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XK9xLAd1aeWWLl9nHV9gcamA18VQvE5iTgorpb3Ql3Q=; b=gPYR/JQd4Jzhntwma2ESDQyHfuJZbxEap8m17FUdoYb3JVMkD/MIZ23SWOWCgpQrM4 jcGyNT1WptMjXRWVezNFvMusioLTKbnDNN1YY86G4g8938pO1fO+Z7S385s4GvR6mIKG DEzVf9Psoi4aaj48TTo2y2s2i4aLlAZ0nK3+xAwAWIHyFxh8/FsXkHeSPbWgJUKgZXZU ePAah3AiHMpLOT7/hrfkk8nN1RdvOQ+o6M6TH4cpUYMLJ3GbaLpNgQ4r4F9ZmBqrJcNO /gd/ZoB55mpDKb9D62Rqzy+pv77p3y+FCCLQ5uxFRFYFJljmPHi4QBh3q6UJaURgS5wy jaWw== X-Gm-Message-State: AOJu0Yx2ycauafjM26dxd2uU0u/y+5GCq/ZUxSXMeRztW7VFNgJnmvMI 8JxgvWEZs+SNW8KDD2bvLb+AUobYT5bqEJhz51AzqNSRCsxLM2ZjQ1aXcMe4iXA= X-Google-Smtp-Source: AGHT+IHnYRThLjQtuqQ25g++f96or6jGXpOGKBOHjBYtOEzyfC2Zv3m6NDV/6g7eYEpePNjEvQY05w== X-Received: by 2002:a05:622a:2996:b0:458:37c6:46fa with SMTP id d75a77b69052e-45b5def2036mr66535021cf.28.1727333307294; Wed, 25 Sep 2024 23:48:27 -0700 (PDT) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.150]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-45b5257ff1esm23024611cf.38.2024.09.25.23.48.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 23:48:26 -0700 (PDT) From: Qi Zheng To: david@redhat.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, rppt@kernel.org, vishal.moola@gmail.com, peterx@redhat.com, ryan.roberts@arm.com, christophe.leroy2@cs-soprasteria.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, Qi Zheng Subject: [PATCH v5 13/13] mm: pgtable: remove pte_offset_map_nolock() Date: Thu, 26 Sep 2024 14:46:26 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now no users are using the pte_offset_map_nolock(), remove it. Signed-off-by: Qi Zheng Reviewed-by: Muchun Song Acked-by: David Hildenbrand --- Documentation/mm/split_page_table_lock.rst | 3 --- include/linux/mm.h | 2 -- mm/pgtable-generic.c | 21 --------------------- 3 files changed, 26 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/= split_page_table_lock.rst index 08d0e706a32db..581446d4a4eba 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -16,9 +16,6 @@ There are helpers to lock/unlock a table and other access= or functions: - pte_offset_map_lock() maps PTE and takes PTE table lock, returns pointer to PTE with pointer to its PTE table lock, or returns NULL if no PTE table; - - pte_offset_map_nolock() - maps PTE, returns pointer to PTE with pointer to its PTE table - lock (not taken), or returns NULL if no PTE table; - pte_offset_map_ro_nolock() maps PTE, returns pointer to PTE with pointer to its PTE table lock (not taken), or returns NULL if no PTE table; diff --git a/include/linux/mm.h b/include/linux/mm.h index 46828b9a74f2c..60a55b28cf679 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3015,8 +3015,6 @@ static inline pte_t *pte_offset_map_lock(struct mm_st= ruct *mm, pmd_t *pmd, return pte; } =20 -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp); pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd, diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index daa08b91ab6b2..5297dcc38c37a 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -305,18 +305,6 @@ pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr= , pmd_t *pmdvalp) return NULL; } =20 -pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, - unsigned long addr, spinlock_t **ptlp) -{ - pmd_t pmdval; - pte_t *pte; - - pte =3D __pte_offset_map(pmd, addr, &pmdval); - if (likely(pte)) - *ptlp =3D pte_lockptr(mm, &pmdval); - return pte; -} - pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd, unsigned long addr, spinlock_t **ptlp) { @@ -372,15 +360,6 @@ pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, = pmd_t *pmd, * and disconnected table. Until pte_unmap(pte) unmaps and rcu_read_unloc= k()s * afterwards. * - * pte_offset_map_nolock(mm, pmd, addr, ptlp), above, is like pte_offset_m= ap(); - * but when successful, it also outputs a pointer to the spinlock in ptlp = - as - * pte_offset_map_lock() does, but in this case without locking it. This = helps - * the caller to avoid a later pte_lockptr(mm, *pmd), which might by that = time - * act on a changed *pmd: pte_offset_map_nolock() provides the correct spi= nlock - * pointer for the page table that it returns. In principle, the caller s= hould - * recheck *pmd once the lock is taken; in practice, no callsite needs tha= t - - * either the mmap_lock for write, or pte_same() check on contents, is eno= ugh. - * * pte_offset_map_ro_nolock(mm, pmd, addr, ptlp), above, is like pte_offse= t_map(); * but when successful, it also outputs a pointer to the spinlock in ptlp = - as * pte_offset_map_lock() does, but in this case without locking it. This = helps --=20 2.20.1