From nobody Fri May 17 05:54:37 2024 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E22DD161912 for ; Thu, 18 Apr 2024 13:44:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713447889; cv=none; b=LaY9xcwcuCn+Wl2GnwEnO8k05WmIDoF3RBVdHWigmjvlrdHiBt9K379W/HmqENK1q3qXKNxkCgyW08mX5J7AjnsLHaU5jWrLSCwPheQfyjGxP2lQo1mkCbLKNFgRGiggOkUyMkaaFlZmfiQJ9sxAaB3zBLzMLSyYd4GE/1w2Uc4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713447889; c=relaxed/simple; bh=YS1E84UOxIjSnPESm8G/PnNhVqV2TAef9FqSYBszYkA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Xr90U158FMO9CtUjb935+ei5mYUY8vtaKy7EuibSVOaVurw3055IdQ1VyYF0rLchw4LboDWIidj5hbHOIW8CWK6NS6ne15tHPWOy1pdjJu7SnWboNttkVxwqaVyUX/uSH1RGHPYfrAb50Uw/dHQPiqnZfRZN+5+NyQUPyTkoR1M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Za4/TxTW; arc=none smtp.client-ip=209.85.214.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Za4/TxTW" Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-1e4f341330fso7427265ad.0 for ; Thu, 18 Apr 2024 06:44:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713447887; x=1714052687; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=lkdNcUi/cFtD91tOAv56m6RXgS21TIueDzxwluwn/Lw=; b=Za4/TxTWOepEKyGGfR0FHxnhf2XdyASegoAdc2nKQ+2RExImc/taLIT0k1Msa2C1vM c7WXRd2DtgCPN9qUv0UXj5ekDJZwOlqE6GF0l9VcRJriyQnt3N4wf6tBpsCFh5yiJbVj KDn0vnwxjWhZXTx5utzGWloemW+p83axogczbZ1C0bUM+t9GOZxshyqEjLfG/Z1oxF0J wgOjAyuVBTAYcEh3jo0VVzCuCCvh1jLX+hfbHipcEMG7cB1L4VD7ugrIZiYyYdPrAEdh P/7Ho8uMmQRAnGZ/HdfHXYm/lvaB8ykMvWVsvMXGDcIWxWQ8oVbtVj+OiBv1jOv8OIzL yWTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713447887; x=1714052687; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lkdNcUi/cFtD91tOAv56m6RXgS21TIueDzxwluwn/Lw=; b=CAZJRv1YGY7mXGU9n3LQ+r5Y68Ya5FQzuRaNm/lWZcFQjOqcctq+RfoijXcCGA/vnG kv81XqkReANTv08bsrhmzJYXy9QI7OQyKC4/UNNx8oRC7Al1dB8o0rF6crn3/kyBI5si BpcjPJ55EricNUxqUxaz/KlcsbZogtf4uXDIJfKsmCr0EKoaUs+mbIe6XFAsoIh93Qgb Yfuz7MUpbWmYK92zAEhQ0gMGTwK/g8Ij2Ch53R+ZTK0pDt1iFNMENBizX5dXlKQvKknv Hzh69ww8FSQDJDqQSRwLelMXRkLyfAgdCLZj3DnHSxE+AlhcgKXeL7VADaeBAzHSj4S2 +rzg== X-Forwarded-Encrypted: i=1; AJvYcCVD+f43iHiwu1njspxqs/9PRX4LRJ2OTvX+WIdAtzm3iWEoot1iwbRAaoxJMPP25NdTdDpJ2byVHDRvFUy3wRuiVepqw+3R0ojXoEuB X-Gm-Message-State: AOJu0YzCPVJhrBHC7qp8MgJbcga2mwH00rdCpuBlyWEoF/KIjK/K/i6R untUZjsgSpSzGfmaiGx0qjPj002PCShTVVEN0NMeF9AKcdMpr3Pv X-Google-Smtp-Source: AGHT+IG2GVYbLpKp4yUNNUcOXfACY+2bHpA67aILzi27vexy/CeFyGpA5xeZ5gYV+u3w513tsrVcFA== X-Received: by 2002:a17:902:c94e:b0:1e7:b775:64bd with SMTP id i14-20020a170902c94e00b001e7b77564bdmr3953276pla.53.1713447887193; Thu, 18 Apr 2024 06:44:47 -0700 (PDT) Received: from LancedeMBP.lan ([112.10.225.217]) by smtp.gmail.com with ESMTPSA id d8-20020a170902b70800b001e4fdcf67desm1504837pls.299.2024.04.18.06.44.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 06:44:46 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v10 1/4] mm/madvise: introduce clear_young_dirty_ptes() batch helper Date: Thu, 18 Apr 2024 21:44:32 +0800 Message-Id: <20240418134435.6092-2-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20240418134435.6092-1-ioworker0@gmail.com> References: <20240418134435.6092-1-ioworker0@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit introduces clear_young_dirty_ptes() to replace mkold_ptes(). By doing so, we can use the same function for both use cases (madvise_pageout and madvise_free), and it also provides the flexibility to only clear the dirty flag in the future if needed. Suggested-by: Ryan Roberts Acked-by: David Hildenbrand Reviewed-by: Ryan Roberts Signed-off-by: Lance Yang --- include/linux/mm_types.h | 9 +++++ include/linux/pgtable.h | 74 ++++++++++++++++++++++++---------------- mm/madvise.c | 3 +- 3 files changed, 55 insertions(+), 31 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index db0adf5721cc..24323c7d0bd4 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1368,6 +1368,15 @@ enum fault_flag { =20 typedef unsigned int __bitwise zap_flags_t; =20 +/* Flags for clear_young_dirty_ptes(). */ +typedef int __bitwise cydp_t; + +/* Clear the access bit */ +#define CYDP_CLEAR_YOUNG ((__force cydp_t)BIT(0)) + +/* Clear the dirty bit */ +#define CYDP_CLEAR_DIRTY ((__force cydp_t)BIT(1)) + /* * FOLL_PIN and FOLL_LONGTERM may be used in various combinations with each * other. Here is what they mean, and how to use them: diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e2f45e22a6d1..18019f037bae 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -361,36 +361,6 @@ static inline int ptep_test_and_clear_young(struct vm_= area_struct *vma, } #endif =20 -#ifndef mkold_ptes -/** - * mkold_ptes - Mark PTEs that map consecutive pages of the same folio as = old. - * @vma: VMA the pages are mapped into. - * @addr: Address the first page is mapped at. - * @ptep: Page table pointer for the first entry. - * @nr: Number of entries to mark old. - * - * May be overridden by the architecture; otherwise, implemented as a simp= le - * loop over ptep_test_and_clear_young(). - * - * Note that PTE bits in the PTE range besides the PFN can differ. For exa= mple, - * some PTEs might be write-protected. - * - * Context: The caller holds the page table lock. The PTEs map consecutive - * pages that belong to the same folio. The PTEs are all in the same PMD. - */ -static inline void mkold_ptes(struct vm_area_struct *vma, unsigned long ad= dr, - pte_t *ptep, unsigned int nr) -{ - for (;;) { - ptep_test_and_clear_young(vma, addr, ptep); - if (--nr =3D=3D 0) - break; - ptep++; - addr +=3D PAGE_SIZE; - } -} -#endif - #ifndef __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEA= F_PMD_YOUNG) static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, @@ -489,6 +459,50 @@ static inline pte_t ptep_get_and_clear(struct mm_struc= t *mm, } #endif =20 +#ifndef clear_young_dirty_ptes +/** + * clear_young_dirty_ptes - Mark PTEs that map consecutive pages of the + * same folio as old/clean. + * @mm: Address space the pages are mapped into. + * @addr: Address the first page is mapped at. + * @ptep: Page table pointer for the first entry. + * @nr: Number of entries to mark old/clean. + * @flags: Flags to modify the PTE batch semantics. + * + * May be overridden by the architecture; otherwise, implemented by + * get_and_clear/modify/set for each pte in the range. + * + * Note that PTE bits in the PTE range besides the PFN can differ. For exa= mple, + * some PTEs might be write-protected. + * + * Context: The caller holds the page table lock. The PTEs map consecutive + * pages that belong to the same folio. The PTEs are all in the same PMD. + */ +static inline void clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags) +{ + pte_t pte; + + for (;;) { + if (flags =3D=3D CYDP_CLEAR_YOUNG) + ptep_test_and_clear_young(vma, addr, ptep); + else { + pte =3D ptep_get_and_clear(vma->vm_mm, addr, ptep); + if (flags & CYDP_CLEAR_YOUNG) + pte =3D pte_mkold(pte); + if (flags & CYDP_CLEAR_DIRTY) + pte =3D pte_mkclean(pte); + set_pte_at(vma->vm_mm, addr, ptep, pte); + } + if (--nr =3D=3D 0) + break; + ptep++; + addr +=3D PAGE_SIZE; + } +} +#endif + static inline void ptep_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { diff --git a/mm/madvise.c b/mm/madvise.c index 4b869b682fd5..f5e3699e7b54 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -507,7 +507,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, continue; =20 if (!pageout && pte_young(ptent)) { - mkold_ptes(vma, addr, pte, nr); + clear_young_dirty_ptes(vma, addr, pte, nr, + CYDP_CLEAR_YOUNG); tlb_remove_tlb_entries(tlb, pte, nr, addr); } =20 --=20 2.33.1 From nobody Fri May 17 05:54:37 2024 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E6F8B1635CA for ; Thu, 18 Apr 2024 13:44:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713447893; cv=none; b=WJhYREDlGzsTNXUZ7986ZXOqgkJ2ogFucxeL7CyISn5u6L23iGrCc7Q3DZCpZrHEkyB+cfR3Y2e/IzFL0kkFR7zbhn+e5qcl4OZ0mYWrcp+VkIFWmXZLfJsxzWTANitRUwzdqSEyp0Dd5qe403AyrpVlVenFeyMc6PIuMPGGF9k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713447893; c=relaxed/simple; bh=vs6Beq48+kXNT5qsMOGp08C7Ii4wZzzBthcyqUl9lGQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=nN/6h94PItWrYu8WDQ0IyMOatLnx3CoqG0Fk3Qq7/2XVpOemLu0GpDTXc/VjU7yFX7sW9d2dFFZUjiJU4JuJZmoEBqZCgnNVkNTY5RVHwSMi6c1sl8FjRFKU/2d3LA0QN2T9WtqOfHvzLcncqxtSneYFVB0dNBZeS+9ZwGk+fdg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=fy1jzZe1; arc=none smtp.client-ip=209.85.214.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fy1jzZe1" Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-1e2178b2cf2so7522085ad.0 for ; Thu, 18 Apr 2024 06:44:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713447891; x=1714052691; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XITq4/hgHmAHZ0ZHlNrd2qmUtOIaf33ePb9J0ZwzOSE=; b=fy1jzZe1tieU44OGut/eAKh14C0yk5j3Pr/YZ+XNHWdNgSsVSfAd9mRpYwGKpmYnK2 1OntILn48k1YHKXBCZo6UhZIvgTb5YunBD35J2nGSGEfXDVJ7BGErOON8gXDLg3R1wzC 17en/J/SZ+y9SZrL0xg/yKq4u0i5PyWW3iPNszUwts7y3RrZ7v84PjA7CQu7uDOmJj3v Bol8+mjXtzDM9pKnvgNauVlJgEnLkBF8hzg1ntM8lk1RYhykrNMu6U4YBjN4ZQABpqa6 XH8FdMnJSgE3nPKKryn9mbfToRs/DKkczDy0LY+o0dJXAzL648/v5Je+EkmYA++o+VPe p9qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713447891; x=1714052691; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XITq4/hgHmAHZ0ZHlNrd2qmUtOIaf33ePb9J0ZwzOSE=; b=N1BMGNljkWRgnVt44DlDFrEzRuA54t1A/N76NNO5bVLdKi824YxLj/pdaLS+134KvI 9sNh2KiYD8XlSHKwsG75hk2FDIayBLaWNGv7xx3s9+ggGjYeDCieEKBQIa5lDk90PiX/ bVmci+QIf3rYJAwp01SL8jXCIdAiIWqIvlMDi7v7M1DYXj/zY3FiAdW2m8sI72mgU7EJ fCyTIwIeacKrHZNvYy4U/5pWXBvx1lSYk3WUTximBjz3H/8mEccDbH9Jib08s23ah41O KxOevsxFTxuUGJKWf84xppybcZGd4vgodsJV1QVszSws668X1aTFQFrCMpXGEKUbBv6k SmKQ== X-Forwarded-Encrypted: i=1; AJvYcCXdtxMCRcNOxPQkkVLs4lT/iERWZAkguyf0tlJZ8jVsipaAiXvFsvo2FuzRUBlWrWCq/e9yi0EfHIO+1VH9oCV0KJ1FlkzYe5tcT6l8 X-Gm-Message-State: AOJu0Yx5eRpZy85SUmxifyQcNHb1ZDvhFQbmIlXpQkbKwEuJV8CXbJs3 YJzhdkNXhGHS2ZLHpja/2GvM28LMeunbgLOZcUxD2ZA1vuZFfA3E X-Google-Smtp-Source: AGHT+IFMvPYbcwWGZBEI/RPL6zIG2k4X3bC8gD2IhYt301UAgb/Og2L2LjKZC1QkdaZ8PQQDYIe8QA== X-Received: by 2002:a17:902:db0a:b0:1e2:1db3:ebeb with SMTP id m10-20020a170902db0a00b001e21db3ebebmr3985587plx.57.1713447891167; Thu, 18 Apr 2024 06:44:51 -0700 (PDT) Received: from LancedeMBP.lan ([112.10.225.217]) by smtp.gmail.com with ESMTPSA id d8-20020a170902b70800b001e4fdcf67desm1504837pls.299.2024.04.18.06.44.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 06:44:50 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v10 2/4] mm/arm64: override clear_young_dirty_ptes() batch helper Date: Thu, 18 Apr 2024 21:44:33 +0800 Message-Id: <20240418134435.6092-3-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20240418134435.6092-1-ioworker0@gmail.com> References: <20240418134435.6092-1-ioworker0@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The per-pte get_and_clear/modify/set approach would result in unfolding/refolding for contpte mappings on arm64. So we need to override clear_young_dirty_ptes() for arm64 to avoid it. Suggested-by: Barry Song <21cnbao@gmail.com> Suggested-by: Ryan Roberts Reviewed-by: Ryan Roberts Signed-off-by: Lance Yang --- arch/arm64/include/asm/pgtable.h | 55 ++++++++++++++++++++++++++++++++ arch/arm64/mm/contpte.c | 29 +++++++++++++++++ 2 files changed, 84 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index 9fd8613b2db2..1303d30287dc 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1223,6 +1223,46 @@ static inline void __wrprotect_ptes(struct mm_struct= *mm, unsigned long address, __ptep_set_wrprotect(mm, address, ptep); } =20 +static inline void __clear_young_dirty_pte(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t pte, cydp_t flags) +{ + pte_t old_pte; + + do { + old_pte =3D pte; + + if (flags & CYDP_CLEAR_YOUNG) + pte =3D pte_mkold(pte); + if (flags & CYDP_CLEAR_DIRTY) + pte =3D pte_mkclean(pte); + + pte_val(pte) =3D cmpxchg_relaxed(&pte_val(*ptep), + pte_val(old_pte), pte_val(pte)); + } while (pte_val(pte) !=3D pte_val(old_pte)); +} + +static inline void __clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags) +{ + pte_t pte; + + for (;;) { + pte =3D __ptep_get(ptep); + + if (flags =3D=3D (CYDP_CLEAR_YOUNG | CYDP_CLEAR_DIRTY)) + __set_pte(ptep, pte_mkclean(pte_mkold(pte))); + else + __clear_young_dirty_pte(vma, addr, ptep, pte, flags); + + if (--nr =3D=3D 0) + break; + ptep++; + addr +=3D PAGE_SIZE; + } +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_PMDP_SET_WRPROTECT static inline void pmdp_set_wrprotect(struct mm_struct *mm, @@ -1379,6 +1419,9 @@ extern void contpte_wrprotect_ptes(struct mm_struct *= mm, unsigned long addr, extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty); +extern void contpte_clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags); =20 static __always_inline void contpte_try_fold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte) @@ -1603,6 +1646,17 @@ static inline int ptep_set_access_flags(struct vm_ar= ea_struct *vma, return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); } =20 +#define clear_young_dirty_ptes clear_young_dirty_ptes +static inline void clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags) +{ + if (likely(nr =3D=3D 1 && !pte_cont(__ptep_get(ptep)))) + __clear_young_dirty_ptes(vma, addr, ptep, nr, flags); + else + contpte_clear_young_dirty_ptes(vma, addr, ptep, nr, flags); +} + #else /* CONFIG_ARM64_CONTPTE */ =20 #define ptep_get __ptep_get @@ -1622,6 +1676,7 @@ static inline int ptep_set_access_flags(struct vm_are= a_struct *vma, #define wrprotect_ptes __wrprotect_ptes #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags +#define clear_young_dirty_ptes __clear_young_dirty_ptes =20 #endif /* CONFIG_ARM64_CONTPTE */ =20 diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 1b64b4c3f8bf..9f9486de0004 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -361,6 +361,35 @@ void contpte_wrprotect_ptes(struct mm_struct *mm, unsi= gned long addr, } EXPORT_SYMBOL_GPL(contpte_wrprotect_ptes); =20 +void contpte_clear_young_dirty_ptes(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + unsigned int nr, cydp_t flags) +{ + /* + * We can safely clear access/dirty without needing to unfold from + * the architectures perspective, even when contpte is set. If the + * range starts or ends midway through a contpte block, we can just + * expand to include the full contpte block. While this is not + * exactly what the core-mm asked for, it tracks access/dirty per + * folio, not per page. And since we only create a contpte block + * when it is covered by a single folio, we can get away with + * clearing access/dirty for the whole block. + */ + unsigned long start =3D addr; + unsigned long end =3D start + nr; + + if (pte_cont(__ptep_get(ptep + nr - 1))) + end =3D ALIGN(end, CONT_PTE_SIZE); + + if (pte_cont(__ptep_get(ptep))) { + start =3D ALIGN_DOWN(start, CONT_PTE_SIZE); + ptep =3D contpte_align_down(ptep); + } + + __clear_young_dirty_ptes(vma, start, ptep, end - start, flags); +} +EXPORT_SYMBOL_GPL(contpte_clear_young_dirty_ptes); + int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty) --=20 2.33.1 From nobody Fri May 17 05:54:37 2024 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E7F481649A7 for ; Thu, 18 Apr 2024 13:44:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713447897; cv=none; b=U8K+P/fHEr8syUPTgEu/JiwL0LgyIHrCR3Q6HeQcWxz8vV3v3JoLL4DdRkj0tghx+Fot78LSXORRpy9nSrVJ69HoX3vcULi3CyW+0wA68onsOS1Csc1LkVIwYO2kVr3BFz6dtQi3OAT6EhWsn6uqszMvcu+r40HVJFIhWmMLJPY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713447897; c=relaxed/simple; bh=TeV6kOnPwbJywIxNf+Gh2TXpm1wrP6kxlJyxKo6iva0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=a/EFKk02rqKVTPnaHdLja21GMHAcjgHYFzuH9lUam985GGW/WY6ReT3HjJQVEjwtCVOfFpkHm1D3pIuqnZGvEGJ76DFR9uWuD7IhChFCs5tmPxQMWA68MZw9xGcaKNt9ech9VQcK1Ar+bCtzqkOrKqsRXD49xIjRn/kge6MS/0o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=DQeGm5C6; arc=none smtp.client-ip=209.85.214.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DQeGm5C6" Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-1e65b29f703so7466445ad.3 for ; Thu, 18 Apr 2024 06:44:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713447895; x=1714052695; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2bptmvyfGYVn+izLGYKs1uKRCf7NdX4woHvJn9zQskE=; b=DQeGm5C6oBiGyuho9X5b3r/rOjx1hpBJ/Qh1LjAal7/pxo0krhU9wz4s5AldDdOxSb ZustjUZ+G1UoqG2cMUoSrG1H46aCPknw5xkr2vX6HW0n4vxGkMMhwaKLhxDsQ7Ltx0D9 4NC99SGb2X6LfW3V1uY9DflcVE3LA10PlEO2fEMZvOZ79WWYdai8BvwL/DLZOk2YnVNy 7axO4enxp6cpQxl/Pj383DrKtrfqz/l+OJYnHvqKeumxHrlEkaNBBLUNM9ZvOXnbjAjr i0ZuYBrLt1zEL2P7Hi8f/rFGEAfnFuzpmo9rg6w7H6JwJ1Z3aFUa7SucWhqF256fSEtW sFFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713447895; x=1714052695; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2bptmvyfGYVn+izLGYKs1uKRCf7NdX4woHvJn9zQskE=; b=UcJBeuKGlUVMtrOXjspqI2XeSJIk4fGNLgyRdH1xkrPO4y+/l51mQbznctzpXNGcnj 9hw6A7yUnfGep54IZZX4OSo1i4JDZVm59fFCinUn7TwYVG+PIeH+FiqjZ6+g2TB1I1nh C4utGkXW0xjeoFbNXvvdaDEjqARLNlkUDzib53arSjF6LOivOb5B5UjupC5pJYDJ3o/5 SOB0dHOKt47u7Q/CsaSsqF3QrEEsbNJwem5khUGx67PVoFXit7Rl/Kq9yd3az2KVzj4t yobZPLDAclJPa40IfF2uOqcK7pb6rOY2FsfuDPHWeTp56m+BUKnacw5xDZQNGXBgt7oM PVjQ== X-Forwarded-Encrypted: i=1; AJvYcCVQEU5qivaU7eNZd8xwvMcs47pd/a3BbBfAqugzkCMjOzCv0lnnQkijDpK7Jg14Uy0Siq2vxzo/MVdt7rLEsgO2/CyP4L2vWnovQ9Vo X-Gm-Message-State: AOJu0YzlZZENBoT2FLeOBlfItS6i0bWpgCESPqYSPrPT0+PpXS0FdbUl v6gOJYbuRnhwZM/B2wwlCwI0J1dOCrEXUG4pvdDr7VRVd9jc9M+m X-Google-Smtp-Source: AGHT+IFF3gVncyN3TR7hbP07PirPfqqnSr7IxJxLr9POXIPSQcFBSJypWZU1PP2Je3jZ98XwlW4btg== X-Received: by 2002:a17:902:f687:b0:1e4:3e67:2bbb with SMTP id l7-20020a170902f68700b001e43e672bbbmr2797169plg.48.1713447895139; Thu, 18 Apr 2024 06:44:55 -0700 (PDT) Received: from LancedeMBP.lan ([112.10.225.217]) by smtp.gmail.com with ESMTPSA id d8-20020a170902b70800b001e4fdcf67desm1504837pls.299.2024.04.18.06.44.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 06:44:54 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v10 3/4] mm/memory: add any_dirty optional pointer to folio_pte_batch() Date: Thu, 18 Apr 2024 21:44:34 +0800 Message-Id: <20240418134435.6092-4-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20240418134435.6092-1-ioworker0@gmail.com> References: <20240418134435.6092-1-ioworker0@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit adds the any_dirty pointer as an optional parameter to folio_pte_batch() function. By using both the any_young and any_dirty point= ers, madvise_free can make smarter decisions about whether to clear the PTEs when marking large folios as lazyfree. Suggested-by: David Hildenbrand Acked-by: David Hildenbrand Signed-off-by: Lance Yang --- mm/internal.h | 12 ++++++++++-- mm/madvise.c | 19 ++++++++++++++----- mm/memory.c | 4 ++-- 3 files changed, 26 insertions(+), 9 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index c6483f73ec13..daa59cef85d7 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -134,6 +134,8 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte= , fpb_t flags) * first one is writable. * @any_young: Optional pointer to indicate whether any entry except the * first one is young. + * @any_dirty: Optional pointer to indicate whether any entry except the + * first one is dirty. * * Detect a PTE batch: consecutive (present) PTEs that map consecutive * pages of the same large folio. @@ -149,18 +151,20 @@ static inline pte_t __pte_batch_clear_ignored(pte_t p= te, fpb_t flags) */ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, pte_t *start_ptep, pte_t pte, int max_nr, fpb_t flags, - bool *any_writable, bool *any_young) + bool *any_writable, bool *any_young, bool *any_dirty) { unsigned long folio_end_pfn =3D folio_pfn(folio) + folio_nr_pages(folio); const pte_t *end_ptep =3D start_ptep + max_nr; pte_t expected_pte, *ptep; - bool writable, young; + bool writable, young, dirty; int nr; =20 if (any_writable) *any_writable =3D false; if (any_young) *any_young =3D false; + if (any_dirty) + *any_dirty =3D false; =20 VM_WARN_ON_FOLIO(!pte_present(pte), folio); VM_WARN_ON_FOLIO(!folio_test_large(folio) || max_nr < 1, folio); @@ -176,6 +180,8 @@ static inline int folio_pte_batch(struct folio *folio, = unsigned long addr, writable =3D !!pte_write(pte); if (any_young) young =3D !!pte_young(pte); + if (any_dirty) + dirty =3D !!pte_dirty(pte); pte =3D __pte_batch_clear_ignored(pte, flags); =20 if (!pte_same(pte, expected_pte)) @@ -193,6 +199,8 @@ static inline int folio_pte_batch(struct folio *folio, = unsigned long addr, *any_writable |=3D writable; if (any_young) *any_young |=3D young; + if (any_dirty) + *any_dirty |=3D dirty; =20 nr =3D pte_batch_hint(ptep, pte); expected_pte =3D pte_advance_pfn(expected_pte, nr); diff --git a/mm/madvise.c b/mm/madvise.c index f5e3699e7b54..4597a3568e7e 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -321,6 +321,18 @@ static inline bool can_do_file_pageout(struct vm_area_= struct *vma) file_permission(vma->vm_file, MAY_WRITE) =3D=3D 0; } =20 +static inline int madvise_folio_pte_batch(unsigned long addr, unsigned lon= g end, + struct folio *folio, pte_t *ptep, + pte_t pte, bool *any_young, + bool *any_dirty) +{ + const fpb_t fpb_flags =3D FPB_IGNORE_DIRTY | FPB_IGNORE_SOFT_DIRTY; + int max_nr =3D (end - addr) / PAGE_SIZE; + + return folio_pte_batch(folio, addr, ptep, pte, max_nr, fpb_flags, NULL, + any_young, any_dirty); +} + static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -456,13 +468,10 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *p= md, * next pte in the range. */ if (folio_test_large(folio)) { - const fpb_t fpb_flags =3D FPB_IGNORE_DIRTY | - FPB_IGNORE_SOFT_DIRTY; - int max_nr =3D (end - addr) / PAGE_SIZE; bool any_young; =20 - nr =3D folio_pte_batch(folio, addr, pte, ptent, max_nr, - fpb_flags, NULL, &any_young); + nr =3D madvise_folio_pte_batch(addr, end, folio, pte, + ptent, &any_young, NULL); if (any_young) ptent =3D pte_mkyoung(ptent); =20 diff --git a/mm/memory.c b/mm/memory.c index 33d87b64d15d..9e07d1b9020c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -989,7 +989,7 @@ copy_present_ptes(struct vm_area_struct *dst_vma, struc= t vm_area_struct *src_vma flags |=3D FPB_IGNORE_SOFT_DIRTY; =20 nr =3D folio_pte_batch(folio, addr, src_pte, pte, max_nr, flags, - &any_writable, NULL); + &any_writable, NULL, NULL); folio_ref_add(folio, nr); if (folio_test_anon(folio)) { if (unlikely(folio_try_dup_anon_rmap_ptes(folio, page, @@ -1558,7 +1558,7 @@ static inline int zap_present_ptes(struct mmu_gather = *tlb, */ if (unlikely(folio_test_large(folio) && max_nr !=3D 1)) { nr =3D folio_pte_batch(folio, addr, pte, ptent, max_nr, fpb_flags, - NULL, NULL); + NULL, NULL, NULL); =20 zap_present_folio_ptes(tlb, vma, folio, page, pte, ptent, nr, addr, details, rss, force_flush, --=20 2.33.1 From nobody Fri May 17 05:54:37 2024 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C06AA161900 for ; Thu, 18 Apr 2024 13:44:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713447901; cv=none; b=Qcz+EaoMlfdcmXZUjzpZn46IAajPyCyfmZAbvs0Mey+BRzT7Qyjp6NET5B90WCi6zGPJtchqBAMpvXRkhmypbdPoU5gG1lEeJeoCs6LRxqpW0a+v/hSBD2LXNt7jWWpk3e/v1PMlrbKCHjydHw10TMJgcVdVpRKDx10u+ads3vs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1713447901; c=relaxed/simple; bh=zeLCQtAMPyhlZ+GIepcWKCZyjf329mCJRgPJQfRGg8E=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=gmCkGfr81zoh59GvRU7+lKIUyUcwTh20oCmFvIGTrlAYFxeVC7jolbauU+SHvv9gOMxUDlyuzzcu4TOrh5PgFXM2siOVIGaUNVbvp4NEKAYzCDQ/saqfRZdbgquxeXaWD3Y7sru0lLih9b6Sm5EBQBmKOHAMslngl2jdZyDbNoU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=CmyBFRgo; arc=none smtp.client-ip=209.85.214.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CmyBFRgo" Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-1e4266673bbso8193155ad.2 for ; Thu, 18 Apr 2024 06:44:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1713447899; x=1714052699; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jgQ7Fv4qoOj50C93E985pB5vnNpjpXSugVdGk65lNOA=; b=CmyBFRgoyjSLjVBf2N/JmOGZNfIJhGolbU79rJ15FrkFB8dExT3DuWoCAHZQ0nbLii scK2C7FvhghrDBoh79iMsX94RB28oYIZqreh+913GQUHEimQ51AVu4q0ghkQy4vcMvi9 aQkfOJUVmFl4HNEHZcmeI0Rqy8JN7BjPIOCvO+FMqMBmtbNVxkNVdisb/gxV5DLExe5g 1yeVf4hvu8+Y91QNfIkQrctnj9hW7v5x/GhcOyOmU1ytn6lVasBIOkJWWB3VDPI61lKQ uPiJVn591s3HjcMSTXM0nYNA1n08HrvhY5o7wCtcWP2pP6F4w/vTTLdMwtsZfOrKzhkd JHYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713447899; x=1714052699; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jgQ7Fv4qoOj50C93E985pB5vnNpjpXSugVdGk65lNOA=; b=MLRMi6hll+FeOZTtsYYQql9JgELLqcl37ubrfvvvoUpr+N5uSKx3axVlEEYqD5n71t b9c5L1DMbXtNaDwmLCrSAYnHuyse9sGgHK08U3HyxEHRoucl6F+/4d/nU7ZpXXGq0+Yg hKoxfei91q8WFitXJNJcnImhG3n5WaTz7CTs+wr5GyjtXhyVCnXHRptENULq3k0SpFqt X9mvcBAs6VbQQUdJSw6K3gZ17uFshL17HlQM1Dof4BjKTcZzKChXuhOxrz2RPYHj86cs L3Hd80Sjjr+vrgnDczidxkKGK8lSh2wrkj9/pNah5Q/dQ/04L13TGuIOXYJgDpeZeRx4 LScA== X-Forwarded-Encrypted: i=1; AJvYcCVYy1JHdYa4RIBZy/uKn2z9RcxRARrIvpxziyWDBauR4Nz03Twgzc6bQTRIGosKHrBwb0iXfOhq/Qke+vsNLvuEUXkOKzTW1l17KLg9 X-Gm-Message-State: AOJu0Yw/zblmyRe1ESiQbSTAqoqwSslrp0IvcTUtnKeWAXMsmpNMY5/r q3I8KqirxsIB/N3MqWWJaa3GwfFUdZ92wY8CTXkWZ9heZJwzt0PJSteKr2Nx X-Google-Smtp-Source: AGHT+IGGqaeZJIZi8LM0BZvZXjbrMIOv72J2g3c12uT7zMGxJh9Xc/5vCUs+3eTCkX6ArwbYFFIuZA== X-Received: by 2002:a17:902:c20d:b0:1de:f91:3cf3 with SMTP id 13-20020a170902c20d00b001de0f913cf3mr2714045pll.55.1713447899052; Thu, 18 Apr 2024 06:44:59 -0700 (PDT) Received: from LancedeMBP.lan ([112.10.225.217]) by smtp.gmail.com with ESMTPSA id d8-20020a170902b70800b001e4fdcf67desm1504837pls.299.2024.04.18.06.44.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 18 Apr 2024 06:44:58 -0700 (PDT) From: Lance Yang To: akpm@linux-foundation.org Cc: ryan.roberts@arm.com, david@redhat.com, 21cnbao@gmail.com, mhocko@suse.com, fengwei.yin@intel.com, zokeefe@google.com, shy828301@gmail.com, xiehuan09@gmail.com, wangkefeng.wang@huawei.com, songmuchun@bytedance.com, peterx@redhat.com, minchan@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Lance Yang Subject: [PATCH v10 4/4] mm/madvise: optimize lazyfreeing with mTHP in madvise_free Date: Thu, 18 Apr 2024 21:44:35 +0800 Message-Id: <20240418134435.6092-5-ioworker0@gmail.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20240418134435.6092-1-ioworker0@gmail.com> References: <20240418134435.6092-1-ioworker0@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This patch optimizes lazyfreeing with PTE-mapped mTHP[1] (Inspired by David Hildenbrand[2]). We aim to avoid unnecessary folio splitting if the large folio is fully mapped within the target range. If a large folio is locked or shared, or if we fail to split it, we just leave it in place and advance to the next PTE in the range. But note that the behavior is changed; previously, any failure of this sort would cause the entire operation to give up. As large folios become more common, sticking to the old way could result in wasted opportunities. On an Intel I5 CPU, lazyfreeing a 1GiB VMA backed by PTE-mapped folios of the same size results in the following runtimes for madvise(MADV_FREE) in seconds (shorter is better): Folio Size | Old | New | Change ------------------------------------------ 4KiB | 0.590251 | 0.590259 | 0% 16KiB | 2.990447 | 0.185655 | -94% 32KiB | 2.547831 | 0.104870 | -95% 64KiB | 2.457796 | 0.052812 | -97% 128KiB | 2.281034 | 0.032777 | -99% 256KiB | 2.230387 | 0.017496 | -99% 512KiB | 2.189106 | 0.010781 | -99% 1024KiB | 2.183949 | 0.007753 | -99% 2048KiB | 0.002799 | 0.002804 | 0% [1] https://lkml.kernel.org/r/20231207161211.2374093-5-ryan.roberts@arm.com [2] https://lore.kernel.org/linux-mm/20240214204435.167852-1-david@redhat.c= om Reviewed-by: Ryan Roberts Acked-by: David Hildenbrand Signed-off-by: Lance Yang --- mm/madvise.c | 85 +++++++++++++++++++++++++++------------------------- 1 file changed, 44 insertions(+), 41 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 4597a3568e7e..ed125ad8a21e 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -643,6 +643,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned = long addr, unsigned long end, struct mm_walk *walk) =20 { + const cydp_t cydp_flags =3D CYDP_CLEAR_YOUNG | CYDP_CLEAR_DIRTY; struct mmu_gather *tlb =3D walk->private; struct mm_struct *mm =3D tlb->mm; struct vm_area_struct *vma =3D walk->vma; @@ -697,44 +698,57 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigne= d long addr, continue; =20 /* - * If pmd isn't transhuge but the folio is large and - * is owned by only this process, split it and - * deactivate all pages. + * If we encounter a large folio, only split it if it is not + * fully mapped within the range we are operating on. Otherwise + * leave it as is so that it can be marked as lazyfree. If we + * fail to split a folio, leave it in place and advance to the + * next pte in the range. */ if (folio_test_large(folio)) { - int err; + bool any_young, any_dirty; =20 - if (folio_likely_mapped_shared(folio)) - break; - if (!folio_trylock(folio)) - break; - folio_get(folio); - arch_leave_lazy_mmu_mode(); - pte_unmap_unlock(start_pte, ptl); - start_pte =3D NULL; - err =3D split_folio(folio); - folio_unlock(folio); - folio_put(folio); - if (err) - break; - start_pte =3D pte =3D - pte_offset_map_lock(mm, pmd, addr, &ptl); - if (!start_pte) - break; - arch_enter_lazy_mmu_mode(); - pte--; - addr -=3D PAGE_SIZE; - continue; + nr =3D madvise_folio_pte_batch(addr, end, folio, pte, + ptent, &any_young, &any_dirty); + + if (nr < folio_nr_pages(folio)) { + int err; + + if (folio_likely_mapped_shared(folio)) + continue; + if (!folio_trylock(folio)) + continue; + folio_get(folio); + arch_leave_lazy_mmu_mode(); + pte_unmap_unlock(start_pte, ptl); + start_pte =3D NULL; + err =3D split_folio(folio); + folio_unlock(folio); + folio_put(folio); + pte =3D pte_offset_map_lock(mm, pmd, addr, &ptl); + start_pte =3D pte; + if (!start_pte) + break; + arch_enter_lazy_mmu_mode(); + if (!err) + nr =3D 0; + continue; + } + + if (any_young) + ptent =3D pte_mkyoung(ptent); + if (any_dirty) + ptent =3D pte_mkdirty(ptent); } =20 if (folio_test_swapcache(folio) || folio_test_dirty(folio)) { if (!folio_trylock(folio)) continue; /* - * If folio is shared with others, we mustn't clear - * the folio's dirty flag. + * If we have a large folio at this point, we know it is + * fully mapped so if its mapcount is the same as its + * number of pages, it must be exclusive. */ - if (folio_mapcount(folio) !=3D 1) { + if (folio_mapcount(folio) !=3D folio_nr_pages(folio)) { folio_unlock(folio); continue; } @@ -750,19 +764,8 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned= long addr, } =20 if (pte_young(ptent) || pte_dirty(ptent)) { - /* - * Some of architecture(ex, PPC) don't update TLB - * with set_pte_at and tlb_remove_tlb_entry so for - * the portability, remap the pte with old|clean - * after pte clearing. - */ - ptent =3D ptep_get_and_clear_full(mm, addr, pte, - tlb->fullmm); - - ptent =3D pte_mkold(ptent); - ptent =3D pte_mkclean(ptent); - set_pte_at(mm, addr, pte, ptent); - tlb_remove_tlb_entry(tlb, pte, addr); + clear_young_dirty_ptes(vma, addr, pte, nr, cydp_flags); + tlb_remove_tlb_entries(tlb, pte, nr, addr); } folio_mark_lazyfree(folio); } --=20 2.33.1