From nobody Thu May 7 21:41:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7C17C433EF for ; Wed, 18 May 2022 01:47:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233830AbiERBrC (ORCPT ); Tue, 17 May 2022 21:47:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55942 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233789AbiERBq5 (ORCPT ); Tue, 17 May 2022 21:46:57 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B0F2D49F93 for ; Tue, 17 May 2022 18:46:56 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id z39-20020a25a12a000000b0064dbcd526ffso668449ybh.15 for ; Tue, 17 May 2022 18:46:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc:content-transfer-encoding; bh=wIzudme/4B8OfAhXW9ayrH5B6l0TMc7K1sH/XdKk6Lg=; b=Ezk1gP5WC/D/1WV88OzvSzVZ4sPkJzMOxJAwd7nnYwsV2l9jLcs2v3e0kg9TQVbK+E YaSc9Ks4FZtZpalfaLJOQUXrD/YOc1tZnIOq+5D+UNEkJfQbu0pC/dA8G4O5USdxGvwH FeUsv50i+EaNusCEQjgwPIJXFN7tPxQP4vjI0NuD5dC0tjcumSXINW/Q7neSqIvb0bM/ o02sV5pM6lRXCgxoKdY3uL1EJE42SuPTi1rAPvC+xpihsPlo7zSdBvGmRoAjygmJAuZD +5Hd5VHONznzGjGEvvt+wD+KyavmAKQVFXAAGUBAKvigs4jUsz8prtj/aRMfvwIvBjK9 9kww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc:content-transfer-encoding; bh=wIzudme/4B8OfAhXW9ayrH5B6l0TMc7K1sH/XdKk6Lg=; b=mGAMoBBdCBmN4vpVnux85AKMZEd0wj+6o1yyZDokqMsgO+dADHtIwHwEhPvvVSrCJf Mid/S1bynR3dyyt6QB6or8b6EkLnGDZInKNhU7xIgWYmxVPGVa7h+wBICjlX6LaBZ0Ly Yt8XU/VmQeBr1awfBcOuaaQgvbEJPDLr6IKiGbBHX2PfkS/0fVwqkF6yubU9hjfBrnh4 /87rKXfCowAYYXBkmDYaOmdSTJEQcrck/NrIHwu6EcOKPEmNWt20PeMcUFRqprUPyfYx pW6WhVGTR4a+sPmpVWjUA6zIUyX7V5Xambh8xdDybvHSSOTMpa2ZI61WRU6h5VifmUaP rb3w== X-Gm-Message-State: AOAM533TddTXwvoy7yarzByi5uloIJZT4rHrmmwQ8SOG/W858jvKEejI QvU2zxN3f3jWBoiHNxqKuPYwuVS6jCw= X-Google-Smtp-Source: ABdhPJxeYnceCSmSU4CUNHBHtgjerhI4/gIHWEDe6JDvKsChyDSilUeyGdczcBGPDtLs45mlW8Z4BuDhLyE= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f7bc:90c9:d86e:4ea8]) (user=yuzhao job=sendgmr) by 2002:a81:7b05:0:b0:2f1:7f75:1d1e with SMTP id w5-20020a817b05000000b002f17f751d1emr27839036ywc.520.1652838415864; Tue, 17 May 2022 18:46:55 -0700 (PDT) Date: Tue, 17 May 2022 19:46:20 -0600 In-Reply-To: <20220518014632.922072-1-yuzhao@google.com> Message-Id: <20220518014632.922072-2-yuzhao@google.com> Mime-Version: 1.0 References: <20220518014632.922072-1-yuzhao@google.com> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog Subject: [PATCH v11 01/14] mm: x86, arm64: add arch_has_hw_pte_young() From: Yu Zhao To: Andrew Morton , linux-mm@kvack.org Cc: Andi Kleen , Aneesh Kumar , Catalin Marinas , Dave Hansen , Hillf Danton , Jens Axboe , Johannes Weiner , Jonathan Corbet , Linus Torvalds , Matthew Wilcox , Mel Gorman , Michael Larabel , Michal Hocko , Mike Rapoport , Peter Zijlstra , Tejun Heo , Vlastimil Babka , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, page-reclaim@google.com, Yu Zhao , Barry Song , Brian Geffon , Jan Alexander Steffens , Oleksandr Natalenko , Steven Barrett , Suleiman Souhlal , Daniel Byrne , Donald Carr , "=?UTF-8?q?Holger=20Hoffst=C3=A4tte?=" , Konstantin Kharlamov , Shuang Zhai , Sofia Trinh , Vaibhav Jain Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Some architectures automatically set the accessed bit in PTEs, e.g., x86 and arm64 v8.2. On architectures that do not have this capability, clearing the accessed bit in a PTE usually triggers a page fault following the TLB miss of this PTE (to emulate the accessed bit). Being aware of this capability can help make better decisions, e.g., whether to spread the work out over a period of time to reduce bursty page faults when trying to clear the accessed bit in many PTEs. Note that theoretically this capability can be unreliable, e.g., hotplugged CPUs might be different from builtin ones. Therefore it should not be used in architecture-independent code that involves correctness, e.g., to determine whether TLB flushes are required (in combination with the accessed bit). Signed-off-by: Yu Zhao Reviewed-by: Barry Song Acked-by: Brian Geffon Acked-by: Jan Alexander Steffens (heftig) Acked-by: Oleksandr Natalenko Acked-by: Steven Barrett Acked-by: Suleiman Souhlal Acked-by: Will Deacon Tested-by: Daniel Byrne Tested-by: Donald Carr Tested-by: Holger Hoffst=C3=A4tte Tested-by: Konstantin Kharlamov Tested-by: Shuang Zhai Tested-by: Sofia Trinh Tested-by: Vaibhav Jain --- arch/arm64/include/asm/pgtable.h | 14 ++------------ arch/x86/include/asm/pgtable.h | 6 +++--- include/linux/pgtable.h | 13 +++++++++++++ mm/memory.c | 14 +------------- 4 files changed, 19 insertions(+), 28 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgta= ble.h index dff2b483ea50..8e9c2c837678 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -999,23 +999,13 @@ static inline void update_mmu_cache(struct vm_area_st= ruct *vma, * page after fork() + CoW for pfn mappings. We don't always have a * hardware-managed access flag on arm64. */ -static inline bool arch_faults_on_old_pte(void) -{ - WARN_ON(preemptible()); - - return !cpu_has_hw_af(); -} -#define arch_faults_on_old_pte arch_faults_on_old_pte +#define arch_has_hw_pte_young cpu_has_hw_af =20 /* * Experimentally, it's cheap to set the access flag in hardware and we * benefit from prefaulting mappings as 'old' to start with. */ -static inline bool arch_wants_old_prefaulted_pte(void) -{ - return !arch_faults_on_old_pte(); -} -#define arch_wants_old_prefaulted_pte arch_wants_old_prefaulted_pte +#define arch_wants_old_prefaulted_pte cpu_has_hw_af =20 static inline bool pud_sect_supported(void) { diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 62ab07e24aef..016606a0cf20 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1424,10 +1424,10 @@ static inline bool arch_has_pfn_modify_check(void) return boot_cpu_has_bug(X86_BUG_L1TF); } =20 -#define arch_faults_on_old_pte arch_faults_on_old_pte -static inline bool arch_faults_on_old_pte(void) +#define arch_has_hw_pte_young arch_has_hw_pte_young +static inline bool arch_has_hw_pte_young(void) { - return false; + return true; } =20 #endif /* __ASSEMBLY__ */ diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index f4f4077b97aa..79f64dcff07d 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -259,6 +259,19 @@ static inline int pmdp_clear_flush_young(struct vm_are= a_struct *vma, #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif =20 +#ifndef arch_has_hw_pte_young +/* + * Return whether the accessed bit is supported on the local CPU. + * + * This stub assumes accessing through an old PTE triggers a page fault. + * Architectures that automatically set the access bit should overwrite it. + */ +static inline bool arch_has_hw_pte_young(void) +{ + return false; +} +#endif + #ifndef __HAVE_ARCH_PTEP_CLEAR static inline void ptep_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) diff --git a/mm/memory.c b/mm/memory.c index 76e3af9639d9..44a1ec7a2cac 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -122,18 +122,6 @@ int randomize_va_space __read_mostly =3D 2; #endif =20 -#ifndef arch_faults_on_old_pte -static inline bool arch_faults_on_old_pte(void) -{ - /* - * Those arches which don't have hw access flag feature need to - * implement their own helper. By default, "true" means pagefault - * will be hit on old pte. - */ - return true; -} -#endif - #ifndef arch_wants_old_prefaulted_pte static inline bool arch_wants_old_prefaulted_pte(void) { @@ -2784,7 +2772,7 @@ static inline bool cow_user_page(struct page *dst, st= ruct page *src, * On architectures with software "accessed" bits, we would * take a double page fault, so mark it accessed here. */ - if (arch_faults_on_old_pte() && !pte_young(vmf->orig_pte)) { + if (!arch_has_hw_pte_young() && !pte_young(vmf->orig_pte)) { pte_t entry; =20 vmf->pte =3D pte_offset_map_lock(mm, vmf->pmd, addr, &vmf->ptl); --=20 2.36.0.550.gb090851708-goog From nobody Thu May 7 21:41:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA947C433FE for ; Wed, 18 May 2022 01:47:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233843AbiERBrI (ORCPT ); Tue, 17 May 2022 21:47:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56102 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233790AbiERBq7 (ORCPT ); Tue, 17 May 2022 21:46:59 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6AEEA369F6 for ; Tue, 17 May 2022 18:46:58 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id s196-20020a252ccd000000b0064ea2e6bcb7so229133ybs.3 for ; Tue, 17 May 2022 18:46:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc:content-transfer-encoding; bh=1bKeqo1zCElFP+SckxGjo2derYSk4l4pfi9HINvqfq4=; b=HsLFhyzjsAQkrYPW7tweXLviM2ac4Z/u86RJZLb+b4+uKw9PjWuLuZ/poZpOBXlJ4A EnLzqKLUp8mfbTC3uzZ35oFke3PMrffiZDsnYY+QZqBtNwEhl1uy2E0bpFXnluoh7L67 vklNO5RUH9u2/rP4//lRvouGI3jWzLqmYcmMENQfHoNSFkIu+P5F6Na9vhUAUVt2vqL9 PX1oYXdcS4oygIzDn7r9TSwLhjBQH3HDYzL2D6tZ09CiUA0tGd4Ed4tS7nYHF37XYXVR 2amql6QycMPOP1s7d34tUB6/TFPSvHk6mK85S15EuwCH/j0UXE6zAFdq4orvDLbO5AnQ 5JZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc:content-transfer-encoding; bh=1bKeqo1zCElFP+SckxGjo2derYSk4l4pfi9HINvqfq4=; b=mo8RJBPjH+cp5+jPwh98oAaStWsyChWHd9nH1BaAs+9lPSk7VOB6GrXU5qZAahMpZa 0PesaFfqwWlIJnzqirWvXCu9bH39G4bZGbIdYCJy+o5RLRfVp45jFEGV4KJsZpAlisld f2aCgv/p9bLOeVUZaOOt4/nx0zA4AUrA4UXWKVJ2FsyNHTK59/PDAMK7v6BAwrbGBnMK DBhWBG39brkowTH7Sm3GrWji9u1NpGmuKHIZcnFU5w7rUG7qfQvCtqPFR6V0Oo9IwSZ5 e/OrCGWEyw1kODXBGlsr+IlYrfmg1FaJOAZuM7H6mbg5DCiDikRrmo0clUu9gmrZS+q2 y9UA== X-Gm-Message-State: AOAM5315xG7vfZaMO5x9mH7s5NERDuojBaQnenDEpAqmRBcrCnOhfEsD ViuBi6Sv9AWrfaliYqheBeO2A5GJkto= X-Google-Smtp-Source: ABdhPJwoW0FqVmxJqH6MjT8JiYpV8QeLbjHhvuKY/1xTwLhHOPRrNl6S2kOtFNWFNSQfvLVIKwiqOtORNPY= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f7bc:90c9:d86e:4ea8]) (user=yuzhao job=sendgmr) by 2002:a25:8c87:0:b0:64a:870a:e29d with SMTP id m7-20020a258c87000000b0064a870ae29dmr24844386ybl.596.1652838417591; Tue, 17 May 2022 18:46:57 -0700 (PDT) Date: Tue, 17 May 2022 19:46:21 -0600 In-Reply-To: <20220518014632.922072-1-yuzhao@google.com> Message-Id: <20220518014632.922072-3-yuzhao@google.com> Mime-Version: 1.0 References: <20220518014632.922072-1-yuzhao@google.com> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog Subject: [PATCH v11 02/14] mm: x86: add CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG From: Yu Zhao To: Andrew Morton , linux-mm@kvack.org Cc: Andi Kleen , Aneesh Kumar , Catalin Marinas , Dave Hansen , Hillf Danton , Jens Axboe , Johannes Weiner , Jonathan Corbet , Linus Torvalds , Matthew Wilcox , Mel Gorman , Michael Larabel , Michal Hocko , Mike Rapoport , Peter Zijlstra , Tejun Heo , Vlastimil Babka , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, page-reclaim@google.com, Yu Zhao , Barry Song , Brian Geffon , Jan Alexander Steffens , Oleksandr Natalenko , Steven Barrett , Suleiman Souhlal , Daniel Byrne , Donald Carr , "=?UTF-8?q?Holger=20Hoffst=C3=A4tte?=" , Konstantin Kharlamov , Shuang Zhai , Sofia Trinh , Vaibhav Jain Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Some architectures support the accessed bit in non-leaf PMD entries, e.g., x86 sets the accessed bit in a non-leaf PMD entry when using it as part of linear address translation [1]. Page table walkers that clear the accessed bit may use this capability to reduce their search space. Note that: 1. Although an inline function is preferable, this capability is added as a configuration option for consistency with the existing macros. 2. Due to the little interest in other varieties, this capability was only tested on Intel and AMD CPUs. Thanks to the following developers for their efforts [2][3]. Randy Dunlap Stephen Rothwell [1]: Intel 64 and IA-32 Architectures Software Developer's Manual Volume 3 (June 2021), section 4.8 [2] https://lore.kernel.org/r/bfdcc7c8-922f-61a9-aa15-7e7250f04af7@infradea= d.org/ [3] https://lore.kernel.org/r/20220413151513.5a0d7a7e@canb.auug.org.au/ Signed-off-by: Yu Zhao Reviewed-by: Barry Song Acked-by: Brian Geffon Acked-by: Jan Alexander Steffens (heftig) Acked-by: Oleksandr Natalenko Acked-by: Steven Barrett Acked-by: Suleiman Souhlal Tested-by: Daniel Byrne Tested-by: Donald Carr Tested-by: Holger Hoffst=C3=A4tte Tested-by: Konstantin Kharlamov Tested-by: Shuang Zhai Tested-by: Sofia Trinh Tested-by: Vaibhav Jain --- arch/Kconfig | 8 ++++++++ arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 3 ++- arch/x86/mm/pgtable.c | 5 ++++- include/linux/pgtable.h | 4 ++-- 5 files changed, 17 insertions(+), 4 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 31c4fdc4a4ba..8a2cb732b09e 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1376,6 +1376,14 @@ config DYNAMIC_SIGFRAME config HAVE_ARCH_NODE_DEV_GROUP bool =20 +config ARCH_HAS_NONLEAF_PMD_YOUNG + bool + help + Architectures that select this option are capable of setting the + accessed bit in non-leaf PMD entries when using them as part of linear + address translations. Page table walkers that clear the accessed bit + may use this capability to reduce their search space. + source "kernel/gcov/Kconfig" =20 source "scripts/gcc-plugins/Kconfig" diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 4bed3abf444d..fce1d9f41cc3 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -86,6 +86,7 @@ config X86 select ARCH_HAS_PMEM_API if X86_64 select ARCH_HAS_PTE_DEVMAP if X86_64 select ARCH_HAS_PTE_SPECIAL + select ARCH_HAS_NONLEAF_PMD_YOUNG if PGTABLE_LEVELS > 2 select ARCH_HAS_UACCESS_FLUSHCACHE if X86_64 select ARCH_HAS_COPY_MC if X86_64 select ARCH_HAS_SET_MEMORY diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 016606a0cf20..9cb3cf4cf6dd 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -820,7 +820,8 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd) =20 static inline int pmd_bad(pmd_t pmd) { - return (pmd_flags(pmd) & ~_PAGE_USER) !=3D _KERNPG_TABLE; + return (pmd_flags(pmd) & ~(_PAGE_USER | _PAGE_ACCESSED)) !=3D + (_KERNPG_TABLE & ~_PAGE_ACCESSED); } =20 static inline unsigned long pages_to_mb(unsigned long npg) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 3481b35cb4ec..a224193d84bf 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -550,7 +550,7 @@ int ptep_test_and_clear_young(struct vm_area_struct *vm= a, return ret; } =20 -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEA= F_PMD_YOUNG) int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp) { @@ -562,6 +562,9 @@ int pmdp_test_and_clear_young(struct vm_area_struct *vm= a, =20 return ret; } +#endif + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE int pudp_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pud_t *pudp) { diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 79f64dcff07d..743e7fc4afda 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -212,7 +212,7 @@ static inline int ptep_test_and_clear_young(struct vm_a= rea_struct *vma, #endif =20 #ifndef __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEA= F_PMD_YOUNG) static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) @@ -233,7 +233,7 @@ static inline int pmdp_test_and_clear_young(struct vm_a= rea_struct *vma, BUILD_BUG(); return 0; } -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG= */ #endif =20 #ifndef __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH --=20 2.36.0.550.gb090851708-goog From nobody Thu May 7 21:41:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9E18C4332F for ; Wed, 18 May 2022 01:47:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234004AbiERBrW (ORCPT ); Tue, 17 May 2022 21:47:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56326 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233820AbiERBrC (ORCPT ); Tue, 17 May 2022 21:47:02 -0400 Received: from mail-il1-x149.google.com (mail-il1-x149.google.com [IPv6:2607:f8b0:4864:20::149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 13DA6546A8 for ; Tue, 17 May 2022 18:47:00 -0700 (PDT) Received: by mail-il1-x149.google.com with SMTP id g11-20020a056e021a2b00b002cf48b48824so424976ile.21 for ; Tue, 17 May 2022 18:47:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc:content-transfer-encoding; bh=z51FPcNwNvxNlVX8smn5vw9si1mnZZRQUKLtiLkXezM=; b=bCWp0sxEaC4g9CBSBbnutqBccAuxHsQs55of/Bt/jQWSINgjftl7w0Qt/PscXsiDqm zo8beAASkyshZHQ/ySdD5c9QF2x4BrjCGYmJvRIWAhiyUIgN1LhHmdXB/deeb15U3Tof tsHB+3K36syq88/p+mLTk0E+rKdS4VUlpmSb7BXaN4Grd3kC8N39Xew3W2+xiAJduUS9 p5s7aLH7cPd6cBs31D0FdGB5VV3317OlZ+lXjPOnmjlO3V6JtdBmJGAEyfE7GlHFS8PA qgIqTztNPP+Hk3oqchsF9QItKApyWLPPMxpDQfIkFOWX9dkx74wWxgvcRLr78juMnh/V 1lEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc:content-transfer-encoding; bh=z51FPcNwNvxNlVX8smn5vw9si1mnZZRQUKLtiLkXezM=; b=F/LuGmUEdk1I31xzEjvMtsiQrE4/qk0Z9WoexW04RFIAyl0YM1qrwr7PJBXXCpMVUQ nIeOOfwgPzS/lzjUkqRiftORf6PoYO4onFZZDKjlg1esWJC4d9Fcpn5XSrsYQg14dUCZ FNSegnQsODIGHhrEfZO1FtpFOuDKTma3WV+h5hc/Mp+RH+hv6MKCbHhZgJh/L+fXEOOn NEgG7IUenyohSXvM5+6PzbgziAkyJPpy61J8K9AUqHCXk9QqpFIDGrttr6RzruAPq0Aa tRCkUiwVIZXgmp8ccwJgoKoS6LlobqbL8vNHHOZBuqGpnHMF6mMI/cxfuqK/32sxC/Wd Vxqg== X-Gm-Message-State: AOAM531B6ZpWDlebHmBy+s91swAY55M13s6jBXjjQWMh0zxoBA2VsNSR TWAJRcZA9IjGsg0LHgs5F3B+RUNnrP4= X-Google-Smtp-Source: ABdhPJyaU1KModtGurKuZbvm5GQ4iXAgX0NK43+WD6/J/2/U88t/+Dj+unOyz4fkoFhBf5nRjHUp6XY00V0= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f7bc:90c9:d86e:4ea8]) (user=yuzhao job=sendgmr) by 2002:a6b:3fc2:0:b0:65a:4236:bb42 with SMTP id m185-20020a6b3fc2000000b0065a4236bb42mr11514573ioa.215.1652838419082; Tue, 17 May 2022 18:46:59 -0700 (PDT) Date: Tue, 17 May 2022 19:46:22 -0600 In-Reply-To: <20220518014632.922072-1-yuzhao@google.com> Message-Id: <20220518014632.922072-4-yuzhao@google.com> Mime-Version: 1.0 References: <20220518014632.922072-1-yuzhao@google.com> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog Subject: [PATCH v11 03/14] mm/vmscan.c: refactor shrink_node() From: Yu Zhao To: Andrew Morton , linux-mm@kvack.org Cc: Andi Kleen , Aneesh Kumar , Catalin Marinas , Dave Hansen , Hillf Danton , Jens Axboe , Johannes Weiner , Jonathan Corbet , Linus Torvalds , Matthew Wilcox , Mel Gorman , Michael Larabel , Michal Hocko , Mike Rapoport , Peter Zijlstra , Tejun Heo , Vlastimil Babka , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, page-reclaim@google.com, Yu Zhao , Barry Song , Miaohe Lin , Brian Geffon , Jan Alexander Steffens , Oleksandr Natalenko , Steven Barrett , Suleiman Souhlal , Daniel Byrne , Donald Carr , "=?UTF-8?q?Holger=20Hoffst=C3=A4tte?=" , Konstantin Kharlamov , Shuang Zhai , Sofia Trinh , Vaibhav Jain Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This patch refactors shrink_node() to improve readability for the upcoming changes to mm/vmscan.c. Signed-off-by: Yu Zhao Reviewed-by: Barry Song Reviewed-by: Miaohe Lin Acked-by: Brian Geffon Acked-by: Jan Alexander Steffens (heftig) Acked-by: Oleksandr Natalenko Acked-by: Steven Barrett Acked-by: Suleiman Souhlal Tested-by: Daniel Byrne Tested-by: Donald Carr Tested-by: Holger Hoffst=C3=A4tte Tested-by: Konstantin Kharlamov Tested-by: Shuang Zhai Tested-by: Sofia Trinh Tested-by: Vaibhav Jain --- mm/vmscan.c | 198 +++++++++++++++++++++++++++------------------------- 1 file changed, 104 insertions(+), 94 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 1678802e03e7..2232cb55af41 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2644,6 +2644,109 @@ enum scan_balance { SCAN_FILE, }; =20 +static void prepare_scan_count(pg_data_t *pgdat, struct scan_control *sc) +{ + unsigned long file; + struct lruvec *target_lruvec; + + target_lruvec =3D mem_cgroup_lruvec(sc->target_mem_cgroup, pgdat); + + /* + * Flush the memory cgroup stats, so that we read accurate per-memcg + * lruvec stats for heuristics. + */ + mem_cgroup_flush_stats(); + + /* + * Determine the scan balance between anon and file LRUs. + */ + spin_lock_irq(&target_lruvec->lru_lock); + sc->anon_cost =3D target_lruvec->anon_cost; + sc->file_cost =3D target_lruvec->file_cost; + spin_unlock_irq(&target_lruvec->lru_lock); + + /* + * Target desirable inactive:active list ratios for the anon + * and file LRU lists. + */ + if (!sc->force_deactivate) { + unsigned long refaults; + + refaults =3D lruvec_page_state(target_lruvec, + WORKINGSET_ACTIVATE_ANON); + if (refaults !=3D target_lruvec->refaults[0] || + inactive_is_low(target_lruvec, LRU_INACTIVE_ANON)) + sc->may_deactivate |=3D DEACTIVATE_ANON; + else + sc->may_deactivate &=3D ~DEACTIVATE_ANON; + + /* + * When refaults are being observed, it means a new + * workingset is being established. Deactivate to get + * rid of any stale active pages quickly. + */ + refaults =3D lruvec_page_state(target_lruvec, + WORKINGSET_ACTIVATE_FILE); + if (refaults !=3D target_lruvec->refaults[1] || + inactive_is_low(target_lruvec, LRU_INACTIVE_FILE)) + sc->may_deactivate |=3D DEACTIVATE_FILE; + else + sc->may_deactivate &=3D ~DEACTIVATE_FILE; + } else + sc->may_deactivate =3D DEACTIVATE_ANON | DEACTIVATE_FILE; + + /* + * If we have plenty of inactive file pages that aren't + * thrashing, try to reclaim those first before touching + * anonymous pages. + */ + file =3D lruvec_page_state(target_lruvec, NR_INACTIVE_FILE); + if (file >> sc->priority && !(sc->may_deactivate & DEACTIVATE_FILE)) + sc->cache_trim_mode =3D 1; + else + sc->cache_trim_mode =3D 0; + + /* + * Prevent the reclaimer from falling into the cache trap: as + * cache pages start out inactive, every cache fault will tip + * the scan balance towards the file LRU. And as the file LRU + * shrinks, so does the window for rotation from references. + * This means we have a runaway feedback loop where a tiny + * thrashing file LRU becomes infinitely more attractive than + * anon pages. Try to detect this based on file LRU size. + */ + if (!cgroup_reclaim(sc)) { + unsigned long total_high_wmark =3D 0; + unsigned long free, anon; + int z; + + free =3D sum_zone_node_page_state(pgdat->node_id, NR_FREE_PAGES); + file =3D node_page_state(pgdat, NR_ACTIVE_FILE) + + node_page_state(pgdat, NR_INACTIVE_FILE); + + for (z =3D 0; z < MAX_NR_ZONES; z++) { + struct zone *zone =3D &pgdat->node_zones[z]; + + if (!managed_zone(zone)) + continue; + + total_high_wmark +=3D high_wmark_pages(zone); + } + + /* + * Consider anon: if that's low too, this isn't a + * runaway file reclaim problem, but rather just + * extreme pressure. Reclaim as per usual then. + */ + anon =3D node_page_state(pgdat, NR_INACTIVE_ANON); + + sc->file_is_tiny =3D + file + free <=3D total_high_wmark && + !(sc->may_deactivate & DEACTIVATE_ANON) && + anon >> sc->priority; + } +} + /* * Determine how aggressively the anon and file LRU lists should be * scanned. The relative value of each set of LRU lists is determined @@ -3114,109 +3217,16 @@ static void shrink_node(pg_data_t *pgdat, struct s= can_control *sc) unsigned long nr_reclaimed, nr_scanned; struct lruvec *target_lruvec; bool reclaimable =3D false; - unsigned long file; =20 target_lruvec =3D mem_cgroup_lruvec(sc->target_mem_cgroup, pgdat); =20 again: - /* - * Flush the memory cgroup stats, so that we read accurate per-memcg - * lruvec stats for heuristics. - */ - mem_cgroup_flush_stats(); - memset(&sc->nr, 0, sizeof(sc->nr)); =20 nr_reclaimed =3D sc->nr_reclaimed; nr_scanned =3D sc->nr_scanned; =20 - /* - * Determine the scan balance between anon and file LRUs. - */ - spin_lock_irq(&target_lruvec->lru_lock); - sc->anon_cost =3D target_lruvec->anon_cost; - sc->file_cost =3D target_lruvec->file_cost; - spin_unlock_irq(&target_lruvec->lru_lock); - - /* - * Target desirable inactive:active list ratios for the anon - * and file LRU lists. - */ - if (!sc->force_deactivate) { - unsigned long refaults; - - refaults =3D lruvec_page_state(target_lruvec, - WORKINGSET_ACTIVATE_ANON); - if (refaults !=3D target_lruvec->refaults[0] || - inactive_is_low(target_lruvec, LRU_INACTIVE_ANON)) - sc->may_deactivate |=3D DEACTIVATE_ANON; - else - sc->may_deactivate &=3D ~DEACTIVATE_ANON; - - /* - * When refaults are being observed, it means a new - * workingset is being established. Deactivate to get - * rid of any stale active pages quickly. - */ - refaults =3D lruvec_page_state(target_lruvec, - WORKINGSET_ACTIVATE_FILE); - if (refaults !=3D target_lruvec->refaults[1] || - inactive_is_low(target_lruvec, LRU_INACTIVE_FILE)) - sc->may_deactivate |=3D DEACTIVATE_FILE; - else - sc->may_deactivate &=3D ~DEACTIVATE_FILE; - } else - sc->may_deactivate =3D DEACTIVATE_ANON | DEACTIVATE_FILE; - - /* - * If we have plenty of inactive file pages that aren't - * thrashing, try to reclaim those first before touching - * anonymous pages. - */ - file =3D lruvec_page_state(target_lruvec, NR_INACTIVE_FILE); - if (file >> sc->priority && !(sc->may_deactivate & DEACTIVATE_FILE)) - sc->cache_trim_mode =3D 1; - else - sc->cache_trim_mode =3D 0; - - /* - * Prevent the reclaimer from falling into the cache trap: as - * cache pages start out inactive, every cache fault will tip - * the scan balance towards the file LRU. And as the file LRU - * shrinks, so does the window for rotation from references. - * This means we have a runaway feedback loop where a tiny - * thrashing file LRU becomes infinitely more attractive than - * anon pages. Try to detect this based on file LRU size. - */ - if (!cgroup_reclaim(sc)) { - unsigned long total_high_wmark =3D 0; - unsigned long free, anon; - int z; - - free =3D sum_zone_node_page_state(pgdat->node_id, NR_FREE_PAGES); - file =3D node_page_state(pgdat, NR_ACTIVE_FILE) + - node_page_state(pgdat, NR_INACTIVE_FILE); - - for (z =3D 0; z < MAX_NR_ZONES; z++) { - struct zone *zone =3D &pgdat->node_zones[z]; - if (!managed_zone(zone)) - continue; - - total_high_wmark +=3D high_wmark_pages(zone); - } - - /* - * Consider anon: if that's low too, this isn't a - * runaway file reclaim problem, but rather just - * extreme pressure. Reclaim as per usual then. - */ - anon =3D node_page_state(pgdat, NR_INACTIVE_ANON); - - sc->file_is_tiny =3D - file + free <=3D total_high_wmark && - !(sc->may_deactivate & DEACTIVATE_ANON) && - anon >> sc->priority; - } + prepare_scan_count(pgdat, sc); =20 shrink_node_memcgs(pgdat, sc); =20 --=20 2.36.0.550.gb090851708-goog From nobody Thu May 7 21:41:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB8B1C433FE for ; Wed, 18 May 2022 01:47:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233947AbiERBrh (ORCPT ); Tue, 17 May 2022 21:47:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56332 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233823AbiERBrC (ORCPT ); Tue, 17 May 2022 21:47:02 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 305EA37022 for ; Tue, 17 May 2022 18:47:01 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id c205-20020a254ed6000000b0064d5bd93479so658930ybb.17 for ; Tue, 17 May 2022 18:47:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc:content-transfer-encoding; bh=2k47IzMwcCabKXgicn4noPykM2uGLJnBykS7HlFiI8w=; b=aei3DToBh8ETXlvQnlUeTmPRDPPpHJQq+Tb7kQz3IxXIpTpzsq5AiNIgt96OPzJefp 6ywXqgZpvrjr4352sYucxPL2b6v7FYT8zUV8RzpmGOxTyHPsQwa25kKP4sr+HWx1MXRZ 7zk1VZ+6KuBwT7HiXKqQzsY6TtLni/IPnj2DJC2crApo36Z2G+0ogkkMwSZH4qzMx3Kl ULntFSZZD/SDjcPPnjQfjrgYddGlTlXk277b5lMj6I+m4gKrNvzRcCvZGWtt+cnbaFrF hvqc5aS3Q9mPr6/Abjh2NxZ7IPoxNzL8Em7qV/ep3hOa1cE+mThDls+dmX0zbmVBL0Iw +fXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc:content-transfer-encoding; bh=2k47IzMwcCabKXgicn4noPykM2uGLJnBykS7HlFiI8w=; b=BsRW18LJ/+7H4uOq3EbyFQgrEpBx/BcK2Ej+SOk4gtR7ttZHV+Ix6jDzGEbOcz9Fvn IMdlctnn5qftM3RowXmg/GlXfyhrh7npt4v2cjlarh0Ry7hNRxrhfkeRkXjCBQs40vH8 EnU6mT+KrCn+NXcni3OJky3vPx8detgU4aCP5qsMl7WRQoOcYN5RrRoL8Zrw1L48vYVp amOdomBHVTWdbxjnxdAorHla1PpQ/8DW7iLkXLn2DJA2WOtv0cYhymx48SWzNJ8uy5Hp 4j+68vb01Q4MHbEtlwBtT0bx+pBsDDP2OwSntFoGeBrWbLBZYO/e8abqHoldP0cyqAPp JnQw== X-Gm-Message-State: AOAM530yJhlp5kFcjMkpS1wzJky9hrFni5p7gBmbRiQgvUxh6q8mH7l5 id5ERA9hcIwK96trEqo6x9B+JnuAVCc= X-Google-Smtp-Source: ABdhPJzqOqzng70YQZkxXWVTmQJQwNaUloJzQSF15E4otE3nKg1B2U9N2NHa1PZDXrxFGBjAXED580jBaMU= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f7bc:90c9:d86e:4ea8]) (user=yuzhao job=sendgmr) by 2002:a25:416:0:b0:64d:8829:7fef with SMTP id 22-20020a250416000000b0064d88297fefmr15179083ybe.48.1652838420215; Tue, 17 May 2022 18:47:00 -0700 (PDT) Date: Tue, 17 May 2022 19:46:23 -0600 In-Reply-To: <20220518014632.922072-1-yuzhao@google.com> Message-Id: <20220518014632.922072-5-yuzhao@google.com> Mime-Version: 1.0 References: <20220518014632.922072-1-yuzhao@google.com> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog Subject: [PATCH v11 04/14] Revert "include/linux/mm_inline.h: fold __update_lru_size() into its sole caller" From: Yu Zhao To: Andrew Morton , linux-mm@kvack.org Cc: Andi Kleen , Aneesh Kumar , Catalin Marinas , Dave Hansen , Hillf Danton , Jens Axboe , Johannes Weiner , Jonathan Corbet , Linus Torvalds , Matthew Wilcox , Mel Gorman , Michael Larabel , Michal Hocko , Mike Rapoport , Peter Zijlstra , Tejun Heo , Vlastimil Babka , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, page-reclaim@google.com, Yu Zhao , Miaohe Lin , Brian Geffon , Jan Alexander Steffens , Oleksandr Natalenko , Steven Barrett , Suleiman Souhlal , Daniel Byrne , Donald Carr , "=?UTF-8?q?Holger=20Hoffst=C3=A4tte?=" , Konstantin Kharlamov , Shuang Zhai , Sofia Trinh , Vaibhav Jain Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" This patch undoes the following refactor: commit 289ccba18af4 ("include/linux/mm_inline.h: fold __update_lru_size() i= nto its sole caller") The upcoming changes to include/linux/mm_inline.h will reuse __update_lru_size(). Signed-off-by: Yu Zhao Reviewed-by: Miaohe Lin Acked-by: Brian Geffon Acked-by: Jan Alexander Steffens (heftig) Acked-by: Oleksandr Natalenko Acked-by: Steven Barrett Acked-by: Suleiman Souhlal Tested-by: Daniel Byrne Tested-by: Donald Carr Tested-by: Holger Hoffst=C3=A4tte Tested-by: Konstantin Kharlamov Tested-by: Shuang Zhai Tested-by: Sofia Trinh Tested-by: Vaibhav Jain --- include/linux/mm_inline.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index ac32125745ab..7c9c2157e9a8 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -32,7 +32,7 @@ static inline int page_is_file_lru(struct page *page) return folio_is_file_lru(page_folio(page)); } =20 -static __always_inline void update_lru_size(struct lruvec *lruvec, +static __always_inline void __update_lru_size(struct lruvec *lruvec, enum lru_list lru, enum zone_type zid, long nr_pages) { @@ -41,6 +41,13 @@ static __always_inline void update_lru_size(struct lruve= c *lruvec, __mod_lruvec_state(lruvec, NR_LRU_BASE + lru, nr_pages); __mod_zone_page_state(&pgdat->node_zones[zid], NR_ZONE_LRU_BASE + lru, nr_pages); +} + +static __always_inline void update_lru_size(struct lruvec *lruvec, + enum lru_list lru, enum zone_type zid, + int nr_pages) +{ + __update_lru_size(lruvec, lru, zid, nr_pages); #ifdef CONFIG_MEMCG mem_cgroup_update_lru_size(lruvec, lru, zid, nr_pages); #endif --=20 2.36.0.550.gb090851708-goog From nobody Thu May 7 21:41:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B47B5C4332F for ; Wed, 18 May 2022 01:47:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233829AbiERBr0 (ORCPT ); Tue, 17 May 2022 21:47:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57920 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231903AbiERBrS (ORCPT ); Tue, 17 May 2022 21:47:18 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1D4DC546BA for ; Tue, 17 May 2022 18:47:02 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id z39-20020a25a12a000000b0064dbcd526ffso668449ybh.15 for ; Tue, 17 May 2022 18:47:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc:content-transfer-encoding; bh=BdUc2XW5WiYZ2PyvxlGy/bO/1gRq+jLxJ6Cnbh8ZRtg=; b=qhdyQyyqKmnnf8W0m9OZi2oQ/MrcdAZ/SMOU/6RU5cNGVey0Jy/4RETmNFMqktRUkr 7gRVPaF8+cwHyZyBfbTaInVUSLGeyWxIRCaJbNT0TmcUwOWH5i6EwHDIX78bKm6jll0z sMjEwNCnVNnEOvKV1yNCNHeCz2kq+WmPXGxxfCId9UoaHhnSS0vgBj+jk0ze79ZfDXln 095FKO8g96C1bITtbSSq5b43ArRm8aENU2jqYM35EM0OOuT3QNM9jp8eAIXLWcAZIswx VJFSFevIl/MxIjSSbUloc6omWd3cZlgET2OWyas4ME80t9FXYJMnWn50msgLaE+L/PxR oHmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc:content-transfer-encoding; bh=BdUc2XW5WiYZ2PyvxlGy/bO/1gRq+jLxJ6Cnbh8ZRtg=; b=phqC22lx50WKN+oZgww37wAK4cfE5USyLwuJU1WTNkbQ7v6Cye4whNHqEQ1Ao6+rnd aHCHoWyIyEo7ZI/RgxF5TWKOApu70MwQA/l3awYTyy7qGYW2I7MJpucKqFs0L7YG+E3Q uErGyh3+Zbxd23llOI5alakh8PoWJq1FoZaVfvLzOVb6HLNY736YkWQznifgpVLI5WyV 17shAOxOsJQvFI7v3PhLYjLlZ+85Un3LxuPOCTxALhGzjh0WbYKkSY9hk9L92LxUg8zd 38tqyMm+LVOMRqKNb0t3Owi3XqatvN1B12+MtfN1XYTXvcYFs9p28nzqKR45Ekiw3CvN mR/w== X-Gm-Message-State: AOAM53383v5MoUhAkLJodt42EOywo0/TfwKqnflQjZ3DMvep86sETQks CtJbtKbI3nEM4L/L7RLoZTOrQQzODt0= X-Google-Smtp-Source: ABdhPJzi5vnU5GSPs3vkYq8/ubKvEppWjlekA5su3Hj5yutoUD7xax3AXYkTquX943inyyuVqfA7ffiBesU= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f7bc:90c9:d86e:4ea8]) (user=yuzhao job=sendgmr) by 2002:a81:25d3:0:b0:2ff:111c:a5ce with SMTP id l202-20020a8125d3000000b002ff111ca5cemr10934053ywl.372.1652838421629; Tue, 17 May 2022 18:47:01 -0700 (PDT) Date: Tue, 17 May 2022 19:46:24 -0600 In-Reply-To: <20220518014632.922072-1-yuzhao@google.com> Message-Id: <20220518014632.922072-6-yuzhao@google.com> Mime-Version: 1.0 References: <20220518014632.922072-1-yuzhao@google.com> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog Subject: [PATCH v11 05/14] mm: multi-gen LRU: groundwork From: Yu Zhao To: Andrew Morton , linux-mm@kvack.org Cc: Andi Kleen , Aneesh Kumar , Catalin Marinas , Dave Hansen , Hillf Danton , Jens Axboe , Johannes Weiner , Jonathan Corbet , Linus Torvalds , Matthew Wilcox , Mel Gorman , Michael Larabel , Michal Hocko , Mike Rapoport , Peter Zijlstra , Tejun Heo , Vlastimil Babka , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, page-reclaim@google.com, Yu Zhao , Brian Geffon , Jan Alexander Steffens , Oleksandr Natalenko , Steven Barrett , Suleiman Souhlal , Daniel Byrne , Donald Carr , "=?UTF-8?q?Holger=20Hoffst=C3=A4tte?=" , Konstantin Kharlamov , Shuang Zhai , Sofia Trinh , Vaibhav Jain Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Evictable pages are divided into multiple generations for each lruvec. The youngest generation number is stored in lrugen->max_seq for both anon and file types as they are aged on an equal footing. The oldest generation numbers are stored in lrugen->min_seq[] separately for anon and file types as clean file pages can be evicted regardless of swap constraints. These three variables are monotonically increasing. Generation numbers are truncated into order_base_2(MAX_NR_GENS+1) bits in order to fit into the gen counter in folio->flags. Each truncated generation number is an index to lrugen->lists[]. The sliding window technique is used to track at least MIN_NR_GENS and at most MAX_NR_GENS generations. The gen counter stores a value within [1, MAX_NR_GENS] while a page is on one of lrugen->lists[]. Otherwise it stores 0. There are two conceptually independent procedures: "the aging", which produces young generations, and "the eviction", which consumes old generations. They form a closed-loop system, i.e., "the page reclaim". Both procedures can be invoked from userspace for the purposes of working set estimation and proactive reclaim. These techniques are commonly used to optimize job scheduling (bin packing) in data centers [1][2]. To avoid confusion, the terms "hot" and "cold" will be applied to the multi-gen LRU, as a new convention; the terms "active" and "inactive" will be applied to the active/inactive LRU, as usual. The protection of hot pages and the selection of cold pages are based on page access channels and patterns. There are two access channels: one through page tables and the other through file descriptors. The protection of the former channel is by design stronger because: 1. The uncertainty in determining the access patterns of the former channel is higher due to the approximation of the accessed bit. 2. The cost of evicting the former channel is higher due to the TLB flushes required and the likelihood of encountering the dirty bit. 3. The penalty of underprotecting the former channel is higher because applications usually do not prepare themselves for major page faults like they do for blocked I/O. E.g., GUI applications commonly use dedicated I/O threads to avoid blocking the rendering threads. There are also two access patterns: one with temporal locality and the other without. For the reasons listed above, the former channel is assumed to follow the former pattern unless VM_SEQ_READ or VM_RAND_READ is present; the latter channel is assumed to follow the latter pattern unless outlying refaults have been observed [3][4]. The next patch will address the "outlying refaults". Three macros, i.e., LRU_REFS_WIDTH, LRU_REFS_PGOFF and LRU_REFS_MASK, used later are added in this patch to make the entire patchset less diffy. A page is added to the youngest generation on faulting. The aging needs to check the accessed bit at least twice before handing this page over to the eviction. The first check takes care of the accessed bit set on the initial fault; the second check makes sure this page has not been used since then. This protocol, AKA second chance, requires a minimum of two generations, hence MIN_NR_GENS. [1] https://dl.acm.org/doi/10.1145/3297858.3304053 [2] https://dl.acm.org/doi/10.1145/3503222.3507731 [3] https://lwn.net/Articles/495543/ [4] https://lwn.net/Articles/815342/ Signed-off-by: Yu Zhao Acked-by: Brian Geffon Acked-by: Jan Alexander Steffens (heftig) Acked-by: Oleksandr Natalenko Acked-by: Steven Barrett Acked-by: Suleiman Souhlal Tested-by: Daniel Byrne Tested-by: Donald Carr Tested-by: Holger Hoffst=C3=A4tte Tested-by: Konstantin Kharlamov Tested-by: Shuang Zhai Tested-by: Sofia Trinh Tested-by: Vaibhav Jain --- fs/fuse/dev.c | 3 +- include/linux/mm.h | 2 + include/linux/mm_inline.h | 180 ++++++++++++++++++++++++++++++ include/linux/mmzone.h | 98 ++++++++++++++++ include/linux/page-flags-layout.h | 13 ++- include/linux/page-flags.h | 4 +- include/linux/sched.h | 4 + kernel/bounds.c | 5 + mm/Kconfig | 8 ++ mm/huge_memory.c | 3 +- mm/memcontrol.c | 2 + mm/memory.c | 25 +++++ mm/mm_init.c | 6 +- mm/mmzone.c | 2 + mm/swap.c | 9 +- mm/vmscan.c | 75 +++++++++++++ 16 files changed, 426 insertions(+), 13 deletions(-) diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c index 0e537e580dc1..5d36015071d2 100644 --- a/fs/fuse/dev.c +++ b/fs/fuse/dev.c @@ -777,7 +777,8 @@ static int fuse_check_page(struct page *page) 1 << PG_active | 1 << PG_workingset | 1 << PG_reclaim | - 1 << PG_waiters))) { + 1 << PG_waiters | + LRU_GEN_MASK | LRU_REFS_MASK))) { dump_page(page, "fuse: trying to steal weird page"); return 1; } diff --git a/include/linux/mm.h b/include/linux/mm.h index 9f44254af8ce..894c289c2c06 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1060,6 +1060,8 @@ vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf); #define ZONES_PGOFF (NODES_PGOFF - ZONES_WIDTH) #define LAST_CPUPID_PGOFF (ZONES_PGOFF - LAST_CPUPID_WIDTH) #define KASAN_TAG_PGOFF (LAST_CPUPID_PGOFF - KASAN_TAG_WIDTH) +#define LRU_GEN_PGOFF (KASAN_TAG_PGOFF - LRU_GEN_WIDTH) +#define LRU_REFS_PGOFF (LRU_GEN_PGOFF - LRU_REFS_WIDTH) =20 /* * Define the bit shifts to access each section. For non-existent diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 7c9c2157e9a8..98ae22bfaf12 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -38,6 +38,9 @@ static __always_inline void __update_lru_size(struct lruv= ec *lruvec, { struct pglist_data *pgdat =3D lruvec_pgdat(lruvec); =20 + lockdep_assert_held(&lruvec->lru_lock); + WARN_ON_ONCE(nr_pages !=3D (int)nr_pages); + __mod_lruvec_state(lruvec, NR_LRU_BASE + lru, nr_pages); __mod_zone_page_state(&pgdat->node_zones[zid], NR_ZONE_LRU_BASE + lru, nr_pages); @@ -99,11 +102,182 @@ static __always_inline enum lru_list folio_lru_list(s= truct folio *folio) return lru; } =20 +#ifdef CONFIG_LRU_GEN + +static inline bool lru_gen_enabled(void) +{ + return true; +} + +static inline bool lru_gen_in_fault(void) +{ + return current->in_lru_fault; +} + +static inline int lru_gen_from_seq(unsigned long seq) +{ + return seq % MAX_NR_GENS; +} + +static inline int folio_lru_gen(struct folio *folio) +{ + unsigned long flags =3D READ_ONCE(folio->flags); + + return ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; +} + +static inline bool lru_gen_is_active(struct lruvec *lruvec, int gen) +{ + unsigned long max_seq =3D lruvec->lrugen.max_seq; + + VM_WARN_ON_ONCE(gen >=3D MAX_NR_GENS); + + /* see the comment on MIN_NR_GENS */ + return gen =3D=3D lru_gen_from_seq(max_seq) || gen =3D=3D lru_gen_from_se= q(max_seq - 1); +} + +static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio= *folio, + int old_gen, int new_gen) +{ + int type =3D folio_is_file_lru(folio); + int zone =3D folio_zonenum(folio); + int delta =3D folio_nr_pages(folio); + enum lru_list lru =3D type * LRU_INACTIVE_FILE; + struct lru_gen_struct *lrugen =3D &lruvec->lrugen; + + VM_WARN_ON_ONCE(old_gen !=3D -1 && old_gen >=3D MAX_NR_GENS); + VM_WARN_ON_ONCE(new_gen !=3D -1 && new_gen >=3D MAX_NR_GENS); + VM_WARN_ON_ONCE(old_gen =3D=3D -1 && new_gen =3D=3D -1); + + if (old_gen >=3D 0) + WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone], + lrugen->nr_pages[old_gen][type][zone] - delta); + if (new_gen >=3D 0) + WRITE_ONCE(lrugen->nr_pages[new_gen][type][zone], + lrugen->nr_pages[new_gen][type][zone] + delta); + + /* addition */ + if (old_gen < 0) { + if (lru_gen_is_active(lruvec, new_gen)) + lru +=3D LRU_ACTIVE; + __update_lru_size(lruvec, lru, zone, delta); + return; + } + + /* deletion */ + if (new_gen < 0) { + if (lru_gen_is_active(lruvec, old_gen)) + lru +=3D LRU_ACTIVE; + __update_lru_size(lruvec, lru, zone, -delta); + return; + } +} + +static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *= folio, bool reclaiming) +{ + unsigned long mask, flags; + int gen =3D folio_lru_gen(folio); + int type =3D folio_is_file_lru(folio); + int zone =3D folio_zonenum(folio); + struct lru_gen_struct *lrugen =3D &lruvec->lrugen; + + VM_WARN_ON_ONCE_FOLIO(gen !=3D -1, folio); + + if (folio_test_unevictable(folio)) + return false; + /* + * There are three common cases for this page: + * 1. If it's hot, e.g., freshly faulted in or previously hot and + * migrated, add it to the youngest generation. + * 2. If it's cold but can't be evicted immediately, i.e., an anon page + * not in swapcache or a dirty page pending writeback, add it to the + * second oldest generation. + * 3. Everything else (clean, cold) is added to the oldest generation. + */ + if (folio_test_active(folio)) + gen =3D lru_gen_from_seq(lrugen->max_seq); + else if ((type =3D=3D LRU_GEN_ANON && !folio_test_swapcache(folio)) || + (folio_test_reclaim(folio) && + (folio_test_dirty(folio) || folio_test_writeback(folio)))) + gen =3D lru_gen_from_seq(lrugen->min_seq[type] + 1); + else + gen =3D lru_gen_from_seq(lrugen->min_seq[type]); + + /* see the comment on MIN_NR_GENS */ + mask =3D LRU_GEN_MASK | BIT(PG_active); + flags =3D (gen + 1UL) << LRU_GEN_PGOFF; + set_mask_bits(&folio->flags, mask, flags); + + lru_gen_update_size(lruvec, folio, -1, gen); + /* for folio_rotate_reclaimable() */ + if (reclaiming) + list_add_tail(&folio->lru, &lrugen->lists[gen][type][zone]); + else + list_add(&folio->lru, &lrugen->lists[gen][type][zone]); + + return true; +} + +static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *= folio, bool reclaiming) +{ + unsigned long mask, flags; + int gen =3D folio_lru_gen(folio); + + if (gen < 0) + return false; + + VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio); + VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); + + mask =3D LRU_GEN_MASK; + flags =3D 0; + /* for shrink_page_list() or folio_migrate_flags() */ + if (reclaiming) + mask |=3D BIT(PG_referenced) | BIT(PG_reclaim); + else if (lru_gen_is_active(lruvec, gen)) + flags |=3D BIT(PG_active); + + flags =3D set_mask_bits(&folio->flags, mask, flags); + gen =3D ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; + + lru_gen_update_size(lruvec, folio, gen, -1); + list_del(&folio->lru); + + return true; +} + +#else + +static inline bool lru_gen_enabled(void) +{ + return false; +} + +static inline bool lru_gen_in_fault(void) +{ + return false; +} + +static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *= folio, bool reclaiming) +{ + return false; +} + +static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *= folio, bool reclaiming) +{ + return false; +} + +#endif /* CONFIG_LRU_GEN */ + static __always_inline void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio) { enum lru_list lru =3D folio_lru_list(folio); =20 + if (lru_gen_add_folio(lruvec, folio, false)) + return; + update_lru_size(lruvec, lru, folio_zonenum(folio), folio_nr_pages(folio)); if (lru !=3D LRU_UNEVICTABLE) @@ -121,6 +295,9 @@ void lruvec_add_folio_tail(struct lruvec *lruvec, struc= t folio *folio) { enum lru_list lru =3D folio_lru_list(folio); =20 + if (lru_gen_add_folio(lruvec, folio, true)) + return; + update_lru_size(lruvec, lru, folio_zonenum(folio), folio_nr_pages(folio)); /* This is not expected to be used on LRU_UNEVICTABLE */ @@ -138,6 +315,9 @@ void lruvec_del_folio(struct lruvec *lruvec, struct fol= io *folio) { enum lru_list lru =3D folio_lru_list(folio); =20 + if (lru_gen_del_folio(lruvec, folio, false)) + return; + if (lru !=3D LRU_UNEVICTABLE) list_del(&folio->lru); update_lru_size(lruvec, lru, folio_zonenum(folio), diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 46ffab808f03..6994acef63cb 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -317,6 +317,100 @@ enum lruvec_flags { */ }; =20 +#endif /* !__GENERATING_BOUNDS_H */ + +/* + * Evictable pages are divided into multiple generations. The youngest and= the + * oldest generation numbers, max_seq and min_seq, are monotonically incre= asing. + * They form a sliding window of a variable size [MIN_NR_GENS, MAX_NR_GENS= ]. An + * offset within MAX_NR_GENS, i.e., gen, indexes the LRU list of the + * corresponding generation. The gen counter in folio->flags stores gen+1 = while + * a page is on one of lrugen->lists[]. Otherwise it stores 0. + * + * A page is added to the youngest generation on faulting. The aging needs= to + * check the accessed bit at least twice before handing this page over to = the + * eviction. The first check takes care of the accessed bit set on the ini= tial + * fault; the second check makes sure this page hasn't been used since the= n. + * This process, AKA second chance, requires a minimum of two generations, + * hence MIN_NR_GENS. And to maintain ABI compatibility with the active/in= active + * LRU, e.g., /proc/vmstat, these two generations are considered active; t= he + * rest of generations, if they exist, are considered inactive. See + * lru_gen_is_active(). PG_active is always cleared while a page is on one= of + * lrugen->lists[] so that the aging needs not to worry about it. And it's= set + * again when a page considered active is isolated for non-reclaiming purp= oses, + * e.g., migration. See lru_gen_add_folio() and lru_gen_del_folio(). + * + * MAX_NR_GENS is set to 4 so that the multi-gen LRU can support twice the + * number of categories of the active/inactive LRU when keeping track of + * accesses through page tables. It requires order_base_2(MAX_NR_GENS+1) b= its in + * folio->flags (LRU_GEN_MASK). + */ +#define MIN_NR_GENS 2U +#define MAX_NR_GENS 4U + +#ifndef __GENERATING_BOUNDS_H + +struct lruvec; + +#define LRU_GEN_MASK ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF) +#define LRU_REFS_MASK ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF) + +#ifdef CONFIG_LRU_GEN + +enum { + LRU_GEN_ANON, + LRU_GEN_FILE, +}; + +/* + * The youngest generation number is stored in max_seq for both anon and f= ile + * types as they are aged on an equal footing. The oldest generation numbe= rs are + * stored in min_seq[] separately for anon and file types as clean file pa= ges + * can be evicted regardless of swap constraints. + * + * Normally anon and file min_seq are in sync. But if swapping is constrai= ned, + * e.g., out of swap space, file min_seq is allowed to advance and leave a= non + * min_seq behind. + * + * nr_pages[] are eventually consistent and therefore can be transiently + * negative. + */ +struct lru_gen_struct { + /* the aging increments the youngest generation number */ + unsigned long max_seq; + /* the eviction increments the oldest generation numbers */ + unsigned long min_seq[ANON_AND_FILE]; + /* the multi-gen LRU lists */ + struct list_head lists[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; + /* the sizes of the above lists */ + long nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; +}; + +void lru_gen_init_lruvec(struct lruvec *lruvec); + +#ifdef CONFIG_MEMCG +void lru_gen_init_memcg(struct mem_cgroup *memcg); +void lru_gen_exit_memcg(struct mem_cgroup *memcg); +#endif + +#else /* !CONFIG_LRU_GEN */ + +static inline void lru_gen_init_lruvec(struct lruvec *lruvec) +{ +} + +#ifdef CONFIG_MEMCG +static inline void lru_gen_init_memcg(struct mem_cgroup *memcg) +{ +} + +static inline void lru_gen_exit_memcg(struct mem_cgroup *memcg) +{ +} +#endif + +#endif /* CONFIG_LRU_GEN */ + struct lruvec { struct list_head lists[NR_LRU_LISTS]; /* per lruvec lru_lock for memcg */ @@ -334,6 +428,10 @@ struct lruvec { unsigned long refaults[ANON_AND_FILE]; /* Various lruvec state flags (enum lruvec_flags) */ unsigned long flags; +#ifdef CONFIG_LRU_GEN + /* evictable pages divided into generations */ + struct lru_gen_struct lrugen; +#endif #ifdef CONFIG_MEMCG struct pglist_data *pgdat; #endif diff --git a/include/linux/page-flags-layout.h b/include/linux/page-flags-l= ayout.h index ef1e3e736e14..240905407a18 100644 --- a/include/linux/page-flags-layout.h +++ b/include/linux/page-flags-layout.h @@ -55,7 +55,8 @@ #define SECTIONS_WIDTH 0 #endif =20 -#if ZONES_WIDTH + SECTIONS_WIDTH + NODES_SHIFT <=3D BITS_PER_LONG - NR_PAG= EFLAGS +#if ZONES_WIDTH + LRU_GEN_WIDTH + SECTIONS_WIDTH + NODES_SHIFT \ + <=3D BITS_PER_LONG - NR_PAGEFLAGS #define NODES_WIDTH NODES_SHIFT #elif defined(CONFIG_SPARSEMEM_VMEMMAP) #error "Vmemmap: No space for nodes field in page flags" @@ -89,8 +90,8 @@ #define LAST_CPUPID_SHIFT 0 #endif =20 -#if ZONES_WIDTH + SECTIONS_WIDTH + NODES_WIDTH + KASAN_TAG_WIDTH + LAST_CP= UPID_SHIFT \ - <=3D BITS_PER_LONG - NR_PAGEFLAGS +#if ZONES_WIDTH + LRU_GEN_WIDTH + SECTIONS_WIDTH + NODES_WIDTH + \ + KASAN_TAG_WIDTH + LAST_CPUPID_SHIFT <=3D BITS_PER_LONG - NR_PAGEFLAGS #define LAST_CPUPID_WIDTH LAST_CPUPID_SHIFT #else #define LAST_CPUPID_WIDTH 0 @@ -100,10 +101,12 @@ #define LAST_CPUPID_NOT_IN_PAGE_FLAGS #endif =20 -#if ZONES_WIDTH + SECTIONS_WIDTH + NODES_WIDTH + KASAN_TAG_WIDTH + LAST_CP= UPID_WIDTH \ - > BITS_PER_LONG - NR_PAGEFLAGS +#if ZONES_WIDTH + LRU_GEN_WIDTH + SECTIONS_WIDTH + NODES_WIDTH + \ + KASAN_TAG_WIDTH + LAST_CPUPID_WIDTH > BITS_PER_LONG - NR_PAGEFLAGS #error "Not enough bits in page flags" #endif =20 +#define LRU_REFS_WIDTH 0 + #endif #endif /* _LINUX_PAGE_FLAGS_LAYOUT */ diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 9d8eeaa67d05..5cbde013ce66 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -1017,7 +1017,7 @@ PAGEFLAG(Isolated, isolated, PF_ANY); 1UL << PG_private | 1UL << PG_private_2 | \ 1UL << PG_writeback | 1UL << PG_reserved | \ 1UL << PG_slab | 1UL << PG_active | \ - 1UL << PG_unevictable | __PG_MLOCKED) + 1UL << PG_unevictable | __PG_MLOCKED | LRU_GEN_MASK) =20 /* * Flags checked when a page is prepped for return by the page allocator. @@ -1028,7 +1028,7 @@ PAGEFLAG(Isolated, isolated, PF_ANY); * alloc-free cycle to prevent from reusing the page. */ #define PAGE_FLAGS_CHECK_AT_PREP \ - (PAGEFLAGS_MASK & ~__PG_HWPOISON) + ((PAGEFLAGS_MASK & ~__PG_HWPOISON) | LRU_GEN_MASK | LRU_REFS_MASK) =20 #define PAGE_FLAGS_PRIVATE \ (1UL << PG_private | 1UL << PG_private_2) diff --git a/include/linux/sched.h b/include/linux/sched.h index a8911b1f35aa..448e75a5acc5 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -914,6 +914,10 @@ struct task_struct { #ifdef CONFIG_MEMCG unsigned in_user_fault:1; #endif +#ifdef CONFIG_LRU_GEN + /* whether the LRU algorithm may apply to this access */ + unsigned in_lru_fault:1; +#endif #ifdef CONFIG_COMPAT_BRK unsigned brk_randomized:1; #endif diff --git a/kernel/bounds.c b/kernel/bounds.c index 9795d75b09b2..5ee60777d8e4 100644 --- a/kernel/bounds.c +++ b/kernel/bounds.c @@ -22,6 +22,11 @@ int main(void) DEFINE(NR_CPUS_BITS, ilog2(CONFIG_NR_CPUS)); #endif DEFINE(SPINLOCK_SIZE, sizeof(spinlock_t)); +#ifdef CONFIG_LRU_GEN + DEFINE(LRU_GEN_WIDTH, order_base_2(MAX_NR_GENS + 1)); +#else + DEFINE(LRU_GEN_WIDTH, 0); +#endif /* End of constants */ =20 return 0; diff --git a/mm/Kconfig b/mm/Kconfig index 034d87953600..e62bd501082b 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -909,6 +909,14 @@ config ANON_VMA_NAME area from being merged with adjacent virtual memory areas due to the difference in their name. =20 +config LRU_GEN + bool "Multi-Gen LRU" + depends on MMU + # make sure folio->flags has enough spare bits + depends on 64BIT || !SPARSEMEM || SPARSEMEM_VMEMMAP + help + A high performance LRU implementation to overcommit memory. + source "mm/damon/Kconfig" =20 endmenu diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 910a138e9859..a090514f2bf3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2320,7 +2320,8 @@ static void __split_huge_page_tail(struct page *head,= int tail, #ifdef CONFIG_64BIT (1L << PG_arch_2) | #endif - (1L << PG_dirty))); + (1L << PG_dirty) | + LRU_GEN_MASK | LRU_REFS_MASK)); =20 /* ->mapping in first tail page is compound_mapcount */ VM_BUG_ON_PAGE(tail > 2 && page_tail->mapping !=3D TAIL_MAPPING, diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 598fece89e2b..2ee074f80e72 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -5072,6 +5072,7 @@ static void __mem_cgroup_free(struct mem_cgroup *memc= g) =20 static void mem_cgroup_free(struct mem_cgroup *memcg) { + lru_gen_exit_memcg(memcg); memcg_wb_domain_exit(memcg); __mem_cgroup_free(memcg); } @@ -5130,6 +5131,7 @@ static struct mem_cgroup *mem_cgroup_alloc(void) memcg->deferred_split_queue.split_queue_len =3D 0; #endif idr_replace(&mem_cgroup_idr, memcg, memcg->id.id); + lru_gen_init_memcg(memcg); return memcg; fail: mem_cgroup_id_remove(memcg); diff --git a/mm/memory.c b/mm/memory.c index 44a1ec7a2cac..6df27b84c5aa 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4812,6 +4812,27 @@ static inline void mm_account_fault(struct pt_regs *= regs, perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS_MIN, 1, regs, address); } =20 +#ifdef CONFIG_LRU_GEN +static void lru_gen_enter_fault(struct vm_area_struct *vma) +{ + /* the LRU algorithm doesn't apply to sequential or random reads */ + current->in_lru_fault =3D !(vma->vm_flags & (VM_SEQ_READ | VM_RAND_READ)); +} + +static void lru_gen_exit_fault(void) +{ + current->in_lru_fault =3D false; +} +#else +static void lru_gen_enter_fault(struct vm_area_struct *vma) +{ +} + +static void lru_gen_exit_fault(void) +{ +} +#endif /* CONFIG_LRU_GEN */ + /* * By the time we get here, we already hold the mm semaphore * @@ -4843,11 +4864,15 @@ vm_fault_t handle_mm_fault(struct vm_area_struct *v= ma, unsigned long address, if (flags & FAULT_FLAG_USER) mem_cgroup_enter_user_fault(); =20 + lru_gen_enter_fault(vma); + if (unlikely(is_vm_hugetlb_page(vma))) ret =3D hugetlb_fault(vma->vm_mm, vma, address, flags); else ret =3D __handle_mm_fault(vma, address, flags); =20 + lru_gen_exit_fault(); + if (flags & FAULT_FLAG_USER) { mem_cgroup_exit_user_fault(); /* diff --git a/mm/mm_init.c b/mm/mm_init.c index 9ddaf0e1b0ab..0d7b2bd2454a 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -65,14 +65,16 @@ void __init mminit_verify_pageflags_layout(void) =20 shift =3D 8 * sizeof(unsigned long); width =3D shift - SECTIONS_WIDTH - NODES_WIDTH - ZONES_WIDTH - - LAST_CPUPID_SHIFT - KASAN_TAG_WIDTH; + - LAST_CPUPID_SHIFT - KASAN_TAG_WIDTH - LRU_GEN_WIDTH - LRU_REFS_WIDTH; mminit_dprintk(MMINIT_TRACE, "pageflags_layout_widths", - "Section %d Node %d Zone %d Lastcpupid %d Kasantag %d Flags %d\n", + "Section %d Node %d Zone %d Lastcpupid %d Kasantag %d Gen %d Tier %d Fla= gs %d\n", SECTIONS_WIDTH, NODES_WIDTH, ZONES_WIDTH, LAST_CPUPID_WIDTH, KASAN_TAG_WIDTH, + LRU_GEN_WIDTH, + LRU_REFS_WIDTH, NR_PAGEFLAGS); mminit_dprintk(MMINIT_TRACE, "pageflags_layout_shifts", "Section %d Node %d Zone %d Lastcpupid %d Kasantag %d\n", diff --git a/mm/mmzone.c b/mm/mmzone.c index 0ae7571e35ab..68e1511be12d 100644 --- a/mm/mmzone.c +++ b/mm/mmzone.c @@ -88,6 +88,8 @@ void lruvec_init(struct lruvec *lruvec) * Poison its list head, so that any operations on it would crash. */ list_del(&lruvec->lists[LRU_UNEVICTABLE]); + + lru_gen_init_lruvec(lruvec); } =20 #if defined(CONFIG_NUMA_BALANCING) && !defined(LAST_CPUPID_NOT_IN_PAGE_FLA= GS) diff --git a/mm/swap.c b/mm/swap.c index 7e320ec08c6a..a6870ba0bd83 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -460,6 +460,11 @@ void folio_add_lru(struct folio *folio) VM_BUG_ON_FOLIO(folio_test_active(folio) && folio_test_unevictable(folio)= , folio); VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); =20 + /* see the comment in lru_gen_add_folio() */ + if (lru_gen_enabled() && !folio_test_unevictable(folio) && + lru_gen_in_fault() && !(current->flags & PF_MEMALLOC)) + folio_set_active(folio); + folio_get(folio); local_lock(&lru_pvecs.lock); pvec =3D this_cpu_ptr(&lru_pvecs.lru_add); @@ -551,7 +556,7 @@ static void lru_deactivate_file_fn(struct page *page, s= truct lruvec *lruvec) =20 static void lru_deactivate_fn(struct page *page, struct lruvec *lruvec) { - if (PageActive(page) && !PageUnevictable(page)) { + if (!PageUnevictable(page) && (PageActive(page) || lru_gen_enabled())) { int nr_pages =3D thp_nr_pages(page); =20 del_page_from_lru_list(page, lruvec); @@ -666,7 +671,7 @@ void deactivate_file_folio(struct folio *folio) */ void deactivate_page(struct page *page) { - if (PageLRU(page) && PageActive(page) && !PageUnevictable(page)) { + if (PageLRU(page) && !PageUnevictable(page) && (PageActive(page) || lru_g= en_enabled())) { struct pagevec *pvec; =20 local_lock(&lru_pvecs.lock); diff --git a/mm/vmscan.c b/mm/vmscan.c index 2232cb55af41..b41ff9765cc7 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2968,6 +2968,81 @@ static bool can_age_anon_pages(struct pglist_data *p= gdat, return can_demote(pgdat->node_id, sc); } =20 +#ifdef CONFIG_LRU_GEN + +/*************************************************************************= ***** + * shorthand helpers + *************************************************************************= *****/ + +#define for_each_gen_type_zone(gen, type, zone) \ + for ((gen) =3D 0; (gen) < MAX_NR_GENS; (gen)++) \ + for ((type) =3D 0; (type) < ANON_AND_FILE; (type)++) \ + for ((zone) =3D 0; (zone) < MAX_NR_ZONES; (zone)++) + +static struct lruvec __maybe_unused *get_lruvec(struct mem_cgroup *memcg, = int nid) +{ + struct pglist_data *pgdat =3D NODE_DATA(nid); + +#ifdef CONFIG_MEMCG + if (memcg) { + struct lruvec *lruvec =3D &memcg->nodeinfo[nid]->lruvec; + + /* for hotadd_new_pgdat() */ + if (!lruvec->pgdat) + lruvec->pgdat =3D pgdat; + + return lruvec; + } +#endif + VM_WARN_ON_ONCE(!mem_cgroup_disabled()); + + return pgdat ? &pgdat->__lruvec : NULL; +} + +/*************************************************************************= ***** + * initialization + *************************************************************************= *****/ + +void lru_gen_init_lruvec(struct lruvec *lruvec) +{ + int gen, type, zone; + struct lru_gen_struct *lrugen =3D &lruvec->lrugen; + + lrugen->max_seq =3D MIN_NR_GENS + 1; + + for_each_gen_type_zone(gen, type, zone) + INIT_LIST_HEAD(&lrugen->lists[gen][type][zone]); +} + +#ifdef CONFIG_MEMCG +void lru_gen_init_memcg(struct mem_cgroup *memcg) +{ +} + +void lru_gen_exit_memcg(struct mem_cgroup *memcg) +{ + int nid; + + for_each_node(nid) { + struct lruvec *lruvec =3D get_lruvec(memcg, nid); + + VM_WARN_ON_ONCE(memchr_inv(lruvec->lrugen.nr_pages, 0, + sizeof(lruvec->lrugen.nr_pages))); + } +} +#endif + +static int __init init_lru_gen(void) +{ + BUILD_BUG_ON(MIN_NR_GENS + 1 >=3D MAX_NR_GENS); + BUILD_BUG_ON(BIT(LRU_GEN_WIDTH) <=3D MAX_NR_GENS); + + return 0; +}; +late_initcall(init_lru_gen); + +#endif /* CONFIG_LRU_GEN */ + static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) { unsigned long nr[NR_LRU_LISTS]; --=20 2.36.0.550.gb090851708-goog From nobody Thu May 7 21:41:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4D14C433FE for ; Wed, 18 May 2022 01:47:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233889AbiERBrp (ORCPT ); Tue, 17 May 2022 21:47:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57970 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233860AbiERBrT (ORCPT ); Tue, 17 May 2022 21:47:19 -0400 Received: from mail-il1-x149.google.com (mail-il1-x149.google.com [IPv6:2607:f8b0:4864:20::149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A2E8546A5 for ; Tue, 17 May 2022 18:47:03 -0700 (PDT) Received: by mail-il1-x149.google.com with SMTP id g11-20020a056e021a2b00b002cf48b48824so424976ile.21 for ; Tue, 17 May 2022 18:47:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc:content-transfer-encoding; bh=iMUum4oayASrVd38eQwbqeH3l0KJDrlIs7oQvS50rmk=; b=HOGuPLk4pea7kZOf8/8xEnWpFAvWhETkdy0zM1uAW8txEYwG6qHJfwzRlckeGVXtmT Iht3HG3gRybZ2m5Ek+nKvBKtk/GK9yXDnrTJufjQ3xL61WSbxrteixzt+Ctdf6DKxmes 1yiwZlk1gqbWHHEOePU6pFRupL3i/JNQhuNnwA0LTUIvQXgdXwryEKUOAgjtwv5eaBA4 fRVocraJzdduX/TZyYIYPAeGCpkUi92M+YNY2/xhUvgJYt7eukhfdJyYFy7ir4twh8OD w4rwkX8IWlW/3yIy1/egJ2Trq/oQYB1IGulETTb8pAKfKnn6y2HYn0+tXgdSGv1bUOTt +xPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc:content-transfer-encoding; bh=iMUum4oayASrVd38eQwbqeH3l0KJDrlIs7oQvS50rmk=; b=Uvvn+nUPYS5F7sALWmFvzT3PBeHA/RIV+dbpRbHNARc8RcuQv2S93ISDLlnL7dqtAR PQKU0nEGuxd5JUbp9RNQ1yM4RHRB4hi0xEDJKioiUUxbq4C6pzF4cNn2xjiFjv3ZgtMZ SP+xcVdSwrQDvWHBPLk1LkrEXNchBJZ/Qkffwc92ugSO71CdXzPZcRYfxpevD4jVFdB8 E1jK15qGQjUyI71jW/xks9II3a5Arpj2S7I+oAQFuiaWUxGOe0Js2CVTS7mjt+6iBygB 2xUOiBK/FsANNVO2+E1uVdb7kMtx6l3yigfsW0/3esp4EdVJgNSMp20byk6uN4sUtXgV HFmw== X-Gm-Message-State: AOAM531shUqrJBk5jn+qy4wXcbGbzRMePgs+j10x/XCGPq1obmTYKoai q3jpjzjAmAMY8t/c6Ra1ZpV9xRtvcZ0= X-Google-Smtp-Source: ABdhPJwJ0Tnl/R9U39FiMVm/fDi71yI7+/4DMvC0HIczyZ6y9l2A6E755d8kGMmKnv1Cg0TOt5mKl/500Us= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f7bc:90c9:d86e:4ea8]) (user=yuzhao job=sendgmr) by 2002:a05:6602:1409:b0:5e7:487:133c with SMTP id t9-20020a056602140900b005e70487133cmr11313973iov.196.1652838423113; Tue, 17 May 2022 18:47:03 -0700 (PDT) Date: Tue, 17 May 2022 19:46:25 -0600 In-Reply-To: <20220518014632.922072-1-yuzhao@google.com> Message-Id: <20220518014632.922072-7-yuzhao@google.com> Mime-Version: 1.0 References: <20220518014632.922072-1-yuzhao@google.com> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog Subject: [PATCH v11 06/14] mm: multi-gen LRU: minimal implementation From: Yu Zhao To: Andrew Morton , linux-mm@kvack.org Cc: Andi Kleen , Aneesh Kumar , Catalin Marinas , Dave Hansen , Hillf Danton , Jens Axboe , Johannes Weiner , Jonathan Corbet , Linus Torvalds , Matthew Wilcox , Mel Gorman , Michael Larabel , Michal Hocko , Mike Rapoport , Peter Zijlstra , Tejun Heo , Vlastimil Babka , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, page-reclaim@google.com, Yu Zhao , Brian Geffon , Jan Alexander Steffens , Oleksandr Natalenko , Steven Barrett , Suleiman Souhlal , Daniel Byrne , Donald Carr , "=?UTF-8?q?Holger=20Hoffst=C3=A4tte?=" , Konstantin Kharlamov , Shuang Zhai , Sofia Trinh , Vaibhav Jain Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" To avoid confusion, the terms "promotion" and "demotion" will be applied to the multi-gen LRU, as a new convention; the terms "activation" and "deactivation" will be applied to the active/inactive LRU, as usual. The aging produces young generations. Given an lruvec, it increments max_seq when max_seq-min_seq+1 approaches MIN_NR_GENS. The aging promotes hot pages to the youngest generation when it finds them accessed through page tables; the demotion of cold pages happens consequently when it increments max_seq. The aging has the complexity O(nr_hot_pages), since it is only interested in hot pages. Promotion in the aging path does not involve any LRU list operations, only the updates of the gen counter and lrugen->nr_pages[]; demotion, unless as the result of the increment of max_seq, requires LRU list operations, e.g., lru_deactivate_fn(). The eviction consumes old generations. Given an lruvec, it increments min_seq when the lists indexed by min_seq%MAX_NR_GENS become empty. A feedback loop modeled after the PID controller monitors refaults over anon and file types and decides which type to evict when both types are available from the same generation. Each generation is divided into multiple tiers. Tiers represent different ranges of numbers of accesses through file descriptors. A page accessed N times through file descriptors is in tier order_base_2(N). Tiers do not have dedicated lrugen->lists[], only bits in folio->flags. In contrast to moving across generations, which requires the LRU lock, moving across tiers only involves operations on folio->flags. The feedback loop also monitors refaults over all tiers and decides when to protect pages in which tiers (N>1), using the first tier (N=3D0,1) as a baseline. The first tier contains single-use unmapped clean pages, which are most likely the best choices. The eviction moves a page to the next generation, i.e., min_seq+1, if the feedback loop decides so. This approach has the following advantages: 1. It removes the cost of activation in the buffered access path by inferring whether pages accessed multiple times through file descriptors are statistically hot and thus worth protecting in the eviction path. 2. It takes pages accessed through page tables into account and avoids overprotecting pages accessed multiple times through file descriptors. (Pages accessed through page tables are in the first tier, since N=3D0.) 3. More tiers provide better protection for pages accessed more than twice through file descriptors, when under heavy buffered I/O workloads. Server benchmark results: Single workload: fio (buffered I/O): +[40, 42]% IOPS BW 5.18-rc1: 2463k 9621MiB/s patch1-6: 3484k 13.3GiB/s Single workload: memcached (anon): +[44, 46]% Ops/sec KB/sec 5.18-rc1: 771403.27 30004.17 patch1-6: 1120643.70 43588.06 Configurations: CPU: two Xeon 6154 Mem: total 256G Node 1 was only used as a ram disk to reduce the variance in the results. patch drivers/block/brd.c < gfp_flags =3D GFP_NOIO | __GFP_ZERO | __GFP_HIGHMEM | __GFP_THISNODE; > page =3D alloc_pages_node(1, gfp_flags, 0); EOF cat >>/etc/systemd/system.conf <>/etc/memcached.conf </sys/fs/cgroup/user.slice/test/memory.max echo $$ >/sys/fs/cgroup/user.slice/test/cgroup.procs fio -name=3Dmglru --numjobs=3D72 --directory=3D/mnt --size=3D1408m \ --buffered=3D1 --ioengine=3Dio_uring --iodepth=3D128 \ --iodepth_batch_submit=3D32 --iodepth_batch_complete=3D32 \ --rw=3Drandread --random_distribution=3Drandom --norandommap \ --time_based --ramp_time=3D10m --runtime=3D5m --group_reporting cat memcached.sh modprobe brd rd_nr=3D1 rd_size=3D113246208 swapoff -a mkswap /dev/ram0 swapon /dev/ram0 memtier_benchmark -S /var/run/memcached/memcached.sock \ -P memcache_binary -n allkeys --key-minimum=3D1 \ --key-maximum=3D65000000 --key-pattern=3DP:P -c 1 -t 36 \ --ratio 1:0 --pipeline 8 -d 2000 memtier_benchmark -S /var/run/memcached/memcached.sock \ -P memcache_binary -n allkeys --key-minimum=3D1 \ --key-maximum=3D65000000 --key-pattern=3DR:R -c 1 -t 36 \ --ratio 0:1 --pipeline 8 --randomize --distinct-client-seed Client benchmark results: kswapd profiles: 5.18-rc1 40.53% page_vma_mapped_walk 20.37% lzo1x_1_do_compress (real work) 6.99% do_raw_spin_lock 3.93% _raw_spin_unlock_irq 2.08% vma_interval_tree_subtree_search 2.06% vma_interval_tree_iter_next 1.95% folio_referenced_one 1.93% anon_vma_interval_tree_iter_first 1.51% ptep_clear_flush 1.35% __anon_vma_interval_tree_subtree_search patch1-6 35.99% lzo1x_1_do_compress (real work) 19.40% page_vma_mapped_walk 6.31% _raw_spin_unlock_irq 3.95% do_raw_spin_lock 2.39% anon_vma_interval_tree_iter_first 2.25% ptep_clear_flush 1.92% __anon_vma_interval_tree_subtree_search 1.70% folio_referenced_one 1.68% __zram_bvec_write 1.43% anon_vma_interval_tree_iter_next Configurations: CPU: single Snapdragon 7c Mem: total 4G Chrome OS MemoryPressure [1] [1] https://chromium.googlesource.com/chromiumos/platform/tast-tests/ Signed-off-by: Yu Zhao Acked-by: Brian Geffon Acked-by: Jan Alexander Steffens (heftig) Acked-by: Oleksandr Natalenko Acked-by: Steven Barrett Acked-by: Suleiman Souhlal Tested-by: Daniel Byrne Tested-by: Donald Carr Tested-by: Holger Hoffst=C3=A4tte Tested-by: Konstantin Kharlamov Tested-by: Shuang Zhai Tested-by: Sofia Trinh Tested-by: Vaibhav Jain --- include/linux/mm_inline.h | 36 ++ include/linux/mmzone.h | 42 ++ include/linux/page-flags-layout.h | 5 +- kernel/bounds.c | 2 + mm/Kconfig | 11 + mm/swap.c | 39 ++ mm/vmscan.c | 799 +++++++++++++++++++++++++++++- mm/workingset.c | 110 +++- 8 files changed, 1034 insertions(+), 10 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 98ae22bfaf12..85fe78832436 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -119,6 +119,33 @@ static inline int lru_gen_from_seq(unsigned long seq) return seq % MAX_NR_GENS; } =20 +static inline int lru_hist_from_seq(unsigned long seq) +{ + return seq % NR_HIST_GENS; +} + +static inline int lru_tier_from_refs(int refs) +{ + VM_WARN_ON_ONCE(refs > BIT(LRU_REFS_WIDTH)); + + /* see the comment in folio_lru_refs() */ + return order_base_2(refs + 1); +} + +static inline int folio_lru_refs(struct folio *folio) +{ + unsigned long flags =3D READ_ONCE(folio->flags); + bool workingset =3D flags & BIT(PG_workingset); + + /* + * Return the number of accesses beyond PG_referenced, i.e., N-1 if the + * total number of accesses is N>1, since N=3D0,1 both map to the first + * tier. lru_tier_from_refs() will account for this off-by-one. Also see + * the comment on MAX_NR_TIERS. + */ + return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + workingset; +} + static inline int folio_lru_gen(struct folio *folio) { unsigned long flags =3D READ_ONCE(folio->flags); @@ -171,6 +198,15 @@ static inline void lru_gen_update_size(struct lruvec *= lruvec, struct folio *foli __update_lru_size(lruvec, lru, zone, -delta); return; } + + /* promotion */ + if (!lru_gen_is_active(lruvec, old_gen) && lru_gen_is_active(lruvec, new_= gen)) { + __update_lru_size(lruvec, lru, zone, -delta); + __update_lru_size(lruvec, lru + LRU_ACTIVE, zone, delta); + } + + /* demotion requires isolation, e.g., lru_deactivate_fn() */ + VM_WARN_ON_ONCE(lru_gen_is_active(lruvec, old_gen) && !lru_gen_is_active(= lruvec, new_gen)); } =20 static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *= folio, bool reclaiming) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 6994acef63cb..2d023d243e73 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -348,6 +348,29 @@ enum lruvec_flags { #define MIN_NR_GENS 2U #define MAX_NR_GENS 4U =20 +/* + * Each generation is divided into multiple tiers. Tiers represent differe= nt + * ranges of numbers of accesses through file descriptors. A page accessed= N + * times through file descriptors is in tier order_base_2(N). A page in the + * first tier (N=3D0,1) is marked by PG_referenced unless it was faulted in + * though page tables or read ahead. A page in any other tier (N>1) is mar= ked + * by PG_referenced and PG_workingset. This implies a minimum of two tiers= is + * supported without using additional bits in folio->flags. + * + * In contrast to moving across generations which requires the LRU lock, m= oving + * across tiers only involves atomic operations on folio->flags and theref= ore + * has a negligible cost in the buffered access path. In the eviction path, + * comparisons of refaulted/(evicted+protected) from the first tier and the + * rest infer whether pages accessed multiple times through file descripto= rs + * are statistically hot and thus worth protecting. + * + * MAX_NR_TIERS is set to 4 so that the multi-gen LRU can support twice the + * number of categories of the active/inactive LRU when keeping track of + * accesses through file descriptors. It uses MAX_NR_TIERS-2 spare bits in + * folio->flags (LRU_REFS_MASK). + */ +#define MAX_NR_TIERS 4U + #ifndef __GENERATING_BOUNDS_H =20 struct lruvec; @@ -362,6 +385,16 @@ enum { LRU_GEN_FILE, }; =20 +#define MIN_LRU_BATCH BITS_PER_LONG +#define MAX_LRU_BATCH (MIN_LRU_BATCH * 128) + +/* whether to keep historical stats from evicted generations */ +#ifdef CONFIG_LRU_GEN_STATS +#define NR_HIST_GENS MAX_NR_GENS +#else +#define NR_HIST_GENS 1U +#endif + /* * The youngest generation number is stored in max_seq for both anon and f= ile * types as they are aged on an equal footing. The oldest generation numbe= rs are @@ -384,6 +417,15 @@ struct lru_gen_struct { struct list_head lists[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; /* the sizes of the above lists */ long nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; + /* the exponential moving average of refaulted */ + unsigned long avg_refaulted[ANON_AND_FILE][MAX_NR_TIERS]; + /* the exponential moving average of evicted+protected */ + unsigned long avg_total[ANON_AND_FILE][MAX_NR_TIERS]; + /* the first tier doesn't need protection, hence the minus one */ + unsigned long protected[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS - 1]; + /* can be modified without holding the LRU lock */ + atomic_long_t evicted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS]; + atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS]; }; =20 void lru_gen_init_lruvec(struct lruvec *lruvec); diff --git a/include/linux/page-flags-layout.h b/include/linux/page-flags-l= ayout.h index 240905407a18..7d79818dc065 100644 --- a/include/linux/page-flags-layout.h +++ b/include/linux/page-flags-layout.h @@ -106,7 +106,10 @@ #error "Not enough bits in page flags" #endif =20 -#define LRU_REFS_WIDTH 0 +/* see the comment on MAX_NR_TIERS */ +#define LRU_REFS_WIDTH min(__LRU_REFS_WIDTH, BITS_PER_LONG - NR_PAGEFLAGS = - \ + ZONES_WIDTH - LRU_GEN_WIDTH - SECTIONS_WIDTH - \ + NODES_WIDTH - KASAN_TAG_WIDTH - LAST_CPUPID_WIDTH) =20 #endif #endif /* _LINUX_PAGE_FLAGS_LAYOUT */ diff --git a/kernel/bounds.c b/kernel/bounds.c index 5ee60777d8e4..b529182e8b04 100644 --- a/kernel/bounds.c +++ b/kernel/bounds.c @@ -24,8 +24,10 @@ int main(void) DEFINE(SPINLOCK_SIZE, sizeof(spinlock_t)); #ifdef CONFIG_LRU_GEN DEFINE(LRU_GEN_WIDTH, order_base_2(MAX_NR_GENS + 1)); + DEFINE(__LRU_REFS_WIDTH, MAX_NR_TIERS - 2); #else DEFINE(LRU_GEN_WIDTH, 0); + DEFINE(__LRU_REFS_WIDTH, 0); #endif /* End of constants */ =20 diff --git a/mm/Kconfig b/mm/Kconfig index e62bd501082b..0aeacbd3361c 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -909,6 +909,7 @@ config ANON_VMA_NAME area from being merged with adjacent virtual memory areas due to the difference in their name. =20 +# multi-gen LRU { config LRU_GEN bool "Multi-Gen LRU" depends on MMU @@ -917,6 +918,16 @@ config LRU_GEN help A high performance LRU implementation to overcommit memory. =20 +config LRU_GEN_STATS + bool "Full stats for debugging" + depends on LRU_GEN + help + Do not enable this option unless you plan to look at historical stats + from evicted generations for debugging purpose. + + This option has a per-memcg and per-node memory overhead. +# } + source "mm/damon/Kconfig" =20 endmenu diff --git a/mm/swap.c b/mm/swap.c index a6870ba0bd83..a99d22308f28 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -405,6 +405,40 @@ static void __lru_cache_activate_folio(struct folio *f= olio) local_unlock(&lru_pvecs.lock); } =20 +#ifdef CONFIG_LRU_GEN +static void folio_inc_refs(struct folio *folio) +{ + unsigned long new_flags, old_flags =3D READ_ONCE(folio->flags); + + if (folio_test_unevictable(folio)) + return; + + if (!folio_test_referenced(folio)) { + folio_set_referenced(folio); + return; + } + + if (!folio_test_workingset(folio)) { + folio_set_workingset(folio); + return; + } + + /* see the comment on MAX_NR_TIERS */ + do { + new_flags =3D old_flags & LRU_REFS_MASK; + if (new_flags =3D=3D LRU_REFS_MASK) + break; + + new_flags +=3D BIT(LRU_REFS_PGOFF); + new_flags |=3D old_flags & ~LRU_REFS_MASK; + } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); +} +#else +static void folio_inc_refs(struct folio *folio) +{ +} +#endif /* CONFIG_LRU_GEN */ + /* * Mark a page as having seen activity. * @@ -417,6 +451,11 @@ static void __lru_cache_activate_folio(struct folio *f= olio) */ void folio_mark_accessed(struct folio *folio) { + if (lru_gen_enabled()) { + folio_inc_refs(folio); + return; + } + if (!folio_test_referenced(folio)) { folio_set_referenced(folio); } else if (folio_test_unevictable(folio)) { diff --git a/mm/vmscan.c b/mm/vmscan.c index b41ff9765cc7..891f0ab69b3a 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1275,9 +1275,11 @@ static int __remove_mapping(struct address_space *ma= pping, struct folio *folio, =20 if (folio_test_swapcache(folio)) { swp_entry_t swap =3D folio_swap_entry(folio); - mem_cgroup_swapout(folio, swap); + + /* get a shadow entry before mem_cgroup_swapout() clears folio_memcg() */ if (reclaimed && !mapping_exiting(mapping)) shadow =3D workingset_eviction(folio, target_memcg); + mem_cgroup_swapout(folio, swap); __delete_from_swap_cache(&folio->page, swap, shadow); xa_unlock_irq(&mapping->i_pages); put_swap_page(&folio->page, swap); @@ -2649,6 +2651,9 @@ static void prepare_scan_count(pg_data_t *pgdat, stru= ct scan_control *sc) unsigned long file; struct lruvec *target_lruvec; =20 + if (lru_gen_enabled()) + return; + target_lruvec =3D mem_cgroup_lruvec(sc->target_mem_cgroup, pgdat); =20 /* @@ -2974,6 +2979,17 @@ static bool can_age_anon_pages(struct pglist_data *p= gdat, * shorthand helpers *************************************************************************= *****/ =20 +#define LRU_REFS_FLAGS (BIT(PG_referenced) | BIT(PG_workingset)) + +#define DEFINE_MAX_SEQ(lruvec) \ + unsigned long max_seq =3D READ_ONCE((lruvec)->lrugen.max_seq) + +#define DEFINE_MIN_SEQ(lruvec) \ + unsigned long min_seq[ANON_AND_FILE] =3D { \ + READ_ONCE((lruvec)->lrugen.min_seq[LRU_GEN_ANON]), \ + READ_ONCE((lruvec)->lrugen.min_seq[LRU_GEN_FILE]), \ + } + #define for_each_gen_type_zone(gen, type, zone) \ for ((gen) =3D 0; (gen) < MAX_NR_GENS; (gen)++) \ for ((type) =3D 0; (type) < ANON_AND_FILE; (type)++) \ @@ -2999,6 +3015,753 @@ static struct lruvec __maybe_unused *get_lruvec(str= uct mem_cgroup *memcg, int ni return pgdat ? &pgdat->__lruvec : NULL; } =20 +static int get_swappiness(struct lruvec *lruvec, struct scan_control *sc) +{ + struct mem_cgroup *memcg =3D lruvec_memcg(lruvec); + struct pglist_data *pgdat =3D lruvec_pgdat(lruvec); + + if (!can_demote(pgdat->node_id, sc) && + mem_cgroup_get_nr_swap_pages(memcg) < MIN_LRU_BATCH) + return 0; + + return mem_cgroup_swappiness(memcg); +} + +static int get_nr_gens(struct lruvec *lruvec, int type) +{ + return lruvec->lrugen.max_seq - lruvec->lrugen.min_seq[type] + 1; +} + +static bool __maybe_unused seq_is_valid(struct lruvec *lruvec) +{ + /* see the comment on lru_gen_struct */ + return get_nr_gens(lruvec, LRU_GEN_FILE) >=3D MIN_NR_GENS && + get_nr_gens(lruvec, LRU_GEN_FILE) <=3D get_nr_gens(lruvec, LRU_GEN= _ANON) && + get_nr_gens(lruvec, LRU_GEN_ANON) <=3D MAX_NR_GENS; +} + +/*************************************************************************= ***** + * refault feedback loop + *************************************************************************= *****/ + +/* + * A feedback loop based on Proportional-Integral-Derivative (PID) control= ler. + * + * The P term is refaulted/(evicted+protected) from a tier in the generati= on + * currently being evicted; the I term is the exponential moving average o= f the + * P term over the generations previously evicted, using the smoothing fac= tor + * 1/2; the D term isn't supported. + * + * The setpoint (SP) is always the first tier of one type; the process var= iable + * (PV) is either any tier of the other type or any other tier of the same + * type. + * + * The error is the difference between the SP and the PV; the correction is + * turn off protection when SP>PV or turn on protection when SPlrugen; + int hist =3D lru_hist_from_seq(lrugen->min_seq[type]); + + pos->refaulted =3D lrugen->avg_refaulted[type][tier] + + atomic_long_read(&lrugen->refaulted[hist][type][tier]); + pos->total =3D lrugen->avg_total[type][tier] + + atomic_long_read(&lrugen->evicted[hist][type][tier]); + if (tier) + pos->total +=3D lrugen->protected[hist][type][tier - 1]; + pos->gain =3D gain; +} + +static void reset_ctrl_pos(struct lruvec *lruvec, int type, bool carryover) +{ + int hist, tier; + struct lru_gen_struct *lrugen =3D &lruvec->lrugen; + bool clear =3D carryover ? NR_HIST_GENS =3D=3D 1 : NR_HIST_GENS > 1; + unsigned long seq =3D carryover ? lrugen->min_seq[type] : lrugen->max_seq= + 1; + + lockdep_assert_held(&lruvec->lru_lock); + + if (!carryover && !clear) + return; + + hist =3D lru_hist_from_seq(seq); + + for (tier =3D 0; tier < MAX_NR_TIERS; tier++) { + if (carryover) { + unsigned long sum; + + sum =3D lrugen->avg_refaulted[type][tier] + + atomic_long_read(&lrugen->refaulted[hist][type][tier]); + WRITE_ONCE(lrugen->avg_refaulted[type][tier], sum / 2); + + sum =3D lrugen->avg_total[type][tier] + + atomic_long_read(&lrugen->evicted[hist][type][tier]); + if (tier) + sum +=3D lrugen->protected[hist][type][tier - 1]; + WRITE_ONCE(lrugen->avg_total[type][tier], sum / 2); + } + + if (clear) { + atomic_long_set(&lrugen->refaulted[hist][type][tier], 0); + atomic_long_set(&lrugen->evicted[hist][type][tier], 0); + if (tier) + WRITE_ONCE(lrugen->protected[hist][type][tier - 1], 0); + } + } +} + +static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv) +{ + /* + * Return true if the PV has a limited number of refaults or a lower + * refaulted/total than the SP. + */ + return pv->refaulted < MIN_LRU_BATCH || + pv->refaulted * (sp->total + MIN_LRU_BATCH) * sp->gain <=3D + (sp->refaulted + 1) * pv->total * pv->gain; +} + +/*************************************************************************= ***** + * the aging + *************************************************************************= *****/ + +static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool = reclaiming) +{ + int type =3D folio_is_file_lru(folio); + struct lru_gen_struct *lrugen =3D &lruvec->lrugen; + int new_gen, old_gen =3D lru_gen_from_seq(lrugen->min_seq[type]); + unsigned long new_flags, old_flags =3D READ_ONCE(folio->flags); + + VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio); + + do { + new_gen =3D (old_gen + 1) % MAX_NR_GENS; + + new_flags =3D old_flags & ~(LRU_GEN_MASK | LRU_REFS_MASK | LRU_REFS_FLAG= S); + new_flags |=3D (new_gen + 1UL) << LRU_GEN_PGOFF; + /* for folio_end_writeback() */ + if (reclaiming) + new_flags |=3D BIT(PG_reclaim); + } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); + + lru_gen_update_size(lruvec, folio, old_gen, new_gen); + + return new_gen; +} + +static void inc_min_seq(struct lruvec *lruvec, int type) +{ + struct lru_gen_struct *lrugen =3D &lruvec->lrugen; + + reset_ctrl_pos(lruvec, type, true); + WRITE_ONCE(lrugen->min_seq[type], lrugen->min_seq[type] + 1); +} + +static bool try_to_inc_min_seq(struct lruvec *lruvec, bool can_swap) +{ + int gen, type, zone; + bool success =3D false; + struct lru_gen_struct *lrugen =3D &lruvec->lrugen; + DEFINE_MIN_SEQ(lruvec); + + VM_WARN_ON_ONCE(!seq_is_valid(lruvec)); + + /* find the oldest populated generation */ + for (type =3D !can_swap; type < ANON_AND_FILE; type++) { + while (min_seq[type] + MIN_NR_GENS <=3D lrugen->max_seq) { + gen =3D lru_gen_from_seq(min_seq[type]); + + for (zone =3D 0; zone < MAX_NR_ZONES; zone++) { + if (!list_empty(&lrugen->lists[gen][type][zone])) + goto next; + } + + min_seq[type]++; + } +next: + ; + } + + /* see the comment on lru_gen_struct */ + if (can_swap) { + min_seq[LRU_GEN_ANON] =3D min(min_seq[LRU_GEN_ANON], min_seq[LRU_GEN_FIL= E]); + min_seq[LRU_GEN_FILE] =3D max(min_seq[LRU_GEN_ANON], lrugen->min_seq[LRU= _GEN_FILE]); + } + + for (type =3D !can_swap; type < ANON_AND_FILE; type++) { + if (min_seq[type] =3D=3D lrugen->min_seq[type]) + continue; + + reset_ctrl_pos(lruvec, type, true); + WRITE_ONCE(lrugen->min_seq[type], min_seq[type]); + success =3D true; + } + + return success; +} + +static void inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, bool= can_swap) +{ + int prev, next; + int type, zone; + struct lru_gen_struct *lrugen =3D &lruvec->lrugen; + + spin_lock_irq(&lruvec->lru_lock); + + VM_WARN_ON_ONCE(!seq_is_valid(lruvec)); + + if (max_seq !=3D lrugen->max_seq) + goto unlock; + + for (type =3D 0; type < ANON_AND_FILE; type++) { + if (get_nr_gens(lruvec, type) !=3D MAX_NR_GENS) + continue; + + VM_WARN_ON_ONCE(type =3D=3D LRU_GEN_FILE || can_swap); + + inc_min_seq(lruvec, type); + } + + /* + * Update the active/inactive LRU sizes for compatibility. Both sides of + * the current max_seq need to be covered, since max_seq+1 can overlap + * with min_seq[LRU_GEN_ANON] if swapping is constrained. And if they do + * overlap, cold/hot inversion happens. + */ + prev =3D lru_gen_from_seq(lrugen->max_seq - 1); + next =3D lru_gen_from_seq(lrugen->max_seq + 1); + + for (type =3D 0; type < ANON_AND_FILE; type++) { + for (zone =3D 0; zone < MAX_NR_ZONES; zone++) { + enum lru_list lru =3D type * LRU_INACTIVE_FILE; + long delta =3D lrugen->nr_pages[prev][type][zone] - + lrugen->nr_pages[next][type][zone]; + + if (!delta) + continue; + + __update_lru_size(lruvec, lru, zone, delta); + __update_lru_size(lruvec, lru + LRU_ACTIVE, zone, -delta); + } + } + + for (type =3D 0; type < ANON_AND_FILE; type++) + reset_ctrl_pos(lruvec, type, false); + + /* make sure preceding modifications appear */ + smp_store_release(&lrugen->max_seq, lrugen->max_seq + 1); +unlock: + spin_unlock_irq(&lruvec->lru_lock); +} + +static long get_nr_evictable(struct lruvec *lruvec, unsigned long max_seq, + unsigned long *min_seq, bool can_swap, bool *need_aging) +{ + int gen, type, zone; + long old =3D 0; + long young =3D 0; + long total =3D 0; + struct lru_gen_struct *lrugen =3D &lruvec->lrugen; + + for (type =3D !can_swap; type < ANON_AND_FILE; type++) { + unsigned long seq; + + for (seq =3D min_seq[type]; seq <=3D max_seq; seq++) { + long size =3D 0; + + gen =3D lru_gen_from_seq(seq); + + for (zone =3D 0; zone < MAX_NR_ZONES; zone++) + size +=3D READ_ONCE(lrugen->nr_pages[gen][type][zone]); + + total +=3D size; + if (seq =3D=3D max_seq) + young +=3D size; + if (seq + MIN_NR_GENS =3D=3D max_seq) + old +=3D size; + } + } + + /* + * The aging tries to be lazy to reduce the overhead. On the other hand, + * the eviction stalls when the number of generations reaches + * MIN_NR_GENS. So ideally, there should be MIN_NR_GENS+1 generations, + * hence the first two if's. + * + * Also it's ideal to spread pages out evenly, meaning 1/(MIN_NR_GENS+1) + * of the total number of pages for each generation. A reasonable range + * for this average portion is [1/MIN_NR_GENS, 1/(MIN_NR_GENS+2)]. The + * eviction cares about the lower bound of cold pages, whereas the aging + * cares about the upper bound of hot pages. + */ + if (min_seq[!can_swap] + MIN_NR_GENS > max_seq) + *need_aging =3D true; + else if (min_seq[!can_swap] + MIN_NR_GENS < max_seq) + *need_aging =3D false; + else if (young * MIN_NR_GENS > total) + *need_aging =3D true; + else if (old * (MIN_NR_GENS + 2) < total) + *need_aging =3D true; + else + *need_aging =3D false; + + return total > 0 ? total : 0; +} + +static void age_lruvec(struct lruvec *lruvec, struct scan_control *sc) +{ + bool need_aging; + long nr_to_scan; + int swappiness =3D get_swappiness(lruvec, sc); + struct mem_cgroup *memcg =3D lruvec_memcg(lruvec); + DEFINE_MAX_SEQ(lruvec); + DEFINE_MIN_SEQ(lruvec); + + VM_WARN_ON_ONCE(sc->memcg_low_reclaim); + + mem_cgroup_calculate_protection(NULL, memcg); + + if (mem_cgroup_below_min(memcg)) + return; + + nr_to_scan =3D get_nr_evictable(lruvec, max_seq, min_seq, swappiness, &ne= ed_aging); + if (!nr_to_scan) + return; + + nr_to_scan >>=3D sc->priority; + + if (!mem_cgroup_online(memcg)) + nr_to_scan++; + + if (nr_to_scan && need_aging) + inc_max_seq(lruvec, max_seq, swappiness); +} + +static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_contro= l *sc) +{ + struct mem_cgroup *memcg; + + VM_WARN_ON_ONCE(!current_is_kswapd()); + + memcg =3D mem_cgroup_iter(NULL, NULL, NULL); + do { + struct lruvec *lruvec =3D mem_cgroup_lruvec(memcg, pgdat); + + age_lruvec(lruvec, sc); + + cond_resched(); + } while ((memcg =3D mem_cgroup_iter(NULL, memcg, NULL))); +} + +/*************************************************************************= ***** + * the eviction + *************************************************************************= *****/ + +static bool sort_folio(struct lruvec *lruvec, struct folio *folio, int tie= r_idx) +{ + bool success; + int gen =3D folio_lru_gen(folio); + int type =3D folio_is_file_lru(folio); + int zone =3D folio_zonenum(folio); + int delta =3D folio_nr_pages(folio); + int refs =3D folio_lru_refs(folio); + int tier =3D lru_tier_from_refs(refs); + struct lru_gen_struct *lrugen =3D &lruvec->lrugen; + + VM_WARN_ON_ONCE_FOLIO(gen >=3D MAX_NR_GENS, folio); + + /* unevictable */ + if (!folio_evictable(folio)) { + success =3D lru_gen_del_folio(lruvec, folio, true); + VM_WARN_ON_ONCE_FOLIO(!success, folio); + folio_set_unevictable(folio); + lruvec_add_folio(lruvec, folio); + __count_vm_events(UNEVICTABLE_PGCULLED, delta); + return true; + } + + /* dirtied lazyfree */ + if (type =3D=3D LRU_GEN_FILE && folio_test_anon(folio) && folio_test_dirt= y(folio)) { + success =3D lru_gen_del_folio(lruvec, folio, true); + VM_WARN_ON_ONCE_FOLIO(!success, folio); + folio_set_swapbacked(folio); + lruvec_add_folio_tail(lruvec, folio); + return true; + } + + /* protected */ + if (tier > tier_idx) { + int hist =3D lru_hist_from_seq(lrugen->min_seq[type]); + + gen =3D folio_inc_gen(lruvec, folio, false); + list_move_tail(&folio->lru, &lrugen->lists[gen][type][zone]); + + WRITE_ONCE(lrugen->protected[hist][type][tier - 1], + lrugen->protected[hist][type][tier - 1] + delta); + __mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta); + return true; + } + + /* waiting for writeback */ + if (folio_test_locked(folio) || folio_test_writeback(folio) || + (type =3D=3D LRU_GEN_FILE && folio_test_dirty(folio))) { + gen =3D folio_inc_gen(lruvec, folio, true); + list_move(&folio->lru, &lrugen->lists[gen][type][zone]); + return true; + } + + return false; +} + +static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, stru= ct scan_control *sc) +{ + bool success; + + if (!sc->may_unmap && folio_mapped(folio)) + return false; + + if (!(sc->may_writepage && (sc->gfp_mask & __GFP_IO)) && + (folio_test_dirty(folio) || + (folio_test_anon(folio) && !folio_test_swapcache(folio)))) + return false; + + if (!folio_try_get(folio)) + return false; + + if (!folio_test_clear_lru(folio)) { + folio_put(folio); + return false; + } + + success =3D lru_gen_del_folio(lruvec, folio, true); + VM_WARN_ON_ONCE_FOLIO(!success, folio); + + return true; +} + +static int scan_folios(struct lruvec *lruvec, struct scan_control *sc, + int type, int tier, struct list_head *list) +{ + int gen, zone; + enum vm_event_item item; + int sorted =3D 0; + int scanned =3D 0; + int isolated =3D 0; + int remaining =3D MAX_LRU_BATCH; + struct lru_gen_struct *lrugen =3D &lruvec->lrugen; + struct mem_cgroup *memcg =3D lruvec_memcg(lruvec); + + VM_WARN_ON_ONCE(!list_empty(list)); + + if (get_nr_gens(lruvec, type) =3D=3D MIN_NR_GENS) + return 0; + + gen =3D lru_gen_from_seq(lrugen->min_seq[type]); + + for (zone =3D sc->reclaim_idx; zone >=3D 0; zone--) { + LIST_HEAD(moved); + int skipped =3D 0; + struct list_head *head =3D &lrugen->lists[gen][type][zone]; + + while (!list_empty(head)) { + struct folio *folio =3D lru_to_folio(head); + int delta =3D folio_nr_pages(folio); + + VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); + VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio); + VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) !=3D type, folio); + VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D zone, folio); + + scanned +=3D delta; + + if (sort_folio(lruvec, folio, tier)) + sorted +=3D delta; + else if (isolate_folio(lruvec, folio, sc)) { + list_add(&folio->lru, list); + isolated +=3D delta; + } else { + list_move(&folio->lru, &moved); + skipped +=3D delta; + } + + if (!--remaining || max(isolated, skipped) >=3D MIN_LRU_BATCH) + break; + } + + if (skipped) { + list_splice(&moved, head); + __count_zid_vm_events(PGSCAN_SKIP, zone, skipped); + } + + if (!remaining || isolated >=3D MIN_LRU_BATCH) + break; + } + + item =3D current_is_kswapd() ? PGSCAN_KSWAPD : PGSCAN_DIRECT; + if (!cgroup_reclaim(sc)) { + __count_vm_events(item, isolated); + __count_vm_events(PGREFILL, sorted); + } + __count_memcg_events(memcg, item, isolated); + __count_memcg_events(memcg, PGREFILL, sorted); + __count_vm_events(PGSCAN_ANON + type, isolated); + + /* + * There might not be eligible pages due to reclaim_idx, may_unmap and + * may_writepage. Check the remaining to prevent livelock if there is no + * progress. + */ + return isolated || !remaining ? scanned : 0; +} + +static int get_tier_idx(struct lruvec *lruvec, int type) +{ + int tier; + struct ctrl_pos sp, pv; + + /* + * To leave a margin for fluctuations, use a larger gain factor (1:2). + * This value is chosen because any other tier would have at least twice + * as many refaults as the first tier. + */ + read_ctrl_pos(lruvec, type, 0, 1, &sp); + for (tier =3D 1; tier < MAX_NR_TIERS; tier++) { + read_ctrl_pos(lruvec, type, tier, 2, &pv); + if (!positive_ctrl_err(&sp, &pv)) + break; + } + + return tier - 1; +} + +static int get_type_to_scan(struct lruvec *lruvec, int swappiness, int *ti= er_idx) +{ + int type, tier; + struct ctrl_pos sp, pv; + int gain[ANON_AND_FILE] =3D { swappiness, 200 - swappiness }; + + /* + * Compare the first tier of anon with that of file to determine which + * type to scan. Also need to compare other tiers of the selected type + * with the first tier of the other type to determine the last tier (of + * the selected type) to evict. + */ + read_ctrl_pos(lruvec, LRU_GEN_ANON, 0, gain[LRU_GEN_ANON], &sp); + read_ctrl_pos(lruvec, LRU_GEN_FILE, 0, gain[LRU_GEN_FILE], &pv); + type =3D positive_ctrl_err(&sp, &pv); + + read_ctrl_pos(lruvec, !type, 0, gain[!type], &sp); + for (tier =3D 1; tier < MAX_NR_TIERS; tier++) { + read_ctrl_pos(lruvec, type, tier, gain[type], &pv); + if (!positive_ctrl_err(&sp, &pv)) + break; + } + + *tier_idx =3D tier - 1; + + return type; +} + +static int isolate_folios(struct lruvec *lruvec, struct scan_control *sc, = int swappiness, + int *type_scanned, struct list_head *list) +{ + int i; + int type; + int scanned; + int tier =3D -1; + DEFINE_MIN_SEQ(lruvec); + + /* + * Try to make the obvious choice first. When anon and file are both + * available from the same generation, interpret swappiness 1 as file + * first and 200 as anon first. + */ + if (!swappiness) + type =3D LRU_GEN_FILE; + else if (min_seq[LRU_GEN_ANON] < min_seq[LRU_GEN_FILE]) + type =3D LRU_GEN_ANON; + else if (swappiness =3D=3D 1) + type =3D LRU_GEN_FILE; + else if (swappiness =3D=3D 200) + type =3D LRU_GEN_ANON; + else + type =3D get_type_to_scan(lruvec, swappiness, &tier); + + for (i =3D !swappiness; i < ANON_AND_FILE; i++) { + if (tier < 0) + tier =3D get_tier_idx(lruvec, type); + + scanned =3D scan_folios(lruvec, sc, type, tier, list); + if (scanned) + break; + + type =3D !type; + tier =3D -1; + } + + *type_scanned =3D type; + + return scanned; +} + +static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, in= t swappiness) +{ + int type; + int scanned; + int reclaimed; + LIST_HEAD(list); + struct folio *folio; + enum vm_event_item item; + struct reclaim_stat stat; + struct mem_cgroup *memcg =3D lruvec_memcg(lruvec); + struct pglist_data *pgdat =3D lruvec_pgdat(lruvec); + + spin_lock_irq(&lruvec->lru_lock); + + scanned =3D isolate_folios(lruvec, sc, swappiness, &type, &list); + + if (try_to_inc_min_seq(lruvec, swappiness)) + scanned++; + + if (get_nr_gens(lruvec, !swappiness) =3D=3D MIN_NR_GENS) + scanned =3D 0; + + spin_unlock_irq(&lruvec->lru_lock); + + if (list_empty(&list)) + return scanned; + + reclaimed =3D shrink_page_list(&list, pgdat, sc, &stat, false); + + /* + * To avoid livelock, don't add rejected pages back to the same lists + * they were isolated from. See lru_gen_add_folio(). + */ + list_for_each_entry(folio, &list, lru) { + folio_clear_referenced(folio); + folio_clear_workingset(folio); + + if (folio_test_reclaim(folio) && + (folio_test_dirty(folio) || folio_test_writeback(folio))) + folio_clear_active(folio); + else + folio_set_active(folio); + } + + spin_lock_irq(&lruvec->lru_lock); + + move_pages_to_lru(lruvec, &list); + + item =3D current_is_kswapd() ? PGSTEAL_KSWAPD : PGSTEAL_DIRECT; + if (!cgroup_reclaim(sc)) + __count_vm_events(item, reclaimed); + __count_memcg_events(memcg, item, reclaimed); + __count_vm_events(PGSTEAL_ANON + type, reclaimed); + + spin_unlock_irq(&lruvec->lru_lock); + + mem_cgroup_uncharge_list(&list); + free_unref_page_list(&list); + + sc->nr_reclaimed +=3D reclaimed; + + return scanned; +} + +static long get_nr_to_scan(struct lruvec *lruvec, struct scan_control *sc,= bool can_swap) +{ + bool need_aging; + long nr_to_scan; + struct mem_cgroup *memcg =3D lruvec_memcg(lruvec); + DEFINE_MAX_SEQ(lruvec); + DEFINE_MIN_SEQ(lruvec); + + if (mem_cgroup_below_min(memcg) || + (mem_cgroup_below_low(memcg) && !sc->memcg_low_reclaim)) + return 0; + + nr_to_scan =3D get_nr_evictable(lruvec, max_seq, min_seq, can_swap, &need= _aging); + if (!nr_to_scan) + return 0; + + /* reset the priority if the target has been met */ + nr_to_scan >>=3D sc->nr_reclaimed < sc->nr_to_reclaim ? sc->priority : DE= F_PRIORITY; + + if (!mem_cgroup_online(memcg)) + nr_to_scan++; + + if (!nr_to_scan) + return 0; + + if (!need_aging) + return nr_to_scan; + + /* leave the work to lru_gen_age_node() */ + if (current_is_kswapd()) + return 0; + + /* try other memcgs before going to the aging path */ + if (!cgroup_reclaim(sc) && !sc->force_deactivate) { + sc->skipped_deactivate =3D true; + return 0; + } + + inc_max_seq(lruvec, max_seq, can_swap); + + return nr_to_scan; +} + +static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_contr= ol *sc) +{ + struct blk_plug plug; + long scanned =3D 0; + + lru_add_drain(); + + blk_start_plug(&plug); + + while (true) { + int delta; + int swappiness; + long nr_to_scan; + + if (sc->may_swap) + swappiness =3D get_swappiness(lruvec, sc); + else if (!cgroup_reclaim(sc) && get_swappiness(lruvec, sc)) + swappiness =3D 1; + else + swappiness =3D 0; + + nr_to_scan =3D get_nr_to_scan(lruvec, sc, swappiness); + if (!nr_to_scan) + break; + + delta =3D evict_folios(lruvec, sc, swappiness); + if (!delta) + break; + + scanned +=3D delta; + if (scanned >=3D nr_to_scan) + break; + + cond_resched(); + } + + blk_finish_plug(&plug); +} + /*************************************************************************= ***** * initialization *************************************************************************= *****/ @@ -3041,6 +3804,16 @@ static int __init init_lru_gen(void) }; late_initcall(init_lru_gen); =20 +#else + +static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_contro= l *sc) +{ +} + +static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_contr= ol *sc) +{ +} + #endif /* CONFIG_LRU_GEN */ =20 static void shrink_lruvec(struct lruvec *lruvec, struct scan_control *sc) @@ -3054,6 +3827,11 @@ static void shrink_lruvec(struct lruvec *lruvec, str= uct scan_control *sc) struct blk_plug plug; bool scan_adjusted; =20 + if (lru_gen_enabled()) { + lru_gen_shrink_lruvec(lruvec, sc); + return; + } + get_scan_count(lruvec, sc, nr); =20 /* Record the original scan target for proportional adjustments later */ @@ -3558,6 +4336,9 @@ static void snapshot_refaults(struct mem_cgroup *targ= et_memcg, pg_data_t *pgdat) struct lruvec *target_lruvec; unsigned long refaults; =20 + if (lru_gen_enabled()) + return; + target_lruvec =3D mem_cgroup_lruvec(target_memcg, pgdat); refaults =3D lruvec_page_state(target_lruvec, WORKINGSET_ACTIVATE_ANON); target_lruvec->refaults[0] =3D refaults; @@ -3922,12 +4703,17 @@ unsigned long try_to_free_mem_cgroup_pages(struct m= em_cgroup *memcg, } #endif =20 -static void age_active_anon(struct pglist_data *pgdat, +static void kswapd_age_node(struct pglist_data *pgdat, struct scan_control *sc) { struct mem_cgroup *memcg; struct lruvec *lruvec; =20 + if (lru_gen_enabled()) { + lru_gen_age_node(pgdat, sc); + return; + } + if (!can_age_anon_pages(pgdat, sc)) return; =20 @@ -4247,12 +5033,11 @@ static int balance_pgdat(pg_data_t *pgdat, int orde= r, int highest_zoneidx) sc.may_swap =3D !nr_boost_reclaim; =20 /* - * Do some background aging of the anon list, to give - * pages a chance to be referenced before reclaiming. All - * pages are rotated regardless of classzone as this is - * about consistent aging. + * Do some background aging, to give pages a chance to be + * referenced before reclaiming. All pages are rotated + * regardless of classzone as this is about consistent aging. */ - age_active_anon(pgdat, &sc); + kswapd_age_node(pgdat, &sc); =20 /* * If we're getting trouble reclaiming, start doing writepage diff --git a/mm/workingset.c b/mm/workingset.c index 592569a8974c..db6f0c8a98c2 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -187,7 +187,6 @@ static unsigned int bucket_order __read_mostly; static void *pack_shadow(int memcgid, pg_data_t *pgdat, unsigned long evic= tion, bool workingset) { - eviction >>=3D bucket_order; eviction &=3D EVICTION_MASK; eviction =3D (eviction << MEM_CGROUP_ID_SHIFT) | memcgid; eviction =3D (eviction << NODES_SHIFT) | pgdat->node_id; @@ -212,10 +211,107 @@ static void unpack_shadow(void *shadow, int *memcgid= p, pg_data_t **pgdat, =20 *memcgidp =3D memcgid; *pgdat =3D NODE_DATA(nid); - *evictionp =3D entry << bucket_order; + *evictionp =3D entry; *workingsetp =3D workingset; } =20 +#ifdef CONFIG_LRU_GEN + +static void *lru_gen_eviction(struct folio *folio) +{ + int hist; + unsigned long token; + unsigned long min_seq; + struct lruvec *lruvec; + struct lru_gen_struct *lrugen; + int type =3D folio_is_file_lru(folio); + int delta =3D folio_nr_pages(folio); + int refs =3D folio_lru_refs(folio); + int tier =3D lru_tier_from_refs(refs); + struct mem_cgroup *memcg =3D folio_memcg(folio); + struct pglist_data *pgdat =3D folio_pgdat(folio); + + BUILD_BUG_ON(LRU_GEN_WIDTH + LRU_REFS_WIDTH > BITS_PER_LONG - EVICTION_SH= IFT); + + lruvec =3D mem_cgroup_lruvec(memcg, pgdat); + lrugen =3D &lruvec->lrugen; + min_seq =3D READ_ONCE(lrugen->min_seq[type]); + token =3D (min_seq << LRU_REFS_WIDTH) | max(refs - 1, 0); + + hist =3D lru_hist_from_seq(min_seq); + atomic_long_add(delta, &lrugen->evicted[hist][type][tier]); + + return pack_shadow(mem_cgroup_id(memcg), pgdat, token, refs); +} + +static void lru_gen_refault(struct folio *folio, void *shadow) +{ + int hist, tier, refs; + int memcg_id; + bool workingset; + unsigned long token; + unsigned long min_seq; + struct lruvec *lruvec; + struct lru_gen_struct *lrugen; + struct mem_cgroup *memcg; + struct pglist_data *pgdat; + int type =3D folio_is_file_lru(folio); + int delta =3D folio_nr_pages(folio); + + unpack_shadow(shadow, &memcg_id, &pgdat, &token, &workingset); + + if (folio_pgdat(folio) !=3D pgdat) + return; + + /* see the comment in folio_lru_refs() */ + refs =3D (token & (BIT(LRU_REFS_WIDTH) - 1)) + workingset; + tier =3D lru_tier_from_refs(refs); + + rcu_read_lock(); + memcg =3D folio_memcg_rcu(folio); + if (mem_cgroup_id(memcg) !=3D memcg_id) + goto unlock; + + lruvec =3D mem_cgroup_lruvec(memcg, pgdat); + lrugen =3D &lruvec->lrugen; + min_seq =3D READ_ONCE(lrugen->min_seq[type]); + + token >>=3D LRU_REFS_WIDTH; + if (token !=3D (min_seq & (EVICTION_MASK >> LRU_REFS_WIDTH))) + goto unlock; + + hist =3D lru_hist_from_seq(min_seq); + atomic_long_add(delta, &lrugen->refaulted[hist][type][tier]); + mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + type, delta); + + /* + * Count the following two cases as stalls: + * 1. For pages accessed through page tables, hotter pages pushed out + * hot pages which refaulted immediately. + * 2. For pages accessed through file descriptors, numbers of accesses + * might have been beyond the limit. + */ + if (lru_gen_in_fault() || refs =3D=3D BIT(LRU_REFS_WIDTH)) { + folio_set_workingset(folio); + mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + type, delta); + } +unlock: + rcu_read_unlock(); +} + +#else + +static void *lru_gen_eviction(struct folio *folio) +{ + return NULL; +} + +static void lru_gen_refault(struct folio *folio, void *shadow) +{ +} + +#endif /* CONFIG_LRU_GEN */ + /** * workingset_age_nonresident - age non-resident entries as LRU ages * @lruvec: the lruvec that was aged @@ -264,10 +360,14 @@ void *workingset_eviction(struct folio *folio, struct= mem_cgroup *target_memcg) VM_BUG_ON_FOLIO(folio_ref_count(folio), folio); VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); =20 + if (lru_gen_enabled()) + return lru_gen_eviction(folio); + lruvec =3D mem_cgroup_lruvec(target_memcg, pgdat); /* XXX: target_memcg can be NULL, go through lruvec */ memcgid =3D mem_cgroup_id(lruvec_memcg(lruvec)); eviction =3D atomic_long_read(&lruvec->nonresident_age); + eviction >>=3D bucket_order; workingset_age_nonresident(lruvec, folio_nr_pages(folio)); return pack_shadow(memcgid, pgdat, eviction, folio_test_workingset(folio)); @@ -298,7 +398,13 @@ void workingset_refault(struct folio *folio, void *sha= dow) int memcgid; long nr; =20 + if (lru_gen_enabled()) { + lru_gen_refault(folio, shadow); + return; + } + unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset); + eviction <<=3D bucket_order; =20 rcu_read_lock(); /* --=20 2.36.0.550.gb090851708-goog From nobody Thu May 7 21:41:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C79D2C433EF for ; Wed, 18 May 2022 01:47:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233887AbiERBru (ORCPT ); Tue, 17 May 2022 21:47:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57962 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233884AbiERBrT (ORCPT ); Tue, 17 May 2022 21:47:19 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3A29B54BCA for ; Tue, 17 May 2022 18:47:05 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id j2-20020a2597c2000000b0064b3e54191aso651575ybo.20 for ; Tue, 17 May 2022 18:47:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc:content-transfer-encoding; bh=DIXSaH9RJGkbTuDMTnXDRSvoEI7pCtEeA4wUNDyN3z8=; b=g0wxsGI7R45H2kF0pWERdLKkyLwtp+Yw0IOEBDlqZo/73VltFoNtGsnpTTiSArgRhy AUeYjOQRLucgcT1BOXjwa8iwyjc/FV4gVMQECeZmAm89brEUh/FlMMzDjbGmAz7sRZKS ZYK6UJRksyC2OA6yMzw37fB2ChJivBQPOwqyRwJa9RYiiTEVoLazjeC7EIFt9WUKkiYA /9UmBdWZAazbCzR+VjrjlnlcaeQghun0EDh1um9NtW846e6uJyKDChbVM31G3CYT6qRo Lbpaf47UJD7WZRJaMWGDTBm43aZ4pjSZ2+FY11zz3ohF53ByYbwB3XT+asa39cay+hYZ J/cA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc:content-transfer-encoding; bh=DIXSaH9RJGkbTuDMTnXDRSvoEI7pCtEeA4wUNDyN3z8=; b=NXwhzWyM1vx2BRWPwGSWLqSiBbvR72yuVJovnQbvVElrIVIvtH2YWlTq1QEEX2dYNx yaA0yiDQOArxkZFN0HR8OqaPd7aXWeGSC4TaUFUiqP/vBdY0MqqSvBKleibQ90ZPRkCd ebdk3WqDjZL65wrKYqXnPVdZBIIo46dVQo73Jn1wqau3LQkM7WaiyRrJBwHmxHFxnsGV V0Ov5R5HIsr3VhdBnmGi61DpMrOnMKeGZNx2kTmbhq9beCvWlwegGs3NLF5lv4xpgUWf XNkY9HyPDskl+yujkY8JQZEKqrGacyiVBCuAyphWJtV9wp8HRe32c0X4ZKrizcBCr0sv cnEA== X-Gm-Message-State: AOAM532G/yRt+QP5j00bBECEwvSO9pWhpMzHSmxbQ80C2WoChjWq8GTg iWG6ao1RjimqwtPj4D1b/pJPtC94xPI= X-Google-Smtp-Source: ABdhPJy5gTaQavr8nh6KkIBuBPD+lcWRRCsyNXYFpHXrgvnOQSN+Y4VhOwdpZBaG2OLswrB0zz7ABDWtGC4= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f7bc:90c9:d86e:4ea8]) (user=yuzhao job=sendgmr) by 2002:a81:d86:0:b0:2ff:2258:f9b3 with SMTP id 128-20020a810d86000000b002ff2258f9b3mr8281904ywn.355.1652838424351; Tue, 17 May 2022 18:47:04 -0700 (PDT) Date: Tue, 17 May 2022 19:46:26 -0600 In-Reply-To: <20220518014632.922072-1-yuzhao@google.com> Message-Id: <20220518014632.922072-8-yuzhao@google.com> Mime-Version: 1.0 References: <20220518014632.922072-1-yuzhao@google.com> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog Subject: [PATCH v11 07/14] mm: multi-gen LRU: exploit locality in rmap From: Yu Zhao To: Andrew Morton , linux-mm@kvack.org Cc: Andi Kleen , Aneesh Kumar , Catalin Marinas , Dave Hansen , Hillf Danton , Jens Axboe , Johannes Weiner , Jonathan Corbet , Linus Torvalds , Matthew Wilcox , Mel Gorman , Michael Larabel , Michal Hocko , Mike Rapoport , Peter Zijlstra , Tejun Heo , Vlastimil Babka , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, page-reclaim@google.com, Yu Zhao , Brian Geffon , Jan Alexander Steffens , Oleksandr Natalenko , Steven Barrett , Suleiman Souhlal , Daniel Byrne , Donald Carr , "=?UTF-8?q?Holger=20Hoffst=C3=A4tte?=" , Konstantin Kharlamov , Shuang Zhai , Sofia Trinh , Vaibhav Jain Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Searching the rmap for PTEs mapping each page on an LRU list (to test and clear the accessed bit) can be expensive because pages from different VMAs (PA space) are not cache friendly to the rmap (VA space). For workloads mostly using mapped pages, the rmap has a high CPU cost in the reclaim path. This patch exploits spatial locality to reduce the trips into the rmap. When shrink_page_list() walks the rmap and finds a young PTE, a new function lru_gen_look_around() scans at most BITS_PER_LONG-1 adjacent PTEs. On finding another young PTE, it clears the accessed bit and updates the gen counter of the page mapped by this PTE to (max_seq%MAX_NR_GENS)+1. Server benchmark results: Single workload: fio (buffered I/O): no change Single workload: memcached (anon): +[5.5, 7.5]% Ops/sec KB/sec patch1-6: 1120643.70 43588.06 patch1-7: 1193918.93 46438.15 Configurations: no change Client benchmark results: kswapd profiles: patch1-6 35.99% lzo1x_1_do_compress (real work) 19.40% page_vma_mapped_walk 6.31% _raw_spin_unlock_irq 3.95% do_raw_spin_lock 2.39% anon_vma_interval_tree_iter_first 2.25% ptep_clear_flush 1.92% __anon_vma_interval_tree_subtree_search 1.70% folio_referenced_one 1.68% __zram_bvec_write 1.43% anon_vma_interval_tree_iter_next patch1-7 45.90% lzo1x_1_do_compress (real work) 9.14% page_vma_mapped_walk 6.81% _raw_spin_unlock_irq 2.80% ptep_clear_flush 2.34% __zram_bvec_write 2.29% do_raw_spin_lock 1.84% lru_gen_look_around 1.78% memmove 1.74% obj_malloc 1.50% free_unref_page_list Configurations: no change Signed-off-by: Yu Zhao Acked-by: Brian Geffon Acked-by: Jan Alexander Steffens (heftig) Acked-by: Oleksandr Natalenko Acked-by: Steven Barrett Acked-by: Suleiman Souhlal Tested-by: Daniel Byrne Tested-by: Donald Carr Tested-by: Holger Hoffst=C3=A4tte Tested-by: Konstantin Kharlamov Tested-by: Shuang Zhai Tested-by: Sofia Trinh Tested-by: Vaibhav Jain --- include/linux/memcontrol.h | 31 ++++++++ include/linux/mm.h | 5 ++ include/linux/mmzone.h | 6 ++ mm/internal.h | 1 + mm/memcontrol.c | 1 + mm/rmap.c | 7 ++ mm/swap.c | 4 +- mm/vmscan.c | 157 +++++++++++++++++++++++++++++++++++++ 8 files changed, 210 insertions(+), 2 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 89b14729d59f..2bfdcc77648a 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -438,6 +438,7 @@ static inline struct obj_cgroup *__folio_objcg(struct f= olio *folio) * - LRU isolation * - lock_page_memcg() * - exclusive reference + * - mem_cgroup_trylock_pages() * * For a kmem folio a caller should hold an rcu read lock to protect memcg * associated with a kmem folio from being released. @@ -499,6 +500,7 @@ static inline struct mem_cgroup *folio_memcg_rcu(struct= folio *folio) * - LRU isolation * - lock_page_memcg() * - exclusive reference + * - mem_cgroup_trylock_pages() * * For a kmem page a caller should hold an rcu read lock to protect memcg * associated with a kmem page from being released. @@ -948,6 +950,23 @@ void unlock_page_memcg(struct page *page); =20 void __mod_memcg_state(struct mem_cgroup *memcg, int idx, int val); =20 +/* try to stablize folio_memcg() for all the pages in a memcg */ +static inline bool mem_cgroup_trylock_pages(struct mem_cgroup *memcg) +{ + rcu_read_lock(); + + if (mem_cgroup_disabled() || !atomic_read(&memcg->moving_account)) + return true; + + rcu_read_unlock(); + return false; +} + +static inline void mem_cgroup_unlock_pages(void) +{ + rcu_read_unlock(); +} + /* idx can be of type enum memcg_stat_item or node_stat_item */ static inline void mod_memcg_state(struct mem_cgroup *memcg, int idx, int val) @@ -1386,6 +1405,18 @@ static inline void folio_memcg_unlock(struct folio *= folio) { } =20 +static inline bool mem_cgroup_trylock_pages(struct mem_cgroup *memcg) +{ + /* to match folio_memcg_rcu() */ + rcu_read_lock(); + return true; +} + +static inline void mem_cgroup_unlock_pages(void) +{ + rcu_read_unlock(); +} + static inline void mem_cgroup_handle_over_high(void) { } diff --git a/include/linux/mm.h b/include/linux/mm.h index 894c289c2c06..4e8ab4ad4473 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1523,6 +1523,11 @@ static inline unsigned long folio_pfn(struct folio *= folio) return page_to_pfn(&folio->page); } =20 +static inline struct folio *pfn_folio(unsigned long pfn) +{ + return page_folio(pfn_to_page(pfn)); +} + static inline atomic_t *folio_pincount_ptr(struct folio *folio) { return &folio_page(folio, 1)->compound_pincount; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 2d023d243e73..f0b980362186 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -374,6 +374,7 @@ enum lruvec_flags { #ifndef __GENERATING_BOUNDS_H =20 struct lruvec; +struct page_vma_mapped_walk; =20 #define LRU_GEN_MASK ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF) #define LRU_REFS_MASK ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF) @@ -429,6 +430,7 @@ struct lru_gen_struct { }; =20 void lru_gen_init_lruvec(struct lruvec *lruvec); +void lru_gen_look_around(struct page_vma_mapped_walk *pvmw); =20 #ifdef CONFIG_MEMCG void lru_gen_init_memcg(struct mem_cgroup *memcg); @@ -441,6 +443,10 @@ static inline void lru_gen_init_lruvec(struct lruvec *= lruvec) { } =20 +static inline void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) +{ +} + #ifdef CONFIG_MEMCG static inline void lru_gen_init_memcg(struct mem_cgroup *memcg) { diff --git a/mm/internal.h b/mm/internal.h index cf16280ce132..59d2422b647d 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -68,6 +68,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf); void folio_rotate_reclaimable(struct folio *folio); bool __folio_end_writeback(struct folio *folio); void deactivate_file_folio(struct folio *folio); +void folio_activate(struct folio *folio); =20 void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vm= a, unsigned long floor, unsigned long ceiling); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 2ee074f80e72..98aa720ac639 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2769,6 +2769,7 @@ static void commit_charge(struct folio *folio, struct= mem_cgroup *memcg) * - LRU isolation * - lock_page_memcg() * - exclusive reference + * - mem_cgroup_trylock_pages() */ folio->memcg_data =3D (unsigned long)memcg; } diff --git a/mm/rmap.c b/mm/rmap.c index fedb82371efe..7cb7ef29088a 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -73,6 +73,7 @@ #include #include #include +#include =20 #include =20 @@ -821,6 +822,12 @@ static bool folio_referenced_one(struct folio *folio, } =20 if (pvmw.pte) { + if (lru_gen_enabled() && pte_young(*pvmw.pte) && + !(vma->vm_flags & (VM_SEQ_READ | VM_RAND_READ))) { + lru_gen_look_around(&pvmw); + referenced++; + } + if (ptep_clear_flush_young_notify(vma, address, pvmw.pte)) { /* diff --git a/mm/swap.c b/mm/swap.c index a99d22308f28..0aa1d0b33d42 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -342,7 +342,7 @@ static bool need_activate_page_drain(int cpu) return pagevec_count(&per_cpu(lru_pvecs.activate_page, cpu)) !=3D 0; } =20 -static void folio_activate(struct folio *folio) +void folio_activate(struct folio *folio) { if (folio_test_lru(folio) && !folio_test_active(folio) && !folio_test_unevictable(folio)) { @@ -362,7 +362,7 @@ static inline void activate_page_drain(int cpu) { } =20 -static void folio_activate(struct folio *folio) +void folio_activate(struct folio *folio) { struct lruvec *lruvec; =20 diff --git a/mm/vmscan.c b/mm/vmscan.c index 891f0ab69b3a..cf89a28c3b0e 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1554,6 +1554,11 @@ static unsigned int shrink_page_list(struct list_hea= d *page_list, if (!sc->may_unmap && page_mapped(page)) goto keep_locked; =20 + /* folio_update_gen() tried to promote this page? */ + if (lru_gen_enabled() && !ignore_references && + page_mapped(page) && PageReferenced(page)) + goto keep_locked; + may_enter_fs =3D (sc->gfp_mask & __GFP_FS) || (PageSwapCache(page) && (sc->gfp_mask & __GFP_IO)); =20 @@ -3137,6 +3142,28 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, s= truct ctrl_pos *pv) * the aging *************************************************************************= *****/ =20 +static int folio_update_gen(struct folio *folio, int gen) +{ + unsigned long new_flags, old_flags =3D READ_ONCE(folio->flags); + + VM_WARN_ON_ONCE(gen >=3D MAX_NR_GENS); + VM_WARN_ON_ONCE(!rcu_read_lock_held()); + + do { + /* lru_gen_del_folio() has isolated this page? */ + if (!(old_flags & LRU_GEN_MASK)) { + /* for shrink_page_list() */ + new_flags =3D old_flags | BIT(PG_referenced); + continue; + } + + new_flags =3D old_flags & ~(LRU_GEN_MASK | LRU_REFS_MASK | LRU_REFS_FLAG= S); + new_flags |=3D (gen + 1UL) << LRU_GEN_PGOFF; + } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); + + return ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; +} + static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool = reclaiming) { int type =3D folio_is_file_lru(folio); @@ -3147,6 +3174,11 @@ static int folio_inc_gen(struct lruvec *lruvec, stru= ct folio *folio, bool reclai VM_WARN_ON_ONCE_FOLIO(!(old_flags & LRU_GEN_MASK), folio); =20 do { + new_gen =3D ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; + /* folio_update_gen() has promoted this page? */ + if (new_gen >=3D 0 && new_gen !=3D old_gen) + return new_gen; + new_gen =3D (old_gen + 1) % MAX_NR_GENS; =20 new_flags =3D old_flags & ~(LRU_GEN_MASK | LRU_REFS_MASK | LRU_REFS_FLAG= S); @@ -3365,6 +3397,125 @@ static void lru_gen_age_node(struct pglist_data *pg= dat, struct scan_control *sc) } while ((memcg =3D mem_cgroup_iter(NULL, memcg, NULL))); } =20 +/* + * This function exploits spatial locality when shrink_page_list() walks t= he + * rmap. It scans the adjacent PTEs of a young PTE and promotes hot pages. + */ +void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) +{ + int i; + pte_t *pte; + unsigned long start; + unsigned long end; + unsigned long addr; + unsigned long bitmap[BITS_TO_LONGS(MIN_LRU_BATCH)] =3D {}; + struct folio *folio =3D pfn_folio(pvmw->pfn); + struct mem_cgroup *memcg =3D folio_memcg(folio); + struct pglist_data *pgdat =3D folio_pgdat(folio); + struct lruvec *lruvec =3D mem_cgroup_lruvec(memcg, pgdat); + DEFINE_MAX_SEQ(lruvec); + int old_gen, new_gen =3D lru_gen_from_seq(max_seq); + + lockdep_assert_held(pvmw->ptl); + VM_WARN_ON_ONCE_FOLIO(folio_test_lru(folio), folio); + + if (spin_is_contended(pvmw->ptl)) + return; + + start =3D max(pvmw->address & PMD_MASK, pvmw->vma->vm_start); + end =3D pmd_addr_end(pvmw->address, pvmw->vma->vm_end); + + if (end - start > MIN_LRU_BATCH * PAGE_SIZE) { + if (pvmw->address - start < MIN_LRU_BATCH * PAGE_SIZE / 2) + end =3D start + MIN_LRU_BATCH * PAGE_SIZE; + else if (end - pvmw->address < MIN_LRU_BATCH * PAGE_SIZE / 2) + start =3D end - MIN_LRU_BATCH * PAGE_SIZE; + else { + start =3D pvmw->address - MIN_LRU_BATCH * PAGE_SIZE / 2; + end =3D pvmw->address + MIN_LRU_BATCH * PAGE_SIZE / 2; + } + } + + pte =3D pvmw->pte - (pvmw->address - start) / PAGE_SIZE; + + rcu_read_lock(); + arch_enter_lazy_mmu_mode(); + + for (i =3D 0, addr =3D start; addr !=3D end; i++, addr +=3D PAGE_SIZE) { + unsigned long pfn =3D pte_pfn(pte[i]); + + VM_WARN_ON_ONCE(addr < pvmw->vma->vm_start || addr >=3D pvmw->vma->vm_en= d); + + if (!pte_present(pte[i]) || is_zero_pfn(pfn)) + continue; + + if (WARN_ON_ONCE(pte_devmap(pte[i]) || pte_special(pte[i]))) + continue; + + if (!pte_young(pte[i])) + continue; + + VM_WARN_ON_ONCE(!pfn_valid(pfn)); + if (pfn < pgdat->node_start_pfn || pfn >=3D pgdat_end_pfn(pgdat)) + continue; + + folio =3D pfn_folio(pfn); + if (folio_nid(folio) !=3D pgdat->node_id) + continue; + + if (folio_memcg_rcu(folio) !=3D memcg) + continue; + + if (!ptep_test_and_clear_young(pvmw->vma, addr, pte + i)) + continue; + + if (pte_dirty(pte[i]) && !folio_test_dirty(folio) && + !(folio_test_anon(folio) && folio_test_swapbacked(folio) && + !folio_test_swapcache(folio))) + folio_mark_dirty(folio); + + old_gen =3D folio_lru_gen(folio); + if (old_gen < 0) + folio_set_referenced(folio); + else if (old_gen !=3D new_gen) + __set_bit(i, bitmap); + } + + arch_leave_lazy_mmu_mode(); + rcu_read_unlock(); + + if (bitmap_weight(bitmap, MIN_LRU_BATCH) < PAGEVEC_SIZE) { + for_each_set_bit(i, bitmap, MIN_LRU_BATCH) { + folio =3D pfn_folio(pte_pfn(pte[i])); + folio_activate(folio); + } + return; + } + + /* folio_update_gen() requires stable folio_memcg() */ + if (!mem_cgroup_trylock_pages(memcg)) + return; + + spin_lock_irq(&lruvec->lru_lock); + new_gen =3D lru_gen_from_seq(lruvec->lrugen.max_seq); + + for_each_set_bit(i, bitmap, MIN_LRU_BATCH) { + folio =3D pfn_folio(pte_pfn(pte[i])); + if (folio_memcg_rcu(folio) !=3D memcg) + continue; + + old_gen =3D folio_update_gen(folio, new_gen); + if (old_gen < 0 || old_gen =3D=3D new_gen) + continue; + + lru_gen_update_size(lruvec, folio, old_gen, new_gen); + } + + spin_unlock_irq(&lruvec->lru_lock); + + mem_cgroup_unlock_pages(); +} + /*************************************************************************= ***** * the eviction *************************************************************************= *****/ @@ -3401,6 +3552,12 @@ static bool sort_folio(struct lruvec *lruvec, struct= folio *folio, int tier_idx) return true; } =20 + /* promoted */ + if (gen !=3D lru_gen_from_seq(lrugen->min_seq[type])) { + list_move(&folio->lru, &lrugen->lists[gen][type][zone]); + return true; + } + /* protected */ if (tier > tier_idx) { int hist =3D lru_hist_from_seq(lrugen->min_seq[type]); --=20 2.36.0.550.gb090851708-goog From nobody Thu May 7 21:41:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 488FCC433F5 for ; Wed, 18 May 2022 01:48:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234038AbiERBs3 (ORCPT ); Tue, 17 May 2022 21:48:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58028 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233893AbiERBrT (ORCPT ); Tue, 17 May 2022 21:47:19 -0400 Received: from mail-io1-xd49.google.com (mail-io1-xd49.google.com [IPv6:2607:f8b0:4864:20::d49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E54D54BD4 for ; Tue, 17 May 2022 18:47:06 -0700 (PDT) Received: by mail-io1-xd49.google.com with SMTP id e10-20020a056602158a00b0065a9bede1c4so13539193iow.8 for ; Tue, 17 May 2022 18:47:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc:content-transfer-encoding; bh=PTjd11WATlmfbdtQhfK897D7nFONbou6umNurqc2JZY=; b=k7gwiy9FkEGpQ2cJPTc0CqF7ZhNg9eIRssfySwAeqT/QU/pZu+xcBXpGXhY16eEC+g /9Y5TExpblTU8rZxzrfxyJ0Pyp63osrSRKn7w1RqllWPKoCfe0jyvnQj750nWh4yCSbe sUmYDTC4lVmOF8FJ0kxkgu+JYIGx8+6CNwahInKDUuJAwLGt5ZfxrR4Vj4Sx9x8x/sMe 8TvShq4s8bgIUvnBrzYOcCAAtVrfihqujQxwPxxUkwhGmShdFsLP04seFDkfIXe96ojc otj7kBAULxcPC4x/i4YynKdyvxmVqOBb3oXTOb22VbGG2qfhystG6v09y15F+VCiEPxW qqOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc:content-transfer-encoding; bh=PTjd11WATlmfbdtQhfK897D7nFONbou6umNurqc2JZY=; b=uQJmSvnJxPo6vImF2aAswRXgvI4MgEyly5IEYZ8KsKGXsxP2fPtN5dz0HgodGaHaV7 I4WKG42hnvaUuro68pEkNOxUUlAPpxq71H0ldMGrdPzif0d4z30TH1GRGz8UwTgtW+1k jN11IVKImJy+SlzPuOZIm39UfIIxG8mnArlxRtwzctHvos+aF7umjSF4x+7UBL+wRjPh 4XWyNbcXTCwtUAzvgOKKePHN8thSIt6lIuzKgPH5EYI+nk22ZhUuUwiuiyzXXazv26Q1 1bs1puv+LCUMOmuNM9uLkizIa5CyoNn7h4wRW0uULyhswBlLkTKHFh4qQjjtuLTB+xR6 vxYQ== X-Gm-Message-State: AOAM532RiW25PNzhMkVDsO7T6i8sZ9C2ER94K5StZ/Zi1ZPQJBoSH4dM ko2vxiqK1+6AGoReoKjMWE49jh5uPrk= X-Google-Smtp-Source: ABdhPJwk7Q4yUGqMOHyNREY9VJxifZTmzumohbyC4Q4FMtnQ2dLXH+B7ANRAGX4eThrtnxmA9J5/iIetGTw= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f7bc:90c9:d86e:4ea8]) (user=yuzhao job=sendgmr) by 2002:a05:6602:2f06:b0:65a:cfe5:1778 with SMTP id q6-20020a0566022f0600b0065acfe51778mr11310389iow.192.1652838425868; Tue, 17 May 2022 18:47:05 -0700 (PDT) Date: Tue, 17 May 2022 19:46:27 -0600 In-Reply-To: <20220518014632.922072-1-yuzhao@google.com> Message-Id: <20220518014632.922072-9-yuzhao@google.com> Mime-Version: 1.0 References: <20220518014632.922072-1-yuzhao@google.com> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog Subject: [PATCH v11 08/14] mm: multi-gen LRU: support page table walks From: Yu Zhao To: Andrew Morton , linux-mm@kvack.org Cc: Andi Kleen , Aneesh Kumar , Catalin Marinas , Dave Hansen , Hillf Danton , Jens Axboe , Johannes Weiner , Jonathan Corbet , Linus Torvalds , Matthew Wilcox , Mel Gorman , Michael Larabel , Michal Hocko , Mike Rapoport , Peter Zijlstra , Tejun Heo , Vlastimil Babka , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, page-reclaim@google.com, Yu Zhao , Brian Geffon , Jan Alexander Steffens , Oleksandr Natalenko , Steven Barrett , Suleiman Souhlal , Daniel Byrne , Donald Carr , "=?UTF-8?q?Holger=20Hoffst=C3=A4tte?=" , Konstantin Kharlamov , Shuang Zhai , Sofia Trinh , Vaibhav Jain Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" To further exploit spatial locality, the aging prefers to walk page tables to search for young PTEs and promote hot pages. A kill switch will be added in the next patch to disable this behavior. When disabled, the aging relies on the rmap only. NB: this behavior has nothing similar with the page table scanning in the 2.4 kernel [1], which searches page tables for old PTEs, adds cold pages to swapcache and unmaps them. To avoid confusion, the term "iteration" specifically means the traversal of an entire mm_struct list; the term "walk" will be applied to page tables and the rmap, as usual. An mm_struct list is maintained for each memcg, and an mm_struct follows its owner task to the new memcg when this task is migrated. Given an lruvec, the aging iterates lruvec_memcg()->mm_list and calls walk_page_range() with each mm_struct on this list to promote hot pages before it increments max_seq. When multiple page table walkers iterate the same list, each of them gets a unique mm_struct; therefore they can run concurrently. Page table walkers ignore any misplaced pages, e.g., if an mm_struct was migrated, pages it left in the previous memcg will not be promoted when its current memcg is under reclaim. Similarly, page table walkers will not promote pages from nodes other than the one under reclaim. This patch uses the following optimizations when walking page tables: 1. It tracks the usage of mm_struct's between context switches so that page table walkers can skip processes that have been sleeping since the last iteration. 2. It uses generational Bloom filters to record populated branches so that page table walkers can reduce their search space based on the query results, e.g., to skip page tables containing mostly holes or misplaced pages. 3. It takes advantage of the accessed bit in non-leaf PMD entries when CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG=3Dy. 4. It does not zigzag between a PGD table and the same PMD table spanning multiple VMAs. IOW, it finishes all the VMAs within the range of the same PMD table before it returns to a PGD table. This improves the cache performance for workloads that have large numbers of tiny VMAs [2], especially when CONFIG_PGTABLE_LEVELS=3D5. Server benchmark results: Single workload: fio (buffered I/O): no change Single workload: memcached (anon): +[8, 10]% Ops/sec KB/sec patch1-7: 1193918.93 46438.15 patch1-8: 1301954.44 50640.27 Configurations: no change Client benchmark results: kswapd profiles: patch1-7 45.90% lzo1x_1_do_compress (real work) 9.14% page_vma_mapped_walk 6.81% _raw_spin_unlock_irq 2.80% ptep_clear_flush 2.34% __zram_bvec_write 2.29% do_raw_spin_lock 1.84% lru_gen_look_around 1.78% memmove 1.74% obj_malloc 1.50% free_unref_page_list patch1-8 46.96% lzo1x_1_do_compress (real work) 7.55% page_vma_mapped_walk 5.89% _raw_spin_unlock_irq 3.33% walk_pte_range 2.65% ptep_clear_flush 2.23% __zram_bvec_write 2.08% do_raw_spin_lock 1.83% memmove 1.65% obj_malloc 1.47% free_unref_page_list Configurations: no change Thanks to the following developers for their efforts [3]. kernel test robot [1] https://lwn.net/Articles/23732/ [2] https://source.android.com/devices/tech/debug/scudo [3] https://lore.kernel.org/r/202204160827.ekEARWQo-lkp@intel.com/ Signed-off-by: Yu Zhao Acked-by: Brian Geffon Acked-by: Jan Alexander Steffens (heftig) Acked-by: Oleksandr Natalenko Acked-by: Steven Barrett Acked-by: Suleiman Souhlal Tested-by: Daniel Byrne Tested-by: Donald Carr Tested-by: Holger Hoffst=C3=A4tte Tested-by: Konstantin Kharlamov Tested-by: Shuang Zhai Tested-by: Sofia Trinh Tested-by: Vaibhav Jain --- fs/exec.c | 2 + include/linux/memcontrol.h | 5 + include/linux/mm_types.h | 77 +++ include/linux/mmzone.h | 61 ++- include/linux/swap.h | 4 + kernel/exit.c | 1 + kernel/fork.c | 9 + kernel/sched/core.c | 1 + mm/memcontrol.c | 25 + mm/vmscan.c | 981 ++++++++++++++++++++++++++++++++++++- 10 files changed, 1150 insertions(+), 16 deletions(-) diff --git a/fs/exec.c b/fs/exec.c index e3e55d5e0be1..bba8fc44926f 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -1011,6 +1011,7 @@ static int exec_mmap(struct mm_struct *mm) active_mm =3D tsk->active_mm; tsk->active_mm =3D mm; tsk->mm =3D mm; + lru_gen_add_mm(mm); /* * This prevents preemption while active_mm is being loaded and * it and mm are being updated, which could cause problems for @@ -1023,6 +1024,7 @@ static int exec_mmap(struct mm_struct *mm) activate_mm(active_mm, mm); if (IS_ENABLED(CONFIG_ARCH_WANT_IRQS_OFF_ACTIVATE_MM)) local_irq_enable(); + lru_gen_use_mm(mm); tsk->mm->vmacache_seqnum =3D 0; vmacache_flush(tsk); task_unlock(tsk); diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 2bfdcc77648a..cc16ba262464 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -344,6 +344,11 @@ struct mem_cgroup { struct deferred_split deferred_split_queue; #endif =20 +#ifdef CONFIG_LRU_GEN + /* per-memcg mm_struct list */ + struct lru_gen_mm_list mm_list; +#endif + struct mem_cgroup_per_node *nodeinfo[]; }; =20 diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 8834e38c06a4..ef46d5b3da9b 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -3,6 +3,7 @@ #define _LINUX_MM_TYPES_H =20 #include +#include =20 #include #include @@ -17,6 +18,7 @@ #include #include #include +#include =20 #include =20 @@ -655,6 +657,22 @@ struct mm_struct { #ifdef CONFIG_IOMMU_SVA u32 pasid; #endif +#ifdef CONFIG_LRU_GEN + struct { + /* this mm_struct is on lru_gen_mm_list */ + struct list_head list; +#ifdef CONFIG_MEMCG + /* points to the memcg of "owner" above */ + struct mem_cgroup *memcg; +#endif + /* + * Set when switching to this mm_struct, as a hint of + * whether it has been used since the last time per-node + * page table walkers cleared the corresponding bits. + */ + unsigned long bitmap; + } lru_gen; +#endif /* CONFIG_LRU_GEN */ } __randomize_layout; =20 /* @@ -681,6 +699,65 @@ static inline cpumask_t *mm_cpumask(struct mm_struct *= mm) return (struct cpumask *)&mm->cpu_bitmap; } =20 +#ifdef CONFIG_LRU_GEN + +struct lru_gen_mm_list { + /* mm_struct list for page table walkers */ + struct list_head fifo; + /* protects the list above */ + spinlock_t lock; +}; + +void lru_gen_add_mm(struct mm_struct *mm); +void lru_gen_del_mm(struct mm_struct *mm); +#ifdef CONFIG_MEMCG +void lru_gen_migrate_mm(struct mm_struct *mm); +#endif + +static inline void lru_gen_init_mm(struct mm_struct *mm) +{ + INIT_LIST_HEAD(&mm->lru_gen.list); +#ifdef CONFIG_MEMCG + mm->lru_gen.memcg =3D NULL; +#endif + mm->lru_gen.bitmap =3D 0; +} + +static inline void lru_gen_use_mm(struct mm_struct *mm) +{ + /* unlikely but not a bug when racing with lru_gen_migrate_mm() */ + VM_WARN_ON_ONCE(list_empty(&mm->lru_gen.list)); + + if (!(current->flags & PF_KTHREAD)) + WRITE_ONCE(mm->lru_gen.bitmap, -1); +} + +#else /* !CONFIG_LRU_GEN */ + +static inline void lru_gen_add_mm(struct mm_struct *mm) +{ +} + +static inline void lru_gen_del_mm(struct mm_struct *mm) +{ +} + +#ifdef CONFIG_MEMCG +static inline void lru_gen_migrate_mm(struct mm_struct *mm) +{ +} +#endif + +static inline void lru_gen_init_mm(struct mm_struct *mm) +{ +} + +static inline void lru_gen_use_mm(struct mm_struct *mm) +{ +} + +#endif /* CONFIG_LRU_GEN */ + struct mmu_gather; extern void tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm); extern void tlb_gather_mmu_fullmm(struct mmu_gather *tlb, struct mm_struct= *mm); diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index f0b980362186..6aa715a2fea1 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -407,7 +407,7 @@ enum { * min_seq behind. * * nr_pages[] are eventually consistent and therefore can be transiently - * negative. + * negative when reset_batch_size() is pending. */ struct lru_gen_struct { /* the aging increments the youngest generation number */ @@ -429,6 +429,58 @@ struct lru_gen_struct { atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS]; }; =20 +enum { + MM_PTE_TOTAL, /* total leaf entries */ + MM_PTE_OLD, /* old leaf entries */ + MM_PTE_YOUNG, /* young leaf entries */ + MM_PMD_TOTAL, /* total non-leaf entries */ + MM_PMD_FOUND, /* non-leaf entries found in Bloom filters */ + MM_PMD_ADDED, /* non-leaf entries added to Bloom filters */ + NR_MM_STATS +}; + +/* mnemonic codes for the mm stats above */ +#define MM_STAT_CODES "toydfa" + +/* double-buffering Bloom filters */ +#define NR_BLOOM_FILTERS 2 + +struct lru_gen_mm_state { + /* set to max_seq after each iteration */ + unsigned long seq; + /* where the current iteration starts (inclusive) */ + struct list_head *head; + /* where the last iteration ends (exclusive) */ + struct list_head *tail; + /* to wait for the last page table walker to finish */ + struct wait_queue_head wait; + /* Bloom filters flip after each iteration */ + unsigned long *filters[NR_BLOOM_FILTERS]; + /* the mm stats for debugging */ + unsigned long stats[NR_HIST_GENS][NR_MM_STATS]; + /* the number of concurrent page table walkers */ + int nr_walkers; +}; + +struct lru_gen_mm_walk { + /* the lruvec under reclaim */ + struct lruvec *lruvec; + /* unstable max_seq from lru_gen_struct */ + unsigned long max_seq; + /* the next address within an mm to scan */ + unsigned long next_addr; + /* to batch page table entries */ + unsigned long bitmap[BITS_TO_LONGS(MIN_LRU_BATCH)]; + /* to batch promoted pages */ + int nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; + /* to batch the mm stats */ + int mm_stats[NR_MM_STATS]; + /* total batched items */ + int batched; + bool can_swap; + bool force_scan; +}; + void lru_gen_init_lruvec(struct lruvec *lruvec); void lru_gen_look_around(struct page_vma_mapped_walk *pvmw); =20 @@ -479,6 +531,8 @@ struct lruvec { #ifdef CONFIG_LRU_GEN /* evictable pages divided into generations */ struct lru_gen_struct lrugen; + /* to concurrently iterate lru_gen_mm_list */ + struct lru_gen_mm_state mm_state; #endif #ifdef CONFIG_MEMCG struct pglist_data *pgdat; @@ -1072,6 +1126,11 @@ typedef struct pglist_data { =20 unsigned long flags; =20 +#ifdef CONFIG_LRU_GEN + /* kswap mm walk data */ + struct lru_gen_mm_walk mm_walk; +#endif + ZONE_PADDING(_pad2_) =20 /* Per-node vmstats */ diff --git a/include/linux/swap.h b/include/linux/swap.h index 27093b477c5f..7bdd7bcb135d 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -137,6 +137,10 @@ union swap_header { */ struct reclaim_state { unsigned long reclaimed_slab; +#ifdef CONFIG_LRU_GEN + /* per-thread mm walk data */ + struct lru_gen_mm_walk *mm_walk; +#endif }; =20 #ifdef __KERNEL__ diff --git a/kernel/exit.c b/kernel/exit.c index f072959fcab7..f2d4d48ea790 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -466,6 +466,7 @@ void mm_update_next_owner(struct mm_struct *mm) goto retry; } WRITE_ONCE(mm->owner, c); + lru_gen_migrate_mm(mm); task_unlock(c); put_task_struct(c); } diff --git a/kernel/fork.c b/kernel/fork.c index 35a3beff140b..3e3cab26e66c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1149,6 +1149,7 @@ static struct mm_struct *mm_init(struct mm_struct *mm= , struct task_struct *p, goto fail_nocontext; =20 mm->user_ns =3D get_user_ns(user_ns); + lru_gen_init_mm(mm); return mm; =20 fail_nocontext: @@ -1191,6 +1192,7 @@ static inline void __mmput(struct mm_struct *mm) } if (mm->binfmt) module_put(mm->binfmt->module); + lru_gen_del_mm(mm); mmdrop(mm); } =20 @@ -2660,6 +2662,13 @@ pid_t kernel_clone(struct kernel_clone_args *args) get_task_struct(p); } =20 + if (IS_ENABLED(CONFIG_LRU_GEN) && !(clone_flags & CLONE_VM)) { + /* lock the task to synchronize with memcg migration */ + task_lock(p); + lru_gen_add_mm(p->mm); + task_unlock(p); + } + wake_up_new_task(p); =20 /* forking complete and child started to run, tell ptracer */ diff --git a/kernel/sched/core.c b/kernel/sched/core.c index d58c0389eb23..a8f9e5ebb5dd 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -5057,6 +5057,7 @@ context_switch(struct rq *rq, struct task_struct *pre= v, * finish_task_switch()'s mmdrop(). */ switch_mm_irqs_off(prev->active_mm, next->mm, next); + lru_gen_use_mm(next->mm); =20 if (!prev->mm) { // from kernel /* will mmdrop() in finish_task_switch(). */ diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 98aa720ac639..ff2b0a09e6c8 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6093,6 +6093,30 @@ static void mem_cgroup_move_task(void) } #endif =20 +#ifdef CONFIG_LRU_GEN +static void mem_cgroup_attach(struct cgroup_taskset *tset) +{ + struct task_struct *task; + struct cgroup_subsys_state *css; + + /* find the first leader if there is any */ + cgroup_taskset_for_each_leader(task, css, tset) + break; + + if (!task) + return; + + task_lock(task); + if (task->mm && task->mm->owner =3D=3D task) + lru_gen_migrate_mm(task->mm); + task_unlock(task); +} +#else +static void mem_cgroup_attach(struct cgroup_taskset *tset) +{ +} +#endif /* CONFIG_LRU_GEN */ + static int seq_puts_memcg_tunable(struct seq_file *m, unsigned long value) { if (value =3D=3D PAGE_COUNTER_MAX) @@ -6438,6 +6462,7 @@ struct cgroup_subsys memory_cgrp_subsys =3D { .css_reset =3D mem_cgroup_css_reset, .css_rstat_flush =3D mem_cgroup_css_rstat_flush, .can_attach =3D mem_cgroup_can_attach, + .attach =3D mem_cgroup_attach, .cancel_attach =3D mem_cgroup_cancel_attach, .post_attach =3D mem_cgroup_move_task, .dfl_cftypes =3D memory_files, diff --git a/mm/vmscan.c b/mm/vmscan.c index cf89a28c3b0e..f292e7e761b1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -50,6 +50,8 @@ #include #include #include +#include +#include =20 #include #include @@ -3000,7 +3002,7 @@ static bool can_age_anon_pages(struct pglist_data *pg= dat, for ((type) =3D 0; (type) < ANON_AND_FILE; (type)++) \ for ((zone) =3D 0; (zone) < MAX_NR_ZONES; (zone)++) =20 -static struct lruvec __maybe_unused *get_lruvec(struct mem_cgroup *memcg, = int nid) +static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) { struct pglist_data *pgdat =3D NODE_DATA(nid); =20 @@ -3045,6 +3047,375 @@ static bool __maybe_unused seq_is_valid(struct lruv= ec *lruvec) get_nr_gens(lruvec, LRU_GEN_ANON) <=3D MAX_NR_GENS; } =20 +/*************************************************************************= ***** + * mm_struct list + *************************************************************************= *****/ + +static struct lru_gen_mm_list *get_mm_list(struct mem_cgroup *memcg) +{ + static struct lru_gen_mm_list mm_list =3D { + .fifo =3D LIST_HEAD_INIT(mm_list.fifo), + .lock =3D __SPIN_LOCK_UNLOCKED(mm_list.lock), + }; + +#ifdef CONFIG_MEMCG + if (memcg) + return &memcg->mm_list; +#endif + VM_WARN_ON_ONCE(!mem_cgroup_disabled()); + + return &mm_list; +} + +void lru_gen_add_mm(struct mm_struct *mm) +{ + int nid; + struct mem_cgroup *memcg =3D get_mem_cgroup_from_mm(mm); + struct lru_gen_mm_list *mm_list =3D get_mm_list(memcg); + + VM_WARN_ON_ONCE(!list_empty(&mm->lru_gen.list)); +#ifdef CONFIG_MEMCG + VM_WARN_ON_ONCE(mm->lru_gen.memcg); + mm->lru_gen.memcg =3D memcg; +#endif + spin_lock(&mm_list->lock); + + for_each_node_state(nid, N_MEMORY) { + struct lruvec *lruvec =3D get_lruvec(memcg, nid); + + if (!lruvec) + continue; + + if (lruvec->mm_state.tail =3D=3D &mm_list->fifo) + lruvec->mm_state.tail =3D &mm->lru_gen.list; + } + + list_add_tail(&mm->lru_gen.list, &mm_list->fifo); + + spin_unlock(&mm_list->lock); +} + +void lru_gen_del_mm(struct mm_struct *mm) +{ + int nid; + struct lru_gen_mm_list *mm_list; + struct mem_cgroup *memcg =3D NULL; + + if (list_empty(&mm->lru_gen.list)) + return; + +#ifdef CONFIG_MEMCG + memcg =3D mm->lru_gen.memcg; +#endif + mm_list =3D get_mm_list(memcg); + + spin_lock(&mm_list->lock); + + for_each_node(nid) { + struct lruvec *lruvec =3D get_lruvec(memcg, nid); + + if (!lruvec) + continue; + + if (lruvec->mm_state.tail =3D=3D &mm->lru_gen.list) + lruvec->mm_state.tail =3D lruvec->mm_state.tail->next; + + if (lruvec->mm_state.head !=3D &mm->lru_gen.list) + continue; + + lruvec->mm_state.head =3D lruvec->mm_state.head->next; + if (lruvec->mm_state.head =3D=3D &mm_list->fifo) + WRITE_ONCE(lruvec->mm_state.seq, lruvec->mm_state.seq + 1); + } + + list_del_init(&mm->lru_gen.list); + + spin_unlock(&mm_list->lock); + +#ifdef CONFIG_MEMCG + mem_cgroup_put(mm->lru_gen.memcg); + mm->lru_gen.memcg =3D NULL; +#endif +} + +#ifdef CONFIG_MEMCG +void lru_gen_migrate_mm(struct mm_struct *mm) +{ + struct mem_cgroup *memcg; + + lockdep_assert_held(&mm->owner->alloc_lock); + + /* for mm_update_next_owner() */ + if (mem_cgroup_disabled()) + return; + + rcu_read_lock(); + memcg =3D mem_cgroup_from_task(rcu_dereference(mm->owner)); + rcu_read_unlock(); + if (memcg =3D=3D mm->lru_gen.memcg) + return; + + VM_WARN_ON_ONCE(!mm->lru_gen.memcg); + VM_WARN_ON_ONCE(list_empty(&mm->lru_gen.list)); + + lru_gen_del_mm(mm); + lru_gen_add_mm(mm); +} +#endif + +/* + * Bloom filters with m=3D1<<15, k=3D2 and the false positive rates of ~1/= 5 when + * n=3D10,000 and ~1/2 when n=3D20,000, where, conventionally, m is the nu= mber of + * bits in a bitmap, k is the number of hash functions and n is the number= of + * inserted items. + * + * Page table walkers use one of the two filters to reduce their search sp= ace. + * To get rid of non-leaf entries that no longer have enough leaf entries,= the + * aging uses the double-buffering technique to flip to the other filter e= ach + * time it produces a new generation. For non-leaf entries that have enough + * leaf entries, the aging carries them over to the next generation in + * walk_pmd_range(); the eviction also report them when walking the rmap + * in lru_gen_look_around(). + * + * For future optimizations: + * 1. It's not necessary to keep both filters all the time. The spare one = can be + * freed after the RCU grace period and reallocated if needed again. + * 2. And when reallocating, it's worth scaling its size according to the = number + * of inserted entries in the other filter, to reduce the memory overhe= ad on + * small systems and false positives on large systems. + * 3. Jenkins' hash function is an alternative to Knuth's. + */ +#define BLOOM_FILTER_SHIFT 15 + +static inline int filter_gen_from_seq(unsigned long seq) +{ + return seq % NR_BLOOM_FILTERS; +} + +static void get_item_key(void *item, int *key) +{ + u32 hash =3D hash_ptr(item, BLOOM_FILTER_SHIFT * 2); + + BUILD_BUG_ON(BLOOM_FILTER_SHIFT * 2 > BITS_PER_TYPE(u32)); + + key[0] =3D hash & (BIT(BLOOM_FILTER_SHIFT) - 1); + key[1] =3D hash >> BLOOM_FILTER_SHIFT; +} + +static void reset_bloom_filter(struct lruvec *lruvec, unsigned long seq) +{ + unsigned long *filter; + int gen =3D filter_gen_from_seq(seq); + + lockdep_assert_held(&get_mm_list(lruvec_memcg(lruvec))->lock); + + filter =3D lruvec->mm_state.filters[gen]; + if (filter) { + bitmap_clear(filter, 0, BIT(BLOOM_FILTER_SHIFT)); + return; + } + + filter =3D bitmap_zalloc(BIT(BLOOM_FILTER_SHIFT), GFP_ATOMIC); + WRITE_ONCE(lruvec->mm_state.filters[gen], filter); +} + +static void update_bloom_filter(struct lruvec *lruvec, unsigned long seq, = void *item) +{ + int key[2]; + unsigned long *filter; + int gen =3D filter_gen_from_seq(seq); + + filter =3D READ_ONCE(lruvec->mm_state.filters[gen]); + if (!filter) + return; + + get_item_key(item, key); + + if (!test_bit(key[0], filter)) + set_bit(key[0], filter); + if (!test_bit(key[1], filter)) + set_bit(key[1], filter); +} + +static bool test_bloom_filter(struct lruvec *lruvec, unsigned long seq, vo= id *item) +{ + int key[2]; + unsigned long *filter; + int gen =3D filter_gen_from_seq(seq); + + filter =3D READ_ONCE(lruvec->mm_state.filters[gen]); + if (!filter) + return true; + + get_item_key(item, key); + + return test_bit(key[0], filter) && test_bit(key[1], filter); +} + +static void reset_mm_stats(struct lruvec *lruvec, struct lru_gen_mm_walk *= walk, bool last) +{ + int i; + int hist; + + lockdep_assert_held(&get_mm_list(lruvec_memcg(lruvec))->lock); + + if (walk) { + hist =3D lru_hist_from_seq(walk->max_seq); + + for (i =3D 0; i < NR_MM_STATS; i++) { + WRITE_ONCE(lruvec->mm_state.stats[hist][i], + lruvec->mm_state.stats[hist][i] + walk->mm_stats[i]); + walk->mm_stats[i] =3D 0; + } + } + + if (NR_HIST_GENS > 1 && last) { + hist =3D lru_hist_from_seq(lruvec->mm_state.seq + 1); + + for (i =3D 0; i < NR_MM_STATS; i++) + WRITE_ONCE(lruvec->mm_state.stats[hist][i], 0); + } +} + +static bool should_skip_mm(struct mm_struct *mm, struct lru_gen_mm_walk *w= alk) +{ + int type; + unsigned long size =3D 0; + struct pglist_data *pgdat =3D lruvec_pgdat(walk->lruvec); + int key =3D pgdat->node_id % BITS_PER_TYPE(mm->lru_gen.bitmap); + + if (!walk->force_scan && cpumask_empty(mm_cpumask(mm)) && + !test_bit(key, &mm->lru_gen.bitmap)) + return true; + + clear_bit(key, &mm->lru_gen.bitmap); + + for (type =3D !walk->can_swap; type < ANON_AND_FILE; type++) { + size +=3D type ? get_mm_counter(mm, MM_FILEPAGES) : + get_mm_counter(mm, MM_ANONPAGES) + + get_mm_counter(mm, MM_SHMEMPAGES); + } + + if (size < MIN_LRU_BATCH) + return true; + + if (mm_is_oom_victim(mm)) + return true; + + return !mmget_not_zero(mm); +} + +static bool iterate_mm_list(struct lruvec *lruvec, struct lru_gen_mm_walk = *walk, + struct mm_struct **iter) +{ + bool first =3D false; + bool last =3D true; + struct mm_struct *mm =3D NULL; + struct mem_cgroup *memcg =3D lruvec_memcg(lruvec); + struct lru_gen_mm_list *mm_list =3D get_mm_list(memcg); + struct lru_gen_mm_state *mm_state =3D &lruvec->mm_state; + + /* + * There are four interesting cases for this page table walker: + * 1. It tries to start a new iteration of mm_list with a stale max_seq; + * there is nothing to be done. + * 2. It's the first of the current generation, and it needs to reset + * the Bloom filter for the next generation. + * 3. It reaches the end of mm_list, and it needs to increment + * mm_state->seq; the iteration is done. + * 4. It's the last of the current generation, and it needs to reset the + * mm stats counters for the next generation. + */ + if (*iter) + mmput_async(*iter); + else if (walk->max_seq <=3D READ_ONCE(mm_state->seq)) + return false; + + spin_lock(&mm_list->lock); + + VM_WARN_ON_ONCE(mm_state->seq + 1 < walk->max_seq); + VM_WARN_ON_ONCE(*iter && mm_state->seq > walk->max_seq); + VM_WARN_ON_ONCE(*iter && !mm_state->nr_walkers); + + if (walk->max_seq <=3D mm_state->seq) { + if (!*iter) + last =3D false; + goto done; + } + + if (!mm_state->nr_walkers) { + VM_WARN_ON_ONCE(mm_state->head && mm_state->head !=3D &mm_list->fifo); + + mm_state->head =3D mm_list->fifo.next; + first =3D true; + } + + while (!mm && mm_state->head !=3D &mm_list->fifo) { + mm =3D list_entry(mm_state->head, struct mm_struct, lru_gen.list); + + mm_state->head =3D mm_state->head->next; + + /* force scan for those added after the last iteration */ + if (!mm_state->tail || mm_state->tail =3D=3D &mm->lru_gen.list) { + mm_state->tail =3D mm_state->head; + walk->force_scan =3D true; + } + + if (should_skip_mm(mm, walk)) + mm =3D NULL; + } + + if (mm_state->head =3D=3D &mm_list->fifo) + WRITE_ONCE(mm_state->seq, mm_state->seq + 1); +done: + if (*iter && !mm) + mm_state->nr_walkers--; + if (!*iter && mm) + mm_state->nr_walkers++; + + if (mm_state->nr_walkers) + last =3D false; + + if (mm && first) + reset_bloom_filter(lruvec, walk->max_seq + 1); + + if (*iter || last) + reset_mm_stats(lruvec, walk, last); + + spin_unlock(&mm_list->lock); + + *iter =3D mm; + + return last; +} + +static bool iterate_mm_list_nowalk(struct lruvec *lruvec, unsigned long ma= x_seq) +{ + bool success =3D false; + struct mem_cgroup *memcg =3D lruvec_memcg(lruvec); + struct lru_gen_mm_list *mm_list =3D get_mm_list(memcg); + struct lru_gen_mm_state *mm_state =3D &lruvec->mm_state; + + if (max_seq <=3D READ_ONCE(mm_state->seq)) + return false; + + spin_lock(&mm_list->lock); + + VM_WARN_ON_ONCE(mm_state->seq + 1 < max_seq); + + if (max_seq > mm_state->seq && !mm_state->nr_walkers) { + VM_WARN_ON_ONCE(mm_state->head && mm_state->head !=3D &mm_list->fifo); + + WRITE_ONCE(mm_state->seq, mm_state->seq + 1); + reset_mm_stats(lruvec, NULL, true); + success =3D true; + } + + spin_unlock(&mm_list->lock); + + return success; +} + /*************************************************************************= ***** * refault feedback loop *************************************************************************= *****/ @@ -3193,6 +3564,479 @@ static int folio_inc_gen(struct lruvec *lruvec, str= uct folio *folio, bool reclai return new_gen; } =20 +static void update_batch_size(struct lru_gen_mm_walk *walk, struct folio *= folio, + int old_gen, int new_gen) +{ + int type =3D folio_is_file_lru(folio); + int zone =3D folio_zonenum(folio); + int delta =3D folio_nr_pages(folio); + + VM_WARN_ON_ONCE(old_gen >=3D MAX_NR_GENS); + VM_WARN_ON_ONCE(new_gen >=3D MAX_NR_GENS); + + walk->batched++; + + walk->nr_pages[old_gen][type][zone] -=3D delta; + walk->nr_pages[new_gen][type][zone] +=3D delta; +} + +static void reset_batch_size(struct lruvec *lruvec, struct lru_gen_mm_walk= *walk) +{ + int gen, type, zone; + struct lru_gen_struct *lrugen =3D &lruvec->lrugen; + + walk->batched =3D 0; + + for_each_gen_type_zone(gen, type, zone) { + enum lru_list lru =3D type * LRU_INACTIVE_FILE; + int delta =3D walk->nr_pages[gen][type][zone]; + + if (!delta) + continue; + + walk->nr_pages[gen][type][zone] =3D 0; + WRITE_ONCE(lrugen->nr_pages[gen][type][zone], + lrugen->nr_pages[gen][type][zone] + delta); + + if (lru_gen_is_active(lruvec, gen)) + lru +=3D LRU_ACTIVE; + __update_lru_size(lruvec, lru, zone, delta); + } +} + +static int should_skip_vma(unsigned long start, unsigned long end, struct = mm_walk *args) +{ + struct address_space *mapping; + struct vm_area_struct *vma =3D args->vma; + struct lru_gen_mm_walk *walk =3D args->private; + + if (!vma_is_accessible(vma) || is_vm_hugetlb_page(vma) || + (vma->vm_flags & (VM_LOCKED | VM_SPECIAL | VM_SEQ_READ | VM_RAND_READ= )) || + vma =3D=3D get_gate_vma(vma->vm_mm)) + return true; + + if (vma_is_anonymous(vma)) + return !walk->can_swap; + + if (WARN_ON_ONCE(!vma->vm_file || !vma->vm_file->f_mapping)) + return true; + + mapping =3D vma->vm_file->f_mapping; + if (mapping_unevictable(mapping)) + return true; + + /* check readpage to exclude special mappings like dax, etc. */ + return shmem_mapping(mapping) ? !walk->can_swap : !mapping->a_ops->readpa= ge; +} + +/* + * Some userspace memory allocators map many single-page VMAs. Instead of + * returning back to the PGD table for each of such VMAs, finish an entire= PMD + * table to reduce zigzags and improve cache performance. + */ +static bool get_next_vma(unsigned long mask, unsigned long size, struct mm= _walk *args, + unsigned long *start, unsigned long *end) +{ + unsigned long next =3D round_up(*end, size); + + VM_WARN_ON_ONCE(mask & size); + VM_WARN_ON_ONCE(*start >=3D *end); + VM_WARN_ON_ONCE((next & mask) !=3D (*start & mask)); + + while (args->vma) { + if (next >=3D args->vma->vm_end) { + args->vma =3D args->vma->vm_next; + continue; + } + + if ((next & mask) !=3D (args->vma->vm_start & mask)) + return false; + + if (should_skip_vma(args->vma->vm_start, args->vma->vm_end, args)) { + args->vma =3D args->vma->vm_next; + continue; + } + + *start =3D max(next, args->vma->vm_start); + next =3D (next | ~mask) + 1; + /* rounded-up boundaries can wrap to 0 */ + *end =3D next && next < args->vma->vm_end ? next : args->vma->vm_end; + + return true; + } + + return false; +} + +static bool suitable_to_scan(int total, int young) +{ + int n =3D clamp_t(int, cache_line_size() / sizeof(pte_t), 2, 8); + + /* suitable if the average number of young PTEs per cacheline is >=3D1 */ + return young * n >=3D total; +} + +static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long = end, + struct mm_walk *args) +{ + int i; + pte_t *pte; + spinlock_t *ptl; + unsigned long addr; + int total =3D 0; + int young =3D 0; + struct lru_gen_mm_walk *walk =3D args->private; + struct mem_cgroup *memcg =3D lruvec_memcg(walk->lruvec); + struct pglist_data *pgdat =3D lruvec_pgdat(walk->lruvec); + int old_gen, new_gen =3D lru_gen_from_seq(walk->max_seq); + + VM_WARN_ON_ONCE(pmd_leaf(*pmd)); + + ptl =3D pte_lockptr(args->mm, pmd); + if (!spin_trylock(ptl)) + return false; + + arch_enter_lazy_mmu_mode(); + + pte =3D pte_offset_map(pmd, start & PMD_MASK); +restart: + for (i =3D pte_index(start), addr =3D start; addr !=3D end; i++, addr += =3D PAGE_SIZE) { + struct folio *folio; + unsigned long pfn =3D pte_pfn(pte[i]); + + VM_WARN_ON_ONCE(addr < args->vma->vm_start || addr >=3D args->vma->vm_en= d); + + total++; + walk->mm_stats[MM_PTE_TOTAL]++; + + if (!pte_present(pte[i]) || is_zero_pfn(pfn)) + continue; + + if (WARN_ON_ONCE(pte_devmap(pte[i]) || pte_special(pte[i]))) + continue; + + if (!pte_young(pte[i])) { + walk->mm_stats[MM_PTE_OLD]++; + continue; + } + + VM_WARN_ON_ONCE(!pfn_valid(pfn)); + if (pfn < pgdat->node_start_pfn || pfn >=3D pgdat_end_pfn(pgdat)) + continue; + + folio =3D pfn_folio(pfn); + if (folio_nid(folio) !=3D pgdat->node_id) + continue; + + if (folio_memcg_rcu(folio) !=3D memcg) + continue; + + if (!folio_is_file_lru(folio) && !walk->can_swap) + continue; + + if (!ptep_test_and_clear_young(args->vma, addr, pte + i)) + continue; + + young++; + walk->mm_stats[MM_PTE_YOUNG]++; + + if (pte_dirty(pte[i]) && !folio_test_dirty(folio) && + !(folio_test_anon(folio) && folio_test_swapbacked(folio) && + !folio_test_swapcache(folio))) + folio_mark_dirty(folio); + + old_gen =3D folio_update_gen(folio, new_gen); + if (old_gen >=3D 0 && old_gen !=3D new_gen) + update_batch_size(walk, folio, old_gen, new_gen); + } + + if (i < PTRS_PER_PTE && get_next_vma(PMD_MASK, PAGE_SIZE, args, &start, &= end)) + goto restart; + + pte_unmap(pte); + + arch_leave_lazy_mmu_mode(); + spin_unlock(ptl); + + return suitable_to_scan(total, young); +} + +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEA= F_PMD_YOUNG) +static void walk_pmd_range_locked(pud_t *pud, unsigned long next, struct v= m_area_struct *vma, + struct mm_walk *args, unsigned long *start) +{ + int i; + pmd_t *pmd; + spinlock_t *ptl; + struct lru_gen_mm_walk *walk =3D args->private; + struct mem_cgroup *memcg =3D lruvec_memcg(walk->lruvec); + struct pglist_data *pgdat =3D lruvec_pgdat(walk->lruvec); + int old_gen, new_gen =3D lru_gen_from_seq(walk->max_seq); + + VM_WARN_ON_ONCE(pud_leaf(*pud)); + + /* try to batch at most 1+MIN_LRU_BATCH+1 entries */ + if (*start =3D=3D -1) { + *start =3D next; + return; + } + + i =3D next =3D=3D -1 ? 0 : pmd_index(next) - pmd_index(*start); + if (i && i <=3D MIN_LRU_BATCH) { + __set_bit(i - 1, walk->bitmap); + return; + } + + pmd =3D pmd_offset(pud, *start); + + ptl =3D pmd_lockptr(args->mm, pmd); + if (!spin_trylock(ptl)) + goto done; + + arch_enter_lazy_mmu_mode(); + + do { + struct folio *folio; + unsigned long pfn =3D pmd_pfn(pmd[i]); + unsigned long addr =3D i ? (*start & PMD_MASK) + i * PMD_SIZE : *start; + + VM_WARN_ON_ONCE(addr < vma->vm_start || addr >=3D vma->vm_end); + + if (!pmd_present(pmd[i]) || is_huge_zero_pmd(pmd[i])) + goto next; + + if (WARN_ON_ONCE(pmd_devmap(pmd[i]))) + goto next; + + if (!pmd_trans_huge(pmd[i])) { + if (IS_ENABLED(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG)) + pmdp_test_and_clear_young(vma, addr, pmd + i); + goto next; + } + + VM_WARN_ON_ONCE(!pfn_valid(pfn)); + if (pfn < pgdat->node_start_pfn || pfn >=3D pgdat_end_pfn(pgdat)) + goto next; + + folio =3D pfn_folio(pfn); + if (folio_nid(folio) !=3D pgdat->node_id) + goto next; + + if (folio_memcg_rcu(folio) !=3D memcg) + goto next; + + if (!folio_is_file_lru(folio) && !walk->can_swap) + goto next; + + if (!pmdp_test_and_clear_young(vma, addr, pmd + i)) + goto next; + + walk->mm_stats[MM_PTE_YOUNG]++; + + if (pmd_dirty(pmd[i]) && !folio_test_dirty(folio) && + !(folio_test_anon(folio) && folio_test_swapbacked(folio) && + !folio_test_swapcache(folio))) + folio_mark_dirty(folio); + + old_gen =3D folio_update_gen(folio, new_gen); + if (old_gen >=3D 0 && old_gen !=3D new_gen) + update_batch_size(walk, folio, old_gen, new_gen); +next: + i =3D i > MIN_LRU_BATCH ? 0 : + find_next_bit(walk->bitmap, MIN_LRU_BATCH, i) + 1; + } while (i <=3D MIN_LRU_BATCH); + + arch_leave_lazy_mmu_mode(); + spin_unlock(ptl); +done: + *start =3D -1; + bitmap_zero(walk->bitmap, MIN_LRU_BATCH); +} +#else +static void walk_pmd_range_locked(pud_t *pud, unsigned long next, struct v= m_area_struct *vma, + struct mm_walk *args, unsigned long *start) +{ +} +#endif + +static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long = end, + struct mm_walk *args) +{ + int i; + pmd_t *pmd; + unsigned long next; + unsigned long addr; + struct vm_area_struct *vma; + unsigned long pos =3D -1; + struct lru_gen_mm_walk *walk =3D args->private; + + VM_WARN_ON_ONCE(pud_leaf(*pud)); + + /* + * Finish an entire PMD in two passes: the first only reaches to PTE + * tables to avoid taking the PMD lock; the second, if necessary, takes + * the PMD lock to clear the accessed bit in PMD entries. + */ + pmd =3D pmd_offset(pud, start & PUD_MASK); +restart: + /* walk_pte_range() may call get_next_vma() */ + vma =3D args->vma; + for (i =3D pmd_index(start), addr =3D start; addr !=3D end; i++, addr =3D= next) { + pmd_t val =3D pmd_read_atomic(pmd + i); + + /* for pmd_read_atomic() */ + barrier(); + + next =3D pmd_addr_end(addr, end); + + if (!pmd_present(val)) { + walk->mm_stats[MM_PTE_TOTAL]++; + continue; + } + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + if (pmd_trans_huge(val)) { + unsigned long pfn =3D pmd_pfn(val); + struct pglist_data *pgdat =3D lruvec_pgdat(walk->lruvec); + + walk->mm_stats[MM_PTE_TOTAL]++; + + if (is_huge_zero_pmd(val)) + continue; + + if (!pmd_young(val)) { + walk->mm_stats[MM_PTE_OLD]++; + continue; + } + + if (pfn < pgdat->node_start_pfn || pfn >=3D pgdat_end_pfn(pgdat)) + continue; + + walk_pmd_range_locked(pud, addr, vma, args, &pos); + continue; + } +#endif + walk->mm_stats[MM_PMD_TOTAL]++; + +#ifdef CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG + if (!pmd_young(val)) + continue; + + walk_pmd_range_locked(pud, addr, vma, args, &pos); +#endif + if (!walk->force_scan && !test_bloom_filter(walk->lruvec, walk->max_seq,= pmd + i)) + continue; + + walk->mm_stats[MM_PMD_FOUND]++; + + if (!walk_pte_range(&val, addr, next, args)) + continue; + + walk->mm_stats[MM_PMD_ADDED]++; + + /* carry over to the next generation */ + update_bloom_filter(walk->lruvec, walk->max_seq + 1, pmd + i); + } + + walk_pmd_range_locked(pud, -1, vma, args, &pos); + + if (i < PTRS_PER_PMD && get_next_vma(PUD_MASK, PMD_SIZE, args, &start, &e= nd)) + goto restart; +} + +static int walk_pud_range(p4d_t *p4d, unsigned long start, unsigned long e= nd, + struct mm_walk *args) +{ + int i; + pud_t *pud; + unsigned long addr; + unsigned long next; + struct lru_gen_mm_walk *walk =3D args->private; + + VM_WARN_ON_ONCE(p4d_leaf(*p4d)); + + pud =3D pud_offset(p4d, start & P4D_MASK); +restart: + for (i =3D pud_index(start), addr =3D start; addr !=3D end; i++, addr =3D= next) { + pud_t val =3D READ_ONCE(pud[i]); + + next =3D pud_addr_end(addr, end); + + if (!pud_present(val) || WARN_ON_ONCE(pud_leaf(val))) + continue; + + walk_pmd_range(&val, addr, next, args); + + if (walk->batched >=3D MAX_LRU_BATCH) { + end =3D (addr | ~PUD_MASK) + 1; + goto done; + } + } + + if (i < PTRS_PER_PUD && get_next_vma(P4D_MASK, PUD_SIZE, args, &start, &e= nd)) + goto restart; + + end =3D round_up(end, P4D_SIZE); +done: + /* rounded-up boundaries can wrap to 0 */ + walk->next_addr =3D end && args->vma ? max(end, args->vma->vm_start) : 0; + + return -EAGAIN; +} + +static void walk_mm(struct lruvec *lruvec, struct mm_struct *mm, struct lr= u_gen_mm_walk *walk) +{ + static const struct mm_walk_ops mm_walk_ops =3D { + .test_walk =3D should_skip_vma, + .p4d_entry =3D walk_pud_range, + }; + + int err; + struct mem_cgroup *memcg =3D lruvec_memcg(lruvec); + + walk->next_addr =3D FIRST_USER_ADDRESS; + + do { + err =3D -EBUSY; + + /* folio_update_gen() requires stable folio_memcg() */ + if (!mem_cgroup_trylock_pages(memcg)) + break; + + /* the caller might be holding the lock for write */ + if (mmap_read_trylock(mm)) { + err =3D walk_page_range(mm, walk->next_addr, ULONG_MAX, &mm_walk_ops, w= alk); + + mmap_read_unlock(mm); + + if (walk->batched) { + spin_lock_irq(&lruvec->lru_lock); + reset_batch_size(lruvec, walk); + spin_unlock_irq(&lruvec->lru_lock); + } + } + + mem_cgroup_unlock_pages(); + + cond_resched(); + } while (err =3D=3D -EAGAIN && walk->next_addr && !mm_is_oom_victim(mm)); +} + +static struct lru_gen_mm_walk *alloc_mm_walk(void) +{ + if (current->reclaim_state && current->reclaim_state->mm_walk) + return current->reclaim_state->mm_walk; + + return kzalloc(sizeof(struct lru_gen_mm_walk), + __GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN); +} + +static void free_mm_walk(struct lru_gen_mm_walk *walk) +{ + if (!current->reclaim_state || !current->reclaim_state->mm_walk) + kfree(walk); +} + static void inc_min_seq(struct lruvec *lruvec, int type) { struct lru_gen_struct *lrugen =3D &lruvec->lrugen; @@ -3244,7 +4088,7 @@ static bool try_to_inc_min_seq(struct lruvec *lruvec,= bool can_swap) return success; } =20 -static void inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, bool= can_swap) +static void inc_max_seq(struct lruvec *lruvec, bool can_swap) { int prev, next; int type, zone; @@ -3254,9 +4098,6 @@ static void inc_max_seq(struct lruvec *lruvec, unsign= ed long max_seq, bool can_s =20 VM_WARN_ON_ONCE(!seq_is_valid(lruvec)); =20 - if (max_seq !=3D lrugen->max_seq) - goto unlock; - for (type =3D 0; type < ANON_AND_FILE; type++) { if (get_nr_gens(lruvec, type) !=3D MAX_NR_GENS) continue; @@ -3294,10 +4135,72 @@ static void inc_max_seq(struct lruvec *lruvec, unsi= gned long max_seq, bool can_s =20 /* make sure preceding modifications appear */ smp_store_release(&lrugen->max_seq, lrugen->max_seq + 1); -unlock: + spin_unlock_irq(&lruvec->lru_lock); } =20 +static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_se= q, + struct scan_control *sc, bool can_swap) +{ + bool success; + struct lru_gen_mm_walk *walk; + struct mm_struct *mm =3D NULL; + struct lru_gen_struct *lrugen =3D &lruvec->lrugen; + + VM_WARN_ON_ONCE(max_seq > READ_ONCE(lrugen->max_seq)); + + /* + * If the hardware doesn't automatically set the accessed bit, fallback + * to lru_gen_look_around(), which only clears the accessed bit in a + * handful of PTEs. Spreading the work out over a period of time usually + * is less efficient, but it avoids bursty page faults. + */ + if (!arch_has_hw_pte_young()) { + success =3D iterate_mm_list_nowalk(lruvec, max_seq); + goto done; + } + + walk =3D alloc_mm_walk(); + if (!walk) { + success =3D iterate_mm_list_nowalk(lruvec, max_seq); + goto done; + } + + walk->lruvec =3D lruvec; + walk->max_seq =3D max_seq; + walk->can_swap =3D can_swap; + walk->force_scan =3D false; + + do { + success =3D iterate_mm_list(lruvec, walk, &mm); + if (mm) + walk_mm(lruvec, mm, walk); + + cond_resched(); + } while (mm); + + free_mm_walk(walk); +done: + if (!success) { + if (!current_is_kswapd() && !sc->priority) + wait_event_killable(lruvec->mm_state.wait, + max_seq < READ_ONCE(lrugen->max_seq)); + + return max_seq < READ_ONCE(lrugen->max_seq); + } + + VM_WARN_ON_ONCE(max_seq !=3D READ_ONCE(lrugen->max_seq)); + + inc_max_seq(lruvec, can_swap); + /* either this sees any waiters or they will see updated max_seq */ + if (wq_has_sleeper(&lruvec->mm_state.wait)) + wake_up_all(&lruvec->mm_state.wait); + + wakeup_flusher_threads(WB_REASON_VMSCAN); + + return true; +} + static long get_nr_evictable(struct lruvec *lruvec, unsigned long max_seq, unsigned long *min_seq, bool can_swap, bool *need_aging) { @@ -3378,7 +4281,7 @@ static void age_lruvec(struct lruvec *lruvec, struct = scan_control *sc) nr_to_scan++; =20 if (nr_to_scan && need_aging) - inc_max_seq(lruvec, max_seq, swappiness); + try_to_inc_max_seq(lruvec, max_seq, sc, swappiness); } =20 static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_contro= l *sc) @@ -3387,6 +4290,8 @@ static void lru_gen_age_node(struct pglist_data *pgda= t, struct scan_control *sc) =20 VM_WARN_ON_ONCE(!current_is_kswapd()); =20 + current->reclaim_state->mm_walk =3D &pgdat->mm_walk; + memcg =3D mem_cgroup_iter(NULL, NULL, NULL); do { struct lruvec *lruvec =3D mem_cgroup_lruvec(memcg, pgdat); @@ -3395,11 +4300,16 @@ static void lru_gen_age_node(struct pglist_data *pg= dat, struct scan_control *sc) =20 cond_resched(); } while ((memcg =3D mem_cgroup_iter(NULL, memcg, NULL))); + + current->reclaim_state->mm_walk =3D NULL; } =20 /* * This function exploits spatial locality when shrink_page_list() walks t= he - * rmap. It scans the adjacent PTEs of a young PTE and promotes hot pages. + * rmap. It scans the adjacent PTEs of a young PTE and promotes hot pages.= If + * the scan was done cacheline efficiently, it adds the PMD entry pointing= to + * the PTE table to the Bloom filter. This forms a feedback loop between t= he + * eviction and the aging. */ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) { @@ -3408,6 +4318,8 @@ void lru_gen_look_around(struct page_vma_mapped_walk = *pvmw) unsigned long start; unsigned long end; unsigned long addr; + struct lru_gen_mm_walk *walk; + int young =3D 0; unsigned long bitmap[BITS_TO_LONGS(MIN_LRU_BATCH)] =3D {}; struct folio *folio =3D pfn_folio(pvmw->pfn); struct mem_cgroup *memcg =3D folio_memcg(folio); @@ -3469,6 +4381,8 @@ void lru_gen_look_around(struct page_vma_mapped_walk = *pvmw) if (!ptep_test_and_clear_young(pvmw->vma, addr, pte + i)) continue; =20 + young++; + if (pte_dirty(pte[i]) && !folio_test_dirty(folio) && !(folio_test_anon(folio) && folio_test_swapbacked(folio) && !folio_test_swapcache(folio))) @@ -3484,7 +4398,13 @@ void lru_gen_look_around(struct page_vma_mapped_walk= *pvmw) arch_leave_lazy_mmu_mode(); rcu_read_unlock(); =20 - if (bitmap_weight(bitmap, MIN_LRU_BATCH) < PAGEVEC_SIZE) { + /* feedback from rmap walkers to page table walkers */ + if (suitable_to_scan(i, young)) + update_bloom_filter(lruvec, max_seq, pvmw->pmd); + + walk =3D current->reclaim_state ? current->reclaim_state->mm_walk : NULL; + + if (!walk && bitmap_weight(bitmap, MIN_LRU_BATCH) < PAGEVEC_SIZE) { for_each_set_bit(i, bitmap, MIN_LRU_BATCH) { folio =3D pfn_folio(pte_pfn(pte[i])); folio_activate(folio); @@ -3496,8 +4416,10 @@ void lru_gen_look_around(struct page_vma_mapped_walk= *pvmw) if (!mem_cgroup_trylock_pages(memcg)) return; =20 - spin_lock_irq(&lruvec->lru_lock); - new_gen =3D lru_gen_from_seq(lruvec->lrugen.max_seq); + if (!walk) { + spin_lock_irq(&lruvec->lru_lock); + new_gen =3D lru_gen_from_seq(lruvec->lrugen.max_seq); + } =20 for_each_set_bit(i, bitmap, MIN_LRU_BATCH) { folio =3D pfn_folio(pte_pfn(pte[i])); @@ -3508,10 +4430,14 @@ void lru_gen_look_around(struct page_vma_mapped_wal= k *pvmw) if (old_gen < 0 || old_gen =3D=3D new_gen) continue; =20 - lru_gen_update_size(lruvec, folio, old_gen, new_gen); + if (walk) + update_batch_size(walk, folio, old_gen, new_gen); + else + lru_gen_update_size(lruvec, folio, old_gen, new_gen); } =20 - spin_unlock_irq(&lruvec->lru_lock); + if (!walk) + spin_unlock_irq(&lruvec->lru_lock); =20 mem_cgroup_unlock_pages(); } @@ -3782,6 +4708,7 @@ static int evict_folios(struct lruvec *lruvec, struct= scan_control *sc, int swap struct folio *folio; enum vm_event_item item; struct reclaim_stat stat; + struct lru_gen_mm_walk *walk; struct mem_cgroup *memcg =3D lruvec_memcg(lruvec); struct pglist_data *pgdat =3D lruvec_pgdat(lruvec); =20 @@ -3821,6 +4748,10 @@ static int evict_folios(struct lruvec *lruvec, struc= t scan_control *sc, int swap =20 move_pages_to_lru(lruvec, &list); =20 + walk =3D current->reclaim_state ? current->reclaim_state->mm_walk : NULL; + if (walk && walk->batched) + reset_batch_size(lruvec, walk); + item =3D current_is_kswapd() ? PGSTEAL_KSWAPD : PGSTEAL_DIRECT; if (!cgroup_reclaim(sc)) __count_vm_events(item, reclaimed); @@ -3875,20 +4806,25 @@ static long get_nr_to_scan(struct lruvec *lruvec, s= truct scan_control *sc, bool return 0; } =20 - inc_max_seq(lruvec, max_seq, can_swap); + if (try_to_inc_max_seq(lruvec, max_seq, sc, can_swap)) + return nr_to_scan; =20 - return nr_to_scan; + return min_seq[!can_swap] + MIN_NR_GENS <=3D max_seq ? nr_to_scan : 0; } =20 static void lru_gen_shrink_lruvec(struct lruvec *lruvec, struct scan_contr= ol *sc) { struct blk_plug plug; long scanned =3D 0; + struct pglist_data *pgdat =3D lruvec_pgdat(lruvec); =20 lru_add_drain(); =20 blk_start_plug(&plug); =20 + if (current_is_kswapd()) + current->reclaim_state->mm_walk =3D &pgdat->mm_walk; + while (true) { int delta; int swappiness; @@ -3916,6 +4852,9 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruv= ec, struct scan_control *sc cond_resched(); } =20 + if (current_is_kswapd()) + current->reclaim_state->mm_walk =3D NULL; + blk_finish_plug(&plug); } =20 @@ -3932,15 +4871,21 @@ void lru_gen_init_lruvec(struct lruvec *lruvec) =20 for_each_gen_type_zone(gen, type, zone) INIT_LIST_HEAD(&lrugen->lists[gen][type][zone]); + + lruvec->mm_state.seq =3D MIN_NR_GENS; + init_waitqueue_head(&lruvec->mm_state.wait); } =20 #ifdef CONFIG_MEMCG void lru_gen_init_memcg(struct mem_cgroup *memcg) { + INIT_LIST_HEAD(&memcg->mm_list.fifo); + spin_lock_init(&memcg->mm_list.lock); } =20 void lru_gen_exit_memcg(struct mem_cgroup *memcg) { + int i; int nid; =20 for_each_node(nid) { @@ -3948,6 +4893,11 @@ void lru_gen_exit_memcg(struct mem_cgroup *memcg) =20 VM_WARN_ON_ONCE(memchr_inv(lruvec->lrugen.nr_pages, 0, sizeof(lruvec->lrugen.nr_pages))); + + for (i =3D 0; i < NR_BLOOM_FILTERS; i++) { + bitmap_free(lruvec->mm_state.filters[i]); + lruvec->mm_state.filters[i] =3D NULL; + } } } #endif @@ -3956,6 +4906,7 @@ static int __init init_lru_gen(void) { BUILD_BUG_ON(MIN_NR_GENS + 1 >=3D MAX_NR_GENS); BUILD_BUG_ON(BIT(LRU_GEN_WIDTH) <=3D MAX_NR_GENS); + BUILD_BUG_ON(sizeof(MM_STAT_CODES) !=3D NR_MM_STATS + 1); =20 return 0; }; --=20 2.36.0.550.gb090851708-goog From nobody Thu May 7 21:41:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 567E8C433F5 for ; Wed, 18 May 2022 01:47:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233950AbiERBrz (ORCPT ); Tue, 17 May 2022 21:47:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57982 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233908AbiERBrT (ORCPT ); Tue, 17 May 2022 21:47:19 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4C2D554BDF for ; Tue, 17 May 2022 18:47:08 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-2fefb9975c5so7807467b3.21 for ; Tue, 17 May 2022 18:47:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc:content-transfer-encoding; bh=4FeaUrAcLPRNiyzpAHQ0qcCXUD7YXoCTIslsK0eD41Y=; b=fY1LqfrKQ18+zKLGy9WAd1jhk4VwDPAd0IP4P/GayWs6ndMf1nwMh8ZeYcvmBvuQ86 GpQQbyLBSjF23S7zqVvGdVp+NhPNYUcEUibP9bZO93rvR/t4RaRptXl2EOOp6+M7siuk yyRO+Wl6PuJWn7Xwhyh8gZ85CQPKMf/I/wBMMl0V/SycgsCafQq1iRHTvJ9moWxQmf46 rZyxEQjVEhddig/3bxo3Z7eq6tArA9FnKVE89X+ECO3qXcalVE8q0pmjvTzgQoZD9u8Y 1jNa9K1p0P9tENosGYdbO0KFHBjCCY+zlhZh6sG36xX4v5+dFYKBdKat1qNZevsdhVt/ sD6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc:content-transfer-encoding; bh=4FeaUrAcLPRNiyzpAHQ0qcCXUD7YXoCTIslsK0eD41Y=; b=Yctb6Ps7fdFFAgc75u2RI9XzpoxoE6nBCjDe2PfaL+GQl5xyrQ/j8n6JmTEI+12sR3 aMJlJstWA4S8ko7DgHVpbZmFX5gWVXHu3vYlymy4ejTbSL7ULwMxIgz8XzOXVuL46uaX ZO8Zp1Gp/j/5Lxb7wDR5HzsLcmvcWRhStNJSonFb1Em6d99AQ1UJNnjG7YSJP0/w+xEW gRPscXVFQpKJcssgW8P+izH/NIQjNWduzMog04HMMmVSHqcgneqR5paO90iRQnxNwvfI 9RIKx9GbWdLBGVSK4I3GzOOTNJb+DhjdEPwj6rLOACDZMRt1JhjlFD0jeXogPymF3TMX n7Lg== X-Gm-Message-State: AOAM530x//URpklhvvgeII6Ddh4AVfYXNL9+08kP/K+f1Cj6UInXfaTE Qg0GNcNLYeJd8+ixOqnSEdh+xxymM1U= X-Google-Smtp-Source: ABdhPJxuNk1xSGxsu2QLdajUUp4vZke7U/gS7MXlBwFIqZvnz7q0GxTwsMzXdo4nRCJXJA5BHFLhn+OxTdg= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f7bc:90c9:d86e:4ea8]) (user=yuzhao job=sendgmr) by 2002:a81:2684:0:b0:2f8:46f4:7c6d with SMTP id m126-20020a812684000000b002f846f47c6dmr30129509ywm.444.1652838427172; Tue, 17 May 2022 18:47:07 -0700 (PDT) Date: Tue, 17 May 2022 19:46:28 -0600 In-Reply-To: <20220518014632.922072-1-yuzhao@google.com> Message-Id: <20220518014632.922072-10-yuzhao@google.com> Mime-Version: 1.0 References: <20220518014632.922072-1-yuzhao@google.com> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog Subject: [PATCH v11 09/14] mm: multi-gen LRU: optimize multiple memcgs From: Yu Zhao To: Andrew Morton , linux-mm@kvack.org Cc: Andi Kleen , Aneesh Kumar , Catalin Marinas , Dave Hansen , Hillf Danton , Jens Axboe , Johannes Weiner , Jonathan Corbet , Linus Torvalds , Matthew Wilcox , Mel Gorman , Michael Larabel , Michal Hocko , Mike Rapoport , Peter Zijlstra , Tejun Heo , Vlastimil Babka , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, page-reclaim@google.com, Yu Zhao , Brian Geffon , Jan Alexander Steffens , Oleksandr Natalenko , Steven Barrett , Suleiman Souhlal , Daniel Byrne , Donald Carr , "=?UTF-8?q?Holger=20Hoffst=C3=A4tte?=" , Konstantin Kharlamov , Shuang Zhai , Sofia Trinh , Vaibhav Jain Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When multiple memcgs are available, it is possible to make better choices based on generations and tiers and therefore improve the overall performance under global memory pressure. This patch adds a rudimentary optimization to select memcgs that can drop single-use unmapped clean pages first. Doing so reduces the chance of going into the aging path or swapping. These two operations can be costly. A typical example that benefits from this optimization is a server running mixed types of workloads, e.g., heavy anon workload in one memcg and heavy buffered I/O workload in the other. Though this optimization can be applied to both kswapd and direct reclaim, it is only added to kswapd to keep the patchset manageable. Later improvements will cover the direct reclaim path. Server benchmark results: Mixed workloads: fio (buffered I/O): +[1, 3]% IOPS BW patch1-8: 2154k 8415MiB/s patch1-9: 2205k 8613MiB/s memcached (anon): +[132, 136]% Ops/sec KB/sec patch1-8: 819618.49 31838.48 patch1-9: 1916516.06 74447.92 Mixed workloads: fio (buffered I/O): +[59, 61]% IOPS BW 5.18-rc1: 1378k 5385MiB/s patch1-9: 2205k 8613MiB/s memcached (anon): +[229, 233]% Ops/sec KB/sec 5.18-rc1: 578946.00 22489.44 patch1-9: 1916516.06 74447.92 Configurations: (changes since patch 6) cat mixed.sh modprobe brd rd_nr=3D2 rd_size=3D56623104 swapoff -a mkswap /dev/ram0 swapon /dev/ram0 mkfs.ext4 /dev/ram1 mount -t ext4 /dev/ram1 /mnt memtier_benchmark -S /var/run/memcached/memcached.sock \ -P memcache_binary -n allkeys --key-minimum=3D1 \ --key-maximum=3D50000000 --key-pattern=3DP:P -c 1 -t 36 \ --ratio 1:0 --pipeline 8 -d 2000 fio -name=3Dmglru --numjobs=3D36 --directory=3D/mnt --size=3D1408m \ --buffered=3D1 --ioengine=3Dio_uring --iodepth=3D128 \ --iodepth_batch_submit=3D32 --iodepth_batch_complete=3D32 \ --rw=3Drandread --random_distribution=3Drandom --norandommap \ --time_based --ramp_time=3D10m --runtime=3D90m --group_reporting & pid=3D$! sleep 200 memtier_benchmark -S /var/run/memcached/memcached.sock \ -P memcache_binary -n allkeys --key-minimum=3D1 \ --key-maximum=3D50000000 --key-pattern=3DR:R -c 1 -t 36 \ --ratio 0:1 --pipeline 8 --randomize --distinct-client-seed kill -INT $pid wait Client benchmark results: no change (CONFIG_MEMCG=3Dn) Signed-off-by: Yu Zhao Acked-by: Brian Geffon Acked-by: Jan Alexander Steffens (heftig) Acked-by: Oleksandr Natalenko Acked-by: Steven Barrett Acked-by: Suleiman Souhlal Tested-by: Daniel Byrne Tested-by: Donald Carr Tested-by: Holger Hoffst=C3=A4tte Tested-by: Konstantin Kharlamov Tested-by: Shuang Zhai Tested-by: Sofia Trinh Tested-by: Vaibhav Jain --- mm/vmscan.c | 45 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 41 insertions(+), 4 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index f292e7e761b1..a7e768675707 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -128,6 +128,13 @@ struct scan_control { /* Always discard instead of demoting to lower tier memory */ unsigned int no_demotion:1; =20 +#ifdef CONFIG_LRU_GEN + /* help make better choices when multiple memcgs are available */ + unsigned int memcgs_need_aging:1; + unsigned int memcgs_need_swapping:1; + unsigned int memcgs_avoid_swapping:1; +#endif + /* Allocation order */ s8 order; =20 @@ -4290,6 +4297,22 @@ static void lru_gen_age_node(struct pglist_data *pgd= at, struct scan_control *sc) =20 VM_WARN_ON_ONCE(!current_is_kswapd()); =20 + /* + * To reduce the chance of going into the aging path or swapping, which + * can be costly, optimistically skip them unless their corresponding + * flags were cleared in the eviction path. This improves the overall + * performance when multiple memcgs are available. + */ + if (!sc->memcgs_need_aging) { + sc->memcgs_need_aging =3D true; + sc->memcgs_avoid_swapping =3D !sc->memcgs_need_swapping; + sc->memcgs_need_swapping =3D true; + return; + } + + sc->memcgs_need_swapping =3D true; + sc->memcgs_avoid_swapping =3D true; + current->reclaim_state->mm_walk =3D &pgdat->mm_walk; =20 memcg =3D mem_cgroup_iter(NULL, NULL, NULL); @@ -4699,7 +4722,8 @@ static int isolate_folios(struct lruvec *lruvec, stru= ct scan_control *sc, int sw return scanned; } =20 -static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, in= t swappiness) +static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, in= t swappiness, + bool *swapped) { int type; int scanned; @@ -4765,6 +4789,9 @@ static int evict_folios(struct lruvec *lruvec, struct= scan_control *sc, int swap =20 sc->nr_reclaimed +=3D reclaimed; =20 + if (type =3D=3D LRU_GEN_ANON && swapped) + *swapped =3D true; + return scanned; } =20 @@ -4793,8 +4820,10 @@ static long get_nr_to_scan(struct lruvec *lruvec, st= ruct scan_control *sc, bool if (!nr_to_scan) return 0; =20 - if (!need_aging) + if (!need_aging) { + sc->memcgs_need_aging =3D false; return nr_to_scan; + } =20 /* leave the work to lru_gen_age_node() */ if (current_is_kswapd()) @@ -4816,6 +4845,8 @@ static void lru_gen_shrink_lruvec(struct lruvec *lruv= ec, struct scan_control *sc { struct blk_plug plug; long scanned =3D 0; + bool swapped =3D false; + unsigned long reclaimed =3D sc->nr_reclaimed; struct pglist_data *pgdat =3D lruvec_pgdat(lruvec); =20 lru_add_drain(); @@ -4841,13 +4872,19 @@ static void lru_gen_shrink_lruvec(struct lruvec *lr= uvec, struct scan_control *sc if (!nr_to_scan) break; =20 - delta =3D evict_folios(lruvec, sc, swappiness); + delta =3D evict_folios(lruvec, sc, swappiness, &swapped); if (!delta) break; =20 + if (sc->memcgs_avoid_swapping && swappiness < 200 && swapped) + break; + scanned +=3D delta; - if (scanned >=3D nr_to_scan) + if (scanned >=3D nr_to_scan) { + if (!swapped && sc->nr_reclaimed - reclaimed >=3D MIN_LRU_BATCH) + sc->memcgs_need_swapping =3D false; break; + } =20 cond_resched(); } --=20 2.36.0.550.gb090851708-goog From nobody Thu May 7 21:41:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91267C433EF for ; Wed, 18 May 2022 01:48:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233959AbiERBsC (ORCPT ); Tue, 17 May 2022 21:48:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57960 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233920AbiERBrU (ORCPT ); Tue, 17 May 2022 21:47:20 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1989554BE6 for ; Tue, 17 May 2022 18:47:08 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id s196-20020a252ccd000000b0064ea2e6bcb7so229133ybs.3 for ; Tue, 17 May 2022 18:47:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc:content-transfer-encoding; bh=NTA14FnCNnSPTudLkNszGIL3HdbMAT4nMBCkNLY5YRU=; b=dj/NUzQ7pldQ5p6qQOOz1M0rA++R2i0rkuA/pdh7gKHKfecCVVJbxlcKWafSRzT+io P2/TwGHl8GyP+Mg5hb/SV8E48PYJa7SyPhOeCh2WtveDGvIqR63kJBDvMBRon9bfxFwc cb+4Ti+w2hCv3EbKxHhCjMCuicPeQiavbDKafGr2N+HvVGBiBGlMin4TBHAZPvg5HWiV fsxaCDJXxWjs84OmU9+hKlIJ1ZEGHcKvSILfuBZ+lCwaenIW0SoaNOUqcU3H99tZA0Pp OpihneRVY05VI5/nix7QlSlUnp7oGxB689r8rUG7RiYvAjaJUxEm2rPSva5gOkSmMRqe EN5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc:content-transfer-encoding; bh=NTA14FnCNnSPTudLkNszGIL3HdbMAT4nMBCkNLY5YRU=; b=FPq1AiYyVsKTPOEn16VwWiUTZb0Jbiga3lDiv+eNINBk2hV3aZpaioYknsPzco6oq8 3JVFlQszcxusqV5TEUmxikpICBnb+hh70bgDa9VevJC6r9neqfmcaMftt39KlR+3sQiX eqr+OlpcER6Y/b9uTzMfqzP5S7oOeX4BoQhC7LR2w/GaHvvs9NSaNeH0e2zGKILLGlSy pb3zqPB5+H503gX37lgH0xSk/gzXjNbQ1BdL0TCe0f+Lsmeu0oaGkEeKdMtEIbR/rjLu LnUZJxuhNfQy26/d5od03JDZy1CGVzk5OeXNCp9ejEkgzMAv3DTpf1iczFc2J+acO0iC PyiQ== X-Gm-Message-State: AOAM531KhJGP4phWQQt0aY9y82aaZubL2vzWv52PPio6V1pHoWNjTIeC Clil4jWJZWW2DAWWzBEHvRLHRMv4vlo= X-Google-Smtp-Source: ABdhPJwTsKX58YMZjZU473qHcm2WfVL2bwxTclJIx+w2+3BMEn4WcPcuNwyIzbAE4LxodKTFVAQMvubt74g= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f7bc:90c9:d86e:4ea8]) (user=yuzhao job=sendgmr) by 2002:a81:ac11:0:b0:2fe:ee45:c6bb with SMTP id k17-20020a81ac11000000b002feee45c6bbmr15371827ywh.427.1652838428578; Tue, 17 May 2022 18:47:08 -0700 (PDT) Date: Tue, 17 May 2022 19:46:29 -0600 In-Reply-To: <20220518014632.922072-1-yuzhao@google.com> Message-Id: <20220518014632.922072-11-yuzhao@google.com> Mime-Version: 1.0 References: <20220518014632.922072-1-yuzhao@google.com> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog Subject: [PATCH v11 10/14] mm: multi-gen LRU: kill switch From: Yu Zhao To: Andrew Morton , linux-mm@kvack.org Cc: Andi Kleen , Aneesh Kumar , Catalin Marinas , Dave Hansen , Hillf Danton , Jens Axboe , Johannes Weiner , Jonathan Corbet , Linus Torvalds , Matthew Wilcox , Mel Gorman , Michael Larabel , Michal Hocko , Mike Rapoport , Peter Zijlstra , Tejun Heo , Vlastimil Babka , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, page-reclaim@google.com, Yu Zhao , Brian Geffon , Jan Alexander Steffens , Oleksandr Natalenko , Steven Barrett , Suleiman Souhlal , Daniel Byrne , Donald Carr , "=?UTF-8?q?Holger=20Hoffst=C3=A4tte?=" , Konstantin Kharlamov , Shuang Zhai , Sofia Trinh , Vaibhav Jain Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Add /sys/kernel/mm/lru_gen/enabled as a kill switch. Components that can be disabled include: 0x0001: the multi-gen LRU core 0x0002: walking page table, when arch_has_hw_pte_young() returns true 0x0004: clearing the accessed bit in non-leaf PMD entries, when CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG=3Dy [yYnN]: apply to all the components above E.g., echo y >/sys/kernel/mm/lru_gen/enabled cat /sys/kernel/mm/lru_gen/enabled 0x0007 echo 5 >/sys/kernel/mm/lru_gen/enabled cat /sys/kernel/mm/lru_gen/enabled 0x0005 NB: the page table walks happen on the scale of seconds under heavy memory pressure, in which case the mmap_lock contention is a lesser concern, compared with the LRU lock contention and the I/O congestion. So far the only well-known case of the mmap_lock contention happens on Android, due to Scudo [1] which allocates several thousand VMAs for merely a few hundred MBs. The SPF and the Maple Tree also have provided their own assessments [2][3]. However, if walking page tables does worsen the mmap_lock contention, the kill switch can be used to disable it. In this case the multi-gen LRU will suffer a minor performance degradation, as shown previously. Clearing the accessed bit in non-leaf PMD entries can also be disabled, since this behavior was not tested on x86 varieties other than Intel and AMD. [1] https://source.android.com/devices/tech/debug/scudo [2] https://lore.kernel.org/r/20220128131006.67712-1-michel@lespinasse.org/ [3] https://lore.kernel.org/r/20220426150616.3937571-1-Liam.Howlett@oracle.= com/ Signed-off-by: Yu Zhao Acked-by: Brian Geffon Acked-by: Jan Alexander Steffens (heftig) Acked-by: Oleksandr Natalenko Acked-by: Steven Barrett Acked-by: Suleiman Souhlal Tested-by: Daniel Byrne Tested-by: Donald Carr Tested-by: Holger Hoffst=C3=A4tte Tested-by: Konstantin Kharlamov Tested-by: Shuang Zhai Tested-by: Sofia Trinh Tested-by: Vaibhav Jain --- include/linux/cgroup.h | 15 +- include/linux/mm_inline.h | 12 +- include/linux/mmzone.h | 9 ++ kernel/cgroup/cgroup-internal.h | 1 - mm/Kconfig | 6 + mm/vmscan.c | 238 +++++++++++++++++++++++++++++++- 6 files changed, 272 insertions(+), 9 deletions(-) diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index 0d1ada8968d7..1bc0cabf993f 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -432,6 +432,18 @@ static inline void cgroup_put(struct cgroup *cgrp) css_put(&cgrp->self); } =20 +extern struct mutex cgroup_mutex; + +static inline void cgroup_lock(void) +{ + mutex_lock(&cgroup_mutex); +} + +static inline void cgroup_unlock(void) +{ + mutex_unlock(&cgroup_mutex); +} + /** * task_css_set_check - obtain a task's css_set with extra access conditio= ns * @task: the task to obtain css_set for @@ -446,7 +458,6 @@ static inline void cgroup_put(struct cgroup *cgrp) * as locks used during the cgroup_subsys::attach() methods. */ #ifdef CONFIG_PROVE_RCU -extern struct mutex cgroup_mutex; extern spinlock_t css_set_lock; #define task_css_set_check(task, __c) \ rcu_dereference_check((task)->cgroups, \ @@ -708,6 +719,8 @@ struct cgroup; static inline u64 cgroup_id(const struct cgroup *cgrp) { return 1; } static inline void css_get(struct cgroup_subsys_state *css) {} static inline void css_put(struct cgroup_subsys_state *css) {} +static inline void cgroup_lock(void) {} +static inline void cgroup_unlock(void) {} static inline int cgroup_attach_task_all(struct task_struct *from, struct task_struct *t) { return 0; } static inline int cgroupstats_build(struct cgroupstats *stats, diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 85fe78832436..d3d7b7bb297d 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -106,7 +106,15 @@ static __always_inline enum lru_list folio_lru_list(st= ruct folio *folio) =20 static inline bool lru_gen_enabled(void) { - return true; +#ifdef CONFIG_LRU_GEN_ENABLED + DECLARE_STATIC_KEY_TRUE(lru_gen_caps[NR_LRU_GEN_CAPS]); + + return static_branch_likely(&lru_gen_caps[LRU_GEN_CORE]); +#else + DECLARE_STATIC_KEY_FALSE(lru_gen_caps[NR_LRU_GEN_CAPS]); + + return static_branch_unlikely(&lru_gen_caps[LRU_GEN_CORE]); +#endif } =20 static inline bool lru_gen_in_fault(void) @@ -219,7 +227,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lru= vec, struct folio *folio, =20 VM_WARN_ON_ONCE_FOLIO(gen !=3D -1, folio); =20 - if (folio_test_unevictable(folio)) + if (folio_test_unevictable(folio) || !lrugen->enabled) return false; /* * There are three common cases for this page: diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 6aa715a2fea1..372a7f5a6829 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -386,6 +386,13 @@ enum { LRU_GEN_FILE, }; =20 +enum { + LRU_GEN_CORE, + LRU_GEN_MM_WALK, + LRU_GEN_NONLEAF_YOUNG, + NR_LRU_GEN_CAPS +}; + #define MIN_LRU_BATCH BITS_PER_LONG #define MAX_LRU_BATCH (MIN_LRU_BATCH * 128) =20 @@ -427,6 +434,8 @@ struct lru_gen_struct { /* can be modified without holding the LRU lock */ atomic_long_t evicted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS]; atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS]; + /* whether the multi-gen LRU is enabled */ + bool enabled; }; =20 enum { diff --git a/kernel/cgroup/cgroup-internal.h b/kernel/cgroup/cgroup-interna= l.h index 6e36e854b512..929ed3bf1a7c 100644 --- a/kernel/cgroup/cgroup-internal.h +++ b/kernel/cgroup/cgroup-internal.h @@ -165,7 +165,6 @@ struct cgroup_mgctx { #define DEFINE_CGROUP_MGCTX(name) \ struct cgroup_mgctx name =3D CGROUP_MGCTX_INIT(name) =20 -extern struct mutex cgroup_mutex; extern spinlock_t css_set_lock; extern struct cgroup_subsys *cgroup_subsys[]; extern struct list_head cgroup_roots; diff --git a/mm/Kconfig b/mm/Kconfig index 0aeacbd3361c..426ea5f57d88 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -918,6 +918,12 @@ config LRU_GEN help A high performance LRU implementation to overcommit memory. =20 +config LRU_GEN_ENABLED + bool "Enable by default" + depends on LRU_GEN + help + This option enables the multi-gen LRU by default. + config LRU_GEN_STATS bool "Full stats for debugging" depends on LRU_GEN diff --git a/mm/vmscan.c b/mm/vmscan.c index a7e768675707..09e2c3e6f2be 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -52,6 +52,7 @@ #include #include #include +#include =20 #include #include @@ -2989,6 +2990,12 @@ static bool can_age_anon_pages(struct pglist_data *p= gdat, =20 #ifdef CONFIG_LRU_GEN =20 +#ifdef CONFIG_LRU_GEN_ENABLED +DEFINE_STATIC_KEY_ARRAY_TRUE(lru_gen_caps, NR_LRU_GEN_CAPS); +#else +DEFINE_STATIC_KEY_ARRAY_FALSE(lru_gen_caps, NR_LRU_GEN_CAPS); +#endif + /*************************************************************************= ***** * shorthand helpers *************************************************************************= *****/ @@ -3009,6 +3016,15 @@ static bool can_age_anon_pages(struct pglist_data *p= gdat, for ((type) =3D 0; (type) < ANON_AND_FILE; (type)++) \ for ((zone) =3D 0; (zone) < MAX_NR_ZONES; (zone)++) =20 +static bool get_cap(int cap) +{ +#ifdef CONFIG_LRU_GEN_ENABLED + return static_branch_likely(&lru_gen_caps[cap]); +#else + return static_branch_unlikely(&lru_gen_caps[cap]); +#endif +} + static struct lruvec *get_lruvec(struct mem_cgroup *memcg, int nid) { struct pglist_data *pgdat =3D NODE_DATA(nid); @@ -3816,7 +3832,8 @@ static void walk_pmd_range_locked(pud_t *pud, unsigne= d long next, struct vm_area goto next; =20 if (!pmd_trans_huge(pmd[i])) { - if (IS_ENABLED(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG)) + if (IS_ENABLED(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG) && + get_cap(LRU_GEN_NONLEAF_YOUNG)) pmdp_test_and_clear_young(vma, addr, pmd + i); goto next; } @@ -3926,10 +3943,12 @@ static void walk_pmd_range(pud_t *pud, unsigned lon= g start, unsigned long end, walk->mm_stats[MM_PMD_TOTAL]++; =20 #ifdef CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG - if (!pmd_young(val)) - continue; + if (get_cap(LRU_GEN_NONLEAF_YOUNG)) { + if (!pmd_young(val)) + continue; =20 - walk_pmd_range_locked(pud, addr, vma, args, &pos); + walk_pmd_range_locked(pud, addr, vma, args, &pos); + } #endif if (!walk->force_scan && !test_bloom_filter(walk->lruvec, walk->max_seq,= pmd + i)) continue; @@ -4162,7 +4181,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec,= unsigned long max_seq, * handful of PTEs. Spreading the work out over a period of time usually * is less efficient, but it avoids bursty page faults. */ - if (!arch_has_hw_pte_young()) { + if (!(arch_has_hw_pte_young() && get_cap(LRU_GEN_MM_WALK))) { success =3D iterate_mm_list_nowalk(lruvec, max_seq); goto done; } @@ -4895,6 +4914,211 @@ static void lru_gen_shrink_lruvec(struct lruvec *lr= uvec, struct scan_control *sc blk_finish_plug(&plug); } =20 +/*************************************************************************= ***** + * state change + *************************************************************************= *****/ + +static bool __maybe_unused state_is_valid(struct lruvec *lruvec) +{ + struct lru_gen_struct *lrugen =3D &lruvec->lrugen; + + if (lrugen->enabled) { + enum lru_list lru; + + for_each_evictable_lru(lru) { + if (!list_empty(&lruvec->lists[lru])) + return false; + } + } else { + int gen, type, zone; + + for_each_gen_type_zone(gen, type, zone) { + if (!list_empty(&lrugen->lists[gen][type][zone])) + return false; + + /* unlikely but not a bug when reset_batch_size() is pending */ + VM_WARN_ON_ONCE(lrugen->nr_pages[gen][type][zone]); + } + } + + return true; +} + +static bool fill_evictable(struct lruvec *lruvec) +{ + enum lru_list lru; + int remaining =3D MAX_LRU_BATCH; + + for_each_evictable_lru(lru) { + int type =3D is_file_lru(lru); + bool active =3D is_active_lru(lru); + struct list_head *head =3D &lruvec->lists[lru]; + + while (!list_empty(head)) { + bool success; + struct folio *folio =3D lru_to_folio(head); + + VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); + VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio) !=3D active, folio); + VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) !=3D type, folio); + VM_WARN_ON_ONCE_FOLIO(folio_lru_gen(folio) !=3D -1, folio); + + lruvec_del_folio(lruvec, folio); + success =3D lru_gen_add_folio(lruvec, folio, false); + VM_WARN_ON_ONCE(!success); + + if (!--remaining) + return false; + } + } + + return true; +} + +static bool drain_evictable(struct lruvec *lruvec) +{ + int gen, type, zone; + int remaining =3D MAX_LRU_BATCH; + + for_each_gen_type_zone(gen, type, zone) { + struct list_head *head =3D &lruvec->lrugen.lists[gen][type][zone]; + + while (!list_empty(head)) { + bool success; + struct folio *folio =3D lru_to_folio(head); + + VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); + VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio); + VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) !=3D type, folio); + VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D zone, folio); + + success =3D lru_gen_del_folio(lruvec, folio, false); + VM_WARN_ON_ONCE(!success); + lruvec_add_folio(lruvec, folio); + + if (!--remaining) + return false; + } + } + + return true; +} + +static void lru_gen_change_state(bool enable) +{ + static DEFINE_MUTEX(state_mutex); + + struct mem_cgroup *memcg; + + cgroup_lock(); + cpus_read_lock(); + get_online_mems(); + mutex_lock(&state_mutex); + + if (enable =3D=3D lru_gen_enabled()) + goto unlock; + + if (enable) + static_branch_enable_cpuslocked(&lru_gen_caps[LRU_GEN_CORE]); + else + static_branch_disable_cpuslocked(&lru_gen_caps[LRU_GEN_CORE]); + + memcg =3D mem_cgroup_iter(NULL, NULL, NULL); + do { + int nid; + + for_each_node(nid) { + struct lruvec *lruvec =3D get_lruvec(memcg, nid); + + if (!lruvec) + continue; + + spin_lock_irq(&lruvec->lru_lock); + + VM_WARN_ON_ONCE(!seq_is_valid(lruvec)); + VM_WARN_ON_ONCE(!state_is_valid(lruvec)); + + lruvec->lrugen.enabled =3D enable; + + while (!(enable ? fill_evictable(lruvec) : drain_evictable(lruvec))) { + spin_unlock_irq(&lruvec->lru_lock); + cond_resched(); + spin_lock_irq(&lruvec->lru_lock); + } + + spin_unlock_irq(&lruvec->lru_lock); + } + + cond_resched(); + } while ((memcg =3D mem_cgroup_iter(NULL, memcg, NULL))); +unlock: + mutex_unlock(&state_mutex); + put_online_mems(); + cpus_read_unlock(); + cgroup_unlock(); +} + +/*************************************************************************= ***** + * sysfs interface + *************************************************************************= *****/ + +static ssize_t show_enable(struct kobject *kobj, struct kobj_attribute *at= tr, char *buf) +{ + unsigned int caps =3D 0; + + if (get_cap(LRU_GEN_CORE)) + caps |=3D BIT(LRU_GEN_CORE); + + if (arch_has_hw_pte_young() && get_cap(LRU_GEN_MM_WALK)) + caps |=3D BIT(LRU_GEN_MM_WALK); + + if (IS_ENABLED(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG) && get_cap(LRU_GEN_NONL= EAF_YOUNG)) + caps |=3D BIT(LRU_GEN_NONLEAF_YOUNG); + + return snprintf(buf, PAGE_SIZE, "0x%04x\n", caps); +} + +static ssize_t store_enable(struct kobject *kobj, struct kobj_attribute *a= ttr, + const char *buf, size_t len) +{ + int i; + unsigned int caps; + + if (tolower(*buf) =3D=3D 'n') + caps =3D 0; + else if (tolower(*buf) =3D=3D 'y') + caps =3D -1; + else if (kstrtouint(buf, 0, &caps)) + return -EINVAL; + + for (i =3D 0; i < NR_LRU_GEN_CAPS; i++) { + bool enable =3D caps & BIT(i); + + if (i =3D=3D LRU_GEN_CORE) + lru_gen_change_state(enable); + else if (enable) + static_branch_enable(&lru_gen_caps[i]); + else + static_branch_disable(&lru_gen_caps[i]); + } + + return len; +} + +static struct kobj_attribute lru_gen_enabled_attr =3D __ATTR( + enabled, 0644, show_enable, store_enable +); + +static struct attribute *lru_gen_attrs[] =3D { + &lru_gen_enabled_attr.attr, + NULL +}; + +static struct attribute_group lru_gen_attr_group =3D { + .name =3D "lru_gen", + .attrs =3D lru_gen_attrs, +}; + /*************************************************************************= ***** * initialization *************************************************************************= *****/ @@ -4905,6 +5129,7 @@ void lru_gen_init_lruvec(struct lruvec *lruvec) struct lru_gen_struct *lrugen =3D &lruvec->lrugen; =20 lrugen->max_seq =3D MIN_NR_GENS + 1; + lrugen->enabled =3D lru_gen_enabled(); =20 for_each_gen_type_zone(gen, type, zone) INIT_LIST_HEAD(&lrugen->lists[gen][type][zone]); @@ -4945,6 +5170,9 @@ static int __init init_lru_gen(void) BUILD_BUG_ON(BIT(LRU_GEN_WIDTH) <=3D MAX_NR_GENS); BUILD_BUG_ON(sizeof(MM_STAT_CODES) !=3D NR_MM_STATS + 1); =20 + if (sysfs_create_group(mm_kobj, &lru_gen_attr_group)) + pr_err("lru_gen: failed to create sysfs group\n"); + return 0; }; late_initcall(init_lru_gen); --=20 2.36.0.550.gb090851708-goog From nobody Thu May 7 21:41:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4065BC433F5 for ; Wed, 18 May 2022 01:48:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234082AbiERBsG (ORCPT ); Tue, 17 May 2022 21:48:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58084 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232176AbiERBrU (ORCPT ); Tue, 17 May 2022 21:47:20 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5D49354BF3 for ; Tue, 17 May 2022 18:47:10 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id o7-20020a256b47000000b0064ddc3bea70so699172ybm.4 for ; Tue, 17 May 2022 18:47:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc:content-transfer-encoding; bh=rNx9Wnnb2Q40MxJ2AB9/JnqJGMPM8zlMbTCpT3dj1BA=; b=f6Wu5/emDRNm3TYtxoLVModE1v2Mx1PvCfEpBOqsq/XlyHn9/UOOs77nvwwPtFVJq3 qrFUkNi5NBLe/pBIYKRo2sd6wPItxq8wQFVlYmzUCfvI7M4regAPS6cdxxH6GbcyzC7s pKvcQC7KEQEotcCZ+1xkSH/D3SFaay9M2qrf6cUg4Gv0sWPkUyv6sWJH6O/1xx1wc7xX 2wyH2dzmstBlBSK58XeKDFzZo/5XZ5e62feVsIhlFHehK8hd1JPDsPNJ9nzl/MZ070WZ 9tiMBh18sOlIL8wH0W8wwikED9KXhFQkcZQxa6EH8aBNEeV+I1y+qJTYRmHLKtvRGLGZ WDlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc:content-transfer-encoding; bh=rNx9Wnnb2Q40MxJ2AB9/JnqJGMPM8zlMbTCpT3dj1BA=; b=0RQ2/Xgd+tkN/YE6mwwMmVuwNTJT+BLzPzXfi74xRRrqKhjywFff2jwqfDrA83spZq nJblH2KKvRLZJhNfPVKWTzWpHgNzm35ycCgJUhY8Q0DQiggVjNCWVH2OwKYjxeGsg2To edqeLQKCGcBcMXGUiEdHc2yLIm8Fo1DOlbaG7KvMbNUt1L0EVTkv97yBgzLpgN52gsFS M/RBcsv06mqdcKw9aTdVPayNBkrI4JtjaifHSoVMLFu7b3A7rS+3VacFuzsUKncJRDQT d3qBfHpHlTYXFoPLz4H3OXBOuNAey8Mni7SmiXxBuRmGrCbTwDS7RDft2ELBVP6BA5Bo Yenw== X-Gm-Message-State: AOAM532MTSyuUstySHwROs8Ib8+UlOF65N2FTnqOVBwtmySx38+bK4h8 VgIofGSKRdyjMicFimWdmgg5oh1Gahk= X-Google-Smtp-Source: ABdhPJw/GVYSFwqTPa9IB5eZEw8fBz/qkDDmWwgcQxKWC3IhU7UZHsf9NiWmIU8OqkuOn+EV5gSLDXe4G+Q= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f7bc:90c9:d86e:4ea8]) (user=yuzhao job=sendgmr) by 2002:a81:990f:0:b0:2f8:c347:d11a with SMTP id q15-20020a81990f000000b002f8c347d11amr28245783ywg.507.1652838429715; Tue, 17 May 2022 18:47:09 -0700 (PDT) Date: Tue, 17 May 2022 19:46:30 -0600 In-Reply-To: <20220518014632.922072-1-yuzhao@google.com> Message-Id: <20220518014632.922072-12-yuzhao@google.com> Mime-Version: 1.0 References: <20220518014632.922072-1-yuzhao@google.com> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog Subject: [PATCH v11 11/14] mm: multi-gen LRU: thrashing prevention From: Yu Zhao To: Andrew Morton , linux-mm@kvack.org Cc: Andi Kleen , Aneesh Kumar , Catalin Marinas , Dave Hansen , Hillf Danton , Jens Axboe , Johannes Weiner , Jonathan Corbet , Linus Torvalds , Matthew Wilcox , Mel Gorman , Michael Larabel , Michal Hocko , Mike Rapoport , Peter Zijlstra , Tejun Heo , Vlastimil Babka , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, page-reclaim@google.com, Yu Zhao , Brian Geffon , Jan Alexander Steffens , Oleksandr Natalenko , Steven Barrett , Suleiman Souhlal , Daniel Byrne , Donald Carr , "=?UTF-8?q?Holger=20Hoffst=C3=A4tte?=" , Konstantin Kharlamov , Shuang Zhai , Sofia Trinh , Vaibhav Jain Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Add /sys/kernel/mm/lru_gen/min_ttl_ms for thrashing prevention, as requested by many desktop users [1]. When set to value N, it prevents the working set of N milliseconds from getting evicted. The OOM killer is triggered if this working set cannot be kept in memory. Based on the average human detectable lag (~100ms), N=3D1000 usually eliminates intolerable lags due to thrashing. Larger values like N=3D3000 make lags less noticeable at the risk of premature OOM kills. Compared with the size-based approach, e.g., [2], this time-based approach has the following advantages: 1. It is easier to configure because it is agnostic to applications and memory sizes. 2. It is more reliable because it is directly wired to the OOM killer. [1] https://lore.kernel.org/r/Ydza%2FzXKY9ATRoh6@google.com/ [2] https://lore.kernel.org/r/20101028191523.GA14972@google.com/ Signed-off-by: Yu Zhao Acked-by: Brian Geffon Acked-by: Jan Alexander Steffens (heftig) Acked-by: Oleksandr Natalenko Acked-by: Steven Barrett Acked-by: Suleiman Souhlal Tested-by: Daniel Byrne Tested-by: Donald Carr Tested-by: Holger Hoffst=C3=A4tte Tested-by: Konstantin Kharlamov Tested-by: Shuang Zhai Tested-by: Sofia Trinh Tested-by: Vaibhav Jain --- include/linux/mmzone.h | 2 ++ mm/vmscan.c | 68 +++++++++++++++++++++++++++++++++++++++--- 2 files changed, 66 insertions(+), 4 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 372a7f5a6829..b2e8869be7cf 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -421,6 +421,8 @@ struct lru_gen_struct { unsigned long max_seq; /* the eviction increments the oldest generation numbers */ unsigned long min_seq[ANON_AND_FILE]; + /* the birth time of each generation in jiffies */ + unsigned long timestamps[MAX_NR_GENS]; /* the multi-gen LRU lists */ struct list_head lists[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES]; /* the sizes of the above lists */ diff --git a/mm/vmscan.c b/mm/vmscan.c index 09e2c3e6f2be..823c55e17e64 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4159,6 +4159,7 @@ static void inc_max_seq(struct lruvec *lruvec, bool c= an_swap) for (type =3D 0; type < ANON_AND_FILE; type++) reset_ctrl_pos(lruvec, type, false); =20 + WRITE_ONCE(lrugen->timestamps[next], jiffies); /* make sure preceding modifications appear */ smp_store_release(&lrugen->max_seq, lrugen->max_seq + 1); =20 @@ -4281,7 +4282,8 @@ static long get_nr_evictable(struct lruvec *lruvec, u= nsigned long max_seq, return total > 0 ? total : 0; } =20 -static void age_lruvec(struct lruvec *lruvec, struct scan_control *sc) +static bool age_lruvec(struct lruvec *lruvec, struct scan_control *sc, + unsigned long min_ttl) { bool need_aging; long nr_to_scan; @@ -4292,14 +4294,22 @@ static void age_lruvec(struct lruvec *lruvec, struc= t scan_control *sc) =20 VM_WARN_ON_ONCE(sc->memcg_low_reclaim); =20 + if (min_ttl) { + int gen =3D lru_gen_from_seq(min_seq[LRU_GEN_FILE]); + unsigned long birth =3D READ_ONCE(lruvec->lrugen.timestamps[gen]); + + if (time_is_after_jiffies(birth + min_ttl)) + return false; + } + mem_cgroup_calculate_protection(NULL, memcg); =20 if (mem_cgroup_below_min(memcg)) - return; + return false; =20 nr_to_scan =3D get_nr_evictable(lruvec, max_seq, min_seq, swappiness, &ne= ed_aging); if (!nr_to_scan) - return; + return false; =20 nr_to_scan >>=3D sc->priority; =20 @@ -4308,11 +4318,18 @@ static void age_lruvec(struct lruvec *lruvec, struc= t scan_control *sc) =20 if (nr_to_scan && need_aging) try_to_inc_max_seq(lruvec, max_seq, sc, swappiness); + + return true; } =20 +/* to protect the working set of the last N jiffies */ +static unsigned long lru_gen_min_ttl __read_mostly; + static void lru_gen_age_node(struct pglist_data *pgdat, struct scan_contro= l *sc) { struct mem_cgroup *memcg; + bool success =3D false; + unsigned long min_ttl =3D READ_ONCE(lru_gen_min_ttl); =20 VM_WARN_ON_ONCE(!current_is_kswapd()); =20 @@ -4338,12 +4355,28 @@ static void lru_gen_age_node(struct pglist_data *pg= dat, struct scan_control *sc) do { struct lruvec *lruvec =3D mem_cgroup_lruvec(memcg, pgdat); =20 - age_lruvec(lruvec, sc); + if (age_lruvec(lruvec, sc, min_ttl)) + success =3D true; =20 cond_resched(); } while ((memcg =3D mem_cgroup_iter(NULL, memcg, NULL))); =20 current->reclaim_state->mm_walk =3D NULL; + + /* + * The main goal is to OOM kill if every generation from all memcgs is + * younger than min_ttl. However, another theoretical possibility is all + * memcgs are either below min or empty. + */ + if (!success && !sc->order && mutex_trylock(&oom_lock)) { + struct oom_control oc =3D { + .gfp_mask =3D sc->gfp_mask, + }; + + out_of_memory(&oc); + + mutex_unlock(&oom_lock); + } } =20 /* @@ -5062,6 +5095,28 @@ static void lru_gen_change_state(bool enable) * sysfs interface *************************************************************************= *****/ =20 +static ssize_t show_min_ttl(struct kobject *kobj, struct kobj_attribute *a= ttr, char *buf) +{ + return sprintf(buf, "%u\n", jiffies_to_msecs(READ_ONCE(lru_gen_min_ttl))); +} + +static ssize_t store_min_ttl(struct kobject *kobj, struct kobj_attribute *= attr, + const char *buf, size_t len) +{ + unsigned int msecs; + + if (kstrtouint(buf, 0, &msecs)) + return -EINVAL; + + WRITE_ONCE(lru_gen_min_ttl, msecs_to_jiffies(msecs)); + + return len; +} + +static struct kobj_attribute lru_gen_min_ttl_attr =3D __ATTR( + min_ttl_ms, 0644, show_min_ttl, store_min_ttl +); + static ssize_t show_enable(struct kobject *kobj, struct kobj_attribute *at= tr, char *buf) { unsigned int caps =3D 0; @@ -5110,6 +5165,7 @@ static struct kobj_attribute lru_gen_enabled_attr =3D= __ATTR( ); =20 static struct attribute *lru_gen_attrs[] =3D { + &lru_gen_min_ttl_attr.attr, &lru_gen_enabled_attr.attr, NULL }; @@ -5125,12 +5181,16 @@ static struct attribute_group lru_gen_attr_group = =3D { =20 void lru_gen_init_lruvec(struct lruvec *lruvec) { + int i; int gen, type, zone; struct lru_gen_struct *lrugen =3D &lruvec->lrugen; =20 lrugen->max_seq =3D MIN_NR_GENS + 1; lrugen->enabled =3D lru_gen_enabled(); =20 + for (i =3D 0; i <=3D MIN_NR_GENS + 1; i++) + lrugen->timestamps[i] =3D jiffies; + for_each_gen_type_zone(gen, type, zone) INIT_LIST_HEAD(&lrugen->lists[gen][type][zone]); =20 --=20 2.36.0.550.gb090851708-goog From nobody Thu May 7 21:41:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E05BAC433F5 for ; Wed, 18 May 2022 01:48:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234027AbiERBsZ (ORCPT ); Tue, 17 May 2022 21:48:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58026 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233949AbiERBrV (ORCPT ); Tue, 17 May 2022 21:47:21 -0400 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7649D54F89 for ; Tue, 17 May 2022 18:47:12 -0700 (PDT) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-2f4dfd09d7fso8828667b3.0 for ; Tue, 17 May 2022 18:47:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc:content-transfer-encoding; bh=W1lmCBMSvh8WM0XdwNyp+PodjxHLyHRsBa8QQZ0AbFw=; b=WvVuu9D+7nu5qVB8kWHqWcYHQbsJXYb8i6L7M+/DrCSC3GN8OWxaa1DizXgf3xnJHk fINbVn4vyZ+HHizxwYyklCrtZpgPwGn/iXVnNmj3tdRYejslmJNHRnltXf0eTNUkPhVp 53hGUOxOYFsZoTAKx6Jns5j5nJuzgmQlW4j/Fo5aiV5yxTE2g1rqp7ewPQH5u7cIk2LA iPYvljtc464NjJPnlOpsNyj0iBzRPWSV33Cuy0ChBxojzIYeXCiYSDrT7zCZwkTIITvG rlxySrkZ7IiIyK0h8z8aIC7pRAwXwsbZdVx59yNzngzWnm2cAP0ZmlzB7HdF4LFhEP30 gvSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc:content-transfer-encoding; bh=W1lmCBMSvh8WM0XdwNyp+PodjxHLyHRsBa8QQZ0AbFw=; b=WJBMoNT3HsrOr0dYIGJzLfgZvhZ+M4bg3ayj5iTtHeDYS/6RtCmB9bETqbEMWWzvYW +yMT5TLyhVwcAFtY9CSOQalzL7b5Js5bHRP/G9iHSFPascoG4ByWH70b7uwct18mviAs ULA7CxQfv+lkcdV7gMvirSQ0gfVoJOTKwkIIk0oo4xQ5RcP7dY1ovM4xaebqYtb+hxSc lRHFthWL3GKUw6MHPZNEg/BiyNobqRQgWR6xzVjX5BGYVzbswJgTZ1ypSubJk+qIU65u F2W6EKRjg+7tfHu5BmU82dAg67Quuy3dWH57dZb0V17CJSCpszwFCFgZtan8T/B0pc1B bGIA== X-Gm-Message-State: AOAM530p0N6xXr9BgMQisddrxULmX0dinQ1iO/N38s0WhMjQUIrLUOX4 FBMO0F7Q7Yz9aD8cqJA4J225PvVZ/fY= X-Google-Smtp-Source: ABdhPJylNicMwjJM0mSLGeu5WFwH+Uvqs4PoDUUOGjHkU7Hmd7go+9S3O62WXfmdYFAyReYmXT0hT8SNDRY= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f7bc:90c9:d86e:4ea8]) (user=yuzhao job=sendgmr) by 2002:a81:5212:0:b0:2fe:e667:ee6e with SMTP id g18-20020a815212000000b002fee667ee6emr16390918ywb.155.1652838431235; Tue, 17 May 2022 18:47:11 -0700 (PDT) Date: Tue, 17 May 2022 19:46:31 -0600 In-Reply-To: <20220518014632.922072-1-yuzhao@google.com> Message-Id: <20220518014632.922072-13-yuzhao@google.com> Mime-Version: 1.0 References: <20220518014632.922072-1-yuzhao@google.com> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog Subject: [PATCH v11 12/14] mm: multi-gen LRU: debugfs interface From: Yu Zhao To: Andrew Morton , linux-mm@kvack.org Cc: Andi Kleen , Aneesh Kumar , Catalin Marinas , Dave Hansen , Hillf Danton , Jens Axboe , Johannes Weiner , Jonathan Corbet , Linus Torvalds , Matthew Wilcox , Mel Gorman , Michael Larabel , Michal Hocko , Mike Rapoport , Peter Zijlstra , Tejun Heo , Vlastimil Babka , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, page-reclaim@google.com, Yu Zhao , Brian Geffon , Jan Alexander Steffens , Oleksandr Natalenko , Steven Barrett , Suleiman Souhlal , Daniel Byrne , Donald Carr , "=?UTF-8?q?Holger=20Hoffst=C3=A4tte?=" , Konstantin Kharlamov , Shuang Zhai , Sofia Trinh , Vaibhav Jain Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Add /sys/kernel/debug/lru_gen for working set estimation and proactive reclaim. These techniques are commonly used to optimize job scheduling (bin packing) in data centers [1][2]. Compared with the page table-based approach and the PFN-based approach, this lruvec-based approach has the following advantages: 1. It offers better choices because it is aware of memcgs, NUMA nodes, shared mappings and unmapped page cache. 2. It is more scalable because it is O(nr_hot_pages), whereas the PFN-based approach is O(nr_total_pages). Add /sys/kernel/debug/lru_gen_full for debugging. [1] https://dl.acm.org/doi/10.1145/3297858.3304053 [2] https://dl.acm.org/doi/10.1145/3503222.3507731 Signed-off-by: Yu Zhao Acked-by: Brian Geffon Acked-by: Jan Alexander Steffens (heftig) Acked-by: Oleksandr Natalenko Acked-by: Steven Barrett Acked-by: Suleiman Souhlal Tested-by: Daniel Byrne Tested-by: Donald Carr Tested-by: Holger Hoffst=C3=A4tte Tested-by: Konstantin Kharlamov Tested-by: Shuang Zhai Tested-by: Sofia Trinh Tested-by: Vaibhav Jain --- include/linux/nodemask.h | 1 + mm/vmscan.c | 403 ++++++++++++++++++++++++++++++++++++++- 2 files changed, 394 insertions(+), 10 deletions(-) diff --git a/include/linux/nodemask.h b/include/linux/nodemask.h index 567c3ddba2c4..90840c459abc 100644 --- a/include/linux/nodemask.h +++ b/include/linux/nodemask.h @@ -486,6 +486,7 @@ static inline int num_node_state(enum node_states state) #define first_online_node 0 #define first_memory_node 0 #define next_online_node(nid) (MAX_NUMNODES) +#define next_memory_node(nid) (MAX_NUMNODES) #define nr_node_ids 1U #define nr_online_nodes 1U =20 diff --git a/mm/vmscan.c b/mm/vmscan.c index 823c55e17e64..aef71c45c424 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -53,6 +53,7 @@ #include #include #include +#include =20 #include #include @@ -4063,12 +4064,39 @@ static void free_mm_walk(struct lru_gen_mm_walk *wa= lk) kfree(walk); } =20 -static void inc_min_seq(struct lruvec *lruvec, int type) +static bool inc_min_seq(struct lruvec *lruvec, int type, bool can_swap) { + int zone; + int remaining =3D MAX_LRU_BATCH; struct lru_gen_struct *lrugen =3D &lruvec->lrugen; + int new_gen, old_gen =3D lru_gen_from_seq(lrugen->min_seq[type]); =20 + if (type =3D=3D LRU_GEN_ANON && !can_swap) + goto done; + + for (zone =3D 0; zone < MAX_NR_ZONES; zone++) { + struct list_head *head =3D &lrugen->lists[old_gen][type][zone]; + + while (!list_empty(head)) { + struct folio *folio =3D lru_to_folio(head); + + VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); + VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio); + VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) !=3D type, folio); + VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D zone, folio); + + new_gen =3D folio_inc_gen(lruvec, folio, false); + list_move_tail(&folio->lru, &lrugen->lists[new_gen][type][zone]); + + if (!--remaining) + return false; + } + } +done: reset_ctrl_pos(lruvec, type, true); WRITE_ONCE(lrugen->min_seq[type], lrugen->min_seq[type] + 1); + + return true; } =20 static bool try_to_inc_min_seq(struct lruvec *lruvec, bool can_swap) @@ -4114,7 +4142,7 @@ static bool try_to_inc_min_seq(struct lruvec *lruvec,= bool can_swap) return success; } =20 -static void inc_max_seq(struct lruvec *lruvec, bool can_swap) +static void inc_max_seq(struct lruvec *lruvec, bool can_swap, bool force_s= can) { int prev, next; int type, zone; @@ -4128,9 +4156,13 @@ static void inc_max_seq(struct lruvec *lruvec, bool = can_swap) if (get_nr_gens(lruvec, type) !=3D MAX_NR_GENS) continue; =20 - VM_WARN_ON_ONCE(type =3D=3D LRU_GEN_FILE || can_swap); + VM_WARN_ON_ONCE(!force_scan && (type =3D=3D LRU_GEN_FILE || can_swap)); =20 - inc_min_seq(lruvec, type); + while (!inc_min_seq(lruvec, type, can_swap)) { + spin_unlock_irq(&lruvec->lru_lock); + cond_resched(); + spin_lock_irq(&lruvec->lru_lock); + } } =20 /* @@ -4167,7 +4199,7 @@ static void inc_max_seq(struct lruvec *lruvec, bool c= an_swap) } =20 static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_se= q, - struct scan_control *sc, bool can_swap) + struct scan_control *sc, bool can_swap, bool force_scan) { bool success; struct lru_gen_mm_walk *walk; @@ -4182,7 +4214,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec,= unsigned long max_seq, * handful of PTEs. Spreading the work out over a period of time usually * is less efficient, but it avoids bursty page faults. */ - if (!(arch_has_hw_pte_young() && get_cap(LRU_GEN_MM_WALK))) { + if (!force_scan && !(arch_has_hw_pte_young() && get_cap(LRU_GEN_MM_WALK))= ) { success =3D iterate_mm_list_nowalk(lruvec, max_seq); goto done; } @@ -4196,7 +4228,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec,= unsigned long max_seq, walk->lruvec =3D lruvec; walk->max_seq =3D max_seq; walk->can_swap =3D can_swap; - walk->force_scan =3D false; + walk->force_scan =3D force_scan; =20 do { success =3D iterate_mm_list(lruvec, walk, &mm); @@ -4218,7 +4250,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec,= unsigned long max_seq, =20 VM_WARN_ON_ONCE(max_seq !=3D READ_ONCE(lrugen->max_seq)); =20 - inc_max_seq(lruvec, can_swap); + inc_max_seq(lruvec, can_swap, force_scan); /* either this sees any waiters or they will see updated max_seq */ if (wq_has_sleeper(&lruvec->mm_state.wait)) wake_up_all(&lruvec->mm_state.wait); @@ -4317,7 +4349,7 @@ static bool age_lruvec(struct lruvec *lruvec, struct = scan_control *sc, nr_to_scan++; =20 if (nr_to_scan && need_aging) - try_to_inc_max_seq(lruvec, max_seq, sc, swappiness); + try_to_inc_max_seq(lruvec, max_seq, sc, swappiness, false); =20 return true; } @@ -4887,7 +4919,7 @@ static long get_nr_to_scan(struct lruvec *lruvec, str= uct scan_control *sc, bool return 0; } =20 - if (try_to_inc_max_seq(lruvec, max_seq, sc, can_swap)) + if (try_to_inc_max_seq(lruvec, max_seq, sc, can_swap, false)) return nr_to_scan; =20 return min_seq[!can_swap] + MIN_NR_GENS <=3D max_seq ? nr_to_scan : 0; @@ -5175,6 +5207,354 @@ static struct attribute_group lru_gen_attr_group = =3D { .attrs =3D lru_gen_attrs, }; =20 +/*************************************************************************= ***** + * debugfs interface + *************************************************************************= *****/ + +static void *lru_gen_seq_start(struct seq_file *m, loff_t *pos) +{ + struct mem_cgroup *memcg; + loff_t nr_to_skip =3D *pos; + + m->private =3D kvmalloc(PATH_MAX, GFP_KERNEL); + if (!m->private) + return ERR_PTR(-ENOMEM); + + memcg =3D mem_cgroup_iter(NULL, NULL, NULL); + do { + int nid; + + for_each_node_state(nid, N_MEMORY) { + if (!nr_to_skip--) + return get_lruvec(memcg, nid); + } + } while ((memcg =3D mem_cgroup_iter(NULL, memcg, NULL))); + + return NULL; +} + +static void lru_gen_seq_stop(struct seq_file *m, void *v) +{ + if (!IS_ERR_OR_NULL(v)) + mem_cgroup_iter_break(NULL, lruvec_memcg(v)); + + kvfree(m->private); + m->private =3D NULL; +} + +static void *lru_gen_seq_next(struct seq_file *m, void *v, loff_t *pos) +{ + int nid =3D lruvec_pgdat(v)->node_id; + struct mem_cgroup *memcg =3D lruvec_memcg(v); + + ++*pos; + + nid =3D next_memory_node(nid); + if (nid =3D=3D MAX_NUMNODES) { + memcg =3D mem_cgroup_iter(NULL, memcg, NULL); + if (!memcg) + return NULL; + + nid =3D first_memory_node; + } + + return get_lruvec(memcg, nid); +} + +static void lru_gen_seq_show_full(struct seq_file *m, struct lruvec *lruve= c, + unsigned long max_seq, unsigned long *min_seq, + unsigned long seq) +{ + int i; + int type, tier; + int hist =3D lru_hist_from_seq(seq); + struct lru_gen_struct *lrugen =3D &lruvec->lrugen; + + for (tier =3D 0; tier < MAX_NR_TIERS; tier++) { + seq_printf(m, " %10d", tier); + for (type =3D 0; type < ANON_AND_FILE; type++) { + unsigned long n[3] =3D {}; + + if (seq =3D=3D max_seq) { + n[0] =3D READ_ONCE(lrugen->avg_refaulted[type][tier]); + n[1] =3D READ_ONCE(lrugen->avg_total[type][tier]); + + seq_printf(m, " %10luR %10luT %10lu ", n[0], n[1], n[2]); + } else if (seq =3D=3D min_seq[type] || NR_HIST_GENS > 1) { + n[0] =3D atomic_long_read(&lrugen->refaulted[hist][type][tier]); + n[1] =3D atomic_long_read(&lrugen->evicted[hist][type][tier]); + if (tier) + n[2] =3D READ_ONCE(lrugen->protected[hist][type][tier - 1]); + + seq_printf(m, " %10lur %10lue %10lup", n[0], n[1], n[2]); + } else + seq_puts(m, " 0 0 0 "); + } + seq_putc(m, '\n'); + } + + seq_puts(m, " "); + for (i =3D 0; i < NR_MM_STATS; i++) { + if (seq =3D=3D max_seq && NR_HIST_GENS =3D=3D 1) + seq_printf(m, " %10lu%c", READ_ONCE(lruvec->mm_state.stats[hist][i]), + toupper(MM_STAT_CODES[i])); + else if (seq !=3D max_seq && NR_HIST_GENS > 1) + seq_printf(m, " %10lu%c", READ_ONCE(lruvec->mm_state.stats[hist][i]), + MM_STAT_CODES[i]); + else + seq_puts(m, " 0 "); + } + seq_putc(m, '\n'); +} + +static int lru_gen_seq_show(struct seq_file *m, void *v) +{ + unsigned long seq; + bool full =3D !debugfs_real_fops(m->file)->write; + struct lruvec *lruvec =3D v; + struct lru_gen_struct *lrugen =3D &lruvec->lrugen; + int nid =3D lruvec_pgdat(lruvec)->node_id; + struct mem_cgroup *memcg =3D lruvec_memcg(lruvec); + DEFINE_MAX_SEQ(lruvec); + DEFINE_MIN_SEQ(lruvec); + + if (nid =3D=3D first_memory_node) { + const char *path =3D memcg ? m->private : ""; + +#ifdef CONFIG_MEMCG + if (memcg) + cgroup_path(memcg->css.cgroup, m->private, PATH_MAX); +#endif + seq_printf(m, "memcg %5hu %s\n", mem_cgroup_id(memcg), path); + } + + seq_printf(m, " node %5d\n", nid); + + if (!full) + seq =3D min_seq[LRU_GEN_ANON]; + else if (max_seq >=3D MAX_NR_GENS) + seq =3D max_seq - MAX_NR_GENS + 1; + else + seq =3D 0; + + for (; seq <=3D max_seq; seq++) { + int type, zone; + int gen =3D lru_gen_from_seq(seq); + unsigned long birth =3D READ_ONCE(lruvec->lrugen.timestamps[gen]); + + seq_printf(m, " %10lu %10u", seq, jiffies_to_msecs(jiffies - birth)); + + for (type =3D 0; type < ANON_AND_FILE; type++) { + long size =3D 0; + char mark =3D full && seq < min_seq[type] ? 'x' : ' '; + + for (zone =3D 0; zone < MAX_NR_ZONES; zone++) + size +=3D READ_ONCE(lrugen->nr_pages[gen][type][zone]); + + seq_printf(m, " %10lu%c", max(size, 0L), mark); + } + + seq_putc(m, '\n'); + + if (full) + lru_gen_seq_show_full(m, lruvec, max_seq, min_seq, seq); + } + + return 0; +} + +static const struct seq_operations lru_gen_seq_ops =3D { + .start =3D lru_gen_seq_start, + .stop =3D lru_gen_seq_stop, + .next =3D lru_gen_seq_next, + .show =3D lru_gen_seq_show, +}; + +static int run_aging(struct lruvec *lruvec, unsigned long seq, struct scan= _control *sc, + bool can_swap, bool force_scan) +{ + DEFINE_MAX_SEQ(lruvec); + DEFINE_MIN_SEQ(lruvec); + + if (seq < max_seq) + return 0; + + if (seq > max_seq || (!force_scan && min_seq[!can_swap] + MAX_NR_GENS - 1= <=3D max_seq)) + return -EINVAL; + + try_to_inc_max_seq(lruvec, max_seq, sc, can_swap, force_scan); + + return 0; +} + +static int run_eviction(struct lruvec *lruvec, unsigned long seq, struct s= can_control *sc, + int swappiness, unsigned long nr_to_reclaim) +{ + struct blk_plug plug; + int err =3D -EINTR; + DEFINE_MAX_SEQ(lruvec); + + if (seq + MIN_NR_GENS > max_seq) + return -EINVAL; + + sc->nr_reclaimed =3D 0; + + blk_start_plug(&plug); + + while (!signal_pending(current)) { + DEFINE_MIN_SEQ(lruvec); + + if (seq < min_seq[!swappiness] || sc->nr_reclaimed >=3D nr_to_reclaim || + !evict_folios(lruvec, sc, swappiness, NULL)) { + err =3D 0; + break; + } + + cond_resched(); + } + + blk_finish_plug(&plug); + + return err; +} + +static int run_cmd(char cmd, int memcg_id, int nid, unsigned long seq, + struct scan_control *sc, int swappiness, unsigned long opt) +{ + struct lruvec *lruvec; + int err =3D -EINVAL; + struct mem_cgroup *memcg =3D NULL; + + if (!mem_cgroup_disabled()) { + rcu_read_lock(); + memcg =3D mem_cgroup_from_id(memcg_id); +#ifdef CONFIG_MEMCG + if (memcg && !css_tryget(&memcg->css)) + memcg =3D NULL; +#endif + rcu_read_unlock(); + + if (!memcg) + goto done; + } + if (memcg_id !=3D mem_cgroup_id(memcg)) + goto done; + + if (nid < 0 || nid >=3D MAX_NUMNODES || !node_state(nid, N_MEMORY)) + goto done; + + lruvec =3D get_lruvec(memcg, nid); + + if (swappiness < 0) + swappiness =3D get_swappiness(lruvec, sc); + else if (swappiness > 200) + goto done; + + switch (cmd) { + case '+': + err =3D run_aging(lruvec, seq, sc, swappiness, opt); + break; + case '-': + err =3D run_eviction(lruvec, seq, sc, swappiness, opt); + break; + } +done: + mem_cgroup_put(memcg); + + return err; +} + +static ssize_t lru_gen_seq_write(struct file *file, const char __user *src, + size_t len, loff_t *pos) +{ + void *buf; + char *cur, *next; + unsigned int flags; + int err =3D 0; + struct scan_control sc =3D { + .may_writepage =3D true, + .may_unmap =3D true, + .may_swap =3D true, + .reclaim_idx =3D MAX_NR_ZONES - 1, + .gfp_mask =3D GFP_KERNEL, + }; + + buf =3D kvmalloc(len + 1, GFP_KERNEL); + if (!buf) + return -ENOMEM; + + if (copy_from_user(buf, src, len)) { + kvfree(buf); + return -EFAULT; + } + + next =3D buf; + next[len] =3D '\0'; + + sc.reclaim_state.mm_walk =3D alloc_mm_walk(); + if (!sc.reclaim_state.mm_walk) { + kvfree(buf); + return -ENOMEM; + } + + set_task_reclaim_state(current, &sc.reclaim_state); + flags =3D memalloc_noreclaim_save(); + + while ((cur =3D strsep(&next, ",;\n"))) { + int n; + int end; + char cmd; + unsigned int memcg_id; + unsigned int nid; + unsigned long seq; + unsigned int swappiness =3D -1; + unsigned long opt =3D -1; + + cur =3D skip_spaces(cur); + if (!*cur) + continue; + + n =3D sscanf(cur, "%c %u %u %lu %n %u %n %lu %n", &cmd, &memcg_id, &nid, + &seq, &end, &swappiness, &end, &opt, &end); + if (n < 4 || cur[end]) { + err =3D -EINVAL; + break; + } + + err =3D run_cmd(cmd, memcg_id, nid, seq, &sc, swappiness, opt); + if (err) + break; + } + + memalloc_noreclaim_restore(flags); + set_task_reclaim_state(current, NULL); + + free_mm_walk(sc.reclaim_state.mm_walk); + kvfree(buf); + + return err ? : len; +} + +static int lru_gen_seq_open(struct inode *inode, struct file *file) +{ + return seq_open(file, &lru_gen_seq_ops); +} + +static const struct file_operations lru_gen_rw_fops =3D { + .open =3D lru_gen_seq_open, + .read =3D seq_read, + .write =3D lru_gen_seq_write, + .llseek =3D seq_lseek, + .release =3D seq_release, +}; + +static const struct file_operations lru_gen_ro_fops =3D { + .open =3D lru_gen_seq_open, + .read =3D seq_read, + .llseek =3D seq_lseek, + .release =3D seq_release, +}; + /*************************************************************************= ***** * initialization *************************************************************************= *****/ @@ -5233,6 +5613,9 @@ static int __init init_lru_gen(void) if (sysfs_create_group(mm_kobj, &lru_gen_attr_group)) pr_err("lru_gen: failed to create sysfs group\n"); =20 + debugfs_create_file("lru_gen", 0644, NULL, NULL, &lru_gen_rw_fops); + debugfs_create_file("lru_gen_full", 0444, NULL, NULL, &lru_gen_ro_fops); + return 0; }; late_initcall(init_lru_gen); --=20 2.36.0.550.gb090851708-goog From nobody Thu May 7 21:41:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C438C433EF for ; Wed, 18 May 2022 01:48:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233976AbiERBsN (ORCPT ); Tue, 17 May 2022 21:48:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57982 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233886AbiERBrV (ORCPT ); Tue, 17 May 2022 21:47:21 -0400 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 72C40546BC for ; Tue, 17 May 2022 18:47:13 -0700 (PDT) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-2f4dfd09d7fso8829047b3.0 for ; Tue, 17 May 2022 18:47:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc:content-transfer-encoding; bh=pC2YGHfY+9xAk1NrS3hVRRGbc59d6jA/kKpqRhmXjZU=; b=d4/5u6MDRmW4dQggwuo45gVHOGcNR7n57VvoOlxwEtWbUwad4GE8aoDysmKFGLTaVE q+zAo3ySyTwPb9hp05gRFncfuWt2nDjGuN6VfZpZuMbuyUxFM11g/agQlrVYIWn1IUxs XkipmOMieNU1uw2D5zUVznCFAV+YUSCTxsr5zTWuavGi78sw6myMYTLZC8XhvytNpf3w IdBNhbj1+CBfR+UZPTYwWusujy0kZWjqR/t/5FIxrla+ArWZXJekE7zvVGx0uTRTrMsM zI9Ssfi5ZbP1ZDD5VM0Y1nW3UsmRKOCaVKiAoWD7WX2rZnxkta80sIocB8kK4eb0Qz/t MIog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc:content-transfer-encoding; bh=pC2YGHfY+9xAk1NrS3hVRRGbc59d6jA/kKpqRhmXjZU=; b=ZRnXWUcOO4QqiAAQRhk4VLyxUO74vwpOOgDq2hupHi5ETB4aTEpFIFBs191iM/NpFR +0ftTwuWl6ymxypki1wnVNs+klw1JCh5ML0B8X2hgQSxVFBrCHsOKOeqTBeNLdpBCCx4 o8T7S9L/0EzetbSVulw96kvmNyKSvUusUUT7Vy3ec/ZIqhhua4x6Ph0QzdypMUWXCZ6x 3nh04+xGT9SY/NU8BVZKNpdQLC/6OBUMm5KODLD92p0OqoWmYIoGDibSIa27OB8/UyM+ SC9hkP6q634vk+FZB9TxT16YKFoaLTx2Egn6Tv4/xcNyvypxFtJchmqH8pjaDJXDzzFw zDfQ== X-Gm-Message-State: AOAM532zVvVqVFSfzfGVQco8L4lIcpbTEahQ7y2MPBM0xKMccKY64jJh nP8CCJKcDZTFCrrFb2vmkXwUCQhm9Vw= X-Google-Smtp-Source: ABdhPJyp1Ue7r2nSH4HDYmkzwihmYW9ugw8ScU0gZU8yTvmMpO6V3hTtIt1pn8lV2v+Jf8Sc9X480fPnoAo= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f7bc:90c9:d86e:4ea8]) (user=yuzhao job=sendgmr) by 2002:a0d:ccc3:0:b0:2ff:4e0:2be5 with SMTP id o186-20020a0dccc3000000b002ff04e02be5mr12603557ywd.225.1652838432493; Tue, 17 May 2022 18:47:12 -0700 (PDT) Date: Tue, 17 May 2022 19:46:32 -0600 In-Reply-To: <20220518014632.922072-1-yuzhao@google.com> Message-Id: <20220518014632.922072-14-yuzhao@google.com> Mime-Version: 1.0 References: <20220518014632.922072-1-yuzhao@google.com> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog Subject: [PATCH v11 13/14] mm: multi-gen LRU: admin guide From: Yu Zhao To: Andrew Morton , linux-mm@kvack.org Cc: Andi Kleen , Aneesh Kumar , Catalin Marinas , Dave Hansen , Hillf Danton , Jens Axboe , Johannes Weiner , Jonathan Corbet , Linus Torvalds , Matthew Wilcox , Mel Gorman , Michael Larabel , Michal Hocko , Mike Rapoport , Peter Zijlstra , Tejun Heo , Vlastimil Babka , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, page-reclaim@google.com, Yu Zhao , Brian Geffon , Jan Alexander Steffens , Oleksandr Natalenko , Steven Barrett , Suleiman Souhlal , Daniel Byrne , Donald Carr , "=?UTF-8?q?Holger=20Hoffst=C3=A4tte?=" , Konstantin Kharlamov , Shuang Zhai , Sofia Trinh , Vaibhav Jain Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Add an admin guide. Signed-off-by: Yu Zhao Acked-by: Brian Geffon Acked-by: Jan Alexander Steffens (heftig) Acked-by: Oleksandr Natalenko Acked-by: Steven Barrett Acked-by: Suleiman Souhlal Tested-by: Daniel Byrne Tested-by: Donald Carr Tested-by: Holger Hoffst=C3=A4tte Tested-by: Konstantin Kharlamov Tested-by: Shuang Zhai Tested-by: Sofia Trinh Tested-by: Vaibhav Jain --- Documentation/admin-guide/mm/index.rst | 1 + Documentation/admin-guide/mm/multigen_lru.rst | 156 ++++++++++++++++++ mm/Kconfig | 3 +- 3 files changed, 159 insertions(+), 1 deletion(-) create mode 100644 Documentation/admin-guide/mm/multigen_lru.rst diff --git a/Documentation/admin-guide/mm/index.rst b/Documentation/admin-g= uide/mm/index.rst index c21b5823f126..2cf5bae62036 100644 --- a/Documentation/admin-guide/mm/index.rst +++ b/Documentation/admin-guide/mm/index.rst @@ -32,6 +32,7 @@ the Linux memory management. idle_page_tracking ksm memory-hotplug + multigen_lru nommu-mmap numa_memory_policy numaperf diff --git a/Documentation/admin-guide/mm/multigen_lru.rst b/Documentation/= admin-guide/mm/multigen_lru.rst new file mode 100644 index 000000000000..6355f2b5019d --- /dev/null +++ b/Documentation/admin-guide/mm/multigen_lru.rst @@ -0,0 +1,156 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Multi-Gen LRU +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +The multi-gen LRU is an alternative LRU implementation that optimizes +page reclaim and improves performance under memory pressure. Page +reclaim decides the kernel's caching policy and ability to overcommit +memory. It directly impacts the kswapd CPU usage and RAM efficiency. + +Quick start +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Build the kernel with the following configurations. + +* ``CONFIG_LRU_GEN=3Dy`` +* ``CONFIG_LRU_GEN_ENABLED=3Dy`` + +All set! + +Runtime options +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +``/sys/kernel/mm/lru_gen/`` contains stable ABIs described in the +following subsections. + +Kill switch +----------- +``enabled`` accepts different values to enable or disable the +following components. Its default value depends on +``CONFIG_LRU_GEN_ENABLED``. All the components should be enabled +unless some of them have unforeseen side effects. Writing to +``enabled`` has no effect when a component is not supported by the +hardware, and valid values will be accepted even when the main switch +is off. + +=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Values Components +=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +0x0001 The main switch for the multi-gen LRU. +0x0002 Clearing the accessed bit in leaf page table entries in large + batches, when MMU sets it (e.g., on x86). This behavior can + theoretically worsen lock contention (mmap_lock). If it is + disabled, the multi-gen LRU will suffer a minor performance + degradation for workloads that contiguously map hot pages, + whose accessed bits can be otherwise cleared by fewer larger + batches. +0x0004 Clearing the accessed bit in non-leaf page table entries as + well, when MMU sets it (e.g., on x86). This behavior was not + verified on x86 varieties other than Intel and AMD. If it is + disabled, the multi-gen LRU will suffer a negligible + performance degradation. +[yYnN] Apply to all the components above. +=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +E.g., +:: + + echo y >/sys/kernel/mm/lru_gen/enabled + cat /sys/kernel/mm/lru_gen/enabled + 0x0007 + echo 5 >/sys/kernel/mm/lru_gen/enabled + cat /sys/kernel/mm/lru_gen/enabled + 0x0005 + +Thrashing prevention +-------------------- +Personal computers are more sensitive to thrashing because it can +cause janks (lags when rendering UI) and negatively impact user +experience. The multi-gen LRU offers thrashing prevention to the +majority of laptop and desktop users who do not have ``oomd``. + +Users can write ``N`` to ``min_ttl_ms`` to prevent the working set of +``N`` milliseconds from getting evicted. The OOM killer is triggered +if this working set cannot be kept in memory. In other words, this +option works as an adjustable pressure relief valve, and when open, it +terminates applications that are hopefully not being used. + +Based on the average human detectable lag (~100ms), ``N=3D1000`` usually +eliminates intolerable janks due to thrashing. Larger values like +``N=3D3000`` make janks less noticeable at the risk of premature OOM +kills. + +The default value ``0`` means disabled. + +Experimental features +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +``/sys/kernel/debug/lru_gen`` accepts commands described in the +following subsections. Multiple command lines are supported, so does +concatenation with delimiters ``,`` and ``;``. + +``/sys/kernel/debug/lru_gen_full`` provides additional stats for +debugging. ``CONFIG_LRU_GEN_STATS=3Dy`` keeps historical stats from +evicted generations in this file. + +Working set estimation +---------------------- +Working set estimation measures how much memory an application needs +in a given time interval, and it is usually done with little impact on +the performance of the application. E.g., data centers want to +optimize job scheduling (bin packing) to improve memory utilizations. +When a new job comes in, the job scheduler needs to find out whether +each server it manages can allocate a certain amount of memory for +this new job before it can pick a candidate. To do so, the job +scheduler needs to estimate the working sets of the existing jobs. + +When it is read, ``lru_gen`` returns a histogram of numbers of pages +accessed over different time intervals for each memcg and node. +``MAX_NR_GENS`` decides the number of bins for each histogram. The +histograms are noncumulative. +:: + + memcg memcg_id memcg_path + node node_id + min_gen_nr age_in_ms nr_anon_pages nr_file_pages + ... + max_gen_nr age_in_ms nr_anon_pages nr_file_pages + +Each bin contains an estimated number of pages that have been accessed +within ``age_in_ms``. E.g., ``min_gen_nr`` contains the coldest pages +and ``max_gen_nr`` contains the hottest pages, since ``age_in_ms`` of +the former is the largest and that of the latter is the smallest. + +Users can write ``+ memcg_id node_id max_gen_nr +[can_swap [force_scan]]`` to ``lru_gen`` to create a new generation +``max_gen_nr+1``. ``can_swap`` defaults to the swap setting and, if it +is set to ``1``, it forces the scan of anon pages when swap is off, +and vice versa. ``force_scan`` defaults to ``1`` and, if it is set to +``0``, it employs heuristics to reduce the overhead, which is likely +to reduce the coverage as well. + +A typical use case is that a job scheduler writes to ``lru_gen`` at a +certain time interval to create new generations, and it ranks the +servers it manages based on the sizes of their cold pages defined by +this time interval. + +Proactive reclaim +----------------- +Proactive reclaim induces page reclaim when there is no memory +pressure. It usually targets cold pages only. E.g., when a new job +comes in, the job scheduler wants to proactively reclaim cold pages on +the server it selected to improve the chance of successfully landing +this new job. + +Users can write ``- memcg_id node_id min_gen_nr [swappiness +[nr_to_reclaim]]`` to ``lru_gen`` to evict generations less than or +equal to ``min_gen_nr``. Note that ``min_gen_nr`` should be less than +``max_gen_nr-1`` as ``max_gen_nr`` and ``max_gen_nr-1`` are not fully +aged and therefore cannot be evicted. ``swappiness`` overrides the +default value in ``/proc/sys/vm/swappiness``. ``nr_to_reclaim`` limits +the number of pages to evict. + +A typical use case is that a job scheduler writes to ``lru_gen`` +before it tries to land a new job on a server. If it fails to +materialize enough cold pages because of the overestimation, it +retries on the next server according to the ranking result obtained +from the working set estimation step. This less forceful approach +limits the impacts on the existing jobs. diff --git a/mm/Kconfig b/mm/Kconfig index 426ea5f57d88..05291697055a 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -916,7 +916,8 @@ config LRU_GEN # make sure folio->flags has enough spare bits depends on 64BIT || !SPARSEMEM || SPARSEMEM_VMEMMAP help - A high performance LRU implementation to overcommit memory. + A high performance LRU implementation to overcommit memory. See + Documentation/admin-guide/mm/multigen_lru.rst for details. =20 config LRU_GEN_ENABLED bool "Enable by default" --=20 2.36.0.550.gb090851708-goog From nobody Thu May 7 21:41:32 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E93ADC433F5 for ; Wed, 18 May 2022 01:48:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233970AbiERBsV (ORCPT ); Tue, 17 May 2022 21:48:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58242 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233960AbiERBrW (ORCPT ); Tue, 17 May 2022 21:47:22 -0400 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0FD7B54F98 for ; Tue, 17 May 2022 18:47:15 -0700 (PDT) Received: by mail-yb1-xb4a.google.com with SMTP id a144-20020a25ca96000000b0064d6e10dd6aso712378ybg.1 for ; Tue, 17 May 2022 18:47:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc:content-transfer-encoding; bh=oDRmYIZkRXCw47tYWNpQTMfPwHaD4jUBbPvynchhw84=; b=c+DuDWogV3HUcoWs4xOcKlCpTX2mcy0Y2GE+VRTwOZ3IGxptGhK3wtyVpZ1KZLVr6G ubDmscCXwFxrBzjkr/Ygt5cBLoClV4EBSwQEBVS7PbHQXSZE+oIoBG4OAKfluE+ysBVt IJvt0Raa+iq69htrMLP/mTP2Lq3ABUm5XGZOh82NrJkfoQYNgRl1dVm/cC2va7Urmw42 umm3BQS2hZ9k2lfKUT/xHL1mM4cA0iBJeAI8n7vemQdx+oQ7+0FGE4Uybg74fyZz8H+b Cjj95J8cDULNDZ8Di11h3iq+2csNPQIlyGVPIPuHLJLyqGYSL8RH2MmDVMhOvVtfFnji JPdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc:content-transfer-encoding; bh=oDRmYIZkRXCw47tYWNpQTMfPwHaD4jUBbPvynchhw84=; b=TiUtfN9pMzHTCWRfeFU8boglrREFnvyWexyPBvaC06Bm2onArDo5msT5W7tBbT197Y rruMVjOW+IZzqnqBBX7St67nr2AAxMgIUE2HE8zNM+0hA0HoSuiCOt5/+FXDkLNlWSKW xLYA7VmXYE93pzVc5aiUjXGr9dq3FD4ezJRq7fLK6+nj39bY3khv18vclXWiFurvLSzE PQLy3pUeZVNpS+3GN6UqE+WNzyTIJ7weS/LAT9Hs6IYYNsYG3XGo8UWcpQ5t7WNMevMs nKCG6dpL4KohINWgTihU0xAJYB2rpZ60J5pRGGwJTr8BhnyJ1N2YZhsRhU3mhB8+WXDC kmuQ== X-Gm-Message-State: AOAM531p2QTvu5Q+GQPShHeYC5+ISOM7bQTONEDZrfpKXbYG6Kr8u1Vs NBuQpqI8zsSWHAVLVmdM46vQJzVSmuk= X-Google-Smtp-Source: ABdhPJy5LKbybCArJ3J0UcjPW1XS/7cnnazrh0shp3SXLKG8Hnk85hqzn3x0uDIfq5b0e31ZBFpJzzd/I1Q= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f7bc:90c9:d86e:4ea8]) (user=yuzhao job=sendgmr) by 2002:a25:9f86:0:b0:64e:238e:9a87 with SMTP id u6-20020a259f86000000b0064e238e9a87mr1178014ybq.208.1652838434145; Tue, 17 May 2022 18:47:14 -0700 (PDT) Date: Tue, 17 May 2022 19:46:33 -0600 In-Reply-To: <20220518014632.922072-1-yuzhao@google.com> Message-Id: <20220518014632.922072-15-yuzhao@google.com> Mime-Version: 1.0 References: <20220518014632.922072-1-yuzhao@google.com> X-Mailer: git-send-email 2.36.0.550.gb090851708-goog Subject: [PATCH v11 14/14] mm: multi-gen LRU: design doc From: Yu Zhao To: Andrew Morton , linux-mm@kvack.org Cc: Andi Kleen , Aneesh Kumar , Catalin Marinas , Dave Hansen , Hillf Danton , Jens Axboe , Johannes Weiner , Jonathan Corbet , Linus Torvalds , Matthew Wilcox , Mel Gorman , Michael Larabel , Michal Hocko , Mike Rapoport , Peter Zijlstra , Tejun Heo , Vlastimil Babka , Will Deacon , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, x86@kernel.org, page-reclaim@google.com, Yu Zhao , Brian Geffon , Jan Alexander Steffens , Oleksandr Natalenko , Steven Barrett , Suleiman Souhlal , Daniel Byrne , Donald Carr , "=?UTF-8?q?Holger=20Hoffst=C3=A4tte?=" , Konstantin Kharlamov , Shuang Zhai , Sofia Trinh , Vaibhav Jain Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Add a design doc. Signed-off-by: Yu Zhao Acked-by: Brian Geffon Acked-by: Jan Alexander Steffens (heftig) Acked-by: Oleksandr Natalenko Acked-by: Steven Barrett Acked-by: Suleiman Souhlal Tested-by: Daniel Byrne Tested-by: Donald Carr Tested-by: Holger Hoffst=C3=A4tte Tested-by: Konstantin Kharlamov Tested-by: Shuang Zhai Tested-by: Sofia Trinh Tested-by: Vaibhav Jain --- Documentation/vm/index.rst | 1 + Documentation/vm/multigen_lru.rst | 159 ++++++++++++++++++++++++++++++ 2 files changed, 160 insertions(+) create mode 100644 Documentation/vm/multigen_lru.rst diff --git a/Documentation/vm/index.rst b/Documentation/vm/index.rst index 44365c4574a3..b48434300226 100644 --- a/Documentation/vm/index.rst +++ b/Documentation/vm/index.rst @@ -25,6 +25,7 @@ algorithms. If you are looking for advice on simply allo= cating memory, see the ksm memory-model mmu_notifier + multigen_lru numa overcommit-accounting page_migration diff --git a/Documentation/vm/multigen_lru.rst b/Documentation/vm/multigen_= lru.rst new file mode 100644 index 000000000000..bc8eaf1b956c --- /dev/null +++ b/Documentation/vm/multigen_lru.rst @@ -0,0 +1,159 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Multi-Gen LRU +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +The multi-gen LRU is an alternative LRU implementation that optimizes +page reclaim and improves performance under memory pressure. Page +reclaim decides the kernel's caching policy and ability to overcommit +memory. It directly impacts the kswapd CPU usage and RAM efficiency. + +Design overview +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Objectives +---------- +The design objectives are: + +* Good representation of access recency +* Try to profit from spatial locality +* Fast paths to make obvious choices +* Simple self-correcting heuristics + +The representation of access recency is at the core of all LRU +implementations. In the multi-gen LRU, each generation represents a +group of pages with similar access recency. Generations establish a +(time-based) common frame of reference and therefore help make better +choices, e.g., between different memcgs on a computer or different +computers in a data center (for job scheduling). + +Exploiting spatial locality improves efficiency when gathering the +accessed bit. A rmap walk targets a single page and does not try to +profit from discovering a young PTE. A page table walk can sweep all +the young PTEs in an address space, but the address space can be too +sparse to make a profit. The key is to optimize both methods and use +them in combination. + +Fast paths reduce code complexity and runtime overhead. Unmapped pages +do not require TLB flushes; clean pages do not require writeback. +These facts are only helpful when other conditions, e.g., access +recency, are similar. With generations as a common frame of reference, +additional factors stand out. But obvious choices might not be good +choices; thus self-correction is necessary. + +The benefits of simple self-correcting heuristics are self-evident. +Again, with generations as a common frame of reference, this becomes +attainable. Specifically, pages in the same generation can be +categorized based on additional factors, and a feedback loop can +statistically compare the refault percentages across those categories +and infer which of them are better choices. + +Assumptions +----------- +The protection of hot pages and the selection of cold pages are based +on page access channels and patterns. There are two access channels: + +* Accesses through page tables +* Accesses through file descriptors + +The protection of the former channel is by design stronger because: + +1. The uncertainty in determining the access patterns of the former + channel is higher due to the approximation of the accessed bit. +2. The cost of evicting the former channel is higher due to the TLB + flushes required and the likelihood of encountering the dirty bit. +3. The penalty of underprotecting the former channel is higher because + applications usually do not prepare themselves for major page + faults like they do for blocked I/O. E.g., GUI applications + commonly use dedicated I/O threads to avoid blocking rendering + threads. + +There are also two access patterns: + +* Accesses exhibiting temporal locality +* Accesses not exhibiting temporal locality + +For the reasons listed above, the former channel is assumed to follow +the former pattern unless ``VM_SEQ_READ`` or ``VM_RAND_READ`` is +present, and the latter channel is assumed to follow the latter +pattern unless outlying refaults have been observed. + +Workflow overview +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Evictable pages are divided into multiple generations for each +``lruvec``. The youngest generation number is stored in +``lrugen->max_seq`` for both anon and file types as they are aged on +an equal footing. The oldest generation numbers are stored in +``lrugen->min_seq[]`` separately for anon and file types as clean file +pages can be evicted regardless of swap constraints. These three +variables are monotonically increasing. + +Generation numbers are truncated into ``order_base_2(MAX_NR_GENS+1)`` +bits in order to fit into the gen counter in ``folio->flags``. Each +truncated generation number is an index to ``lrugen->lists[]``. The +sliding window technique is used to track at least ``MIN_NR_GENS`` and +at most ``MAX_NR_GENS`` generations. The gen counter stores a value +within ``[1, MAX_NR_GENS]`` while a page is on one of +``lrugen->lists[]``; otherwise it stores zero. + +Each generation is divided into multiple tiers. Tiers represent +different ranges of numbers of accesses through file descriptors. A +page accessed ``N`` times through file descriptors is in tier +``order_base_2(N)``. In contrast to moving across generations, which +requires the LRU lock, moving across tiers only involves atomic +operations on ``folio->flags`` and therefore has a negligible cost. A +feedback loop modeled after the PID controller monitors refaults over +all the tiers from anon and file types and decides which tiers from +which types to evict or protect. + +There are two conceptually independent procedures: the aging and the +eviction. They form a closed-loop system, i.e., the page reclaim. + +Aging +----- +The aging produces young generations. Given an ``lruvec``, it +increments ``max_seq`` when ``max_seq-min_seq+1`` approaches +``MIN_NR_GENS``. The aging promotes hot pages to the youngest +generation when it finds them accessed through page tables; the +demotion of cold pages happens consequently when it increments +``max_seq``. The aging uses page table walks and rmap walks to find +young PTEs. For the former, it iterates ``lruvec_memcg()->mm_list`` +and calls ``walk_page_range()`` with each ``mm_struct`` on this list +to scan PTEs, and after each iteration, it increments ``max_seq``. For +the latter, when the eviction walks the rmap and finds a young PTE, +the aging scans the adjacent PTEs. For both, on finding a young PTE, +the aging clears the accessed bit and updates the gen counter of the +page mapped by this PTE to ``(max_seq%MAX_NR_GENS)+1``. + +Eviction +-------- +The eviction consumes old generations. Given an ``lruvec``, it +increments ``min_seq`` when ``lrugen->lists[]`` indexed by +``min_seq%MAX_NR_GENS`` becomes empty. To select a type and a tier to +evict from, it first compares ``min_seq[]`` to select the older type. +If both types are equally old, it selects the one whose first tier has +a lower refault percentage. The first tier contains single-use +unmapped clean pages, which are the best bet. The eviction sorts a +page according to its gen counter if the aging has found this page +accessed through page tables and updated its gen counter. It also +moves a page to the next generation, i.e., ``min_seq+1``, if this page +was accessed multiple times through file descriptors and the feedback +loop has detected outlying refaults from the tier this page is in. To +do this, the feedback loop uses the first tier as the baseline, for +the reason stated earlier. + +Summary +------- +The multi-gen LRU can be disassembled into the following parts: + +* Generations +* Page table walks +* Rmap walks +* Bloom filters +* PID controller + +The aging and the eviction form a producer-consumer model; +specifically, the latter drives the former by the sliding window over +generations. Within the aging, rmap walks drive page table walks by +inserting hot densely populated page tables to the Bloom filters. +Within the eviction, the PID controller uses refaults as the feedback +to select types to evict and tiers to protect. --=20 2.36.0.550.gb090851708-goog