From nobody Mon Apr 13 18:44:42 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6803C433FE for ; Tue, 15 Nov 2022 18:29:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229995AbiKOS3m (ORCPT ); Tue, 15 Nov 2022 13:29:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38880 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231855AbiKOS3O (ORCPT ); Tue, 15 Nov 2022 13:29:14 -0500 Received: from mail-pl1-x634.google.com (mail-pl1-x634.google.com [IPv6:2607:f8b0:4864:20::634]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 532022FFD6 for ; Tue, 15 Nov 2022 10:29:06 -0800 (PST) Received: by mail-pl1-x634.google.com with SMTP id v17so13959997plo.1 for ; Tue, 15 Nov 2022 10:29:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XlpGuGY3sJe9DLKH41vLTGd0LKIzHOuJ8+yi4Jjp058=; b=BBr8ahhuM2MCFKMZ/s05AiGPyeVkO2G9z/hbSkzjX0lY2G19nKIRAyPv4LqvFGtRk9 R/WM5NlfWEmS9TsTF3hsJV3HDoHwfpujZtu2vUrwPpgvEIK5BZ0G1j3cyWbnL1r2MFyz 8MbBg0kNP/rP4TSeJTRUMU2aPplm/armvyU4+xP6PDyGvSjgWIiD1K866M+41yLSVI0a +pZl3mCaixU/HKYf3lfmmlU6uq1HHpJyv/p2iuhS+flKIkzR7Ga0FDiXHMovKRhHeOtp MSQIVWlms2IcBe2h73fwCwkXDSDzXrO5UbL7MUmk49FBcfVCFk+7CiHIlWv+oKhEVENS DSGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XlpGuGY3sJe9DLKH41vLTGd0LKIzHOuJ8+yi4Jjp058=; b=t3sz1zVnKLBG7RnvlmzrRgBeawfnLW4xAPtbKpoBaD+Y3a/fGmRTjXfL2JZNZd3CXT UwT86ivehVK5Q+A1h88WNmN9gDt2ftsVyaqK1tsq9SVWwkCAqFFRfVKYepqk/eBecR6U V8QizRPAjf3FnmHSp3f7W1n40DSlwpxl75yYkkrKeW404z59UdOUBBzUepebnCs4ULhM R2GlNwQEd+E5J/fwiDgOr4NMGLL1BCEgDt2n0itq7/p1sV1TBiyuPJg1C1aRMCgqV0dZ w5Agld/ynI6OGJ3+HzqkFblhHnCQncSj+QxHL4/5akGPsDxlc2+iVc3O47D6H+dBiujR LYag== X-Gm-Message-State: ANoB5pl0mfH90nV/XOrBlGs1THyvhTE3AbcAB9T/cPs+jSy+yM4vDPq4 NC32n41bYjAld2iGLYoPyO6e1fuizxvVjA== X-Google-Smtp-Source: AA0mqf5IfXE8x2kz25f2TNlUhdzOFwCYbVwyJMG2ZT25VVWRdudCRU17PIpKVGJDsyeTpvz7/Udf6g== X-Received: by 2002:a17:90b:268f:b0:212:ce2d:9fd7 with SMTP id pl15-20020a17090b268f00b00212ce2d9fd7mr3529401pjb.157.1668536945693; Tue, 15 Nov 2022 10:29:05 -0800 (PST) Received: from localhost (fwdproxy-prn-014.fbsv.net. [2a03:2880:ff:e::face:b00c]) by smtp.gmail.com with ESMTPSA id b24-20020aa79518000000b0056afd55722asm9062101pfp.153.2022.11.15.10.29.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Nov 2022 10:29:05 -0800 (PST) From: Nhat Pham To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: hannes@cmpxchg.org Subject: [PATCH 2/4] workingset: refactor LRU refault to expose refault recency check Date: Tue, 15 Nov 2022 10:28:59 -0800 Message-Id: <20221115182901.2755368-3-nphamcs@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221115182901.2755368-1-nphamcs@gmail.com> References: <20221115182901.2755368-1-nphamcs@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" In preparation for computing recently evicted pages in cachestat, refactor workingset_refault and lru_gen_refault to expose a helper function that would test if an evicted page is recently evicted. Signed-off-by: Nhat Pham --- include/linux/swap.h | 1 + mm/workingset.c | 141 +++++++++++++++++++++++++++++-------------- 2 files changed, 98 insertions(+), 44 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index a18cf4b7c724..dae6f6f955eb 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -361,6 +361,7 @@ static inline void folio_set_swap_entry(struct folio *f= olio, swp_entry_t entry) } =20 /* linux/mm/workingset.c */ +bool workingset_test_recent(void *shadow, bool file, bool *workingset); void workingset_age_nonresident(struct lruvec *lruvec, unsigned long nr_pa= ges); void *workingset_eviction(struct folio *folio, struct mem_cgroup *target_m= emcg); void workingset_refault(struct folio *folio, void *shadow); diff --git a/mm/workingset.c b/mm/workingset.c index 79585d55c45d..51ed8c836467 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -244,6 +244,30 @@ static void *lru_gen_eviction(struct folio *folio) return pack_shadow(mem_cgroup_id(memcg), pgdat, token, refs); } =20 +/** + * Test if the folio is recently evicted. + * + * As a side effect, also populates the references with + * values unpacked from the shadow of the evicted folio. + */ +static bool lru_gen_test_recent(void *shadow, bool file, int *memcgid, + struct pglist_data **pgdat, unsigned long *token, bool *workingset) +{ + struct mem_cgroup *eviction_memcg; + struct lruvec *lruvec; + struct lru_gen_struct *lrugen; + unsigned long min_seq; + + unpack_shadow(shadow, memcgid, pgdat, token, workingset); + eviction_memcg =3D mem_cgroup_from_id(*memcgid); + + lruvec =3D mem_cgroup_lruvec(eviction_memcg, *pgdat); + lrugen =3D &lruvec->lrugen; + + min_seq =3D READ_ONCE(lrugen->min_seq[file]); + return !((*token >> LRU_REFS_WIDTH) !=3D (min_seq & (EVICTION_MASK >> LRU= _REFS_WIDTH))); +} + static void lru_gen_refault(struct folio *folio, void *shadow) { int hist, tier, refs; @@ -258,23 +282,24 @@ static void lru_gen_refault(struct folio *folio, void= *shadow) int type =3D folio_is_file_lru(folio); int delta =3D folio_nr_pages(folio); =20 - unpack_shadow(shadow, &memcg_id, &pgdat, &token, &workingset); - - if (pgdat !=3D folio_pgdat(folio)) - return; =20 rcu_read_lock(); =20 memcg =3D folio_memcg_rcu(folio); + + if (!lru_gen_test_recent(shadow, type, &memcg_id, &pgdat, &token, + &workingset)) + goto unlock; + if (memcg_id !=3D mem_cgroup_id(memcg)) goto unlock; =20 + if (pgdat !=3D folio_pgdat(folio)) + goto unlock; + lruvec =3D mem_cgroup_lruvec(memcg, pgdat); lrugen =3D &lruvec->lrugen; - min_seq =3D READ_ONCE(lrugen->min_seq[type]); - if ((token >> LRU_REFS_WIDTH) !=3D (min_seq & (EVICTION_MASK >> LRU_REFS_= WIDTH))) - goto unlock; =20 hist =3D lru_hist_from_seq(min_seq); /* see the comment in folio_lru_refs() */ @@ -306,6 +331,12 @@ static void *lru_gen_eviction(struct folio *folio) return NULL; } =20 +static bool lru_gen_test_recent(void *shadow, int type, int *memcgid, + struct pglist_data **pgdat, unsigned long *token, bool *workingset) +{ + return true; +} + static void lru_gen_refault(struct folio *folio, void *shadow) { } @@ -374,39 +405,31 @@ void *workingset_eviction(struct folio *folio, struct= mem_cgroup *target_memcg) } =20 /** - * workingset_refault - Evaluate the refault of a previously evicted folio. - * @folio: The freshly allocated replacement folio. - * @shadow: Shadow entry of the evicted folio. + * Test if the folio is recently evicted by checking if + * refault distance of shadow exceeds workingset size. * - * Calculates and evaluates the refault distance of the previously - * evicted folio in the context of the node and the memcg whose memory - * pressure caused the eviction. + * As a side effect, populate workingset with the value + * unpacked from shadow. */ -void workingset_refault(struct folio *folio, void *shadow) +bool workingset_test_recent(void *shadow, bool file, bool *workingset) { - bool file =3D folio_is_file_lru(folio); struct mem_cgroup *eviction_memcg; struct lruvec *eviction_lruvec; unsigned long refault_distance; unsigned long workingset_size; - struct pglist_data *pgdat; - struct mem_cgroup *memcg; - unsigned long eviction; - struct lruvec *lruvec; unsigned long refault; - bool workingset; + int memcgid; - long nr; + struct pglist_data *pgdat; + unsigned long eviction; =20 - if (lru_gen_enabled()) { - lru_gen_refault(folio, shadow); - return; - } + if (lru_gen_enabled()) + return lru_gen_test_recent(shadow, file, &memcgid, + &pgdat, &eviction, workingset); =20 - unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset); + unpack_shadow(shadow, &memcgid, &pgdat, &eviction, workingset); eviction <<=3D bucket_order; =20 - rcu_read_lock(); /* * Look up the memcg associated with the stored ID. It might * have been deleted since the folio's eviction. @@ -425,7 +448,8 @@ void workingset_refault(struct folio *folio, void *shad= ow) */ eviction_memcg =3D mem_cgroup_from_id(memcgid); if (!mem_cgroup_disabled() && !eviction_memcg) - goto out; + return false; + eviction_lruvec =3D mem_cgroup_lruvec(eviction_memcg, pgdat); refault =3D atomic_long_read(&eviction_lruvec->nonresident_age); =20 @@ -447,21 +471,6 @@ void workingset_refault(struct folio *folio, void *sha= dow) */ refault_distance =3D (refault - eviction) & EVICTION_MASK; =20 - /* - * The activation decision for this folio is made at the level - * where the eviction occurred, as that is where the LRU order - * during folio reclaim is being determined. - * - * However, the cgroup that will own the folio is the one that - * is actually experiencing the refault event. - */ - nr =3D folio_nr_pages(folio); - memcg =3D folio_memcg(folio); - pgdat =3D folio_pgdat(folio); - lruvec =3D mem_cgroup_lruvec(memcg, pgdat); - - mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); - mem_cgroup_flush_stats_delayed(); /* * Compare the distance to the existing workingset size. We @@ -483,8 +492,51 @@ void workingset_refault(struct folio *folio, void *sha= dow) NR_INACTIVE_ANON); } } - if (refault_distance > workingset_size) + + return refault_distance <=3D workingset_size; +} + +/** + * workingset_refault - Evaluate the refault of a previously evicted folio. + * @folio: The freshly allocated replacement folio. + * @shadow: Shadow entry of the evicted folio. + * + * Calculates and evaluates the refault distance of the previously + * evicted folio in the context of the node and the memcg whose memory + * pressure caused the eviction. + */ +void workingset_refault(struct folio *folio, void *shadow) +{ + bool file =3D folio_is_file_lru(folio); + struct pglist_data *pgdat; + struct mem_cgroup *memcg; + struct lruvec *lruvec; + bool workingset; + long nr; + + if (lru_gen_enabled()) { + lru_gen_refault(folio, shadow); + return; + } + + rcu_read_lock(); + + nr =3D folio_nr_pages(folio); + memcg =3D folio_memcg(folio); + pgdat =3D folio_pgdat(folio); + lruvec =3D mem_cgroup_lruvec(memcg, pgdat); + + if (!workingset_test_recent(shadow, file, &workingset)) { + /* + * The activation decision for this folio is made at the level + * where the eviction occurred, as that is where the LRU order + * during folio reclaim is being determined. + * + * However, the cgroup that will own the folio is the one that + * is actually experiencing the refault event. + */ goto out; + } =20 folio_set_active(folio); workingset_age_nonresident(lruvec, nr); @@ -498,6 +550,7 @@ void workingset_refault(struct folio *folio, void *shad= ow) mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file, nr); } out: + mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); rcu_read_unlock(); } =20 --=20 2.30.2