From nobody Fri Dec 19 18:53:41 2025 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B45FB168BE for ; Fri, 22 Dec 2023 10:24:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ZuBZ0SE8" Received: by mail-pf1-f178.google.com with SMTP id d2e1a72fcca58-6d0a679fca7so1038979b3a.2 for ; Fri, 22 Dec 2023 02:24:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703240649; x=1703845449; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=TmeNvQjyOn7MVyb9xd+5eOKYiNc8no2eFt793P8sn6s=; b=ZuBZ0SE8tHRCwX5jB7kLowB8qUZMm+WOeoBl4MjV6UyykT7OLY4x9Ix7jA3gIiw+hl nos6CKLfbk0l6zMX87cuzgze3z8RMQhm2ud1L9PPOA8zZuDfo91BJg3IkbhsWaG5Q3gz I5Jt9suR4BGqy1kf+nTx+UShAoEN9XUo2Jg/G8x1TpnOOyysn/LTBN/m9mpF7pd9hjqo JMs+jVFomaPbV5ogKF9WA52pncNEBOYRmQzKZhkd27tVz1aCbq2gPfMXChZ5ft6/+bAz VDHMIstPkhOXaQ3IvYi5Z2m5PIp7cBZqSG0ayvMCyw8ckA/oDso6UjmVQ8Vc5UXLGBMr dLIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703240649; x=1703845449; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=TmeNvQjyOn7MVyb9xd+5eOKYiNc8no2eFt793P8sn6s=; b=OORBLwZu7cdqLzyPfXcRi2ykeuqM8p6b/En2TA9K2DOB1epHS3O9cQeLMU4M+bWDMC f8ofN4GKgL7De3NrgTqe154peLUL0+IQ+vyKCZ257EhZ3619IdFpy0EcB6TZX19VXxGg CHGHo6K5quYwkyn3l5BQU6JPrubFhfyn8y09BzY03OC6OYezAgwzJHJm5yWBdbZwQGlt K5xkTgiX6GTVX7c6Kwk83UjodIqmq+zBdMkkXAFoGNhKSD//AZcTJ2vFZY/juSvP6Omr wLD7EHibqFXPZX2gU1D5pBfmCrkaokY19rHKPSCY1WV7hqSbemmVKFoZou203t4ihRnE XJng== X-Gm-Message-State: AOJu0YzkVZ4mMRj5IcrIiehQqMSP3ZVkb9mAp0G1/Z9wFSSRO4dKp81H QMnl32LyU4Gv16lc0qLM1eY= X-Google-Smtp-Source: AGHT+IFqSFyTsdaDJbnz34Tju5Rl8kxf/zGRHLlyWWOVYHMpESFATB5jFyXEKy/Qqs+cjCDyhnGHDQ== X-Received: by 2002:aa7:8554:0:b0:6d8:bef3:4182 with SMTP id y20-20020aa78554000000b006d8bef34182mr721760pfn.1.1703240648841; Fri, 22 Dec 2023 02:24:08 -0800 (PST) Received: from KASONG-MB2.tencent.com ([103.7.29.31]) by smtp.gmail.com with ESMTPSA id gx1-20020a056a001e0100b006d9912e9a77sm364074pfb.6.2023.12.22.02.24.06 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 22 Dec 2023 02:24:08 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Yu Zhao , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 1/3] mm, lru_gen: batch update counters on againg Date: Fri, 22 Dec 2023 18:22:53 +0800 Message-ID: <20231222102255.56993-2-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20231222102255.56993-1-ryncsn@gmail.com> References: <20231222102255.56993-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song When lru_gen is aging, it will update mm counters page by page, which causes a higher overhead if age happens frequently or there are a lot of pages in one generation getting moved. Optimize this by doing the counter update in batch. Although most __mod_*_state has its own caches the overhead is still observable. Tested in a 4G memcg on a EPYC 7K62 with: memcached -u nobody -m 16384 -s /tmp/memcached.socket \ -a 0766 -t 16 -B binary & memtier_benchmark -S /tmp/memcached.socket \ -P memcache_binary -n allkeys \ --key-minimum=3D1 --key-maximum=3D16000000 -d 1024 \ --ratio=3D1:0 --key-pattern=3DP:P -c 2 -t 16 --pipeline 8 -x 6 Average result of 18 test runs: Before: 44017.78 Ops/sec After: 44687.08 Ops/sec (+1.5%) Signed-off-by: Kairui Song --- mm/vmscan.c | 64 +++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 55 insertions(+), 9 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index b4ca3563bcf4..e3b4797b9729 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3095,9 +3095,47 @@ static int folio_update_gen(struct folio *folio, int= gen) return ((old_flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1; } =20 +/* + * Update LRU gen in batch for each lru_gen LRU list. The batch is limited= to + * each gen / type / zone level LRU. Batch is applied after finished or ab= orted + * scanning one LRU list. + */ +struct gen_update_batch { + int delta[MAX_NR_GENS]; +}; + +static void lru_gen_update_batch(struct lruvec *lruvec, bool type, int zon= e, + struct gen_update_batch *batch) +{ + int gen; + int promoted =3D 0; + struct lru_gen_folio *lrugen =3D &lruvec->lrugen; + enum lru_list lru =3D type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON; + + for (gen =3D 0; gen < MAX_NR_GENS; gen++) { + int delta =3D batch->delta[gen]; + + if (!delta) + continue; + + WRITE_ONCE(lrugen->nr_pages[gen][type][zone], + lrugen->nr_pages[gen][type][zone] + delta); + + if (lru_gen_is_active(lruvec, gen)) + promoted +=3D delta; + } + + if (promoted) { + __update_lru_size(lruvec, lru, zone, -promoted); + __update_lru_size(lruvec, lru + LRU_ACTIVE, zone, promoted); + } +} + /* protect pages accessed multiple times through file descriptors */ -static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, bool = reclaiming) +static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio, + bool reclaiming, struct gen_update_batch *batch) { + int delta =3D folio_nr_pages(folio); int type =3D folio_is_file_lru(folio); struct lru_gen_folio *lrugen =3D &lruvec->lrugen; int new_gen, old_gen =3D lru_gen_from_seq(lrugen->min_seq[type]); @@ -3120,7 +3158,8 @@ static int folio_inc_gen(struct lruvec *lruvec, struc= t folio *folio, bool reclai new_flags |=3D BIT(PG_reclaim); } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); =20 - lru_gen_update_size(lruvec, folio, old_gen, new_gen); + batch->delta[old_gen] -=3D delta; + batch->delta[new_gen] +=3D delta; =20 return new_gen; } @@ -3663,6 +3702,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int ty= pe, bool can_swap) { int zone; int remaining =3D MAX_LRU_BATCH; + struct gen_update_batch batch =3D { }; struct lru_gen_folio *lrugen =3D &lruvec->lrugen; int new_gen, old_gen =3D lru_gen_from_seq(lrugen->min_seq[type]); =20 @@ -3681,12 +3721,15 @@ static bool inc_min_seq(struct lruvec *lruvec, int = type, bool can_swap) VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) !=3D type, folio); VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D zone, folio); =20 - new_gen =3D folio_inc_gen(lruvec, folio, false); + new_gen =3D folio_inc_gen(lruvec, folio, false, &batch); list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); =20 - if (!--remaining) + if (!--remaining) { + lru_gen_update_batch(lruvec, type, zone, &batch); return false; + } } + lru_gen_update_batch(lruvec, type, zone, &batch); } done: reset_ctrl_pos(lruvec, type, true); @@ -4197,7 +4240,7 @@ static int lru_gen_memcg_seg(struct lruvec *lruvec) *************************************************************************= *****/ =20 static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct = scan_control *sc, - int tier_idx) + int tier_idx, struct gen_update_batch *batch) { bool success; int gen =3D folio_lru_gen(folio); @@ -4239,7 +4282,7 @@ static bool sort_folio(struct lruvec *lruvec, struct = folio *folio, struct scan_c if (tier > tier_idx || refs =3D=3D BIT(LRU_REFS_WIDTH)) { int hist =3D lru_hist_from_seq(lrugen->min_seq[type]); =20 - gen =3D folio_inc_gen(lruvec, folio, false); + gen =3D folio_inc_gen(lruvec, folio, false, batch); list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); =20 WRITE_ONCE(lrugen->protected[hist][type][tier - 1], @@ -4249,7 +4292,7 @@ static bool sort_folio(struct lruvec *lruvec, struct = folio *folio, struct scan_c =20 /* ineligible */ if (zone > sc->reclaim_idx || skip_cma(folio, sc)) { - gen =3D folio_inc_gen(lruvec, folio, false); + gen =3D folio_inc_gen(lruvec, folio, false, batch); list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); return true; } @@ -4257,7 +4300,7 @@ static bool sort_folio(struct lruvec *lruvec, struct = folio *folio, struct scan_c /* waiting for writeback */ if (folio_test_locked(folio) || folio_test_writeback(folio) || (type =3D=3D LRU_GEN_FILE && folio_test_dirty(folio))) { - gen =3D folio_inc_gen(lruvec, folio, true); + gen =3D folio_inc_gen(lruvec, folio, true, batch); list_move(&folio->lru, &lrugen->folios[gen][type][zone]); return true; } @@ -4323,6 +4366,7 @@ static int scan_folios(struct lruvec *lruvec, struct = scan_control *sc, for (i =3D MAX_NR_ZONES; i > 0; i--) { LIST_HEAD(moved); int skipped_zone =3D 0; + struct gen_update_batch batch =3D { }; int zone =3D (sc->reclaim_idx + i) % MAX_NR_ZONES; struct list_head *head =3D &lrugen->folios[gen][type][zone]; =20 @@ -4337,7 +4381,7 @@ static int scan_folios(struct lruvec *lruvec, struct = scan_control *sc, =20 scanned +=3D delta; =20 - if (sort_folio(lruvec, folio, sc, tier)) + if (sort_folio(lruvec, folio, sc, tier, &batch)) sorted +=3D delta; else if (isolate_folio(lruvec, folio, sc)) { list_add(&folio->lru, list); @@ -4357,6 +4401,8 @@ static int scan_folios(struct lruvec *lruvec, struct = scan_control *sc, skipped +=3D skipped_zone; } =20 + lru_gen_update_batch(lruvec, type, zone, &batch); + if (!remaining || isolated >=3D MIN_LRU_BATCH) break; } --=20 2.43.0 From nobody Fri Dec 19 18:53:41 2025 Received: from mail-oa1-f52.google.com (mail-oa1-f52.google.com [209.85.160.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 01E1D18054 for ; Fri, 22 Dec 2023 10:24:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="EkBUT+4j" Received: by mail-oa1-f52.google.com with SMTP id 586e51a60fabf-203fed05a31so1116439fac.0 for ; Fri, 22 Dec 2023 02:24:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703240652; x=1703845452; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=gsAcvd86KuP2ISw06xLpP7WDqEO9VY9n54CovJHHIls=; b=EkBUT+4jVO5UYs3Cgaf5ikrKSKqM1Oknk/QO7sd1R+9evYo9hRvjSM4gUcGwIaf67Z KYNk/kK33+jPs/LfnTPRKvkUVFOoDtWL2q5g14lw+jEFaW7u8NCBu5dynJ3Tbh8UYQgN wdLA7wxYhzIyRGZFyQVMTgP64p3CDSmsr//2ZbfqIOM02ngsX20NqA13h3zTAaXvOYar PZIF7seYGGElVd0yIPEFQ4GlTvnjjM7LaBuAKcLF//y4Z/BnqGyiWRLayq+dHI+K7ZFT u6DZcnXKevdFgBlIfUg6IN23b5auMO7RMax7C+1J9tDmCW6jsz7bVUDMs89huYmKWy2o +FGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703240652; x=1703845452; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=gsAcvd86KuP2ISw06xLpP7WDqEO9VY9n54CovJHHIls=; b=nJjMtva++m0JBfDBI/dtxuHxDFLQfP5V7JQ7hVlaD/9ZK1Rmud2leFXg9DuHu2soJx pJ3OX8f6VVED5aPVJotR+1iw6Y2BEbPo24RmfJj7Y/uY/Jya5fTK+TVBkqVqEeI9x3Fx MsEXi2QIZxhttWO3VMOd9ae5qknpMERmqYBzhuc0nr7/TQMhbUHUiyroCtpawS1CRcKI g9Z4FMJUstIhav9VRMZybrQRQu6BlCOiD782psM6nKjL7/65jiI3QTSaKPqrHK8oNRt1 cBx+kBD+Yew5LEMHYl4MPbbTIDufUgJ5U3yiwoENeUyAoE1+LbaiAz30EdrlI39pFt44 Qiew== X-Gm-Message-State: AOJu0YzJ3zl1lMTyD9NPmNS+7dt+N1dGk6JNbnXqrmWS8y0j+3IjKHsh NC+KRHHxWIkVpULFgjQKYWI= X-Google-Smtp-Source: AGHT+IFM08JVkja56Id51LMPVYml7/oIhHyVGlp94LC9R8zx3T0/LUwfVjdY7Ob42uuEZ9jE1hgJMg== X-Received: by 2002:a05:6870:ce0b:b0:204:5455:c44b with SMTP id xc11-20020a056870ce0b00b002045455c44bmr41106oab.107.1703240651957; Fri, 22 Dec 2023 02:24:11 -0800 (PST) Received: from KASONG-MB2.tencent.com ([103.7.29.31]) by smtp.gmail.com with ESMTPSA id gx1-20020a056a001e0100b006d9912e9a77sm364074pfb.6.2023.12.22.02.24.09 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 22 Dec 2023 02:24:11 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Yu Zhao , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 2/3] mm, lru_gen: move pages in bulk when aging Date: Fri, 22 Dec 2023 18:22:54 +0800 Message-ID: <20231222102255.56993-3-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20231222102255.56993-1-ryncsn@gmail.com> References: <20231222102255.56993-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song Another overhead of aging is page moving. Actually, in most cases, pages are being moved to the same gen after folio_inc_gen is called, especially the protected pages. So it's better to move them in bulk. This also has a good effect on LRU ordering. Currently when MGLRU ages, it walks the LRU backward, and the protected pages are moved to the tail of newer gen one by one, which reverses the order of pages in LRU. Moving them in batches can help keep their order, only in a small scope though due to the scan limit of MAX_LRU_BATCH pages. After this commit, we can see a performance gain: Tested in a 4G memcg on a EPYC 7K62 with: memcached -u nobody -m 16384 -s /tmp/memcached.socket \ -a 0766 -t 16 -B binary & memtier_benchmark -S /tmp/memcached.socket \ -P memcache_binary -n allkeys \ --key-minimum=3D1 --key-maximum=3D16000000 -d 1024 \ --ratio=3D1:0 --key-pattern=3DP:P -c 2 -t 16 --pipeline 8 -x 6 Average result of 18 test runs: Before: 44017.78 Ops/sec After patch 1-2: 44810.01 Ops/sec (+1.8%) Signed-off-by: Kairui Song --- mm/vmscan.c | 84 ++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 71 insertions(+), 13 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index e3b4797b9729..af1266129c1b 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3102,9 +3102,46 @@ static int folio_update_gen(struct folio *folio, int= gen) */ struct gen_update_batch { int delta[MAX_NR_GENS]; + struct folio *head, *tail; }; =20 -static void lru_gen_update_batch(struct lruvec *lruvec, bool type, int zon= e, +static void inline lru_gen_inc_bulk_finish(struct lru_gen_folio *lrugen, + int bulk_gen, bool type, int zone, + struct gen_update_batch *batch) +{ + if (!batch->head) + return; + + list_bulk_move_tail(&lrugen->folios[bulk_gen][type][zone], + &batch->head->lru, + &batch->tail->lru); + + batch->head =3D NULL; +} + +/* + * When aging, protected pages will go to the tail of the same higher + * gen, so the can be moved in batches. Besides reduced overhead, this + * also avoids changing their LRU order in a small scope. + */ +static void inline lru_gen_try_inc_bulk(struct lru_gen_folio *lrugen, stru= ct folio *folio, + int bulk_gen, int gen, bool type, int zone, + struct gen_update_batch *batch) +{ + /* + * If folio not moving to the bulk_gen, it's raced with promotion + * so it need to go to the head of another LRU. + */ + if (bulk_gen !=3D gen) + list_move(&folio->lru, &lrugen->folios[gen][type][zone]); + + if (!batch->head) + batch->tail =3D folio; + + batch->head =3D folio; +} + +static void lru_gen_update_batch(struct lruvec *lruvec, int bulk_gen, bool= type, int zone, struct gen_update_batch *batch) { int gen; @@ -3112,6 +3149,8 @@ static void lru_gen_update_batch(struct lruvec *lruve= c, bool type, int zone, struct lru_gen_folio *lrugen =3D &lruvec->lrugen; enum lru_list lru =3D type ? LRU_INACTIVE_FILE : LRU_INACTIVE_ANON; =20 + lru_gen_inc_bulk_finish(lrugen, bulk_gen, type, zone, batch); + for (gen =3D 0; gen < MAX_NR_GENS; gen++) { int delta =3D batch->delta[gen]; =20 @@ -3705,6 +3744,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int ty= pe, bool can_swap) struct gen_update_batch batch =3D { }; struct lru_gen_folio *lrugen =3D &lruvec->lrugen; int new_gen, old_gen =3D lru_gen_from_seq(lrugen->min_seq[type]); + int bulk_gen =3D (old_gen + 1) % MAX_NR_GENS; =20 if (type =3D=3D LRU_GEN_ANON && !can_swap) goto done; @@ -3712,24 +3752,33 @@ static bool inc_min_seq(struct lruvec *lruvec, int = type, bool can_swap) /* prevent cold/hot inversion if force_scan is true */ for (zone =3D 0; zone < MAX_NR_ZONES; zone++) { struct list_head *head =3D &lrugen->folios[old_gen][type][zone]; + struct folio *prev =3D NULL; =20 - while (!list_empty(head)) { - struct folio *folio =3D lru_to_folio(head); + if (!list_empty(head)) + prev =3D lru_to_folio(head); =20 + while (prev) { + struct folio *folio =3D prev; VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio); VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) !=3D type, folio); VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D zone, folio); =20 + if (unlikely(list_is_first(&folio->lru, head))) + prev =3D NULL; + else + prev =3D lru_to_folio(&folio->lru); + new_gen =3D folio_inc_gen(lruvec, folio, false, &batch); - list_move_tail(&folio->lru, &lrugen->folios[new_gen][type][zone]); + lru_gen_try_inc_bulk(lrugen, folio, bulk_gen, new_gen, type, zone, &bat= ch); =20 if (!--remaining) { - lru_gen_update_batch(lruvec, type, zone, &batch); + lru_gen_update_batch(lruvec, bulk_gen, type, zone, &batch); return false; } } - lru_gen_update_batch(lruvec, type, zone, &batch); + + lru_gen_update_batch(lruvec, bulk_gen, type, zone, &batch); } done: reset_ctrl_pos(lruvec, type, true); @@ -4240,7 +4289,7 @@ static int lru_gen_memcg_seg(struct lruvec *lruvec) *************************************************************************= *****/ =20 static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct = scan_control *sc, - int tier_idx, struct gen_update_batch *batch) + int tier_idx, int bulk_gen, struct gen_update_batch *batch) { bool success; int gen =3D folio_lru_gen(folio); @@ -4283,7 +4332,7 @@ static bool sort_folio(struct lruvec *lruvec, struct = folio *folio, struct scan_c int hist =3D lru_hist_from_seq(lrugen->min_seq[type]); =20 gen =3D folio_inc_gen(lruvec, folio, false, batch); - list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); + lru_gen_try_inc_bulk(lrugen, folio, bulk_gen, gen, type, zone, batch); =20 WRITE_ONCE(lrugen->protected[hist][type][tier - 1], lrugen->protected[hist][type][tier - 1] + delta); @@ -4293,7 +4342,7 @@ static bool sort_folio(struct lruvec *lruvec, struct = folio *folio, struct scan_c /* ineligible */ if (zone > sc->reclaim_idx || skip_cma(folio, sc)) { gen =3D folio_inc_gen(lruvec, folio, false, batch); - list_move_tail(&folio->lru, &lrugen->folios[gen][type][zone]); + lru_gen_try_inc_bulk(lrugen, folio, bulk_gen, gen, type, zone, batch); return true; } =20 @@ -4367,11 +4416,16 @@ static int scan_folios(struct lruvec *lruvec, struc= t scan_control *sc, LIST_HEAD(moved); int skipped_zone =3D 0; struct gen_update_batch batch =3D { }; + int bulk_gen =3D (gen + 1) % MAX_NR_GENS; int zone =3D (sc->reclaim_idx + i) % MAX_NR_ZONES; struct list_head *head =3D &lrugen->folios[gen][type][zone]; + struct folio *prev =3D NULL; =20 - while (!list_empty(head)) { - struct folio *folio =3D lru_to_folio(head); + if (!list_empty(head)) + prev =3D lru_to_folio(head); + + while (prev) { + struct folio *folio =3D prev; int delta =3D folio_nr_pages(folio); =20 VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio); @@ -4380,8 +4434,12 @@ static int scan_folios(struct lruvec *lruvec, struct= scan_control *sc, VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D zone, folio); =20 scanned +=3D delta; + if (unlikely(list_is_first(&folio->lru, head))) + prev =3D NULL; + else + prev =3D lru_to_folio(&folio->lru); =20 - if (sort_folio(lruvec, folio, sc, tier, &batch)) + if (sort_folio(lruvec, folio, sc, tier, bulk_gen, &batch)) sorted +=3D delta; else if (isolate_folio(lruvec, folio, sc)) { list_add(&folio->lru, list); @@ -4401,7 +4459,7 @@ static int scan_folios(struct lruvec *lruvec, struct = scan_control *sc, skipped +=3D skipped_zone; } =20 - lru_gen_update_batch(lruvec, type, zone, &batch); + lru_gen_update_batch(lruvec, bulk_gen, type, zone, &batch); =20 if (!remaining || isolated >=3D MIN_LRU_BATCH) break; --=20 2.43.0 From nobody Fri Dec 19 18:53:41 2025 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B90C618AEF for ; Fri, 22 Dec 2023 10:24:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aEswjhFV" Received: by mail-pf1-f181.google.com with SMTP id d2e1a72fcca58-6d4bef65786so1552202b3a.2 for ; Fri, 22 Dec 2023 02:24:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1703240655; x=1703845455; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=ZVtAklavGTH0ChBjbCr10B7CRN0g8Fxr7pwyrunV9Wg=; b=aEswjhFVbtl4tVqHQuo2E9+S5v6MEYNdo5S31anwKqFyeU0cNQleA+jGCiaBeq7mN/ 2fm8BST1OUgrwKqdAITKNmD4J84yfKOZWYf44uk7rGy4NjaNfFvocdH9MLsDaI+P7QyZ 61eYYtvFUpk+7Qa7PlDFoMf0hMTUDQO+2orFET4SJUFP5PGsT5Pde0JotUmhpevJOhlt l9wY6fZek+PHsXWSnZdyXrZ0a0EQTtWJSX/2gcP64+OUbqBc/D89HuRivBskF3MDdZw3 zhVFijIf+OE8XF2xgK4tl6uC98OxKyNVivupqMhhgUgVOvhG2quoKAUnZHwPlG44Z3OD QFGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1703240655; x=1703845455; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ZVtAklavGTH0ChBjbCr10B7CRN0g8Fxr7pwyrunV9Wg=; b=G1WLOHZCSfXN8SjQfRI/MAMSkutEHmLee5nhHVs9h2Gp+zLUUe0Yk4AxiCUD70KYD7 HhkQxdmrN85iqbLZvZffgH4rxg+MTbD+E70b8otJhTr+Vf1F0xeZknqeOcZrcBZdE657 bCV0RRk6+Y/8o0Ot2iCDvskCCg58ytMCCxnuaF8uSL9eV0GqswoyFjQaKANAKnaqxb2H P+mt7dqyqI5w3VK/0ZfCuAHMyIUlwOBfAnVzEGxP4bpK23O5U7yXWRCwIm5qBRTYvU+t Dq+lWEyPswXkXNXTjdWhGs7jALxfOLwh/kTkSly8bVJJGjRmpPEDdYWP+Vvneulj8DyY 1USg== X-Gm-Message-State: AOJu0YyzyTbN4ZNiF+eNlIZ9s6DAHdt21/NxedV8/IpQ1ngMZgGnVOlv 1hfugSE2mjmcPQo8yd8JU/PKU7QHrmEoES53 X-Google-Smtp-Source: AGHT+IFG6ajXG6vALhMWO57ARn52j3Kb6LdriotQIo1jxt6hAcExEanAl4pobD+1sBFGtt0fLivXqg== X-Received: by 2002:a05:6a20:12c9:b0:190:85d9:9d4a with SMTP id v9-20020a056a2012c900b0019085d99d4amr1304221pzg.97.1703240655043; Fri, 22 Dec 2023 02:24:15 -0800 (PST) Received: from KASONG-MB2.tencent.com ([103.7.29.31]) by smtp.gmail.com with ESMTPSA id gx1-20020a056a001e0100b006d9912e9a77sm364074pfb.6.2023.12.22.02.24.12 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 22 Dec 2023 02:24:14 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Yu Zhao , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 3/3] mm, lru_gen: try to prefetch next page when canning LRU Date: Fri, 22 Dec 2023 18:22:55 +0800 Message-ID: <20231222102255.56993-4-ryncsn@gmail.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20231222102255.56993-1-ryncsn@gmail.com> References: <20231222102255.56993-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song Prefetch for inactive/active LRU have been long exiting, apply the same optimization for MGLRU. Tested in a 4G memcg on a EPYC 7K62 with: memcached -u nobody -m 16384 -s /tmp/memcached.socket \ -a 0766 -t 16 -B binary & memtier_benchmark -S /tmp/memcached.socket \ -P memcache_binary -n allkeys \ --key-minimum=3D1 --key-maximum=3D16000000 -d 1024 \ --ratio=3D1:0 --key-pattern=3DP:P -c 2 -t 16 --pipeline 8 -x 6 Average result of 18 test runs: Before: 44017.78 Ops/sec After patch 1-3: 44890.50 Ops/sec (+1.8%) Signed-off-by: Kairui Song --- mm/vmscan.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index af1266129c1b..1e9d69e18443 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3764,10 +3764,12 @@ static bool inc_min_seq(struct lruvec *lruvec, int = type, bool can_swap) VM_WARN_ON_ONCE_FOLIO(folio_is_file_lru(folio) !=3D type, folio); VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D zone, folio); =20 - if (unlikely(list_is_first(&folio->lru, head))) + if (unlikely(list_is_first(&folio->lru, head))) { prev =3D NULL; - else + } else { prev =3D lru_to_folio(&folio->lru); + prefetchw(&prev->flags); + } =20 new_gen =3D folio_inc_gen(lruvec, folio, false, &batch); lru_gen_try_inc_bulk(lrugen, folio, bulk_gen, new_gen, type, zone, &bat= ch); @@ -4434,10 +4436,12 @@ static int scan_folios(struct lruvec *lruvec, struc= t scan_control *sc, VM_WARN_ON_ONCE_FOLIO(folio_zonenum(folio) !=3D zone, folio); =20 scanned +=3D delta; - if (unlikely(list_is_first(&folio->lru, head))) + if (unlikely(list_is_first(&folio->lru, head))) { prev =3D NULL; - else + } else { prev =3D lru_to_folio(&folio->lru); + prefetchw(&prev->flags); + } =20 if (sort_folio(lruvec, folio, sc, tier, bulk_gen, &batch)) sorted +=3D delta; --=20 2.43.0