From nobody Sun Dec 14 21:43:45 2025 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BDC162475C9 for ; Wed, 15 Jan 2025 12:06:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736942806; cv=none; b=alBuAFeY6qvvecplCi7MRoqFtXVIxR+tsVJFF6ZiCw6kGSLttiEhsRtLQLQbT0Aq4w4IRHlGHVIhEnztIteDVnwSP5ZqnZ9dFQ6FHNXe7WFaLElGK7X96ijPPapXmkwgBReMZiFtNbVV2trx2rveeMxSIZiGbJ0o8zfM/40k7GQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1736942806; c=relaxed/simple; bh=0iNu0ZqnCgSRvfLDS8SJ3fpeOCW7lm/qfjXz/J2rWA4=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version:Content-Type; b=b0HEzCWAiiVSPf/njWrGKrSQ/BIpwr87M+KBHsz6LnjhOnPJ8XY6er+Fl5lMk5y8c06QOQGQDG028i9dD7ban8x0ZPo6HMa1R+WY01Voniv31fLTtKKxXKMVvNyLOt6bJomkKEclY4n1GMvQDLCZTen4fmoDFHN9EPY9TCy/jos= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=sYcRaUr/; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="sYcRaUr/" Received: from pps.filterd (m0353725.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 50F1PkAF013432; Wed, 15 Jan 2025 12:06:31 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:content-type:date:from:message-id :mime-version:subject:to; s=pp1; bh=Q68++GGoXmKPwhjOkPGV8ZviSVNK RTXNxz2brydr0VQ=; b=sYcRaUr/+MjKgtzi8+8jvnBc92lu0ZBxgxbxwjq4Lsou SAuo/q4KIaU/rV0IUfHxoc6WGodWk9D+uYck1Nv3ZYUhVkWnSNm6aDULdFFm7xz7 XjvRoHcEhyrvw9Mj8q47lPvdVItZN7+ifRVcE36Di6GbXRmthBFCg2+P6mxyCwNx ld5rBF+9NAga7lecIdRstclzSOIBHX68FDqHYhZvPLtLwmSl4RKu0unT8j3KfUyf u73JLpRf/LPahgCoUWoM8lE0TEvdGggdtbI5jNZhYjuxtEmZxJ8mQ9yl8ayvQ+qk vlkQkq70sCeekebGhpLgBsnkoKm6+HDDsQSmVDbpkg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 445sd64vp1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 15 Jan 2025 12:06:31 +0000 (GMT) Received: from m0353725.ppops.net (m0353725.ppops.net [127.0.0.1]) by pps.reinject (8.18.0.8/8.18.0.8) with ESMTP id 50FC5Q6b028736; Wed, 15 Jan 2025 12:06:30 GMT Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 445sd64vny-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 15 Jan 2025 12:06:30 +0000 (GMT) Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 50FBr1ZL001110; Wed, 15 Jan 2025 12:06:30 GMT Received: from smtprelay03.fra02v.mail.ibm.com ([9.218.2.224]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 44456jyw9y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 15 Jan 2025 12:06:29 +0000 Received: from smtpav01.fra02v.mail.ibm.com (smtpav01.fra02v.mail.ibm.com [10.20.54.100]) by smtprelay03.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 50FC6Sd335520936 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 15 Jan 2025 12:06:28 GMT Received: from smtpav01.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 384C920043; Wed, 15 Jan 2025 12:06:28 +0000 (GMT) Received: from smtpav01.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 78F0B20040; Wed, 15 Jan 2025 12:06:26 +0000 (GMT) Received: from ltczz402-lp1.aus.stglabs.ibm.com (unknown [9.40.194.31]) by smtpav01.fra02v.mail.ibm.com (Postfix) with ESMTP; Wed, 15 Jan 2025 12:06:26 +0000 (GMT) From: Donet Tom To: Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Gregory Price Cc: Yu Zhao , Ritesh Harjani , "Aneesh Kumar K . V" , Matthew Wilcox , David Hildenbrand , Huang Ying , Johannes Weiner Subject: [RFC] mm/swap.c: Enable promotion of unmapped MGLRU page cache pages Date: Wed, 15 Jan 2025 06:06:25 -0600 Message-ID: <20250115120625.3785-1-donettom@linux.ibm.com> X-Mailer: git-send-email 2.43.5 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-GUID: LGMfVWDP16kUO1s0Ia1wCWkUkb321qRc X-Proofpoint-ORIG-GUID: YmM6d4CDL8wePF6S2SgvrI69I_a86sT1 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-01-15_04,2025-01-15_02,2024-11-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 adultscore=0 clxscore=1015 phishscore=0 spamscore=0 suspectscore=0 priorityscore=1501 mlxlogscore=999 bulkscore=0 lowpriorityscore=0 malwarescore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2411120000 definitions=main-2501150091 This patch is based on patch [1], which introduced support for promoting unmapped normal LRU page cache pages. Here, we extend the functionality to support promotion of MGLRU page cache pages. An MGLRU page cache page is eligible for promotion when: 1. Memory Tiering and pagecache_promotion_enabled are enabled 2. It resides in a lower memory tier. 3. It is referenced. 4. It is part of the working set. 5. It belongs to the active list. For MGLRU, the youngest generation and the youngest generation - 1 are treated as the active list. When a page is accessed through a file descriptor, folio_inc_refs() is invoked. In this function, we check whether the page is referenced, is part of the working set, and Belongs to the active list. If all these conditions are met, the page is added to the promotion list. The per-process task task_numa_promotion_work() take the pages from the promotion list and promotes them to a higher memory tier. Test process: We measured the read time in below scenarios for both LRU and MGLRU. Scenario 1: Pages are on Lower tier + promotion off Scenario 2: Pages are on Lower tier + promotion on Scenario 3: Pages are on higher tier Test Results MGLRU --------------------------------------------------------------- Pages on higher | Pages Lower tier | Pages on Lower Tier | Tier | promotion off | Promotion On | --------------------------------------------------------------- 1.5s | 3.2s |During Promotion - 6.6s | | |After Promotion - 1.6s | | | | --------------------------------------------------------------- Test Results LRU --------------------------------------------------------------- Pages on higher | Pages Lower tier | Pages on Lower Tier | Tier | promotion off | Promotion On | --------------------------------------------------------------- 1.5s | 3.2s |During Promotion - 4.2s | | | - 3.4s | | | - 5.6s | | |After Promotion - 1.6s | | | | --------------------------------------------------------------- In LRU, pages are initially added to the inactive list. When a page is referenced, it is moved to the active list, and promotion to a higher memory tier occurs from the active list. This process often requires multiple reads to trigger promotion. In contrast, MGLRU adds new pages to the youngest generation immediately. As a result, we observe that MGLRU promotes pages on the first read itself, unlike LRU which takes multiple reads to trigger promotion. This difference also impacts read latency: For MGLRU, the first read shows higher latency due to the combined overhead of accessing a lower tier and performing promotion. For LRU, the first 3=E2=80=934 reads typically exhibit lower latency since promotion does not occur immediately. MGLRU and LRU are showing similar performance benefit. [1] https://lore.kernel.org/all/20250107000346.1338481-1-gourry@gourry.net/ Signed-off-by: Donet Tom --- mm/swap.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index b2341bc18452..121de1d7e938 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -386,6 +386,9 @@ static void __lru_cache_activate_folio(struct folio *fo= lio) =20 static void lru_gen_inc_refs(struct folio *folio) { + struct mem_cgroup *memcg; + struct lruvec *lruvec; + int gen; unsigned long new_flags, old_flags =3D READ_ONCE(folio->flags); =20 if (folio_test_unevictable(folio)) @@ -399,13 +402,29 @@ static void lru_gen_inc_refs(struct folio *folio) =20 do { if ((old_flags & LRU_REFS_MASK) =3D=3D LRU_REFS_MASK) { - if (!folio_test_workingset(folio)) + if (!folio_test_workingset(folio)) { folio_set_workingset(folio); - return; + return; + } + goto promo_candid; } =20 new_flags =3D old_flags + BIT(LRU_REFS_PGOFF); } while (!try_cmpxchg(&folio->flags, &old_flags, new_flags)); + +promo_candid: + if (!folio_test_isolated(folio) && + (sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING) && + numa_pagecache_promotion_enabled) { + memcg =3D folio_memcg(folio); + if (memcg) { + lruvec =3D mem_cgroup_lruvec(memcg, folio_pgdat(folio)); + gen =3D folio_lru_gen(folio); + + if ((gen < MAX_NR_GENS) && lru_gen_is_active(lruvec, gen)) + promotion_candidate(folio); + } + } } =20 static bool lru_gen_clear_refs(struct folio *folio) --=20 2.43.5