From nobody Mon Apr 6 18:38:42 2026 Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 893F83A7846 for ; Wed, 18 Mar 2026 20:04:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773864253; cv=none; b=nKwJ9sgM6EpbDJlPgi4o6Wl3hwBH8ZBSt2h6yc4h9p1FemQ3VYTQF9Zx3PiwV23Fa6ZWruYkyHSxbyUY1pAELR0GI+GhL1TR1qNo8Us/zsqu3tV89MLnkyR/JECha5jvjPxncSt2gvgxiLU74vNWYl1J8UfIg/RgE0RHP0CBAeg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773864253; c=relaxed/simple; bh=W0/oQRBD0+7E2k72mq3zznO1a3Q+lDyYOj6Osyx4hg8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=I/6wbKwNOZLHv6JLQ1D/eLmj9kvDGe0O/VA+LhBQomUGosx7OY/TuV3dpcapSp0wnCq/crqKNeMWjHvBAI35K8om3DOYdQ3SqkYB74wTOZ+dpKw3SL9UmweudtXXMfFIok6La5obEUzJhUaVwcbdI+L7Ip2mQV8SBMM8RkChdIM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=QrGPls/L; arc=none smtp.client-ip=209.85.222.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="QrGPls/L" Received: by mail-qk1-f180.google.com with SMTP id af79cd13be357-8cd80bea5f3so23740285a.3 for ; Wed, 18 Mar 2026 13:04:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1773864251; x=1774469051; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XKEn0eNu+Wls6kueeJsU2hAaS0QRR2i3TclFtAuyFDs=; b=QrGPls/LBwkL7WlrzfIEaGfvgOSFIBid6dftlH7GX4xEgoK333ZY2ZnoAdXWvJpzSu ANk8DzjIgWqJqnWIMrQW21v9wtSf+4azbTKgMuPFpoTZe1PIGZFWhkmbfmWBowNIpgh3 lUHZl/fqo3ynnY/iZZLL2rZ+wVLELx40sAdmp7w6j/39e29hCgYaqZrgnQComLcIBTXJ q7N92mctYSCBUP1ByS+/AftXUmb6FSe2Ikaapz69WVWBdEgv3PfPc/3Mnp2ioRKXblHH nAZWsjPPdd+MyPn9KnK/FojlIPtGn/KFQRly8U9/Ait/xS+kjlI2V2vGzelOk38ArUVu 7INA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1773864251; x=1774469051; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=XKEn0eNu+Wls6kueeJsU2hAaS0QRR2i3TclFtAuyFDs=; b=ETWWR2DTXiGYe7aNYEnKhGs2GI+3/5V9Jlk/id5MjGs6/SIiMsbUhsjID8w4he+ILM RDdS3VYMkSI9Dftg8SGGeC+bgEZEFhqC+tIN6dRXzJMkpnINHH6joqYQkYkT20rik3Op yYnvP0ElMPLXiLEONbklEVgjKGln6cfZXCBztOItWNzV6nGtD2WfjHzDUuLnDKyngoNX R32wD9IcN2Byj/YWZOJSmKYrBIJZWAcuNk4DGhFJ1AHDF7IrYKT70trKJV+7UKCBxfpq 0UBV3kehpXH2imgnvexUIqYxTywrjq1plMnB7Or6rsJmynGMGMYXLG+aGzstFe0z9k78 tLEQ== X-Forwarded-Encrypted: i=1; AJvYcCVJWlzHkqvlblAT7iGdKyjN65AEvd8TpwZBcX8EUvIwsdxoTA/OpwjC3JG9AVsbd+0Z/v+/GmFIan/CbZE=@vger.kernel.org X-Gm-Message-State: AOJu0YzOVC4464Ok9K4inDvZSvKhioeT8Hs0i+kkjU75EhhedSAPwK57 ETFvBR7YCmoiD9zva689fQqvz4jRxCTiCPW3Dtk6mXYnbghy3B1A5Nd01T5EgPQ+FWM= X-Gm-Gg: ATEYQzwzJuvAshfTb3NVIyAmpvqFQ61oTVQIYYaJ0MYvKjYp5PxlOchRVh7ky39kKO5 iHN+8akYQER918sHUSqw1KcMRnn7PosmLHHMp281CW6cGzIecK7yB2OLxj+hAtFafVw8aCk3gci Z186VGcND1h+UjxULiTXPg4TbvdrzRzlf2PQXUt4yB7lpISxcspWPYpCdeh4w1Tw6ZD26ZV8DSy VCt40yr/L7fPflmnxJlE111v4+z16DhpPbh6KUExtGdz3tfPUMB/6UBMquWZCughTkd9Y4T2ABx wVQRKcWyhoLAoxO/dwAoOqj64Sk8MYE6/5wyFFW2VBYEvtWJtw9XGId7FYw+DGc2eXkQNPdWAFZ /jJDHBF1BLmGtsfZsGkNOSOKMgrmD3dhl72VEmn1LLHQ+QXxfYhUVmaOHQL0XzE/9IR3NJkTqXE 7XjvXADxLa/Sp5rQqIp0BIWA== X-Received: by 2002:ad4:4ee8:0:b0:899:ea9e:31c4 with SMTP id 6a1803df08f44-89c6b5c828emr66247796d6.51.1773864251348; Wed, 18 Mar 2026 13:04:11 -0700 (PDT) Received: from localhost ([2603:7000:c00:3a00:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-89c6b9cea7bsm33041426d6.27.2026.03.18.13.04.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Mar 2026 13:04:10 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: David Hildenbrand , Shakeel Butt , Yosry Ahmed , Zi Yan , "Liam R. Howlett" , Usama Arif , Kiryl Shutsemau , Dave Chinner , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 6/7] mm: list_lru: introduce folio_memcg_list_lru_alloc() Date: Wed, 18 Mar 2026 15:53:24 -0400 Message-ID: <20260318200352.1039011-7-hannes@cmpxchg.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260318200352.1039011-1-hannes@cmpxchg.org> References: <20260318200352.1039011-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" memcg_list_lru_alloc() is called every time an object that may end up on the list_lru is created. It needs to quickly check if the list_lru heads for the memcg already exist, and allocate them when they don't. Doing this with folio objects is tricky: folio_memcg() is not stable and requires either RCU protection or pinning the cgroup. But it's desirable to make the existence check lightweight under RCU, and only pin the memcg when we need to allocate list_lru heads and may block. In preparation for switching the THP shrinker to list_lru, add a helper function for allocating list_lru heads coming from a folio. Reviewed-by: David Hildenbrand (Arm) Signed-off-by: Johannes Weiner Acked-by: Shakeel Butt Reviewed-by: Lorenzo Stoakes (Oracle) --- include/linux/list_lru.h | 12 ++++++++++++ mm/list_lru.c | 39 ++++++++++++++++++++++++++++++++++----- 2 files changed, 46 insertions(+), 5 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index 4afc02deb44d..4bd29b61c59a 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -81,6 +81,18 @@ static inline int list_lru_init_memcg_key(struct list_lr= u *lru, struct shrinker =20 int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, gfp_t gfp); + +#ifdef CONFIG_MEMCG +int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru, + gfp_t gfp); +#else +static inline int folio_memcg_list_lru_alloc(struct folio *folio, + struct list_lru *lru, gfp_t gfp) +{ + return 0; +} +#endif + void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup = *parent); =20 /** diff --git a/mm/list_lru.c b/mm/list_lru.c index b817c0f48f73..1ccdd45b1d14 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -537,17 +537,14 @@ static inline bool memcg_list_lru_allocated(struct me= m_cgroup *memcg, return idx < 0 || xa_load(&lru->xa, idx); } =20 -int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, - gfp_t gfp) +static int __memcg_list_lru_alloc(struct mem_cgroup *memcg, + struct list_lru *lru, gfp_t gfp) { unsigned long flags; struct list_lru_memcg *mlru =3D NULL; struct mem_cgroup *pos, *parent; XA_STATE(xas, &lru->xa, 0); =20 - if (!list_lru_memcg_aware(lru) || memcg_list_lru_allocated(memcg, lru)) - return 0; - gfp &=3D GFP_RECLAIM_MASK; /* * Because the list_lru can be reparented to the parent cgroup's @@ -588,6 +585,38 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, str= uct list_lru *lru, =20 return xas_error(&xas); } + +int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, + gfp_t gfp) +{ + if (!list_lru_memcg_aware(lru) || memcg_list_lru_allocated(memcg, lru)) + return 0; + return __memcg_list_lru_alloc(memcg, lru, gfp); +} + +int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru, + gfp_t gfp) +{ + struct mem_cgroup *memcg; + int res; + + if (!list_lru_memcg_aware(lru)) + return 0; + + /* Fast path when list_lru heads already exist */ + rcu_read_lock(); + memcg =3D folio_memcg(folio); + res =3D memcg_list_lru_allocated(memcg, lru); + rcu_read_unlock(); + if (likely(res)) + return 0; + + /* Allocation may block, pin the memcg */ + memcg =3D get_mem_cgroup_from_folio(folio); + res =3D __memcg_list_lru_alloc(memcg, lru, gfp); + mem_cgroup_put(memcg); + return res; +} #else static inline void memcg_init_list_lru(struct list_lru *lru, bool memcg_aw= are) { --=20 2.53.0