From nobody Tue Apr 7 16:16:03 2026 Received: from mail-qk1-f169.google.com (mail-qk1-f169.google.com [209.85.222.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 64B043B6C07 for ; Thu, 12 Mar 2026 20:53:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773348825; cv=none; b=nADBFT+NjnucMV5m0DNQLk/XCr+NSH735jlXCsXiZiHVLVKHmEqJPmVheHy4DRP+KTrfG9tzFmSvLEnl6y+ljqziIN7JRGOYK6AHtbTwCJHMzE2jQsZrZjFdg4FsStHR8bUbprgG7JwcppZ5rDu9jlf3/Is+Zkksuto4PXJ+fjc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773348825; c=relaxed/simple; bh=r6j7CBaD6bfWmHnyhYFjVx4fURy7OqHj6oW9zFCS6k4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=l2AxB/48/lpiYBxw5EJBB/BaRezbXY87B5rIphJ10p2URPu1cUoYd3Cbjq4vEC5AXuFyijDedAvLdRH2b7SxxpmQELk6/JiB8hsk6XWxR4gO5FNlI1rYZm+DuTvTNYZeUT8mmlfE46qDVZsI/DdoEIFIKfAQXg2IU8QEXamlCjM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=N5GS6orp; arc=none smtp.client-ip=209.85.222.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="N5GS6orp" Received: by mail-qk1-f169.google.com with SMTP id af79cd13be357-8cd8c6e365eso144347685a.0 for ; Thu, 12 Mar 2026 13:53:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1773348821; x=1773953621; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H+fNWmGr+tcL8ctiy45XUsI2tWZ66Zdo3aVc9uo+wjs=; b=N5GS6orpHZ4POXe/aNuIKE5UlIZWNCl0qcf5DWrGo1yYSmBiqpIcbD62Etca7qLKHD E1jiL8OMVdzT2UDrY/In1eHok0jaJH7v9Jd8MvqmPvL84gyslF7BKGPYPYC+Nxowx0Yb l3HOcUAuz8+byheMtrlDvSU+Bc5NP6G+mfuloaLIvFzNPK77ALwjxEpMn121z07WGQeK jZGDuSmDxQ8XEr4G4RyS/uTiiaHfnVce4MBXD4yEw31IeYdBsHRzCw/3DeWqEZ8TuEsb mAf6iRAgs9faGVCphAz9hWZn0/LOHXG8r5N4kstEI70pgD8uH/8ON20H/+DOAOAJBe1r lR/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773348821; x=1773953621; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=H+fNWmGr+tcL8ctiy45XUsI2tWZ66Zdo3aVc9uo+wjs=; b=KqmvOoyeh7loGgGvU8Es/4P/gyiBSRUQsFkr5pTJ1+glMCu2c7OCWzzgUq5aiKN4Xu XnOwPfCTu6Ql2CRW+ivMI55taNE+qGGpbdkXrbCOTSBBYNo7/Qe0C2RJpU9Vwvsp9krT SfPNCsXsE/2lwi+csi/D9yjZTmN8Bczi4ERQYbEh0OhigzwfZPeXU4qnTqF0uS6/+JNb PhiBH7ZtIuz3qNaH4o+aZsiXPmlwfQmXIyRjrMZzBy4/TszWHve7I9NxxNcIqwR5XmnE NUItajmoo5xobpUq16SwZ/dYAwHC/5Z07bwL57fxH7VdTUyAlnDY2o2wsAPLad2+JDp3 B6aw== X-Forwarded-Encrypted: i=1; AJvYcCW3MHVHRTaxGhFKjdRHvtQ11ihx54UzKu11K8zm4+VtaOErg5rmz7SDXxFhyqZJlrcrz6h41Gr3K8IEOFw=@vger.kernel.org X-Gm-Message-State: AOJu0Yxy+pQx6vlqk9F+o3zS/KIH6Oe31BVFAdJqWK/cDBK+HjNYJN0k ipMToaDYzl8on6tEcSu1fBuVFsZZooA23BZ5vTE8wV47VMfza2KmIAvTjqWuUlESInk= X-Gm-Gg: ATEYQzxI19SJKQisEOp58QN/4w3weZ+cuy+m3NgGEf9Zx+gQIKGVj3Drcoj0mt0ghNk sou/W3keOGAwgR5Vh2WGXXJRVd0tTix7u24YP+m56q21NO3FkSvja/LZUmE6L9Kyv4DGgsFgXnt GVWNvMeO4R4dzC6J4KzaHtOJrePKwgrzDCBYg4V1Y3/DYTHAzQTo8KeFUgqh0cwGavCqH+HS8dq zRu4hmHGdygnDVVvL/5dX6a8tmKCwp5iAgZKACwZE9YccfjQHNiyclhRvvD/nNKRryJDhV/1fEF NP/g2MCt7iSL6CAw4swHVZGXi/MtNxjSuJ0VVUrcCENOk57Cwemlt5H40dqZKhUoU+d0BLlkUzV UdSCQRIUV/Y+6z7irXBrb9oXDeTId4h7Yqg2OtCtJarcplhHPHokT0atAJX/7DYdxEA1EAAFwua EnLPZWfQafT6mzmVSNclmINQ== X-Received: by 2002:a05:620a:40d3:b0:8ca:3d7c:e767 with SMTP id af79cd13be357-8cdb5b5f305mr157157385a.52.1773348821047; Thu, 12 Mar 2026 13:53:41 -0700 (PDT) Received: from localhost ([2603:7000:c00:3a00:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id af79cd13be357-8cda1fbcc5fsm465928085a.4.2026.03.12.13.53.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2026 13:53:40 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: David Hildenbrand , Shakeel Butt , Yosry Ahmed , Zi Yan , "Liam R. Howlett" , Usama Arif , Kiryl Shutsemau , Dave Chinner , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 6/7] mm: list_lru: introduce memcg_list_lru_alloc_folio() Date: Thu, 12 Mar 2026 16:51:54 -0400 Message-ID: <20260312205321.638053-7-hannes@cmpxchg.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260312205321.638053-1-hannes@cmpxchg.org> References: <20260312205321.638053-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" memcg_list_lru_alloc() is called every time an object that may end up on the list_lru is created. It needs to quickly check if the list_lru heads for the memcg already exist, and allocate them when they don't. Doing this with folio objects is tricky: folio_memcg() is not stable and requires either RCU protection or pinning the cgroup. But it's desirable to make the existence check lightweight under RCU, and only pin the memcg when we need to allocate list_lru heads and may block. In preparation for switching the THP shrinker to list_lru, add a helper function for allocating list_lru heads coming from a folio. Signed-off-by: Johannes Weiner Reviewed-by: David Hildenbrand (Arm) --- include/linux/list_lru.h | 12 ++++++++++++ mm/list_lru.c | 39 ++++++++++++++++++++++++++++++++++----- 2 files changed, 46 insertions(+), 5 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index 4afc02deb44d..df6bd3c64b06 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -81,6 +81,18 @@ static inline int list_lru_init_memcg_key(struct list_lr= u *lru, struct shrinker =20 int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, gfp_t gfp); + +#ifdef CONFIG_MEMCG +int memcg_list_lru_alloc_folio(struct folio *folio, struct list_lru *lru, + gfp_t gfp); +#else +static inline int memcg_list_lru_alloc_folio(struct folio *folio, + struct list_lru *lru, gfp_t gfp) +{ + return 0; +} +#endif + void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup = *parent); =20 /** diff --git a/mm/list_lru.c b/mm/list_lru.c index 779cb26cec84..562b2b1f8c41 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -534,17 +534,14 @@ static inline bool memcg_list_lru_allocated(struct me= m_cgroup *memcg, return idx < 0 || xa_load(&lru->xa, idx); } =20 -int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, - gfp_t gfp) +static int __memcg_list_lru_alloc(struct mem_cgroup *memcg, + struct list_lru *lru, gfp_t gfp) { unsigned long flags; struct list_lru_memcg *mlru =3D NULL; struct mem_cgroup *pos, *parent; XA_STATE(xas, &lru->xa, 0); =20 - if (!list_lru_memcg_aware(lru) || memcg_list_lru_allocated(memcg, lru)) - return 0; - gfp &=3D GFP_RECLAIM_MASK; /* * Because the list_lru can be reparented to the parent cgroup's @@ -585,6 +582,38 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, str= uct list_lru *lru, =20 return xas_error(&xas); } + +int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, + gfp_t gfp) +{ + if (!list_lru_memcg_aware(lru) || memcg_list_lru_allocated(memcg, lru)) + return 0; + return __memcg_list_lru_alloc(memcg, lru, gfp); +} + +int memcg_list_lru_alloc_folio(struct folio *folio, struct list_lru *lru, + gfp_t gfp) +{ + struct mem_cgroup *memcg; + int res; + + if (!list_lru_memcg_aware(lru)) + return 0; + + /* Fast path when list_lru heads already exist */ + rcu_read_lock(); + memcg =3D folio_memcg(folio); + res =3D memcg_list_lru_allocated(memcg, lru); + rcu_read_unlock(); + if (likely(res)) + return 0; + + /* Allocation may block, pin the memcg */ + memcg =3D get_mem_cgroup_from_folio(folio); + res =3D __memcg_list_lru_alloc(memcg, lru, gfp); + mem_cgroup_put(memcg); + return res; +} #else static inline void memcg_init_list_lru(struct list_lru *lru, bool memcg_aw= are) { --=20 2.53.0