From nobody Tue Apr 7 16:31:44 2026 Received: from mail-qk1-f174.google.com (mail-qk1-f174.google.com [209.85.222.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 174EA3B0ADB for ; Thu, 12 Mar 2026 20:53:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773348823; cv=none; b=T5PhgXHLIQqw2hVp5VNfQkKP8UMYYb5CGKVFyZVh0oHCJ8WP8uwgQ53R0OjlCzJYqo+YlX3BtV4WFCIIBUo2Oko9h/ijqhclQlNvOrsBSjKHE59SaYPAdb3FHeRgMkMQbwRT81eN8yb4XI9wKGB1VBgD9/66MQkqsrAO74Yhmrs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773348823; c=relaxed/simple; bh=MVDI0Bqiurks4MrwaZ4b5LImWjwgxUmFmMP1z/OmQWE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Jg2Zqc5X09QtInKLhC8tNwsNTI/THnGrtHHjO5BpgUpALQ5t/WctztYBIVdJrQsSH/CCthcVb2ZlvygJ/y7j8s/Idou08ELluv6eUYp/Oywjz8UfgHo3uRNDOEGJTzyNSTNjkXVcKWKMMhJn4tJkX3suYxzPrGVxCxsNLAxjEBs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=Jg67ecby; arc=none smtp.client-ip=209.85.222.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="Jg67ecby" Received: by mail-qk1-f174.google.com with SMTP id af79cd13be357-8cbb6d5f780so134514485a.1 for ; Thu, 12 Mar 2026 13:53:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1773348819; x=1773953619; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MfCdMvkJxAeZG14cTtGmM/ZN7/Bzr7yHmjt9XpyWCt8=; b=Jg67ecbyJX//UzSjM3saOVg6FBhO7AcH3C7sYmfXH2HkcQuzyJZWniB9fpIdD6KIUZ 6/l+Eucj43bGIsmgCr+Ov6EmsFTSy//nTmZfzNTLEitrbnQSMoCk6Q+Xj3WclU10EYfW mcHBJdMKap9TeHdRVjPhNT2LynM+6hjwc962MOVh1IN7VN98QvhrwD3fOVAYREAJcJqG Dkj7oUEzz+e1lifvO9m7w+DVBYwVjaXfNJSUFVgDmhNqQO1asBdzyHv+b7cNB68mMH/z p3hvM3qc3S/NZzdieAxIYHLR1vsgwqv7uKEB/F+Jikz5QAVCVxWBbbwjFECMH2Xjwmfq zv9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773348819; x=1773953619; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=MfCdMvkJxAeZG14cTtGmM/ZN7/Bzr7yHmjt9XpyWCt8=; b=gmYpoEGhUrdX+uH7IslDAQTJoD1gknkoLmNiarWChtEAF63xEXMy6BN1mKXbP3pCvX FfjM+AvU/Q8cw/HUJZeGZDoOFXraLrCghQx34ltbVCeAcCYO5723GEsZy5FpG0IbMwAL fhWk9YQa/KYM/752BZey0MEqOebFAwG5ZZEHQ0WAkOs531T3yDVT2rECy4BX2xL7S2Wx u8nTm7n8oLRCMb5XIOTDGYc45Boq3Fv09+8lkadgPoKEV1+x9BscwnMG8xafSpNYAjBt AGvk3RwBEVZqnnJ2EtdmrZ05bKjrr5w6ytj64v1ToB9xdkAdM0wIPS7Dl3LVP5y37gOR XTSg== X-Forwarded-Encrypted: i=1; AJvYcCVOiB2feB7ddnloYX5qH0LsDHernpo3StN7ogu70MsuHNDI4kN+GYKHWOvVh4zecKm1VWWg+GqVqF2MgHI=@vger.kernel.org X-Gm-Message-State: AOJu0YyEp7LDx0xPjI9F3q3EXpsMeLjB05WI2mMJjyWyn/5AeSOpP25s krvgiyOVrbUz3dRcvTnHlQXNw/6KZ3LQjxdKtX5uC9CRHjc9b/6lE2O3JJegpoFjUVk= X-Gm-Gg: ATEYQzzWnH0mQQlbdHokWaw5i2EW1+Q2YV5xzp/JDmG7naOcDc7U+rAFQFUNsr1ZTtb bpTNmAXJlgL9lITKa2IESuOUDBaVdrLFoOn773NNIq7DiWrJzgayQZ8RywBR4dmfycGDSfZU/xj QroFOM5ZfXCVz9D7y7H0R48S+DzXXpHr1PgJTUqaerXasGRpg/xe7AmEZCTWqXjJgW0O+YJowj7 +OVaQOAU9ZhhPX8gU19DwpmoqDgwGgmMdnT5EdAC52vvm9W0CutcwZDHqVjcbLMg9cWxqehKgeU 3JmrCDR3Lg7K+ZN8p+Cv81cLZUHqYudIBC1R9YthGueeJL+VNyAufM0Kta2sjtQwfnQNc1z+RKO lH0MH5t9KLXIrtrJoZmFjtqmAfGBVQkGJIVfbjeOH7/G8aJaSTpD50K9mYXSBXQUqWja5uhXARJ H1Ia+q5jodIk9nL+SW4q4qYg== X-Received: by 2002:a05:620a:1724:b0:8cd:7f7d:b091 with SMTP id af79cd13be357-8cdb5bce7bcmr154603785a.71.1773348818838; Thu, 12 Mar 2026 13:53:38 -0700 (PDT) Received: from localhost ([2603:7000:c00:3a00:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id af79cd13be357-8cda1fbd120sm412873485a.7.2026.03.12.13.53.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2026 13:53:37 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: David Hildenbrand , Shakeel Butt , Yosry Ahmed , Zi Yan , "Liam R. Howlett" , Usama Arif , Kiryl Shutsemau , Dave Chinner , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 5/7] mm: list_lru: introduce caller locking for additions and deletions Date: Thu, 12 Mar 2026 16:51:53 -0400 Message-ID: <20260312205321.638053-6-hannes@cmpxchg.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260312205321.638053-1-hannes@cmpxchg.org> References: <20260312205321.638053-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Locking is currently internal to the list_lru API. However, a caller might want to keep auxiliary state synchronized with the LRU state. For example, the THP shrinker uses the lock of its custom LRU to keep PG_partially_mapped and vmstats consistent. To allow the THP shrinker to switch to list_lru, provide normal and irqsafe locking primitives as well as caller-locked variants of the addition and deletion functions. Signed-off-by: Johannes Weiner Reviewed-by: David Hildenbrand (Arm) --- include/linux/list_lru.h | 34 +++++++++++++ mm/list_lru.c | 104 +++++++++++++++++++++++++++------------ 2 files changed, 107 insertions(+), 31 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index fe739d35a864..4afc02deb44d 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -83,6 +83,40 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, struc= t list_lru *lru, gfp_t gfp); void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup = *parent); =20 +/** + * list_lru_lock: lock the sublist for the given node and memcg + * @lru: the lru pointer + * @nid: the node id of the sublist to lock. + * @memcg: the cgroup of the sublist to lock. + * + * Returns the locked list_lru_one sublist. The caller must call + * list_lru_unlock() when done. + * + * You must ensure that the memcg is not freed during this call (e.g., with + * rcu or by taking a css refcnt). + * + * Return: the locked list_lru_one, or NULL on failure + */ +struct list_lru_one *list_lru_lock(struct list_lru *lru, int nid, + struct mem_cgroup *memcg); + +/** + * list_lru_unlock: unlock a sublist locked by list_lru_lock() + * @l: the list_lru_one to unlock + */ +void list_lru_unlock(struct list_lru_one *l); + +struct list_lru_one *list_lru_lock_irqsave(struct list_lru *lru, int nid, + struct mem_cgroup *memcg, unsigned long *irq_flags); +void list_lru_unlock_irqrestore(struct list_lru_one *l, + unsigned long *irq_flags); + +/* Caller-locked variants, see list_lru_add() etc for documentation */ +bool __list_lru_add(struct list_lru *lru, struct list_lru_one *l, + struct list_head *item, int nid, struct mem_cgroup *memcg); +bool __list_lru_del(struct list_lru *lru, struct list_lru_one *l, + struct list_head *item, int nid); + /** * list_lru_add: add an element to the lru list's tail * @lru: the lru pointer diff --git a/mm/list_lru.c b/mm/list_lru.c index 4d74c2e9c2a5..779cb26cec84 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -15,17 +15,23 @@ #include "slab.h" #include "internal.h" =20 -static inline void lock_list_lru(struct list_lru_one *l, bool irq) +static inline void lock_list_lru(struct list_lru_one *l, bool irq, + unsigned long *irq_flags) { - if (irq) + if (irq_flags) + spin_lock_irqsave(&l->lock, *irq_flags); + else if (irq) spin_lock_irq(&l->lock); else spin_lock(&l->lock); } =20 -static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off) +static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off, + unsigned long *irq_flags) { - if (irq_off) + if (irq_flags) + spin_unlock_irqrestore(&l->lock, *irq_flags); + else if (irq_off) spin_unlock_irq(&l->lock); else spin_unlock(&l->lock); @@ -78,7 +84,7 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, in= t idx) =20 static inline struct list_lru_one * lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *m= emcg, - bool irq, bool skip_empty) + bool irq, unsigned long *irq_flags, bool skip_empty) { struct list_lru_one *l; =20 @@ -86,12 +92,12 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, s= truct mem_cgroup *memcg, again: l =3D list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); if (likely(l)) { - lock_list_lru(l, irq); + lock_list_lru(l, irq, irq_flags); if (likely(READ_ONCE(l->nr_items) !=3D LONG_MIN)) { rcu_read_unlock(); return l; } - unlock_list_lru(l, irq); + unlock_list_lru(l, irq, irq_flags); } /* * Caller may simply bail out if raced with reparenting or @@ -132,37 +138,79 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid= , int idx) =20 static inline struct list_lru_one * lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *m= emcg, - bool irq, bool skip_empty) + bool irq, unsigned long *irq_flags, bool skip_empty) { struct list_lru_one *l =3D &lru->node[nid].lru; =20 - lock_list_lru(l, irq); + lock_list_lru(l, irq, irq_flags); =20 return l; } #endif /* CONFIG_MEMCG */ =20 -/* The caller must ensure the memcg lifetime. */ -bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid, - struct mem_cgroup *memcg) +struct list_lru_one *list_lru_lock(struct list_lru *lru, int nid, + struct mem_cgroup *memcg) { - struct list_lru_node *nlru =3D &lru->node[nid]; - struct list_lru_one *l; + return lock_list_lru_of_memcg(lru, nid, memcg, false, NULL, false); +} + +void list_lru_unlock(struct list_lru_one *l) +{ + unlock_list_lru(l, false, NULL); +} + +struct list_lru_one *list_lru_lock_irqsave(struct list_lru *lru, int nid, + struct mem_cgroup *memcg, + unsigned long *flags) +{ + return lock_list_lru_of_memcg(lru, nid, memcg, true, flags, false); +} + +void list_lru_unlock_irqrestore(struct list_lru_one *l, unsigned long *fla= gs) +{ + unlock_list_lru(l, true, flags); +} =20 - l =3D lock_list_lru_of_memcg(lru, nid, memcg, false, false); +bool __list_lru_add(struct list_lru *lru, struct list_lru_one *l, + struct list_head *item, int nid, + struct mem_cgroup *memcg) +{ if (list_empty(item)) { list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ if (!l->nr_items++) set_shrinker_bit(memcg, nid, lru_shrinker_id(lru)); - unlock_list_lru(l, false); - atomic_long_inc(&nlru->nr_items); + atomic_long_inc(&lru->node[nid].nr_items); + return true; + } + return false; +} + +bool __list_lru_del(struct list_lru *lru, struct list_lru_one *l, + struct list_head *item, int nid) +{ + if (!list_empty(item)) { + list_del_init(item); + l->nr_items--; + atomic_long_dec(&lru->node[nid].nr_items); return true; } - unlock_list_lru(l, false); return false; } =20 +/* The caller must ensure the memcg lifetime. */ +bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid, + struct mem_cgroup *memcg) +{ + struct list_lru_one *l; + bool ret; + + l =3D list_lru_lock(lru, nid, memcg); + ret =3D __list_lru_add(lru, l, item, nid, memcg); + list_lru_unlock(l); + return ret; +} + bool list_lru_add_obj(struct list_lru *lru, struct list_head *item) { bool ret; @@ -184,19 +232,13 @@ EXPORT_SYMBOL_GPL(list_lru_add_obj); bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid, struct mem_cgroup *memcg) { - struct list_lru_node *nlru =3D &lru->node[nid]; struct list_lru_one *l; + bool ret; =20 - l =3D lock_list_lru_of_memcg(lru, nid, memcg, false, false); - if (!list_empty(item)) { - list_del_init(item); - l->nr_items--; - unlock_list_lru(l, false); - atomic_long_dec(&nlru->nr_items); - return true; - } - unlock_list_lru(l, false); - return false; + l =3D list_lru_lock(lru, nid, memcg); + ret =3D __list_lru_del(lru, l, item, nid); + list_lru_unlock(l); + return ret; } =20 bool list_lru_del_obj(struct list_lru *lru, struct list_head *item) @@ -269,7 +311,7 @@ __list_lru_walk_one(struct list_lru *lru, int nid, stru= ct mem_cgroup *memcg, unsigned long isolated =3D 0; =20 restart: - l =3D lock_list_lru_of_memcg(lru, nid, memcg, irq_off, true); + l =3D lock_list_lru_of_memcg(lru, nid, memcg, irq_off, NULL, true); if (!l) return isolated; list_for_each_safe(item, n, &l->list) { @@ -310,7 +352,7 @@ __list_lru_walk_one(struct list_lru *lru, int nid, stru= ct mem_cgroup *memcg, BUG(); } } - unlock_list_lru(l, irq_off); + unlock_list_lru(l, irq_off, NULL); out: return isolated; } --=20 2.53.0