From nobody Tue Apr 7 14:38:27 2026 Received: from mail-qv1-f45.google.com (mail-qv1-f45.google.com [209.85.219.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D91EC38AC87 for ; Thu, 12 Mar 2026 20:53:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773348815; cv=none; b=Bihprp9F4UV7NrQ0d2r3Qxl6vR+WNjyXIHaIt5d/kp0jLicgEaAOUMaT8TTL8jiF9wvtvFdC/2sWjLeqlM3wRYs/5H1B0wU8rG1IUsoHdTFaZcBFUSvCy/pwM0my8eiS8NFZRC55u0EQrVv2xLl28SU/ilKL0m6lV5FTfJMccJs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773348815; c=relaxed/simple; bh=Idr2THtP4YQirdU2zLaM8WYEx2j3lup/qgmJYBXFWVc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jp5v2+PzpW6GEO94k4celYZ/+i2eztJ2GKuIknEFagG5ZBhu4xwYKdKFaxNO5SxvnR/KDOT/JSFaT9Olyp0yGRhhn+mxEzVgypvuQavMqfTNI7w3jaZG9SdduLfD2qSeELOKyeNP/arizn17eEdMrlZCdzxvlkiYo2lgmH/RCoA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=YWyGB41B; arc=none smtp.client-ip=209.85.219.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="YWyGB41B" Received: by mail-qv1-f45.google.com with SMTP id 6a1803df08f44-89a1d7cc7f0so13427456d6.1 for ; Thu, 12 Mar 2026 13:53:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1773348811; x=1773953611; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Aqbg0K2IsgJ/qrqREwPFlBusaMsCq6IazOmM3TBzhTA=; b=YWyGB41BzS3caxKVY1COYMJB3FBJwTUYpXOUCtqUpaiXGoy6kmE1n/qxZUAI5qW1LB 8EAcfQ9itTmKEh60SgcdWPIeYLcnz1wyHebuneLxe5FCT7+C+C74G1ikLukj7Shsx8gw hodj6miAb7b4RKv5OmqXnE8aghBFS02QghrDVhsEHlyx62lMHH/3cluZ26H3ZDh2lcvR XudQevu8JwNUCkZ5Cq+UFjRSpPrYxGnb33Jh4x06OylU3UHOtifS8h/z/ZoQGz6DaPpe UsUW54zm/jlcJER8VsRDIM13UYI1Z79iImpAOsAo02jhjZcdSgwpDuA90kIHy/GLPYvL L6+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773348811; x=1773953611; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Aqbg0K2IsgJ/qrqREwPFlBusaMsCq6IazOmM3TBzhTA=; b=ErrnscF04mgT609NhxJ3sliMi6KpoIjtsRGfMZxyTSSgn2e388Wj82+UYSN9dyZ/Hg dvsazX8LBQzoxs2ivImFZo5i/fBeRo/Z1MYA2iI8+zvaIrFYfItR0qwj2fSPAJStw51t JpnqwaTpCQePoal80oHZkUDAYLZTHkzKxoQ4lLrecEfeOh0nQrB3tbexNad7Snp/BCpt +6NYj5g3FVW3h6rWhKdfLJYe+N9ih72KAjPzxfU1ck02XiILJJqe+dGWZG78Ba2ZRy/I e044SKZ0KEEvD2TPPWVJWZ5Wd9mghdlAowzcS+hrPP2+7K2JrAGPga4P1GLWz0ISH+6a GMcw== X-Forwarded-Encrypted: i=1; AJvYcCUcokNBleN+HQkx4mzya4hoCzq5frP9YFPClR2rMuTfs9zYy1T6Wg9TXGIdw2zJvfkykZWqivIDSDBPLJ0=@vger.kernel.org X-Gm-Message-State: AOJu0YxWe1eBUCGjFqRiUNao/BfalhwHKSP5PqYrMxjDQ9GYKyoGQ6W5 n90m3Zxig8L+DixCOAdutc/D4wpOzX9Ek30DPQMim3II0QH9NlONhA6JWBJLPJpD8To= X-Gm-Gg: ATEYQzwwKI7rOiPbJjejUa/eUd8mzuwzC/253HJQt5rw0zsQ1hLiIeeR9hgkeKI6qhr TpnDNh4laYFo+bfBCxCJFWdUGgHfqyk1qeUI1bZkOLbEoFvzyEUNXqLqrkV9k1PjD+PHctTXxxD BsrOvloB5wpTTMMUPRGrgltMdOK+DgOpixM3Kydvx40MFiyFmMQSnIcuqbTa+4ZBldiNA94QKV3 LGXSUqROiISY2yMS260GNJ2XtL+IvUWAem+cLLms/CteYoxC6nM6ksUB7vc2/NlEv3jXXSdSysp 2MYo88i8DspaocWrYparnFQf9E9t6Ogu5zBSke9Z31il/I06DjYbMwS7h/0fQ99ddnAWP0Nxl9e 5sj2pt0m6ynFeM0x17wz4vwP6JHUh9itL05pucPrXk7j3kbFE6dpwWzWuwO7oJwFFx6buxfcLOH qB2rvDB7jdiApUBaEy36NrEQ== X-Received: by 2002:a05:6214:1c41:b0:896:fa50:4c0c with SMTP id 6a1803df08f44-89a81f5282emr14261866d6.53.1773348811497; Thu, 12 Mar 2026 13:53:31 -0700 (PDT) Received: from localhost ([2603:7000:c00:3a00:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-89a65c0d327sm42509556d6.22.2026.03.12.13.53.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2026 13:53:30 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: David Hildenbrand , Shakeel Butt , Yosry Ahmed , Zi Yan , "Liam R. Howlett" , Usama Arif , Kiryl Shutsemau , Dave Chinner , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 1/7] mm: list_lru: lock_list_lru_of_memcg() cannot return NULL if !skip_empty Date: Thu, 12 Mar 2026 16:51:49 -0400 Message-ID: <20260312205321.638053-2-hannes@cmpxchg.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260312205321.638053-1-hannes@cmpxchg.org> References: <20260312205321.638053-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" skip_empty is only for the shrinker to abort and skip a list that's empty or whose cgroup is being deleted. For list additions and deletions, the cgroup hierarchy is walked upwards until a valid list_lru head is found, or it will fall back to the node list. Acquiring the lock won't fail. Remove the NULL checks in those callers. Signed-off-by: Johannes Weiner Reviewed-by: David Hildenbrand (Arm) --- mm/list_lru.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index 26463ae29c64..d96fd50fc9af 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -165,8 +165,6 @@ bool list_lru_add(struct list_lru *lru, struct list_hea= d *item, int nid, struct list_lru_one *l; =20 l =3D lock_list_lru_of_memcg(lru, nid, memcg, false, false); - if (!l) - return false; if (list_empty(item)) { list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ @@ -203,9 +201,8 @@ bool list_lru_del(struct list_lru *lru, struct list_hea= d *item, int nid, { struct list_lru_node *nlru =3D &lru->node[nid]; struct list_lru_one *l; + l =3D lock_list_lru_of_memcg(lru, nid, memcg, false, false); - if (!l) - return false; if (!list_empty(item)) { list_del_init(item); l->nr_items--; --=20 2.53.0 From nobody Tue Apr 7 14:38:27 2026 Received: from mail-qk1-f175.google.com (mail-qk1-f175.google.com [209.85.222.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8E956374E43 for ; Thu, 12 Mar 2026 20:53:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773348817; cv=none; b=AM+arMQmNFwTvr1q/ex2rFXttzMgVPtjxNzzQcJVX8xo/QJWwd+YKWM1tAjLzGdI1xeNGtRuXP4fyzn/dUBbhmuq1wMJwcNV8kV+OEMYs83a75nIU/F1t8IgPs5xGa7Rhc5ZpUvDsoX5nWHANLzltYMGoLhOQWGDL0znQCcOpL8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773348817; c=relaxed/simple; bh=qbsbyItfFL8uygu+Ulf4MLtwpF0DZ+jfdHZf6+XmqYY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=X/1ux08Lf7rc+OkSBb6PnXrnnzCGVZzUjl8SwG1cnVf0YfeYSI9gSMO70FlSx388QBmopnaXu+gGhrkPGNBxk7OoySEm2wu7W+7mAmYQ7LaLRf2H82LixtC36M+Aafkct3JMTofW7ooVDBJaHLiC8nQ7xUcutEMHIfV09e5GxOY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=E9FR12Q/; arc=none smtp.client-ip=209.85.222.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="E9FR12Q/" Received: by mail-qk1-f175.google.com with SMTP id af79cd13be357-8cd90401034so164016885a.0 for ; Thu, 12 Mar 2026 13:53:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1773348813; x=1773953613; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bKktsnvx+4HbrqJvE5JfNhKHypqzO+issASPe3WGHl8=; b=E9FR12Q/IdYtkXB9HKflM9kJccr68Z4+xe7oZY92kIjHjwXAYwE2/hem8lva5l6kMU V5k1WszgsP+kH7v2/q8Ki1N3vu5G9xzXnCWAvOrOfiYGpJlHhnn6X0IqARPV8WenxQ54 k1kXj3h+zol2Nc+Ihj7k8IMmq1jLWQuklEmg7RCiDhXwxBV3zEKbVq0k7NN1S5vHNpV4 Ca3KCP4JgpOxNVNvxipKOeUgydYXphIH7hYg9dbxCDfGsSfQzhD7552lc1UU1M2Rry1s xcKUGlJJz10gaCh4xn9OXrzA/oN4NfQRCZPFhFISy+o+aGbkjBRdtOFTsYRN1ufwwNRc P3qQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773348813; x=1773953613; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=bKktsnvx+4HbrqJvE5JfNhKHypqzO+issASPe3WGHl8=; b=H2qrdmve+UKnYpdWbVIFeBnzXuunrPVhcr7ytWHa9HNV7H2xCX3uuCxTxUa2gZZZ6B lC1Yj3LIVd2WEuslzQysq4vYx0RopSsElqiOv066bBqCCSOWbfLl3tB3v9E+jCB7yFgA PW6yDmykNJkBOOZKoiyElKx9F7SBTofmEYAiJfT9NyyWBQx+n1713mGhDRSrwXSa0ZjD 5OkklaLGYx+6bBNZHXPkQErPhOKaW9pf0dGBn/2wIIQLou5BARLnP9bXljzeHvp9FRy9 /eEQSNec8jBWwalmftUeQsLcjgcpyyQZCDFeHk9uP1SNDnubP+jWYDgxbUQhWPqqG1C5 Enig== X-Forwarded-Encrypted: i=1; AJvYcCXfFCmDgrvsAJlNaaBZhEzzK0oKo7pYzuw30rYSr4iv7b/9la/scB9sqxyRVNVgVRluAZlrRIsbo9gKDwE=@vger.kernel.org X-Gm-Message-State: AOJu0YxOHt9qgy5J9tAclCmC6MuhJaueefKGAby0cftEjGYtN7MrOcPP 8dTu8kweVN3mc1RSgDIgxwSQMoehLZaW4vexwlGSa04Kf2QKceCDfVt7s/+0yWPqsiM= X-Gm-Gg: ATEYQzxhhSI/QM5IuYFGD4tGlx4k1019EzjVSZ8/HGowEozM9aIDsHVz+lXX/e8WeW1 hm+ruhTKFEfhqBlxpK2wSmmB95f+IHZ8TceeVfmP+0O2qOVotdrjhNx78h+SHRKzkpOVp5+oYer HZlK5ncPZkVQRTNkehpb07d+CZfD2fNAAE2g1JxCpWvrMIXCCaPcY6mQEJg/dLq7B401OMDdyO4 rixPKC1iejxETmApjsPi7fOrercTTJ21UN3JRBYcIhn/6NT1NbzHFIT3XXSbMGFWCxuhcsg1kE/ 3oTRhAhYV3/8C+ecIQSzCYqhwPacuyZBFIgH3zRtWzDYlEtFVd+l58rIKBj38mSgaAsSiDL3HaQ YmjJIg/uJIGezC4IXR8vCeOTBtaBzUvZseWeC1ekK8eFYvj2KrMtDwsu/3eGQPRdlB0gbGifg6N pVtvtxJQHWkibKlW/bSTNXQQ== X-Received: by 2002:a05:620a:4510:b0:8cd:79aa:6931 with SMTP id af79cd13be357-8cdb5b6f77amr159581585a.58.1773348813306; Thu, 12 Mar 2026 13:53:33 -0700 (PDT) Received: from localhost ([2603:7000:c00:3a00:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id af79cd13be357-8cda2112f5bsm473183885a.32.2026.03.12.13.53.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2026 13:53:32 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: David Hildenbrand , Shakeel Butt , Yosry Ahmed , Zi Yan , "Liam R. Howlett" , Usama Arif , Kiryl Shutsemau , Dave Chinner , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 2/7] mm: list_lru: deduplicate unlock_list_lru() Date: Thu, 12 Mar 2026 16:51:50 -0400 Message-ID: <20260312205321.638053-3-hannes@cmpxchg.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260312205321.638053-1-hannes@cmpxchg.org> References: <20260312205321.638053-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The MEMCG and !MEMCG variants are the same. lock_list_lru() has the same pattern when bailing. Consolidate into a common implementation. Signed-off-by: Johannes Weiner Acked-by: Shakeel Butt Reviewed-by: David Hildenbrand (Arm) --- mm/list_lru.c | 29 +++++++++-------------------- 1 file changed, 9 insertions(+), 20 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index d96fd50fc9af..e873bc26a7ef 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -15,6 +15,14 @@ #include "slab.h" #include "internal.h" =20 +static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off) +{ + if (irq_off) + spin_unlock_irq(&l->lock); + else + spin_unlock(&l->lock); +} + #ifdef CONFIG_MEMCG static LIST_HEAD(memcg_list_lrus); static DEFINE_MUTEX(list_lrus_mutex); @@ -67,10 +75,7 @@ static inline bool lock_list_lru(struct list_lru_one *l,= bool irq) else spin_lock(&l->lock); if (unlikely(READ_ONCE(l->nr_items) =3D=3D LONG_MIN)) { - if (irq) - spin_unlock_irq(&l->lock); - else - spin_unlock(&l->lock); + unlock_list_lru(l, irq); return false; } return true; @@ -101,14 +106,6 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, = struct mem_cgroup *memcg, memcg =3D parent_mem_cgroup(memcg); goto again; } - -static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off) -{ - if (irq_off) - spin_unlock_irq(&l->lock); - else - spin_unlock(&l->lock); -} #else static void list_lru_register(struct list_lru *lru) { @@ -147,14 +144,6 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, = struct mem_cgroup *memcg, =20 return l; } - -static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off) -{ - if (irq_off) - spin_unlock_irq(&l->lock); - else - spin_unlock(&l->lock); -} #endif /* CONFIG_MEMCG */ =20 /* The caller must ensure the memcg lifetime. */ --=20 2.53.0 From nobody Tue Apr 7 14:38:27 2026 Received: from mail-qk1-f172.google.com (mail-qk1-f172.google.com [209.85.222.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AF9A43B38B6 for ; Thu, 12 Mar 2026 20:53:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773348819; cv=none; b=Sh+jvtifLFvf+gCPYT5Un7CztG54tEMbUGsQtwf8jjOytVV/qanH7HNmE/YLrzOjJDicaalrTcJiku8CzzDsMNM7OvzXnRnCiXXCNW7QeRv0NczlTqQ8sjj8V0icBu/Z/klvlHMsbroK8ABLAqgUX1LHsvqa2oSVKhfp1TUlgek= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773348819; c=relaxed/simple; bh=Ml3MGP27gKnwVk3o42TvyYhVQZlEVcnjQHGgFhpn3W4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=chB//TuvFpjBmIqq5yUj0Sq4IEESy2Svy1O/eSCsPZBIS1IcsGaQ+1FJ1L3zPZWG4vvqP+mlNF2MbuuBvAZrnXvUjrm87MJsAsXM5a+o00NSOEBeEErSvY4n+zM40YKyuBMQ1Y94okU0NbLAzxzuyZ0gy0dz94//Bt952vzYol0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=MsH4v6d/; arc=none smtp.client-ip=209.85.222.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="MsH4v6d/" Received: by mail-qk1-f172.google.com with SMTP id af79cd13be357-8cd77786e97so146165685a.3 for ; Thu, 12 Mar 2026 13:53:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1773348815; x=1773953615; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=eafNr9BwjJfwIWHhaZifojoDqJ10dT0nuYTLu1eoj30=; b=MsH4v6d/6ueTDZgjc7G+k0793yCy+xMyuegsgGXOF4CUaNfIWwlJYRryqZpmAknwN3 e2lTFghWM4REiT2748sRrrWCHeFVS/Ih8BzczXA4jd2hr5YgFR4Ta9V+EDAyWtyI7/9I F8bjEd9BYqkXM6gMZmhbI0cJedyCUw0JhtteetJ/cnCai5dpGeXTk28kgtujBJHjMklW cPbYBLbKd1n39kkLCnD3Rfiyw1fbryz+7gcZ0T7DBuiOYM/b/YGz/kKhuVCCnz+LIVN9 wOqTwwq9i5KAfRwpSiDqe+Naamk8uPCs503C+gCgrYNjnovDd0zwE8a1tAg41Fes7207 K0kA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773348815; x=1773953615; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=eafNr9BwjJfwIWHhaZifojoDqJ10dT0nuYTLu1eoj30=; b=QdzpELh3kY5qjVZVMBsWoH6HyvxJKwIrgkMxoJyrEr+46fTCkoxw5Jiag+iwYntI9b 5Vlu2J6BBOMDL791+Qq/3FeenDc8/X9CSPO6gZEIe3ujzFMfKSjyPg3e+xY7HRNdoEji G/MlX6dVdSczrGlF6HQtGbavFEwZCTOrwi3VLWjcSehSnz9VYerUxro6OiMqRb+IyMgR zWw4g88+XRx1oXBUNGLUWidUU9yfo2gMUY5H0kc2RjsWi3/8+5lfG5XD5BhsMgyGA9R5 M3MrXx1UpWPJM8SZBtNiCN0UU6BB957+Z9ACxZ65Cv0oHUuhJUtFruUrJQUKWk+QSufo IIzQ== X-Forwarded-Encrypted: i=1; AJvYcCX+TmTUMHFvsJpitSleK71n2IJBhD0wLZt7STWtvJusVtQnyvuh2SV9ybla0nEU8l+domCtUCXhsiHH/4M=@vger.kernel.org X-Gm-Message-State: AOJu0YwASMnvEHS7nE8Z9Pp7i9ThxmBSDWUb8FqUtLXK6WQgBCh3urBv 07M1+2UiIQC4Bu3RDobpaeoq1yUyscqsZxJmhyHjdrNq0rVX9/2MwPEHSad6M8dCjEA= X-Gm-Gg: ATEYQzwkZ2D9B/Lfd3zIjP64+ask/ngYyBNcRkwx8PH+kY1IFImsczEGjMWILkr1VOk oQ4u9ykjuUE4JjjaDaaRvCuVfHE7bsF5e3dowTyVfQ0DUgH8sD+rjfa4n2OiQib/qOvAVbGbNMb O8YkPi6aw0zF7+I0X/XjsrKJ2EcmH0lW2LsHnvCc6gGS2TyLzLEU4/FEEURHqfxr7egTAoW525m Yur+U0zeUHxYZEzzTqDqXYMKdqpWadXZTh7F6omkhEd77Y1yECxCd/xxNOnUzAwhRTnlvmDP/b5 cs5Rx/j/oe+9OPcjQCS2cmWSLFN/rtEIGUeqe/IOkIKzGdsf+iFbhq1S9woNtc7bImS22ox8slS E6/u1pQ4xG0M0Iadupcx4IhX74UFw8mFCIacISRnId42SGRWCF92TYWMW9HSkpwVTJYeR1EB6Cr XJpgfEfxZj8xzNMZOAWwSWPA== X-Received: by 2002:a05:620a:1a0d:b0:8ca:41af:32cb with SMTP id af79cd13be357-8cdb5bb05f4mr160267485a.77.1773348815504; Thu, 12 Mar 2026 13:53:35 -0700 (PDT) Received: from localhost ([2603:7000:c00:3a00:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-89a6e736043sm29875566d6.10.2026.03.12.13.53.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2026 13:53:34 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: David Hildenbrand , Shakeel Butt , Yosry Ahmed , Zi Yan , "Liam R. Howlett" , Usama Arif , Kiryl Shutsemau , Dave Chinner , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 3/7] mm: list_lru: move list dead check to lock_list_lru_of_memcg() Date: Thu, 12 Mar 2026 16:51:51 -0400 Message-ID: <20260312205321.638053-4-hannes@cmpxchg.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260312205321.638053-1-hannes@cmpxchg.org> References: <20260312205321.638053-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Only the MEMCG variant of lock_list_lru() needs to check if there is a race with cgroup deletion and list reparenting. Move the check to the caller, so that the next patch can unify the lock_list_lru() variants. Signed-off-by: Johannes Weiner Reviewed-by: David Hildenbrand (Arm) --- mm/list_lru.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index e873bc26a7ef..1a39ff490643 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -68,17 +68,12 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, = int idx) return &lru->node[nid].lru; } =20 -static inline bool lock_list_lru(struct list_lru_one *l, bool irq) +static inline void lock_list_lru(struct list_lru_one *l, bool irq) { if (irq) spin_lock_irq(&l->lock); else spin_lock(&l->lock); - if (unlikely(READ_ONCE(l->nr_items) =3D=3D LONG_MIN)) { - unlock_list_lru(l, irq); - return false; - } - return true; } =20 static inline struct list_lru_one * @@ -90,9 +85,13 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, st= ruct mem_cgroup *memcg, rcu_read_lock(); again: l =3D list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); - if (likely(l) && lock_list_lru(l, irq)) { - rcu_read_unlock(); - return l; + if (likely(l)) { + lock_list_lru(l, irq); + if (likely(READ_ONCE(l->nr_items) !=3D LONG_MIN)) { + rcu_read_unlock(); + return l; + } + unlock_list_lru(l, irq); } /* * Caller may simply bail out if raced with reparenting or --=20 2.53.0 From nobody Tue Apr 7 14:38:27 2026 Received: from mail-qv1-f47.google.com (mail-qv1-f47.google.com [209.85.219.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 412B43AC0FC for ; Thu, 12 Mar 2026 20:53:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773348821; cv=none; b=QKAksyyf4gZG3AQuxoykFyCFrtspxsoRw0gRv5m4b2qFd2jr9/NukRUhNjrO6MGstHXIFuFgHwpczEIgVsUEtW5PFonoXcaCdxbB7F2FWUmIpgW2tSdcZ1rrrVgIePKAj9e12xY3jg2zDj917EmKwYmWvxZBTg2Rf+wjMUpazdU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773348821; c=relaxed/simple; bh=lXGs8kmFC+AdsUro3ZawPlwIzcatYTm8zi/IbrDcSnc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=t7UsKgPQUiT5YcnIYDl5zs8qgVKLQtRCZmblC4QrxDqcCO7D6bWTYqG63rOUJd4Ks6HElkQb8NTae3Xhn+tz3+kH6p+VeZtvsBxaLROfaILMDKGlyACa3NesKY3bMJPgOLv17pQwONfWNP/50CouS56dv/B6gqj9qA99gBudS6I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=qszlxOle; arc=none smtp.client-ip=209.85.219.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="qszlxOle" Received: by mail-qv1-f47.google.com with SMTP id 6a1803df08f44-89a465bd7edso11375776d6.0 for ; Thu, 12 Mar 2026 13:53:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1773348817; x=1773953617; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kXGfWWKzNbxM79mVGdQBIJH3/Y1RdaXH5DLrm3GtnoY=; b=qszlxOleg9F8YFov+eLvmhdxBeDe7MgdN96lCJevnBOLnRW1x9UjjGCVEatI2hFo0S 3LJUhsXroXgf9DN9vSZBRRuxIQDOEFqwiXcEvq3HiYBmdf1DdrZecrss2c1yswvcXsB3 zKdLdut1K7MbwBI4J5G9HeXyJooZPOTdSN8805OvojhhQNItkSPHzphRBihKFkVQeG3g bNyxANDoWYqFR+cxfUgNokIHnkEnP4ONzhwY22CHNp+OFliWo5IIrrQYwaCh9KpTHhKv w9pRHFIM+l3XqC6RHWfRCh56wwdEsC6x/EU5b+jcC7Ka72GDHqVmKUkzSRazondaJ5UB Wi6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773348817; x=1773953617; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=kXGfWWKzNbxM79mVGdQBIJH3/Y1RdaXH5DLrm3GtnoY=; b=pu8j0jGkR4KWNl8Fnb5mOJ2jYcHK6JKwaSIV3rArlECwpdpRry85nzUAzFZ0DWF/Bc tmoubNVzxSuOM6Vupcyazfa42p4zHCMrCP/6JBDafLCrZmHAubG5PvmGAgNLrqKo4nLE 7Mi36hUPZKfvn9Ip0bEKGZczqVV1wQm8oPifvawmtxV+jJQslrLtNSJ3ZziqzYfOlBRC xdDiY6Sl7pRjsOfH6fTEi4WXL+YmBrAyMcZyKQHpx6Lx5lFaK2i9UYv3yM4jyRZKnQ20 JUwfvHbypTt8XwL+tok9rvAvrhLeDFgdPw13Hb7/Khphkf4uQW0lbt+j4tL4Bq6OCJy5 /48Q== X-Forwarded-Encrypted: i=1; AJvYcCUqdILzNE8ZYs24cnfr+Ho8wSc5xJazAvE4VcR4l+f27LjmxQHY26fOOPTpWO2PfWtme3iGzF3ADYFK/qs=@vger.kernel.org X-Gm-Message-State: AOJu0YwW/JeZbMW1/HOoF1Joh8ikN3ckk21hzpLwwNLhpHXLIt0eBHJt VGdDlw5ZgEkcpkUSw79/eC1X7IegWIDAaZbBkJAGPii3QF/l8WG+Wc7IFOCHdYTLvMw= X-Gm-Gg: ATEYQzzCWcgqWogwB/eNHzSwVL0DNnER+BSVLqOEdsyX8KYsLGmtjyLvCFZQLl2w2kL b7ZWDYqRPLSi+QGeShPDuwD1H+W18qMc0nIVXWOY49zrcnAJts3WJMyVIdesarkir/KV6yPgj3r i4a6+VJARqMnET9HgRGU7qU+IwJ5nlOskLCrj3RmfHcEa+h428wkKoPbCmOOqOZHVUj1prRFhHp L3hZg0HG3Wl1hIcKpAhifa4Jw+IvPCUkbqi1Eovt1bfy+S9bp+vLKUII4DgYdxj4p3cX3PT9U4P 4hvnPecJMvlujT00VOYQmgTwYpEDNjLiX35gR5dLnfki+8lemYXZc2mrvakg9NobihZgREClAML nNf6EFUGxtuzqOOcs3vpkpyJjqPQAtsCUHq8rP4gBt/EuVBfAazoOhxidTU05sobklmJHRgjCBq chGP76qAo+x9+neZWE+3nWvg== X-Received: by 2002:ad4:5c6f:0:b0:89a:6e62:1dd1 with SMTP id 6a1803df08f44-89a81fb92c8mr19060666d6.60.1773348817112; Thu, 12 Mar 2026 13:53:37 -0700 (PDT) Received: from localhost ([2603:7000:c00:3a00:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-89a65d17023sm42324226d6.49.2026.03.12.13.53.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2026 13:53:36 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: David Hildenbrand , Shakeel Butt , Yosry Ahmed , Zi Yan , "Liam R. Howlett" , Usama Arif , Kiryl Shutsemau , Dave Chinner , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 4/7] mm: list_lru: deduplicate lock_list_lru() Date: Thu, 12 Mar 2026 16:51:52 -0400 Message-ID: <20260312205321.638053-5-hannes@cmpxchg.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260312205321.638053-1-hannes@cmpxchg.org> References: <20260312205321.638053-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The MEMCG and !MEMCG paths have the same pattern. Share the code. Signed-off-by: Johannes Weiner Reviewed-by: David Hildenbrand (Arm) --- mm/list_lru.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index 1a39ff490643..4d74c2e9c2a5 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -15,6 +15,14 @@ #include "slab.h" #include "internal.h" =20 +static inline void lock_list_lru(struct list_lru_one *l, bool irq) +{ + if (irq) + spin_lock_irq(&l->lock); + else + spin_lock(&l->lock); +} + static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off) { if (irq_off) @@ -68,14 +76,6 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, i= nt idx) return &lru->node[nid].lru; } =20 -static inline void lock_list_lru(struct list_lru_one *l, bool irq) -{ - if (irq) - spin_lock_irq(&l->lock); - else - spin_lock(&l->lock); -} - static inline struct list_lru_one * lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *m= emcg, bool irq, bool skip_empty) @@ -136,10 +136,7 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, = struct mem_cgroup *memcg, { struct list_lru_one *l =3D &lru->node[nid].lru; =20 - if (irq) - spin_lock_irq(&l->lock); - else - spin_lock(&l->lock); + lock_list_lru(l, irq); =20 return l; } --=20 2.53.0 From nobody Tue Apr 7 14:38:27 2026 Received: from mail-qk1-f174.google.com (mail-qk1-f174.google.com [209.85.222.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 174EA3B0ADB for ; Thu, 12 Mar 2026 20:53:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773348823; cv=none; b=T5PhgXHLIQqw2hVp5VNfQkKP8UMYYb5CGKVFyZVh0oHCJ8WP8uwgQ53R0OjlCzJYqo+YlX3BtV4WFCIIBUo2Oko9h/ijqhclQlNvOrsBSjKHE59SaYPAdb3FHeRgMkMQbwRT81eN8yb4XI9wKGB1VBgD9/66MQkqsrAO74Yhmrs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773348823; c=relaxed/simple; bh=MVDI0Bqiurks4MrwaZ4b5LImWjwgxUmFmMP1z/OmQWE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Jg2Zqc5X09QtInKLhC8tNwsNTI/THnGrtHHjO5BpgUpALQ5t/WctztYBIVdJrQsSH/CCthcVb2ZlvygJ/y7j8s/Idou08ELluv6eUYp/Oywjz8UfgHo3uRNDOEGJTzyNSTNjkXVcKWKMMhJn4tJkX3suYxzPrGVxCxsNLAxjEBs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=Jg67ecby; arc=none smtp.client-ip=209.85.222.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="Jg67ecby" Received: by mail-qk1-f174.google.com with SMTP id af79cd13be357-8cbb6d5f780so134514485a.1 for ; Thu, 12 Mar 2026 13:53:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1773348819; x=1773953619; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MfCdMvkJxAeZG14cTtGmM/ZN7/Bzr7yHmjt9XpyWCt8=; b=Jg67ecbyJX//UzSjM3saOVg6FBhO7AcH3C7sYmfXH2HkcQuzyJZWniB9fpIdD6KIUZ 6/l+Eucj43bGIsmgCr+Ov6EmsFTSy//nTmZfzNTLEitrbnQSMoCk6Q+Xj3WclU10EYfW mcHBJdMKap9TeHdRVjPhNT2LynM+6hjwc962MOVh1IN7VN98QvhrwD3fOVAYREAJcJqG Dkj7oUEzz+e1lifvO9m7w+DVBYwVjaXfNJSUFVgDmhNqQO1asBdzyHv+b7cNB68mMH/z p3hvM3qc3S/NZzdieAxIYHLR1vsgwqv7uKEB/F+Jikz5QAVCVxWBbbwjFECMH2Xjwmfq zv9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773348819; x=1773953619; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=MfCdMvkJxAeZG14cTtGmM/ZN7/Bzr7yHmjt9XpyWCt8=; b=gmYpoEGhUrdX+uH7IslDAQTJoD1gknkoLmNiarWChtEAF63xEXMy6BN1mKXbP3pCvX FfjM+AvU/Q8cw/HUJZeGZDoOFXraLrCghQx34ltbVCeAcCYO5723GEsZy5FpG0IbMwAL fhWk9YQa/KYM/752BZey0MEqOebFAwG5ZZEHQ0WAkOs531T3yDVT2rECy4BX2xL7S2Wx u8nTm7n8oLRCMb5XIOTDGYc45Boq3Fv09+8lkadgPoKEV1+x9BscwnMG8xafSpNYAjBt AGvk3RwBEVZqnnJ2EtdmrZ05bKjrr5w6ytj64v1ToB9xdkAdM0wIPS7Dl3LVP5y37gOR XTSg== X-Forwarded-Encrypted: i=1; AJvYcCVOiB2feB7ddnloYX5qH0LsDHernpo3StN7ogu70MsuHNDI4kN+GYKHWOvVh4zecKm1VWWg+GqVqF2MgHI=@vger.kernel.org X-Gm-Message-State: AOJu0YyEp7LDx0xPjI9F3q3EXpsMeLjB05WI2mMJjyWyn/5AeSOpP25s krvgiyOVrbUz3dRcvTnHlQXNw/6KZ3LQjxdKtX5uC9CRHjc9b/6lE2O3JJegpoFjUVk= X-Gm-Gg: ATEYQzzWnH0mQQlbdHokWaw5i2EW1+Q2YV5xzp/JDmG7naOcDc7U+rAFQFUNsr1ZTtb bpTNmAXJlgL9lITKa2IESuOUDBaVdrLFoOn773NNIq7DiWrJzgayQZ8RywBR4dmfycGDSfZU/xj QroFOM5ZfXCVz9D7y7H0R48S+DzXXpHr1PgJTUqaerXasGRpg/xe7AmEZCTWqXjJgW0O+YJowj7 +OVaQOAU9ZhhPX8gU19DwpmoqDgwGgmMdnT5EdAC52vvm9W0CutcwZDHqVjcbLMg9cWxqehKgeU 3JmrCDR3Lg7K+ZN8p+Cv81cLZUHqYudIBC1R9YthGueeJL+VNyAufM0Kta2sjtQwfnQNc1z+RKO lH0MH5t9KLXIrtrJoZmFjtqmAfGBVQkGJIVfbjeOH7/G8aJaSTpD50K9mYXSBXQUqWja5uhXARJ H1Ia+q5jodIk9nL+SW4q4qYg== X-Received: by 2002:a05:620a:1724:b0:8cd:7f7d:b091 with SMTP id af79cd13be357-8cdb5bce7bcmr154603785a.71.1773348818838; Thu, 12 Mar 2026 13:53:38 -0700 (PDT) Received: from localhost ([2603:7000:c00:3a00:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id af79cd13be357-8cda1fbd120sm412873485a.7.2026.03.12.13.53.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2026 13:53:37 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: David Hildenbrand , Shakeel Butt , Yosry Ahmed , Zi Yan , "Liam R. Howlett" , Usama Arif , Kiryl Shutsemau , Dave Chinner , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 5/7] mm: list_lru: introduce caller locking for additions and deletions Date: Thu, 12 Mar 2026 16:51:53 -0400 Message-ID: <20260312205321.638053-6-hannes@cmpxchg.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260312205321.638053-1-hannes@cmpxchg.org> References: <20260312205321.638053-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Locking is currently internal to the list_lru API. However, a caller might want to keep auxiliary state synchronized with the LRU state. For example, the THP shrinker uses the lock of its custom LRU to keep PG_partially_mapped and vmstats consistent. To allow the THP shrinker to switch to list_lru, provide normal and irqsafe locking primitives as well as caller-locked variants of the addition and deletion functions. Signed-off-by: Johannes Weiner Reviewed-by: David Hildenbrand (Arm) --- include/linux/list_lru.h | 34 +++++++++++++ mm/list_lru.c | 104 +++++++++++++++++++++++++++------------ 2 files changed, 107 insertions(+), 31 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index fe739d35a864..4afc02deb44d 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -83,6 +83,40 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, struc= t list_lru *lru, gfp_t gfp); void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup = *parent); =20 +/** + * list_lru_lock: lock the sublist for the given node and memcg + * @lru: the lru pointer + * @nid: the node id of the sublist to lock. + * @memcg: the cgroup of the sublist to lock. + * + * Returns the locked list_lru_one sublist. The caller must call + * list_lru_unlock() when done. + * + * You must ensure that the memcg is not freed during this call (e.g., with + * rcu or by taking a css refcnt). + * + * Return: the locked list_lru_one, or NULL on failure + */ +struct list_lru_one *list_lru_lock(struct list_lru *lru, int nid, + struct mem_cgroup *memcg); + +/** + * list_lru_unlock: unlock a sublist locked by list_lru_lock() + * @l: the list_lru_one to unlock + */ +void list_lru_unlock(struct list_lru_one *l); + +struct list_lru_one *list_lru_lock_irqsave(struct list_lru *lru, int nid, + struct mem_cgroup *memcg, unsigned long *irq_flags); +void list_lru_unlock_irqrestore(struct list_lru_one *l, + unsigned long *irq_flags); + +/* Caller-locked variants, see list_lru_add() etc for documentation */ +bool __list_lru_add(struct list_lru *lru, struct list_lru_one *l, + struct list_head *item, int nid, struct mem_cgroup *memcg); +bool __list_lru_del(struct list_lru *lru, struct list_lru_one *l, + struct list_head *item, int nid); + /** * list_lru_add: add an element to the lru list's tail * @lru: the lru pointer diff --git a/mm/list_lru.c b/mm/list_lru.c index 4d74c2e9c2a5..779cb26cec84 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -15,17 +15,23 @@ #include "slab.h" #include "internal.h" =20 -static inline void lock_list_lru(struct list_lru_one *l, bool irq) +static inline void lock_list_lru(struct list_lru_one *l, bool irq, + unsigned long *irq_flags) { - if (irq) + if (irq_flags) + spin_lock_irqsave(&l->lock, *irq_flags); + else if (irq) spin_lock_irq(&l->lock); else spin_lock(&l->lock); } =20 -static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off) +static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off, + unsigned long *irq_flags) { - if (irq_off) + if (irq_flags) + spin_unlock_irqrestore(&l->lock, *irq_flags); + else if (irq_off) spin_unlock_irq(&l->lock); else spin_unlock(&l->lock); @@ -78,7 +84,7 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, in= t idx) =20 static inline struct list_lru_one * lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *m= emcg, - bool irq, bool skip_empty) + bool irq, unsigned long *irq_flags, bool skip_empty) { struct list_lru_one *l; =20 @@ -86,12 +92,12 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, s= truct mem_cgroup *memcg, again: l =3D list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); if (likely(l)) { - lock_list_lru(l, irq); + lock_list_lru(l, irq, irq_flags); if (likely(READ_ONCE(l->nr_items) !=3D LONG_MIN)) { rcu_read_unlock(); return l; } - unlock_list_lru(l, irq); + unlock_list_lru(l, irq, irq_flags); } /* * Caller may simply bail out if raced with reparenting or @@ -132,37 +138,79 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid= , int idx) =20 static inline struct list_lru_one * lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *m= emcg, - bool irq, bool skip_empty) + bool irq, unsigned long *irq_flags, bool skip_empty) { struct list_lru_one *l =3D &lru->node[nid].lru; =20 - lock_list_lru(l, irq); + lock_list_lru(l, irq, irq_flags); =20 return l; } #endif /* CONFIG_MEMCG */ =20 -/* The caller must ensure the memcg lifetime. */ -bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid, - struct mem_cgroup *memcg) +struct list_lru_one *list_lru_lock(struct list_lru *lru, int nid, + struct mem_cgroup *memcg) { - struct list_lru_node *nlru =3D &lru->node[nid]; - struct list_lru_one *l; + return lock_list_lru_of_memcg(lru, nid, memcg, false, NULL, false); +} + +void list_lru_unlock(struct list_lru_one *l) +{ + unlock_list_lru(l, false, NULL); +} + +struct list_lru_one *list_lru_lock_irqsave(struct list_lru *lru, int nid, + struct mem_cgroup *memcg, + unsigned long *flags) +{ + return lock_list_lru_of_memcg(lru, nid, memcg, true, flags, false); +} + +void list_lru_unlock_irqrestore(struct list_lru_one *l, unsigned long *fla= gs) +{ + unlock_list_lru(l, true, flags); +} =20 - l =3D lock_list_lru_of_memcg(lru, nid, memcg, false, false); +bool __list_lru_add(struct list_lru *lru, struct list_lru_one *l, + struct list_head *item, int nid, + struct mem_cgroup *memcg) +{ if (list_empty(item)) { list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ if (!l->nr_items++) set_shrinker_bit(memcg, nid, lru_shrinker_id(lru)); - unlock_list_lru(l, false); - atomic_long_inc(&nlru->nr_items); + atomic_long_inc(&lru->node[nid].nr_items); + return true; + } + return false; +} + +bool __list_lru_del(struct list_lru *lru, struct list_lru_one *l, + struct list_head *item, int nid) +{ + if (!list_empty(item)) { + list_del_init(item); + l->nr_items--; + atomic_long_dec(&lru->node[nid].nr_items); return true; } - unlock_list_lru(l, false); return false; } =20 +/* The caller must ensure the memcg lifetime. */ +bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid, + struct mem_cgroup *memcg) +{ + struct list_lru_one *l; + bool ret; + + l =3D list_lru_lock(lru, nid, memcg); + ret =3D __list_lru_add(lru, l, item, nid, memcg); + list_lru_unlock(l); + return ret; +} + bool list_lru_add_obj(struct list_lru *lru, struct list_head *item) { bool ret; @@ -184,19 +232,13 @@ EXPORT_SYMBOL_GPL(list_lru_add_obj); bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid, struct mem_cgroup *memcg) { - struct list_lru_node *nlru =3D &lru->node[nid]; struct list_lru_one *l; + bool ret; =20 - l =3D lock_list_lru_of_memcg(lru, nid, memcg, false, false); - if (!list_empty(item)) { - list_del_init(item); - l->nr_items--; - unlock_list_lru(l, false); - atomic_long_dec(&nlru->nr_items); - return true; - } - unlock_list_lru(l, false); - return false; + l =3D list_lru_lock(lru, nid, memcg); + ret =3D __list_lru_del(lru, l, item, nid); + list_lru_unlock(l); + return ret; } =20 bool list_lru_del_obj(struct list_lru *lru, struct list_head *item) @@ -269,7 +311,7 @@ __list_lru_walk_one(struct list_lru *lru, int nid, stru= ct mem_cgroup *memcg, unsigned long isolated =3D 0; =20 restart: - l =3D lock_list_lru_of_memcg(lru, nid, memcg, irq_off, true); + l =3D lock_list_lru_of_memcg(lru, nid, memcg, irq_off, NULL, true); if (!l) return isolated; list_for_each_safe(item, n, &l->list) { @@ -310,7 +352,7 @@ __list_lru_walk_one(struct list_lru *lru, int nid, stru= ct mem_cgroup *memcg, BUG(); } } - unlock_list_lru(l, irq_off); + unlock_list_lru(l, irq_off, NULL); out: return isolated; } --=20 2.53.0 From nobody Tue Apr 7 14:38:27 2026 Received: from mail-qk1-f169.google.com (mail-qk1-f169.google.com [209.85.222.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 64B043B6C07 for ; Thu, 12 Mar 2026 20:53:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773348825; cv=none; b=nADBFT+NjnucMV5m0DNQLk/XCr+NSH735jlXCsXiZiHVLVKHmEqJPmVheHy4DRP+KTrfG9tzFmSvLEnl6y+ljqziIN7JRGOYK6AHtbTwCJHMzE2jQsZrZjFdg4FsStHR8bUbprgG7JwcppZ5rDu9jlf3/Is+Zkksuto4PXJ+fjc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773348825; c=relaxed/simple; bh=r6j7CBaD6bfWmHnyhYFjVx4fURy7OqHj6oW9zFCS6k4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=l2AxB/48/lpiYBxw5EJBB/BaRezbXY87B5rIphJ10p2URPu1cUoYd3Cbjq4vEC5AXuFyijDedAvLdRH2b7SxxpmQELk6/JiB8hsk6XWxR4gO5FNlI1rYZm+DuTvTNYZeUT8mmlfE46qDVZsI/DdoEIFIKfAQXg2IU8QEXamlCjM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=N5GS6orp; arc=none smtp.client-ip=209.85.222.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="N5GS6orp" Received: by mail-qk1-f169.google.com with SMTP id af79cd13be357-8cd8c6e365eso144347685a.0 for ; Thu, 12 Mar 2026 13:53:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1773348821; x=1773953621; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=H+fNWmGr+tcL8ctiy45XUsI2tWZ66Zdo3aVc9uo+wjs=; b=N5GS6orpHZ4POXe/aNuIKE5UlIZWNCl0qcf5DWrGo1yYSmBiqpIcbD62Etca7qLKHD E1jiL8OMVdzT2UDrY/In1eHok0jaJH7v9Jd8MvqmPvL84gyslF7BKGPYPYC+Nxowx0Yb l3HOcUAuz8+byheMtrlDvSU+Bc5NP6G+mfuloaLIvFzNPK77ALwjxEpMn121z07WGQeK jZGDuSmDxQ8XEr4G4RyS/uTiiaHfnVce4MBXD4yEw31IeYdBsHRzCw/3DeWqEZ8TuEsb mAf6iRAgs9faGVCphAz9hWZn0/LOHXG8r5N4kstEI70pgD8uH/8ON20H/+DOAOAJBe1r lR/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773348821; x=1773953621; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=H+fNWmGr+tcL8ctiy45XUsI2tWZ66Zdo3aVc9uo+wjs=; b=KqmvOoyeh7loGgGvU8Es/4P/gyiBSRUQsFkr5pTJ1+glMCu2c7OCWzzgUq5aiKN4Xu XnOwPfCTu6Ql2CRW+ivMI55taNE+qGGpbdkXrbCOTSBBYNo7/Qe0C2RJpU9Vwvsp9krT SfPNCsXsE/2lwi+csi/D9yjZTmN8Bczi4ERQYbEh0OhigzwfZPeXU4qnTqF0uS6/+JNb PhiBH7ZtIuz3qNaH4o+aZsiXPmlwfQmXIyRjrMZzBy4/TszWHve7I9NxxNcIqwR5XmnE NUItajmoo5xobpUq16SwZ/dYAwHC/5Z07bwL57fxH7VdTUyAlnDY2o2wsAPLad2+JDp3 B6aw== X-Forwarded-Encrypted: i=1; AJvYcCW3MHVHRTaxGhFKjdRHvtQ11ihx54UzKu11K8zm4+VtaOErg5rmz7SDXxFhyqZJlrcrz6h41Gr3K8IEOFw=@vger.kernel.org X-Gm-Message-State: AOJu0Yxy+pQx6vlqk9F+o3zS/KIH6Oe31BVFAdJqWK/cDBK+HjNYJN0k ipMToaDYzl8on6tEcSu1fBuVFsZZooA23BZ5vTE8wV47VMfza2KmIAvTjqWuUlESInk= X-Gm-Gg: ATEYQzxI19SJKQisEOp58QN/4w3weZ+cuy+m3NgGEf9Zx+gQIKGVj3Drcoj0mt0ghNk sou/W3keOGAwgR5Vh2WGXXJRVd0tTix7u24YP+m56q21NO3FkSvja/LZUmE6L9Kyv4DGgsFgXnt GVWNvMeO4R4dzC6J4KzaHtOJrePKwgrzDCBYg4V1Y3/DYTHAzQTo8KeFUgqh0cwGavCqH+HS8dq zRu4hmHGdygnDVVvL/5dX6a8tmKCwp5iAgZKACwZE9YccfjQHNiyclhRvvD/nNKRryJDhV/1fEF NP/g2MCt7iSL6CAw4swHVZGXi/MtNxjSuJ0VVUrcCENOk57Cwemlt5H40dqZKhUoU+d0BLlkUzV UdSCQRIUV/Y+6z7irXBrb9oXDeTId4h7Yqg2OtCtJarcplhHPHokT0atAJX/7DYdxEA1EAAFwua EnLPZWfQafT6mzmVSNclmINQ== X-Received: by 2002:a05:620a:40d3:b0:8ca:3d7c:e767 with SMTP id af79cd13be357-8cdb5b5f305mr157157385a.52.1773348821047; Thu, 12 Mar 2026 13:53:41 -0700 (PDT) Received: from localhost ([2603:7000:c00:3a00:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id af79cd13be357-8cda1fbcc5fsm465928085a.4.2026.03.12.13.53.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2026 13:53:40 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: David Hildenbrand , Shakeel Butt , Yosry Ahmed , Zi Yan , "Liam R. Howlett" , Usama Arif , Kiryl Shutsemau , Dave Chinner , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 6/7] mm: list_lru: introduce memcg_list_lru_alloc_folio() Date: Thu, 12 Mar 2026 16:51:54 -0400 Message-ID: <20260312205321.638053-7-hannes@cmpxchg.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260312205321.638053-1-hannes@cmpxchg.org> References: <20260312205321.638053-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" memcg_list_lru_alloc() is called every time an object that may end up on the list_lru is created. It needs to quickly check if the list_lru heads for the memcg already exist, and allocate them when they don't. Doing this with folio objects is tricky: folio_memcg() is not stable and requires either RCU protection or pinning the cgroup. But it's desirable to make the existence check lightweight under RCU, and only pin the memcg when we need to allocate list_lru heads and may block. In preparation for switching the THP shrinker to list_lru, add a helper function for allocating list_lru heads coming from a folio. Signed-off-by: Johannes Weiner Reviewed-by: David Hildenbrand (Arm) --- include/linux/list_lru.h | 12 ++++++++++++ mm/list_lru.c | 39 ++++++++++++++++++++++++++++++++++----- 2 files changed, 46 insertions(+), 5 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index 4afc02deb44d..df6bd3c64b06 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -81,6 +81,18 @@ static inline int list_lru_init_memcg_key(struct list_lr= u *lru, struct shrinker =20 int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, gfp_t gfp); + +#ifdef CONFIG_MEMCG +int memcg_list_lru_alloc_folio(struct folio *folio, struct list_lru *lru, + gfp_t gfp); +#else +static inline int memcg_list_lru_alloc_folio(struct folio *folio, + struct list_lru *lru, gfp_t gfp) +{ + return 0; +} +#endif + void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup = *parent); =20 /** diff --git a/mm/list_lru.c b/mm/list_lru.c index 779cb26cec84..562b2b1f8c41 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -534,17 +534,14 @@ static inline bool memcg_list_lru_allocated(struct me= m_cgroup *memcg, return idx < 0 || xa_load(&lru->xa, idx); } =20 -int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, - gfp_t gfp) +static int __memcg_list_lru_alloc(struct mem_cgroup *memcg, + struct list_lru *lru, gfp_t gfp) { unsigned long flags; struct list_lru_memcg *mlru =3D NULL; struct mem_cgroup *pos, *parent; XA_STATE(xas, &lru->xa, 0); =20 - if (!list_lru_memcg_aware(lru) || memcg_list_lru_allocated(memcg, lru)) - return 0; - gfp &=3D GFP_RECLAIM_MASK; /* * Because the list_lru can be reparented to the parent cgroup's @@ -585,6 +582,38 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, str= uct list_lru *lru, =20 return xas_error(&xas); } + +int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, + gfp_t gfp) +{ + if (!list_lru_memcg_aware(lru) || memcg_list_lru_allocated(memcg, lru)) + return 0; + return __memcg_list_lru_alloc(memcg, lru, gfp); +} + +int memcg_list_lru_alloc_folio(struct folio *folio, struct list_lru *lru, + gfp_t gfp) +{ + struct mem_cgroup *memcg; + int res; + + if (!list_lru_memcg_aware(lru)) + return 0; + + /* Fast path when list_lru heads already exist */ + rcu_read_lock(); + memcg =3D folio_memcg(folio); + res =3D memcg_list_lru_allocated(memcg, lru); + rcu_read_unlock(); + if (likely(res)) + return 0; + + /* Allocation may block, pin the memcg */ + memcg =3D get_mem_cgroup_from_folio(folio); + res =3D __memcg_list_lru_alloc(memcg, lru, gfp); + mem_cgroup_put(memcg); + return res; +} #else static inline void memcg_init_list_lru(struct list_lru *lru, bool memcg_aw= are) { --=20 2.53.0 From nobody Tue Apr 7 14:38:27 2026 Received: from mail-qk1-f178.google.com (mail-qk1-f178.google.com [209.85.222.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3EE773B8949 for ; Thu, 12 Mar 2026 20:53:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773348828; cv=none; b=dcxSJjgp1XyQzdPcXuK+tdUTJdOJsFexK+cqzQV33gWUUNMa/J+HbSoWxiQvddFFb5MQvT4qlzZchd0liZTpCMiydjc+reSj57BFmWIPFWEM4UBltouhMRcer3DtIM8bFKPcT3yql9TnZZ2kgXITFTGx/V40qn7wStn4/R/6X2k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1773348828; c=relaxed/simple; bh=MQGaM5CV2ObaJ4pw7IH3WbEoXvQIWj5TUBhbfs7ntBE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tcc+OhHcgqOpd0Lif/rjTH4D9FAwj77mhsHCV14CqN/Da3YLbiw1Tc3KM6LMGSiy9oNgDBXlJpMj2QMqUWQtkTP6gt6mn4ByC3QWVl/wdtXrH+omK/RvCHsm/Fu2hic7hSLPodHIhc6ioT+BlYfKMar/xCoAMzW5uz6IQeBdF94= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=HYu8wy6X; arc=none smtp.client-ip=209.85.222.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="HYu8wy6X" Received: by mail-qk1-f178.google.com with SMTP id af79cd13be357-8cd7aab92dfso181488785a.0 for ; Thu, 12 Mar 2026 13:53:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1773348823; x=1773953623; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=meNuwYi96fVzujigXO/A5TlsI0kRYb25p0jCYCvGzAg=; b=HYu8wy6XajOrbxfl6G16JHWV9YyvxswKgqWhetQV8WerxB8gJJoK7KKDYquDnNyage G1sT0Im7BOfz0nH2k7rlUH+3J5T4qhunjgPR/aRMkdxjbnHTCGsczMCq2UHD41Sfs+AG Tc27Xy5S7fgfl+gyL6vkH6bU6XD38uS2yPZFKYR7KPFCGfPpgcOvynW0fzfCScPLWKvv s7VHpi2f+eRZR1nk/xJ4j39jEAKOG8U8HwgKTeLTo/7Iv1uKPXvIAgA0yvhgNpmpCFDz RqhWw/pqUT3P2mHx462hWt8poMyQaxKj0p62n7K728DHRIv6FVcwP71UEs+ZYK0fsjxn ELsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1773348823; x=1773953623; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=meNuwYi96fVzujigXO/A5TlsI0kRYb25p0jCYCvGzAg=; b=rtrLe1buwxSFVndXv+6nDuJJRKz/xZzhzwMltH9vg+TOzvJyfM0JpyfypPuf8DdKfs hj432UPu9ueajVQF1Pb+O7X2Bsco5EDPmPizqYnk9qXssOIlF6DJmBglkoxbX1CcDfZ9 YwZTdbvQL/RXa/xbOMLNybsqr65bN12kSpf7lMBtMM3Onizs69OWfJgoiyUXWxldUg3F qD9Po+sErSQhIhx55klt/Kd+mvL4VYrjLNaX65SCuPCOvdUHps1O+LdxYM9hi1qB2JEY PfZfu2Iv43Jv/sb6uXux7H682dZwqYbECIAijD0xAOCzGCZi25yr6URlo6ngRbgOO/RY +9Uw== X-Forwarded-Encrypted: i=1; AJvYcCWX+aGbzLXZv2trK4K/VGXAjdWX9RFNifTytGBFeM6z6FwRoceiWJhyHfwgoeejgxPLM5Xr5tfmBSHD6jw=@vger.kernel.org X-Gm-Message-State: AOJu0YwMv2z+D/i0JHUZ1AJHy0u35fbwSgBD4I2S1qd1riFWjp15vqRR 7zRi1+u+3Y70AuOVgMmKUAgV0Bz4OTX5fPTSoBjbipuf+wnTMs78rj2k7s4swt8Avio/yf4Avhc n8iJ6 X-Gm-Gg: ATEYQzwqlvldgTs3uaflE7MqvPQZDrBoU2+VLgr+TInIHSdzt6ALzUuL2QXI5U/PXdW MHafVwp269YN5XCSpIF3yweaLMzirGmyJeCMdgR5yX2jSGlVNnZ9RAnw7SEUneByA0nFX2J2pxO 67XUd/pEU/wQyFyW5JkOxqqoFDTfaa9NBCOwNKnixyV7eLaRNBK3T5eDHkYJSJ4X0eFqh3EqX+a buOdMk6GkitC1UeAUUm1QB5d0NomQUmn7wyGC3wosY6bIjwdYgkXzg0am6uBbwQ6usQV6q0kvq3 RPcx13FY46D7Yt3nvIpKHAVe2fw33d5AoBGNW70hF6GUY9fd1FNWG77VtI2K+3qIxhtK/MRq3ZP 4wPV32kg2aCiIh5Djapt/0z0iuVLTi5M1GKwXz6gACEJewDkzwKmz3XRI+1grdZXzyhhRCsa3cq ifqKmmhloNiq47vHDITly2WQ== X-Received: by 2002:a05:620a:4694:b0:8cd:8987:41b3 with SMTP id af79cd13be357-8cdaa7a33c8mr614912485a.11.1773348822904; Thu, 12 Mar 2026 13:53:42 -0700 (PDT) Received: from localhost ([2603:7000:c00:3a00:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id af79cd13be357-8cda21100b8sm417782085a.29.2026.03.12.13.53.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 12 Mar 2026 13:53:42 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: David Hildenbrand , Shakeel Butt , Yosry Ahmed , Zi Yan , "Liam R. Howlett" , Usama Arif , Kiryl Shutsemau , Dave Chinner , Roman Gushchin , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 7/7] mm: switch deferred split shrinker to list_lru Date: Thu, 12 Mar 2026 16:51:55 -0400 Message-ID: <20260312205321.638053-8-hannes@cmpxchg.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260312205321.638053-1-hannes@cmpxchg.org> References: <20260312205321.638053-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The deferred split queue handles cgroups in a suboptimal fashion. The queue is per-NUMA node or per-cgroup, not the intersection. That means on a cgrouped system, a node-restricted allocation entering reclaim can end up splitting large pages on other nodes: alloc/unmap deferred_split_folio() list_add_tail(memcg->split_queue) set_shrinker_bit(memcg, node, deferred_shrinker_id) for_each_zone_zonelist_nodemask(restricted_nodes) mem_cgroup_iter() shrink_slab(node, memcg) shrink_slab_memcg(node, memcg) if test_shrinker_bit(memcg, node, deferred_shrinker_id) deferred_split_scan() walks memcg->split_queue The shrinker bit adds an imperfect guard rail. As soon as the cgroup has a single large page on the node of interest, all large pages owned by that memcg, including those on other nodes, will be split. list_lru properly sets up per-node, per-cgroup lists. As a bonus, it streamlines a lot of the list operations and reclaim walks. It's used widely by other major shrinkers already. Convert the deferred split queue as well. The list_lru per-memcg heads are instantiated on demand when the first object of interest is allocated for a cgroup, by calling memcg_list_lru_alloc_folio(). Add calls to where splittable pages are created: anon faults, swapin faults, khugepaged collapse. These calls create all possible node heads for the cgroup at once, so the migration code (between nodes) doesn't need any special care. Signed-off-by: Johannes Weiner --- include/linux/huge_mm.h | 6 +- include/linux/memcontrol.h | 4 - include/linux/mmzone.h | 12 -- mm/huge_memory.c | 330 +++++++++++-------------------------- mm/internal.h | 2 +- mm/khugepaged.c | 7 + mm/memcontrol.c | 12 +- mm/memory.c | 52 +++--- mm/mm_init.c | 15 -- 9 files changed, 140 insertions(+), 300 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index a4d9f964dfde..2d0d0c797dd8 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -414,10 +414,9 @@ static inline int split_huge_page(struct page *page) { return split_huge_page_to_list_to_order(page, NULL, 0); } + +extern struct list_lru deferred_split_lru; void deferred_split_folio(struct folio *folio, bool partially_mapped); -#ifdef CONFIG_MEMCG -void reparent_deferred_split_queue(struct mem_cgroup *memcg); -#endif =20 void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, unsigned long address, bool freeze); @@ -650,7 +649,6 @@ static inline int try_folio_split_to_order(struct folio= *folio, } =20 static inline void deferred_split_folio(struct folio *folio, bool partiall= y_mapped) {} -static inline void reparent_deferred_split_queue(struct mem_cgroup *memcg)= {} #define split_huge_pmd(__vma, __pmd, __address) \ do { } while (0) =20 diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 086158969529..0782c72a1997 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -277,10 +277,6 @@ struct mem_cgroup { struct memcg_cgwb_frn cgwb_frn[MEMCG_CGWB_FRN_CNT]; #endif =20 -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - struct deferred_split deferred_split_queue; -#endif - #ifdef CONFIG_LRU_GEN_WALKS_MMU /* per-memcg mm_struct list */ struct lru_gen_mm_list mm_list; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 7bd0134c241c..232b7a71fd69 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1429,14 +1429,6 @@ struct zonelist { */ extern struct page *mem_map; =20 -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -struct deferred_split { - spinlock_t split_queue_lock; - struct list_head split_queue; - unsigned long split_queue_len; -}; -#endif - #ifdef CONFIG_MEMORY_FAILURE /* * Per NUMA node memory failure handling statistics. @@ -1562,10 +1554,6 @@ typedef struct pglist_data { unsigned long first_deferred_pfn; #endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ =20 -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - struct deferred_split deferred_split_queue; -#endif - #ifdef CONFIG_NUMA_BALANCING /* start time in ms of current promote rate limit period */ unsigned int nbp_rl_start; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 7d0a64033b18..ed9b98e2e166 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -67,6 +68,7 @@ unsigned long transparent_hugepage_flags __read_mostly = =3D (1<count_objects =3D deferred_split_count; deferred_split_shrinker->scan_objects =3D deferred_split_scan; shrinker_register(deferred_split_shrinker); @@ -886,6 +893,7 @@ static int __init thp_shrinker_init(void) =20 huge_zero_folio_shrinker =3D shrinker_alloc(0, "thp-zero"); if (!huge_zero_folio_shrinker) { + list_lru_destroy(&deferred_split_lru); shrinker_free(deferred_split_shrinker); return -ENOMEM; } @@ -900,6 +908,7 @@ static int __init thp_shrinker_init(void) static void __init thp_shrinker_exit(void) { shrinker_free(huge_zero_folio_shrinker); + list_lru_destroy(&deferred_split_lru); shrinker_free(deferred_split_shrinker); } =20 @@ -1080,119 +1089,6 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_s= truct *vma) return pmd; } =20 -static struct deferred_split *split_queue_node(int nid) -{ - struct pglist_data *pgdata =3D NODE_DATA(nid); - - return &pgdata->deferred_split_queue; -} - -#ifdef CONFIG_MEMCG -static inline -struct mem_cgroup *folio_split_queue_memcg(struct folio *folio, - struct deferred_split *queue) -{ - if (mem_cgroup_disabled()) - return NULL; - if (split_queue_node(folio_nid(folio)) =3D=3D queue) - return NULL; - return container_of(queue, struct mem_cgroup, deferred_split_queue); -} - -static struct deferred_split *memcg_split_queue(int nid, struct mem_cgroup= *memcg) -{ - return memcg ? &memcg->deferred_split_queue : split_queue_node(nid); -} -#else -static inline -struct mem_cgroup *folio_split_queue_memcg(struct folio *folio, - struct deferred_split *queue) -{ - return NULL; -} - -static struct deferred_split *memcg_split_queue(int nid, struct mem_cgroup= *memcg) -{ - return split_queue_node(nid); -} -#endif - -static struct deferred_split *split_queue_lock(int nid, struct mem_cgroup = *memcg) -{ - struct deferred_split *queue; - -retry: - queue =3D memcg_split_queue(nid, memcg); - spin_lock(&queue->split_queue_lock); - /* - * There is a period between setting memcg to dying and reparenting - * deferred split queue, and during this period the THPs in the deferred - * split queue will be hidden from the shrinker side. - */ - if (unlikely(memcg_is_dying(memcg))) { - spin_unlock(&queue->split_queue_lock); - memcg =3D parent_mem_cgroup(memcg); - goto retry; - } - - return queue; -} - -static struct deferred_split * -split_queue_lock_irqsave(int nid, struct mem_cgroup *memcg, unsigned long = *flags) -{ - struct deferred_split *queue; - -retry: - queue =3D memcg_split_queue(nid, memcg); - spin_lock_irqsave(&queue->split_queue_lock, *flags); - if (unlikely(memcg_is_dying(memcg))) { - spin_unlock_irqrestore(&queue->split_queue_lock, *flags); - memcg =3D parent_mem_cgroup(memcg); - goto retry; - } - - return queue; -} - -static struct deferred_split *folio_split_queue_lock(struct folio *folio) -{ - struct deferred_split *queue; - - rcu_read_lock(); - queue =3D split_queue_lock(folio_nid(folio), folio_memcg(folio)); - /* - * The memcg destruction path is acquiring the split queue lock for - * reparenting. Once you have it locked, it's safe to drop the rcu lock. - */ - rcu_read_unlock(); - - return queue; -} - -static struct deferred_split * -folio_split_queue_lock_irqsave(struct folio *folio, unsigned long *flags) -{ - struct deferred_split *queue; - - rcu_read_lock(); - queue =3D split_queue_lock_irqsave(folio_nid(folio), folio_memcg(folio), = flags); - rcu_read_unlock(); - - return queue; -} - -static inline void split_queue_unlock(struct deferred_split *queue) -{ - spin_unlock(&queue->split_queue_lock); -} - -static inline void split_queue_unlock_irqrestore(struct deferred_split *qu= eue, - unsigned long flags) -{ - spin_unlock_irqrestore(&queue->split_queue_lock, flags); -} - static inline bool is_transparent_hugepage(const struct folio *folio) { if (!folio_test_large(folio)) @@ -1293,6 +1189,14 @@ static struct folio *vma_alloc_anon_folio_pmd(struct= vm_area_struct *vma, count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); return NULL; } + + if (memcg_list_lru_alloc_folio(folio, &deferred_split_lru, gfp)) { + folio_put(folio); + count_vm_event(THP_FAULT_FALLBACK); + count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK); + return NULL; + } + folio_throttle_swaprate(folio, gfp); =20 /* @@ -3802,33 +3706,28 @@ static int __folio_freeze_and_split_unmapped(struct= folio *folio, unsigned int n struct folio *new_folio, *next; int old_order =3D folio_order(folio); int ret =3D 0; - struct deferred_split *ds_queue; + struct list_lru_one *l; =20 VM_WARN_ON_ONCE(!mapping && end); /* Prevent deferred_split_scan() touching ->_refcount */ - ds_queue =3D folio_split_queue_lock(folio); + rcu_read_lock(); + l =3D list_lru_lock(&deferred_split_lru, folio_nid(folio), folio_memcg(fo= lio)); if (folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1)) { struct swap_cluster_info *ci =3D NULL; struct lruvec *lruvec; =20 if (old_order > 1) { - if (!list_empty(&folio->_deferred_list)) { - ds_queue->split_queue_len--; - /* - * Reinitialize page_deferred_list after removing the - * page from the split_queue, otherwise a subsequent - * split will see list corruption when checking the - * page_deferred_list. - */ - list_del_init(&folio->_deferred_list); - } + __list_lru_del(&deferred_split_lru, l, + &folio->_deferred_list, folio_nid(folio)); if (folio_test_partially_mapped(folio)) { folio_clear_partially_mapped(folio); mod_mthp_stat(old_order, MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1); } } - split_queue_unlock(ds_queue); + list_lru_unlock(l); + rcu_read_unlock(); + if (mapping) { int nr =3D folio_nr_pages(folio); =20 @@ -3929,7 +3828,8 @@ static int __folio_freeze_and_split_unmapped(struct f= olio *folio, unsigned int n if (ci) swap_cluster_unlock(ci); } else { - split_queue_unlock(ds_queue); + list_lru_unlock(l); + rcu_read_unlock(); return -EAGAIN; } =20 @@ -4296,33 +4196,35 @@ int split_folio_to_list(struct folio *folio, struct= list_head *list) * queueing THP splits, and that list is (racily observed to be) non-empty. * * It is unsafe to call folio_unqueue_deferred_split() until folio refcoun= t is - * zero: because even when split_queue_lock is held, a non-empty _deferred= _list - * might be in use on deferred_split_scan()'s unlocked on-stack list. + * zero: because even when the list_lru lock is held, a non-empty + * _deferred_list might be in use on deferred_split_scan()'s unlocked + * on-stack list. * - * If memory cgroups are enabled, split_queue_lock is in the mem_cgroup: i= t is - * therefore important to unqueue deferred split before changing folio mem= cg. + * The list_lru sublist is determined by folio's memcg: it is therefore + * important to unqueue deferred split before changing folio memcg. */ bool __folio_unqueue_deferred_split(struct folio *folio) { - struct deferred_split *ds_queue; + struct list_lru_one *l; + int nid =3D folio_nid(folio); unsigned long flags; bool unqueued =3D false; =20 WARN_ON_ONCE(folio_ref_count(folio)); WARN_ON_ONCE(!mem_cgroup_disabled() && !folio_memcg_charged(folio)); =20 - ds_queue =3D folio_split_queue_lock_irqsave(folio, &flags); - if (!list_empty(&folio->_deferred_list)) { - ds_queue->split_queue_len--; + rcu_read_lock(); + l =3D list_lru_lock_irqsave(&deferred_split_lru, nid, folio_memcg(folio),= &flags); + if (__list_lru_del(&deferred_split_lru, l, &folio->_deferred_list, nid)) { if (folio_test_partially_mapped(folio)) { folio_clear_partially_mapped(folio); mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1); } - list_del_init(&folio->_deferred_list); unqueued =3D true; } - split_queue_unlock_irqrestore(ds_queue, flags); + list_lru_unlock_irqrestore(l, &flags); + rcu_read_unlock(); =20 return unqueued; /* useful for debug warnings */ } @@ -4330,7 +4232,9 @@ bool __folio_unqueue_deferred_split(struct folio *fol= io) /* partially_mapped=3Dfalse won't clear PG_partially_mapped folio flag */ void deferred_split_folio(struct folio *folio, bool partially_mapped) { - struct deferred_split *ds_queue; + struct list_lru_one *l; + int nid; + struct mem_cgroup *memcg; unsigned long flags; =20 /* @@ -4353,7 +4257,11 @@ void deferred_split_folio(struct folio *folio, bool = partially_mapped) if (folio_test_swapcache(folio)) return; =20 - ds_queue =3D folio_split_queue_lock_irqsave(folio, &flags); + nid =3D folio_nid(folio); + + rcu_read_lock(); + memcg =3D folio_memcg(folio); + l =3D list_lru_lock_irqsave(&deferred_split_lru, nid, memcg, &flags); if (partially_mapped) { if (!folio_test_partially_mapped(folio)) { folio_set_partially_mapped(folio); @@ -4361,36 +4269,20 @@ void deferred_split_folio(struct folio *folio, bool= partially_mapped) count_vm_event(THP_DEFERRED_SPLIT_PAGE); count_mthp_stat(folio_order(folio), MTHP_STAT_SPLIT_DEFERRED); mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, 1= ); - } } else { /* partially mapped folios cannot become non-partially mapped */ VM_WARN_ON_FOLIO(folio_test_partially_mapped(folio), folio); } - if (list_empty(&folio->_deferred_list)) { - struct mem_cgroup *memcg; - - memcg =3D folio_split_queue_memcg(folio, ds_queue); - list_add_tail(&folio->_deferred_list, &ds_queue->split_queue); - ds_queue->split_queue_len++; - if (memcg) - set_shrinker_bit(memcg, folio_nid(folio), - shrinker_id(deferred_split_shrinker)); - } - split_queue_unlock_irqrestore(ds_queue, flags); + __list_lru_add(&deferred_split_lru, l, &folio->_deferred_list, nid, memcg= ); + list_lru_unlock_irqrestore(l, &flags); + rcu_read_unlock(); } =20 static unsigned long deferred_split_count(struct shrinker *shrink, struct shrink_control *sc) { - struct pglist_data *pgdata =3D NODE_DATA(sc->nid); - struct deferred_split *ds_queue =3D &pgdata->deferred_split_queue; - -#ifdef CONFIG_MEMCG - if (sc->memcg) - ds_queue =3D &sc->memcg->deferred_split_queue; -#endif - return READ_ONCE(ds_queue->split_queue_len); + return list_lru_shrink_count(&deferred_split_lru, sc); } =20 static bool thp_underused(struct folio *folio) @@ -4420,45 +4312,47 @@ static bool thp_underused(struct folio *folio) return false; } =20 +static enum lru_status deferred_split_isolate(struct list_head *item, + struct list_lru_one *lru, + void *cb_arg) +{ + struct folio *folio =3D container_of(item, struct folio, _deferred_list); + struct list_head *freeable =3D cb_arg; + + if (folio_try_get(folio)) { + list_lru_isolate_move(lru, item, freeable); + return LRU_REMOVED; + } + + /* We lost race with folio_put() */ + list_lru_isolate(lru, item); + if (folio_test_partially_mapped(folio)) { + folio_clear_partially_mapped(folio); + mod_mthp_stat(folio_order(folio), + MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1); + } + return LRU_REMOVED; +} + static unsigned long deferred_split_scan(struct shrinker *shrink, struct shrink_control *sc) { - struct deferred_split *ds_queue; - unsigned long flags; + LIST_HEAD(dispose); struct folio *folio, *next; - int split =3D 0, i; - struct folio_batch fbatch; - - folio_batch_init(&fbatch); + int split =3D 0; + unsigned long isolated; =20 -retry: - ds_queue =3D split_queue_lock_irqsave(sc->nid, sc->memcg, &flags); - /* Take pin on all head pages to avoid freeing them under us */ - list_for_each_entry_safe(folio, next, &ds_queue->split_queue, - _deferred_list) { - if (folio_try_get(folio)) { - folio_batch_add(&fbatch, folio); - } else if (folio_test_partially_mapped(folio)) { - /* We lost race with folio_put() */ - folio_clear_partially_mapped(folio); - mod_mthp_stat(folio_order(folio), - MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1); - } - list_del_init(&folio->_deferred_list); - ds_queue->split_queue_len--; - if (!--sc->nr_to_scan) - break; - if (!folio_batch_space(&fbatch)) - break; - } - split_queue_unlock_irqrestore(ds_queue, flags); + isolated =3D list_lru_shrink_walk_irq(&deferred_split_lru, sc, + deferred_split_isolate, &dispose); =20 - for (i =3D 0; i < folio_batch_count(&fbatch); i++) { + list_for_each_entry_safe(folio, next, &dispose, _deferred_list) { bool did_split =3D false; bool underused =3D false; - struct deferred_split *fqueue; + struct list_lru_one *l; + unsigned long flags; + + list_del_init(&folio->_deferred_list); =20 - folio =3D fbatch.folios[i]; if (!folio_test_partially_mapped(folio)) { /* * See try_to_map_unused_to_zeropage(): we cannot @@ -4481,64 +4375,32 @@ static unsigned long deferred_split_scan(struct shr= inker *shrink, } folio_unlock(folio); next: - if (did_split || !folio_test_partially_mapped(folio)) - continue; /* * Only add back to the queue if folio is partially mapped. * If thp_underused returns false, or if split_folio fails * in the case it was underused, then consider it used and * don't add it back to split_queue. */ - fqueue =3D folio_split_queue_lock_irqsave(folio, &flags); - if (list_empty(&folio->_deferred_list)) { - list_add_tail(&folio->_deferred_list, &fqueue->split_queue); - fqueue->split_queue_len++; + if (!did_split && folio_test_partially_mapped(folio)) { + rcu_read_lock(); + l =3D list_lru_lock_irqsave(&deferred_split_lru, + folio_nid(folio), + folio_memcg(folio), + &flags); + __list_lru_add(&deferred_split_lru, l, + &folio->_deferred_list, + folio_nid(folio), folio_memcg(folio)); + list_lru_unlock_irqrestore(l, &flags); + rcu_read_unlock(); } - split_queue_unlock_irqrestore(fqueue, flags); - } - folios_put(&fbatch); - - if (sc->nr_to_scan && !list_empty(&ds_queue->split_queue)) { - cond_resched(); - goto retry; + folio_put(folio); } =20 - /* - * Stop shrinker if we didn't split any page, but the queue is empty. - * This can happen if pages were freed under us. - */ - if (!split && list_empty(&ds_queue->split_queue)) + if (!split && !isolated) return SHRINK_STOP; return split; } =20 -#ifdef CONFIG_MEMCG -void reparent_deferred_split_queue(struct mem_cgroup *memcg) -{ - struct mem_cgroup *parent =3D parent_mem_cgroup(memcg); - struct deferred_split *ds_queue =3D &memcg->deferred_split_queue; - struct deferred_split *parent_ds_queue =3D &parent->deferred_split_queue; - int nid; - - spin_lock_irq(&ds_queue->split_queue_lock); - spin_lock_nested(&parent_ds_queue->split_queue_lock, SINGLE_DEPTH_NESTING= ); - - if (!ds_queue->split_queue_len) - goto unlock; - - list_splice_tail_init(&ds_queue->split_queue, &parent_ds_queue->split_que= ue); - parent_ds_queue->split_queue_len +=3D ds_queue->split_queue_len; - ds_queue->split_queue_len =3D 0; - - for_each_node(nid) - set_shrinker_bit(parent, nid, shrinker_id(deferred_split_shrinker)); - -unlock: - spin_unlock(&parent_ds_queue->split_queue_lock); - spin_unlock_irq(&ds_queue->split_queue_lock); -} -#endif - #ifdef CONFIG_DEBUG_FS static void split_huge_pages_all(void) { diff --git a/mm/internal.h b/mm/internal.h index 95b583e7e4f7..71d2605f8040 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -857,7 +857,7 @@ static inline bool folio_unqueue_deferred_split(struct = folio *folio) /* * At this point, there is no one trying to add the folio to * deferred_list. If folio is not in deferred_list, it's safe - * to check without acquiring the split_queue_lock. + * to check without acquiring the list_lru lock. */ if (data_race(list_empty(&folio->_deferred_list))) return false; diff --git a/mm/khugepaged.c b/mm/khugepaged.c index b7b4680d27ab..01fd3d5933c5 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1076,6 +1076,7 @@ static enum scan_result alloc_charge_folio(struct fol= io **foliop, struct mm_stru } =20 count_vm_event(THP_COLLAPSE_ALLOC); + if (unlikely(mem_cgroup_charge(folio, mm, gfp))) { folio_put(folio); *foliop =3D NULL; @@ -1084,6 +1085,12 @@ static enum scan_result alloc_charge_folio(struct fo= lio **foliop, struct mm_stru =20 count_memcg_folio_events(folio, THP_COLLAPSE_ALLOC, 1); =20 + if (memcg_list_lru_alloc_folio(folio, &deferred_split_lru, gfp)) { + folio_put(folio); + *foliop =3D NULL; + return SCAN_CGROUP_CHARGE_FAIL; + } + *foliop =3D folio; return SCAN_SUCCEED; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a47fb68dd65f..f381cb6bdff1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4015,11 +4015,6 @@ static struct mem_cgroup *mem_cgroup_alloc(struct me= m_cgroup *parent) for (i =3D 0; i < MEMCG_CGWB_FRN_CNT; i++) memcg->cgwb_frn[i].done =3D __WB_COMPLETION_INIT(&memcg_cgwb_frn_waitq); -#endif -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - spin_lock_init(&memcg->deferred_split_queue.split_queue_lock); - INIT_LIST_HEAD(&memcg->deferred_split_queue.split_queue); - memcg->deferred_split_queue.split_queue_len =3D 0; #endif lru_gen_init_memcg(memcg); return memcg; @@ -4167,11 +4162,10 @@ static void mem_cgroup_css_offline(struct cgroup_su= bsys_state *css) zswap_memcg_offline_cleanup(memcg); =20 memcg_offline_kmem(memcg); - reparent_deferred_split_queue(memcg); /* - * The reparenting of objcg must be after the reparenting of the - * list_lru and deferred_split_queue above, which ensures that they will - * not mistakenly get the parent list_lru and deferred_split_queue. + * The reparenting of objcg must be after the reparenting of + * the list_lru in memcg_offline_kmem(), which ensures that + * they will not mistakenly get the parent list_lru. */ memcg_reparent_objcgs(memcg); reparent_shrinker_deferred(memcg); diff --git a/mm/memory.c b/mm/memory.c index 38062f8e1165..4dad1a7890aa 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4651,13 +4651,19 @@ static struct folio *alloc_swap_folio(struct vm_fau= lt *vmf) while (orders) { addr =3D ALIGN_DOWN(vmf->address, PAGE_SIZE << order); folio =3D vma_alloc_folio(gfp, order, vma, addr); - if (folio) { - if (!mem_cgroup_swapin_charge_folio(folio, vma->vm_mm, - gfp, entry)) - return folio; + if (!folio) + goto next; + if (mem_cgroup_swapin_charge_folio(folio, vma->vm_mm, gfp, entry)) { count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK_CHARGE); folio_put(folio); + goto next; } + if (memcg_list_lru_alloc_folio(folio, &deferred_split_lru, gfp)) { + folio_put(folio); + goto fallback; + } + return folio; +next: count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK); order =3D next_order(&orders, order); } @@ -5168,24 +5174,28 @@ static struct folio *alloc_anon_folio(struct vm_fau= lt *vmf) while (orders) { addr =3D ALIGN_DOWN(vmf->address, PAGE_SIZE << order); folio =3D vma_alloc_folio(gfp, order, vma, addr); - if (folio) { - if (mem_cgroup_charge(folio, vma->vm_mm, gfp)) { - count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); - folio_put(folio); - goto next; - } - folio_throttle_swaprate(folio, gfp); - /* - * When a folio is not zeroed during allocation - * (__GFP_ZERO not used) or user folios require special - * handling, folio_zero_user() is used to make sure - * that the page corresponding to the faulting address - * will be hot in the cache after zeroing. - */ - if (user_alloc_needs_zeroing()) - folio_zero_user(folio, vmf->address); - return folio; + if (!folio) + goto next; + if (mem_cgroup_charge(folio, vma->vm_mm, gfp)) { + count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); + folio_put(folio); + goto next; } + if (memcg_list_lru_alloc_folio(folio, &deferred_split_lru, gfp)) { + folio_put(folio); + goto fallback; + } + folio_throttle_swaprate(folio, gfp); + /* + * When a folio is not zeroed during allocation + * (__GFP_ZERO not used) or user folios require special + * handling, folio_zero_user() is used to make sure + * that the page corresponding to the faulting address + * will be hot in the cache after zeroing. + */ + if (user_alloc_needs_zeroing()) + folio_zero_user(folio, vmf->address); + return folio; next: count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK); order =3D next_order(&orders, order); diff --git a/mm/mm_init.c b/mm/mm_init.c index cec7bb758bdd..f293a62e652a 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1388,19 +1388,6 @@ static void __init calculate_node_totalpages(struct = pglist_data *pgdat, pr_debug("On node %d totalpages: %lu\n", pgdat->node_id, realtotalpages); } =20 -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -static void pgdat_init_split_queue(struct pglist_data *pgdat) -{ - struct deferred_split *ds_queue =3D &pgdat->deferred_split_queue; - - spin_lock_init(&ds_queue->split_queue_lock); - INIT_LIST_HEAD(&ds_queue->split_queue); - ds_queue->split_queue_len =3D 0; -} -#else -static void pgdat_init_split_queue(struct pglist_data *pgdat) {} -#endif - #ifdef CONFIG_COMPACTION static void pgdat_init_kcompactd(struct pglist_data *pgdat) { @@ -1416,8 +1403,6 @@ static void __meminit pgdat_init_internals(struct pgl= ist_data *pgdat) =20 pgdat_resize_init(pgdat); pgdat_kswapd_lock_init(pgdat); - - pgdat_init_split_queue(pgdat); pgdat_init_kcompactd(pgdat); =20 init_waitqueue_head(&pgdat->kswapd_wait); --=20 2.53.0