From nobody Sun May 24 20:35:35 2026 Received: from mail-oi1-f175.google.com (mail-oi1-f175.google.com [209.85.167.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4AC873346B4 for ; Thu, 21 May 2026 15:03:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779375826; cv=none; b=dIriSQtRD8HNnlYK2URYLThud8tIhaZ+T1dvWjk98XlCGQ8YZhvlM2OY6KeOcGyMG2uTGPN8hteFMwI5ZrtG4Bd6EyDO+A53eTD+Yqir2TiRhn09I6Re33i6iZFqmqVF7otMicR1b+HCpXSGJ6SFhXE4DavVJ6uAGxOMfu88HXo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779375826; c=relaxed/simple; bh=U8Z3an3hNgZrCxyekQRPWZceUjB6u0warXZQZI9zDco=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TKCnS+GfcJxFgecIEs26spyPzr3t8opb5dVlaBpyt1ifjfJyD1sJhYqm4PGQtSmfK8YR0wvZcGP10vUcvG9E5gnWcAwrjWyG7FmKng/RiXLVgoeAx4eDLgKMZzB0nbV+2yJn0OwMdK2BURQq7dxdz8UKYjGHVcyrsZofexe0Amo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=MPLRq5Vc; arc=none smtp.client-ip=209.85.167.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="MPLRq5Vc" Received: by mail-oi1-f175.google.com with SMTP id 5614622812f47-484d3c0855aso2439979b6e.3 for ; Thu, 21 May 2026 08:03:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1779375824; x=1779980624; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1cbAO2KW+lIj5NuloFoDpyrLcHnbVUzz5CiT+zyYy0s=; b=MPLRq5Vcn/sK/aXPgD0CaHvegRRF9M1+l6r0nhCwNEvOD9feeAEykYn1oixuTqPTH5 BLplBfvM0g/y/PrmXWRgraUG/7uslPVrpAoauviD0iQjAO20owdE44O/3Peczfyk7yRG E17LbfVG+8AwU7wLlDpTRzKHddUYIlXSI36UBxyCyNxTeL1DBqHAgbTVlw+h1YmgUx3+ YpvsIR5bgt3OHUyVXRhck7Sf/RuTURNQ/uAEHML9mK4VLN0b0uJo8iy1gD27ybn2HwXU izgu3Erd4l5jS3zsNOPp/EzMw3YpnVEG+45F0ximUypIFk4hH/M9zZhennlZFeD6qsP+ 2iBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779375824; x=1779980624; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=1cbAO2KW+lIj5NuloFoDpyrLcHnbVUzz5CiT+zyYy0s=; b=S0Ync3voJV2PJvkEcmsoWOG+ujKpvtCV+4ANZa6t8MZCHMvfS6lCLODRSDhGFl5BOX UcgXCHiQ/cRwhh3CQ/XRoPVTEl0o3Txh6eZTbmTvn9hJGe59ooK7QAuHYemkdTWNNgTt Yp9MaY54f/b9AcGVMeCrNk+vyTsc1zwRpDBQVwstKQPKxrJNlO+AniwJ8NFj8odCs0LZ f7lWD9AIgc52rQJ397AM75syccg4kD+u8I+IpW8iCh3ggfun/2rXtVVEFWJoHRRvqbDv C4ZkVTDuv3GH0BDJOzxDayl1zPCutAe8+oZMiRzYD5T797ehav6y26DxrD00yG/DKWjY e7IQ== X-Forwarded-Encrypted: i=1; AFNElJ84zUH+frULdOnHjtJ3IgRQVTukED6h5aorErVIGh7pENm2Cw/jw9VUWvi00nSyKNKra3vMgRkTxkQ+lWA=@vger.kernel.org X-Gm-Message-State: AOJu0YzK5hcydioC43SWry24LkFJ3aIQkpLiRjFSzJNAikqGAtBV0zC2 4+14zJsgndh64rWCY3yROvJ0NVY68l8iGb1SOMnyDYoFPFa62Y1NRAQYuVuB/K4LUnA= X-Gm-Gg: Acq92OHph2NWm4B5VvGFvUqrz/mswLg2JW/PgkX06l7jsdITtGGz9WvpkRZM4HLl3cI 1i4ztsTwcBl46z1BN0m17+V/a+lPKjs3Co0eLoEO48HocqQ98OKN7T38MFftDcbPNLuXVRp6ELx 2WIw1s+CWXZsSHGFxO7eqbpv/goaiduBi6fiKCDapRrqjwU+70jqX+2l+JlzY3RSHEoSxk3LVni qsT8mFzR9RMAsBOYL9xehL6B58oX4AGT53rdRYsv26fkEE50sGoA0dsG9Hu7yNlKM6xFPYQkISy I+Y5Z82C58wp7ZTw+CwNmtnpGOnXFXojomlSrcAttQ4ba+FlM0klfCPOB9R443hv1IC4qmjKF5F 7s6d0QFjaetFoNjcD7OyIzbEKv31qfaYqn43OeZ1Jy5HgQFK1P7SmGssDEaF7TAB1kW0vp64k4j NFav+/39tRY5voIR8bic4RyA== X-Received: by 2002:a05:6808:2221:b0:467:1941:1f0d with SMTP id 5614622812f47-4852eb590aemr1685571b6e.11.1779375823781; Thu, 21 May 2026 08:03:43 -0700 (PDT) Received: from localhost ([2603:7001:f100:500:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-516cce102f7sm8647231cf.22.2026.05.21.08.03.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 May 2026 08:03:42 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: David Hildenbrand , Lorenzo Stoakes , Shakeel Butt , Michal Hocko , Dave Chinner , Roman Gushchin , Muchun Song , Qi Zheng , Yosry Ahmed , Zi Yan , "Liam R . Howlett" , Usama Arif , Kiryl Shutsemau , Vlastimil Babka , Kairui Song , Mikhail Zaslonko , Vasily Gorbik , Baolin Wang , Barry Song , Dev Jain , Lance Yang , Nico Pache , Ryan Roberts , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 1/8] mm: list_lru: lock_list_lru_of_memcg() cannot return NULL if !skip_empty Date: Thu, 21 May 2026 11:02:07 -0400 Message-ID: <20260521150330.1955924-2-hannes@cmpxchg.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260521150330.1955924-1-hannes@cmpxchg.org> References: <20260521150330.1955924-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" skip_empty is only for the shrinker to abort and skip a list that's empty or whose cgroup is being deleted. For list additions and deletions, the cgroup hierarchy is walked upwards until a valid list_lru head is found, or it will fall back to the node list. Acquiring the lock won't fail. Remove the NULL checks in those callers. Reviewed-by: David Hildenbrand (Arm) Acked-by: Shakeel Butt Reviewed-by: Lorenzo Stoakes (Oracle) Signed-off-by: Johannes Weiner Reviewed-by: Liam R. Howlett (Oracle) --- mm/list_lru.c | 5 +---- 1 file changed, 1 insertion(+), 4 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index dd29bcf8eb5f..d3619961a7ac 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -165,8 +165,6 @@ bool list_lru_add(struct list_lru *lru, struct list_hea= d *item, int nid, struct list_lru_one *l; =20 l =3D lock_list_lru_of_memcg(lru, nid, memcg, false, false); - if (!l) - return false; if (list_empty(item)) { list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ @@ -204,9 +202,8 @@ bool list_lru_del(struct list_lru *lru, struct list_hea= d *item, int nid, { struct list_lru_node *nlru =3D &lru->node[nid]; struct list_lru_one *l; + l =3D lock_list_lru_of_memcg(lru, nid, memcg, false, false); - if (!l) - return false; if (!list_empty(item)) { list_del_init(item); l->nr_items--; --=20 2.54.0 From nobody Sun May 24 20:35:35 2026 Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 291723655DF for ; Thu, 21 May 2026 15:03:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779375829; cv=none; b=O+BgdtrLiRcinatXunQBIysNGuVUrLNnT/W3qPe7mk3hz0vsWT/2HGiCtFNI9nRqt/YRT4mkyYK+0wY2VdJkrqcG5g5HciJKKK/ub2rlhTkVdm/h8Acg+k8TOGaIweCaE0wCaAow8qSFYSeIB2UgqIOHCCUWFm332q9hcRyveEI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779375829; c=relaxed/simple; bh=7vbHWCiV+CF1x1UjKtGMwHGjchNHDlG16hVGTCF4ge0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=B3jf1Zn3NHVIrt/T5qus7i3lDGC2zo4gpmMJfibzUzQSVz7RnJn9oeFchIkZA5+z2f+ZS+uslJoaR9LlJu3glXKoRI7B6fU6EQsQ52CwH94YHbsXb2O45+imjfOp43jnNXmJ9neCaFiMhwlubUAbosSNYBGJ+csclAvOA5uarxc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=gNbx8Jg0; arc=none smtp.client-ip=209.85.160.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="gNbx8Jg0" Received: by mail-qt1-f179.google.com with SMTP id d75a77b69052e-50e5dbd8e0eso70980741cf.1 for ; Thu, 21 May 2026 08:03:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1779375826; x=1779980626; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Vn9mXwqyXlTnHz6VCNHiXrEOY97snOle2kaY6Iy8PrM=; b=gNbx8Jg0alMVz9JNUPXbPoGIZ3SHR5Jro41ueL12iC3N0cugFsxoj5gN15arKmwObG DQZ5cBwNdLrzKbY4YOivqpHx7Qq1PIJ/kpKAy+2Gp53pkyLitjiafP0ggaEjs2q2hrQS fDlTiBKyDN4zweU4Sap6EkZL60Z6Dd1HxUJe3NXLvmQOgrXwv5S1/kfdtfScRQ7SJpqD scVUjeKp8cAhSRUhHAg512oQ4TDei0NX7gXEV+ZY/TUSpCuRAaey3+x2jKC1xrmBBjT1 N1SEIlEtbN832hogAMHA+pxWy2JLVfzYNnfE06YwDBULRtBAXIaXc/9R0927RfDffuNi 49wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779375826; x=1779980626; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=Vn9mXwqyXlTnHz6VCNHiXrEOY97snOle2kaY6Iy8PrM=; b=U/ENL0oic16uZys7AqeZco5rbsYrydPQhU46ZirTNq2vVgwSrIcLW5S9Nd4UBKq0w2 eA7crrxjDTWBKpgKfgEjCOALZDKaHDLqfKSogNzLL0z+AJvo1MGtSnLeIf0myy/BMsV3 WGuayj2tYLiStzAHHh/2HkQppuI0HgTLGqUiCEpz1/peOuB0GAkf61i9wufhFa7/Mn1M 7z2BNU6DJTzqoSZ315zJcCUlCmxwhePUz4zOKgZPBPIIQqoGR4MdCLTZ3nfv+fjydAKf Jhbpw/QcRQ5fKjHYQddeexZwLkKnGT29poVgSrLtjIoWlW9aqY1YuKz9MwSIx4st62cj vLng== X-Forwarded-Encrypted: i=1; AFNElJ96ME051MPRbZ3XKAIhzIY5a7MFm1EqmgfzBeMmlwL0OgjyEYLpTfK9w4Yyv7sO+QSTa+Ked4/Ax9/WwPs=@vger.kernel.org X-Gm-Message-State: AOJu0YyeJu3re9txASNtqAZ2WjlVc68A4Fpg1cBqGoUa30ZK8bRH3l6S KckIzWMfY6uyqL79xUPCfDlEcwADR2f8NTJiL6xHKqfuoSpkZ/8vBUzxVMBANUT1nuo= X-Gm-Gg: Acq92OF/2QqKaGTSfIiPAkkHtIemSQZ5BEZsCGnFCZKqcxxvYlv2DCkateNILgra0Ad 8oRLPHhdwqM3svASzQOdryELJDPPrmmiBzQhKaN3ds85kQWT8RvjV5eX0kIJNr7FGLFxAPdpIne jeOP4QZ6FSqGNfqaHqFDf+GeoorwmUKCqjpt8LRloEzmMRPvUQ2F/S48/vM4XYVLf3KHVqWLU8k yuO+dxyHyRvMgWw7d3Zlla78assMaX1Zl/KPaMxZhlYcFkvEd+dFQ+bVVC7mrQmNjoLHOYwX/3g YliZmGNYiuEDnmak2ipLkDe0W88Sz7BtDiL/yTlo1y2DSimnPc4l2T2VNLo/iFueKymAVjJ+GD+ RSMhG+GsAaFOQQloXLvldIQIYm3Vfcj1j4pfAzNZnZB5sXQo0ZM7Votkm6EzkSINeXNNASfz2GN Wt3t+6PD6KSB3QEj0S/h8nUA== X-Received: by 2002:a05:622a:4296:b0:4f1:ab79:fb18 with SMTP id d75a77b69052e-516c54e89a9mr45144641cf.25.1779375825779; Thu, 21 May 2026 08:03:45 -0700 (PDT) Received: from localhost ([2603:7001:f100:500:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-516cce01aafsm8868781cf.21.2026.05.21.08.03.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 May 2026 08:03:44 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: David Hildenbrand , Lorenzo Stoakes , Shakeel Butt , Michal Hocko , Dave Chinner , Roman Gushchin , Muchun Song , Qi Zheng , Yosry Ahmed , Zi Yan , "Liam R . Howlett" , Usama Arif , Kiryl Shutsemau , Vlastimil Babka , Kairui Song , Mikhail Zaslonko , Vasily Gorbik , Baolin Wang , Barry Song , Dev Jain , Lance Yang , Nico Pache , Ryan Roberts , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 2/8] mm: list_lru: deduplicate unlock_list_lru() Date: Thu, 21 May 2026 11:02:08 -0400 Message-ID: <20260521150330.1955924-3-hannes@cmpxchg.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260521150330.1955924-1-hannes@cmpxchg.org> References: <20260521150330.1955924-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The MEMCG and !MEMCG variants are the same. lock_list_lru() has the same pattern when bailing. Consolidate into a common implementation. Reviewed-by: David Hildenbrand (Arm) Acked-by: Shakeel Butt Reviewed-by: Lorenzo Stoakes (Oracle) Signed-off-by: Johannes Weiner Reviewed-by: Liam R. Howlett (Oracle) --- mm/list_lru.c | 29 +++++++++-------------------- 1 file changed, 9 insertions(+), 20 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index d3619961a7ac..9a68177619bf 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -15,6 +15,14 @@ #include "slab.h" #include "internal.h" =20 +static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off) +{ + if (irq_off) + spin_unlock_irq(&l->lock); + else + spin_unlock(&l->lock); +} + #ifdef CONFIG_MEMCG static LIST_HEAD(memcg_list_lrus); static DEFINE_MUTEX(list_lrus_mutex); @@ -67,10 +75,7 @@ static inline bool lock_list_lru(struct list_lru_one *l,= bool irq) else spin_lock(&l->lock); if (unlikely(READ_ONCE(l->nr_items) =3D=3D LONG_MIN)) { - if (irq) - spin_unlock_irq(&l->lock); - else - spin_unlock(&l->lock); + unlock_list_lru(l, irq); return false; } return true; @@ -101,14 +106,6 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, = struct mem_cgroup *memcg, memcg =3D parent_mem_cgroup(memcg); goto again; } - -static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off) -{ - if (irq_off) - spin_unlock_irq(&l->lock); - else - spin_unlock(&l->lock); -} #else static void list_lru_register(struct list_lru *lru) { @@ -147,14 +144,6 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, = struct mem_cgroup *memcg, =20 return l; } - -static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off) -{ - if (irq_off) - spin_unlock_irq(&l->lock); - else - spin_unlock(&l->lock); -} #endif /* CONFIG_MEMCG */ =20 /* The caller must ensure the memcg lifetime. */ --=20 2.54.0 From nobody Sun May 24 20:35:35 2026 Received: from mail-qk1-f170.google.com (mail-qk1-f170.google.com [209.85.222.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B1AD33672AB for ; Thu, 21 May 2026 15:03:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779375830; cv=none; b=dYqyEOn78jmEMChY7Uo3PEJTNsBVswmqzjTMVR5pR3rtNIXEq9wK1xF0QPAJyjeHcBtknysMEjEBwTRlxKXwyFhMhMyiPEnkUb5IWypFd58CBnYBwLBxrtbzZrmat/dwVBQYBcBTx4EaKEEN64rlqlbeMaTropcYaUvXKjfCY1U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779375830; c=relaxed/simple; bh=0c1+pP8SAG7u+WBWdlkZnBVO4585VjMZzm3levEOLwQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WcMFo1wXeODzchrSCvBAXSZWx6db2KPxiu23uxxdG7sVtkh3b3NWtLRzT5PCPIOqlIABH2wr0gdm27giMazopJ1Qj0ZWPcfqwr9vtg92wIQD3BU5sAgxI+YxgVMlIS4oG784M8xm5FOJv6zChi2w3CuBj248+jKIoV2HN2zlN3g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=hiS8Hizj; arc=none smtp.client-ip=209.85.222.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="hiS8Hizj" Received: by mail-qk1-f170.google.com with SMTP id af79cd13be357-90b2fcf90a0so871224685a.1 for ; Thu, 21 May 2026 08:03:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1779375828; x=1779980628; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=XJv7QOlxyTdHx0IQuT1IvjERhQqwxSN8j0WlihPGf50=; b=hiS8HizjDgDC+cgfCUmN1YcM4hJ/xRmwlwKHYzRrYmFK2CIki00iFM6eM+/B6AORNo jyYbQusNCa5Hy3hPmrHr7LZtrdp00HzYEq/hz71NIk11dob8AszeYC08EJYoA0e4es/z GYhmCtFwP7YkkjwDM9ik9cKqoFlrGmOckjP9Emx4fJWkPpxboS2O9UUWQcMS3247I6ax X8kl+SkegB1e2ReZeGpqzkjRBdDDUyEg09t8SQcoUDPVGbIkzRe3NyMM4LCtik7UcsLE V0S/broZx+ZoIJA/qfNG+eqpFftVHNfSeG1ObZ+1YTTfEon2nw+sbfPIvRFd4Ek0qNwl szPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779375828; x=1779980628; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=XJv7QOlxyTdHx0IQuT1IvjERhQqwxSN8j0WlihPGf50=; b=NA9X59ct+kAJPfFz8oHl16fw4Wo80RHLwQ0ERzkgfSwaniBa6koEAVHCjRqOmRSmtu I2+Srxf+bRvU8LVY4i68z7wFmMufA36JTOvL0A2CMewM0J+ACiVubCT3hJGmR7d4Zar+ 2blMyv90tkxD4VttiKKRS19WOl1070RsmTgKcgKUjpqkjxGtMKdn+3fYcIrKk6yipUvW Irqh8sNhB6FcTSJmmbsMLEP1YQZ5zLyX2S8IjaFIBJVq/fpLjaVoBDyOWrvVZwaDvl4N xcMPMUp/KQyfu4fGy1pSkwcJQjwb1mZ+wPrMtRtn6EjATI0HR8FcGX10R8201yxU9tNf txLQ== X-Forwarded-Encrypted: i=1; AFNElJ9VhHB440puSCDPWfTDVb/Qb2Cmc6CQKpDV0qy9QadDKeIFMkErQFgoVt5RSH1cdshFJZkicWhFUnNyzMc=@vger.kernel.org X-Gm-Message-State: AOJu0YzZ5mUy5Qs7+U97Sxx5l4kIyNpx/2S52INatmHG8JQTs+LDRJD/ q3dR6CxLcVbrNnW4GADQQhqMYuKSDMY5VtsME/NrDzmTxqsu1y2ZDwmBs2OPnACCbcg= X-Gm-Gg: Acq92OE0sAe4Kgzn4cWg2edltfJqEnqxRoLQTOslDN3NhU7ZaicbZyzucv2QlOK7hT0 yQn4ZH/rS+YNKl8EylMeQ5G/VQiblPYvNE0XpNaRezRSA9QtIKO9btwCJzcyx/wSm20iP8RCaVu GjGMwi8OYgIcuNjgZHz2a7Ea4ulAvCM3ZCsxbacQi48Rp+qvJu7k5mqegy3mbG4EawmQNjdSylq vf0GVwFCQfBK1Nxt5izLZDL5vnpzZxq1iC+FMos/QGXrTo+m2wLtpCpNMCdzaF0icMs0PosrVFM GtvK8kTStprK1ttHEp2s9Li/NEUv/krETF75Gv9Bv5c3F/NS4P+/HRhsky3biK6THGzlYQH2WZB bL23JAMTgMwJ6bAj5fVOWxZVbbtABGu5bbXAeUWEfyEEF//8kzvdACRyYsmr/O5+mzyiDHVel8I QBzP4RQmLyHLD3nj66e/mJBEuPtdsLZcgs X-Received: by 2002:a05:620a:47ae:b0:914:a824:6695 with SMTP id af79cd13be357-914a8246d0bmr214593485a.37.1779375827591; Thu, 21 May 2026 08:03:47 -0700 (PDT) Received: from localhost ([2603:7001:f100:500:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id af79cd13be357-914ab9faed2sm114273485a.6.2026.05.21.08.03.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 May 2026 08:03:46 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: David Hildenbrand , Lorenzo Stoakes , Shakeel Butt , Michal Hocko , Dave Chinner , Roman Gushchin , Muchun Song , Qi Zheng , Yosry Ahmed , Zi Yan , "Liam R . Howlett" , Usama Arif , Kiryl Shutsemau , Vlastimil Babka , Kairui Song , Mikhail Zaslonko , Vasily Gorbik , Baolin Wang , Barry Song , Dev Jain , Lance Yang , Nico Pache , Ryan Roberts , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 3/8] mm: list_lru: move list dead check to lock_list_lru_of_memcg() Date: Thu, 21 May 2026 11:02:09 -0400 Message-ID: <20260521150330.1955924-4-hannes@cmpxchg.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260521150330.1955924-1-hannes@cmpxchg.org> References: <20260521150330.1955924-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Only the MEMCG variant of lock_list_lru() needs to check if there is a race with cgroup deletion and list reparenting. Move the check to the caller, so that the next patch can unify the lock_list_lru() variants. Reviewed-by: David Hildenbrand (Arm) Acked-by: Shakeel Butt Reviewed-by: Lorenzo Stoakes (Oracle) Signed-off-by: Johannes Weiner Reviewed-by: Liam R. Howlett (Oracle) --- mm/list_lru.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index 9a68177619bf..9da6fce19832 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -68,17 +68,12 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, = int idx) return &lru->node[nid].lru; } =20 -static inline bool lock_list_lru(struct list_lru_one *l, bool irq) +static inline void lock_list_lru(struct list_lru_one *l, bool irq) { if (irq) spin_lock_irq(&l->lock); else spin_lock(&l->lock); - if (unlikely(READ_ONCE(l->nr_items) =3D=3D LONG_MIN)) { - unlock_list_lru(l, irq); - return false; - } - return true; } =20 static inline struct list_lru_one * @@ -90,9 +85,13 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, st= ruct mem_cgroup *memcg, rcu_read_lock(); again: l =3D list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); - if (likely(l) && lock_list_lru(l, irq)) { - rcu_read_unlock(); - return l; + if (likely(l)) { + lock_list_lru(l, irq); + if (likely(READ_ONCE(l->nr_items) !=3D LONG_MIN)) { + rcu_read_unlock(); + return l; + } + unlock_list_lru(l, irq); } /* * Caller may simply bail out if raced with reparenting or --=20 2.54.0 From nobody Sun May 24 20:35:35 2026 Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B6C61368D65 for ; Thu, 21 May 2026 15:03:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779375832; cv=none; b=G++JFfjpN5gro5+7Bowr19MWkEb6qZ8ZYvW+Txq4HnKrAiQiRg/HpVcueipRwGG53s8z45GaIygmwKyMhUWEr1YoFShgmEIhY82wtl7C9STYq7aT4Xuh7wdYZoJB0i2uCbVnsDn5tsAUgQ85btF2ZAKEuWnOdvGs0hih9IS4ibk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779375832; c=relaxed/simple; bh=j8U8aUC7GF7hZAPSnE7T1xSkXrx5X2zPqG7sVz0ma50=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iFrLQdUNLVeSdwdoUr3/M8RX7/i2zMWqC8SgyXBFq1XxbRkIwbza8Qry94JZTP1pH/LNWuWIElvHbSJpYLCNmF2ehk4SDuxd5aq5WJzVb5fi1uCi8XsDXdXr8V+7I1diLciOEZA0K7AlmW/K7OahBhjm/E+nZd0i9orBwjmdtBU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=HvW18WP2; arc=none smtp.client-ip=209.85.219.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="HvW18WP2" Received: by mail-qv1-f46.google.com with SMTP id 6a1803df08f44-8acb3daf2aaso103800406d6.0 for ; Thu, 21 May 2026 08:03:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1779375830; x=1779980630; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BH9Ib0oi5E8fnc2UPrjbKjmGT1EOdGnzHGNmtCTvMlI=; b=HvW18WP22E+fJyxY94AuFRyyP2CZZf2oCfjPP2btq8n0Gux83gMzljmOzLBGa7kZU5 iDh+SVMvN0EYNwYLHC2Aa2JSFYJo9qrV1zHzyiusB3+d4BLLb6BoRpQv2ASOz6K/yZwY kK18JARRo98uIJmQxScM6obmznrTT4uhrYE6hoX5Q5QdTtWd++D2+RVFkXFIh1xdPZCo coKBWy9K6N7YE6jjdkwFwsZwXNMJj0hb83hM1uGjjXpHbR6+2qjH+Kap93/OwhSmvoqC wx54awSNzNqAf6vM1cbBPG+/ekdC+rYPnyvX/ylazQ3r3YKMakCBJGZp/r6YmlamqFKc g36A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779375830; x=1779980630; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=BH9Ib0oi5E8fnc2UPrjbKjmGT1EOdGnzHGNmtCTvMlI=; b=gmj4/j3UXVqoMTPWMZu/0b3zqJYwdKd2DaBsQW1XWOUNYia65maP/kQ5gsCdAFHgQb 9WqEr7ff1DGJjRxLvpZnGNa6AfOZnqmAlK7zuQ/TMznDn1AwBApxYHcAOlfU30AbJAJN GbarK/48ERsPRaUAAbmOtrmjEqDSgFOgRaYyXdeIH4heiiZBcWx1oF6Uwm8lpq8ClQ/2 jmwXHKy+MI7OPy1Im2q4hVSrrC8fctgb2R740IB7Lnv4oNeHbaRCk6OylWTwNS8j9K39 YZ9lwswjkkYXoNt7zR2h9RrMk/hphws0rmWtok/uflELXBRujiKcfllUw3ULBHmen8kM LbNw== X-Forwarded-Encrypted: i=1; AFNElJ8eHG63B4kL/c2BIN5JcbST1GTKgl83hXTTeO9F80+au4oejVTDsOR7ortAZecbuY/YUuD6693fgL9mqvQ=@vger.kernel.org X-Gm-Message-State: AOJu0YxLsfPjsjofaMlryXCXRoRYYVZoCprGSFakqvRXOWcV/Iarl232 ayA6zpcqCkp6xJNMwusDQQl2QWIcl4x+7zLbAG5VQuM6UJ5IeRmyxSEuK11eiHvQ4vM= X-Gm-Gg: Acq92OHnZjDgNzcMocvHpHoHZCRM0g8TFB23cW1svL+XXmIxFsbjJ+84/y/S03c2lTQ AjBJBdxzdcixh8kzn2pkm0wfsGJ+uRCFXSDgxFRSrZh8k6vSSvWKdl9fOoxjPsmv1+JUdIoaf2Q 8oKknK3E3JOsdGTut2ioW31CmhWzK+uoB5KeERHzu0FKgjmuuGT7Wb6ZGdux6wbz57uLzNLqEYo TPoUroK0eAGQJNRqHUFPevXeU6cMEl8Wx4X9aK0Hw1hfkCeEExPEfyUqdrEQ9efMrG7ChuKkSr1 Xo5XEIDz3h1wv0Irn1os1RyPTI7QFfT0whjyug7Xlq+A1gDPYuTaxe8dCe6049KgeittQdT8653 Eq/HWG0VPLcdI5WV9SCvZ/dp0Qe7DRIh4bQhiMrWUH4ixws4Gju89Vp+eIOASYcOXNXXgJttjnV Alw1zJGOGknwXXZLYOHfo0dpyHl01GaCIH X-Received: by 2002:a05:6214:3a8a:b0:8bd:6baa:6a0c with SMTP id 6a1803df08f44-8cc6e31e7ffmr51218506d6.11.1779375829379; Thu, 21 May 2026 08:03:49 -0700 (PDT) Received: from localhost ([2603:7001:f100:500:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-8cc768b7032sm7149496d6.4.2026.05.21.08.03.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 May 2026 08:03:48 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: David Hildenbrand , Lorenzo Stoakes , Shakeel Butt , Michal Hocko , Dave Chinner , Roman Gushchin , Muchun Song , Qi Zheng , Yosry Ahmed , Zi Yan , "Liam R . Howlett" , Usama Arif , Kiryl Shutsemau , Vlastimil Babka , Kairui Song , Mikhail Zaslonko , Vasily Gorbik , Baolin Wang , Barry Song , Dev Jain , Lance Yang , Nico Pache , Ryan Roberts , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 4/8] mm: list_lru: deduplicate lock_list_lru() Date: Thu, 21 May 2026 11:02:10 -0400 Message-ID: <20260521150330.1955924-5-hannes@cmpxchg.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260521150330.1955924-1-hannes@cmpxchg.org> References: <20260521150330.1955924-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The MEMCG and !MEMCG paths have the same pattern. Share the code. Reviewed-by: David Hildenbrand (Arm) Acked-by: Shakeel Butt Reviewed-by: Lorenzo Stoakes (Oracle) Signed-off-by: Johannes Weiner Reviewed-by: Liam R. Howlett (Oracle) --- mm/list_lru.c | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/mm/list_lru.c b/mm/list_lru.c index 9da6fce19832..65962dbf6dda 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -15,6 +15,14 @@ #include "slab.h" #include "internal.h" =20 +static inline void lock_list_lru(struct list_lru_one *l, bool irq) +{ + if (irq) + spin_lock_irq(&l->lock); + else + spin_lock(&l->lock); +} + static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off) { if (irq_off) @@ -68,14 +76,6 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, i= nt idx) return &lru->node[nid].lru; } =20 -static inline void lock_list_lru(struct list_lru_one *l, bool irq) -{ - if (irq) - spin_lock_irq(&l->lock); - else - spin_lock(&l->lock); -} - static inline struct list_lru_one * lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *m= emcg, bool irq, bool skip_empty) @@ -136,10 +136,7 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, = struct mem_cgroup *memcg, { struct list_lru_one *l =3D &lru->node[nid].lru; =20 - if (irq) - spin_lock_irq(&l->lock); - else - spin_lock(&l->lock); + lock_list_lru(l, irq); =20 return l; } --=20 2.54.0 From nobody Sun May 24 20:35:35 2026 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8323336A370 for ; Thu, 21 May 2026 15:03:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779375834; cv=none; b=SMFbswkfxk9sVFGBSNA6lPed5G359U4v4IHHfWKN3Kj2OhscDDoM6H47kwyHV6DL33NaLVQoem3FiHEeF0KMGuYNn8ZCnoME+/9IMyYjAUKK9PzaWSjebT3MJGUPs9WiqvmJjCSP69zDuwWa7b8grjMgFS2DBCt+xdh9IAWoKE4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779375834; c=relaxed/simple; bh=jPUR/BukKVEfdy9PQWQstfIwiip/659dV2dK1M/R6KE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RnsC0OUB2A/Z9pjxPjIfy5fbmSrOxQR3M8hStMuenki1BLFJVWmz+aODpToLxWqxcq651SX/IIBUSgG95FVQxunYCOtA7Oc5dtKnpwEob+xuheutcQ8E/XxihLjgSlCVLI6fTfQKBSIKiPNE0ul5p8lOVK1zmhBmS24EuzsCNNk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=SCOi4vRt; arc=none smtp.client-ip=209.85.216.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="SCOi4vRt" Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-366be8040a9so2638544a91.3 for ; Thu, 21 May 2026 08:03:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1779375832; x=1779980632; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=USyWRFDNXHImycXHNr3l4mW9eMlhPd90cPYpqyp43EY=; b=SCOi4vRt1MEuazsd+OQXt3NN7g8/YZhF7A0NTn6F+gsBRviY3gwxyt2aLC6jnW1Y8S MzHRr9mhXenhzJ6q7rhJiNcfty5BUXD4XpdiZ4s0aOj3QLahmHv4u02k5+N2m45IE3U/ 2v6l7SVcnoCPlU3/KMc2R8cCkeWfJzGfh/M5HlaQ6iPOo8prBNMZQBuCsjJYWhGeR3ui fZvqi+8AjFf1ICSsRUo2CX9Yo/FXoMVbk8XOlXAKaJ/Ax9LV91r1M/2ZaECCXh1ali8g Y/luLOmDmvZdSlIfuB4IFjIFOGkj/H3PzbYD/k/EVwRZ6tP/KX+anfpfPEJXeGS8cwk9 XfJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779375832; x=1779980632; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=USyWRFDNXHImycXHNr3l4mW9eMlhPd90cPYpqyp43EY=; b=Iv/K4Yp34xauZifDR4NvtSJT7SHTpZbEXUyCSC+FimlFvvDJOjoLG9HhFG8MdJOQYS wFjaZzuHz/cUat1LHyYaV7okVTd6c9gzlKGK1FaFx6m/nVVE95JEN1wCcQVzLUFqkcj9 dry1J2gnhsMevR+imF/6zCuRxKfKiYxzjfQDtoLurLibTXjDZSsFuD953hz3lNVubeXP hslA0DM0aK6I9bbgb/JX27TuEBjZKvpW7AdGEJpUebLZ0UWWBS0TLzBUg0oF51GCm3uU mwPIlPfiLKR1vB0fnXRVeXx6nvJCyvnNMKRatWkL0TyxhGSNHljT0Q8vZ5/DgagJzYV0 IP1Q== X-Forwarded-Encrypted: i=1; AFNElJ/02yGE4L5Qap1C+tg4mbPpwmIYI924SUnL24teXPr19IViy9EZO6I44MBmaxVXnRUXeX/VNFNt+fpkhmM=@vger.kernel.org X-Gm-Message-State: AOJu0Yxx+CoKT/HJ4QE8KITZi4CGXJa7mwI7QUaVxlzavVcOlmV694Yi JtnMl69vYaCnjsMScoYNTqg6o1i1BBWzirEh8852C2Wm0P+3yOddF28sjpx2WZtFXRk= X-Gm-Gg: Acq92OHEePd1gO7EJ67Z6Rm2CzTZUwAsE1W4HfKVtUo7z0RSI4OFpR2l7SIkfHmXQcb 2PwJy8CJ43b9GdZFWJh0VdBjUzS/t06YDEdQdbUz7SdvURv2GY6dHSunZuJP79aAPHB0AqHb7bv Zaw1f+29rsuK3GQ+72dRX3VuBqQmh+fEAzuE50pVVym43XxGsQwwmokslI6Rp5X0HkeVwHOZUIC 9lg919cYIiSSJkEzg/Wei5L4Vkdi9SojdxwmtuUdPuC7gQ21qeqklPEE1KvGLagxiY++BEPOn2t cO4NO2aWNFGVVypXsZ+ZNEwJd1UuRBCNIdk6Tn526PVzAQ3wOcw/cP6f/UZ1WLZTsNCqThv7ppQ FUSUm9qjdmFn8xJTGIWwqM5OiMBtt6zLsLK5eTC5mmWaJm4NQhbIOTvY/744h3P/Y537LFX37TD BwAsiW0n7jyVqWZLthl5Gxdw== X-Received: by 2002:a17:90b:548b:b0:369:996d:282c with SMTP id 98e67ed59e1d1-36a4518224amr3266558a91.9.1779375831567; Thu, 21 May 2026 08:03:51 -0700 (PDT) Received: from localhost ([2603:7001:f100:500:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id af79cd13be357-914abb9935bsm114647585a.31.2026.05.21.08.03.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 May 2026 08:03:50 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: David Hildenbrand , Lorenzo Stoakes , Shakeel Butt , Michal Hocko , Dave Chinner , Roman Gushchin , Muchun Song , Qi Zheng , Yosry Ahmed , Zi Yan , "Liam R . Howlett" , Usama Arif , Kiryl Shutsemau , Vlastimil Babka , Kairui Song , Mikhail Zaslonko , Vasily Gorbik , Baolin Wang , Barry Song , Dev Jain , Lance Yang , Nico Pache , Ryan Roberts , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 5/8] mm: list_lru: introduce caller locking for additions and deletions Date: Thu, 21 May 2026 11:02:11 -0400 Message-ID: <20260521150330.1955924-6-hannes@cmpxchg.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260521150330.1955924-1-hannes@cmpxchg.org> References: <20260521150330.1955924-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Locking is currently internal to the list_lru API. However, a caller might want to keep auxiliary state synchronized with the LRU state. For example, the THP shrinker uses the lock of its custom LRU to keep PG_partially_mapped and vmstats consistent. To allow the THP shrinker to switch to list_lru, provide normal and irqsafe locking primitives as well as caller-locked variants of the addition and deletion functions. Reviewed-by: David Hildenbrand (Arm) Acked-by: Shakeel Butt Reviewed-by: Lorenzo Stoakes (Oracle) Signed-off-by: Johannes Weiner Reviewed-by: Liam R. Howlett (Oracle) --- include/linux/list_lru.h | 41 ++++++++++++ mm/list_lru.c | 131 ++++++++++++++++++++++++++++++--------- 2 files changed, 141 insertions(+), 31 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index fe739d35a864..c79ed378311f 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -83,6 +83,44 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, struc= t list_lru *lru, gfp_t gfp); void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup = *parent); =20 +/** + * list_lru_lock: lock the sublist for the given node and memcg + * @lru: the lru pointer + * @nid: the node id of the sublist to lock. + * @memcg: the cgroup of the sublist to lock. + * + * Returns the locked list_lru_one sublist. The caller must call + * list_lru_unlock() when done. + * + * You must ensure that the memcg is not freed during this call (e.g., with + * rcu or by taking a css refcnt). + * + * Return: the locked list_lru_one, or NULL on failure + */ +struct list_lru_one *list_lru_lock(struct list_lru *lru, int nid, + struct mem_cgroup *memcg); + +/** + * list_lru_unlock: unlock a sublist locked by list_lru_lock() + * @l: the list_lru_one to unlock + */ +void list_lru_unlock(struct list_lru_one *l); + +struct list_lru_one *list_lru_lock_irq(struct list_lru *lru, int nid, + struct mem_cgroup *memcg); +void list_lru_unlock_irq(struct list_lru_one *l); + +struct list_lru_one *list_lru_lock_irqsave(struct list_lru *lru, int nid, + struct mem_cgroup *memcg, unsigned long *irq_flags); +void list_lru_unlock_irqrestore(struct list_lru_one *l, + unsigned long *irq_flags); + +/* Caller-locked variants, see list_lru_add() etc for documentation */ +bool __list_lru_add(struct list_lru *lru, struct list_lru_one *l, + struct list_head *item, int nid, struct mem_cgroup *memcg); +bool __list_lru_del(struct list_lru *lru, struct list_lru_one *l, + struct list_head *item, int nid); + /** * list_lru_add: add an element to the lru list's tail * @lru: the lru pointer @@ -115,6 +153,9 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg,= struct mem_cgroup *paren bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid, struct mem_cgroup *memcg); =20 +bool list_lru_add_irq(struct list_lru *lru, struct list_head *item, int ni= d, + struct mem_cgroup *memcg); + /** * list_lru_add_obj: add an element to the lru list's tail * @lru: the lru pointer diff --git a/mm/list_lru.c b/mm/list_lru.c index 65962dbf6dda..df58226eea8c 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -15,17 +15,23 @@ #include "slab.h" #include "internal.h" =20 -static inline void lock_list_lru(struct list_lru_one *l, bool irq) +static inline void lock_list_lru(struct list_lru_one *l, bool irq, + unsigned long *irq_flags) { - if (irq) + if (irq_flags) + spin_lock_irqsave(&l->lock, *irq_flags); + else if (irq) spin_lock_irq(&l->lock); else spin_lock(&l->lock); } =20 -static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off) +static inline void unlock_list_lru(struct list_lru_one *l, bool irq_off, + unsigned long *irq_flags) { - if (irq_off) + if (irq_flags) + spin_unlock_irqrestore(&l->lock, *irq_flags); + else if (irq_off) spin_unlock_irq(&l->lock); else spin_unlock(&l->lock); @@ -78,7 +84,7 @@ list_lru_from_memcg_idx(struct list_lru *lru, int nid, in= t idx) =20 static inline struct list_lru_one * lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *m= emcg, - bool irq, bool skip_empty) + bool irq, unsigned long *irq_flags, bool skip_empty) { struct list_lru_one *l; =20 @@ -86,12 +92,12 @@ lock_list_lru_of_memcg(struct list_lru *lru, int nid, s= truct mem_cgroup *memcg, again: l =3D list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); if (likely(l)) { - lock_list_lru(l, irq); + lock_list_lru(l, irq, irq_flags); if (likely(READ_ONCE(l->nr_items) !=3D LONG_MIN)) { rcu_read_unlock(); return l; } - unlock_list_lru(l, irq); + unlock_list_lru(l, irq, irq_flags); } /* * Caller may simply bail out if raced with reparenting or @@ -132,38 +138,106 @@ list_lru_from_memcg_idx(struct list_lru *lru, int ni= d, int idx) =20 static inline struct list_lru_one * lock_list_lru_of_memcg(struct list_lru *lru, int nid, struct mem_cgroup *m= emcg, - bool irq, bool skip_empty) + bool irq, unsigned long *irq_flags, bool skip_empty) { struct list_lru_one *l =3D &lru->node[nid].lru; =20 - lock_list_lru(l, irq); + lock_list_lru(l, irq, irq_flags); =20 return l; } #endif /* CONFIG_MEMCG */ =20 -/* The caller must ensure the memcg lifetime. */ -bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid, - struct mem_cgroup *memcg) +struct list_lru_one *list_lru_lock(struct list_lru *lru, int nid, + struct mem_cgroup *memcg) { - struct list_lru_node *nlru =3D &lru->node[nid]; - struct list_lru_one *l; + return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=3D*/false, + /*irq_flags=3D*/NULL, /*skip_empty=3D*/false); +} + +void list_lru_unlock(struct list_lru_one *l) +{ + unlock_list_lru(l, /*irq_off=3D*/false, /*irq_flags=3D*/NULL); +} + +struct list_lru_one *list_lru_lock_irq(struct list_lru *lru, int nid, + struct mem_cgroup *memcg) +{ + return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=3D*/true, + /*irq_flags=3D*/NULL, /*skip_empty=3D*/false); +} + +void list_lru_unlock_irq(struct list_lru_one *l) +{ + unlock_list_lru(l, /*irq_off=3D*/true, /*irq_flags=3D*/NULL); +} =20 - l =3D lock_list_lru_of_memcg(lru, nid, memcg, false, false); +struct list_lru_one *list_lru_lock_irqsave(struct list_lru *lru, int nid, + struct mem_cgroup *memcg, + unsigned long *flags) +{ + return lock_list_lru_of_memcg(lru, nid, memcg, /*irq=3D*/true, + /*irq_flags=3D*/flags, /*skip_empty=3D*/false); +} + +void list_lru_unlock_irqrestore(struct list_lru_one *l, unsigned long *fla= gs) +{ + unlock_list_lru(l, /*irq_off=3D*/true, /*irq_flags=3D*/flags); +} + +bool __list_lru_add(struct list_lru *lru, struct list_lru_one *l, + struct list_head *item, int nid, + struct mem_cgroup *memcg) +{ if (list_empty(item)) { list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ if (!l->nr_items++) set_shrinker_bit(memcg, nid, lru_shrinker_id(lru)); - unlock_list_lru(l, false); - atomic_long_inc(&nlru->nr_items); + atomic_long_inc(&lru->node[nid].nr_items); return true; } - unlock_list_lru(l, false); return false; } EXPORT_SYMBOL_GPL(list_lru_add); =20 +bool __list_lru_del(struct list_lru *lru, struct list_lru_one *l, + struct list_head *item, int nid) +{ + if (!list_empty(item)) { + list_del_init(item); + l->nr_items--; + atomic_long_dec(&lru->node[nid].nr_items); + return true; + } + return false; +} + +/* The caller must ensure the memcg lifetime. */ +bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid, + struct mem_cgroup *memcg) +{ + struct list_lru_one *l; + bool ret; + + l =3D list_lru_lock(lru, nid, memcg); + ret =3D __list_lru_add(lru, l, item, nid, memcg); + list_lru_unlock(l); + return ret; +} + +bool list_lru_add_irq(struct list_lru *lru, struct list_head *item, + int nid, struct mem_cgroup *memcg) +{ + struct list_lru_one *l; + bool ret; + + l =3D list_lru_lock_irq(lru, nid, memcg); + ret =3D __list_lru_add(lru, l, item, nid, memcg); + list_lru_unlock_irq(l); + return ret; +} + bool list_lru_add_obj(struct list_lru *lru, struct list_head *item) { bool ret; @@ -185,19 +259,13 @@ EXPORT_SYMBOL_GPL(list_lru_add_obj); bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid, struct mem_cgroup *memcg) { - struct list_lru_node *nlru =3D &lru->node[nid]; struct list_lru_one *l; + bool ret; =20 - l =3D lock_list_lru_of_memcg(lru, nid, memcg, false, false); - if (!list_empty(item)) { - list_del_init(item); - l->nr_items--; - unlock_list_lru(l, false); - atomic_long_dec(&nlru->nr_items); - return true; - } - unlock_list_lru(l, false); - return false; + l =3D list_lru_lock(lru, nid, memcg); + ret =3D __list_lru_del(lru, l, item, nid); + list_lru_unlock(l); + return ret; } =20 bool list_lru_del_obj(struct list_lru *lru, struct list_head *item) @@ -270,7 +338,8 @@ __list_lru_walk_one(struct list_lru *lru, int nid, stru= ct mem_cgroup *memcg, unsigned long isolated =3D 0; =20 restart: - l =3D lock_list_lru_of_memcg(lru, nid, memcg, irq_off, true); + l =3D lock_list_lru_of_memcg(lru, nid, memcg, /*irq=3D*/irq_off, + /*irq_flags=3D*/NULL, /*skip_empty=3D*/true); if (!l) return isolated; list_for_each_safe(item, n, &l->list) { @@ -311,7 +380,7 @@ __list_lru_walk_one(struct list_lru *lru, int nid, stru= ct mem_cgroup *memcg, BUG(); } } - unlock_list_lru(l, irq_off); + unlock_list_lru(l, irq_off, NULL); out: return isolated; } --=20 2.54.0 From nobody Sun May 24 20:35:35 2026 Received: from mail-qv1-f44.google.com (mail-qv1-f44.google.com [209.85.219.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 34FAB36F8FE for ; Thu, 21 May 2026 15:03:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779375836; cv=none; b=aat34tcqQ9OVmLGi+cO+HMGmQmmMsHOeBZYdtJSRH+KfvkbbYcP+6hDeNvQzzxW5iM7oix9TaBc2C5p3QSrUMhXJwu/t4+O4wOytuq72Yd+e+0jnywELgK5+dhqymKkYECLzPZgZXKiuV5R9/yK7JdPlsMKmpJssTeFQXpEaWXM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779375836; c=relaxed/simple; bh=XDHwiXK7DhsPrkpYFuamx5LXb9zFWccd6v2gAdweAKE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=VufUHxjzNPwGMrOOqy3EGlGDE79ZZTr/2vsMw2K2n2DLNwgCwWfj6SHj/SYLlu6DiC8gKYxcP8t0bXhEXlXiW9TLiDzEedqa/NiG5Vy7oYr4mpa0JIEHB82sn7nkmnpMtJcJ9eVYcNN8qZsLc8MVcr4xAOuA1WDWdSfb1PUGmAM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=tU/jJ+b3; arc=none smtp.client-ip=209.85.219.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="tU/jJ+b3" Received: by mail-qv1-f44.google.com with SMTP id 6a1803df08f44-8b74b460d77so81635836d6.3 for ; Thu, 21 May 2026 08:03:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1779375834; x=1779980634; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rR/PuWQM6I0Sn5LudGZTwz22IXjqotRj7EvDPtJsrZA=; b=tU/jJ+b3fBUHI05Ny9Rarn81/d3LqESIs0nKlG4J1eEJheSlCQtFCP/32+74qqqEtI bAdTQ++1D4trnZNETWRTCBHKF53dmE9Tyvsd96vybtCVHR2JQQRegJe7wvcKO/7IcKzq 1FdlSwn2Hle05Fe9IlqugXYvLCJUUiinjbqjw1z8PcSwbroD53IhGY00Yr51xftalFC0 2NKvSdt13c5hasmceAlg26aaIRKe4AzRI/CEhBcAMmrjkfVE+EdfuiVp37UhR6LjzfEn puQRixSBwQ2yiNxRWMiDFJ8e+HG6Op9aZa39+k5t6yoRzbHL36MLau3xZbd0zS/h/pyh ZzwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779375834; x=1779980634; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=rR/PuWQM6I0Sn5LudGZTwz22IXjqotRj7EvDPtJsrZA=; b=XG6MoEkTH2HboOmuUftGVDHqMc17q2FDgd5N+6bH0E6ZxP6AflIpFR9q1hm8UJGRVB R5UEIY1277DC+3XO4pqeXUQGal9NIUBQuaKFtU8lq05H6O+yc+G/+XfsxehcpnJhkd9S zqf2F6XG+M6ej6VXOzIcNiC5g/AOf9hO6TFfUc1/esK+uazGlbZYpGOraXGC+Kf5KjID 5GTrCqY2aPsRhlmyc4nvJD3XOZgYrcmMnNDUFGZ6V02C25Upx/i2tqW7zj1jEXZSLMRQ 8V3DZmKSwZDKHf3wtpw6o4RQQfO88jn1bVUYOgb6SH0CGnF+z8XlL4t5IYmi4oqBmY1Y bHqw== X-Forwarded-Encrypted: i=1; AFNElJ8dKxAhzU35RztFrfWeK4B1EfIq7+9pMlbD4Sto5T7DaMkvuc2z08C+e0z2soYx0863PEr8Ocgt1GlOdn4=@vger.kernel.org X-Gm-Message-State: AOJu0YzgnfG+C/ScDsHrX/L5jI54nn3Upzd7sYC5mqIDoLl+2jKpaWpm cw74ECgybt0j77wW72C+UrORLHmOX4kKzXSafGaOhNyrL5v6s4bqRudTqs4035QNOqQ= X-Gm-Gg: Acq92OGL9bh4wCKnUwzLcAAhjfHVTqlv0b2S1pWvf2afrjxO8wWD5WaUJhG5EnH8YgL EeNPPerxbMdUf1OHwddUVxBpKOiX02xlNwyxgiosL3L0LWoXVC/LfwIR8Vv6DPd42s65DMuxStd xolV0XTSzgC22Kl8jVePIuaT+d5+pksswXKRIbo3jo1whWlMLG4x5JZzxCELTCSq9E1n4TYyMuv 8GbWAS8fzjj7Dztm7KoZGY7A/qexROyDnXAF+BGPC87dPK0VsyEbuSQVvlageXspdZ4mkUuzxz8 4js4oU3gfsMEfrAI+e5wO7PXXeCl4PA0OMYeOGZjFbTG7gYw7ftGulY3EmzSrwKzxWZQqfSoz5V QHFHWwfsS+jOQqtuytXh81u9dOciAKYEeh/FzC99DOHS+z7VBe1mMmjDC3hndxFi4Cokh7lfcAa Zr9P/KWpbMJBPmLqzmXNTa6GZZel7d3R06 X-Received: by 2002:ac8:5e0f:0:b0:50f:b3d2:6ee1 with SMTP id d75a77b69052e-516c555a206mr43518571cf.31.1779375834075; Thu, 21 May 2026 08:03:54 -0700 (PDT) Received: from localhost ([2603:7001:f100:500:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-516ccba5ffdsm9168671cf.5.2026.05.21.08.03.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 May 2026 08:03:52 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: David Hildenbrand , Lorenzo Stoakes , Shakeel Butt , Michal Hocko , Dave Chinner , Roman Gushchin , Muchun Song , Qi Zheng , Yosry Ahmed , Zi Yan , "Liam R . Howlett" , Usama Arif , Kiryl Shutsemau , Vlastimil Babka , Kairui Song , Mikhail Zaslonko , Vasily Gorbik , Baolin Wang , Barry Song , Dev Jain , Lance Yang , Nico Pache , Ryan Roberts , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 6/8] mm: list_lru: introduce folio_memcg_list_lru_alloc() Date: Thu, 21 May 2026 11:02:12 -0400 Message-ID: <20260521150330.1955924-7-hannes@cmpxchg.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260521150330.1955924-1-hannes@cmpxchg.org> References: <20260521150330.1955924-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" memcg_list_lru_alloc() is called every time an object that may end up on the list_lru is created. It needs to quickly check if the list_lru heads for the memcg already exist, and allocate them when they don't. Doing this with folio objects is tricky: folio_memcg() is not stable and requires either RCU protection or pinning the cgroup. But it's desirable to make the existence check lightweight under RCU, and only pin the memcg when we need to allocate list_lru heads and may block. In preparation for switching the THP shrinker to list_lru, add a helper function for allocating list_lru heads coming from a folio. Reviewed-by: David Hildenbrand (Arm) Acked-by: Shakeel Butt Reviewed-by: Lorenzo Stoakes (Oracle) Signed-off-by: Johannes Weiner --- include/linux/list_lru.h | 27 +++++++++++++++++++++++++++ mm/list_lru.c | 39 ++++++++++++++++++++++++++++++++++----- 2 files changed, 61 insertions(+), 5 deletions(-) diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index c79ed378311f..733a262b91e5 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -81,6 +81,33 @@ static inline int list_lru_init_memcg_key(struct list_lr= u *lru, struct shrinker =20 int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, gfp_t gfp); + +#ifdef CONFIG_MEMCG +/** + * folio_memcg_list_lru_alloc - allocate list_lru heads for shrinkable fol= io + * @folio: the newly allocated & charged folio + * @lru: the list_lru this might be queued on + * @gfp: gfp mask + * + * Allocate list_lru heads (per-memcg, per-node) needed to queue this + * particular folio down the line. + * + * This does memcg_list_lru_alloc(), but on the memcg that @folio is + * associated with. Handles folio_memcg() access rules in the fast + * path (list_lru heads allocated) and the allocation slowpath. + * + * Returns 0 on success, a negative error value otherwise. + */ +int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru, + gfp_t gfp); +#else +static inline int folio_memcg_list_lru_alloc(struct folio *folio, + struct list_lru *lru, gfp_t gfp) +{ + return 0; +} +#endif + void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup = *parent); =20 /** diff --git a/mm/list_lru.c b/mm/list_lru.c index df58226eea8c..4eeb4351c475 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -562,17 +562,14 @@ static inline bool memcg_list_lru_allocated(struct me= m_cgroup *memcg, return idx < 0 || xa_load(&lru->xa, idx); } =20 -int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, - gfp_t gfp) +static int __memcg_list_lru_alloc(struct mem_cgroup *memcg, + struct list_lru *lru, gfp_t gfp) { unsigned long flags; struct list_lru_memcg *mlru =3D NULL; struct mem_cgroup *pos, *parent; XA_STATE(xas, &lru->xa, 0); =20 - if (!list_lru_memcg_aware(lru) || memcg_list_lru_allocated(memcg, lru)) - return 0; - gfp &=3D GFP_RECLAIM_MASK; /* * Because the list_lru can be reparented to the parent cgroup's @@ -613,6 +610,38 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, str= uct list_lru *lru, =20 return xas_error(&xas); } + +int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, + gfp_t gfp) +{ + if (!list_lru_memcg_aware(lru) || memcg_list_lru_allocated(memcg, lru)) + return 0; + return __memcg_list_lru_alloc(memcg, lru, gfp); +} + +int folio_memcg_list_lru_alloc(struct folio *folio, struct list_lru *lru, + gfp_t gfp) +{ + struct mem_cgroup *memcg; + int res; + + if (!list_lru_memcg_aware(lru)) + return 0; + + /* Fast path when list_lru heads already exist */ + rcu_read_lock(); + memcg =3D folio_memcg(folio); + res =3D memcg_list_lru_allocated(memcg, lru); + rcu_read_unlock(); + if (likely(res)) + return 0; + + /* Allocation may block, pin the memcg */ + memcg =3D get_mem_cgroup_from_folio(folio); + res =3D __memcg_list_lru_alloc(memcg, lru, gfp); + mem_cgroup_put(memcg); + return res; +} #else static inline void memcg_init_list_lru(struct list_lru *lru, bool memcg_aw= are) { --=20 2.54.0 From nobody Sun May 24 20:35:35 2026 Received: from mail-qk1-f182.google.com (mail-qk1-f182.google.com [209.85.222.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 25020379C47 for ; Thu, 21 May 2026 15:03:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779375838; cv=none; b=bF1goqkDBAfPXtaXq2BxSBL3Jvz/4oWBqpMUWW0ixnIi1QxFi8vNWrr3fPSV47qq7dYGWJv/JIAHbH1r8VzMbQT1knXKpsRiz3Zd3G+OLVrDvJZIZBQCDBgvbCG18/sUIT4Xfi5ffX5hlwV33LgyNpyCzDcCPEdHlUtA5OyfCuI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779375838; c=relaxed/simple; bh=71dbN/lwsz2RYbZhDRLT9fuF1mdojGIG9wIFF+AZShE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WWloYk8yt152d0rfZcZZuCyTys4wwRq11TNSFTZ4Cl33LHk6EW2GykfIDgvX4HH38iBZavLcvzJXuxl8qaObpXEgszmUQov0Y1bSkr7d5XX+XbLEFlha59z7kKWF/x4QvCJs5D5VfpMYUCn+/znD/pBGvyXtowCXfPNBiascgRA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=TVfi7btK; arc=none smtp.client-ip=209.85.222.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="TVfi7btK" Received: by mail-qk1-f182.google.com with SMTP id af79cd13be357-9144163319fso576175285a.2 for ; Thu, 21 May 2026 08:03:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1779375836; x=1779980636; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VJJEuZOvug6rcak+Osyt/tu7+OHIrKr4x7Xajjb2SaI=; b=TVfi7btKfs52Ud8FsysSxZtMH1F5OjgJIxCHty5Taq8U6zqvCxwhhJGjfd4rZp99iJ YvHwhwX45E03r1y92lMLy1Ea79jxYvIhvbGdMqxkqAowfW6fY8Dnk++zefxBAF2jtHcb OZlLnOn4nxAK6UQ7VnWWrZGuV02Lx5xNcpwsWEGY4YBezdtJH8aGhj2EXo/uM+nKgrVu WTCNLFRr0AdeOrLOCpIG8oMjFyOJvSWnwhDBDKrSc6K3e+fsdfrcA9ExSrXtnI9zL3Eh nmU3C+QxETIMewceTKi3fOZRC2kGLteTY8xJvhmOO1hP09saDG6XHlkkBBLMZXs94eRF HlQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779375836; x=1779980636; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=VJJEuZOvug6rcak+Osyt/tu7+OHIrKr4x7Xajjb2SaI=; b=D2SPs9w+9qga3OQndUSApUFCrecaWIJjaLrM4SG+tJbbaYzT5IXTfoxm/NQZpwDnLc 4ipdrKSDahslti2xc0W0D+cgRQ/aNuJfUeNawBtFOf+9onsRPrvU0t+dbaetOLv1y3yA Ft5YRE3P+c7oByFb4IWr5IYyNTao4dsM3jtRfaZ4FBLxSKuhwCYJLYKcloYfJoxQ3iYW 7lyt2DM55+Qxnq32hwyi0ledCs8ilMZOusumSs2m4CTkBZyjulfbnDH/16ryQ59HZwjk JFQWZ7LVmTi9efsEKRrgUQiNFbF84FT82g/hcVqD6/l+UfKJOwVq/eJjEYFDYtuVsNuQ vYYg== X-Forwarded-Encrypted: i=1; AFNElJ9+KeSUH9TvKmdcYUvA1c7uyN5l5ad1+WCggMHTP+SzcfB70NqM8vwz9c8QfdUZcbZ8yLhDUmxovC8M1+o=@vger.kernel.org X-Gm-Message-State: AOJu0YxBHO5Dk1iR8vbhWgFkh5wECo8HOEZnHXWOhrWF+JKMM2bOJy7N 14aOedl+Zr4eXwQA5JzPkWy2UReA7SYxNB2B+t5yCwpW1f1xeOS1aGUIg0GGqDRCXhY= X-Gm-Gg: Acq92OEIxnqSlTn4nrfYAxaIomF0I/GbXRZ2wy7p3H+VtMBwI5khiZ/16esE1u2EieP BYCLiwkGLQrX06tcm4sHZRidroaodVFpBWgrBvADn2rd1fdHvLf8TYpVdmDrzqq1ATOUmTf6J0k QOZozeUFunpxd80iSHOGR0xcBJUvzNcdZnVVr58N1NLxjIJ0QPioW6kPEpB9bxWv0pdvAyXSHMw 6jxFa5RA5R8sm+c8p8P4Wn3dd8YQ305BtM9u7rOza7VwTi9DFW23fKCEZv8kRHIXaNyWfpjcTSi pNvVTQpCIz6U3PRMyoeROFWwin14v6nAliQ7mMZHUtqoIP7Ny2tLKdyT58OVfJa58p/49zp5bjh QR/Lh4c6OTrcs9YaqvfeK18fRil+cFHmms9TiyWDKCtJ56RK8xit6iZ//pQlVsIo8ALX6BGt4FJ qSCnv4mSGC1MdRysq4Wgptyw== X-Received: by 2002:a05:622a:608f:b0:50d:860a:8fb0 with SMTP id d75a77b69052e-516c54c70e8mr41818231cf.21.1779375835903; Thu, 21 May 2026 08:03:55 -0700 (PDT) Received: from localhost ([2603:7001:f100:500:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-516ccba5ffdsm9169611cf.5.2026.05.21.08.03.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 May 2026 08:03:54 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: David Hildenbrand , Lorenzo Stoakes , Shakeel Butt , Michal Hocko , Dave Chinner , Roman Gushchin , Muchun Song , Qi Zheng , Yosry Ahmed , Zi Yan , "Liam R . Howlett" , Usama Arif , Kiryl Shutsemau , Vlastimil Babka , Kairui Song , Mikhail Zaslonko , Vasily Gorbik , Baolin Wang , Barry Song , Dev Jain , Lance Yang , Nico Pache , Ryan Roberts , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 7/8] mm/memory: flatten folio allocation retry loops Date: Thu, 21 May 2026 11:02:13 -0400 Message-ID: <20260521150330.1955924-8-hannes@cmpxchg.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260521150330.1955924-1-hannes@cmpxchg.org> References: <20260521150330.1955924-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" alloc_swap_folio() and alloc_anon_folio() use a top-level if (folio) that buries the success path four levels deep. This makes for awkward long lines and wrapping. The next patch will add more code here, so flatten this now to keep things clean and simple. alloc_anon_folio() already has a next label, use it for !folio. Add the equivalent to alloc_swap_folio(). No functional change intended. Suggested-by: Lorenzo Stoakes (Oracle) Signed-off-by: Johannes Weiner --- mm/memory.c | 44 +++++++++++++++++++++++--------------------- 1 file changed, 23 insertions(+), 21 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 0c9d9c2cbf0e..552fe26a042a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4752,13 +4752,15 @@ static struct folio *alloc_swap_folio(struct vm_fau= lt *vmf) while (orders) { addr =3D ALIGN_DOWN(vmf->address, PAGE_SIZE << order); folio =3D vma_alloc_folio(gfp, order, vma, addr); - if (folio) { - if (!mem_cgroup_swapin_charge_folio(folio, vma->vm_mm, - gfp, entry)) - return folio; + if (!folio) + goto next; + if (mem_cgroup_swapin_charge_folio(folio, vma->vm_mm, gfp, entry)) { count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK_CHARGE); folio_put(folio); + goto next; } + return folio; +next: count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK); order =3D next_order(&orders, order); } @@ -5270,24 +5272,24 @@ static struct folio *alloc_anon_folio(struct vm_fau= lt *vmf) while (orders) { addr =3D ALIGN_DOWN(vmf->address, PAGE_SIZE << order); folio =3D vma_alloc_folio(gfp, order, vma, addr); - if (folio) { - if (mem_cgroup_charge(folio, vma->vm_mm, gfp)) { - count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); - folio_put(folio); - goto next; - } - folio_throttle_swaprate(folio, gfp); - /* - * When a folio is not zeroed during allocation - * (__GFP_ZERO not used) or user folios require special - * handling, folio_zero_user() is used to make sure - * that the page corresponding to the faulting address - * will be hot in the cache after zeroing. - */ - if (user_alloc_needs_zeroing()) - folio_zero_user(folio, vmf->address); - return folio; + if (!folio) + goto next; + if (mem_cgroup_charge(folio, vma->vm_mm, gfp)) { + count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); + folio_put(folio); + goto next; } + folio_throttle_swaprate(folio, gfp); + /* + * When a folio is not zeroed during allocation + * (__GFP_ZERO not used) or user folios require special + * handling, folio_zero_user() is used to make sure + * that the page corresponding to the faulting address + * will be hot in the cache after zeroing. + */ + if (user_alloc_needs_zeroing()) + folio_zero_user(folio, vmf->address); + return folio; next: count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK); order =3D next_order(&orders, order); --=20 2.54.0 From nobody Sun May 24 20:35:35 2026 Received: from mail-qv1-f50.google.com (mail-qv1-f50.google.com [209.85.219.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6013D37F00E for ; Thu, 21 May 2026 15:03:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779375842; cv=none; b=NHoHjJPCrxRKE3g0QgnQGZNvAHb+uixnTDPdv2cl8M5E4bROt9ReCjzb19K0nKO+Nb84Bxxbfl647nVGgWNgL9RUYEJJ7XguW9P5k9WM77i1G4xVyE5tWNC5VuSg9XkyFQq+12wNjPxuLGeJRAFZP3txrRkYz55+x19sbo1qOIo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779375842; c=relaxed/simple; bh=Nmeq3RhVZYsL7P4s7GZ0oJJ7Ne/2HWLOnqCA4ajgemA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=fScsxWlswJgh6zvDKRx3e9PQzmFzMuHjd+wIRLUWZ5yKwr6gggFwReaRBUyqu68SjzH/NzqyZ6E0R6WnhZ82NoRHWqxXcypPEt8eQtXUdMBut/zzUVaL7cjBvJZDTEV9NEDFnuS/Skl/ZFLS0tPFsjVsRu6oTBdTkYLvnhiIeX0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=qf9hvNoU; arc=none smtp.client-ip=209.85.219.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="qf9hvNoU" Received: by mail-qv1-f50.google.com with SMTP id 6a1803df08f44-8b74b460d77so81636666d6.3 for ; Thu, 21 May 2026 08:03:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1779375838; x=1779980638; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=67fTJ2GsUTWnFBVp+cp93tYrnY33hKOC/aTP0xtFSh8=; b=qf9hvNoUtyjs4WfikNx1+2x6ltpuJxtdAnz/2X49OoCtLRTZlwoVfm+PvMYiAavls5 VADwzf68KYxUVS4zWCouuYdVWQYrD7Wu0sB0C5lm8zNNU7jvJRcD1r5clyS2CN6cOHKt cK8V+LUJ78QTEMw/E8uP/cZJnXJR/glEQTgutqpoCUdgkpWDdaXoRLG61Un7mDDgis1A LRa59aKPwUXgCKPocrJ4r4lZHuF8I70EqQVl1J+VNwa+84wlwMEhcw+hqrVAiIotUFYP p0+93v2A/PN4ZEC9nulJtj6XKe9Qj3JOP7pOeMZz8+myD/VR4ShNaWWOGVTSpOIj3PGl +I4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779375838; x=1779980638; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=67fTJ2GsUTWnFBVp+cp93tYrnY33hKOC/aTP0xtFSh8=; b=p3LM5VHt9lg7q2CBg7nA0YCN/jrObCVsrkS805J9E1F5Y2rGZ+Ml/pxE+OsTiTi2UR vLoxkIs4/i2GkwROIl5EBGJ8H/kib4SvpqNesAiAl4tUcuimB4LtcIr7y9kP9iYenJoK WaPameFPuEgAmmPC236iWzwszm0C4gKs/lqAGlymzSq0KQBi7SrcA2EIFFErKC3AVG9o /iNqjaSMKXFVg2Z/VR1wNjKFycZfpEzZ2H8Vl/qscc9dFCwIafdwD8rffH61vJJpHWHj aVNpWnjw5oGzNPGdbG+E7GhV6a0g/hl7S0p9gqDs/aKe3MLt9ejE3Dr2nEETQt053Wiu KU3Q== X-Forwarded-Encrypted: i=1; AFNElJ/UC+qEHhJm2bmH2H7M9DDHNuJi5DEbQtHpvZOGks0Y2sYjzK5dgOsrZxCqN3NOlaO3eWfjAoM7H34SEp4=@vger.kernel.org X-Gm-Message-State: AOJu0YyriXOqJ8ssG38VdjpLtfZ0pIE2x+Clf0DSaoaFycDQIXPUYjHk KzXJZPNJb9O2Pe2KiSCskES7Ab/45YgPqTgOdE0CmodY8dN67T222MrzUuDI9OcvV0M= X-Gm-Gg: Acq92OFgfgbke1FYzggB7GmWuwLhPLsJjxH+tT73MldR+LrdsVn/XqT2+ViYaGqFKG6 Vzvhk+pr7mrj5MxpDjYXDV9upzegZ8GDrmr7Pap6NXxxog8ohTt+2POpPGc93j9votUUQtS9ebk zJgUecvWhX6289rm3g9kx2Ew9CKmivi41TwdJO8M5a2wNiY0FzQS8xt6V3H3uD3NXY60CfAL+Em XG/HWtvgj31hkN+MyyczQ26IzrVydkwVmVED0954sYJ+DKakySJvvjOacmYW+xIOLWTQ/v0xQBB am1xHyyxmVc4qXGkAb0wsUKBnkHQ1+d1ui/ZLDex0Pgexh+eucZMOCt/0Frup5XMqA/qajfwUtp M6DIXlkMPOmehypmbkmiCK/9FmFVzDlP9sw9rKIEJOrvMA8LDfFc/M06WLus7zRbyQcnVlgP7Hj ND0Ou+ikZy4sU2RHwtSOiS0S5SiwgG2ECeUYY0ox02eCI= X-Received: by 2002:a05:6214:1c82:b0:8ae:6d63:3066 with SMTP id 6a1803df08f44-8cc6e386740mr51339906d6.38.1779375837977; Thu, 21 May 2026 08:03:57 -0700 (PDT) Received: from localhost ([2603:7001:f100:500:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-8cc781b3611sm7627886d6.26.2026.05.21.08.03.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 May 2026 08:03:56 -0700 (PDT) From: Johannes Weiner To: Andrew Morton Cc: David Hildenbrand , Lorenzo Stoakes , Shakeel Butt , Michal Hocko , Dave Chinner , Roman Gushchin , Muchun Song , Qi Zheng , Yosry Ahmed , Zi Yan , "Liam R . Howlett" , Usama Arif , Kiryl Shutsemau , Vlastimil Babka , Kairui Song , Mikhail Zaslonko , Vasily Gorbik , Baolin Wang , Barry Song , Dev Jain , Lance Yang , Nico Pache , Ryan Roberts , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 8/8] mm: switch deferred split shrinker to list_lru Date: Thu, 21 May 2026 11:02:14 -0400 Message-ID: <20260521150330.1955924-9-hannes@cmpxchg.org> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260521150330.1955924-1-hannes@cmpxchg.org> References: <20260521150330.1955924-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The deferred split queue handles cgroups in a suboptimal fashion. The queue is per-NUMA node or per-cgroup, not the intersection. That means on a cgrouped system, a node-restricted allocation entering reclaim can end up splitting large pages on other nodes: alloc/unmap deferred_split_folio() list_add_tail(memcg->split_queue) set_shrinker_bit(memcg, node, deferred_shrinker_id) for_each_zone_zonelist_nodemask(restricted_nodes) mem_cgroup_iter() shrink_slab(node, memcg) shrink_slab_memcg(node, memcg) if test_shrinker_bit(memcg, node, deferred_shrinker_id) deferred_split_scan() walks memcg->split_queue The shrinker bit adds an imperfect guard rail. As soon as the cgroup has a single large page on the node of interest, all large pages owned by that memcg, including those on other nodes, will be split. list_lru properly sets up per-node, per-cgroup lists. As a bonus, it streamlines a lot of the list operations and reclaim walks. It's used widely by other major shrinkers already. Convert the deferred split queue as well. The list_lru per-memcg heads are instantiated on demand when the first object of interest is allocated for a cgroup, by calling folio_memcg_alloc_deferred(). Add calls to where splittable pages are created: anon faults, swapin faults, khugepaged collapse. These calls create all possible node heads for the cgroup at once, so the migration code (between nodes) doesn't need any special care. Reported-by: Mikhail Zaslonko Tested-by: Mikhail Zaslonko Acked-by: Shakeel Butt Reviewed-by: Lorenzo Stoakes (Oracle) Signed-off-by: Johannes Weiner --- include/linux/huge_mm.h | 7 +- include/linux/memcontrol.h | 4 - include/linux/mmzone.h | 12 -- mm/huge_memory.c | 355 ++++++++++++------------------------- mm/internal.h | 2 +- mm/khugepaged.c | 3 + mm/memcontrol.c | 12 +- mm/memory.c | 8 + mm/mm_init.c | 15 -- 9 files changed, 133 insertions(+), 285 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 127f9e1e7604..dc939873f5df 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -398,10 +398,10 @@ static inline int split_huge_page(struct page *page) { return split_huge_page_to_list_to_order(page, NULL, 0); } + +int folio_memcg_alloc_deferred(struct folio *folio); + void deferred_split_folio(struct folio *folio, bool partially_mapped); -#ifdef CONFIG_MEMCG -void reparent_deferred_split_queue(struct mem_cgroup *memcg); -#endif =20 void __split_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd, unsigned long address, bool freeze); @@ -634,7 +634,6 @@ static inline int folio_split(struct folio *folio, unsi= gned int new_order, } =20 static inline void deferred_split_folio(struct folio *folio, bool partiall= y_mapped) {} -static inline void reparent_deferred_split_queue(struct mem_cgroup *memcg)= {} #define split_huge_pmd(__vma, __pmd, __address) \ do { } while (0) =20 diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index dc3fa687759b..4a7d8c4f55b4 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -277,10 +277,6 @@ struct mem_cgroup { struct memcg_cgwb_frn cgwb_frn[MEMCG_CGWB_FRN_CNT]; #endif =20 -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - struct deferred_split deferred_split_queue; -#endif - #ifdef CONFIG_LRU_GEN_WALKS_MMU /* per-memcg mm_struct list */ struct lru_gen_mm_list mm_list; diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 1331a7b93f33..8e449f524f26 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1431,14 +1431,6 @@ struct zonelist { */ extern struct page *mem_map; =20 -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -struct deferred_split { - spinlock_t split_queue_lock; - struct list_head split_queue; - unsigned long split_queue_len; -}; -#endif - #ifdef CONFIG_MEMORY_FAILURE /* * Per NUMA node memory failure handling statistics. @@ -1564,10 +1556,6 @@ typedef struct pglist_data { unsigned long first_deferred_pfn; #endif /* CONFIG_DEFERRED_STRUCT_PAGE_INIT */ =20 -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - struct deferred_split deferred_split_queue; -#endif - #ifdef CONFIG_NUMA_BALANCING /* start time in ms of current promote rate limit period */ unsigned int nbp_rl_start; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c565b2a651e0..67be09a58d5a 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -67,6 +68,8 @@ unsigned long transparent_hugepage_flags __read_mostly = =3D (1<count_objects =3D deferred_split_count; deferred_split_shrinker->scan_objects =3D deferred_split_scan; shrinker_register(deferred_split_shrinker); @@ -973,6 +990,7 @@ static int __init thp_shrinker_init(void) huge_zero_folio_shrinker =3D shrinker_alloc(0, "thp-zero"); if (!huge_zero_folio_shrinker) { shrinker_free(deferred_split_shrinker); + list_lru_destroy(&deferred_split_lru); return -ENOMEM; } =20 @@ -987,6 +1005,7 @@ static void __init thp_shrinker_exit(void) { shrinker_free(huge_zero_folio_shrinker); shrinker_free(deferred_split_shrinker); + list_lru_destroy(&deferred_split_lru); } =20 static int __init hugepage_init(void) @@ -1166,119 +1185,6 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_s= truct *vma) return pmd; } =20 -static struct deferred_split *split_queue_node(int nid) -{ - struct pglist_data *pgdata =3D NODE_DATA(nid); - - return &pgdata->deferred_split_queue; -} - -#ifdef CONFIG_MEMCG -static inline -struct mem_cgroup *folio_split_queue_memcg(struct folio *folio, - struct deferred_split *queue) -{ - if (mem_cgroup_disabled()) - return NULL; - if (split_queue_node(folio_nid(folio)) =3D=3D queue) - return NULL; - return container_of(queue, struct mem_cgroup, deferred_split_queue); -} - -static struct deferred_split *memcg_split_queue(int nid, struct mem_cgroup= *memcg) -{ - return memcg ? &memcg->deferred_split_queue : split_queue_node(nid); -} -#else -static inline -struct mem_cgroup *folio_split_queue_memcg(struct folio *folio, - struct deferred_split *queue) -{ - return NULL; -} - -static struct deferred_split *memcg_split_queue(int nid, struct mem_cgroup= *memcg) -{ - return split_queue_node(nid); -} -#endif - -static struct deferred_split *split_queue_lock(int nid, struct mem_cgroup = *memcg) -{ - struct deferred_split *queue; - -retry: - queue =3D memcg_split_queue(nid, memcg); - spin_lock(&queue->split_queue_lock); - /* - * There is a period between setting memcg to dying and reparenting - * deferred split queue, and during this period the THPs in the deferred - * split queue will be hidden from the shrinker side. - */ - if (unlikely(memcg_is_dying(memcg))) { - spin_unlock(&queue->split_queue_lock); - memcg =3D parent_mem_cgroup(memcg); - goto retry; - } - - return queue; -} - -static struct deferred_split * -split_queue_lock_irqsave(int nid, struct mem_cgroup *memcg, unsigned long = *flags) -{ - struct deferred_split *queue; - -retry: - queue =3D memcg_split_queue(nid, memcg); - spin_lock_irqsave(&queue->split_queue_lock, *flags); - if (unlikely(memcg_is_dying(memcg))) { - spin_unlock_irqrestore(&queue->split_queue_lock, *flags); - memcg =3D parent_mem_cgroup(memcg); - goto retry; - } - - return queue; -} - -static struct deferred_split *folio_split_queue_lock(struct folio *folio) -{ - struct deferred_split *queue; - - rcu_read_lock(); - queue =3D split_queue_lock(folio_nid(folio), folio_memcg(folio)); - /* - * The memcg destruction path is acquiring the split queue lock for - * reparenting. Once you have it locked, it's safe to drop the rcu lock. - */ - rcu_read_unlock(); - - return queue; -} - -static struct deferred_split * -folio_split_queue_lock_irqsave(struct folio *folio, unsigned long *flags) -{ - struct deferred_split *queue; - - rcu_read_lock(); - queue =3D split_queue_lock_irqsave(folio_nid(folio), folio_memcg(folio), = flags); - rcu_read_unlock(); - - return queue; -} - -static inline void split_queue_unlock(struct deferred_split *queue) -{ - spin_unlock(&queue->split_queue_lock); -} - -static inline void split_queue_unlock_irqrestore(struct deferred_split *qu= eue, - unsigned long flags) -{ - spin_unlock_irqrestore(&queue->split_queue_lock, flags); -} - static inline bool is_transparent_hugepage(const struct folio *folio) { if (!folio_test_large(folio)) @@ -1379,6 +1285,14 @@ static struct folio *vma_alloc_anon_folio_pmd(struct= vm_area_struct *vma, count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); return NULL; } + + if (folio_memcg_alloc_deferred(folio)) { + folio_put(folio); + count_vm_event(THP_FAULT_FALLBACK); + count_mthp_stat(order, MTHP_STAT_ANON_FAULT_FALLBACK); + return NULL; + } + folio_throttle_swaprate(folio, gfp); =20 /* @@ -3888,34 +3802,40 @@ static int __folio_freeze_and_split_unmapped(struct= folio *folio, unsigned int n struct folio *end_folio =3D folio_next(folio); struct folio *new_folio, *next; int old_order =3D folio_order(folio); + struct list_lru_one *lru; + bool dequeue_deferred; int ret =3D 0; - struct deferred_split *ds_queue; =20 VM_WARN_ON_ONCE(!mapping && end); - /* Prevent deferred_split_scan() touching ->_refcount */ - ds_queue =3D folio_split_queue_lock(folio); + /* + * If this folio can be on the deferred split queue, lock out + * the shrinker before freezing the ref. If the shrinker sees + * a 0-ref folio, it assumes it beat folio_put() to the list + * lock and must clean up the LRU state - the same dequeue we + * will do below as part of the split. + */ + dequeue_deferred =3D folio_test_anon(folio) && old_order > 1; + if (dequeue_deferred) { + rcu_read_lock(); + lru =3D list_lru_lock(&deferred_split_lru, + folio_nid(folio), folio_memcg(folio)); + } if (folio_ref_freeze(folio, folio_cache_ref_count(folio) + 1)) { struct swap_cluster_info *ci =3D NULL; struct lruvec *lruvec; =20 - if (old_order > 1) { - if (!list_empty(&folio->_deferred_list)) { - ds_queue->split_queue_len--; - /* - * Reinitialize page_deferred_list after removing the - * page from the split_queue, otherwise a subsequent - * split will see list corruption when checking the - * page_deferred_list. - */ - list_del_init(&folio->_deferred_list); - } + if (dequeue_deferred) { + __list_lru_del(&deferred_split_lru, lru, + &folio->_deferred_list, folio_nid(folio)); if (folio_test_partially_mapped(folio)) { folio_clear_partially_mapped(folio); mod_mthp_stat(old_order, MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1); } + list_lru_unlock(lru); + rcu_read_unlock(); } - split_queue_unlock(ds_queue); + if (mapping) { int nr =3D folio_nr_pages(folio); =20 @@ -4015,7 +3935,10 @@ static int __folio_freeze_and_split_unmapped(struct = folio *folio, unsigned int n if (ci) swap_cluster_unlock(ci); } else { - split_queue_unlock(ds_queue); + if (dequeue_deferred) { + list_lru_unlock(lru); + rcu_read_unlock(); + } return -EAGAIN; } =20 @@ -4381,33 +4304,35 @@ int split_folio_to_list(struct folio *folio, struct= list_head *list) * queueing THP splits, and that list is (racily observed to be) non-empty. * * It is unsafe to call folio_unqueue_deferred_split() until folio refcoun= t is - * zero: because even when split_queue_lock is held, a non-empty _deferred= _list - * might be in use on deferred_split_scan()'s unlocked on-stack list. + * zero: because even when the list_lru lock is held, a non-empty + * _deferred_list might be in use on deferred_split_scan()'s unlocked + * on-stack list. * - * If memory cgroups are enabled, split_queue_lock is in the mem_cgroup: i= t is - * therefore important to unqueue deferred split before changing folio mem= cg. + * The list_lru sublist is determined by folio's memcg: it is therefore + * important to unqueue deferred split before changing folio memcg. */ bool __folio_unqueue_deferred_split(struct folio *folio) { - struct deferred_split *ds_queue; + struct list_lru_one *lru; + int nid =3D folio_nid(folio); unsigned long flags; bool unqueued =3D false; =20 WARN_ON_ONCE(folio_ref_count(folio)); WARN_ON_ONCE(!mem_cgroup_disabled() && !folio_memcg_charged(folio)); =20 - ds_queue =3D folio_split_queue_lock_irqsave(folio, &flags); - if (!list_empty(&folio->_deferred_list)) { - ds_queue->split_queue_len--; + rcu_read_lock(); + lru =3D list_lru_lock_irqsave(&deferred_split_lru, nid, folio_memcg(folio= ), &flags); + if (__list_lru_del(&deferred_split_lru, lru, &folio->_deferred_list, nid)= ) { if (folio_test_partially_mapped(folio)) { folio_clear_partially_mapped(folio); mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1); } - list_del_init(&folio->_deferred_list); unqueued =3D true; } - split_queue_unlock_irqrestore(ds_queue, flags); + list_lru_unlock_irqrestore(lru, &flags); + rcu_read_unlock(); =20 return unqueued; /* useful for debug warnings */ } @@ -4415,7 +4340,9 @@ bool __folio_unqueue_deferred_split(struct folio *fol= io) /* partially_mapped=3Dfalse won't clear PG_partially_mapped folio flag */ void deferred_split_folio(struct folio *folio, bool partially_mapped) { - struct deferred_split *ds_queue; + struct list_lru_one *lru; + int nid; + struct mem_cgroup *memcg; unsigned long flags; =20 /* @@ -4438,7 +4365,11 @@ void deferred_split_folio(struct folio *folio, bool = partially_mapped) if (folio_test_swapcache(folio)) return; =20 - ds_queue =3D folio_split_queue_lock_irqsave(folio, &flags); + nid =3D folio_nid(folio); + + rcu_read_lock(); + memcg =3D folio_memcg(folio); + lru =3D list_lru_lock_irqsave(&deferred_split_lru, nid, memcg, &flags); if (partially_mapped) { if (!folio_test_partially_mapped(folio)) { folio_set_partially_mapped(folio); @@ -4446,36 +4377,20 @@ void deferred_split_folio(struct folio *folio, bool= partially_mapped) count_vm_event(THP_DEFERRED_SPLIT_PAGE); count_mthp_stat(folio_order(folio), MTHP_STAT_SPLIT_DEFERRED); mod_mthp_stat(folio_order(folio), MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, 1= ); - } } else { /* partially mapped folios cannot become non-partially mapped */ VM_WARN_ON_FOLIO(folio_test_partially_mapped(folio), folio); } - if (list_empty(&folio->_deferred_list)) { - struct mem_cgroup *memcg; - - memcg =3D folio_split_queue_memcg(folio, ds_queue); - list_add_tail(&folio->_deferred_list, &ds_queue->split_queue); - ds_queue->split_queue_len++; - if (memcg) - set_shrinker_bit(memcg, folio_nid(folio), - shrinker_id(deferred_split_shrinker)); - } - split_queue_unlock_irqrestore(ds_queue, flags); + __list_lru_add(&deferred_split_lru, lru, &folio->_deferred_list, nid, mem= cg); + list_lru_unlock_irqrestore(lru, &flags); + rcu_read_unlock(); } =20 static unsigned long deferred_split_count(struct shrinker *shrink, struct shrink_control *sc) { - struct pglist_data *pgdata =3D NODE_DATA(sc->nid); - struct deferred_split *ds_queue =3D &pgdata->deferred_split_queue; - -#ifdef CONFIG_MEMCG - if (sc->memcg) - ds_queue =3D &sc->memcg->deferred_split_queue; -#endif - return READ_ONCE(ds_queue->split_queue_len); + return list_lru_shrink_count(&deferred_split_lru, sc); } =20 static bool thp_underused(struct folio *folio) @@ -4505,45 +4420,45 @@ static bool thp_underused(struct folio *folio) return false; } =20 +static enum lru_status deferred_split_isolate(struct list_head *item, + struct list_lru_one *lru, + void *cb_arg) +{ + struct folio *folio =3D container_of(item, struct folio, _deferred_list); + struct list_head *freeable =3D cb_arg; + + if (folio_try_get(folio)) { + list_lru_isolate_move(lru, item, freeable); + return LRU_REMOVED; + } + + /* We lost race with folio_put() */ + list_lru_isolate(lru, item); + if (folio_test_partially_mapped(folio)) { + folio_clear_partially_mapped(folio); + mod_mthp_stat(folio_order(folio), + MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1); + } + return LRU_REMOVED; +} + static unsigned long deferred_split_scan(struct shrinker *shrink, struct shrink_control *sc) { - struct deferred_split *ds_queue; - unsigned long flags; + LIST_HEAD(dispose); struct folio *folio, *next; - int split =3D 0, i; - struct folio_batch fbatch; + int split =3D 0; + unsigned long isolated; =20 - folio_batch_init(&fbatch); + isolated =3D list_lru_shrink_walk_irq(&deferred_split_lru, sc, + deferred_split_isolate, &dispose); =20 -retry: - ds_queue =3D split_queue_lock_irqsave(sc->nid, sc->memcg, &flags); - /* Take pin on all head pages to avoid freeing them under us */ - list_for_each_entry_safe(folio, next, &ds_queue->split_queue, - _deferred_list) { - if (folio_try_get(folio)) { - folio_batch_add(&fbatch, folio); - } else if (folio_test_partially_mapped(folio)) { - /* We lost race with folio_put() */ - folio_clear_partially_mapped(folio); - mod_mthp_stat(folio_order(folio), - MTHP_STAT_NR_ANON_PARTIALLY_MAPPED, -1); - } - list_del_init(&folio->_deferred_list); - ds_queue->split_queue_len--; - if (!--sc->nr_to_scan) - break; - if (!folio_batch_space(&fbatch)) - break; - } - split_queue_unlock_irqrestore(ds_queue, flags); - - for (i =3D 0; i < folio_batch_count(&fbatch); i++) { + list_for_each_entry_safe(folio, next, &dispose, _deferred_list) { bool did_split =3D false; bool underused =3D false; - struct deferred_split *fqueue; =20 - folio =3D fbatch.folios[i]; + list_del_init(&folio->_deferred_list); + if (!folio_test_partially_mapped(folio)) { /* * See try_to_map_unused_to_zeropage(): we cannot @@ -4572,63 +4487,23 @@ static unsigned long deferred_split_scan(struct shr= inker *shrink, * underused, then consider it used and don't add it back to * split_queue. */ - if (did_split || !folio_test_partially_mapped(folio)) - continue; + if (!did_split && folio_test_partially_mapped(folio)) { requeue: - /* - * Add back partially mapped folios, or underused folios that - * we could not lock this round. - */ - fqueue =3D folio_split_queue_lock_irqsave(folio, &flags); - if (list_empty(&folio->_deferred_list)) { - list_add_tail(&folio->_deferred_list, &fqueue->split_queue); - fqueue->split_queue_len++; + rcu_read_lock(); + list_lru_add_irq(&deferred_split_lru, + &folio->_deferred_list, + folio_nid(folio), + folio_memcg(folio)); + rcu_read_unlock(); } - split_queue_unlock_irqrestore(fqueue, flags); - } - folios_put(&fbatch); - - if (sc->nr_to_scan && !list_empty(&ds_queue->split_queue)) { - cond_resched(); - goto retry; + folio_put(folio); } =20 - /* - * Stop shrinker if we didn't split any page, but the queue is empty. - * This can happen if pages were freed under us. - */ - if (!split && list_empty(&ds_queue->split_queue)) + if (!split && !isolated) return SHRINK_STOP; return split; } =20 -#ifdef CONFIG_MEMCG -void reparent_deferred_split_queue(struct mem_cgroup *memcg) -{ - struct mem_cgroup *parent =3D parent_mem_cgroup(memcg); - struct deferred_split *ds_queue =3D &memcg->deferred_split_queue; - struct deferred_split *parent_ds_queue =3D &parent->deferred_split_queue; - int nid; - - spin_lock_irq(&ds_queue->split_queue_lock); - spin_lock_nested(&parent_ds_queue->split_queue_lock, SINGLE_DEPTH_NESTING= ); - - if (!ds_queue->split_queue_len) - goto unlock; - - list_splice_tail_init(&ds_queue->split_queue, &parent_ds_queue->split_que= ue); - parent_ds_queue->split_queue_len +=3D ds_queue->split_queue_len; - ds_queue->split_queue_len =3D 0; - - for_each_node(nid) - set_shrinker_bit(parent, nid, shrinker_id(deferred_split_shrinker)); - -unlock: - spin_unlock(&parent_ds_queue->split_queue_lock); - spin_unlock_irq(&ds_queue->split_queue_lock); -} -#endif - #ifdef CONFIG_DEBUG_FS static void split_huge_pages_all(void) { diff --git a/mm/internal.h b/mm/internal.h index 09931b1e535f..e5bf549ad78e 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -861,7 +861,7 @@ static inline bool folio_unqueue_deferred_split(struct = folio *folio) /* * At this point, there is no one trying to add the folio to * deferred_list. If folio is not in deferred_list, it's safe - * to check without acquiring the split_queue_lock. + * to check without acquiring the list_lru lock. */ if (data_race(list_empty(&folio->_deferred_list))) return false; diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 1a25af3d6d0f..644de0410c12 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1301,6 +1301,9 @@ static enum scan_result collapse_huge_page(struct mm_= struct *mm, unsigned long s if (result !=3D SCAN_SUCCEED) goto out_nolock; =20 + if (folio_memcg_alloc_deferred(folio)) + goto out_nolock; + mmap_read_lock(mm); result =3D hugepage_vma_revalidate(mm, pmd_addr, /*expect_anon=3D*/ true, &vma, cc, order); diff --git a/mm/memcontrol.c b/mm/memcontrol.c index aa7361b0d2da..fe896824078b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -4065,11 +4065,6 @@ static struct mem_cgroup *mem_cgroup_alloc(struct me= m_cgroup *parent) for (i =3D 0; i < MEMCG_CGWB_FRN_CNT; i++) memcg->cgwb_frn[i].done =3D __WB_COMPLETION_INIT(&memcg_cgwb_frn_waitq); -#endif -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - spin_lock_init(&memcg->deferred_split_queue.split_queue_lock); - INIT_LIST_HEAD(&memcg->deferred_split_queue.split_queue); - memcg->deferred_split_queue.split_queue_len =3D 0; #endif lru_gen_init_memcg(memcg); return memcg; @@ -4221,11 +4216,10 @@ static void mem_cgroup_css_offline(struct cgroup_su= bsys_state *css) zswap_memcg_offline_cleanup(memcg); =20 memcg_offline_kmem(memcg); - reparent_deferred_split_queue(memcg); /* - * The reparenting of objcg must be after the reparenting of the - * list_lru and deferred_split_queue above, which ensures that they will - * not mistakenly get the parent list_lru and deferred_split_queue. + * The reparenting of objcg must be after the reparenting of + * the list_lru in memcg_offline_kmem(), which ensures that + * they will not mistakenly get the parent list_lru. */ memcg_reparent_objcgs(memcg); reparent_shrinker_deferred(memcg); diff --git a/mm/memory.c b/mm/memory.c index 552fe26a042a..b97e3145982a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4759,6 +4759,10 @@ static struct folio *alloc_swap_folio(struct vm_faul= t *vmf) folio_put(folio); goto next; } + if (order > 1 && folio_memcg_alloc_deferred(folio)) { + folio_put(folio); + goto fallback; + } return folio; next: count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK); @@ -5279,6 +5283,10 @@ static struct folio *alloc_anon_folio(struct vm_faul= t *vmf) folio_put(folio); goto next; } + if (order > 1 && folio_memcg_alloc_deferred(folio)) { + folio_put(folio); + goto fallback; + } folio_throttle_swaprate(folio, gfp); /* * When a folio is not zeroed during allocation diff --git a/mm/mm_init.c b/mm/mm_init.c index db5568cf36e1..c0a7f1cf6fef 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1373,19 +1373,6 @@ static void __init calculate_node_totalpages(struct = pglist_data *pgdat, pr_debug("On node %d totalpages: %lu\n", pgdat->node_id, realtotalpages); } =20 -#ifdef CONFIG_TRANSPARENT_HUGEPAGE -static void pgdat_init_split_queue(struct pglist_data *pgdat) -{ - struct deferred_split *ds_queue =3D &pgdat->deferred_split_queue; - - spin_lock_init(&ds_queue->split_queue_lock); - INIT_LIST_HEAD(&ds_queue->split_queue); - ds_queue->split_queue_len =3D 0; -} -#else -static void pgdat_init_split_queue(struct pglist_data *pgdat) {} -#endif - #ifdef CONFIG_COMPACTION static void pgdat_init_kcompactd(struct pglist_data *pgdat) { @@ -1401,8 +1388,6 @@ static void __meminit pgdat_init_internals(struct pgl= ist_data *pgdat) =20 pgdat_resize_init(pgdat); pgdat_kswapd_lock_init(pgdat); - - pgdat_init_split_queue(pgdat); pgdat_init_kcompactd(pgdat); =20 init_waitqueue_head(&pgdat->kswapd_wait); --=20 2.54.0