From nobody Sat May 18 18:27:39 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1382EC43334 for ; Tue, 21 Jun 2022 12:57:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350905AbiFUM5t (ORCPT ); Tue, 21 Jun 2022 08:57:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55984 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350894AbiFUM5r (ORCPT ); Tue, 21 Jun 2022 08:57:47 -0400 Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9D6CE193E4 for ; Tue, 21 Jun 2022 05:57:45 -0700 (PDT) Received: by mail-pj1-x1035.google.com with SMTP id g16-20020a17090a7d1000b001ea9f820449so13569318pjl.5 for ; Tue, 21 Jun 2022 05:57:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EillBei8kFa/6bw4XJ6MyIzznZyI10pUVH5tAnTdeuw=; b=Jv9GaHJN86Q1C8Y+YpP2HCf9JPFImIvPmz0BU9q0OZd0+4E84aO4jvX7w2JJE8HLii xXvy2BsIyrtipyPfO3JiQ1L4gabF6BNM/uSE7D/v4vl5Vc36NMI2yEg+YZ5KSpNsGptU HO2lidk6QI25ifAr3ya3fadEcyz9Zbz8Qo1Yg9d9O1CQjjvVGXgpg23fllv4PPkdtuY7 rKxObA5Q8AuofZlnIdgY5syfjw3uLfKsvjs7i/wXGnHIM/h1h8t1/P4ZXW1c8yFLligG siardVDC/TnIZX7H68y5GKiYC9yQyG4FPJP7v8L66B47QZvSC5zmsPVyqmAM4Ajp6REh K7Pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EillBei8kFa/6bw4XJ6MyIzznZyI10pUVH5tAnTdeuw=; b=BM153laaom6uhY1X6JSlISBVGIaH2qlPabsuOoOnnq5Hk5ZXKCFKuwawruuNIAG0bH RKRm99+SGGBPEg9b3ncm73QubQcKiKFxb1yzdch1gHfNoBXRyInEvz5amxXHV9wH0LEd /CFS1yC7ARe0CM4ZVjBSyVfRid+oYSzIkVJpDBK5vzoBisWfNL2h5w5mDpv/PvxhE/Qy PZee0bSoI/V1Rp2+dY4qFeD+Gh2p2aD3ONeNbHgqpOSRflhzFmxFk+ssOclX1+cdtvfA fcCFJldEiNALwJaDbGTpeR+KRTvaGGwiymO4p0HLklQiyRiqPxqFQEFjc6dYYIHuqb8Q wcQQ== X-Gm-Message-State: AJIora8PfVz2SoxCTuvm24sutarsAgxHf7kKA31QM2+9WnVF9W+W6nez lQf0yphyxgJT/OZGYY5idj3/4Q== X-Google-Smtp-Source: AGRyM1tSXaqiZZOhIChBYioUnW0DVCRqwFclVFY428N7/E3eP36ixMt0VVOaIjmyV7ksj0qm20VtAQ== X-Received: by 2002:a17:902:c2c6:b0:168:d8ce:90b2 with SMTP id c6-20020a170902c2c600b00168d8ce90b2mr29357788pla.110.1655816265091; Tue, 21 Jun 2022 05:57:45 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id e3-20020a170903240300b0015ea3a491a1sm10643134plo.191.2022.06.21.05.57.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jun 2022 05:57:44 -0700 (PDT) From: Muchun Song To: akpm@linux-foundation.org, hannes@cmpxchg.org, longman@redhat.com, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, duanxiongchun@bytedance.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH v6 01/11] mm: memcontrol: remove dead code and comments Date: Tue, 21 Jun 2022 20:56:48 +0800 Message-Id: <20220621125658.64935-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220621125658.64935-1-songmuchun@bytedance.com> References: <20220621125658.64935-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Since no-hierarchy mode is deprecated after commit bef8620cd8e0 ("mm: memcg: deprecate the non-hierarchical mode") so parent_mem_cgroup() cannot return a NULL except root memcg, however, root memcg cannot be offline, so it is safe to drop the check of returned value of parent_mem_cgroup(). Remove those dead code. The comments in memcg_offline_kmem() above memcg_reparent_list_lrus() are out of date since commit 5abc1e37afa0 ("mm: list_lru: allocate list_lru_one only when neede= d") There is no ordering requirement between memcg_reparent_list_lrus() and memcg_reparent_objcgs(), so remove those outdated comments. Signed-off-by: Muchun Song Acked-by: Roman Gushchin --- include/linux/memcontrol.h | 3 +-- mm/memcontrol.c | 12 ------------ mm/vmscan.c | 6 +----- 3 files changed, 2 insertions(+), 19 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 4d31ce55b1c0..318d8880d62a 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -866,8 +866,7 @@ static inline struct mem_cgroup *lruvec_memcg(struct lr= uvec *lruvec) * parent_mem_cgroup - find the accounting parent of a memcg * @memcg: memcg whose parent to find * - * Returns the parent memcg, or NULL if this is the root or the memory - * controller is in legacy no-hierarchy mode. + * Returns the parent memcg, or NULL if this is the root. */ static inline struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *memc= g) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 163492b9efa9..fc706d6fc265 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3684,17 +3684,7 @@ static void memcg_offline_kmem(struct mem_cgroup *me= mcg) return; =20 parent =3D parent_mem_cgroup(memcg); - if (!parent) - parent =3D root_mem_cgroup; - memcg_reparent_objcgs(memcg, parent); - - /* - * After we have finished memcg_reparent_objcgs(), all list_lrus - * corresponding to this cgroup are guaranteed to remain empty. - * The ordering is imposed by list_lru_node->lock taken by - * memcg_reparent_list_lrus(). - */ memcg_reparent_list_lrus(memcg, parent); } #else @@ -7195,8 +7185,6 @@ static struct mem_cgroup *mem_cgroup_id_get_online(st= ruct mem_cgroup *memcg) break; } memcg =3D parent_mem_cgroup(memcg); - if (!memcg) - memcg =3D root_mem_cgroup; } return memcg; } diff --git a/mm/vmscan.c b/mm/vmscan.c index 88fce64cfa96..b68b0216424d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -409,13 +409,9 @@ void reparent_shrinker_deferred(struct mem_cgroup *mem= cg) { int i, nid; long nr; - struct mem_cgroup *parent; + struct mem_cgroup *parent =3D parent_mem_cgroup(memcg); struct shrinker_info *child_info, *parent_info; =20 - parent =3D parent_mem_cgroup(memcg); - if (!parent) - parent =3D root_mem_cgroup; - /* Prevent from concurrent shrinker_info expand */ down_read(&shrinker_rwsem); for_each_node(nid) { --=20 2.11.0 From nobody Sat May 18 18:27:39 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C605FC43334 for ; Tue, 21 Jun 2022 12:58:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351262AbiFUM6z (ORCPT ); Tue, 21 Jun 2022 08:58:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56504 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350942AbiFUM6P (ORCPT ); Tue, 21 Jun 2022 08:58:15 -0400 Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9CEBB1A05C for ; Tue, 21 Jun 2022 05:57:50 -0700 (PDT) Received: by mail-pg1-x536.google.com with SMTP id q140so13045102pgq.6 for ; Tue, 21 Jun 2022 05:57:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gTiFdIy6o2Yi1gJX4lzQx8P4N9gDSqx+3EHpcopFmMs=; b=H6I4gmqjOFEdoEBuSJPzHJ6dCyaEaCmJXpSxrAZPjyTDv2Q8SbONI8KHfphtzLSqAT 9lc44ETpllLe4mOMH3oB/7XjlxsXvUWOJI2lPGD9OdboEEo7oWRGt/sNWjQjg6uKTUaT OOdb2lpx/vJUedABwJ4o/OcHKlu8smj+MJLQbU2n/KlYlyJj9eG75Hj1MsTmf/JkzZra PWAdrUqyJuZ9KAi7TS7rUa7eL8821+6CHCEga0i4PVqxW6rgphVWRtshdCB+Q8T8vkBM KF9731XEaigVx6tvEzqN6j/Hfbzh2otQQV4EaGCl0EvE7UfQNhJl6MSBUPoAkj5QJX62 XTrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gTiFdIy6o2Yi1gJX4lzQx8P4N9gDSqx+3EHpcopFmMs=; b=XkULqluFlMl1253yDEJB4dJAhnkb6HdLooHaPqf1hs6mnj7Sks7CTt5hYyvwqaOonZ gdiMoO9IL1DpoCa5Pkk3yPVcJwV+8p5tFhysdJ5hHVV3Fgp6rqNPwIOfr8xTUHTTnBLK 9StX3/CmPrRNbJez0ZG07VwK4OpQEcoI42j6u3dIZr/AXWkvOtnDnfCONHtKoqDUITpm bzhMUuzgn9ZzLq793bKq/f4liKDbyzVVYKhzeNUDNXH8YQ38KN8jN8H5DdTxJGm26FGR pF7desra4GsWWLhElW3xLztjtyD1tGc1FNnjcktmlW0RZzZHZBnZU0OS7B0WH17acli7 ZgeQ== X-Gm-Message-State: AJIora8kshjFeP+nXvi9JGCStUasKPQ70piZcnXcNOAvQmxITX3XPqLf dqs125k4uO6VoGbTRxpfB2vbVA== X-Google-Smtp-Source: AGRyM1vpTJTblWfSYnZ+Nh7sMujWCZHB4wqmjhM8d1a4YN3w990DlToN0ZohIO62P6JEDRcNdGSzyg== X-Received: by 2002:a63:f453:0:b0:3fc:7939:9654 with SMTP id p19-20020a63f453000000b003fc79399654mr26239787pgk.341.1655816270064; Tue, 21 Jun 2022 05:57:50 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id e3-20020a170903240300b0015ea3a491a1sm10643134plo.191.2022.06.21.05.57.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jun 2022 05:57:49 -0700 (PDT) From: Muchun Song To: akpm@linux-foundation.org, hannes@cmpxchg.org, longman@redhat.com, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, duanxiongchun@bytedance.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH v6 02/11] mm: rename unlock_page_lruvec{_irq, _irqrestore} to lruvec_unlock{_irq, _irqrestore} Date: Tue, 21 Jun 2022 20:56:49 +0800 Message-Id: <20220621125658.64935-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220621125658.64935-1-songmuchun@bytedance.com> References: <20220621125658.64935-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" It is weird to use folio_lruvec_lock() variants and unlock_page_lruvec() va= riants together, e.g. locking folio and unlocking page. So rename unlock_page_lruvec{_irq, _irqrestore} to lruvec_unlock{_irq, _irqrestore}. Signed-off-by: Muchun Song Acked-by: Roman Gushchin --- include/linux/memcontrol.h | 10 +++++----- mm/compaction.c | 12 ++++++------ mm/huge_memory.c | 2 +- mm/mlock.c | 2 +- mm/swap.c | 14 +++++++------- mm/vmscan.c | 4 ++-- 6 files changed, 22 insertions(+), 22 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 318d8880d62a..d0c0da7cafb7 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1579,17 +1579,17 @@ static inline struct lruvec *parent_lruvec(struct l= ruvec *lruvec) return mem_cgroup_lruvec(memcg, lruvec_pgdat(lruvec)); } =20 -static inline void unlock_page_lruvec(struct lruvec *lruvec) +static inline void lruvec_unlock(struct lruvec *lruvec) { spin_unlock(&lruvec->lru_lock); } =20 -static inline void unlock_page_lruvec_irq(struct lruvec *lruvec) +static inline void lruvec_unlock_irq(struct lruvec *lruvec) { spin_unlock_irq(&lruvec->lru_lock); } =20 -static inline void unlock_page_lruvec_irqrestore(struct lruvec *lruvec, +static inline void lruvec_unlock_irqrestore(struct lruvec *lruvec, unsigned long flags) { spin_unlock_irqrestore(&lruvec->lru_lock, flags); @@ -1611,7 +1611,7 @@ static inline struct lruvec *folio_lruvec_relock_irq(= struct folio *folio, if (folio_matches_lruvec(folio, locked_lruvec)) return locked_lruvec; =20 - unlock_page_lruvec_irq(locked_lruvec); + lruvec_unlock_irq(locked_lruvec); } =20 return folio_lruvec_lock_irq(folio); @@ -1625,7 +1625,7 @@ static inline struct lruvec *folio_lruvec_relock_irqs= ave(struct folio *folio, if (folio_matches_lruvec(folio, locked_lruvec)) return locked_lruvec; =20 - unlock_page_lruvec_irqrestore(locked_lruvec, *flags); + lruvec_unlock_irqrestore(locked_lruvec, *flags); } =20 return folio_lruvec_lock_irqsave(folio, flags); diff --git a/mm/compaction.c b/mm/compaction.c index 1f89b969c12b..46351a14eed2 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -864,7 +864,7 @@ isolate_migratepages_block(struct compact_control *cc, = unsigned long low_pfn, */ if (!(low_pfn % COMPACT_CLUSTER_MAX)) { if (locked) { - unlock_page_lruvec_irqrestore(locked, flags); + lruvec_unlock_irqrestore(locked, flags); locked =3D NULL; } =20 @@ -977,7 +977,7 @@ isolate_migratepages_block(struct compact_control *cc, = unsigned long low_pfn, if (unlikely(__PageMovable(page)) && !PageIsolated(page)) { if (locked) { - unlock_page_lruvec_irqrestore(locked, flags); + lruvec_unlock_irqrestore(locked, flags); locked =3D NULL; } =20 @@ -1060,7 +1060,7 @@ isolate_migratepages_block(struct compact_control *cc= , unsigned long low_pfn, /* If we already hold the lock, we can skip some rechecking */ if (lruvec !=3D locked) { if (locked) - unlock_page_lruvec_irqrestore(locked, flags); + lruvec_unlock_irqrestore(locked, flags); =20 compact_lock_irqsave(&lruvec->lru_lock, &flags, cc); locked =3D lruvec; @@ -1119,7 +1119,7 @@ isolate_migratepages_block(struct compact_control *cc= , unsigned long low_pfn, isolate_fail_put: /* Avoid potential deadlock in freeing page under lru_lock */ if (locked) { - unlock_page_lruvec_irqrestore(locked, flags); + lruvec_unlock_irqrestore(locked, flags); locked =3D NULL; } put_page(page); @@ -1135,7 +1135,7 @@ isolate_migratepages_block(struct compact_control *cc= , unsigned long low_pfn, */ if (nr_isolated) { if (locked) { - unlock_page_lruvec_irqrestore(locked, flags); + lruvec_unlock_irqrestore(locked, flags); locked =3D NULL; } putback_movable_pages(&cc->migratepages); @@ -1167,7 +1167,7 @@ isolate_migratepages_block(struct compact_control *cc= , unsigned long low_pfn, =20 isolate_abort: if (locked) - unlock_page_lruvec_irqrestore(locked, flags); + lruvec_unlock_irqrestore(locked, flags); if (page) { SetPageLRU(page); put_page(page); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 2e2a8b5bc567..66d9ed8a1289 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2515,7 +2515,7 @@ static void __split_huge_page(struct page *page, stru= ct list_head *list, } =20 ClearPageCompound(head); - unlock_page_lruvec(lruvec); + lruvec_unlock(lruvec); /* Caller disabled irqs, so they are still disabled here */ =20 split_page_owner(head, nr); diff --git a/mm/mlock.c b/mm/mlock.c index 7032f6dd0ce1..d9039fb9c56b 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -205,7 +205,7 @@ static void mlock_pagevec(struct pagevec *pvec) } =20 if (lruvec) - unlock_page_lruvec_irq(lruvec); + lruvec_unlock_irq(lruvec); release_pages(pvec->pages, pvec->nr); pagevec_reinit(pvec); } diff --git a/mm/swap.c b/mm/swap.c index 1f563d857768..127ef4db394f 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -86,7 +86,7 @@ static void __page_cache_release(struct folio *folio) lruvec =3D folio_lruvec_lock_irqsave(folio, &flags); lruvec_del_folio(lruvec, folio); __folio_clear_lru_flags(folio); - unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec_unlock_irqrestore(lruvec, flags); } /* See comment on folio_test_mlocked in release_pages() */ if (unlikely(folio_test_mlocked(folio))) { @@ -249,7 +249,7 @@ static void folio_batch_move_lru(struct folio_batch *fb= atch, move_fn_t move_fn) } =20 if (lruvec) - unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec_unlock_irqrestore(lruvec, flags); folios_put(fbatch->folios, folio_batch_count(fbatch)); folio_batch_init(fbatch); } @@ -392,7 +392,7 @@ static void folio_activate(struct folio *folio) if (folio_test_clear_lru(folio)) { lruvec =3D folio_lruvec_lock_irq(folio); folio_activate_fn(lruvec, folio); - unlock_page_lruvec_irq(lruvec); + lruvec_unlock_irq(lruvec); folio_set_lru(folio); } } @@ -948,7 +948,7 @@ void release_pages(struct page **pages, int nr) * same lruvec. The lock is held only if lruvec !=3D NULL. */ if (lruvec && ++lock_batch =3D=3D SWAP_CLUSTER_MAX) { - unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec_unlock_irqrestore(lruvec, flags); lruvec =3D NULL; } =20 @@ -957,7 +957,7 @@ void release_pages(struct page **pages, int nr) =20 if (folio_is_zone_device(folio)) { if (lruvec) { - unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec_unlock_irqrestore(lruvec, flags); lruvec =3D NULL; } if (put_devmap_managed_page(&folio->page)) @@ -972,7 +972,7 @@ void release_pages(struct page **pages, int nr) =20 if (folio_test_large(folio)) { if (lruvec) { - unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec_unlock_irqrestore(lruvec, flags); lruvec =3D NULL; } __folio_put_large(folio); @@ -1006,7 +1006,7 @@ void release_pages(struct page **pages, int nr) list_add(&folio->lru, &pages_to_free); } if (lruvec) - unlock_page_lruvec_irqrestore(lruvec, flags); + lruvec_unlock_irqrestore(lruvec, flags); =20 mem_cgroup_uncharge_list(&pages_to_free); free_unref_page_list(&pages_to_free); diff --git a/mm/vmscan.c b/mm/vmscan.c index b68b0216424d..6a554712ef5d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2257,7 +2257,7 @@ int folio_isolate_lru(struct folio *folio) folio_get(folio); lruvec =3D folio_lruvec_lock_irq(folio); lruvec_del_folio(lruvec, folio); - unlock_page_lruvec_irq(lruvec); + lruvec_unlock_irq(lruvec); ret =3D 0; } =20 @@ -4886,7 +4886,7 @@ void check_move_unevictable_pages(struct pagevec *pve= c) if (lruvec) { __count_vm_events(UNEVICTABLE_PGRESCUED, pgrescued); __count_vm_events(UNEVICTABLE_PGSCANNED, pgscanned); - unlock_page_lruvec_irq(lruvec); + lruvec_unlock_irq(lruvec); } else if (pgscanned) { count_vm_events(UNEVICTABLE_PGSCANNED, pgscanned); } --=20 2.11.0 From nobody Sat May 18 18:27:39 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B756C433EF for ; Tue, 21 Jun 2022 12:59:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351294AbiFUM7i (ORCPT ); Tue, 21 Jun 2022 08:59:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56866 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351017AbiFUM6X (ORCPT ); Tue, 21 Jun 2022 08:58:23 -0400 Received: from mail-pf1-x42e.google.com (mail-pf1-x42e.google.com [IPv6:2607:f8b0:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 605D6B2C for ; Tue, 21 Jun 2022 05:57:57 -0700 (PDT) Received: by mail-pf1-x42e.google.com with SMTP id x4so6705063pfq.2 for ; Tue, 21 Jun 2022 05:57:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6Y6NgpPc42NYBvW73tDuppRYStJt3qQWgJd6fl6Oef4=; b=uF9OogsqG1RyhL7U+jblJtPj5YWhZ4AiZjE+EQ8j0xRHnV/O89W7gU9jS5UW7Ou5bk ZD/BJsWxtuzMf2V5MCtrrLDGRQKEb68PPNjcqskQI1e8ES4jEzP5uvODBovx4ljRpfFm URbzN150jWgQm4hYBvEsrmj8rhGXLX85rM84Iu+JLS+u0dihChhDk+5OZJPSxxf/Oqad VeEhlbrM6KTndZCc9GE36AjtfHdnyNwn4fq45ma+q1Ua0zo9M43j5cWF8tG4wN4FA4LC nFGonUPSl7pV6F76pm3vWS+XdEQak09XaD2SO1sRKIBVG64sz0Zi2lGZomsjaCGKNCBu NHFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6Y6NgpPc42NYBvW73tDuppRYStJt3qQWgJd6fl6Oef4=; b=J9ROyvJ6iTMTe96GZxlbKWO6i6eqL3PYa0twCsbSn15LdUe+KMRSMMD0pPA/V10ccP J9MGHcnyu2ODekrPTPtMaOf7Z+i2x7pKc/KlJru8X9D7qv7Fxug581pN/w0PICidP4wF 18LTq1u5B3LW4JOX8AIsUoxyAV40ATC2JVdp7NIsVOwYK4AzfjCuJhq3hY8etUeMLH7C AWv+3TNp4ywTRY4O43PYnAPOvIMZBpAc0pWp6tM/wP9SnnICvYf+a5RjuvB3zHEgdMSz w6JKjI3zpmVLr6V2NHQWmk8rFiOvf57YwlzxJH+E5VZ7LRRDalK5LiuMdavq507iRn4+ YSfg== X-Gm-Message-State: AJIora8+gjt5Y4QVvtWZRQOb5cjoi46cNoUCHcg0mxvAkB8fdeU4/lEy BIX8VZZ07ewQlHYkqJW4JsFzfA== X-Google-Smtp-Source: AGRyM1sQC5dliKs7RaIb0VSNxBH7/RLYnMsWURiGVbZONIJEcJX14MVUbAJGDDw7hQbIAjk0G7rmSA== X-Received: by 2002:aa7:88cc:0:b0:51c:319e:772c with SMTP id k12-20020aa788cc000000b0051c319e772cmr30033261pff.41.1655816276884; Tue, 21 Jun 2022 05:57:56 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id e3-20020a170903240300b0015ea3a491a1sm10643134plo.191.2022.06.21.05.57.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jun 2022 05:57:56 -0700 (PDT) From: Muchun Song To: akpm@linux-foundation.org, hannes@cmpxchg.org, longman@redhat.com, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, duanxiongchun@bytedance.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song , =?UTF-8?q?Michal=20Koutn=C3=BD?= Subject: [PATCH v6 03/11] mm: memcontrol: prepare objcg API for non-kmem usage Date: Tue, 21 Jun 2022 20:56:50 +0800 Message-Id: <20220621125658.64935-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220621125658.64935-1-songmuchun@bytedance.com> References: <20220621125658.64935-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Pagecache pages are charged at the allocation time and holding a reference to the original memory cgroup until being reclaimed. Depending on the memory pressure, specific patterns of the page sharing between different cgroups and the cgroup creation and destruction rates, a large number of dying memory cgroups can be pinned by pagecache pages. It makes the page reclaim less efficient and wastes memory. We can convert LRU pages and most other raw memcg pins to the objcg direction to fix this problem, and then the page->memcg will always point to an object cgroup pointer. Therefore, the infrastructure of objcg no longer only serves CONFIG_MEMCG_KMEM. In this patch, we move the infrastructure of the objcg out of the scope of the CONFIG_MEMCG_KMEM so that the LRU pages can reuse it to charge pages. We know that the LRU pages are not accounted at the root level. But the page->memcg_data points to the root_mem_cgroup. So the page->memcg_data of the LRU pages always points to a valid pointer. But the root_mem_cgroup dose not have an object cgroup. If we use obj_cgroup APIs to charge the LRU pages, we should set the page->memcg_data to a root object cgroup. So we also allocate an object cgroup for the root_mem_cgroup. Signed-off-by: Muchun Song Acked-by: Johannes Weiner Reviewed-by: Michal Koutn=C3=BD Acked-by: Roman Gushchin --- include/linux/memcontrol.h | 2 +- mm/memcontrol.c | 56 +++++++++++++++++++++++++++---------------= ---- 2 files changed, 34 insertions(+), 24 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index d0c0da7cafb7..111eda6ff1ce 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -321,10 +321,10 @@ struct mem_cgroup { =20 #ifdef CONFIG_MEMCG_KMEM int kmemcg_id; +#endif struct obj_cgroup __rcu *objcg; /* list of inherited objcgs, protected by objcg_lock */ struct list_head objcg_list; -#endif =20 MEMCG_PADDING(_pad2_); =20 diff --git a/mm/memcontrol.c b/mm/memcontrol.c index fc706d6fc265..3c489651d312 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -252,9 +252,9 @@ struct mem_cgroup *vmpressure_to_memcg(struct vmpressur= e *vmpr) return container_of(vmpr, struct mem_cgroup, vmpressure); } =20 -#ifdef CONFIG_MEMCG_KMEM static DEFINE_SPINLOCK(objcg_lock); =20 +#ifdef CONFIG_MEMCG_KMEM bool mem_cgroup_kmem_disabled(void) { return cgroup_memory_nokmem; @@ -263,12 +263,10 @@ bool mem_cgroup_kmem_disabled(void) static void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg, unsigned int nr_pages); =20 -static void obj_cgroup_release(struct percpu_ref *ref) +static void obj_cgroup_release_bytes(struct obj_cgroup *objcg) { - struct obj_cgroup *objcg =3D container_of(ref, struct obj_cgroup, refcnt); unsigned int nr_bytes; unsigned int nr_pages; - unsigned long flags; =20 /* * At this point all allocated objects are freed, and @@ -282,9 +280,9 @@ static void obj_cgroup_release(struct percpu_ref *ref) * 3) CPU1: a process from another memcg is allocating something, * the stock if flushed, * objcg->nr_charged_bytes =3D PAGE_SIZE - 92 - * 5) CPU0: we do release this object, + * 4) CPU0: we do release this object, * 92 bytes are added to stock->nr_bytes - * 6) CPU0: stock is flushed, + * 5) CPU0: stock is flushed, * 92 bytes are added to objcg->nr_charged_bytes * * In the result, nr_charged_bytes =3D=3D PAGE_SIZE. @@ -296,6 +294,19 @@ static void obj_cgroup_release(struct percpu_ref *ref) =20 if (nr_pages) obj_cgroup_uncharge_pages(objcg, nr_pages); +} +#else +static inline void obj_cgroup_release_bytes(struct obj_cgroup *objcg) +{ +} +#endif + +static void obj_cgroup_release(struct percpu_ref *ref) +{ + struct obj_cgroup *objcg =3D container_of(ref, struct obj_cgroup, refcnt); + unsigned long flags; + + obj_cgroup_release_bytes(objcg); =20 spin_lock_irqsave(&objcg_lock, flags); list_del(&objcg->list); @@ -324,10 +335,10 @@ static struct obj_cgroup *obj_cgroup_alloc(void) return objcg; } =20 -static void memcg_reparent_objcgs(struct mem_cgroup *memcg, - struct mem_cgroup *parent) +static void memcg_reparent_objcgs(struct mem_cgroup *memcg) { struct obj_cgroup *objcg, *iter; + struct mem_cgroup *parent =3D parent_mem_cgroup(memcg); =20 objcg =3D rcu_replace_pointer(memcg->objcg, NULL, true); =20 @@ -346,6 +357,7 @@ static void memcg_reparent_objcgs(struct mem_cgroup *me= mcg, percpu_ref_kill(&objcg->refcnt); } =20 +#ifdef CONFIG_MEMCG_KMEM /* * A lot of the calls to the cache allocation functions are expected to be * inlined by the compiler. Since the calls to memcg_slab_pre_alloc_hook()= are @@ -3651,21 +3663,12 @@ static u64 mem_cgroup_read_u64(struct cgroup_subsys= _state *css, #ifdef CONFIG_MEMCG_KMEM static int memcg_online_kmem(struct mem_cgroup *memcg) { - struct obj_cgroup *objcg; - if (cgroup_memory_nokmem) return 0; =20 if (unlikely(mem_cgroup_is_root(memcg))) return 0; =20 - objcg =3D obj_cgroup_alloc(); - if (!objcg) - return -ENOMEM; - - objcg->memcg =3D memcg; - rcu_assign_pointer(memcg->objcg, objcg); - static_branch_enable(&memcg_kmem_enabled_key); =20 memcg->kmemcg_id =3D memcg->id.id; @@ -3675,17 +3678,13 @@ static int memcg_online_kmem(struct mem_cgroup *mem= cg) =20 static void memcg_offline_kmem(struct mem_cgroup *memcg) { - struct mem_cgroup *parent; - if (cgroup_memory_nokmem) return; =20 if (unlikely(mem_cgroup_is_root(memcg))) return; =20 - parent =3D parent_mem_cgroup(memcg); - memcg_reparent_objcgs(memcg, parent); - memcg_reparent_list_lrus(memcg, parent); + memcg_reparent_list_lrus(memcg, parent_mem_cgroup(memcg)); } #else static int memcg_online_kmem(struct mem_cgroup *memcg) @@ -5190,8 +5189,8 @@ static struct mem_cgroup *mem_cgroup_alloc(void) memcg->socket_pressure =3D jiffies; #ifdef CONFIG_MEMCG_KMEM memcg->kmemcg_id =3D -1; - INIT_LIST_HEAD(&memcg->objcg_list); #endif + INIT_LIST_HEAD(&memcg->objcg_list); #ifdef CONFIG_CGROUP_WRITEBACK INIT_LIST_HEAD(&memcg->cgwb_list); for (i =3D 0; i < MEMCG_CGWB_FRN_CNT; i++) @@ -5256,6 +5255,7 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *pare= nt_css) static int mem_cgroup_css_online(struct cgroup_subsys_state *css) { struct mem_cgroup *memcg =3D mem_cgroup_from_css(css); + struct obj_cgroup *objcg; =20 if (memcg_online_kmem(memcg)) goto remove_id; @@ -5268,6 +5268,13 @@ static int mem_cgroup_css_online(struct cgroup_subsy= s_state *css) if (alloc_shrinker_info(memcg)) goto offline_kmem; =20 + objcg =3D obj_cgroup_alloc(); + if (!objcg) + goto free_shrinker; + + objcg->memcg =3D memcg; + rcu_assign_pointer(memcg->objcg, objcg); + /* Online state pins memcg ID, memcg ID pins CSS */ refcount_set(&memcg->id.ref, 1); css_get(css); @@ -5276,6 +5283,8 @@ static int mem_cgroup_css_online(struct cgroup_subsys= _state *css) queue_delayed_work(system_unbound_wq, &stats_flush_dwork, 2UL*HZ); return 0; +free_shrinker: + free_shrinker_info(memcg); offline_kmem: memcg_offline_kmem(memcg); remove_id: @@ -5303,6 +5312,7 @@ static void mem_cgroup_css_offline(struct cgroup_subs= ys_state *css) page_counter_set_min(&memcg->memory, 0); page_counter_set_low(&memcg->memory, 0); =20 + memcg_reparent_objcgs(memcg); memcg_offline_kmem(memcg); reparent_shrinker_deferred(memcg); wb_memcg_offline(memcg); --=20 2.11.0 From nobody Sat May 18 18:27:39 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4900DC43334 for ; Tue, 21 Jun 2022 12:59:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351043AbiFUM74 (ORCPT ); Tue, 21 Jun 2022 08:59:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351033AbiFUM6Y (ORCPT ); Tue, 21 Jun 2022 08:58:24 -0400 Received: from mail-pf1-x42d.google.com (mail-pf1-x42d.google.com [IPv6:2607:f8b0:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61E33EB9 for ; Tue, 21 Jun 2022 05:58:02 -0700 (PDT) Received: by mail-pf1-x42d.google.com with SMTP id n12so6098596pfq.0 for ; Tue, 21 Jun 2022 05:58:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=SlTUvNn4DYl954HH+rppOxcDjVgo6GqCQhQnv1Q1EhI=; b=fL58wbGMa2SB+egmCsOvHcDN9eGBoafFpvGg3so9U2DmhwQApgE7KCSCjC0WqjJSEn U5vzUEd9XhSPlTzg2SbtZtzSKOqEvUMV1cOHyWmMvvC1jYprYDJl3s9sItS89pDeS9ju RecNr0jOEyi1FNnA5SBkQiHhh6ThScEZzdCM/AQHdtKi4/YLil617oq14JDUMS09tcfU JhfTJo28sCf+ZmeDAmSLtgBqjE0xZ7p6panlmEzDWBJKFE3B7RHOsAdD+pYq2fj0tFhP A3Y94ty0igohGF4sTFirkZWPJzp4qvQtmS9/W3uwgeypNIZuA5gd8Qkxq/Tz6WFjN4ly lPiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SlTUvNn4DYl954HH+rppOxcDjVgo6GqCQhQnv1Q1EhI=; b=07Dhoxku/st9pMI+HvC23cM6vt0+3jJPDNhV5Z+khnd/LhIL7qk/Hi/gjh2Z68El7R 2VgHvXKGzZ3oBQLe6mmquZcEoLNGAkz6jgUgxcZSmqrHjA44WBPK7h1fzI4iwJKEqWJr ZsJzL84U9Tl7Dcwfly6JdaA76nXUyhtR/VG4nqpXXMa8CwHRVvxwwBcVYYYLSxymg1dQ Q0Yt+Lag0M4rDRbLryJ0eOikYo9fipKMmwYj36sSUBIP4dTiWHQW0SyYOhqNNheWPgEP Co+n68FulXoswZFaFcNVP6mmbN8rQYyPT0jSfNwNm3ghQZLE4taQmT6DmUf2JNzPsW4G x+sQ== X-Gm-Message-State: AJIora+/bDta/fTx2FECTriD84hf3/0v4BKsotUDQGnydoh2bSw+k7pQ ARg3qOTung8HGepSxXQO8StrAA== X-Google-Smtp-Source: AGRyM1uuuY5reI7WuH0uENLbGB2vH7kDjDFb906OSYMlWq65zYFJjEPKnW83COUBx7RaIXd8cyjJnw== X-Received: by 2002:a05:6a00:c92:b0:51c:1030:5eef with SMTP id a18-20020a056a000c9200b0051c10305eefmr30060253pfv.76.1655816282163; Tue, 21 Jun 2022 05:58:02 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id e3-20020a170903240300b0015ea3a491a1sm10643134plo.191.2022.06.21.05.57.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jun 2022 05:58:01 -0700 (PDT) From: Muchun Song To: akpm@linux-foundation.org, hannes@cmpxchg.org, longman@redhat.com, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, duanxiongchun@bytedance.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH v6 04/11] mm: memcontrol: make lruvec lock safe when LRU pages are reparented Date: Tue, 21 Jun 2022 20:56:51 +0800 Message-Id: <20220621125658.64935-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220621125658.64935-1-songmuchun@bytedance.com> References: <20220621125658.64935-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The diagram below shows how to make the folio lruvec lock safe when LRU pages are reparented. folio_lruvec_lock(folio) rcu_read_lock(); retry: lruvec =3D folio_lruvec(folio); // The folio is reparented at this time. spin_lock(&lruvec->lru_lock); if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) // Acquired the wrong lruvec lock and need to retry. // Because this folio is on the parent memcg lruvec list. spin_unlock(&lruvec->lru_lock); goto retry; // If we reach here, it means that folio_memcg(folio) is stable. memcg_reparent_objcgs(memcg) // lruvec belongs to memcg and lruvec_parent belongs to parent memcg. spin_lock(&lruvec->lru_lock); spin_lock(&lruvec_parent->lru_lock); // Move all the pages from the lruvec list to the parent lruvec list. spin_unlock(&lruvec_parent->lru_lock); spin_unlock(&lruvec->lru_lock); After we acquire the lruvec lock, we need to check whether the folio is reparented. If so, we need to reacquire the new lruvec lock. On the routine of the LRU pages reparenting, we will also acquire the lruvec lock (will be implemented in the later patch). So folio_memcg() cannot be changed when we hold the lruvec lock. Since lruvec_memcg(lruvec) is always equal to folio_memcg(folio) after we hold the lruvec lock, lruvec_memcg_debug() check is pointless. So remove it. This is a preparation for reparenting the LRU pages. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 18 +++------------- mm/compaction.c | 27 +++++++++++++++++++---- mm/memcontrol.c | 53 ++++++++++++++++++++++++++----------------= ---- mm/swap.c | 5 +++++ 4 files changed, 61 insertions(+), 42 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 111eda6ff1ce..ff3106eca6f3 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -758,7 +758,9 @@ static inline struct lruvec *mem_cgroup_lruvec(struct m= em_cgroup *memcg, * folio_lruvec - return lruvec for isolating/putting an LRU folio * @folio: Pointer to the folio. * - * This function relies on folio->mem_cgroup being stable. + * The lruvec can be changed to its parent lruvec when the page reparented. + * The caller need to recheck if it cares about this changes (just like + * folio_lruvec_lock() does). */ static inline struct lruvec *folio_lruvec(struct folio *folio) { @@ -777,15 +779,6 @@ struct lruvec *folio_lruvec_lock_irq(struct folio *fol= io); struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio, unsigned long *flags); =20 -#ifdef CONFIG_DEBUG_VM -void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio); -#else -static inline -void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio) -{ -} -#endif - static inline struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){ return css ? container_of(css, struct mem_cgroup, css) : NULL; @@ -1260,11 +1253,6 @@ static inline struct lruvec *folio_lruvec(struct fol= io *folio) return &pgdat->__lruvec; } =20 -static inline -void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio) -{ -} - static inline struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *memc= g) { return NULL; diff --git a/mm/compaction.c b/mm/compaction.c index 46351a14eed2..fe49ac9aedd8 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -508,6 +508,25 @@ static bool compact_lock_irqsave(spinlock_t *lock, uns= igned long *flags, return true; } =20 +static struct lruvec * +compact_folio_lruvec_lock_irqsave(struct folio *folio, unsigned long *flag= s, + struct compact_control *cc) +{ + struct lruvec *lruvec; + + rcu_read_lock(); +retry: + lruvec =3D folio_lruvec(folio); + compact_lock_irqsave(&lruvec->lru_lock, flags, cc); + if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) { + spin_unlock_irqrestore(&lruvec->lru_lock, *flags); + goto retry; + } + rcu_read_unlock(); + + return lruvec; +} + /* * Compaction requires the taking of some coarse locks that are potentially * very heavily contended. The lock should be periodically unlocked to avo= id @@ -834,6 +853,7 @@ isolate_migratepages_block(struct compact_control *cc, = unsigned long low_pfn, =20 /* Time to isolate some pages for migration */ for (; low_pfn < end_pfn; low_pfn++) { + struct folio *folio; =20 if (skip_on_failure && low_pfn >=3D next_skip_pfn) { /* @@ -1055,18 +1075,17 @@ isolate_migratepages_block(struct compact_control *= cc, unsigned long low_pfn, if (!TestClearPageLRU(page)) goto isolate_fail_put; =20 - lruvec =3D folio_lruvec(page_folio(page)); + folio =3D page_folio(page); + lruvec =3D folio_lruvec(folio); =20 /* If we already hold the lock, we can skip some rechecking */ if (lruvec !=3D locked) { if (locked) lruvec_unlock_irqrestore(locked, flags); =20 - compact_lock_irqsave(&lruvec->lru_lock, &flags, cc); + lruvec =3D compact_folio_lruvec_lock_irqsave(folio, &flags, cc); locked =3D lruvec; =20 - lruvec_memcg_debug(lruvec, page_folio(page)); - /* Try get exclusive access under lock */ if (!skip_updated) { skip_updated =3D true; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 3c489651d312..6f171480b2f2 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1195,23 +1195,6 @@ int mem_cgroup_scan_tasks(struct mem_cgroup *memcg, return ret; } =20 -#ifdef CONFIG_DEBUG_VM -void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio) -{ - struct mem_cgroup *memcg; - - if (mem_cgroup_disabled()) - return; - - memcg =3D folio_memcg(folio); - - if (!memcg) - VM_BUG_ON_FOLIO(lruvec_memcg(lruvec) !=3D root_mem_cgroup, folio); - else - VM_BUG_ON_FOLIO(lruvec_memcg(lruvec) !=3D memcg, folio); -} -#endif - /** * folio_lruvec_lock - Lock the lruvec for a folio. * @folio: Pointer to the folio. @@ -1226,10 +1209,18 @@ void lruvec_memcg_debug(struct lruvec *lruvec, stru= ct folio *folio) */ struct lruvec *folio_lruvec_lock(struct folio *folio) { - struct lruvec *lruvec =3D folio_lruvec(folio); + struct lruvec *lruvec; =20 + rcu_read_lock(); +retry: + lruvec =3D folio_lruvec(folio); spin_lock(&lruvec->lru_lock); - lruvec_memcg_debug(lruvec, folio); + + if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) { + spin_unlock(&lruvec->lru_lock); + goto retry; + } + rcu_read_unlock(); =20 return lruvec; } @@ -1249,10 +1240,18 @@ struct lruvec *folio_lruvec_lock(struct folio *foli= o) */ struct lruvec *folio_lruvec_lock_irq(struct folio *folio) { - struct lruvec *lruvec =3D folio_lruvec(folio); + struct lruvec *lruvec; =20 + rcu_read_lock(); +retry: + lruvec =3D folio_lruvec(folio); spin_lock_irq(&lruvec->lru_lock); - lruvec_memcg_debug(lruvec, folio); + + if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) { + spin_unlock_irq(&lruvec->lru_lock); + goto retry; + } + rcu_read_unlock(); =20 return lruvec; } @@ -1274,10 +1273,18 @@ struct lruvec *folio_lruvec_lock_irq(struct folio *= folio) struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio, unsigned long *flags) { - struct lruvec *lruvec =3D folio_lruvec(folio); + struct lruvec *lruvec; =20 + rcu_read_lock(); +retry: + lruvec =3D folio_lruvec(folio); spin_lock_irqsave(&lruvec->lru_lock, *flags); - lruvec_memcg_debug(lruvec, folio); + + if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) { + spin_unlock_irqrestore(&lruvec->lru_lock, *flags); + goto retry; + } + rcu_read_unlock(); =20 return lruvec; } diff --git a/mm/swap.c b/mm/swap.c index 127ef4db394f..987dcbd93ffa 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -337,6 +337,11 @@ void lru_note_cost(struct lruvec *lruvec, bool file, u= nsigned int nr_pages) =20 void lru_note_cost_folio(struct folio *folio) { + WARN_ON_ONCE(!rcu_read_lock_held()); + /* + * The rcu read lock is held by the caller, so we do not need to + * care about the lruvec returned by folio_lruvec() being released. + */ lru_note_cost(folio_lruvec(folio), folio_is_file_lru(folio), folio_nr_pages(folio)); } --=20 2.11.0 From nobody Sat May 18 18:27:39 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13211C433EF for ; Tue, 21 Jun 2022 12:59:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351311AbiFUM7q (ORCPT ); Tue, 21 Jun 2022 08:59:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56720 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351014AbiFUM6X (ORCPT ); Tue, 21 Jun 2022 08:58:23 -0400 Received: from mail-pg1-x52a.google.com (mail-pg1-x52a.google.com [IPv6:2607:f8b0:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 609B5BC3 for ; Tue, 21 Jun 2022 05:58:07 -0700 (PDT) Received: by mail-pg1-x52a.google.com with SMTP id l4so13021911pgh.13 for ; Tue, 21 Jun 2022 05:58:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JQmNu1Pi61kiVMPJr8aGyJU8VoXoYJEPNAmwcIY/MLE=; b=uRoZYgA2WDdO9ticBdRw4R7/aUXk5r9fqjKxKJCs1SkEG4UdxhjFsO/X/PheSrZDg+ Q23F5hcZd9t0a7k63chGSWb3GHjtfCKitwmgpkZIMzFXmeCoTtJZt1WI45hmjAbSNvjU /5wRPFqGLSXBhc/RogtGwjV9RN90CBGRxKIoHDUJysh3Wr6Y4yJWuUtBVejjCQihuCj0 CtlKQVvYF4ukD9CjCMcdRjlepscIFcl4DKR7BS4rzRSK7Mf6uRzcc1U6qlPUu8rUerNI JY12moVGemqnvgCfDy9ZpJmjCyPFcuQ5O6SQvqUei19rSVIpmLCFjVox8NrK4WND0BYX Cqsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JQmNu1Pi61kiVMPJr8aGyJU8VoXoYJEPNAmwcIY/MLE=; b=hXQXhRNPbXPGKxii4LYiSjD7h6371sa6KljonwGGoU5LSOFiFlDpW0SNoiITXEpyaF WgS/qoDWe5oMiAX+jRVeipszKAGvGUF8m/YCVEW6boWjjUiicy/5wYSwKOcWXLVgPjkL DFL8faUmtWZKutPlsnCxP72dPaisoQ4fW7tfPDPuIwpbsgnXyx+IcteWT70ZTGlrdnFD Kbev258jC3lSOkxyCsd48OA9P8oRCii1JpmNsXGLmO8j2ogcRx0XazV6k1NgXNmqOfRN oxQjQ9CTrSbwv/ewIRkhNoxTvPLaM5o8s+xRb23cWe5eaDGa5/SfDJteuiseE/kcKtP3 KvzA== X-Gm-Message-State: AJIora+7jqFtDLkXpKA5NHPi0GTRnOyKQaIAEsmD+vpbwDyJb1dBn8ai tEoyZ8juHxug+7AtZ3gergMwfg== X-Google-Smtp-Source: AGRyM1upaoIcJWVsnc39MXQBtGK/kVVeFupBrAq4ZDu6K/3ZNA+9pZV71aSGLs8BkJ4IPnyCNTw3hA== X-Received: by 2002:a63:2b16:0:b0:3fa:faf9:e6d7 with SMTP id r22-20020a632b16000000b003fafaf9e6d7mr25873846pgr.325.1655816286866; Tue, 21 Jun 2022 05:58:06 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id e3-20020a170903240300b0015ea3a491a1sm10643134plo.191.2022.06.21.05.58.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jun 2022 05:58:06 -0700 (PDT) From: Muchun Song To: akpm@linux-foundation.org, hannes@cmpxchg.org, longman@redhat.com, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, duanxiongchun@bytedance.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH v6 05/11] mm: vmscan: rework move_pages_to_lru() Date: Tue, 21 Jun 2022 20:56:52 +0800 Message-Id: <20220621125658.64935-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220621125658.64935-1-songmuchun@bytedance.com> References: <20220621125658.64935-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" In the later patch, we will reparent the LRU pages. The pages moved to appropriate LRU list can be reparented during the process of the move_pages_to_lru(). So holding a lruvec lock by the caller is wrong, we should use the more general interface of folio_lruvec_relock_irq() to acquire the correct lruvec lock. Signed-off-by: Muchun Song Acked-by: Johannes Weiner Acked-by: Roman Gushchin --- mm/vmscan.c | 39 +++++++++++++++++++-------------------- 1 file changed, 19 insertions(+), 20 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 6a554712ef5d..697656151431 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2312,23 +2312,26 @@ static int too_many_isolated(struct pglist_data *pg= dat, int file, * move_pages_to_lru() moves folios from private @list to appropriate LRU = list. * On return, @list is reused as a list of folios to be freed by the calle= r. * - * Returns the number of pages moved to the given lruvec. + * Returns the number of pages moved to the appropriate LRU list. + * + * Note: The caller must not hold any lruvec lock. */ -static unsigned int move_pages_to_lru(struct lruvec *lruvec, - struct list_head *list) +static unsigned int move_pages_to_lru(struct list_head *list) { int nr_pages, nr_moved =3D 0; + struct lruvec *lruvec =3D NULL; LIST_HEAD(folios_to_free); =20 while (!list_empty(list)) { struct folio *folio =3D lru_to_folio(list); =20 + lruvec =3D folio_lruvec_relock_irq(folio, lruvec); VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); list_del(&folio->lru); if (unlikely(!folio_evictable(folio))) { - spin_unlock_irq(&lruvec->lru_lock); + lruvec_unlock_irq(lruvec); folio_putback_lru(folio); - spin_lock_irq(&lruvec->lru_lock); + lruvec =3D NULL; continue; } =20 @@ -2349,19 +2352,15 @@ static unsigned int move_pages_to_lru(struct lruvec= *lruvec, __folio_clear_lru_flags(folio); =20 if (unlikely(folio_test_large(folio))) { - spin_unlock_irq(&lruvec->lru_lock); + lruvec_unlock_irq(lruvec); destroy_large_folio(folio); - spin_lock_irq(&lruvec->lru_lock); + lruvec =3D NULL; } else list_add(&folio->lru, &folios_to_free); =20 continue; } =20 - /* - * All pages were isolated from the same lruvec (and isolation - * inhibits memcg migration). - */ VM_BUG_ON_FOLIO(!folio_matches_lruvec(folio, lruvec), folio); lruvec_add_folio(lruvec, folio); nr_pages =3D folio_nr_pages(folio); @@ -2370,6 +2369,8 @@ static unsigned int move_pages_to_lru(struct lruvec *= lruvec, workingset_age_nonresident(lruvec, nr_pages); } =20 + if (lruvec) + lruvec_unlock_irq(lruvec); /* * To save our caller's stack, now use input list for pages to free. */ @@ -2440,16 +2441,16 @@ shrink_inactive_list(unsigned long nr_to_scan, stru= ct lruvec *lruvec, =20 nr_reclaimed =3D shrink_page_list(&page_list, pgdat, sc, &stat, false); =20 - spin_lock_irq(&lruvec->lru_lock); - move_pages_to_lru(lruvec, &page_list); + move_pages_to_lru(&page_list); =20 + local_irq_disable(); __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); item =3D current_is_kswapd() ? PGSTEAL_KSWAPD : PGSTEAL_DIRECT; if (!cgroup_reclaim(sc)) __count_vm_events(item, nr_reclaimed); __count_memcg_events(lruvec_memcg(lruvec), item, nr_reclaimed); __count_vm_events(PGSTEAL_ANON + file, nr_reclaimed); - spin_unlock_irq(&lruvec->lru_lock); + local_irq_enable(); =20 lru_note_cost(lruvec, file, stat.nr_pageout); mem_cgroup_uncharge_list(&page_list); @@ -2578,18 +2579,16 @@ static void shrink_active_list(unsigned long nr_to_= scan, /* * Move folios back to the lru list. */ - spin_lock_irq(&lruvec->lru_lock); - - nr_activate =3D move_pages_to_lru(lruvec, &l_active); - nr_deactivate =3D move_pages_to_lru(lruvec, &l_inactive); + nr_activate =3D move_pages_to_lru(&l_active); + nr_deactivate =3D move_pages_to_lru(&l_inactive); /* Keep all free folios in l_active list */ list_splice(&l_inactive, &l_active); =20 + local_irq_disable(); __count_vm_events(PGDEACTIVATE, nr_deactivate); __count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate); - __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); - spin_unlock_irq(&lruvec->lru_lock); + local_irq_enable(); =20 mem_cgroup_uncharge_list(&l_active); free_unref_page_list(&l_active); --=20 2.11.0 From nobody Sat May 18 18:27:39 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6EDA6C433EF for ; Tue, 21 Jun 2022 12:59:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351353AbiFUM7u (ORCPT ); Tue, 21 Jun 2022 08:59:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351028AbiFUM6Y (ORCPT ); Tue, 21 Jun 2022 08:58:24 -0400 Received: from mail-pl1-x62f.google.com (mail-pl1-x62f.google.com [IPv6:2607:f8b0:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1698A389C for ; Tue, 21 Jun 2022 05:58:12 -0700 (PDT) Received: by mail-pl1-x62f.google.com with SMTP id d5so12431148plo.12 for ; Tue, 21 Jun 2022 05:58:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nIZConLAu+BQmuX/Tvb9zBVZtwEjqQiDdIX8payO2zo=; b=Zk51NSDrVMZPvZN3TRE8By5zbzxn4KfxBk7mAPZIGImd6sXrZQTiixROziyC4GK7i2 tjX3Uc/E0v4rj4SK78xIySUmqwcFv2/QqpusGj+so6BOEcPZY8a1BVGjEQEYoGh7bkkl OE/0bXu3mXp+aWvIm0cGBgL12WKiQodj2ZzTuAk2u3eLaReU+cqH1yGlGUUfgMhXcIsa pMZ2j0weSC5IJyI7Bg2HlgmEVFzD4vXJM29uR5XzcRBOtii439vkQTRNzX2HOBB+Ru/L ngbXnl0WALO2G1k49DSGVvpkOu3AFIwpwC5BwKEXviwXxw5OpAxEnvkkS7Z0wzEZO0oN dG0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nIZConLAu+BQmuX/Tvb9zBVZtwEjqQiDdIX8payO2zo=; b=Yrp/rhY6S4pkcCtPaELP+3zaSatdvNTJJk1Oj7gfJm0wn5T1C3M77N32JHoqydkSrv GPyvSFY+opCnHNjOTg9PxIAf1vAZuEoIZHW/2FhGjW5yXNcoPNzMoyXI/VhAmz3uQzdt S7OUbwLZliTmzCb9BKCtJT5oMtTgRd+vx15pmC3r0KXA77pGkjK0h6nlZniaCe+a4E1A 4rY4xQvXWXVjWrIuED1dvE1a0d/0lUZzDqPTliWEAuuMCw1OMLKOWHsKBDBqxrZMsdKV KBdqkUoY0T99qAXiSBVv2pBnlRwq9uEzwnuy48E/mNGPfcpItsUEgN410fTgmFa6GXLE DVfg== X-Gm-Message-State: AJIora+g0YVPrjfoTlqWGH8w5ic1xL8PJINB/EzWDTbjkYmDBFPA7pzd 5fANhm+dA++HvWQfuAKo/YCwIA== X-Google-Smtp-Source: AGRyM1sfMqtOuqujc10J2sSlGnIc2sn0IXLs0wAthMZegVzDh1Wtv0OMoc9MoJz2Lr6MQuwf390UoQ== X-Received: by 2002:a17:902:c40c:b0:16a:252c:ec82 with SMTP id k12-20020a170902c40c00b0016a252cec82mr10424049plk.5.1655816291697; Tue, 21 Jun 2022 05:58:11 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id e3-20020a170903240300b0015ea3a491a1sm10643134plo.191.2022.06.21.05.58.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jun 2022 05:58:11 -0700 (PDT) From: Muchun Song To: akpm@linux-foundation.org, hannes@cmpxchg.org, longman@redhat.com, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, duanxiongchun@bytedance.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH v6 06/11] mm: thp: make split queue lock safe when LRU pages are reparented Date: Tue, 21 Jun 2022 20:56:53 +0800 Message-Id: <20220621125658.64935-7-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220621125658.64935-1-songmuchun@bytedance.com> References: <20220621125658.64935-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Similar to the lruvec lock, we use the same approach to make the split queue lock safe when LRU pages are reparented. Signed-off-by: Muchun Song Acked-by: Roman Gushchin --- include/linux/memcontrol.h | 10 ++++ mm/huge_memory.c | 116 +++++++++++++++++++++++++++++++++++------= ---- 2 files changed, 100 insertions(+), 26 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index ff3106eca6f3..026b62b206b1 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1691,6 +1691,11 @@ int alloc_shrinker_info(struct mem_cgroup *memcg); void free_shrinker_info(struct mem_cgroup *memcg); void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); void reparent_shrinker_deferred(struct mem_cgroup *memcg); + +static inline int shrinker_id(struct shrinker *shrinker) +{ + return shrinker->id; +} #else #define mem_cgroup_sockets_enabled 0 static inline void mem_cgroup_sk_alloc(struct sock *sk) { }; @@ -1704,6 +1709,11 @@ static inline void set_shrinker_bit(struct mem_cgrou= p *memcg, int nid, int shrinker_id) { } + +static inline int shrinker_id(struct shrinker *shrinker) +{ + return -1; +} #endif =20 #ifdef CONFIG_MEMCG_KMEM diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 66d9ed8a1289..11ec92783b37 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -558,25 +558,90 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_str= uct *vma) } =20 #ifdef CONFIG_MEMCG -static inline struct deferred_split *get_deferred_split_queue(struct page = *page) +static inline struct mem_cgroup *folio_split_queue_memcg(struct folio *fol= io, + struct deferred_split *queue) { - struct mem_cgroup *memcg =3D page_memcg(compound_head(page)); - struct pglist_data *pgdat =3D NODE_DATA(page_to_nid(page)); + if (mem_cgroup_disabled()) + return NULL; + if (&NODE_DATA(folio_nid(folio))->deferred_split_queue =3D=3D queue) + return NULL; + return container_of(queue, struct mem_cgroup, deferred_split_queue); +} =20 - if (memcg) - return &memcg->deferred_split_queue; - else - return &pgdat->deferred_split_queue; +static inline struct deferred_split *folio_memcg_split_queue(struct folio = *folio) +{ + struct mem_cgroup *memcg =3D folio_memcg(folio); + + return memcg ? &memcg->deferred_split_queue : NULL; } #else -static inline struct deferred_split *get_deferred_split_queue(struct page = *page) +static inline struct mem_cgroup *folio_split_queue_memcg(struct folio *fol= io, + struct deferred_split *queue) { - struct pglist_data *pgdat =3D NODE_DATA(page_to_nid(page)); + return NULL; +} =20 - return &pgdat->deferred_split_queue; +static inline struct deferred_split *folio_memcg_split_queue(struct folio = *folio) +{ + return NULL; } #endif =20 +static struct deferred_split *folio_split_queue(struct folio *folio) +{ + struct deferred_split *queue =3D folio_memcg_split_queue(folio); + + return queue ? : &NODE_DATA(folio_nid(folio))->deferred_split_queue; +} + +static struct deferred_split *folio_split_queue_lock(struct folio *folio) +{ + struct deferred_split *queue; + + rcu_read_lock(); +retry: + queue =3D folio_split_queue(folio); + spin_lock(&queue->split_queue_lock); + + if (unlikely(folio_split_queue_memcg(folio, queue) !=3D folio_memcg(folio= ))) { + spin_unlock(&queue->split_queue_lock); + goto retry; + } + rcu_read_unlock(); + + return queue; +} + +static struct deferred_split * +folio_split_queue_lock_irqsave(struct folio *folio, unsigned long *flags) +{ + struct deferred_split *queue; + + rcu_read_lock(); +retry: + queue =3D folio_split_queue(folio); + spin_lock_irqsave(&queue->split_queue_lock, *flags); + + if (unlikely(folio_split_queue_memcg(folio, queue) !=3D folio_memcg(folio= ))) { + spin_unlock_irqrestore(&queue->split_queue_lock, *flags); + goto retry; + } + rcu_read_unlock(); + + return queue; +} + +static inline void split_queue_unlock(struct deferred_split *queue) +{ + spin_unlock(&queue->split_queue_lock); +} + +static inline void split_queue_unlock_irqrestore(struct deferred_split *qu= eue, + unsigned long flags) +{ + spin_unlock_irqrestore(&queue->split_queue_lock, flags); +} + void prep_transhuge_page(struct page *page) { /* @@ -2600,7 +2665,7 @@ int split_huge_page_to_list(struct page *page, struct= list_head *list) { struct folio *folio =3D page_folio(page); struct page *head =3D &folio->page; - struct deferred_split *ds_queue =3D get_deferred_split_queue(head); + struct deferred_split *ds_queue; XA_STATE(xas, &head->mapping->i_pages, head->index); struct anon_vma *anon_vma =3D NULL; struct address_space *mapping =3D NULL; @@ -2692,13 +2757,13 @@ int split_huge_page_to_list(struct page *page, stru= ct list_head *list) } =20 /* Prevent deferred_split_scan() touching ->_refcount */ - spin_lock(&ds_queue->split_queue_lock); + ds_queue =3D folio_split_queue_lock(folio); if (page_ref_freeze(head, 1 + extra_pins)) { if (!list_empty(page_deferred_list(head))) { ds_queue->split_queue_len--; list_del(page_deferred_list(head)); } - spin_unlock(&ds_queue->split_queue_lock); + split_queue_unlock(ds_queue); if (mapping) { int nr =3D thp_nr_pages(head); =20 @@ -2716,7 +2781,7 @@ int split_huge_page_to_list(struct page *page, struct= list_head *list) __split_huge_page(page, list, end); ret =3D 0; } else { - spin_unlock(&ds_queue->split_queue_lock); + split_queue_unlock(ds_queue); fail: if (mapping) xas_unlock(&xas); @@ -2740,25 +2805,23 @@ int split_huge_page_to_list(struct page *page, stru= ct list_head *list) =20 void free_transhuge_page(struct page *page) { - struct deferred_split *ds_queue =3D get_deferred_split_queue(page); + struct deferred_split *ds_queue; unsigned long flags; =20 - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); + ds_queue =3D folio_split_queue_lock_irqsave(page_folio(page), &flags); if (!list_empty(page_deferred_list(page))) { ds_queue->split_queue_len--; list_del(page_deferred_list(page)); } - spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); + split_queue_unlock_irqrestore(ds_queue, flags); free_compound_page(page); } =20 void deferred_split_huge_page(struct page *page) { - struct deferred_split *ds_queue =3D get_deferred_split_queue(page); -#ifdef CONFIG_MEMCG - struct mem_cgroup *memcg =3D page_memcg(compound_head(page)); -#endif + struct deferred_split *ds_queue; unsigned long flags; + struct folio *folio =3D page_folio(page); =20 VM_BUG_ON_PAGE(!PageTransHuge(page), page); =20 @@ -2775,18 +2838,19 @@ void deferred_split_huge_page(struct page *page) if (PageSwapCache(page)) return; =20 - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); + ds_queue =3D folio_split_queue_lock_irqsave(folio, &flags); if (list_empty(page_deferred_list(page))) { + struct mem_cgroup *memcg; + + memcg =3D folio_split_queue_memcg(folio, ds_queue); count_vm_event(THP_DEFERRED_SPLIT_PAGE); list_add_tail(page_deferred_list(page), &ds_queue->split_queue); ds_queue->split_queue_len++; -#ifdef CONFIG_MEMCG if (memcg) set_shrinker_bit(memcg, page_to_nid(page), - deferred_split_shrinker.id); -#endif + shrinker_id(&deferred_split_shrinker)); } - spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); + split_queue_unlock_irqrestore(ds_queue, flags); } =20 static unsigned long deferred_split_count(struct shrinker *shrink, --=20 2.11.0 From nobody Sat May 18 18:27:39 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85F77C43334 for ; Tue, 21 Jun 2022 12:59:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351079AbiFUM7D (ORCPT ); Tue, 21 Jun 2022 08:59:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56880 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351100AbiFUM62 (ORCPT ); Tue, 21 Jun 2022 08:58:28 -0400 Received: from mail-pj1-x1034.google.com (mail-pj1-x1034.google.com [IPv6:2607:f8b0:4864:20::1034]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DD42ADEC3 for ; Tue, 21 Jun 2022 05:58:17 -0700 (PDT) Received: by mail-pj1-x1034.google.com with SMTP id y13-20020a17090a154d00b001eaaa3b9b8dso13297773pja.2 for ; Tue, 21 Jun 2022 05:58:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FY/yyaqIA+0UtXl+c+2/CMp1DjrnJ12MKuKhyE1HfHU=; b=Zea++/WGVZTO0KrQJEfldWpAcOWAaI48Jtbz0zDDjspXNq0Je6rF/v2EPq9xOK1S1J 0oPhK7zmf1lsxmyg+s4mVdjYZTk9eTqeXka+6G5SUo0W5JEV9GtmUfgAtsLcNOu9EFKG ddgnYJxu4rhdMX9NoV28KjnkyEgOZblD6QlVqounYQqDKyWB2zPn2wDoGDpD/YDPyLm8 DfPqoOodi0yktIftubkOblm5fzrj9TgFJLmg7fCEOYGluDlrclpzFOhBPnam04gtAZk9 Nd3Hi7J1YIALTLQhbEKF3eF8bwhRi96aOqxpeD/Vshcr+0kJv7OARDDPY8i3fcxkudVs GPzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FY/yyaqIA+0UtXl+c+2/CMp1DjrnJ12MKuKhyE1HfHU=; b=fQ6W5a5Q7ClPSBD8mYrfTaZ3eNf8HK6mApM3/RTFSKlWweog8VLRfs1snaWTvRQ4yJ 4sQYIcyKmipQ/vJDlOfvM7A2fSvtUSocZ1sR/9yGg1gDsPzW8Jlo8JDNCXwul9QIP8UP +EEYEoQ50675Gr94gjjsdRa1Tbp3XyaySO0bjKgYv8O4C5bY8dL5Fty89QqbAmwgWrAX J97x2G0Wnzkq2rZ1Xk6ufq1+0nhuKnF07GkKcwkbnKLG3S69Ev0Xfd0YSyptOVNSUFos 4QiCUPQzEcFSt4H79keOjQy3hE1vv3FqSk1E5ueCg8o0Z3BTbv+RbZ6wtI4CZq9cR7Uw Po/g== X-Gm-Message-State: AJIora834b6IKdgQ8XuK+kTp/zL2diWqCxHwjCXUTuNre9UUDcF/vWtx BSHuZvq7VXIs7PYX9kA7r/9vgA== X-Google-Smtp-Source: AGRyM1tEcyaTtEUyzQ5EMeEUCXTRMvihuBmKI3S8O+v82E+buq0zDoMqkvtDqvccflyBf0mVxJ/TyQ== X-Received: by 2002:a17:90a:8b05:b0:1ea:f5e7:daf2 with SMTP id y5-20020a17090a8b0500b001eaf5e7daf2mr29921210pjn.160.1655816296535; Tue, 21 Jun 2022 05:58:16 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id e3-20020a170903240300b0015ea3a491a1sm10643134plo.191.2022.06.21.05.58.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jun 2022 05:58:16 -0700 (PDT) From: Muchun Song To: akpm@linux-foundation.org, hannes@cmpxchg.org, longman@redhat.com, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, duanxiongchun@bytedance.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH v6 07/11] mm: memcontrol: make all the callers of {folio,page}_memcg() safe Date: Tue, 21 Jun 2022 20:56:54 +0800 Message-Id: <20220621125658.64935-8-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220621125658.64935-1-songmuchun@bytedance.com> References: <20220621125658.64935-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When we use objcg APIs to charge the LRU pages, the page will not hold a reference to the memcg associated with the page. So the caller of the {folio,page}_memcg() should hold an rcu read lock or obtain a reference to the memcg associated with the page to protect memcg from being released. So introduce get_mem_cgroup_from_{page,folio}() to obtain a reference to the memory cgroup associated with the page. In this patch, make all the callers hold an rcu read lock or obtain a reference to the memcg to protect memcg from being released when the LRU pages reparented. We do not need to adjust the callers of {folio,page}_memcg() during the whole process of mem_cgroup_move_task(). Because the cgroup migration and memory cgroup offlining are serialized by @cgroup_mutex. In this routine, the LRU pages cannot be reparented to its parent memory cgroup. So {folio,page}_memcg() is stable and cannot be released. This is a preparation for reparenting the LRU pages. Signed-off-by: Muchun Song Acked-by: Roman Gushchin --- fs/buffer.c | 4 +-- fs/fs-writeback.c | 23 +++++++------- include/linux/memcontrol.h | 66 +++++++++++++++++++++++++++++++++---= -- include/trace/events/writeback.h | 5 +++ mm/memcontrol.c | 68 +++++++++++++++++++++++++++++-------= ---- mm/migrate.c | 4 +++ mm/page_io.c | 5 +-- 7 files changed, 135 insertions(+), 40 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index 898c7f301b1b..04ec53f327e4 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -819,8 +819,7 @@ struct buffer_head *alloc_page_buffers(struct page *pag= e, unsigned long size, if (retry) gfp |=3D __GFP_NOFAIL; =20 - /* The page lock pins the memcg */ - memcg =3D page_memcg(page); + memcg =3D get_mem_cgroup_from_page(page); old_memcg =3D set_active_memcg(memcg); =20 head =3D NULL; @@ -840,6 +839,7 @@ struct buffer_head *alloc_page_buffers(struct page *pag= e, unsigned long size, set_bh_page(bh, page, offset); } out: + mem_cgroup_put(memcg); set_active_memcg(old_memcg); return head; /* diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c index 05221366a16d..1cbac56c810b 100644 --- a/fs/fs-writeback.c +++ b/fs/fs-writeback.c @@ -244,15 +244,13 @@ void __inode_attach_wb(struct inode *inode, struct pa= ge *page) if (inode_cgwb_enabled(inode)) { struct cgroup_subsys_state *memcg_css; =20 - if (page) { - memcg_css =3D mem_cgroup_css_from_page(page); - wb =3D wb_get_create(bdi, memcg_css, GFP_ATOMIC); - } else { - /* must pin memcg_css, see wb_get_create() */ + /* must pin memcg_css, see wb_get_create() */ + if (page) + memcg_css =3D get_mem_cgroup_css_from_page(page); + else memcg_css =3D task_get_css(current, memory_cgrp_id); - wb =3D wb_get_create(bdi, memcg_css, GFP_ATOMIC); - css_put(memcg_css); - } + wb =3D wb_get_create(bdi, memcg_css, GFP_ATOMIC); + css_put(memcg_css); } =20 if (!wb) @@ -869,16 +867,16 @@ void wbc_account_cgroup_owner(struct writeback_contro= l *wbc, struct page *page, if (!wbc->wb || wbc->no_cgroup_owner) return; =20 - css =3D mem_cgroup_css_from_page(page); + css =3D get_mem_cgroup_css_from_page(page); /* dead cgroups shouldn't contribute to inode ownership arbitration */ if (!(css->flags & CSS_ONLINE)) - return; + goto out; =20 id =3D css->id; =20 if (id =3D=3D wbc->wb_id) { wbc->wb_bytes +=3D bytes; - return; + goto out; } =20 if (id =3D=3D wbc->wb_lcand_id) @@ -891,6 +889,9 @@ void wbc_account_cgroup_owner(struct writeback_control = *wbc, struct page *page, wbc->wb_tcand_bytes +=3D bytes; else wbc->wb_tcand_bytes -=3D min(bytes, wbc->wb_tcand_bytes); + +out: + css_put(css); } EXPORT_SYMBOL_GPL(wbc_account_cgroup_owner); =20 diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 026b62b206b1..a8bd4bb39502 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -379,7 +379,7 @@ static inline bool folio_memcg_kmem(struct folio *folio= ); * a valid memcg, but can be atomically swapped to the parent memcg. * * The caller must ensure that the returned memcg won't be released: - * e.g. acquire the rcu_read_lock or css_set_lock. + * e.g. acquire the rcu_read_lock or objcg_lock or cgroup_mutex. */ static inline struct mem_cgroup *obj_cgroup_memcg(struct obj_cgroup *objcg) { @@ -445,8 +445,8 @@ static inline struct obj_cgroup *__folio_objcg(struct f= olio *folio) * - lock_page_memcg() * - exclusive reference * - * For a kmem folio a caller should hold an rcu read lock to protect memcg - * associated with a kmem folio from being released. + * Note: The caller should hold an rcu read lock to protect memcg associat= ed + * with a folio from being released. */ static inline struct mem_cgroup *folio_memcg(struct folio *folio) { @@ -455,12 +455,48 @@ static inline struct mem_cgroup *folio_memcg(struct f= olio *folio) return __folio_memcg(folio); } =20 +/* + * page_memcg - Get the memory cgroup associated with a page. + * @page: Pointer to the page. + * + * See the cooments in folio_memcg(). + */ static inline struct mem_cgroup *page_memcg(struct page *page) { return folio_memcg(page_folio(page)); } =20 -/** +/* + * get_mem_cgroup_from_folio - Obtain a reference on the memory cgroup + * associated with a folio. + * @folio: Pointer to the folio. + * + * Returns a pointer to the memory cgroup (and obtain a reference on it) + * associated with the folio, or NULL. This function assumes that the + * folio is known to have a proper memory cgroup pointer. It's not safe + * to call this function against some type of pages, e.g. slab pages or + * ex-slab pages. + */ +static inline struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *f= olio) +{ + struct mem_cgroup *memcg; + + rcu_read_lock(); +retry: + memcg =3D folio_memcg(folio); + if (unlikely(memcg && !css_tryget(&memcg->css))) + goto retry; + rcu_read_unlock(); + + return memcg; +} + +static inline struct mem_cgroup *get_mem_cgroup_from_page(struct page *pag= e) +{ + return get_mem_cgroup_from_folio(page_folio(page)); +} + +/* * folio_memcg_rcu - Locklessly get the memory cgroup associated with a fo= lio. * @folio: Pointer to the folio. * @@ -888,7 +924,7 @@ static inline bool mm_match_cgroup(struct mm_struct *mm, return match; } =20 -struct cgroup_subsys_state *mem_cgroup_css_from_page(struct page *page); +struct cgroup_subsys_state *get_mem_cgroup_css_from_page(struct page *page= ); ino_t page_cgroup_ino(struct page *page); =20 static inline bool mem_cgroup_online(struct mem_cgroup *memcg) @@ -1058,19 +1094,25 @@ static inline void count_memcg_events(struct mem_cg= roup *memcg, static inline void count_memcg_page_event(struct page *page, enum vm_event_item idx) { - struct mem_cgroup *memcg =3D page_memcg(page); + struct mem_cgroup *memcg; =20 + rcu_read_lock(); + memcg =3D page_memcg(page); if (memcg) count_memcg_events(memcg, idx, 1); + rcu_read_unlock(); } =20 static inline void count_memcg_folio_events(struct folio *folio, enum vm_event_item idx, unsigned long nr) { - struct mem_cgroup *memcg =3D folio_memcg(folio); + struct mem_cgroup *memcg; =20 + rcu_read_lock(); + memcg =3D folio_memcg(folio); if (memcg) count_memcg_events(memcg, idx, nr); + rcu_read_unlock(); } =20 static inline void count_memcg_event_mm(struct mm_struct *mm, @@ -1149,6 +1191,16 @@ static inline struct mem_cgroup *page_memcg(struct p= age *page) return NULL; } =20 +static inline struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *f= olio) +{ + return NULL; +} + +static inline struct mem_cgroup *get_mem_cgroup_from_page(struct page *pag= e) +{ + return NULL; +} + static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio) { WARN_ON_ONCE(!rcu_read_lock_held()); diff --git a/include/trace/events/writeback.h b/include/trace/events/writeb= ack.h index 86b2a82da546..cdb822339f13 100644 --- a/include/trace/events/writeback.h +++ b/include/trace/events/writeback.h @@ -258,6 +258,11 @@ TRACE_EVENT(track_foreign_dirty, __entry->ino =3D inode ? inode->i_ino : 0; __entry->memcg_id =3D wb->memcg_css->id; __entry->cgroup_ino =3D __trace_wb_assign_cgroup(wb); + /* + * TP_fast_assign() is under preemption disabled which can + * serve as an RCU read-side critical section so that the + * memcg returned by folio_memcg() cannot be freed. + */ __entry->page_cgroup_ino =3D cgroup_ino(folio_memcg(folio)->css.cgroup); ), =20 diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 6f171480b2f2..346a954e190e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -369,7 +369,7 @@ EXPORT_SYMBOL(memcg_kmem_enabled_key); #endif =20 /** - * mem_cgroup_css_from_page - css of the memcg associated with a page + * get_mem_cgroup_css_from_page - get css of the memcg associated with a p= age * @page: page of interest * * If memcg is bound to the default hierarchy, css of the memcg associated @@ -379,13 +379,15 @@ EXPORT_SYMBOL(memcg_kmem_enabled_key); * If memcg is bound to a traditional hierarchy, the css of root_mem_cgroup * is returned. */ -struct cgroup_subsys_state *mem_cgroup_css_from_page(struct page *page) +struct cgroup_subsys_state *get_mem_cgroup_css_from_page(struct page *page) { struct mem_cgroup *memcg; =20 - memcg =3D page_memcg(page); + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + return &root_mem_cgroup->css; =20 - if (!memcg || !cgroup_subsys_on_dfl(memory_cgrp_subsys)) + memcg =3D get_mem_cgroup_from_page(page); + if (!memcg) memcg =3D root_mem_cgroup; =20 return &memcg->css; @@ -768,13 +770,13 @@ void __mod_lruvec_state(struct lruvec *lruvec, enum n= ode_stat_item idx, void __mod_lruvec_page_state(struct page *page, enum node_stat_item idx, int val) { - struct page *head =3D compound_head(page); /* rmap on tail pages */ + struct folio *folio =3D page_folio(page); /* rmap on tail pages */ struct mem_cgroup *memcg; pg_data_t *pgdat =3D page_pgdat(page); struct lruvec *lruvec; =20 rcu_read_lock(); - memcg =3D page_memcg(head); + memcg =3D folio_memcg(folio); /* Untracked pages have no memcg, no lruvec. Update only the node */ if (!memcg) { rcu_read_unlock(); @@ -2056,7 +2058,9 @@ void folio_memcg_lock(struct folio *folio) * The RCU lock is held throughout the transaction. The fast * path can get away without acquiring the memcg->move_lock * because page moving starts with an RCU grace period. - */ + * + * The RCU lock also protects the memcg from being freed. + */ rcu_read_lock(); =20 if (mem_cgroup_disabled()) @@ -3353,7 +3357,7 @@ void obj_cgroup_uncharge(struct obj_cgroup *objcg, si= ze_t size) void split_page_memcg(struct page *head, unsigned int nr) { struct folio *folio =3D page_folio(head); - struct mem_cgroup *memcg =3D folio_memcg(folio); + struct mem_cgroup *memcg =3D get_mem_cgroup_from_folio(folio); int i; =20 if (mem_cgroup_disabled() || !memcg) @@ -3366,6 +3370,8 @@ void split_page_memcg(struct page *head, unsigned int= nr) obj_cgroup_get_many(__folio_objcg(folio), nr - 1); else css_get_many(&memcg->css, nr - 1); + + css_put(&memcg->css); } =20 #ifdef CONFIG_MEMCG_SWAP @@ -4558,7 +4564,7 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, un= signed long *pfilepages, void mem_cgroup_track_foreign_dirty_slowpath(struct folio *folio, struct bdi_writeback *wb) { - struct mem_cgroup *memcg =3D folio_memcg(folio); + struct mem_cgroup *memcg =3D get_mem_cgroup_from_folio(folio); struct memcg_cgwb_frn *frn; u64 now =3D get_jiffies_64(); u64 oldest_at =3D now; @@ -4605,6 +4611,7 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct f= olio *folio, frn->memcg_id =3D wb->memcg_css->id; frn->at =3D now; } + css_put(&memcg->css); } =20 /* issue foreign writeback flushes for recorded foreign dirtying events */ @@ -6167,6 +6174,14 @@ static void mem_cgroup_move_charge(void) atomic_dec(&mc.from->moving_account); } =20 +/* + * The cgroup migration and memory cgroup offlining are serialized by + * @cgroup_mutex. If we reach here, it means that the LRU pages cannot + * be reparented to its parent memory cgroup. So during the whole process + * of mem_cgroup_move_task(), page_memcg(page) is stable. So we do not + * need to worry about the memcg (returned from page_memcg()) being + * released even if we do not hold an rcu read lock. + */ static void mem_cgroup_move_task(void) { if (mc.to) { @@ -7025,7 +7040,7 @@ void mem_cgroup_migrate(struct folio *old, struct fol= io *new) if (folio_memcg(new)) return; =20 - memcg =3D folio_memcg(old); + memcg =3D get_mem_cgroup_from_folio(old); VM_WARN_ON_ONCE_FOLIO(!memcg, old); if (!memcg) return; @@ -7044,6 +7059,8 @@ void mem_cgroup_migrate(struct folio *old, struct fol= io *new) mem_cgroup_charge_statistics(memcg, nr_pages); memcg_check_events(memcg, folio_nid(new)); local_irq_restore(flags); + + css_put(&memcg->css); } =20 DEFINE_STATIC_KEY_FALSE(memcg_sockets_enabled_key); @@ -7228,6 +7245,10 @@ void mem_cgroup_swapout(struct folio *folio, swp_ent= ry_t entry) if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) return; =20 + /* + * Interrupts should be disabled by the caller (see the comments below), + * which can serve as RCU read-side critical sections. + */ memcg =3D folio_memcg(folio); =20 VM_WARN_ON_ONCE_FOLIO(!memcg, folio); @@ -7289,19 +7310,21 @@ int __mem_cgroup_try_charge_swap(struct folio *foli= o, swp_entry_t entry) struct page_counter *counter; struct mem_cgroup *memcg; unsigned short oldid; + int ret =3D 0; =20 if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) return 0; =20 + rcu_read_lock(); memcg =3D folio_memcg(folio); =20 VM_WARN_ON_ONCE_FOLIO(!memcg, folio); if (!memcg) - return 0; + goto out; =20 if (!entry.val) { memcg_memory_event(memcg, MEMCG_SWAP_FAIL); - return 0; + goto out; } =20 memcg =3D mem_cgroup_id_get_online(memcg); @@ -7311,7 +7334,8 @@ int __mem_cgroup_try_charge_swap(struct folio *folio,= swp_entry_t entry) memcg_memory_event(memcg, MEMCG_SWAP_MAX); memcg_memory_event(memcg, MEMCG_SWAP_FAIL); mem_cgroup_id_put(memcg); - return -ENOMEM; + ret =3D -ENOMEM; + goto out; } =20 /* Get references for the tail pages, too */ @@ -7320,8 +7344,10 @@ int __mem_cgroup_try_charge_swap(struct folio *folio= , swp_entry_t entry) oldid =3D swap_cgroup_record(entry, mem_cgroup_id(memcg), nr_pages); VM_BUG_ON_FOLIO(oldid, folio); mod_memcg_state(memcg, MEMCG_SWAP, nr_pages); +out: + rcu_read_unlock(); =20 - return 0; + return ret; } =20 /** @@ -7366,6 +7392,7 @@ long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *= memcg) bool mem_cgroup_swap_full(struct page *page) { struct mem_cgroup *memcg; + bool ret =3D false; =20 VM_BUG_ON_PAGE(!PageLocked(page), page); =20 @@ -7374,19 +7401,24 @@ bool mem_cgroup_swap_full(struct page *page) if (cgroup_memory_noswap || !cgroup_subsys_on_dfl(memory_cgrp_subsys)) return false; =20 + rcu_read_lock(); memcg =3D page_memcg(page); if (!memcg) - return false; + goto out; =20 for (; memcg !=3D root_mem_cgroup; memcg =3D parent_mem_cgroup(memcg)) { unsigned long usage =3D page_counter_read(&memcg->swap); =20 if (usage * 2 >=3D READ_ONCE(memcg->swap.high) || - usage * 2 >=3D READ_ONCE(memcg->swap.max)) - return true; + usage * 2 >=3D READ_ONCE(memcg->swap.max)) { + ret =3D true; + goto out; + } } +out: + rcu_read_unlock(); =20 - return false; + return ret; } =20 static int __init setup_swap_account(char *s) diff --git a/mm/migrate.c b/mm/migrate.c index 1ece23d80bc4..2e49b96fa339 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -451,6 +451,10 @@ int folio_migrate_mapping(struct address_space *mappin= g, struct lruvec *old_lruvec, *new_lruvec; struct mem_cgroup *memcg; =20 + /* + * Irq is disabled, which can serve as RCU read-side critical + * sections. + */ memcg =3D folio_memcg(folio); old_lruvec =3D mem_cgroup_lruvec(memcg, oldzone->zone_pgdat); new_lruvec =3D mem_cgroup_lruvec(memcg, newzone->zone_pgdat); diff --git a/mm/page_io.c b/mm/page_io.c index 68318134dc92..f75ebbc95ee6 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -222,13 +222,14 @@ static void bio_associate_blkg_from_page(struct bio *= bio, struct page *page) struct cgroup_subsys_state *css; struct mem_cgroup *memcg; =20 + rcu_read_lock(); memcg =3D page_memcg(page); if (!memcg) - return; + goto out; =20 - rcu_read_lock(); css =3D cgroup_e_css(memcg->css.cgroup, &io_cgrp_subsys); bio_associate_blkg_from_css(bio, css); +out: rcu_read_unlock(); } #else --=20 2.11.0 From nobody Sat May 18 18:27:39 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 528D0C43334 for ; Tue, 21 Jun 2022 12:59:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351287AbiFUM7U (ORCPT ); Tue, 21 Jun 2022 08:59:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56892 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351126AbiFUM62 (ORCPT ); Tue, 21 Jun 2022 08:58:28 -0400 Received: from mail-pf1-x42a.google.com (mail-pf1-x42a.google.com [IPv6:2607:f8b0:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 525E62612E for ; Tue, 21 Jun 2022 05:58:22 -0700 (PDT) Received: by mail-pf1-x42a.google.com with SMTP id t21so6773624pfq.1 for ; Tue, 21 Jun 2022 05:58:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=R9q0ch1BG5yy3uNoTKVJadkT0JjG8vQh5rsCMZAiDeM=; b=X1ymgAzrECJrNEqT0zu/qYqioj+MK14RMF+IxcAS9y0uCpcuMThML5qsMfwkh5Syqb a+aAQz+na0wNKw5ONdOQH1YzYWTCg9aI9VoZg1AiFiGcFm5GNrdmqoniS/jFYAWZB9U0 x7hnUcvnD2Uwvu2dL/27+ftKyQYgBnzgc38FLw8CWdGOZanoyKBMLvP036HHHz7f77de /57RM1HmPaKRHnE6aN43rnTmSuQnUSdoWtTVyMJC9jbL9ijPukp0Xg3q3OywkZdjchvG /bQ8emT1SlpJguuihXT//Xhcgv32+AVxjjynF7sfuZCkEaISt9UJisvUrCMCsdduD4Ug QLPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=R9q0ch1BG5yy3uNoTKVJadkT0JjG8vQh5rsCMZAiDeM=; b=qZdRGNwoB08lCMWQf5na9M+A+tzcrGG3CQxcv//N3YHQOoZp6LE18Onu/scrCPZ7+6 dOjkMLZdV8gE7czxsxqe5pFM0os7//9n+PG87a1DECV+jsJxrdhcnS9KOT8ixbKf9KzX IN3rzDPhCY6cIYIQlfFF7eAs1V9621XR/OCESLiTStXmXtKwTKBOs9KdkhZFpgK8icCM 6ZW8izOdKVzLN2ggiilr0xdXqGOghrj6Y0RmhC/bd49T3f/mYIIiR8Xz6s1PWzoA5xHt jlgo/2Vi5Vxa5DTl2PHWv/ikzssBvDivh+aeXdh/B448UPjgjK52j5GQhDct9fnb/ZZE MHqA== X-Gm-Message-State: AJIora9H5tpHiWWyn2l9SjirthGE48BYfnA2XnrloXbCHYTwiOQFEd78 rN+Dchm0iPCoLZHyiTyROirddg== X-Google-Smtp-Source: AGRyM1vJmCnTypP6KX9NHCb9AyM5aDswtoVUk0Wth7I38QLP7dlJmEjPt2tfj2n1yyM2/5DwhBTAJQ== X-Received: by 2002:a63:5155:0:b0:400:14af:a1e6 with SMTP id r21-20020a635155000000b0040014afa1e6mr25830864pgl.71.1655816301516; Tue, 21 Jun 2022 05:58:21 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id e3-20020a170903240300b0015ea3a491a1sm10643134plo.191.2022.06.21.05.58.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jun 2022 05:58:21 -0700 (PDT) From: Muchun Song To: akpm@linux-foundation.org, hannes@cmpxchg.org, longman@redhat.com, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, duanxiongchun@bytedance.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH v6 08/11] mm: memcontrol: introduce memcg_reparent_ops Date: Tue, 21 Jun 2022 20:56:55 +0800 Message-Id: <20220621125658.64935-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220621125658.64935-1-songmuchun@bytedance.com> References: <20220621125658.64935-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" In the previous patch, we know how to make the lruvec lock safe when LRU pages are reparented. We should do something like following. memcg_reparent_objcgs(memcg) 1) lock // lruvec belongs to memcg and lruvec_parent belongs to parent memc= g. spin_lock(&lruvec->lru_lock); spin_lock(&lruvec_parent->lru_lock); 2) relocate from current memcg to its parent // Move all the pages from the lruvec list to the parent lruvec lis= t. 3) unlock spin_unlock(&lruvec_parent->lru_lock); spin_unlock(&lruvec->lru_lock); Apart from the page lruvec lock, the deferred split queue lock (THP only) also needs to do something similar. So we extract the necessary three steps in the memcg_reparent_objcgs(). memcg_reparent_objcgs(memcg) 1) lock memcg_reparent_ops->lock(memcg, parent); 2) relocate memcg_reparent_ops->relocate(memcg, reparent); 3) unlock memcg_reparent_ops->unlock(memcg, reparent); Now there are two different locks (e.g. lruvec lock and deferred split queue lock) need to use this infrastructure. In the next patch, we will use those APIs to make those locks safe when the LRU pages reparented. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 20 +++++++++++++++ mm/memcontrol.c | 62 ++++++++++++++++++++++++++++++++++++------= ---- 2 files changed, 69 insertions(+), 13 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index a8bd4bb39502..63dbdef60cbd 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -353,6 +353,26 @@ struct mem_cgroup { struct mem_cgroup_per_node *nodeinfo[]; }; =20 +struct memcg_reparent_ops { + /* + * Note that interrupt is disabled before calling those callbacks, + * so the interrupt should remain disabled when leaving those callbacks. + */ + void (*lock)(struct mem_cgroup *src, struct mem_cgroup *dst); + void (*relocate)(struct mem_cgroup *src, struct mem_cgroup *dst); + void (*unlock)(struct mem_cgroup *src, struct mem_cgroup *dst); +}; + +#define DEFINE_MEMCG_REPARENT_OPS(name) \ + const struct memcg_reparent_ops memcg_##name##_reparent_ops =3D { \ + .lock =3D name##_reparent_lock, \ + .relocate =3D name##_reparent_relocate, \ + .unlock =3D name##_reparent_unlock, \ + } + +#define DECLARE_MEMCG_REPARENT_OPS(name) \ + extern const struct memcg_reparent_ops memcg_##name##_reparent_ops + /* * size of first charge trial. "32" comes from vmscan.c's magic value. * TODO: maybe necessary to use big numbers in big irons. diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 346a954e190e..6ef3a264054e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -335,24 +335,60 @@ static struct obj_cgroup *obj_cgroup_alloc(void) return objcg; } =20 -static void memcg_reparent_objcgs(struct mem_cgroup *memcg) +static void objcg_reparent_lock(struct mem_cgroup *src, struct mem_cgroup = *dst) +{ + spin_lock(&objcg_lock); +} + +static void objcg_reparent_relocate(struct mem_cgroup *src, struct mem_cgr= oup *dst) { struct obj_cgroup *objcg, *iter; - struct mem_cgroup *parent =3D parent_mem_cgroup(memcg); =20 - objcg =3D rcu_replace_pointer(memcg->objcg, NULL, true); + objcg =3D rcu_replace_pointer(src->objcg, NULL, true); + /* 1) Ready to reparent active objcg. */ + list_add(&objcg->list, &src->objcg_list); + /* 2) Reparent active objcg and already reparented objcgs to dst. */ + list_for_each_entry(iter, &src->objcg_list, list) + WRITE_ONCE(iter->memcg, dst); + /* 3) Move already reparented objcgs to the dst's list */ + list_splice(&src->objcg_list, &dst->objcg_list); +} + +static void objcg_reparent_unlock(struct mem_cgroup *src, struct mem_cgrou= p *dst) +{ + spin_unlock(&objcg_lock); +} =20 - spin_lock_irq(&objcg_lock); +static DEFINE_MEMCG_REPARENT_OPS(objcg); =20 - /* 1) Ready to reparent active objcg. */ - list_add(&objcg->list, &memcg->objcg_list); - /* 2) Reparent active objcg and already reparented objcgs to parent. */ - list_for_each_entry(iter, &memcg->objcg_list, list) - WRITE_ONCE(iter->memcg, parent); - /* 3) Move already reparented objcgs to the parent's list */ - list_splice(&memcg->objcg_list, &parent->objcg_list); - - spin_unlock_irq(&objcg_lock); +static const struct memcg_reparent_ops *memcg_reparent_ops[] =3D { + &memcg_objcg_reparent_ops, +}; + +#define DEFINE_MEMCG_REPARENT_FUNC(phase) \ + static void memcg_reparent_##phase(struct mem_cgroup *src, \ + struct mem_cgroup *dst) \ + { \ + int i; \ + \ + for (i =3D 0; i < ARRAY_SIZE(memcg_reparent_ops); i++) \ + memcg_reparent_ops[i]->phase(src, dst); \ + } + +DEFINE_MEMCG_REPARENT_FUNC(lock) +DEFINE_MEMCG_REPARENT_FUNC(relocate) +DEFINE_MEMCG_REPARENT_FUNC(unlock) + +static void memcg_reparent_objcgs(struct mem_cgroup *src) +{ + struct mem_cgroup *dst =3D parent_mem_cgroup(src); + struct obj_cgroup *objcg =3D rcu_dereference_protected(src->objcg, true); + + local_irq_disable(); + memcg_reparent_lock(src, dst); + memcg_reparent_relocate(src, dst); + memcg_reparent_unlock(src, dst); + local_irq_enable(); =20 percpu_ref_kill(&objcg->refcnt); } --=20 2.11.0 From nobody Sat May 18 18:27:39 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD18BC433EF for ; Tue, 21 Jun 2022 12:59:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351162AbiFUM7K (ORCPT ); Tue, 21 Jun 2022 08:59:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57084 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351136AbiFUM6c (ORCPT ); Tue, 21 Jun 2022 08:58:32 -0400 Received: from mail-pg1-x531.google.com (mail-pg1-x531.google.com [IPv6:2607:f8b0:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4793827C for ; Tue, 21 Jun 2022 05:58:27 -0700 (PDT) Received: by mail-pg1-x531.google.com with SMTP id 184so13037170pga.12 for ; Tue, 21 Jun 2022 05:58:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5AnDR3mseJ3BEgaMu0BeAFWG5+UZ/bYL9bd8XK/xdNA=; b=ITb3mmcKa2a/9qMdm4l7JXtXf2d2YXg9lRRzkyXz8AmY6Ne+Se4c1Ya/dhgZnZz/yb ea2qQSDYfxXUhtpoNqD3TAHbZkYFygTSKsUre8MnK50H+Z8RyVbdraHhlF/4NxzD80N2 l3LjsH0u0t3PE3jNhTqyVgOtchf149MP+6D6J6Btqb7Wkwz640T8p/PYGueeWMHmlBBH jEf/pHbpoWvu5X1lU5A8SRv908Z53aIc6ymmmJV7G4NZAAc00NLHpLgjNFkRVCVM1Q+8 CF1FFzv3yoZhDszg7NyF/8bLxNlz60yua87Ara7LeiSgC4Qy4QQVAtHIMCiaTYn63OoO +pSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5AnDR3mseJ3BEgaMu0BeAFWG5+UZ/bYL9bd8XK/xdNA=; b=zpFVmPo5juQU0QDZgBly8tYdcNAUVmQilyJuur8fAgv1NzfgGQSb80E1l6oiBiKcpC ioE2cCHT/Kun12pHoq4BQDCQkFQvgnw+3w4g2G8pLYIWUXeXUcnT5mSbnNrk+XAep6he 36/j2RE1uP8RFOvV6dZMhiCbCF7MvGtqC9xxr4TI3QuvlsU5Vf3g8i5ButoK8F8nsCfM 1bETIXyhZYH1gAS5lcSFNprYRtwmJ/cEk98rAbev5W8E9B/lrJmfiaYeGrON/tzJvF0L uA/uf+2d3AWfzPCW2COFZe7cqEdi5nTQk+DDWwO1iNkGCH//7/qmWPbkUN+MVF4rgVk5 IPpQ== X-Gm-Message-State: AJIora/xSDeAkgtXSFgiS0Z7Z7bJaKefLX1PLhD0LbEmBMR2Xo1UyhLk 0QWhQn5UsBdQdWaUinUAAw+gEgUhjBNTsJW7jZA= X-Google-Smtp-Source: AGRyM1sj3aPdZ8lhgydJRpzBu/R8JCbuRxXWOmbP6T5OPoEj5KpuF5zlxiv1h2/OwPAKciW/uyoSSw== X-Received: by 2002:aa7:83d0:0:b0:50c:eb2b:8e8a with SMTP id j16-20020aa783d0000000b0050ceb2b8e8amr29920968pfn.31.1655816306533; Tue, 21 Jun 2022 05:58:26 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id e3-20020a170903240300b0015ea3a491a1sm10643134plo.191.2022.06.21.05.58.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jun 2022 05:58:26 -0700 (PDT) From: Muchun Song To: akpm@linux-foundation.org, hannes@cmpxchg.org, longman@redhat.com, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, duanxiongchun@bytedance.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song , =?UTF-8?q?Michal=20Koutn=C3=BD?= Subject: [PATCH v6 09/11] mm: memcontrol: use obj_cgroup APIs to charge the LRU pages Date: Tue, 21 Jun 2022 20:56:56 +0800 Message-Id: <20220621125658.64935-10-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220621125658.64935-1-songmuchun@bytedance.com> References: <20220621125658.64935-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org We will reuse the obj_cgroup APIs to charge the LRU pages. Finally, page->memcg_data will have 2 different meanings. - For the slab pages, page->memcg_data points to an object cgroups vector. - For the kmem pages (exclude the slab pages) and the LRU pages, page->memcg_data points to an object cgroup. In this patch, we reuse obj_cgroup APIs to charge LRU pages. In the end, The page cache cannot prevent long-living objects from pinning the original memory cgroup in the memory. At the same time we also changed the rules of page and objcg or memcg binding stability. The new rules are as follows. For a page any of the following ensures page and objcg binding stability: - the page lock - LRU isolation - lock_page_memcg() - exclusive reference Based on the stable binding of page and objcg, for a page any of the following ensures page and memcg binding stability: - objcg_lock - cgroup_mutex - the lruvec lock - the split queue lock (only THP page) If the caller only want to ensure that the page counters of memcg are updated correctly, ensure that the binding stability of page and objcg is sufficient. Signed-off-by: Muchun Song Reviewed-by: Michal Koutn=C3=BD Acked-by: Roman Gushchin --- include/linux/memcontrol.h | 89 +++++-------- mm/huge_memory.c | 35 +++++ mm/memcontrol.c | 317 ++++++++++++++++++++++++++++++-----------= ---- 3 files changed, 282 insertions(+), 159 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 63dbdef60cbd..744cde2b2368 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -392,8 +392,6 @@ enum page_memcg_data_flags { =20 #define MEMCG_DATA_FLAGS_MASK (__NR_MEMCG_DATA_FLAGS - 1) =20 -static inline bool folio_memcg_kmem(struct folio *folio); - /* * After the initialization objcg->memcg is always pointing at * a valid memcg, but can be atomically swapped to the parent memcg. @@ -407,43 +405,19 @@ static inline struct mem_cgroup *obj_cgroup_memcg(str= uct obj_cgroup *objcg) } =20 /* - * __folio_memcg - Get the memory cgroup associated with a non-kmem folio - * @folio: Pointer to the folio. - * - * Returns a pointer to the memory cgroup associated with the folio, - * or NULL. This function assumes that the folio is known to have a - * proper memory cgroup pointer. It's not safe to call this function - * against some type of folios, e.g. slab folios or ex-slab folios or - * kmem folios. - */ -static inline struct mem_cgroup *__folio_memcg(struct folio *folio) -{ - unsigned long memcg_data =3D folio->memcg_data; - - VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); - VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJCGS, folio); - VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_KMEM, folio); - - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); -} - -/* - * __folio_objcg - get the object cgroup associated with a kmem folio. + * folio_objcg - get the object cgroup associated with a folio. * @folio: Pointer to the folio. * * Returns a pointer to the object cgroup associated with the folio, * or NULL. This function assumes that the folio is known to have a - * proper object cgroup pointer. It's not safe to call this function - * against some type of folios, e.g. slab folios or ex-slab folios or - * LRU folios. + * proper object cgroup pointer. */ -static inline struct obj_cgroup *__folio_objcg(struct folio *folio) +static inline struct obj_cgroup *folio_objcg(struct folio *folio) { unsigned long memcg_data =3D folio->memcg_data; =20 VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJCGS, folio); - VM_BUG_ON_FOLIO(!(memcg_data & MEMCG_DATA_KMEM), folio); =20 return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); } @@ -457,22 +431,33 @@ static inline struct obj_cgroup *__folio_objcg(struct= folio *folio) * proper memory cgroup pointer. It's not safe to call this function * against some type of folios, e.g. slab folios or ex-slab folios. * - * For a non-kmem folio any of the following ensures folio and memcg bindi= ng - * stability: + * For a folio any of the following ensures folio and objcg binding stabil= ity: * * - the folio lock * - LRU isolation * - lock_page_memcg() * - exclusive reference * + * Based on the stable binding of folio and objcg, for a folio any of the + * following ensures folio and memcg binding stability: + * + * - objcg_lock + * - cgroup_mutex + * - the lruvec lock + * - the split queue lock (only THP page) + * + * If the caller only want to ensure that the page counters of memcg are + * updated correctly, ensure that the binding stability of folio and objcg + * is sufficient. + * * Note: The caller should hold an rcu read lock to protect memcg associat= ed * with a folio from being released. */ static inline struct mem_cgroup *folio_memcg(struct folio *folio) { - if (folio_memcg_kmem(folio)) - return obj_cgroup_memcg(__folio_objcg(folio)); - return __folio_memcg(folio); + struct obj_cgroup *objcg =3D folio_objcg(folio); + + return objcg ? obj_cgroup_memcg(objcg) : NULL; } =20 /* @@ -496,6 +481,8 @@ static inline struct mem_cgroup *page_memcg(struct page= *page) * folio is known to have a proper memory cgroup pointer. It's not safe * to call this function against some type of pages, e.g. slab pages or * ex-slab pages. + * + * The page and objcg or memcg binding rules can refer to folio_memcg(). */ static inline struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *f= olio) { @@ -526,22 +513,20 @@ static inline struct mem_cgroup *get_mem_cgroup_from_= page(struct page *page) * * Return: A pointer to the memory cgroup associated with the folio, * or NULL. + * + * The folio and objcg or memcg binding rules can refer to folio_memcg(). */ static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio) { unsigned long memcg_data =3D READ_ONCE(folio->memcg_data); + struct obj_cgroup *objcg; =20 VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); WARN_ON_ONCE(!rcu_read_lock_held()); =20 - if (memcg_data & MEMCG_DATA_KMEM) { - struct obj_cgroup *objcg; + objcg =3D (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); =20 - objcg =3D (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); - return obj_cgroup_memcg(objcg); - } - - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return objcg ? obj_cgroup_memcg(objcg) : NULL; } =20 /* @@ -554,16 +539,10 @@ static inline struct mem_cgroup *folio_memcg_rcu(stru= ct folio *folio) * has an associated memory cgroup pointer or an object cgroups vector or * an object cgroup. * - * For a non-kmem page any of the following ensures page and memcg binding - * stability: + * The page and objcg or memcg binding rules can refer to page_memcg(). * - * - the page lock - * - LRU isolation - * - lock_page_memcg() - * - exclusive reference - * - * For a kmem page a caller should hold an rcu read lock to protect memcg - * associated with a kmem page from being released. + * A caller should hold an rcu read lock to protect memcg associated with a + * page from being released. */ static inline struct mem_cgroup *page_memcg_check(struct page *page) { @@ -572,18 +551,14 @@ static inline struct mem_cgroup *page_memcg_check(str= uct page *page) * for slab pages, READ_ONCE() should be used here. */ unsigned long memcg_data =3D READ_ONCE(page->memcg_data); + struct obj_cgroup *objcg; =20 if (memcg_data & MEMCG_DATA_OBJCGS) return NULL; =20 - if (memcg_data & MEMCG_DATA_KMEM) { - struct obj_cgroup *objcg; - - objcg =3D (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); - return obj_cgroup_memcg(objcg); - } + objcg =3D (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); =20 - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return objcg ? obj_cgroup_memcg(objcg) : NULL; } =20 static inline struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgro= up *objcg) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 11ec92783b37..4f276ae6e840 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -558,6 +558,8 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struc= t *vma) } =20 #ifdef CONFIG_MEMCG +static struct shrinker deferred_split_shrinker; + static inline struct mem_cgroup *folio_split_queue_memcg(struct folio *fol= io, struct deferred_split *queue) { @@ -574,6 +576,39 @@ static inline struct deferred_split *folio_memcg_split= _queue(struct folio *folio =20 return memcg ? &memcg->deferred_split_queue : NULL; } + +static void thp_sq_reparent_lock(struct mem_cgroup *src, struct mem_cgroup= *dst) +{ + spin_lock(&src->deferred_split_queue.split_queue_lock); + spin_lock_nested(&dst->deferred_split_queue.split_queue_lock, + SINGLE_DEPTH_NESTING); +} + +static void thp_sq_reparent_relocate(struct mem_cgroup *src, struct mem_cg= roup *dst) +{ + int nid; + struct deferred_split *src_queue, *dst_queue; + + src_queue =3D &src->deferred_split_queue; + dst_queue =3D &dst->deferred_split_queue; + + if (!src_queue->split_queue_len) + return; + + list_splice_tail_init(&src_queue->split_queue, &dst_queue->split_queue); + dst_queue->split_queue_len +=3D src_queue->split_queue_len; + src_queue->split_queue_len =3D 0; + + for_each_node(nid) + set_shrinker_bit(dst, nid, deferred_split_shrinker.id); +} + +static void thp_sq_reparent_unlock(struct mem_cgroup *src, struct mem_cgro= up *dst) +{ + spin_unlock(&dst->deferred_split_queue.split_queue_lock); + spin_unlock(&src->deferred_split_queue.split_queue_lock); +} +DEFINE_MEMCG_REPARENT_OPS(thp_sq); #else static inline struct mem_cgroup *folio_split_queue_memcg(struct folio *fol= io, struct deferred_split *queue) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 6ef3a264054e..803dbdf5f233 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -77,6 +77,7 @@ struct cgroup_subsys memory_cgrp_subsys __read_mostly; EXPORT_SYMBOL(memory_cgrp_subsys); =20 struct mem_cgroup *root_mem_cgroup __read_mostly; +static struct obj_cgroup *root_obj_cgroup __read_mostly; =20 /* Active memory cgroup to use from an interrupt context */ DEFINE_PER_CPU(struct mem_cgroup *, int_active_memcg); @@ -254,6 +255,11 @@ struct mem_cgroup *vmpressure_to_memcg(struct vmpressu= re *vmpr) =20 static DEFINE_SPINLOCK(objcg_lock); =20 +static inline bool obj_cgroup_is_root(struct obj_cgroup *objcg) +{ + return objcg =3D=3D root_obj_cgroup; +} + #ifdef CONFIG_MEMCG_KMEM bool mem_cgroup_kmem_disabled(void) { @@ -361,8 +367,78 @@ static void objcg_reparent_unlock(struct mem_cgroup *s= rc, struct mem_cgroup *dst =20 static DEFINE_MEMCG_REPARENT_OPS(objcg); =20 +static void lruvec_reparent_lock(struct mem_cgroup *src, struct mem_cgroup= *dst) +{ + int nid, nest =3D 0; + + for_each_node(nid) { + spin_lock_nested(&mem_cgroup_lruvec(src, + NODE_DATA(nid))->lru_lock, nest++); + spin_lock_nested(&mem_cgroup_lruvec(dst, + NODE_DATA(nid))->lru_lock, nest++); + } +} + +static void lruvec_reparent_lru(struct lruvec *src, struct lruvec *dst, + enum lru_list lru) +{ + int zid; + struct mem_cgroup_per_node *mz_src, *mz_dst; + + mz_src =3D container_of(src, struct mem_cgroup_per_node, lruvec); + mz_dst =3D container_of(dst, struct mem_cgroup_per_node, lruvec); + + if (lru !=3D LRU_UNEVICTABLE) + list_splice_tail_init(&src->lists[lru], &dst->lists[lru]); + + for (zid =3D 0; zid < MAX_NR_ZONES; zid++) { + mz_dst->lru_zone_size[zid][lru] +=3D mz_src->lru_zone_size[zid][lru]; + mz_src->lru_zone_size[zid][lru] =3D 0; + } +} + +static void lruvec_reparent_relocate(struct mem_cgroup *src, struct mem_cg= roup *dst) +{ + int nid; + + for_each_node(nid) { + enum lru_list lru; + struct lruvec *src_lruvec, *dst_lruvec; + + src_lruvec =3D mem_cgroup_lruvec(src, NODE_DATA(nid)); + dst_lruvec =3D mem_cgroup_lruvec(dst, NODE_DATA(nid)); + + dst_lruvec->anon_cost +=3D src_lruvec->anon_cost; + dst_lruvec->file_cost +=3D src_lruvec->file_cost; + + for_each_lru(lru) + lruvec_reparent_lru(src_lruvec, dst_lruvec, lru); + } +} + +static void lruvec_reparent_unlock(struct mem_cgroup *src, struct mem_cgro= up *dst) +{ + int nid; + + for_each_node(nid) { + spin_unlock(&mem_cgroup_lruvec(dst, NODE_DATA(nid))->lru_lock); + spin_unlock(&mem_cgroup_lruvec(src, NODE_DATA(nid))->lru_lock); + } +} + +static DEFINE_MEMCG_REPARENT_OPS(lruvec); + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +DECLARE_MEMCG_REPARENT_OPS(thp_sq); +#endif + +/* The lock order depends on the order of elements in this array. */ static const struct memcg_reparent_ops *memcg_reparent_ops[] =3D { &memcg_objcg_reparent_ops, + &memcg_lruvec_reparent_ops, +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + &memcg_thp_sq_reparent_ops, +#endif }; =20 #define DEFINE_MEMCG_REPARENT_FUNC(phase) \ @@ -2825,18 +2901,33 @@ static inline void cancel_charge(struct mem_cgroup = *memcg, unsigned int nr_pages page_counter_uncharge(&memcg->memsw, nr_pages); } =20 -static void commit_charge(struct folio *folio, struct mem_cgroup *memcg) +static void commit_charge(struct folio *folio, struct obj_cgroup *objcg) { - VM_BUG_ON_FOLIO(folio_memcg(folio), folio); + VM_BUG_ON_FOLIO(folio_objcg(folio), folio); /* - * Any of the following ensures page's memcg stability: + * Any of the following ensures page's objcg stability: * * - the page lock * - LRU isolation * - lock_page_memcg() * - exclusive reference */ - folio->memcg_data =3D (unsigned long)memcg; + folio->memcg_data =3D (unsigned long)objcg; +} + +static struct obj_cgroup *get_obj_cgroup_from_memcg(struct mem_cgroup *mem= cg) +{ + struct obj_cgroup *objcg =3D NULL; + + rcu_read_lock(); + for (; memcg; memcg =3D parent_mem_cgroup(memcg)) { + objcg =3D rcu_dereference(memcg->objcg); + if (objcg && obj_cgroup_tryget(objcg)) + break; + } + rcu_read_unlock(); + + return objcg; } =20 #ifdef CONFIG_MEMCG_KMEM @@ -2982,19 +3073,6 @@ struct mem_cgroup *mem_cgroup_from_slab_obj(void *p) return mem_cgroup_from_obj_folio(virt_to_folio(p), p); } =20 -static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *m= emcg) -{ - struct obj_cgroup *objcg =3D NULL; - - for (; memcg !=3D root_mem_cgroup; memcg =3D parent_mem_cgroup(memcg)) { - objcg =3D rcu_dereference(memcg->objcg); - if (objcg && obj_cgroup_tryget(objcg)) - break; - objcg =3D NULL; - } - return objcg; -} - __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) { struct obj_cgroup *objcg =3D NULL; @@ -3008,7 +3086,16 @@ __always_inline struct obj_cgroup *get_obj_cgroup_fr= om_current(void) memcg =3D active_memcg(); else memcg =3D mem_cgroup_from_task(current); - objcg =3D __get_obj_cgroup_from_memcg(memcg); + + if (mem_cgroup_is_root(memcg)) + goto out; + + objcg =3D get_obj_cgroup_from_memcg(memcg); + if (obj_cgroup_is_root(objcg)) { + obj_cgroup_put(objcg); + objcg =3D NULL; + } +out: rcu_read_unlock(); return objcg; } @@ -3020,20 +3107,10 @@ struct obj_cgroup *get_obj_cgroup_from_page(struct = page *page) if (!memcg_kmem_enabled() || memcg_kmem_bypass()) return NULL; =20 - if (PageMemcgKmem(page)) { - objcg =3D __folio_objcg(page_folio(page)); + objcg =3D folio_objcg(page_folio(page)); + if (objcg) obj_cgroup_get(objcg); - } else { - struct mem_cgroup *memcg; =20 - rcu_read_lock(); - memcg =3D __folio_memcg(page_folio(page)); - if (memcg) - objcg =3D __get_obj_cgroup_from_memcg(memcg); - else - objcg =3D NULL; - rcu_read_unlock(); - } return objcg; } =20 @@ -3128,13 +3205,13 @@ int __memcg_kmem_charge_page(struct page *page, gfp= _t gfp, int order) void __memcg_kmem_uncharge_page(struct page *page, int order) { struct folio *folio =3D page_folio(page); - struct obj_cgroup *objcg; + struct obj_cgroup *objcg =3D folio_objcg(folio); unsigned int nr_pages =3D 1 << order; =20 - if (!folio_memcg_kmem(folio)) + if (!objcg) return; =20 - objcg =3D __folio_objcg(folio); + VM_BUG_ON_FOLIO(!folio_memcg_kmem(folio), folio); obj_cgroup_uncharge_pages(objcg, nr_pages); folio->memcg_data =3D 0; obj_cgroup_put(objcg); @@ -3388,26 +3465,21 @@ void obj_cgroup_uncharge(struct obj_cgroup *objcg, = size_t size) #endif /* CONFIG_MEMCG_KMEM */ =20 /* - * Because page_memcg(head) is not set on tails, set it now. + * Because page_objcg(head) is not set on tails, set it now. */ void split_page_memcg(struct page *head, unsigned int nr) { struct folio *folio =3D page_folio(head); - struct mem_cgroup *memcg =3D get_mem_cgroup_from_folio(folio); + struct obj_cgroup *objcg =3D folio_objcg(folio); int i; =20 - if (mem_cgroup_disabled() || !memcg) + if (mem_cgroup_disabled() || !objcg) return; =20 for (i =3D 1; i < nr; i++) folio_page(folio, i)->memcg_data =3D folio->memcg_data; =20 - if (folio_memcg_kmem(folio)) - obj_cgroup_get_many(__folio_objcg(folio), nr - 1); - else - css_get_many(&memcg->css, nr - 1); - - css_put(&memcg->css); + obj_cgroup_get_many(objcg, nr - 1); } =20 #ifdef CONFIG_MEMCG_SWAP @@ -5325,6 +5397,9 @@ static int mem_cgroup_css_online(struct cgroup_subsys= _state *css) objcg->memcg =3D memcg; rcu_assign_pointer(memcg->objcg, objcg); =20 + if (unlikely(mem_cgroup_is_root(memcg))) + root_obj_cgroup =3D objcg; + /* Online state pins memcg ID, memcg ID pins CSS */ refcount_set(&memcg->id.ref, 1); css_get(css); @@ -5729,10 +5804,12 @@ static int mem_cgroup_move_account(struct page *pag= e, */ smp_mb(); =20 - css_get(&to->css); - css_put(&from->css); + rcu_read_lock(); + obj_cgroup_get(rcu_dereference(to->objcg)); + obj_cgroup_put(rcu_dereference(from->objcg)); + rcu_read_unlock(); =20 - folio->memcg_data =3D (unsigned long)to; + folio->memcg_data =3D (unsigned long)rcu_access_pointer(to->objcg); =20 __folio_memcg_unlock(from); =20 @@ -6208,6 +6285,42 @@ static void mem_cgroup_move_charge(void) walk_page_range(mc.mm, 0, ULONG_MAX, &charge_walk_ops, NULL); mmap_read_unlock(mc.mm); atomic_dec(&mc.from->moving_account); + + /* + * Moving its pages to another memcg is finished. Wait for already + * started RCU-only updates to finish to make sure that the caller + * of lock_page_memcg() can unlock the correct move_lock. The + * possible bad scenario would like: + * + * CPU0: CPU1: + * mem_cgroup_move_charge() + * walk_page_range() + * + * lock_page_memcg(page) + * memcg =3D folio_memcg() + * spin_lock_irqsave(&memcg->move_lock) + * memcg->move_lock_task =3D current + * + * atomic_dec(&mc.from->moving_account) + * + * mem_cgroup_css_offline() + * memcg_offline_kmem() + * memcg_reparent_objcgs() <=3D=3D reparented + * + * unlock_page_memcg(page) + * memcg =3D folio_memcg() <=3D=3D memcg has been changed + * if (memcg->move_lock_task =3D=3D current) <=3D=3D false + * spin_unlock_irqrestore(&memcg->move_lock) + * + * Once mem_cgroup_move_charge() returns (it means that the cgroup_mutex + * would be released soon), the page can be reparented to its parent + * memcg. When the unlock_page_memcg() is called for the page, we will + * miss unlock the move_lock. So using synchronize_rcu to wait for + * already started RCU-only updates to finish before this function + * returns (mem_cgroup_move_charge() and mem_cgroup_css_offline() are + * serialized by cgroup_mutex). + */ + synchronize_rcu(); } =20 /* @@ -6822,21 +6935,26 @@ void mem_cgroup_calculate_protection(struct mem_cgr= oup *root, static int charge_memcg(struct folio *folio, struct mem_cgroup *memcg, gfp_t gfp) { + struct obj_cgroup *objcg; long nr_pages =3D folio_nr_pages(folio); - int ret; + int ret =3D 0; =20 - ret =3D try_charge(memcg, gfp, nr_pages); + objcg =3D get_obj_cgroup_from_memcg(memcg); + /* Do not account at the root objcg level. */ + if (!obj_cgroup_is_root(objcg)) + ret =3D try_charge(memcg, gfp, nr_pages); if (ret) goto out; =20 - css_get(&memcg->css); - commit_charge(folio, memcg); + obj_cgroup_get(objcg); + commit_charge(folio, objcg); =20 local_irq_disable(); mem_cgroup_charge_statistics(memcg, nr_pages); memcg_check_events(memcg, folio_nid(folio)); local_irq_enable(); out: + obj_cgroup_put(objcg); return ret; } =20 @@ -6922,7 +7040,7 @@ void mem_cgroup_swapin_uncharge_swap(swp_entry_t entr= y) } =20 struct uncharge_gather { - struct mem_cgroup *memcg; + struct obj_cgroup *objcg; unsigned long nr_memory; unsigned long pgpgout; unsigned long nr_kmem; @@ -6937,63 +7055,56 @@ static inline void uncharge_gather_clear(struct unc= harge_gather *ug) static void uncharge_batch(const struct uncharge_gather *ug) { unsigned long flags; + struct mem_cgroup *memcg; =20 + rcu_read_lock(); + memcg =3D obj_cgroup_memcg(ug->objcg); if (ug->nr_memory) { - page_counter_uncharge(&ug->memcg->memory, ug->nr_memory); + page_counter_uncharge(&memcg->memory, ug->nr_memory); if (do_memsw_account()) - page_counter_uncharge(&ug->memcg->memsw, ug->nr_memory); + page_counter_uncharge(&memcg->memsw, ug->nr_memory); if (ug->nr_kmem) - memcg_account_kmem(ug->memcg, -ug->nr_kmem); - memcg_oom_recover(ug->memcg); + memcg_account_kmem(memcg, -ug->nr_kmem); + memcg_oom_recover(memcg); } =20 local_irq_save(flags); - __count_memcg_events(ug->memcg, PGPGOUT, ug->pgpgout); - __this_cpu_add(ug->memcg->vmstats_percpu->nr_page_events, ug->nr_memory); - memcg_check_events(ug->memcg, ug->nid); + __count_memcg_events(memcg, PGPGOUT, ug->pgpgout); + __this_cpu_add(memcg->vmstats_percpu->nr_page_events, ug->nr_memory); + memcg_check_events(memcg, ug->nid); local_irq_restore(flags); + rcu_read_unlock(); =20 /* drop reference from uncharge_folio */ - css_put(&ug->memcg->css); + obj_cgroup_put(ug->objcg); } =20 static void uncharge_folio(struct folio *folio, struct uncharge_gather *ug) { long nr_pages; - struct mem_cgroup *memcg; struct obj_cgroup *objcg; =20 VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); =20 /* * Nobody should be changing or seriously looking at - * folio memcg or objcg at this point, we have fully - * exclusive access to the folio. + * folio objcg at this point, we have fully exclusive + * access to the folio. */ - if (folio_memcg_kmem(folio)) { - objcg =3D __folio_objcg(folio); - /* - * This get matches the put at the end of the function and - * kmem pages do not hold memcg references anymore. - */ - memcg =3D get_mem_cgroup_from_objcg(objcg); - } else { - memcg =3D __folio_memcg(folio); - } - - if (!memcg) + objcg =3D folio_objcg(folio); + if (!objcg) return; =20 - if (ug->memcg !=3D memcg) { - if (ug->memcg) { + if (ug->objcg !=3D objcg) { + if (ug->objcg) { uncharge_batch(ug); uncharge_gather_clear(ug); } - ug->memcg =3D memcg; + ug->objcg =3D objcg; ug->nid =3D folio_nid(folio); =20 - /* pairs with css_put in uncharge_batch */ - css_get(&memcg->css); + /* pairs with obj_cgroup_put in uncharge_batch */ + obj_cgroup_get(objcg); } =20 nr_pages =3D folio_nr_pages(folio); @@ -7001,19 +7112,15 @@ static void uncharge_folio(struct folio *folio, str= uct uncharge_gather *ug) if (folio_memcg_kmem(folio)) { ug->nr_memory +=3D nr_pages; ug->nr_kmem +=3D nr_pages; - - folio->memcg_data =3D 0; - obj_cgroup_put(objcg); } else { /* LRU pages aren't accounted at the root level */ - if (!mem_cgroup_is_root(memcg)) + if (!obj_cgroup_is_root(objcg)) ug->nr_memory +=3D nr_pages; ug->pgpgout++; - - folio->memcg_data =3D 0; } =20 - css_put(&memcg->css); + folio->memcg_data =3D 0; + obj_cgroup_put(objcg); } =20 void __mem_cgroup_uncharge(struct folio *folio) @@ -7021,7 +7128,7 @@ void __mem_cgroup_uncharge(struct folio *folio) struct uncharge_gather ug; =20 /* Don't touch folio->lru of any random page, pre-check: */ - if (!folio_memcg(folio)) + if (!folio_objcg(folio)) return; =20 uncharge_gather_clear(&ug); @@ -7044,7 +7151,7 @@ void __mem_cgroup_uncharge_list(struct list_head *pag= e_list) uncharge_gather_clear(&ug); list_for_each_entry(folio, page_list, lru) uncharge_folio(folio, &ug); - if (ug.memcg) + if (ug.objcg) uncharge_batch(&ug); } =20 @@ -7061,6 +7168,7 @@ void __mem_cgroup_uncharge_list(struct list_head *pag= e_list) void mem_cgroup_migrate(struct folio *old, struct folio *new) { struct mem_cgroup *memcg; + struct obj_cgroup *objcg; long nr_pages =3D folio_nr_pages(new); unsigned long flags; =20 @@ -7073,30 +7181,33 @@ void mem_cgroup_migrate(struct folio *old, struct f= olio *new) return; =20 /* Page cache replacement: new folio already charged? */ - if (folio_memcg(new)) + if (folio_objcg(new)) return; =20 - memcg =3D get_mem_cgroup_from_folio(old); - VM_WARN_ON_ONCE_FOLIO(!memcg, old); - if (!memcg) + objcg =3D folio_objcg(old); + VM_WARN_ON_ONCE_FOLIO(!objcg, old); + if (!objcg) return; =20 + rcu_read_lock(); + memcg =3D obj_cgroup_memcg(objcg); + /* Force-charge the new page. The old one will be freed soon */ - if (!mem_cgroup_is_root(memcg)) { + if (!obj_cgroup_is_root(objcg)) { page_counter_charge(&memcg->memory, nr_pages); if (do_memsw_account()) page_counter_charge(&memcg->memsw, nr_pages); } =20 - css_get(&memcg->css); - commit_charge(new, memcg); + obj_cgroup_get(objcg); + commit_charge(new, objcg); =20 local_irq_save(flags); mem_cgroup_charge_statistics(memcg, nr_pages); memcg_check_events(memcg, folio_nid(new)); local_irq_restore(flags); =20 - css_put(&memcg->css); + rcu_read_unlock(); } =20 DEFINE_STATIC_KEY_FALSE(memcg_sockets_enabled_key); @@ -7269,6 +7380,7 @@ static struct mem_cgroup *mem_cgroup_id_get_online(st= ruct mem_cgroup *memcg) void mem_cgroup_swapout(struct folio *folio, swp_entry_t entry) { struct mem_cgroup *memcg, *swap_memcg; + struct obj_cgroup *objcg; unsigned int nr_entries; unsigned short oldid; =20 @@ -7281,15 +7393,16 @@ void mem_cgroup_swapout(struct folio *folio, swp_en= try_t entry) if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) return; =20 + objcg =3D folio_objcg(folio); + VM_WARN_ON_ONCE_FOLIO(!objcg, folio); + if (!objcg) + return; + /* * Interrupts should be disabled by the caller (see the comments below), * which can serve as RCU read-side critical sections. */ - memcg =3D folio_memcg(folio); - - VM_WARN_ON_ONCE_FOLIO(!memcg, folio); - if (!memcg) - return; + memcg =3D obj_cgroup_memcg(objcg); =20 /* * In case the memcg owning these pages has been offlined and doesn't @@ -7308,7 +7421,7 @@ void mem_cgroup_swapout(struct folio *folio, swp_entr= y_t entry) =20 folio->memcg_data =3D 0; =20 - if (!mem_cgroup_is_root(memcg)) + if (!obj_cgroup_is_root(objcg)) page_counter_uncharge(&memcg->memory, nr_entries); =20 if (!cgroup_memory_noswap && memcg !=3D swap_memcg) { @@ -7328,7 +7441,7 @@ void mem_cgroup_swapout(struct folio *folio, swp_entr= y_t entry) memcg_stats_unlock(); memcg_check_events(memcg, folio_nid(folio)); =20 - css_put(&memcg->css); + obj_cgroup_put(objcg); } =20 /** --=20 2.11.0 From nobody Sat May 18 18:27:39 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1780C433EF for ; Tue, 21 Jun 2022 12:59:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1351121AbiFUM7O (ORCPT ); Tue, 21 Jun 2022 08:59:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57082 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351135AbiFUM6c (ORCPT ); Tue, 21 Jun 2022 08:58:32 -0400 Received: from mail-pf1-x42c.google.com (mail-pf1-x42c.google.com [IPv6:2607:f8b0:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7136D300 for ; Tue, 21 Jun 2022 05:58:31 -0700 (PDT) Received: by mail-pf1-x42c.google.com with SMTP id i64so13030649pfc.8 for ; Tue, 21 Jun 2022 05:58:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=shtH7PIMf3J8dcya3xtagcl4W8V8R+/sKr+kfrLIY6Y=; b=pKDiUQIgj4YaZxnL6yCeczDMoqlNkY/fF9chdQFzHfGrTICA/YrcbElxeDDgKm++4f 6Cxu565+659Mbpd+/mE7x+gUVXxoISE/HC0AN/7bQGI7T2zC2G+4yGW6GKGsgGLiNSbf y7UEpRzdy/p0Tq93+aIN9Ik2CU7UNoB3XcKtE1bfePMgY7rTkTwwqscPFdMO3RNmhIfY 9QxdQEr3AcmrE33LiJxuDqt+aW0wit9ADGaAiEt4Kw2pPCT1u6q7cr38r7hY7aybgITy DvosnAH5V/sCJGZMM25qDYSkhP3+ATsBrPLgsuH6JxtYlZNQUeAcXTsUPlnNGDexLJCF QdIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=shtH7PIMf3J8dcya3xtagcl4W8V8R+/sKr+kfrLIY6Y=; b=zfUpzKThLnsDsgO1wy2a1KN3h87RW3WRG+6sYPSajgvSgZxS35lxPikVA0Eh1V9Gd0 EvXJBN3BVnib/WjZtlfQrHTv2eUlw3rfsaISdwIRisVhWIr/ru1etx2+rGbExTA4vTGR KpXs3yJYPLFzLY8PieUhK/34E343QEynWD/byl13/oR5syOFyQht/T1AikJEmFoZ0HPY V6YvacVSt2CnFytjBwSvPSbsL4JjgvYQdgHyr8gfOPcN7KXHLWNDZ/pemRJ6045VUm0T IPLeucwJF2x0hA6lYV0oMYdWKkT4c7YWRol4AUCHlCC0ChWzJ/Er3JK4HDZBvKE/lU0i j4Iw== X-Gm-Message-State: AJIora9K9ltdAtvP9tNMc/xov4Kb6/I61cNPYOSrsI8O0UvLPIBhGFBi JI5CimH0g4U84go8H8iaL/PIOw== X-Google-Smtp-Source: AGRyM1u8ksQyZcIYOxcN0FytSUT2CO9PHE09w2KWxt2Sj0ZFsOL55NgBQbL9f4jw1sdjQKIB9ybP3Q== X-Received: by 2002:a05:6a00:430e:b0:525:26c1:973e with SMTP id cb14-20020a056a00430e00b0052526c1973emr9219577pfb.52.1655816311195; Tue, 21 Jun 2022 05:58:31 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id e3-20020a170903240300b0015ea3a491a1sm10643134plo.191.2022.06.21.05.58.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jun 2022 05:58:30 -0700 (PDT) From: Muchun Song To: akpm@linux-foundation.org, hannes@cmpxchg.org, longman@redhat.com, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, duanxiongchun@bytedance.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH v6 10/11] mm: lru: add VM_WARN_ON_ONCE_FOLIO to lru maintenance function Date: Tue, 21 Jun 2022 20:56:57 +0800 Message-Id: <20220621125658.64935-11-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220621125658.64935-1-songmuchun@bytedance.com> References: <20220621125658.64935-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" We need to make sure that the page is deleted from or added to the correct lruvec list. So add a VM_WARN_ON_ONCE_FOLIO() to catch invalid users. Then the VM_BUG_ON_PAGE() in move_pages_to_lru() could be removed since add_page_to_lru_list() will check that. Signed-off-by: Muchun Song Acked-by: Roman Gushchin --- include/linux/mm_inline.h | 6 ++++++ mm/vmscan.c | 1 - 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 7b25b53c474a..6585198b19e2 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -99,6 +99,8 @@ void lruvec_add_folio(struct lruvec *lruvec, struct folio= *folio) { enum lru_list lru =3D folio_lru_list(folio); =20 + VM_WARN_ON_ONCE_FOLIO(!folio_matches_lruvec(folio, lruvec), folio); + update_lru_size(lruvec, lru, folio_zonenum(folio), folio_nr_pages(folio)); if (lru !=3D LRU_UNEVICTABLE) @@ -116,6 +118,8 @@ void lruvec_add_folio_tail(struct lruvec *lruvec, struc= t folio *folio) { enum lru_list lru =3D folio_lru_list(folio); =20 + VM_WARN_ON_ONCE_FOLIO(!folio_matches_lruvec(folio, lruvec), folio); + update_lru_size(lruvec, lru, folio_zonenum(folio), folio_nr_pages(folio)); /* This is not expected to be used on LRU_UNEVICTABLE */ @@ -133,6 +137,8 @@ void lruvec_del_folio(struct lruvec *lruvec, struct fol= io *folio) { enum lru_list lru =3D folio_lru_list(folio); =20 + VM_WARN_ON_ONCE_FOLIO(!folio_matches_lruvec(folio, lruvec), folio); + if (lru !=3D LRU_UNEVICTABLE) list_del(&folio->lru); update_lru_size(lruvec, lru, folio_zonenum(folio), diff --git a/mm/vmscan.c b/mm/vmscan.c index 697656151431..51b1607c81e4 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2361,7 +2361,6 @@ static unsigned int move_pages_to_lru(struct list_hea= d *list) continue; } =20 - VM_BUG_ON_FOLIO(!folio_matches_lruvec(folio, lruvec), folio); lruvec_add_folio(lruvec, folio); nr_pages =3D folio_nr_pages(folio); nr_moved +=3D nr_pages; --=20 2.11.0 From nobody Sat May 18 18:27:39 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D917FC433EF for ; Tue, 21 Jun 2022 12:59:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350999AbiFUM70 (ORCPT ); Tue, 21 Jun 2022 08:59:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1351157AbiFUM6h (ORCPT ); Tue, 21 Jun 2022 08:58:37 -0400 Received: from mail-pg1-x52e.google.com (mail-pg1-x52e.google.com [IPv6:2607:f8b0:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 701B0B2C for ; Tue, 21 Jun 2022 05:58:36 -0700 (PDT) Received: by mail-pg1-x52e.google.com with SMTP id z14so13085514pgh.0 for ; Tue, 21 Jun 2022 05:58:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FXYyMqCuMn8N8AtiePCadIk2ihaRxjS8+sCa4dkDZyY=; b=ID4gt1H5XAXZc2WTCkcM4F1wzX1sAe3JOWtacCk0iQKQ+aLMZXAoV2LeDm9V/qhVXx 6tpAVEejJhQBmJjVUK+nIaNyniVWz7hJGtPgx1OZl1uOfj2pujAFTcbGChoVU9DBG1kQ iSr/YRd45uGdFIuCRF+nZy0GIGBavXDFzjNLsbIp/9bZz+yRlZan+nvKfKhVHEPvjiD5 P/VKwyJbbrvB0eRUS26CEwFTb/hg3CWoix8AAXQYFzZU3v7OZBDBODtMEhL6UOydPlDK QPnx0hfY8bRXub0eY6AxeQXF/rpFM66dPqaBxFcGtr4OkljZGqrs6ATOyrSkIf9U3exr 0w8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FXYyMqCuMn8N8AtiePCadIk2ihaRxjS8+sCa4dkDZyY=; b=1Ho6hQdYop/3SMfjykvCh8G6aGxYHbuvzmQt4r/rW0P+7+qJlCmWkEl7cTQj2UePYF 5ppcYjmDt+lGPEJOen5b77FqMzgWHQr52Xv8WlRSbSZs9TIT0yQPq3f4NWf4kLBrYEym tosP6LX6Y4WIoh0QzFvYbnr4I2i8MbZMWuWgCIcYK50Bc9u9pVmdLvZn1iw6KnpGVLL+ CAf2QPvfbg3I5eMNmbVq+/yxKQi6JJZdiW7C0GP9cWFG/4Vy0MEtf5usHkIyZizsdAOO zMd4M/iME34yuwanpOLXqj82LYcSfmArlT1c0ipeK5ekADAQ35jWmVyqD/e/Mu1g0T88 9/fg== X-Gm-Message-State: AJIora9jd46xjSYXFEF0V8PmxMbPnD2eGPbtmxVbl9VUawfSPi0qvayk P+S06Lm7BOwPKhtU3EOwuYsxVg== X-Google-Smtp-Source: AGRyM1s3jBpILIrDx7KtbyRkegEjXT/bYyiPJT5PmXEB7MsPCFZC/X2wELk53MsJbksfq61xEhFUrw== X-Received: by 2002:a63:7c4e:0:b0:380:8ae9:c975 with SMTP id l14-20020a637c4e000000b003808ae9c975mr26959498pgn.25.1655816315865; Tue, 21 Jun 2022 05:58:35 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id e3-20020a170903240300b0015ea3a491a1sm10643134plo.191.2022.06.21.05.58.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Jun 2022 05:58:35 -0700 (PDT) From: Muchun Song To: akpm@linux-foundation.org, hannes@cmpxchg.org, longman@redhat.com, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, duanxiongchun@bytedance.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH v6 11/11] mm: lru: use lruvec lock to serialize memcg changes Date: Tue, 21 Jun 2022 20:56:58 +0800 Message-Id: <20220621125658.64935-12-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220621125658.64935-1-songmuchun@bytedance.com> References: <20220621125658.64935-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" As described by commit fc574c23558c ("mm/swap.c: serialize memcg changes in pagevec_lru_move_fn"), TestClearPageLRU() aims to serialize mem_cgroup_move_account() during pagevec_lru_move_fn(). Now folio_lruvec_lock*() has the ability to detect whether page memcg has been changed. So we can use lruvec lock to serialize mem_cgroup_move_account() during pagevec_lru_move_fn(). This change is a partial revert of the commit fc574c23558c ("mm/swap.c: serialize memcg changes in pagevec_lru_move_fn"). And pagevec_lru_move_fn() is more hot compare with mem_cgroup_move_account(), removing an atomic operation would be an optimization. Also this change would not dirty cacheline for a page which isn't on the LRU. Signed-off-by: Muchun Song --- mm/memcontrol.c | 34 ++++++++++++++++++++++++++++++++++ mm/swap.c | 32 +++++++++++++++----------------- mm/vmscan.c | 16 +++++++--------- 3 files changed, 56 insertions(+), 26 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 803dbdf5f233..85adc43c5a25 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1330,10 +1330,39 @@ struct lruvec *folio_lruvec_lock(struct folio *foli= o) lruvec =3D folio_lruvec(folio); spin_lock(&lruvec->lru_lock); =20 + /* + * The memcg of the page can be changed by any the following routines: + * + * 1) mem_cgroup_move_account() or + * 2) memcg_reparent_objcgs() + * + * The possible bad scenario would like: + * + * CPU0: CPU1: CPU2: + * lruvec =3D folio_lruvec() + * + * if (!isolate_lru_page()) + * mem_cgroup_move_account() + * + * memcg_reparent_objcgs() + * + * spin_lock(&lruvec->lru_lock) + * ^^^^^^ + * wrong lock + * + * Either CPU1 or CPU2 can change page memcg, so we need to check + * whether page memcg is changed, if so, we should reacquire the + * new lruvec lock. + */ if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) { spin_unlock(&lruvec->lru_lock); goto retry; } + + /* + * When we reach here, it means that the folio_memcg(folio) is + * stable. + */ rcu_read_unlock(); =20 return lruvec; @@ -1361,6 +1390,7 @@ struct lruvec *folio_lruvec_lock_irq(struct folio *fo= lio) lruvec =3D folio_lruvec(folio); spin_lock_irq(&lruvec->lru_lock); =20 + /* See the comments in folio_lruvec_lock(). */ if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) { spin_unlock_irq(&lruvec->lru_lock); goto retry; @@ -1394,6 +1424,7 @@ struct lruvec *folio_lruvec_lock_irqsave(struct folio= *folio, lruvec =3D folio_lruvec(folio); spin_lock_irqsave(&lruvec->lru_lock, *flags); =20 + /* See the comments in folio_lruvec_lock(). */ if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) { spin_unlock_irqrestore(&lruvec->lru_lock, *flags); goto retry; @@ -5809,7 +5840,10 @@ static int mem_cgroup_move_account(struct page *page, obj_cgroup_put(rcu_dereference(from->objcg)); rcu_read_unlock(); =20 + /* See the comments in folio_lruvec_lock(). */ + spin_lock(&from_vec->lru_lock); folio->memcg_data =3D (unsigned long)rcu_access_pointer(to->objcg); + spin_unlock(&from_vec->lru_lock); =20 __folio_memcg_unlock(from); =20 diff --git a/mm/swap.c b/mm/swap.c index 987dcbd93ffa..0fc59409e27d 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -196,6 +196,7 @@ static void lru_add_fn(struct lruvec *lruvec, struct fo= lio *folio) =20 VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); =20 + folio_set_lru(folio); /* * Is an smp_mb__after_atomic() still required here, before * folio_evictable() tests the mlocked flag, to rule out the possibility @@ -238,14 +239,8 @@ static void folio_batch_move_lru(struct folio_batch *f= batch, move_fn_t move_fn) for (i =3D 0; i < folio_batch_count(fbatch); i++) { struct folio *folio =3D fbatch->folios[i]; =20 - /* block memcg migration while the folio moves between lru */ - if (move_fn !=3D lru_add_fn && !folio_test_clear_lru(folio)) - continue; - lruvec =3D folio_lruvec_relock_irqsave(folio, lruvec, &flags); move_fn(lruvec, folio); - - folio_set_lru(folio); } =20 if (lruvec) @@ -265,7 +260,7 @@ static void folio_batch_add_and_move(struct folio_batch= *fbatch, =20 static void lru_move_tail_fn(struct lruvec *lruvec, struct folio *folio) { - if (!folio_test_unevictable(folio)) { + if (folio_test_lru(folio) && !folio_test_unevictable(folio)) { lruvec_del_folio(lruvec, folio); folio_clear_active(folio); lruvec_add_folio_tail(lruvec, folio); @@ -348,7 +343,8 @@ void lru_note_cost_folio(struct folio *folio) =20 static void folio_activate_fn(struct lruvec *lruvec, struct folio *folio) { - if (!folio_test_active(folio) && !folio_test_unevictable(folio)) { + if (folio_test_lru(folio) && !folio_test_active(folio) && + !folio_test_unevictable(folio)) { long nr_pages =3D folio_nr_pages(folio); =20 lruvec_del_folio(lruvec, folio); @@ -394,12 +390,9 @@ static void folio_activate(struct folio *folio) { struct lruvec *lruvec; =20 - if (folio_test_clear_lru(folio)) { - lruvec =3D folio_lruvec_lock_irq(folio); - folio_activate_fn(lruvec, folio); - lruvec_unlock_irq(lruvec); - folio_set_lru(folio); - } + lruvec =3D folio_lruvec_lock_irq(folio); + folio_activate_fn(lruvec, folio); + lruvec_unlock_irq(lruvec); } #endif =20 @@ -542,6 +535,9 @@ static void lru_deactivate_file_fn(struct lruvec *lruve= c, struct folio *folio) bool active =3D folio_test_active(folio); long nr_pages =3D folio_nr_pages(folio); =20 + if (!folio_test_lru(folio)) + return; + if (folio_test_unevictable(folio)) return; =20 @@ -580,7 +576,8 @@ static void lru_deactivate_file_fn(struct lruvec *lruve= c, struct folio *folio) =20 static void lru_deactivate_fn(struct lruvec *lruvec, struct folio *folio) { - if (folio_test_active(folio) && !folio_test_unevictable(folio)) { + if (folio_test_lru(folio) && folio_test_active(folio) && + !folio_test_unevictable(folio)) { long nr_pages =3D folio_nr_pages(folio); =20 lruvec_del_folio(lruvec, folio); @@ -596,8 +593,9 @@ static void lru_deactivate_fn(struct lruvec *lruvec, st= ruct folio *folio) =20 static void lru_lazyfree_fn(struct lruvec *lruvec, struct folio *folio) { - if (folio_test_anon(folio) && folio_test_swapbacked(folio) && - !folio_test_swapcache(folio) && !folio_test_unevictable(folio)) { + if (folio_test_lru(folio) && folio_test_anon(folio) && + folio_test_swapbacked(folio) && !folio_test_swapcache(folio) && + !folio_test_unevictable(folio)) { long nr_pages =3D folio_nr_pages(folio); =20 lruvec_del_folio(lruvec, folio); diff --git a/mm/vmscan.c b/mm/vmscan.c index 51b1607c81e4..11e1f6fc5898 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4864,21 +4864,19 @@ void check_move_unevictable_pages(struct pagevec *p= vec) if (PageTransTail(page)) continue; =20 - nr_pages =3D thp_nr_pages(page); + nr_pages =3D folio_nr_pages(folio); pgscanned +=3D nr_pages; =20 - /* block memcg migration during page moving between lru */ - if (!TestClearPageLRU(page)) + lruvec =3D folio_lruvec_relock_irq(folio, lruvec); + if (!folio_test_lru(folio) || !folio_test_unevictable(folio)) continue; =20 - lruvec =3D folio_lruvec_relock_irq(folio, lruvec); - if (page_evictable(page) && PageUnevictable(page)) { - del_page_from_lru_list(page, lruvec); - ClearPageUnevictable(page); - add_page_to_lru_list(page, lruvec); + if (folio_evictable(folio)) { + lruvec_del_folio(lruvec, folio); + folio_clear_unevictable(folio); + lruvec_add_folio(lruvec, folio); pgrescued +=3D nr_pages; } - SetPageLRU(page); } =20 if (lruvec) { --=20 2.11.0