From nobody Sun May 5 20:57:33 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8AE6C433EF for ; Tue, 24 May 2022 06:27:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234977AbiEXG11 (ORCPT ); Tue, 24 May 2022 02:27:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47194 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234953AbiEXG1T (ORCPT ); Tue, 24 May 2022 02:27:19 -0400 Received: from mail-pl1-x630.google.com (mail-pl1-x630.google.com [IPv6:2607:f8b0:4864:20::630]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4FA4572229 for ; Mon, 23 May 2022 23:27:17 -0700 (PDT) Received: by mail-pl1-x630.google.com with SMTP id m1so15039930plx.3 for ; Mon, 23 May 2022 23:27:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Snuh9kH4iZm3sviSaP28BB2r1eDnnNjUUortMndKLbU=; b=0PirmXF7o8ixOp2Gap1kFE0YZXnIJC6frN43iRyf5aoypM1dWTfeQieGP0WOA+VK2C fBNRoSeyGbR0xXtit0xqBhE2UQCsFHRLLbFEajJk1QEqetMQH6u8LN1TAr/6cFwWAVHP tZW3tMMIO8LaVtAFpQqIV2kLiC1sLhYqsswdewCSRp9NyRlqKn17NU9wpnfSXDWSYFRj 8YMhndotc/k5AE9hW9d3ihLS1YqG0hkZNwwwkfv9b7X9eVDkdPRKe9R2lt76FzM4FSJI n7his0xOGjmqBmYNm80AG01800cKPXywYJBosOpj36gQ3cTpW/CaSwiOjXz8wjYEBzKt obSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Snuh9kH4iZm3sviSaP28BB2r1eDnnNjUUortMndKLbU=; b=EO5ptHbxJ4ljW2mdsvcATTJyaqnFKA0NLxChPGgmFaiPmXp65ir4QjEiNX9LbYhs23 cJZUmV4CzFcAuQjC6P5KwF3I8Iu/pTGdNYIdUd0G3L1G83GeX9oxFVP8jnaNAAE1O013 1c56pCR3xC7+w2G1zWwfpupVJrpW+cbvOcF2N3FIr+MEryAqb3FsPL16fMMPiuca3rvi XoUws4sNFTyTMKaXLBA3bg/iSgKtGMXtl1pjYQZoQFSZ3voZ1MJckFfSQweWgGSu8qP9 whGZkseSG5xNaC4kzU+rgSOH8ua7TPqZ2B59plja330/+yTdi5UCUHo7Yx+5i1qOYYAU y2dQ== X-Gm-Message-State: AOAM533hszzC9+WfLG1v4FmIAnJNQLOcXWsh2pg0+CWqCA9yfNez4BRu CtWoOy+uEZLAjby/qY6c8vZG8ljFWQ9/GQ== X-Google-Smtp-Source: ABdhPJzjHagZRT6kX9J4hdRChv9TYsas2yUTDw3/NgMRdwNEtdr1pWT0pzzStyqD12/f5A5zeK4pfQ== X-Received: by 2002:a17:902:9349:b0:158:a6f7:e280 with SMTP id g9-20020a170902934900b00158a6f7e280mr25975740plp.155.1653373636644; Mon, 23 May 2022 23:27:16 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([2408:8207:18da:2310:f940:af17:c2f5:8656]) by smtp.gmail.com with ESMTPSA id h5-20020a170902f54500b0016168e90f2dsm6254455plf.219.2022.05.23.23.27.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 May 2022 23:27:16 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, duanxiongchun@bytedance.com, longman@redhat.com, Muchun Song Subject: [PATCH v4 01/11] mm: memcontrol: prepare objcg API for non-kmem usage Date: Tue, 24 May 2022 14:05:41 +0800 Message-Id: <20220524060551.80037-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220524060551.80037-1-songmuchun@bytedance.com> References: <20220524060551.80037-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Pagecache pages are charged at the allocation time and holding a reference to the original memory cgroup until being reclaimed. Depending on the memory pressure, specific patterns of the page sharing between different cgroups and the cgroup creation and destruction rates, a large number of dying memory cgroups can be pinned by pagecache pages. It makes the page reclaim less efficient and wastes memory. We can convert LRU pages and most other raw memcg pins to the objcg direction to fix this problem, and then the page->memcg will always point to an object cgroup pointer. Therefore, the infrastructure of objcg no longer only serves CONFIG_MEMCG_KMEM. In this patch, we move the infrastructure of the objcg out of the scope of the CONFIG_MEMCG_KMEM so that the LRU pages can reuse it to charge pages. We know that the LRU pages are not accounted at the root level. But the page->memcg_data points to the root_mem_cgroup. So the page->memcg_data of the LRU pages always points to a valid pointer. But the root_mem_cgroup dose not have an object cgroup. If we use obj_cgroup APIs to charge the LRU pages, we should set the page->memcg_data to a root object cgroup. So we also allocate an object cgroup for the root_mem_cgroup. Signed-off-by: Muchun Song Acked-by: Johannes Weiner --- include/linux/memcontrol.h | 5 ++-- mm/memcontrol.c | 60 +++++++++++++++++++++++++-----------------= ---- 2 files changed, 35 insertions(+), 30 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 89b14729d59f..ff1c1dd7e762 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -315,10 +315,10 @@ struct mem_cgroup { =20 #ifdef CONFIG_MEMCG_KMEM int kmemcg_id; +#endif struct obj_cgroup __rcu *objcg; /* list of inherited objcgs, protected by objcg_lock */ struct list_head objcg_list; -#endif =20 MEMCG_PADDING(_pad2_); =20 @@ -851,8 +851,7 @@ static inline struct mem_cgroup *lruvec_memcg(struct lr= uvec *lruvec) * parent_mem_cgroup - find the accounting parent of a memcg * @memcg: memcg whose parent to find * - * Returns the parent memcg, or NULL if this is the root or the memory - * controller is in legacy no-hierarchy mode. + * Returns the parent memcg, or NULL if this is the root. */ static inline struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *memc= g) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 598fece89e2b..6de0d3e53eb1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -254,9 +254,9 @@ struct mem_cgroup *vmpressure_to_memcg(struct vmpressur= e *vmpr) return container_of(vmpr, struct mem_cgroup, vmpressure); } =20 -#ifdef CONFIG_MEMCG_KMEM static DEFINE_SPINLOCK(objcg_lock); =20 +#ifdef CONFIG_MEMCG_KMEM bool mem_cgroup_kmem_disabled(void) { return cgroup_memory_nokmem; @@ -265,12 +265,10 @@ bool mem_cgroup_kmem_disabled(void) static void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg, unsigned int nr_pages); =20 -static void obj_cgroup_release(struct percpu_ref *ref) +static void obj_cgroup_release_bytes(struct obj_cgroup *objcg) { - struct obj_cgroup *objcg =3D container_of(ref, struct obj_cgroup, refcnt); unsigned int nr_bytes; unsigned int nr_pages; - unsigned long flags; =20 /* * At this point all allocated objects are freed, and @@ -284,9 +282,9 @@ static void obj_cgroup_release(struct percpu_ref *ref) * 3) CPU1: a process from another memcg is allocating something, * the stock if flushed, * objcg->nr_charged_bytes =3D PAGE_SIZE - 92 - * 5) CPU0: we do release this object, + * 4) CPU0: we do release this object, * 92 bytes are added to stock->nr_bytes - * 6) CPU0: stock is flushed, + * 5) CPU0: stock is flushed, * 92 bytes are added to objcg->nr_charged_bytes * * In the result, nr_charged_bytes =3D=3D PAGE_SIZE. @@ -298,6 +296,19 @@ static void obj_cgroup_release(struct percpu_ref *ref) =20 if (nr_pages) obj_cgroup_uncharge_pages(objcg, nr_pages); +} +#else +static inline void obj_cgroup_release_bytes(struct obj_cgroup *objcg) +{ +} +#endif + +static void obj_cgroup_release(struct percpu_ref *ref) +{ + struct obj_cgroup *objcg =3D container_of(ref, struct obj_cgroup, refcnt); + unsigned long flags; + + obj_cgroup_release_bytes(objcg); =20 spin_lock_irqsave(&objcg_lock, flags); list_del(&objcg->list); @@ -326,10 +337,10 @@ static struct obj_cgroup *obj_cgroup_alloc(void) return objcg; } =20 -static void memcg_reparent_objcgs(struct mem_cgroup *memcg, - struct mem_cgroup *parent) +static void memcg_reparent_objcgs(struct mem_cgroup *memcg) { struct obj_cgroup *objcg, *iter; + struct mem_cgroup *parent =3D parent_mem_cgroup(memcg); =20 objcg =3D rcu_replace_pointer(memcg->objcg, NULL, true); =20 @@ -348,6 +359,7 @@ static void memcg_reparent_objcgs(struct mem_cgroup *me= mcg, percpu_ref_kill(&objcg->refcnt); } =20 +#ifdef CONFIG_MEMCG_KMEM /* * A lot of the calls to the cache allocation functions are expected to be * inlined by the compiler. Since the calls to memcg_slab_pre_alloc_hook()= are @@ -3589,21 +3601,12 @@ static u64 mem_cgroup_read_u64(struct cgroup_subsys= _state *css, #ifdef CONFIG_MEMCG_KMEM static int memcg_online_kmem(struct mem_cgroup *memcg) { - struct obj_cgroup *objcg; - if (cgroup_memory_nokmem) return 0; =20 if (unlikely(mem_cgroup_is_root(memcg))) return 0; =20 - objcg =3D obj_cgroup_alloc(); - if (!objcg) - return -ENOMEM; - - objcg->memcg =3D memcg; - rcu_assign_pointer(memcg->objcg, objcg); - static_branch_enable(&memcg_kmem_enabled_key); =20 memcg->kmemcg_id =3D memcg->id.id; @@ -3613,27 +3616,19 @@ static int memcg_online_kmem(struct mem_cgroup *mem= cg) =20 static void memcg_offline_kmem(struct mem_cgroup *memcg) { - struct mem_cgroup *parent; - if (cgroup_memory_nokmem) return; =20 if (unlikely(mem_cgroup_is_root(memcg))) return; =20 - parent =3D parent_mem_cgroup(memcg); - if (!parent) - parent =3D root_mem_cgroup; - - memcg_reparent_objcgs(memcg, parent); - /* * After we have finished memcg_reparent_objcgs(), all list_lrus * corresponding to this cgroup are guaranteed to remain empty. * The ordering is imposed by list_lru_node->lock taken by * memcg_reparent_list_lrus(). */ - memcg_reparent_list_lrus(memcg, parent); + memcg_reparent_list_lrus(memcg, parent_mem_cgroup(memcg)); } #else static int memcg_online_kmem(struct mem_cgroup *memcg) @@ -5116,8 +5111,8 @@ static struct mem_cgroup *mem_cgroup_alloc(void) memcg->socket_pressure =3D jiffies; #ifdef CONFIG_MEMCG_KMEM memcg->kmemcg_id =3D -1; - INIT_LIST_HEAD(&memcg->objcg_list); #endif + INIT_LIST_HEAD(&memcg->objcg_list); #ifdef CONFIG_CGROUP_WRITEBACK INIT_LIST_HEAD(&memcg->cgwb_list); for (i =3D 0; i < MEMCG_CGWB_FRN_CNT; i++) @@ -5179,6 +5174,7 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *pare= nt_css) static int mem_cgroup_css_online(struct cgroup_subsys_state *css) { struct mem_cgroup *memcg =3D mem_cgroup_from_css(css); + struct obj_cgroup *objcg; =20 if (memcg_online_kmem(memcg)) goto remove_id; @@ -5191,6 +5187,13 @@ static int mem_cgroup_css_online(struct cgroup_subsy= s_state *css) if (alloc_shrinker_info(memcg)) goto offline_kmem; =20 + objcg =3D obj_cgroup_alloc(); + if (!objcg) + goto free_shrinker; + + objcg->memcg =3D memcg; + rcu_assign_pointer(memcg->objcg, objcg); + /* Online state pins memcg ID, memcg ID pins CSS */ refcount_set(&memcg->id.ref, 1); css_get(css); @@ -5199,6 +5202,8 @@ static int mem_cgroup_css_online(struct cgroup_subsys= _state *css) queue_delayed_work(system_unbound_wq, &stats_flush_dwork, 2UL*HZ); return 0; +free_shrinker: + free_shrinker_info(memcg); offline_kmem: memcg_offline_kmem(memcg); remove_id: @@ -5226,6 +5231,7 @@ static void mem_cgroup_css_offline(struct cgroup_subs= ys_state *css) page_counter_set_min(&memcg->memory, 0); page_counter_set_low(&memcg->memory, 0); =20 + memcg_reparent_objcgs(memcg); memcg_offline_kmem(memcg); reparent_shrinker_deferred(memcg); wb_memcg_offline(memcg); --=20 2.11.0 From nobody Sun May 5 20:57:33 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 963B4C433F5 for ; Tue, 24 May 2022 06:27:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234968AbiEXG1j (ORCPT ); Tue, 24 May 2022 02:27:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47772 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234979AbiEXG1c (ORCPT ); Tue, 24 May 2022 02:27:32 -0400 Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1816772E1F for ; Mon, 23 May 2022 23:27:25 -0700 (PDT) Received: by mail-pj1-x1036.google.com with SMTP id v5-20020a17090a7c0500b001df84fa82f8so1295615pjf.5 for ; Mon, 23 May 2022 23:27:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YQmbzG/x1UDz5U0yQVXExNL9W+QNwv5yyDN0QDopwLM=; b=SbFCQ3vJYsGUAEX8nyldT87US7H/Zi6r40ReQrz8S1DFmwuWGEoV2ZsawlDNV2F+b9 cBbWPLt6WyLG/WEs6j6or7s/bkrS5fhQCtCJYspYV7YbDoGUD1Kpw7Cz19/bYZvOSmvn C7kr1O/qam5v/KZxRh4lh9qzOJtWk0NaaM5tm7fJrsj7KBzxuKuwY074AEzpGvXjXhRa OSdKGe9kRdi5MZEOyUCYPsJ6eblgDNZpMXrb7JA12Yd9Uc5u620Py3qTO2xYaCVNavdX iwklkT8fllKtpNXMNWn4waK8JwI1CM/uTxteV3N+Nq25H+HgDd0owN20+UIAf1N6TPfK 7S+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YQmbzG/x1UDz5U0yQVXExNL9W+QNwv5yyDN0QDopwLM=; b=tlKRQVh/vYI7h+n8ut3nCMH9OYWN43Z4xBitrDi5jpK30qic1P4pkLkkSc/ebBBAWp +VveAwbeOafHdeTjunMkXELitswrBpjDj8HOXDuxjbn/kSrp5U/oAf/3meulscrXAKE7 HQU/odQa4f6EK9nscnkAUefMb0mHRcrCLOUa3CxRl071pS2SaaAP9l7d+wH5G3BiaMo4 NbNLGORGAkZW76TjVFPeGycZBH9Ymwjbu1VFGbPWRBlI6KnGgYb+o61OQZZ/e07oJPFL KgP5kKf280XHcbss4wFSKpmY8MghIW/hS8KAFZzwqQK6JH5X2nQsBhUH3yBdE0JUdrrm uk7w== X-Gm-Message-State: AOAM531rGgzcz1FQH7BimBqbb+biRPVzdkFWfBnBu3ZjUdpiae4z7Vy9 pq5p5Lc4D+oPgeiKaYCK3V+yvA== X-Google-Smtp-Source: ABdhPJyvk99DuGpFOsiT2uRB5kw+R+itIfmHziS8mh58MTBgrfJrSmlj+pJB/4xFtnMj2WkZMk1iTA== X-Received: by 2002:a17:90b:4ace:b0:1df:cb33:5e7e with SMTP id mh14-20020a17090b4ace00b001dfcb335e7emr2962784pjb.5.1653373644635; Mon, 23 May 2022 23:27:24 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([2408:8207:18da:2310:f940:af17:c2f5:8656]) by smtp.gmail.com with ESMTPSA id h5-20020a170902f54500b0016168e90f2dsm6254455plf.219.2022.05.23.23.27.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 May 2022 23:27:24 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, duanxiongchun@bytedance.com, longman@redhat.com, Muchun Song Subject: [PATCH v4 02/11] mm: memcontrol: introduce compact_folio_lruvec_lock_irqsave Date: Tue, 24 May 2022 14:05:42 +0800 Message-Id: <20220524060551.80037-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220524060551.80037-1-songmuchun@bytedance.com> References: <20220524060551.80037-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" If we reuse the objcg APIs to charge LRU pages, the folio_memcg() can be changed when the LRU pages reparented. In this case, we need to acquire the new lruvec lock. lruvec =3D folio_lruvec(folio); // The page is reparented. compact_lock_irqsave(&lruvec->lru_lock, &flags, cc); // Acquired the wrong lruvec lock and need to retry. But compact_lock_irqsave() only take lruvec lock as the parameter, we cannot aware this change. If it can take the page as parameter to acquire the lruvec lock. When the page memcg is changed, we can use the folio_memcg() detect whether we need to reacquire the new lruvec lock. So compact_lock_irqsave() is not suitable for us. Similar to folio_lruvec_lock_irqsave(), introduce compact_folio_lruvec_lock_irqsave() to acquire the lruvec lock in the compaction routine. Signed-off-by: Muchun Song --- mm/compaction.c | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index fe915db6149b..817098817302 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -509,6 +509,29 @@ static bool compact_lock_irqsave(spinlock_t *lock, uns= igned long *flags, return true; } =20 +static struct lruvec * +compact_folio_lruvec_lock_irqsave(struct folio *folio, unsigned long *flag= s, + struct compact_control *cc) +{ + struct lruvec *lruvec; + + lruvec =3D folio_lruvec(folio); + + /* Track if the lock is contended in async mode */ + if (cc->mode =3D=3D MIGRATE_ASYNC && !cc->contended) { + if (spin_trylock_irqsave(&lruvec->lru_lock, *flags)) + goto out; + + cc->contended =3D true; + } + + spin_lock_irqsave(&lruvec->lru_lock, *flags); +out: + lruvec_memcg_debug(lruvec, folio); + + return lruvec; +} + /* * Compaction requires the taking of some coarse locks that are potentially * very heavily contended. The lock should be periodically unlocked to avo= id @@ -844,6 +867,7 @@ isolate_migratepages_block(struct compact_control *cc, = unsigned long low_pfn, =20 /* Time to isolate some pages for migration */ for (; low_pfn < end_pfn; low_pfn++) { + struct folio *folio; =20 if (skip_on_failure && low_pfn >=3D next_skip_pfn) { /* @@ -1065,18 +1089,17 @@ isolate_migratepages_block(struct compact_control *= cc, unsigned long low_pfn, if (!TestClearPageLRU(page)) goto isolate_fail_put; =20 - lruvec =3D folio_lruvec(page_folio(page)); + folio =3D page_folio(page); + lruvec =3D folio_lruvec(folio); =20 /* If we already hold the lock, we can skip some rechecking */ if (lruvec !=3D locked) { if (locked) unlock_page_lruvec_irqrestore(locked, flags); =20 - compact_lock_irqsave(&lruvec->lru_lock, &flags, cc); + lruvec =3D compact_folio_lruvec_lock_irqsave(folio, &flags, cc); locked =3D lruvec; =20 - lruvec_memcg_debug(lruvec, page_folio(page)); - /* Try get exclusive access under lock */ if (!skip_updated) { skip_updated =3D true; --=20 2.11.0 From nobody Sun May 5 20:57:33 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A0FAC433EF for ; Tue, 24 May 2022 06:27:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234987AbiEXG1l (ORCPT ); Tue, 24 May 2022 02:27:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47978 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234982AbiEXG1h (ORCPT ); Tue, 24 May 2022 02:27:37 -0400 Received: from mail-pg1-x52d.google.com (mail-pg1-x52d.google.com [IPv6:2607:f8b0:4864:20::52d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E7A3C95DC7 for ; Mon, 23 May 2022 23:27:32 -0700 (PDT) Received: by mail-pg1-x52d.google.com with SMTP id j21so15322883pga.13 for ; Mon, 23 May 2022 23:27:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Ps+xkks9BA0IHbJ7e2Vp4q031wMw8KJxnkj0GJ9dypU=; b=tifRAsabe/xz0iekv4yC+Zh0Vn2uMaOfgndI2cnHadPnG1WYTFhrpunDnj5seDRE/W KgsNtldScqY9J5t7REermRIAZWnR+n5sVNHU8MCs5Jo0rXscKfdhZwMOaMfD/AHmhtx8 4SpEvEVw9VKd7rcOFcmrDrG4teOq+leHY0+fuLbOFSJ74c4x9O/JtXynpf11F25m638x IS++bfkZ5xRuuCpHynZrX3rrE+PpLWvxyIkJX3LYUw1+QBP/QQFqwloPiiDQqyKvoBB5 uk+bO+IFJoRgl9tDxy2r75yhWUUm8+olsPngbOGfrNRsu3de95nV1s4PmqgHAHBF5TtG L13Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ps+xkks9BA0IHbJ7e2Vp4q031wMw8KJxnkj0GJ9dypU=; b=cj83AotIcLUvZY3Pu+x2fK/TuoQ4lCcgM3+zlXBjIrQB8IAom/hIFYTvf2n+xWhMrb YVKUzqi7oAYvp/7/FOw06dFvUseOg+ScvHT9JNxkN4N+Cd/BKtzqfm+M+gM51HG84r5R vBCHwqCj9OSMOzdTmRhC+tgOnXl3UEZxk7mkdM3oC8FexAxgfD/8qoWW7cjyOfngTXkM j69My8sWtT+pmr+k18jrq2jJ0E1AsTDJx/2BdSkXLR2sGkF2VZukUyfRn4DXFBGHKvQs D98150JVnxSunT48uvFqoK+NrNjmOfU60U3D0ZshfvTafQfCi4wWmDu5lseycNa4ljDl RkkQ== X-Gm-Message-State: AOAM530wwv0PpjtIkIbS7mRmnlUWsOsm1Ss9hkKMGJv9bEtpuYgUW0Nl Kwi1rZxUzLFWfIpPex2jtd82AQ== X-Google-Smtp-Source: ABdhPJwxDpeHOj9+v5T/IxrO944plygT05jhexQ150vcKGGXcVWfFtzRH51uaiJCjcHoU5zrf/xxQA== X-Received: by 2002:a65:6c12:0:b0:3db:a518:ff1b with SMTP id y18-20020a656c12000000b003dba518ff1bmr22603702pgu.315.1653373652331; Mon, 23 May 2022 23:27:32 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([2408:8207:18da:2310:f940:af17:c2f5:8656]) by smtp.gmail.com with ESMTPSA id h5-20020a170902f54500b0016168e90f2dsm6254455plf.219.2022.05.23.23.27.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 May 2022 23:27:32 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, duanxiongchun@bytedance.com, longman@redhat.com, Muchun Song Subject: [PATCH v4 03/11] mm: memcontrol: make lruvec lock safe when LRU pages are reparented Date: Tue, 24 May 2022 14:05:43 +0800 Message-Id: <20220524060551.80037-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220524060551.80037-1-songmuchun@bytedance.com> References: <20220524060551.80037-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" The diagram below shows how to make the folio lruvec lock safe when LRU pages are reparented. folio_lruvec_lock(folio) retry: lruvec =3D folio_lruvec(folio); // The folio is reparented at this time. spin_lock(&lruvec->lru_lock); if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) // Acquired the wrong lruvec lock and need to retry. // Because this folio is on the parent memcg lruvec list. goto retry; // If we reach here, it means that folio_memcg(folio) is stable. memcg_reparent_objcgs(memcg) // lruvec belongs to memcg and lruvec_parent belongs to parent memcg. spin_lock(&lruvec->lru_lock); spin_lock(&lruvec_parent->lru_lock); // Move all the pages from the lruvec list to the parent lruvec list. spin_unlock(&lruvec_parent->lru_lock); spin_unlock(&lruvec->lru_lock); After we acquire the lruvec lock, we need to check whether the folio is reparented. If so, we need to reacquire the new lruvec lock. On the routine of the LRU pages reparenting, we will also acquire the lruvec lock (will be implemented in the later patch). So folio_memcg() cannot be changed when we hold the lruvec lock. Since lruvec_memcg(lruvec) is always equal to folio_memcg(folio) after we hold the lruvec lock, lruvec_memcg_debug() check is pointless. So remove it. This is a preparation for reparenting the LRU pages. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 18 +++----------- mm/compaction.c | 10 +++++++- mm/memcontrol.c | 62 +++++++++++++++++++++++++++++-------------= ---- mm/swap.c | 4 +++ 4 files changed, 55 insertions(+), 39 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index ff1c1dd7e762..4042e4d21fe2 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -752,7 +752,9 @@ static inline struct lruvec *mem_cgroup_lruvec(struct m= em_cgroup *memcg, * folio_lruvec - return lruvec for isolating/putting an LRU folio * @folio: Pointer to the folio. * - * This function relies on folio->mem_cgroup being stable. + * The lruvec can be changed to its parent lruvec when the page reparented. + * The caller need to recheck if it cares about this changes (just like + * folio_lruvec_lock() does). */ static inline struct lruvec *folio_lruvec(struct folio *folio) { @@ -771,15 +773,6 @@ struct lruvec *folio_lruvec_lock_irq(struct folio *fol= io); struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio, unsigned long *flags); =20 -#ifdef CONFIG_DEBUG_VM -void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio); -#else -static inline -void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio) -{ -} -#endif - static inline struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css){ return css ? container_of(css, struct mem_cgroup, css) : NULL; @@ -1240,11 +1233,6 @@ static inline struct lruvec *folio_lruvec(struct fol= io *folio) return &pgdat->__lruvec; } =20 -static inline -void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio) -{ -} - static inline struct mem_cgroup *parent_mem_cgroup(struct mem_cgroup *memc= g) { return NULL; diff --git a/mm/compaction.c b/mm/compaction.c index 817098817302..1692b17db781 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -515,6 +515,8 @@ compact_folio_lruvec_lock_irqsave(struct folio *folio, = unsigned long *flags, { struct lruvec *lruvec; =20 + rcu_read_lock(); +retry: lruvec =3D folio_lruvec(folio); =20 /* Track if the lock is contended in async mode */ @@ -527,7 +529,13 @@ compact_folio_lruvec_lock_irqsave(struct folio *folio,= unsigned long *flags, =20 spin_lock_irqsave(&lruvec->lru_lock, *flags); out: - lruvec_memcg_debug(lruvec, folio); + if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) { + spin_unlock_irqrestore(&lruvec->lru_lock, *flags); + goto retry; + } + + /* See the comments in folio_lruvec_lock(). */ + rcu_read_unlock(); =20 return lruvec; } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 6de0d3e53eb1..b38a77f6696f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1199,23 +1199,6 @@ int mem_cgroup_scan_tasks(struct mem_cgroup *memcg, return ret; } =20 -#ifdef CONFIG_DEBUG_VM -void lruvec_memcg_debug(struct lruvec *lruvec, struct folio *folio) -{ - struct mem_cgroup *memcg; - - if (mem_cgroup_disabled()) - return; - - memcg =3D folio_memcg(folio); - - if (!memcg) - VM_BUG_ON_FOLIO(lruvec_memcg(lruvec) !=3D root_mem_cgroup, folio); - else - VM_BUG_ON_FOLIO(lruvec_memcg(lruvec) !=3D memcg, folio); -} -#endif - /** * folio_lruvec_lock - Lock the lruvec for a folio. * @folio: Pointer to the folio. @@ -1230,10 +1213,23 @@ void lruvec_memcg_debug(struct lruvec *lruvec, stru= ct folio *folio) */ struct lruvec *folio_lruvec_lock(struct folio *folio) { - struct lruvec *lruvec =3D folio_lruvec(folio); + struct lruvec *lruvec; =20 + rcu_read_lock(); +retry: + lruvec =3D folio_lruvec(folio); spin_lock(&lruvec->lru_lock); - lruvec_memcg_debug(lruvec, folio); + + if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) { + spin_unlock(&lruvec->lru_lock); + goto retry; + } + + /* + * Preemption is disabled in the internal of spin_lock, which can serve + * as RCU read-side critical sections. + */ + rcu_read_unlock(); =20 return lruvec; } @@ -1253,10 +1249,20 @@ struct lruvec *folio_lruvec_lock(struct folio *foli= o) */ struct lruvec *folio_lruvec_lock_irq(struct folio *folio) { - struct lruvec *lruvec =3D folio_lruvec(folio); + struct lruvec *lruvec; =20 + rcu_read_lock(); +retry: + lruvec =3D folio_lruvec(folio); spin_lock_irq(&lruvec->lru_lock); - lruvec_memcg_debug(lruvec, folio); + + if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) { + spin_unlock_irq(&lruvec->lru_lock); + goto retry; + } + + /* See the comments in folio_lruvec_lock(). */ + rcu_read_unlock(); =20 return lruvec; } @@ -1278,10 +1284,20 @@ struct lruvec *folio_lruvec_lock_irq(struct folio *= folio) struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio, unsigned long *flags) { - struct lruvec *lruvec =3D folio_lruvec(folio); + struct lruvec *lruvec; =20 + rcu_read_lock(); +retry: + lruvec =3D folio_lruvec(folio); spin_lock_irqsave(&lruvec->lru_lock, *flags); - lruvec_memcg_debug(lruvec, folio); + + if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) { + spin_unlock_irqrestore(&lruvec->lru_lock, *flags); + goto retry; + } + + /* See the comments in folio_lruvec_lock(). */ + rcu_read_unlock(); =20 return lruvec; } diff --git a/mm/swap.c b/mm/swap.c index 7e320ec08c6a..9680f2fc48b1 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -303,6 +303,10 @@ void lru_note_cost(struct lruvec *lruvec, bool file, u= nsigned int nr_pages) =20 void lru_note_cost_folio(struct folio *folio) { + /* + * The rcu read lock is held by the caller, so we do not need to + * care about the lruvec returned by folio_lruvec() being released. + */ lru_note_cost(folio_lruvec(folio), folio_is_file_lru(folio), folio_nr_pages(folio)); } --=20 2.11.0 From nobody Sun May 5 20:57:33 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D6ACC433EF for ; Tue, 24 May 2022 06:28:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235002AbiEXG2D (ORCPT ); Tue, 24 May 2022 02:28:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48128 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235023AbiEXG1w (ORCPT ); Tue, 24 May 2022 02:27:52 -0400 Received: from mail-pf1-x42f.google.com (mail-pf1-x42f.google.com [IPv6:2607:f8b0:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C524895DDD for ; Mon, 23 May 2022 23:27:42 -0700 (PDT) Received: by mail-pf1-x42f.google.com with SMTP id c14so15640159pfn.2 for ; Mon, 23 May 2022 23:27:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=C4QIRjaHJeiHbHgGHewQVJI5rixr5jafar8TUaLFgWA=; b=MkCTHNsxUNBMErlaDCUFDA8ek9BRW+c1NG6n7H9gxyOA2KBsh++mlxbHmVlgRQXAON wvVeQGECKUuL2Py5+B7jUql9VwKcy287DQOLCPZJhDIBHWfNSk2OipPfzEyRNGouNdJX 3TJ7C6xANQxgkGQlsnWVvRiUlyeFR2U4oiozTqrQOSfpykGln3SFiPDydFQOlnVsU77I 0A47+5k3HyNlKjGZLkGkxq1lpjQh0d5uscl1GKB80laEELcMf5YaqYc+R0gwP3lx+Fqx A+5X6VDGns5zFewPT8ut4ZfqX2MWolvowSpMiv4xLng/W887aOhxY7yntiIcWLOnL+rk nxDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=C4QIRjaHJeiHbHgGHewQVJI5rixr5jafar8TUaLFgWA=; b=W+T1APnKJ6CxqrD004VRsFPWoeviEEl/ucJjs//7rPYmcSSHtAKpv/FmeBj+sXzsUr E3sy1QPOe5ndqZ32m50l8ulAzOonPljlswoeddMVAbS/5C8Et8e2BD2K63iGXFzkvXYp oonDZaO/Ir5tVEMJpiBDEncwh0W4ye2hZNiIJr0p/exgG0sLyuyFSO3ugBOJSJcjHmzj WrTNFYW8km3h8nOxXlTb9GQ13Cv/aYnT2g/o13z35ZMsK69fjMsFzEROKe/dT0G0RbSN UTslm2aKmBqFIHOQe5aVzpCNTHTJ8JUcBGGAYbqSFCqE1mYxR524CjTtbI4BQ1HOkiSK J52w== X-Gm-Message-State: AOAM5310CISU4whgxYNYHrjcTDcX1Da2nQ9Mg8/UYB+BgquXMOhNbuig xjLfoDGOL0LTgXWbkdQTkNo/uw== X-Google-Smtp-Source: ABdhPJxRnJODI5LFjdRDbVjSWigQvz3LogjzZlyMVA5cK9WUh3QvAyB3wQNtKrYi6SKeFqyIERSv8A== X-Received: by 2002:a63:5625:0:b0:3db:55dd:37ae with SMTP id k37-20020a635625000000b003db55dd37aemr22489609pgb.611.1653373662140; Mon, 23 May 2022 23:27:42 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([2408:8207:18da:2310:f940:af17:c2f5:8656]) by smtp.gmail.com with ESMTPSA id h5-20020a170902f54500b0016168e90f2dsm6254455plf.219.2022.05.23.23.27.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 May 2022 23:27:41 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, duanxiongchun@bytedance.com, longman@redhat.com, Muchun Song Subject: [PATCH v4 04/11] mm: vmscan: rework move_pages_to_lru() Date: Tue, 24 May 2022 14:05:44 +0800 Message-Id: <20220524060551.80037-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220524060551.80037-1-songmuchun@bytedance.com> References: <20220524060551.80037-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" In the later patch, we will reparent the LRU pages. The pages moved to appropriate LRU list can be reparented during the process of the move_pages_to_lru(). So holding a lruvec lock by the caller is wrong, we should use the more general interface of folio_lruvec_relock_irq() to acquire the correct lruvec lock. Signed-off-by: Muchun Song Acked-by: Johannes Weiner Acked-by: Roman Gushchin --- mm/vmscan.c | 49 +++++++++++++++++++++++++------------------------ 1 file changed, 25 insertions(+), 24 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 1678802e03e7..761d5e0dd78d 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2230,23 +2230,28 @@ static int too_many_isolated(struct pglist_data *pg= dat, int file, * move_pages_to_lru() moves pages from private @list to appropriate LRU l= ist. * On return, @list is reused as a list of pages to be freed by the caller. * - * Returns the number of pages moved to the given lruvec. + * Returns the number of pages moved to the appropriate LRU list. + * + * Note: The caller must not hold any lruvec lock. */ -static unsigned int move_pages_to_lru(struct lruvec *lruvec, - struct list_head *list) +static unsigned int move_pages_to_lru(struct list_head *list) { - int nr_pages, nr_moved =3D 0; + int nr_moved =3D 0; + struct lruvec *lruvec =3D NULL; LIST_HEAD(pages_to_free); - struct page *page; =20 while (!list_empty(list)) { - page =3D lru_to_page(list); + int nr_pages; + struct folio *folio =3D lru_to_folio(list); + struct page *page =3D &folio->page; + + lruvec =3D folio_lruvec_relock_irq(folio, lruvec); VM_BUG_ON_PAGE(PageLRU(page), page); list_del(&page->lru); if (unlikely(!page_evictable(page))) { - spin_unlock_irq(&lruvec->lru_lock); + unlock_page_lruvec_irq(lruvec); putback_lru_page(page); - spin_lock_irq(&lruvec->lru_lock); + lruvec =3D NULL; continue; } =20 @@ -2267,20 +2272,16 @@ static unsigned int move_pages_to_lru(struct lruvec= *lruvec, __clear_page_lru_flags(page); =20 if (unlikely(PageCompound(page))) { - spin_unlock_irq(&lruvec->lru_lock); + unlock_page_lruvec_irq(lruvec); destroy_compound_page(page); - spin_lock_irq(&lruvec->lru_lock); + lruvec =3D NULL; } else list_add(&page->lru, &pages_to_free); =20 continue; } =20 - /* - * All pages were isolated from the same lruvec (and isolation - * inhibits memcg migration). - */ - VM_BUG_ON_PAGE(!folio_matches_lruvec(page_folio(page), lruvec), page); + VM_BUG_ON_PAGE(!folio_matches_lruvec(folio, lruvec), page); add_page_to_lru_list(page, lruvec); nr_pages =3D thp_nr_pages(page); nr_moved +=3D nr_pages; @@ -2288,6 +2289,8 @@ static unsigned int move_pages_to_lru(struct lruvec *= lruvec, workingset_age_nonresident(lruvec, nr_pages); } =20 + if (lruvec) + unlock_page_lruvec_irq(lruvec); /* * To save our caller's stack, now use input list for pages to free. */ @@ -2359,16 +2362,16 @@ shrink_inactive_list(unsigned long nr_to_scan, stru= ct lruvec *lruvec, =20 nr_reclaimed =3D shrink_page_list(&page_list, pgdat, sc, &stat, false); =20 - spin_lock_irq(&lruvec->lru_lock); - move_pages_to_lru(lruvec, &page_list); + move_pages_to_lru(&page_list); =20 + local_irq_disable(); __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); item =3D current_is_kswapd() ? PGSTEAL_KSWAPD : PGSTEAL_DIRECT; if (!cgroup_reclaim(sc)) __count_vm_events(item, nr_reclaimed); __count_memcg_events(lruvec_memcg(lruvec), item, nr_reclaimed); __count_vm_events(PGSTEAL_ANON + file, nr_reclaimed); - spin_unlock_irq(&lruvec->lru_lock); + local_irq_enable(); =20 lru_note_cost(lruvec, file, stat.nr_pageout); mem_cgroup_uncharge_list(&page_list); @@ -2498,18 +2501,16 @@ static void shrink_active_list(unsigned long nr_to_= scan, /* * Move pages back to the lru list. */ - spin_lock_irq(&lruvec->lru_lock); - - nr_activate =3D move_pages_to_lru(lruvec, &l_active); - nr_deactivate =3D move_pages_to_lru(lruvec, &l_inactive); + nr_activate =3D move_pages_to_lru(&l_active); + nr_deactivate =3D move_pages_to_lru(&l_inactive); /* Keep all free pages in l_active list */ list_splice(&l_inactive, &l_active); =20 + local_irq_disable(); __count_vm_events(PGDEACTIVATE, nr_deactivate); __count_memcg_events(lruvec_memcg(lruvec), PGDEACTIVATE, nr_deactivate); - __mod_node_page_state(pgdat, NR_ISOLATED_ANON + file, -nr_taken); - spin_unlock_irq(&lruvec->lru_lock); + local_irq_enable(); =20 mem_cgroup_uncharge_list(&l_active); free_unref_page_list(&l_active); --=20 2.11.0 From nobody Sun May 5 20:57:33 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B352C433FE for ; Tue, 24 May 2022 06:28:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234924AbiEXG16 (ORCPT ); Tue, 24 May 2022 02:27:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48592 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235057AbiEXG1y (ORCPT ); Tue, 24 May 2022 02:27:54 -0400 Received: from mail-pf1-x42f.google.com (mail-pf1-x42f.google.com [IPv6:2607:f8b0:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6766A9729B for ; Mon, 23 May 2022 23:27:49 -0700 (PDT) Received: by mail-pf1-x42f.google.com with SMTP id bo5so15623584pfb.4 for ; Mon, 23 May 2022 23:27:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=348SaC7WcgSM/sY/xPmE22D0TXDMzza3SSCcL487100=; b=0+r2wt6T13k+HnQR9gTMlwFAXOvTOPrOAa7wvRnvNatQt+nuKscaUIr/xDsTQt8dx7 cmfbnMdBcmrFZqxpV056iUOHc17yG9yoWbd+kugli0Dk0ER+5jrqBy6nLydMe2HVyu0K OCM5YN0lcvsR0oPmGjixkjfBfZUFhKNLmDfjV/qiA7CdjRwKrE7G1bega5RFViTRveKf ALxIf3bdEqBOsFd/aAlY8MjkwjKJ89pjPsIsxgUqSMzjoKQgMas8Mm73sZaR4KHMyrko r0xabTvv49viltamLfcZA36N9pC4w3xW7PPrxZfYBNexHQZrpO0Dd5cde8KkIupcosQ0 O5Sw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=348SaC7WcgSM/sY/xPmE22D0TXDMzza3SSCcL487100=; b=wgRXnoAep5ekSrpF8xH5IQb+p0tla7T9J27t/u64/FShDRDELkA8x2I5IJEAcaxf+i 9XZbpaSbCV+AP2wC7Tki7rllp9RK2fDOLLH6jUf/b3G7jfQ/6pZRmxu4Quh4/Y2QK7Ks rdoV2eMcvTnoS8reka0VITW99AsM/p8/gw6olNgRrlbkVK+X4nZjzvYeq7ZBs1gPbCRt ue4Jo+lHen0QsBcbTaaTWxyCIO9Ah/5EDIOgKV5RJrxM3zfTtW39UW/EfTaQcHpJO8DA QkOvKHh7+WFbPlO2NCYM4fyRBloScRj1+ufooWHDCRoC1ksZQMfuANhLHKEWmzqk7I9h h06w== X-Gm-Message-State: AOAM532gQiqzVJqev/hNolCXqgR4lXePCYGJkTzs4ngIKwAWqe2UM9G1 D8rGSm8XmN44ZZDmDzMP9TR8wg== X-Google-Smtp-Source: ABdhPJyBGXSY/zzDQO6+7gsa+C6gYcL6fGofFEFJj++nDfwdp0pTKf3/FX7KPrkYhj3cyiG5CXaRWg== X-Received: by 2002:a63:5f83:0:b0:3fa:8154:dddd with SMTP id t125-20020a635f83000000b003fa8154ddddmr4679828pgb.486.1653373668901; Mon, 23 May 2022 23:27:48 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([2408:8207:18da:2310:f940:af17:c2f5:8656]) by smtp.gmail.com with ESMTPSA id h5-20020a170902f54500b0016168e90f2dsm6254455plf.219.2022.05.23.23.27.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 May 2022 23:27:48 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, duanxiongchun@bytedance.com, longman@redhat.com, Muchun Song Subject: [PATCH v4 05/11] mm: thp: introduce folio_split_queue_lock{_irqsave}() Date: Tue, 24 May 2022 14:05:45 +0800 Message-Id: <20220524060551.80037-6-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220524060551.80037-1-songmuchun@bytedance.com> References: <20220524060551.80037-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" We should make thp deferred split queue lock safe when LRU pages are reparented. Similar to folio_lruvec_lock{_irqsave, _irq}(), we introduce folio_split_queue_lock{_irqsave}() to make the deferred split queue lock easier to be reparented. And in the next patch, we can use a similar approach (just like lruvec lock does) to make thp deferred split queue lock safe when the LRU pages reparented. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 10 +++++ mm/huge_memory.c | 100 +++++++++++++++++++++++++++++++++--------= ---- 2 files changed, 84 insertions(+), 26 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 4042e4d21fe2..8c2f1ba2f471 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1650,6 +1650,11 @@ int alloc_shrinker_info(struct mem_cgroup *memcg); void free_shrinker_info(struct mem_cgroup *memcg); void set_shrinker_bit(struct mem_cgroup *memcg, int nid, int shrinker_id); void reparent_shrinker_deferred(struct mem_cgroup *memcg); + +static inline int shrinker_id(struct shrinker *shrinker) +{ + return shrinker->id; +} #else #define mem_cgroup_sockets_enabled 0 static inline void mem_cgroup_sk_alloc(struct sock *sk) { }; @@ -1663,6 +1668,11 @@ static inline void set_shrinker_bit(struct mem_cgrou= p *memcg, int nid, int shrinker_id) { } + +static inline int shrinker_id(struct shrinker *shrinker) +{ + return -1; +} #endif =20 #ifdef CONFIG_MEMCG_KMEM diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 910a138e9859..ea152bde441e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -503,25 +503,74 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_str= uct *vma) } =20 #ifdef CONFIG_MEMCG -static inline struct deferred_split *get_deferred_split_queue(struct page = *page) +static inline struct mem_cgroup *folio_split_queue_memcg(struct folio *fol= io, + struct deferred_split *queue) { - struct mem_cgroup *memcg =3D page_memcg(compound_head(page)); - struct pglist_data *pgdat =3D NODE_DATA(page_to_nid(page)); + if (mem_cgroup_disabled()) + return NULL; + if (&NODE_DATA(folio_nid(folio))->deferred_split_queue =3D=3D queue) + return NULL; + return container_of(queue, struct mem_cgroup, deferred_split_queue); +} =20 - if (memcg) - return &memcg->deferred_split_queue; - else - return &pgdat->deferred_split_queue; +static inline struct deferred_split *folio_memcg_split_queue(struct folio = *folio) +{ + struct mem_cgroup *memcg =3D folio_memcg(folio); + + return memcg ? &memcg->deferred_split_queue : NULL; } #else -static inline struct deferred_split *get_deferred_split_queue(struct page = *page) +static inline struct mem_cgroup *folio_split_queue_memcg(struct folio *fol= io, + struct deferred_split *queue) { - struct pglist_data *pgdat =3D NODE_DATA(page_to_nid(page)); + return NULL; +} =20 - return &pgdat->deferred_split_queue; +static inline struct deferred_split *folio_memcg_split_queue(struct folio = *folio) +{ + return NULL; } #endif =20 +static struct deferred_split *folio_split_queue(struct folio *folio) +{ + struct deferred_split *queue =3D folio_memcg_split_queue(folio); + + return queue ? : &NODE_DATA(folio_nid(folio))->deferred_split_queue; +} + +static struct deferred_split *folio_split_queue_lock(struct folio *folio) +{ + struct deferred_split *queue; + + queue =3D folio_split_queue(folio); + spin_lock(&queue->split_queue_lock); + + return queue; +} + +static struct deferred_split * +folio_split_queue_lock_irqsave(struct folio *folio, unsigned long *flags) +{ + struct deferred_split *queue; + + queue =3D folio_split_queue(folio); + spin_lock_irqsave(&queue->split_queue_lock, *flags); + + return queue; +} + +static inline void split_queue_unlock(struct deferred_split *queue) +{ + spin_unlock(&queue->split_queue_lock); +} + +static inline void split_queue_unlock_irqrestore(struct deferred_split *qu= eue, + unsigned long flags) +{ + spin_unlock_irqrestore(&queue->split_queue_lock, flags); +} + void prep_transhuge_page(struct page *page) { /* @@ -2489,7 +2538,7 @@ int split_huge_page_to_list(struct page *page, struct= list_head *list) { struct folio *folio =3D page_folio(page); struct page *head =3D &folio->page; - struct deferred_split *ds_queue =3D get_deferred_split_queue(head); + struct deferred_split *ds_queue; XA_STATE(xas, &head->mapping->i_pages, head->index); struct anon_vma *anon_vma =3D NULL; struct address_space *mapping =3D NULL; @@ -2581,13 +2630,13 @@ int split_huge_page_to_list(struct page *page, stru= ct list_head *list) } =20 /* Prevent deferred_split_scan() touching ->_refcount */ - spin_lock(&ds_queue->split_queue_lock); + ds_queue =3D folio_split_queue_lock(folio); if (page_ref_freeze(head, 1 + extra_pins)) { if (!list_empty(page_deferred_list(head))) { ds_queue->split_queue_len--; list_del(page_deferred_list(head)); } - spin_unlock(&ds_queue->split_queue_lock); + split_queue_unlock(ds_queue); if (mapping) { int nr =3D thp_nr_pages(head); =20 @@ -2605,7 +2654,7 @@ int split_huge_page_to_list(struct page *page, struct= list_head *list) __split_huge_page(page, list, end); ret =3D 0; } else { - spin_unlock(&ds_queue->split_queue_lock); + split_queue_unlock(ds_queue); fail: if (mapping) xas_unlock(&xas); @@ -2630,25 +2679,23 @@ int split_huge_page_to_list(struct page *page, stru= ct list_head *list) =20 void free_transhuge_page(struct page *page) { - struct deferred_split *ds_queue =3D get_deferred_split_queue(page); + struct deferred_split *ds_queue; unsigned long flags; =20 - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); + ds_queue =3D folio_split_queue_lock_irqsave(page_folio(page), &flags); if (!list_empty(page_deferred_list(page))) { ds_queue->split_queue_len--; list_del(page_deferred_list(page)); } - spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); + split_queue_unlock_irqrestore(ds_queue, flags); free_compound_page(page); } =20 void deferred_split_huge_page(struct page *page) { - struct deferred_split *ds_queue =3D get_deferred_split_queue(page); -#ifdef CONFIG_MEMCG - struct mem_cgroup *memcg =3D page_memcg(compound_head(page)); -#endif + struct deferred_split *ds_queue; unsigned long flags; + struct folio *folio =3D page_folio(page); =20 VM_BUG_ON_PAGE(!PageTransHuge(page), page); =20 @@ -2665,18 +2712,19 @@ void deferred_split_huge_page(struct page *page) if (PageSwapCache(page)) return; =20 - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); + ds_queue =3D folio_split_queue_lock_irqsave(folio, &flags); if (list_empty(page_deferred_list(page))) { + struct mem_cgroup *memcg; + + memcg =3D folio_split_queue_memcg(folio, ds_queue); count_vm_event(THP_DEFERRED_SPLIT_PAGE); list_add_tail(page_deferred_list(page), &ds_queue->split_queue); ds_queue->split_queue_len++; -#ifdef CONFIG_MEMCG if (memcg) set_shrinker_bit(memcg, page_to_nid(page), - deferred_split_shrinker.id); -#endif + shrinker_id(&deferred_split_shrinker)); } - spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); + split_queue_unlock_irqrestore(ds_queue, flags); } =20 static unsigned long deferred_split_count(struct shrinker *shrink, --=20 2.11.0 From nobody Sun May 5 20:57:33 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDC0AC433EF for ; Tue, 24 May 2022 06:28:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235011AbiEXG2I (ORCPT ); Tue, 24 May 2022 02:28:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48466 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234985AbiEXG17 (ORCPT ); Tue, 24 May 2022 02:27:59 -0400 Received: from mail-pg1-x52f.google.com (mail-pg1-x52f.google.com [IPv6:2607:f8b0:4864:20::52f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 813F172E1F for ; Mon, 23 May 2022 23:27:57 -0700 (PDT) Received: by mail-pg1-x52f.google.com with SMTP id a9so13513213pgv.12 for ; Mon, 23 May 2022 23:27:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2mx9bPTY7AEcp+j24lhXXPGVls/yhivZ1agLRVxoLiw=; b=v+S3AKD3IZY1715alsr8yTzbsVUYEJKe+49LFvD+HdqE9G/w+5EjzEN0EwaEDyGrgn i9Ya3PVga4iPV58AJIeN1oTrMCpdjatANgHWEgSW8H8UHrlDSX4k0sGGqWcV7AXdFW13 EAk3nLfKEjQm6om5mFrDB7o4f02o242GQEXT/WIFEL4yR/3PpF7M5gtnBfzi+dSVbTxu StmM8BZkmo2YtgyD2u3N/KxcgJyaZoQ5QI6XjNdjDbH/gX13NzTb6mqHMmzWnw0wjnjt 6G2U6k80RGyCUtx5VNIlrbZZrmZIkm6UOOp3C70ojMsYgXdvqPAfokDNwMjQRVy8THGC Q/Ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2mx9bPTY7AEcp+j24lhXXPGVls/yhivZ1agLRVxoLiw=; b=AFCkqLzTMjh69gAuhKv01elZcabg6YuuvBgKYM+X3BgrlExUoER5jgI2GZyat4EYx+ XBgQ7hIXkZZUhoZKMt56Pj0C77SEjpqQzVHZJaI6s1VKFrjY78UwjX7TtdTxi7jh+nNW LVK11KAB2F7d7KtclClldRbBP3sDFPUC8+9QnLNXuBqARnsja8olnv4uU8q05VMefwML rvjtsykQdQqLMGfcqPqbHO6yFZuodWZfj6ZEOb6TtUvuUpNuAXKjgmy4TWFptPwkaz8R u0FChzD5fc7kHaDySRzm0l5/1OdlmWYLPXGJhAc5ZMirdd2yWt99TxAQ4yYeuz9IWEtP 58/g== X-Gm-Message-State: AOAM530LzaPpFPyUTerh8MzsQyRRglUf/9v/FoCGCGEegf2DaFgJHA4j ND8mH/dazOsJSyZDIu3zKf6KJg== X-Google-Smtp-Source: ABdhPJx7zUzEoXA0gBdm9ca0a2T/LuHVii8K0J7cqNEy85oapg7WfQ/2eJWO3OwS2W/T0/ARUmA1jA== X-Received: by 2002:a62:f901:0:b0:518:307a:b392 with SMTP id o1-20020a62f901000000b00518307ab392mr27007106pfh.44.1653373677075; Mon, 23 May 2022 23:27:57 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([2408:8207:18da:2310:f940:af17:c2f5:8656]) by smtp.gmail.com with ESMTPSA id h5-20020a170902f54500b0016168e90f2dsm6254455plf.219.2022.05.23.23.27.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 May 2022 23:27:56 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, duanxiongchun@bytedance.com, longman@redhat.com, Muchun Song Subject: [PATCH v4 06/11] mm: thp: make split queue lock safe when LRU pages are reparented Date: Tue, 24 May 2022 14:05:46 +0800 Message-Id: <20220524060551.80037-7-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220524060551.80037-1-songmuchun@bytedance.com> References: <20220524060551.80037-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" Similar to the lruvec lock, we use the same approach to make the split queue lock safe when LRU pages are reparented. Signed-off-by: Muchun Song Acked-by: Roman Gushchin . --- mm/huge_memory.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index ea152bde441e..cc596034c487 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -543,9 +543,22 @@ static struct deferred_split *folio_split_queue_lock(s= truct folio *folio) { struct deferred_split *queue; =20 + rcu_read_lock(); +retry: queue =3D folio_split_queue(folio); spin_lock(&queue->split_queue_lock); =20 + if (unlikely(folio_split_queue_memcg(folio, queue) !=3D folio_memcg(folio= ))) { + spin_unlock(&queue->split_queue_lock); + goto retry; + } + + /* + * Preemption is disabled in the internal of spin_lock, which can serve + * as RCU read-side critical sections. + */ + rcu_read_unlock(); + return queue; } =20 @@ -554,9 +567,19 @@ folio_split_queue_lock_irqsave(struct folio *folio, un= signed long *flags) { struct deferred_split *queue; =20 + rcu_read_lock(); +retry: queue =3D folio_split_queue(folio); spin_lock_irqsave(&queue->split_queue_lock, *flags); =20 + if (unlikely(folio_split_queue_memcg(folio, queue) !=3D folio_memcg(folio= ))) { + spin_unlock_irqrestore(&queue->split_queue_lock, *flags); + goto retry; + } + + /* See the comments in folio_split_queue_lock(). */ + rcu_read_unlock(); + return queue; } =20 --=20 2.11.0 From nobody Sun May 5 20:57:33 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2968C433F5 for ; Tue, 24 May 2022 06:28:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234606AbiEXG21 (ORCPT ); Tue, 24 May 2022 02:28:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49218 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235055AbiEXG2N (ORCPT ); Tue, 24 May 2022 02:28:13 -0400 Received: from mail-pf1-x42f.google.com (mail-pf1-x42f.google.com [IPv6:2607:f8b0:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B496C95A3F for ; Mon, 23 May 2022 23:28:10 -0700 (PDT) Received: by mail-pf1-x42f.google.com with SMTP id bo5so15623584pfb.4 for ; Mon, 23 May 2022 23:28:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=75mWj9rD1FYusXwu3300I1N1+yPz4lqDem3tyIK/8iA=; b=0mI2DnWXNZfKzi1VTaGi6mnvdQeegw3U+SJgq23SfLkYlQKleDrETfbZEdU6xOWAtE bCZ5ddaCZVykd6rvye2FXZn/i2bzNeSBL0K5W3Hx5TIYO+/mxFVzdGUbRufjOSj1StQd 1Z5KY4SW8ALZiDBV4mukfsSuqPRX6UsudRhj/mHxntCU72WAD7zM2KtK4qGIsX1j8qdw iAxj4TtovNkwlv6vmBK5R0Pr9DlPoWtoHJ+NQ5Fzz/5TUkJqif6z60d3wZF9SoYR2DJ0 J8oxIy3VK5v4Rd9SpiAk0zFNw1CkGUs2szEwWl9fDB497m/yUD3QiQzsZEZg4ea5ZXaG YKZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=75mWj9rD1FYusXwu3300I1N1+yPz4lqDem3tyIK/8iA=; b=i6J27hG0C+nVy29k0Mg7sXG3FIS1R7xUuiCCAbr6ItB9nJ2VAyboSYUk2hluvslx5f Xc6pLzKaV3WjRHCqE9gy2o5JS+QRWWZqSa2Z9L2w8uYlAwBOxT1DA/7Mrf+vET2J3cfM eTRT0lupFkYst6D70dIuHvuftprwyWed2WaZ5/SsuE/SDIabMwF6DgGZHDE3meqEmqTw ZvKqjdEjUUojyOLPm6uTYnUefMvIq+BXo/ZLDs17lsuVVw88aKtTFvWmX68dhtYVNAjq DfG6qcrK+BmtMyWsbNzTGv9jkW2I7W3B7pW0N8j6WLPyM4ecLUsO2LFxFUkh6QXL14ga YoIA== X-Gm-Message-State: AOAM531lcpqv7QFqhEaYlybMPZZaemRMlWilt1daDpJIss4YrTez7JsF TTeXlM1tXmcBDcRMcOg5YNGmvA== X-Google-Smtp-Source: ABdhPJwq2pm6rKdBcWg38OlxbDskxHfLDlbD+/R0a1N0nafcrmm/lkvYoV9/VEYOEw1QS+yVER3/XQ== X-Received: by 2002:a05:6a00:10d0:b0:4f7:5af4:47b6 with SMTP id d16-20020a056a0010d000b004f75af447b6mr27430471pfu.6.1653373690124; Mon, 23 May 2022 23:28:10 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([2408:8207:18da:2310:f940:af17:c2f5:8656]) by smtp.gmail.com with ESMTPSA id h5-20020a170902f54500b0016168e90f2dsm6254455plf.219.2022.05.23.23.27.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 May 2022 23:28:09 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, duanxiongchun@bytedance.com, longman@redhat.com, Muchun Song Subject: [PATCH v4 07/11] mm: memcontrol: make all the callers of {folio,page}_memcg() safe Date: Tue, 24 May 2022 14:05:47 +0800 Message-Id: <20220524060551.80037-8-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220524060551.80037-1-songmuchun@bytedance.com> References: <20220524060551.80037-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" When we use objcg APIs to charge the LRU pages, the page will not hold a reference to the memcg associated with the page. So the caller of the {folio,page}_memcg() should hold an rcu read lock or obtain a reference to the memcg associated with the page to protect memcg from being released. So introduce get_mem_cgroup_from_{page,folio}() to obtain a reference to the memory cgroup associated with the page. In this patch, make all the callers hold an rcu read lock or obtain a reference to the memcg to protect memcg from being released when the LRU pages reparented. We do not need to adjust the callers of {folio,page}_memcg() during the whole process of mem_cgroup_move_task(). Because the cgroup migration and memory cgroup offlining are serialized by @cgroup_mutex. In this routine, the LRU pages cannot be reparented to its parent memory cgroup. So {folio,page}_memcg() is stable and cannot be released. This is a preparation for reparenting the LRU pages. Signed-off-by: Muchun Song --- fs/buffer.c | 4 +-- fs/fs-writeback.c | 23 ++++++++-------- include/linux/memcontrol.h | 51 ++++++++++++++++++++++++++++++++--- include/trace/events/writeback.h | 5 ++++ mm/memcontrol.c | 58 ++++++++++++++++++++++++++++++------= ---- mm/migrate.c | 4 +++ mm/page_io.c | 5 ++-- 7 files changed, 117 insertions(+), 33 deletions(-) diff --git a/fs/buffer.c b/fs/buffer.c index 2b5561ae5d0b..80975a457670 100644 --- a/fs/buffer.c +++ b/fs/buffer.c @@ -819,8 +819,7 @@ struct buffer_head *alloc_page_buffers(struct page *pag= e, unsigned long size, if (retry) gfp |=3D __GFP_NOFAIL; =20 - /* The page lock pins the memcg */ - memcg =3D page_memcg(page); + memcg =3D get_mem_cgroup_from_page(page); old_memcg =3D set_active_memcg(memcg); =20 head =3D NULL; @@ -840,6 +839,7 @@ struct buffer_head *alloc_page_buffers(struct page *pag= e, unsigned long size, set_bh_page(bh, page, offset); } out: + mem_cgroup_put(memcg); set_active_memcg(old_memcg); return head; /* diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c index 1fae0196292a..56612ace8778 100644 --- a/fs/fs-writeback.c +++ b/fs/fs-writeback.c @@ -243,15 +243,13 @@ void __inode_attach_wb(struct inode *inode, struct pa= ge *page) if (inode_cgwb_enabled(inode)) { struct cgroup_subsys_state *memcg_css; =20 - if (page) { - memcg_css =3D mem_cgroup_css_from_page(page); - wb =3D wb_get_create(bdi, memcg_css, GFP_ATOMIC); - } else { - /* must pin memcg_css, see wb_get_create() */ + /* must pin memcg_css, see wb_get_create() */ + if (page) + memcg_css =3D get_mem_cgroup_css_from_page(page); + else memcg_css =3D task_get_css(current, memory_cgrp_id); - wb =3D wb_get_create(bdi, memcg_css, GFP_ATOMIC); - css_put(memcg_css); - } + wb =3D wb_get_create(bdi, memcg_css, GFP_ATOMIC); + css_put(memcg_css); } =20 if (!wb) @@ -868,16 +866,16 @@ void wbc_account_cgroup_owner(struct writeback_contro= l *wbc, struct page *page, if (!wbc->wb || wbc->no_cgroup_owner) return; =20 - css =3D mem_cgroup_css_from_page(page); + css =3D get_mem_cgroup_css_from_page(page); /* dead cgroups shouldn't contribute to inode ownership arbitration */ if (!(css->flags & CSS_ONLINE)) - return; + goto out; =20 id =3D css->id; =20 if (id =3D=3D wbc->wb_id) { wbc->wb_bytes +=3D bytes; - return; + goto out; } =20 if (id =3D=3D wbc->wb_lcand_id) @@ -890,6 +888,9 @@ void wbc_account_cgroup_owner(struct writeback_control = *wbc, struct page *page, wbc->wb_tcand_bytes +=3D bytes; else wbc->wb_tcand_bytes -=3D min(bytes, wbc->wb_tcand_bytes); + +out: + css_put(css); } EXPORT_SYMBOL_GPL(wbc_account_cgroup_owner); =20 diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 8c2f1ba2f471..3a0e2592434e 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -373,7 +373,7 @@ static inline bool folio_memcg_kmem(struct folio *folio= ); * a valid memcg, but can be atomically swapped to the parent memcg. * * The caller must ensure that the returned memcg won't be released: - * e.g. acquire the rcu_read_lock or css_set_lock. + * e.g. acquire the rcu_read_lock or objcg_lock or cgroup_mutex. */ static inline struct mem_cgroup *obj_cgroup_memcg(struct obj_cgroup *objcg) { @@ -454,7 +454,37 @@ static inline struct mem_cgroup *page_memcg(struct pag= e *page) return folio_memcg(page_folio(page)); } =20 -/** +/* + * get_mem_cgroup_from_folio - Obtain a reference on the memory cgroup + * associated with a folio. + * @folio: Pointer to the folio. + * + * Returns a pointer to the memory cgroup (and obtain a reference on it) + * associated with the folio, or NULL. This function assumes that the + * folio is known to have a proper memory cgroup pointer. It's not safe + * to call this function against some type of pages, e.g. slab pages or + * ex-slab pages. + */ +static inline struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *f= olio) +{ + struct mem_cgroup *memcg; + + rcu_read_lock(); +retry: + memcg =3D folio_memcg(folio); + if (unlikely(memcg && !css_tryget(&memcg->css))) + goto retry; + rcu_read_unlock(); + + return memcg; +} + +static inline struct mem_cgroup *get_mem_cgroup_from_page(struct page *pag= e) +{ + return get_mem_cgroup_from_folio(page_folio(page)); +} + +/* * folio_memcg_rcu - Locklessly get the memory cgroup associated with a fo= lio. * @folio: Pointer to the folio. * @@ -873,7 +903,7 @@ static inline bool mm_match_cgroup(struct mm_struct *mm, return match; } =20 -struct cgroup_subsys_state *mem_cgroup_css_from_page(struct page *page); +struct cgroup_subsys_state *get_mem_cgroup_css_from_page(struct page *page= ); ino_t page_cgroup_ino(struct page *page); =20 static inline bool mem_cgroup_online(struct mem_cgroup *memcg) @@ -1047,10 +1077,13 @@ static inline void count_memcg_events(struct mem_cg= roup *memcg, static inline void count_memcg_page_event(struct page *page, enum vm_event_item idx) { - struct mem_cgroup *memcg =3D page_memcg(page); + struct mem_cgroup *memcg; =20 + rcu_read_lock(); + memcg =3D page_memcg(page); if (memcg) count_memcg_events(memcg, idx, 1); + rcu_read_unlock(); } =20 static inline void count_memcg_event_mm(struct mm_struct *mm, @@ -1129,6 +1162,16 @@ static inline struct mem_cgroup *page_memcg(struct p= age *page) return NULL; } =20 +static inline struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *f= olio) +{ + return NULL; +} + +static inline struct mem_cgroup *get_mem_cgroup_from_page(struct page *pag= e) +{ + return NULL; +} + static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio) { WARN_ON_ONCE(!rcu_read_lock_held()); diff --git a/include/trace/events/writeback.h b/include/trace/events/writeb= ack.h index 86b2a82da546..cdb822339f13 100644 --- a/include/trace/events/writeback.h +++ b/include/trace/events/writeback.h @@ -258,6 +258,11 @@ TRACE_EVENT(track_foreign_dirty, __entry->ino =3D inode ? inode->i_ino : 0; __entry->memcg_id =3D wb->memcg_css->id; __entry->cgroup_ino =3D __trace_wb_assign_cgroup(wb); + /* + * TP_fast_assign() is under preemption disabled which can + * serve as an RCU read-side critical section so that the + * memcg returned by folio_memcg() cannot be freed. + */ __entry->page_cgroup_ino =3D cgroup_ino(folio_memcg(folio)->css.cgroup); ), =20 diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b38a77f6696f..dcaf6cf5dc74 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -371,7 +371,7 @@ EXPORT_SYMBOL(memcg_kmem_enabled_key); #endif =20 /** - * mem_cgroup_css_from_page - css of the memcg associated with a page + * get_mem_cgroup_css_from_page - get css of the memcg associated with a p= age * @page: page of interest * * If memcg is bound to the default hierarchy, css of the memcg associated @@ -381,13 +381,15 @@ EXPORT_SYMBOL(memcg_kmem_enabled_key); * If memcg is bound to a traditional hierarchy, the css of root_mem_cgroup * is returned. */ -struct cgroup_subsys_state *mem_cgroup_css_from_page(struct page *page) +struct cgroup_subsys_state *get_mem_cgroup_css_from_page(struct page *page) { struct mem_cgroup *memcg; =20 - memcg =3D page_memcg(page); + if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) + return &root_mem_cgroup->css; =20 - if (!memcg || !cgroup_subsys_on_dfl(memory_cgrp_subsys)) + memcg =3D get_mem_cgroup_from_page(page); + if (!memcg) memcg =3D root_mem_cgroup; =20 return &memcg->css; @@ -770,13 +772,13 @@ void __mod_lruvec_state(struct lruvec *lruvec, enum n= ode_stat_item idx, void __mod_lruvec_page_state(struct page *page, enum node_stat_item idx, int val) { - struct page *head =3D compound_head(page); /* rmap on tail pages */ + struct folio *folio =3D page_folio(page); /* rmap on tail pages */ struct mem_cgroup *memcg; pg_data_t *pgdat =3D page_pgdat(page); struct lruvec *lruvec; =20 rcu_read_lock(); - memcg =3D page_memcg(head); + memcg =3D folio_memcg(folio); /* Untracked pages have no memcg, no lruvec. Update only the node */ if (!memcg) { rcu_read_unlock(); @@ -2058,7 +2060,9 @@ void folio_memcg_lock(struct folio *folio) * The RCU lock is held throughout the transaction. The fast * path can get away without acquiring the memcg->move_lock * because page moving starts with an RCU grace period. - */ + * + * The RCU lock also protects the memcg from being freed. + */ rcu_read_lock(); =20 if (mem_cgroup_disabled()) @@ -3296,7 +3300,7 @@ void obj_cgroup_uncharge(struct obj_cgroup *objcg, si= ze_t size) void split_page_memcg(struct page *head, unsigned int nr) { struct folio *folio =3D page_folio(head); - struct mem_cgroup *memcg =3D folio_memcg(folio); + struct mem_cgroup *memcg =3D get_mem_cgroup_from_folio(folio); int i; =20 if (mem_cgroup_disabled() || !memcg) @@ -3309,6 +3313,8 @@ void split_page_memcg(struct page *head, unsigned int= nr) obj_cgroup_get_many(__folio_objcg(folio), nr - 1); else css_get_many(&memcg->css, nr - 1); + + css_put(&memcg->css); } =20 #ifdef CONFIG_MEMCG_SWAP @@ -4511,7 +4517,7 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, un= signed long *pfilepages, void mem_cgroup_track_foreign_dirty_slowpath(struct folio *folio, struct bdi_writeback *wb) { - struct mem_cgroup *memcg =3D folio_memcg(folio); + struct mem_cgroup *memcg =3D get_mem_cgroup_from_folio(folio); struct memcg_cgwb_frn *frn; u64 now =3D get_jiffies_64(); u64 oldest_at =3D now; @@ -4558,6 +4564,7 @@ void mem_cgroup_track_foreign_dirty_slowpath(struct f= olio *folio, frn->memcg_id =3D wb->memcg_css->id; frn->at =3D now; } + css_put(&memcg->css); } =20 /* issue foreign writeback flushes for recorded foreign dirtying events */ @@ -6092,6 +6099,14 @@ static void mem_cgroup_move_charge(void) atomic_dec(&mc.from->moving_account); } =20 +/* + * The cgroup migration and memory cgroup offlining are serialized by + * @cgroup_mutex. If we reach here, it means that the LRU pages cannot + * be reparented to its parent memory cgroup. So during the whole process + * of mem_cgroup_move_task(), page_memcg(page) is stable. So we do not + * need to worry about the memcg (returned from page_memcg()) being + * released even if we do not hold an rcu read lock. + */ static void mem_cgroup_move_task(void) { if (mc.to) { @@ -6895,7 +6910,7 @@ void mem_cgroup_migrate(struct folio *old, struct fol= io *new) if (folio_memcg(new)) return; =20 - memcg =3D folio_memcg(old); + memcg =3D get_mem_cgroup_from_folio(old); VM_WARN_ON_ONCE_FOLIO(!memcg, old); if (!memcg) return; @@ -6914,6 +6929,8 @@ void mem_cgroup_migrate(struct folio *old, struct fol= io *new) mem_cgroup_charge_statistics(memcg, nr_pages); memcg_check_events(memcg, folio_nid(new)); local_irq_restore(flags); + + css_put(&memcg->css); } =20 DEFINE_STATIC_KEY_FALSE(memcg_sockets_enabled_key); @@ -7100,6 +7117,10 @@ void mem_cgroup_swapout(struct folio *folio, swp_ent= ry_t entry) if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) return; =20 + /* + * Interrupts should be disabled by the caller (see the comments below), + * which can serve as RCU read-side critical sections. + */ memcg =3D folio_memcg(folio); =20 VM_WARN_ON_ONCE_FOLIO(!memcg, folio); @@ -7165,15 +7186,16 @@ int __mem_cgroup_try_charge_swap(struct page *page,= swp_entry_t entry) if (!cgroup_subsys_on_dfl(memory_cgrp_subsys)) return 0; =20 + rcu_read_lock(); memcg =3D page_memcg(page); =20 VM_WARN_ON_ONCE_PAGE(!memcg, page); if (!memcg) - return 0; + goto out; =20 if (!entry.val) { memcg_memory_event(memcg, MEMCG_SWAP_FAIL); - return 0; + goto out; } =20 memcg =3D mem_cgroup_id_get_online(memcg); @@ -7183,6 +7205,7 @@ int __mem_cgroup_try_charge_swap(struct page *page, s= wp_entry_t entry) memcg_memory_event(memcg, MEMCG_SWAP_MAX); memcg_memory_event(memcg, MEMCG_SWAP_FAIL); mem_cgroup_id_put(memcg); + rcu_read_unlock(); return -ENOMEM; } =20 @@ -7192,6 +7215,8 @@ int __mem_cgroup_try_charge_swap(struct page *page, s= wp_entry_t entry) oldid =3D swap_cgroup_record(entry, mem_cgroup_id(memcg), nr_pages); VM_BUG_ON_PAGE(oldid, page); mod_memcg_state(memcg, MEMCG_SWAP, nr_pages); +out: + rcu_read_unlock(); =20 return 0; } @@ -7246,17 +7271,22 @@ bool mem_cgroup_swap_full(struct page *page) if (cgroup_memory_noswap || !cgroup_subsys_on_dfl(memory_cgrp_subsys)) return false; =20 + rcu_read_lock(); memcg =3D page_memcg(page); if (!memcg) - return false; + goto out; =20 for (; memcg !=3D root_mem_cgroup; memcg =3D parent_mem_cgroup(memcg)) { unsigned long usage =3D page_counter_read(&memcg->swap); =20 if (usage * 2 >=3D READ_ONCE(memcg->swap.high) || - usage * 2 >=3D READ_ONCE(memcg->swap.max)) + usage * 2 >=3D READ_ONCE(memcg->swap.max)) { + rcu_read_unlock(); return true; + } } +out: + rcu_read_unlock(); =20 return false; } diff --git a/mm/migrate.c b/mm/migrate.c index 6c31ee1e1c9b..59e97a8a64a0 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -430,6 +430,10 @@ int folio_migrate_mapping(struct address_space *mappin= g, struct lruvec *old_lruvec, *new_lruvec; struct mem_cgroup *memcg; =20 + /* + * Irq is disabled, which can serve as RCU read-side critical + * sections. + */ memcg =3D folio_memcg(folio); old_lruvec =3D mem_cgroup_lruvec(memcg, oldzone->zone_pgdat); new_lruvec =3D mem_cgroup_lruvec(memcg, newzone->zone_pgdat); diff --git a/mm/page_io.c b/mm/page_io.c index 89fbf3cae30f..a0d9cd68e87a 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -221,13 +221,14 @@ static void bio_associate_blkg_from_page(struct bio *= bio, struct page *page) struct cgroup_subsys_state *css; struct mem_cgroup *memcg; =20 + rcu_read_lock(); memcg =3D page_memcg(page); if (!memcg) - return; + goto out; =20 - rcu_read_lock(); css =3D cgroup_e_css(memcg->css.cgroup, &io_cgrp_subsys); bio_associate_blkg_from_css(bio, css); +out: rcu_read_unlock(); } #else --=20 2.11.0 From nobody Sun May 5 20:57:33 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88C28C433FE for ; Tue, 24 May 2022 06:28:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234998AbiEXG2a (ORCPT ); Tue, 24 May 2022 02:28:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48740 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235014AbiEXG2S (ORCPT ); Tue, 24 May 2022 02:28:18 -0400 Received: from mail-pj1-x1032.google.com (mail-pj1-x1032.google.com [IPv6:2607:f8b0:4864:20::1032]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 174847378E for ; Mon, 23 May 2022 23:28:16 -0700 (PDT) Received: by mail-pj1-x1032.google.com with SMTP id fw21-20020a17090b129500b001df9f62edd6so1417591pjb.0 for ; Mon, 23 May 2022 23:28:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NIoa7/YAUntswrCKfqoiHIwQ2FiZAUKsFw3sXpH2ToA=; b=nmPPdm0n11BOQ5k1iXVhMTgOfUnT5doQW6FEzqyptlVcbRMwfoXIbTPOIt+hcvURYm O/Nzh2C3SdKHRPAyGRnaF7nYOlZpg0prTt9wOpP7vQYDWfArjY7T7ApBJ/JAjAgVvECg 4jls9zpAlcLrVEOlK+ywZNAG7nqEKA2b9BaLbgp81TVUePK7vD9suhJWDXnJXra5Ju0l TyOGR8EJB98F0nb8TEaC6QAaRokmsR0nOX6rWCne6FPTiuZMak68sJLTis5vjtjMKXdP dEE2Vgt89hFC9V3zjfIoJOLpgb3NNQ3m21yXjs+4S62wvOivx9BWVqvSTt1KsMEMAqko jK9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NIoa7/YAUntswrCKfqoiHIwQ2FiZAUKsFw3sXpH2ToA=; b=6MfBBX1KDczVR5J9RvuYJBjmkFf7CvRFwMycTEOVU/qRohOQg4lo9d0Z0B6N/XvgTL Ev1BmJktj68xi+sem+2Wg8kfmSyAlOEcPC1FOgfWixsZxkQTJK9wWGNejyhpgbBtDQik 3bGpM5ifMpcHuvqXGLutnP4kKCbV7TZmSXUhEuJPEQeGQvZTRju7qV9SLbp4Iv+pJJ5Y Na7Q487Q7iCZH6nv2vT/CgrpeStf9/QBzsKOkza0mVkWBZrXeVrOw5uvHHYWleVEIUZb TOhuzGGz52vWxEAXrjKbIRvJSo9h1ZvHXcetLbxx/CNOrD7XbxNO6n4t1UmbS614Gxzy O+Kg== X-Gm-Message-State: AOAM533QmBI6BzOIY2/ohnvs0HhiUyOc7V9GF9OQ7Ya9t1xvDdMn5NAB ph5GqcdeJz+HqxvQ9H+pkkCR+Q== X-Google-Smtp-Source: ABdhPJz3zlcfAkmEkbweh3Xk2zbyA46Z3LjTGnxV11LirLFNvvsfkscs2z1dzbsre43De32HasZ8nw== X-Received: by 2002:a17:90a:6342:b0:1df:763d:4eca with SMTP id v2-20020a17090a634200b001df763d4ecamr2938988pjs.85.1653373695699; Mon, 23 May 2022 23:28:15 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([2408:8207:18da:2310:f940:af17:c2f5:8656]) by smtp.gmail.com with ESMTPSA id h5-20020a170902f54500b0016168e90f2dsm6254455plf.219.2022.05.23.23.28.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 May 2022 23:28:15 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, duanxiongchun@bytedance.com, longman@redhat.com, Muchun Song Subject: [PATCH v4 08/11] mm: memcontrol: introduce memcg_reparent_ops Date: Tue, 24 May 2022 14:05:48 +0800 Message-Id: <20220524060551.80037-9-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220524060551.80037-1-songmuchun@bytedance.com> References: <20220524060551.80037-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" In the previous patch, we know how to make the lruvec lock safe when LRU pages are reparented. We should do something like following. memcg_reparent_objcgs(memcg) 1) lock // lruvec belongs to memcg and lruvec_parent belongs to parent memc= g. spin_lock(&lruvec->lru_lock); spin_lock(&lruvec_parent->lru_lock); 2) relocate from current memcg to its parent // Move all the pages from the lruvec list to the parent lruvec lis= t. 3) unlock spin_unlock(&lruvec_parent->lru_lock); spin_unlock(&lruvec->lru_lock); Apart from the page lruvec lock, the deferred split queue lock (THP only) also needs to do something similar. So we extract the necessary three steps in the memcg_reparent_objcgs(). memcg_reparent_objcgs(memcg) 1) lock memcg_reparent_ops->lock(memcg, parent); 2) relocate memcg_reparent_ops->relocate(memcg, reparent); 3) unlock memcg_reparent_ops->unlock(memcg, reparent); Now there are two different locks (e.g. lruvec lock and deferred split queue lock) need to use this infrastructure. In the next patch, we will use those APIs to make those locks safe when the LRU pages reparented. Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 20 +++++++++++++++ mm/memcontrol.c | 62 ++++++++++++++++++++++++++++++++++++------= ---- 2 files changed, 69 insertions(+), 13 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 3a0e2592434e..e806e743a1fc 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -347,6 +347,26 @@ struct mem_cgroup { struct mem_cgroup_per_node *nodeinfo[]; }; =20 +struct memcg_reparent_ops { + /* + * Note that interrupt is disabled before calling those callbacks, + * so the interrupt should remain disabled when leaving those callbacks. + */ + void (*lock)(struct mem_cgroup *src, struct mem_cgroup *dst); + void (*relocate)(struct mem_cgroup *src, struct mem_cgroup *dst); + void (*unlock)(struct mem_cgroup *src, struct mem_cgroup *dst); +}; + +#define DEFINE_MEMCG_REPARENT_OPS(name) \ + const struct memcg_reparent_ops memcg_##name##_reparent_ops =3D { \ + .lock =3D name##_reparent_lock, \ + .relocate =3D name##_reparent_relocate, \ + .unlock =3D name##_reparent_unlock, \ + } + +#define DECLARE_MEMCG_REPARENT_OPS(name) \ + extern const struct memcg_reparent_ops memcg_##name##_reparent_ops + /* * size of first charge trial. "32" comes from vmscan.c's magic value. * TODO: maybe necessary to use big numbers in big irons. diff --git a/mm/memcontrol.c b/mm/memcontrol.c index dcaf6cf5dc74..7d62764c6380 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -337,24 +337,60 @@ static struct obj_cgroup *obj_cgroup_alloc(void) return objcg; } =20 -static void memcg_reparent_objcgs(struct mem_cgroup *memcg) +static void objcg_reparent_lock(struct mem_cgroup *src, struct mem_cgroup = *dst) +{ + spin_lock(&objcg_lock); +} + +static void objcg_reparent_relocate(struct mem_cgroup *src, struct mem_cgr= oup *dst) { struct obj_cgroup *objcg, *iter; - struct mem_cgroup *parent =3D parent_mem_cgroup(memcg); =20 - objcg =3D rcu_replace_pointer(memcg->objcg, NULL, true); + objcg =3D rcu_replace_pointer(src->objcg, NULL, true); + /* 1) Ready to reparent active objcg. */ + list_add(&objcg->list, &src->objcg_list); + /* 2) Reparent active objcg and already reparented objcgs to dst. */ + list_for_each_entry(iter, &src->objcg_list, list) + WRITE_ONCE(iter->memcg, dst); + /* 3) Move already reparented objcgs to the dst's list */ + list_splice(&src->objcg_list, &dst->objcg_list); +} + +static void objcg_reparent_unlock(struct mem_cgroup *src, struct mem_cgrou= p *dst) +{ + spin_unlock(&objcg_lock); +} =20 - spin_lock_irq(&objcg_lock); +static DEFINE_MEMCG_REPARENT_OPS(objcg); =20 - /* 1) Ready to reparent active objcg. */ - list_add(&objcg->list, &memcg->objcg_list); - /* 2) Reparent active objcg and already reparented objcgs to parent. */ - list_for_each_entry(iter, &memcg->objcg_list, list) - WRITE_ONCE(iter->memcg, parent); - /* 3) Move already reparented objcgs to the parent's list */ - list_splice(&memcg->objcg_list, &parent->objcg_list); - - spin_unlock_irq(&objcg_lock); +static const struct memcg_reparent_ops *memcg_reparent_ops[] =3D { + &memcg_objcg_reparent_ops, +}; + +#define DEFINE_MEMCG_REPARENT_FUNC(phase) \ + static void memcg_reparent_##phase(struct mem_cgroup *src, \ + struct mem_cgroup *dst) \ + { \ + int i; \ + \ + for (i =3D 0; i < ARRAY_SIZE(memcg_reparent_ops); i++) \ + memcg_reparent_ops[i]->phase(src, dst); \ + } + +DEFINE_MEMCG_REPARENT_FUNC(lock) +DEFINE_MEMCG_REPARENT_FUNC(relocate) +DEFINE_MEMCG_REPARENT_FUNC(unlock) + +static void memcg_reparent_objcgs(struct mem_cgroup *src) +{ + struct mem_cgroup *dst =3D parent_mem_cgroup(src); + struct obj_cgroup *objcg =3D rcu_dereference_protected(src->objcg, true); + + local_irq_disable(); + memcg_reparent_lock(src, dst); + memcg_reparent_relocate(src, dst); + memcg_reparent_unlock(src, dst); + local_irq_enable(); =20 percpu_ref_kill(&objcg->refcnt); } --=20 2.11.0 From nobody Sun May 5 20:57:33 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 127F5C433EF for ; Tue, 24 May 2022 06:28:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234918AbiEXG2k (ORCPT ); Tue, 24 May 2022 02:28:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234392AbiEXG2Y (ORCPT ); Tue, 24 May 2022 02:28:24 -0400 Received: from mail-pj1-x1033.google.com (mail-pj1-x1033.google.com [IPv6:2607:f8b0:4864:20::1033]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E25271DB7 for ; Mon, 23 May 2022 23:28:22 -0700 (PDT) Received: by mail-pj1-x1033.google.com with SMTP id l14so15960204pjk.2 for ; Mon, 23 May 2022 23:28:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XOWyBLvW8iT3STG868ovfrJ7kSsobdYF72qyisoggTc=; b=mYoB/etgWqgokImebX4ZLu6iNfVJKXC6JnsHDMPYOr+76ypKKPjwEDyO7611wLd5vC RbE5Mw2mHkLm6l4oU9dkhlWQ3+1Rl1Lqi+2Mk9BHQbUxp+cF2JsnC34+b7AMqFuz2ISu 3955Uk643wbjcwvXkPyUoaYg3QHkUyQozK1N2ghELEMAwkGCEr/inNbCG901oHfD7IUC D4uAexotHz3gCuiK6xQW6CwI75KyD9ZXROE2lcaUTjVhnV9ztmFT6gdRlXdpjckN+pbs Zhw6jFbAo2MM1t6ltCQtH+HvrZzSt77FOdGnQV8HLDkmaGkb6y9LpSj4NWQy/f6vWunF 95vA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XOWyBLvW8iT3STG868ovfrJ7kSsobdYF72qyisoggTc=; b=zaM3GO+UgmsCnITCDh7CXnRjO17a8hC/h0WRsBH3wdJGroZADRdRQRfLqQoQ6cQSCP zd7cj7ce2BuUw/GszPeX2GYU5Ex5qUmYHVaDbu0tnlcOaxg7z3ZgqofzD4fPiHBQj3b6 Gp+tU0c7slKXJ2na5y9a1JMVaKZcJYKB2HLoqo0zzrZHjgwk3WqMTTCCE5clQA7n8giW e4pcTJ+LpmLnIExyBcf9AhdvZCGzyBHhwD3ZCSNXLTl82epFuAE9MghXE0tyrlhQWvTn 3Rmwkkj62UtZV7QsBVB5Q3tlMZr8aiYGrRnp+k1BUjA78jcDU1duH2J7SWFh1WoWhyAg nODQ== X-Gm-Message-State: AOAM530DovXyEjoUt8/G63zOHTulC4H967DuAheuNGH+G+K6K6rPk/A1 sfnQfJrIOBI5mXQvHxFRwtUMVA== X-Google-Smtp-Source: ABdhPJz7JGYmbZ0Ne7dt6jcVNa+7wxQVvm2OfAnw+WmCaM1sejvusIYDGNEoKfGByV5DDwlyDK1KkQ== X-Received: by 2002:a17:90b:390b:b0:1df:b3dc:514a with SMTP id ob11-20020a17090b390b00b001dfb3dc514amr3047993pjb.25.1653373702038; Mon, 23 May 2022 23:28:22 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([2408:8207:18da:2310:f940:af17:c2f5:8656]) by smtp.gmail.com with ESMTPSA id h5-20020a170902f54500b0016168e90f2dsm6254455plf.219.2022.05.23.23.28.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 May 2022 23:28:21 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, duanxiongchun@bytedance.com, longman@redhat.com, Muchun Song Subject: [PATCH v4 09/11] mm: memcontrol: use obj_cgroup APIs to charge the LRU pages Date: Tue, 24 May 2022 14:05:49 +0800 Message-Id: <20220524060551.80037-10-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220524060551.80037-1-songmuchun@bytedance.com> References: <20220524060551.80037-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" We will reuse the obj_cgroup APIs to charge the LRU pages. Finally, page->memcg_data will have 2 different meanings. - For the slab pages, page->memcg_data points to an object cgroups vector. - For the kmem pages (exclude the slab pages) and the LRU pages, page->memcg_data points to an object cgroup. In this patch, we reuse obj_cgroup APIs to charge LRU pages. In the end, The page cache cannot prevent long-living objects from pinning the original memory cgroup in the memory. At the same time we also changed the rules of page and objcg or memcg binding stability. The new rules are as follows. For a page any of the following ensures page and objcg binding stability: - the page lock - LRU isolation - lock_page_memcg() - exclusive reference Based on the stable binding of page and objcg, for a page any of the following ensures page and memcg binding stability: - css_set_lock - cgroup_mutex - the lruvec lock - the split queue lock (only THP page) If the caller only want to ensure that the page counters of memcg are updated correctly, ensure that the binding stability of page and objcg is sufficient. Signed-off-by: Muchun Song Reported-by: kernel test robot Reported-by: kernel test robot --- include/linux/memcontrol.h | 94 ++++++--------- mm/huge_memory.c | 34 ++++++ mm/memcontrol.c | 287 ++++++++++++++++++++++++++++++++---------= ---- 3 files changed, 277 insertions(+), 138 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e806e743a1fc..237ae86f8d8e 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -386,8 +386,6 @@ enum page_memcg_data_flags { =20 #define MEMCG_DATA_FLAGS_MASK (__NR_MEMCG_DATA_FLAGS - 1) =20 -static inline bool folio_memcg_kmem(struct folio *folio); - /* * After the initialization objcg->memcg is always pointing at * a valid memcg, but can be atomically swapped to the parent memcg. @@ -401,43 +399,19 @@ static inline struct mem_cgroup *obj_cgroup_memcg(str= uct obj_cgroup *objcg) } =20 /* - * __folio_memcg - Get the memory cgroup associated with a non-kmem folio - * @folio: Pointer to the folio. - * - * Returns a pointer to the memory cgroup associated with the folio, - * or NULL. This function assumes that the folio is known to have a - * proper memory cgroup pointer. It's not safe to call this function - * against some type of folios, e.g. slab folios or ex-slab folios or - * kmem folios. - */ -static inline struct mem_cgroup *__folio_memcg(struct folio *folio) -{ - unsigned long memcg_data =3D folio->memcg_data; - - VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); - VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJCGS, folio); - VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_KMEM, folio); - - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); -} - -/* - * __folio_objcg - get the object cgroup associated with a kmem folio. + * folio_objcg - get the object cgroup associated with a folio. * @folio: Pointer to the folio. * * Returns a pointer to the object cgroup associated with the folio, * or NULL. This function assumes that the folio is known to have a - * proper object cgroup pointer. It's not safe to call this function - * against some type of folios, e.g. slab folios or ex-slab folios or - * LRU folios. + * proper object cgroup pointer. */ -static inline struct obj_cgroup *__folio_objcg(struct folio *folio) +static inline struct obj_cgroup *folio_objcg(struct folio *folio) { unsigned long memcg_data =3D folio->memcg_data; =20 VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); VM_BUG_ON_FOLIO(memcg_data & MEMCG_DATA_OBJCGS, folio); - VM_BUG_ON_FOLIO(!(memcg_data & MEMCG_DATA_KMEM), folio); =20 return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); } @@ -451,7 +425,7 @@ static inline struct obj_cgroup *__folio_objcg(struct f= olio *folio) * proper memory cgroup pointer. It's not safe to call this function * against some type of folios, e.g. slab folios or ex-slab folios. * - * For a non-kmem folio any of the following ensures folio and memcg bindi= ng + * For a folio any of the following ensures folio and memcg binding * stability: * * - the folio lock @@ -459,14 +433,28 @@ static inline struct obj_cgroup *__folio_objcg(struct= folio *folio) * - lock_page_memcg() * - exclusive reference * - * For a kmem folio a caller should hold an rcu read lock to protect memcg - * associated with a kmem folio from being released. + * Based on the stable binding of folio and objcg, for a folio any of the + * following ensures folio and memcg binding stability: + * + * - css_set_lock + * - cgroup_mutex + * - the lruvec lock + * - the split queue lock (only THP page) + * + * If the caller only want to ensure that the page counters of memcg are + * updated correctly, ensure that the binding stability of folio and objcg + * is sufficient. + * + * A caller should hold an rcu read lock (In addition, regions of code acr= oss + * which interrupts, preemption, or softirqs have been disabled also serve= as + * RCU read-side critical sections) to protect memcg associated with a fol= io + * from being released. */ static inline struct mem_cgroup *folio_memcg(struct folio *folio) { - if (folio_memcg_kmem(folio)) - return obj_cgroup_memcg(__folio_objcg(folio)); - return __folio_memcg(folio); + struct obj_cgroup *objcg =3D folio_objcg(folio); + + return objcg ? obj_cgroup_memcg(objcg) : NULL; } =20 static inline struct mem_cgroup *page_memcg(struct page *page) @@ -484,6 +472,8 @@ static inline struct mem_cgroup *page_memcg(struct page= *page) * folio is known to have a proper memory cgroup pointer. It's not safe * to call this function against some type of pages, e.g. slab pages or * ex-slab pages. + * + * The page and objcg or memcg binding rules can refer to folio_memcg(). */ static inline struct mem_cgroup *get_mem_cgroup_from_folio(struct folio *f= olio) { @@ -514,22 +504,20 @@ static inline struct mem_cgroup *get_mem_cgroup_from_= page(struct page *page) * * Return: A pointer to the memory cgroup associated with the folio, * or NULL. + * + * The folio and objcg or memcg binding rules can refer to folio_memcg(). */ static inline struct mem_cgroup *folio_memcg_rcu(struct folio *folio) { unsigned long memcg_data =3D READ_ONCE(folio->memcg_data); + struct obj_cgroup *objcg; =20 VM_BUG_ON_FOLIO(folio_test_slab(folio), folio); WARN_ON_ONCE(!rcu_read_lock_held()); =20 - if (memcg_data & MEMCG_DATA_KMEM) { - struct obj_cgroup *objcg; - - objcg =3D (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); - return obj_cgroup_memcg(objcg); - } + objcg =3D (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); =20 - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return objcg ? obj_cgroup_memcg(objcg) : NULL; } =20 /* @@ -542,16 +530,10 @@ static inline struct mem_cgroup *folio_memcg_rcu(stru= ct folio *folio) * has an associated memory cgroup pointer or an object cgroups vector or * an object cgroup. * - * For a non-kmem page any of the following ensures page and memcg binding - * stability: + * The page and objcg or memcg binding rules can refer to page_memcg(). * - * - the page lock - * - LRU isolation - * - lock_page_memcg() - * - exclusive reference - * - * For a kmem page a caller should hold an rcu read lock to protect memcg - * associated with a kmem page from being released. + * A caller should hold an rcu read lock to protect memcg associated with a + * page from being released. */ static inline struct mem_cgroup *page_memcg_check(struct page *page) { @@ -560,18 +542,14 @@ static inline struct mem_cgroup *page_memcg_check(str= uct page *page) * for slab pages, READ_ONCE() should be used here. */ unsigned long memcg_data =3D READ_ONCE(page->memcg_data); + struct obj_cgroup *objcg; =20 if (memcg_data & MEMCG_DATA_OBJCGS) return NULL; =20 - if (memcg_data & MEMCG_DATA_KMEM) { - struct obj_cgroup *objcg; - - objcg =3D (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); - return obj_cgroup_memcg(objcg); - } + objcg =3D (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); =20 - return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return objcg ? obj_cgroup_memcg(objcg) : NULL; } =20 static inline struct mem_cgroup *get_mem_cgroup_from_objcg(struct obj_cgro= up *objcg) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index cc596034c487..ec98f346cae6 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -503,6 +503,8 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struc= t *vma) } =20 #ifdef CONFIG_MEMCG +static struct shrinker deferred_split_shrinker; + static inline struct mem_cgroup *folio_split_queue_memcg(struct folio *fol= io, struct deferred_split *queue) { @@ -519,6 +521,38 @@ static inline struct deferred_split *folio_memcg_split= _queue(struct folio *folio =20 return memcg ? &memcg->deferred_split_queue : NULL; } + +static void thp_reparent_lock(struct mem_cgroup *src, struct mem_cgroup *d= st) +{ + spin_lock(&src->deferred_split_queue.split_queue_lock); + spin_lock(&dst->deferred_split_queue.split_queue_lock); +} + +static void thp_reparent_relocate(struct mem_cgroup *src, struct mem_cgrou= p *dst) +{ + int nid; + struct deferred_split *src_queue, *dst_queue; + + src_queue =3D &src->deferred_split_queue; + dst_queue =3D &dst->deferred_split_queue; + + if (!src_queue->split_queue_len) + return; + + list_splice_tail_init(&src_queue->split_queue, &dst_queue->split_queue); + dst_queue->split_queue_len +=3D src_queue->split_queue_len; + src_queue->split_queue_len =3D 0; + + for_each_node(nid) + set_shrinker_bit(dst, nid, deferred_split_shrinker.id); +} + +static void thp_reparent_unlock(struct mem_cgroup *src, struct mem_cgroup = *dst) +{ + spin_unlock(&dst->deferred_split_queue.split_queue_lock); + spin_unlock(&src->deferred_split_queue.split_queue_lock); +} +DEFINE_MEMCG_REPARENT_OPS(thp); #else static inline struct mem_cgroup *folio_split_queue_memcg(struct folio *fol= io, struct deferred_split *queue) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7d62764c6380..1a35f7fde3ed 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -76,6 +76,7 @@ struct cgroup_subsys memory_cgrp_subsys __read_mostly; EXPORT_SYMBOL(memory_cgrp_subsys); =20 struct mem_cgroup *root_mem_cgroup __read_mostly; +static struct obj_cgroup *root_obj_cgroup __read_mostly; =20 /* Active memory cgroup to use from an interrupt context */ DEFINE_PER_CPU(struct mem_cgroup *, int_active_memcg); @@ -256,6 +257,11 @@ struct mem_cgroup *vmpressure_to_memcg(struct vmpressu= re *vmpr) =20 static DEFINE_SPINLOCK(objcg_lock); =20 +static inline bool obj_cgroup_is_root(struct obj_cgroup *objcg) +{ + return objcg =3D=3D root_obj_cgroup; +} + #ifdef CONFIG_MEMCG_KMEM bool mem_cgroup_kmem_disabled(void) { @@ -363,8 +369,75 @@ static void objcg_reparent_unlock(struct mem_cgroup *s= rc, struct mem_cgroup *dst =20 static DEFINE_MEMCG_REPARENT_OPS(objcg); =20 +static void lruvec_reparent_lock(struct mem_cgroup *src, struct mem_cgroup= *dst) +{ + int i; + + for_each_node(i) { + spin_lock(&mem_cgroup_lruvec(src, NODE_DATA(i))->lru_lock); + spin_lock(&mem_cgroup_lruvec(dst, NODE_DATA(i))->lru_lock); + } +} + +static void lruvec_reparent_lru(struct lruvec *src, struct lruvec *dst, + enum lru_list lru) +{ + int zid; + struct mem_cgroup_per_node *mz_src, *mz_dst; + + mz_src =3D container_of(src, struct mem_cgroup_per_node, lruvec); + mz_dst =3D container_of(dst, struct mem_cgroup_per_node, lruvec); + + if (lru !=3D LRU_UNEVICTABLE) + list_splice_tail_init(&src->lists[lru], &dst->lists[lru]); + + for (zid =3D 0; zid < MAX_NR_ZONES; zid++) { + mz_dst->lru_zone_size[zid][lru] +=3D mz_src->lru_zone_size[zid][lru]; + mz_src->lru_zone_size[zid][lru] =3D 0; + } +} + +static void lruvec_reparent_relocate(struct mem_cgroup *src, struct mem_cg= roup *dst) +{ + int i; + + for_each_node(i) { + enum lru_list lru; + struct lruvec *src_lruvec, *dst_lruvec; + + src_lruvec =3D mem_cgroup_lruvec(src, NODE_DATA(i)); + dst_lruvec =3D mem_cgroup_lruvec(dst, NODE_DATA(i)); + + dst_lruvec->anon_cost +=3D src_lruvec->anon_cost; + dst_lruvec->file_cost +=3D src_lruvec->file_cost; + + for_each_lru(lru) + lruvec_reparent_lru(src_lruvec, dst_lruvec, lru); + } +} + +static void lruvec_reparent_unlock(struct mem_cgroup *src, struct mem_cgro= up *dst) +{ + int i; + + for_each_node(i) { + spin_unlock(&mem_cgroup_lruvec(dst, NODE_DATA(i))->lru_lock); + spin_unlock(&mem_cgroup_lruvec(src, NODE_DATA(i))->lru_lock); + } +} + +static DEFINE_MEMCG_REPARENT_OPS(lruvec); + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +DECLARE_MEMCG_REPARENT_OPS(thp); +#endif + static const struct memcg_reparent_ops *memcg_reparent_ops[] =3D { &memcg_objcg_reparent_ops, + &memcg_lruvec_reparent_ops, +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + &memcg_thp_reparent_ops, +#endif }; =20 #define DEFINE_MEMCG_REPARENT_FUNC(phase) \ @@ -2827,18 +2900,18 @@ static inline void cancel_charge(struct mem_cgroup = *memcg, unsigned int nr_pages page_counter_uncharge(&memcg->memsw, nr_pages); } =20 -static void commit_charge(struct folio *folio, struct mem_cgroup *memcg) +static void commit_charge(struct folio *folio, struct obj_cgroup *objcg) { - VM_BUG_ON_FOLIO(folio_memcg(folio), folio); + VM_BUG_ON_FOLIO(folio_objcg(folio), folio); /* - * Any of the following ensures page's memcg stability: + * Any of the following ensures page's objcg stability: * * - the page lock * - LRU isolation * - lock_page_memcg() * - exclusive reference */ - folio->memcg_data =3D (unsigned long)memcg; + folio->memcg_data =3D (unsigned long)objcg; } =20 #ifdef CONFIG_MEMCG_KMEM @@ -2955,6 +3028,21 @@ struct mem_cgroup *mem_cgroup_from_obj(void *p) return page_memcg_check(folio_page(folio, 0)); } =20 +static struct obj_cgroup *__get_obj_cgroup_from_memcg(struct mem_cgroup *m= emcg) +{ + struct obj_cgroup *objcg =3D NULL; + + rcu_read_lock(); + for (; memcg; memcg =3D parent_mem_cgroup(memcg)) { + objcg =3D rcu_dereference(memcg->objcg); + if (objcg && obj_cgroup_tryget(objcg)) + break; + } + rcu_read_unlock(); + + return objcg; +} + __always_inline struct obj_cgroup *get_obj_cgroup_from_current(void) { struct obj_cgroup *objcg =3D NULL; @@ -2969,12 +3057,15 @@ __always_inline struct obj_cgroup *get_obj_cgroup_f= rom_current(void) else memcg =3D mem_cgroup_from_task(current); =20 - for (; memcg !=3D root_mem_cgroup; memcg =3D parent_mem_cgroup(memcg)) { - objcg =3D rcu_dereference(memcg->objcg); - if (objcg && obj_cgroup_tryget(objcg)) - break; + if (mem_cgroup_is_root(memcg)) + goto out; + + objcg =3D __get_obj_cgroup_from_memcg(memcg); + if (obj_cgroup_is_root(objcg)) { + obj_cgroup_put(objcg); objcg =3D NULL; } +out: rcu_read_unlock(); =20 return objcg; @@ -3071,13 +3162,13 @@ int __memcg_kmem_charge_page(struct page *page, gfp= _t gfp, int order) void __memcg_kmem_uncharge_page(struct page *page, int order) { struct folio *folio =3D page_folio(page); - struct obj_cgroup *objcg; + struct obj_cgroup *objcg =3D folio_objcg(folio); unsigned int nr_pages =3D 1 << order; =20 - if (!folio_memcg_kmem(folio)) + if (!objcg) return; =20 - objcg =3D __folio_objcg(folio); + VM_BUG_ON_FOLIO(!folio_memcg_kmem(folio), folio); obj_cgroup_uncharge_pages(objcg, nr_pages); folio->memcg_data =3D 0; obj_cgroup_put(objcg); @@ -3331,26 +3422,21 @@ void obj_cgroup_uncharge(struct obj_cgroup *objcg, = size_t size) #endif /* CONFIG_MEMCG_KMEM */ =20 /* - * Because page_memcg(head) is not set on tails, set it now. + * Because page_objcg(head) is not set on tails, set it now. */ void split_page_memcg(struct page *head, unsigned int nr) { struct folio *folio =3D page_folio(head); - struct mem_cgroup *memcg =3D get_mem_cgroup_from_folio(folio); + struct obj_cgroup *objcg =3D folio_objcg(folio); int i; =20 - if (mem_cgroup_disabled() || !memcg) + if (mem_cgroup_disabled() || !objcg) return; =20 for (i =3D 1; i < nr; i++) folio_page(folio, i)->memcg_data =3D folio->memcg_data; =20 - if (folio_memcg_kmem(folio)) - obj_cgroup_get_many(__folio_objcg(folio), nr - 1); - else - css_get_many(&memcg->css, nr - 1); - - css_put(&memcg->css); + obj_cgroup_get_many(objcg, nr - 1); } =20 #ifdef CONFIG_MEMCG_SWAP @@ -5253,6 +5339,9 @@ static int mem_cgroup_css_online(struct cgroup_subsys= _state *css) objcg->memcg =3D memcg; rcu_assign_pointer(memcg->objcg, objcg); =20 + if (unlikely(mem_cgroup_is_root(memcg))) + root_obj_cgroup =3D objcg; + /* Online state pins memcg ID, memcg ID pins CSS */ refcount_set(&memcg->id.ref, 1); css_get(css); @@ -5657,10 +5746,12 @@ static int mem_cgroup_move_account(struct page *pag= e, */ smp_mb(); =20 - css_get(&to->css); - css_put(&from->css); + rcu_read_lock(); + obj_cgroup_get(rcu_dereference(to->objcg)); + obj_cgroup_put(rcu_dereference(from->objcg)); + rcu_read_unlock(); =20 - folio->memcg_data =3D (unsigned long)to; + folio->memcg_data =3D (unsigned long)rcu_access_pointer(to->objcg); =20 __folio_memcg_unlock(from); =20 @@ -6133,6 +6224,42 @@ static void mem_cgroup_move_charge(void) =20 mmap_read_unlock(mc.mm); atomic_dec(&mc.from->moving_account); + + /* + * Moving its pages to another memcg is finished. Wait for already + * started RCU-only updates to finish to make sure that the caller + * of lock_page_memcg() can unlock the correct move_lock. The + * possible bad scenario would like: + * + * CPU0: CPU1: + * mem_cgroup_move_charge() + * walk_page_range() + * + * lock_page_memcg(page) + * memcg =3D folio_memcg() + * spin_lock_irqsave(&memcg->move_lock) + * memcg->move_lock_task =3D current + * + * atomic_dec(&mc.from->moving_account) + * + * mem_cgroup_css_offline() + * memcg_offline_kmem() + * memcg_reparent_objcgs() <=3D=3D reparented + * + * unlock_page_memcg(page) + * memcg =3D folio_memcg() <=3D=3D memcg has been changed + * if (memcg->move_lock_task =3D=3D current) <=3D=3D false + * spin_unlock_irqrestore(&memcg->move_lock) + * + * Once mem_cgroup_move_charge() returns (it means that the cgroup_mutex + * would be released soon), the page can be reparented to its parent + * memcg. When the unlock_page_memcg() is called for the page, we will + * miss unlock the move_lock. So using synchronize_rcu to wait for + * already started RCU-only updates to finish before this function + * returns (mem_cgroup_move_charge() and mem_cgroup_css_offline() are + * serialized by cgroup_mutex). + */ + synchronize_rcu(); } =20 /* @@ -6692,21 +6819,26 @@ void mem_cgroup_calculate_protection(struct mem_cgr= oup *root, static int charge_memcg(struct folio *folio, struct mem_cgroup *memcg, gfp_t gfp) { + struct obj_cgroup *objcg; long nr_pages =3D folio_nr_pages(folio); - int ret; + int ret =3D 0; =20 - ret =3D try_charge(memcg, gfp, nr_pages); + objcg =3D __get_obj_cgroup_from_memcg(memcg); + /* Do not account at the root objcg level. */ + if (!obj_cgroup_is_root(objcg)) + ret =3D try_charge(memcg, gfp, nr_pages); if (ret) goto out; =20 - css_get(&memcg->css); - commit_charge(folio, memcg); + obj_cgroup_get(objcg); + commit_charge(folio, objcg); =20 local_irq_disable(); mem_cgroup_charge_statistics(memcg, nr_pages); memcg_check_events(memcg, folio_nid(folio)); local_irq_enable(); out: + obj_cgroup_put(objcg); return ret; } =20 @@ -6792,7 +6924,7 @@ void mem_cgroup_swapin_uncharge_swap(swp_entry_t entr= y) } =20 struct uncharge_gather { - struct mem_cgroup *memcg; + struct obj_cgroup *objcg; unsigned long nr_memory; unsigned long pgpgout; unsigned long nr_kmem; @@ -6807,63 +6939,56 @@ static inline void uncharge_gather_clear(struct unc= harge_gather *ug) static void uncharge_batch(const struct uncharge_gather *ug) { unsigned long flags; + struct mem_cgroup *memcg; =20 + rcu_read_lock(); + memcg =3D obj_cgroup_memcg(ug->objcg); if (ug->nr_memory) { - page_counter_uncharge(&ug->memcg->memory, ug->nr_memory); + page_counter_uncharge(&memcg->memory, ug->nr_memory); if (do_memsw_account()) - page_counter_uncharge(&ug->memcg->memsw, ug->nr_memory); + page_counter_uncharge(&memcg->memsw, ug->nr_memory); if (ug->nr_kmem) - memcg_account_kmem(ug->memcg, -ug->nr_kmem); - memcg_oom_recover(ug->memcg); + memcg_account_kmem(memcg, -ug->nr_kmem); + memcg_oom_recover(memcg); } =20 local_irq_save(flags); - __count_memcg_events(ug->memcg, PGPGOUT, ug->pgpgout); - __this_cpu_add(ug->memcg->vmstats_percpu->nr_page_events, ug->nr_memory); - memcg_check_events(ug->memcg, ug->nid); + __count_memcg_events(memcg, PGPGOUT, ug->pgpgout); + __this_cpu_add(memcg->vmstats_percpu->nr_page_events, ug->nr_memory); + memcg_check_events(memcg, ug->nid); local_irq_restore(flags); + rcu_read_unlock(); =20 /* drop reference from uncharge_folio */ - css_put(&ug->memcg->css); + obj_cgroup_put(ug->objcg); } =20 static void uncharge_folio(struct folio *folio, struct uncharge_gather *ug) { long nr_pages; - struct mem_cgroup *memcg; struct obj_cgroup *objcg; =20 VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); =20 /* * Nobody should be changing or seriously looking at - * folio memcg or objcg at this point, we have fully - * exclusive access to the folio. + * folio objcg at this point, we have fully exclusive + * access to the folio. */ - if (folio_memcg_kmem(folio)) { - objcg =3D __folio_objcg(folio); - /* - * This get matches the put at the end of the function and - * kmem pages do not hold memcg references anymore. - */ - memcg =3D get_mem_cgroup_from_objcg(objcg); - } else { - memcg =3D __folio_memcg(folio); - } - - if (!memcg) + objcg =3D folio_objcg(folio); + if (!objcg) return; =20 - if (ug->memcg !=3D memcg) { - if (ug->memcg) { + if (ug->objcg !=3D objcg) { + if (ug->objcg) { uncharge_batch(ug); uncharge_gather_clear(ug); } - ug->memcg =3D memcg; + ug->objcg =3D objcg; ug->nid =3D folio_nid(folio); =20 - /* pairs with css_put in uncharge_batch */ - css_get(&memcg->css); + /* pairs with obj_cgroup_put in uncharge_batch */ + obj_cgroup_get(objcg); } =20 nr_pages =3D folio_nr_pages(folio); @@ -6871,19 +6996,15 @@ static void uncharge_folio(struct folio *folio, str= uct uncharge_gather *ug) if (folio_memcg_kmem(folio)) { ug->nr_memory +=3D nr_pages; ug->nr_kmem +=3D nr_pages; - - folio->memcg_data =3D 0; - obj_cgroup_put(objcg); } else { /* LRU pages aren't accounted at the root level */ - if (!mem_cgroup_is_root(memcg)) + if (!obj_cgroup_is_root(objcg)) ug->nr_memory +=3D nr_pages; ug->pgpgout++; - - folio->memcg_data =3D 0; } =20 - css_put(&memcg->css); + folio->memcg_data =3D 0; + obj_cgroup_put(objcg); } =20 void __mem_cgroup_uncharge(struct folio *folio) @@ -6891,7 +7012,7 @@ void __mem_cgroup_uncharge(struct folio *folio) struct uncharge_gather ug; =20 /* Don't touch folio->lru of any random page, pre-check: */ - if (!folio_memcg(folio)) + if (!folio_objcg(folio)) return; =20 uncharge_gather_clear(&ug); @@ -6914,7 +7035,7 @@ void __mem_cgroup_uncharge_list(struct list_head *pag= e_list) uncharge_gather_clear(&ug); list_for_each_entry(folio, page_list, lru) uncharge_folio(folio, &ug); - if (ug.memcg) + if (ug.objcg) uncharge_batch(&ug); } =20 @@ -6931,6 +7052,7 @@ void __mem_cgroup_uncharge_list(struct list_head *pag= e_list) void mem_cgroup_migrate(struct folio *old, struct folio *new) { struct mem_cgroup *memcg; + struct obj_cgroup *objcg; long nr_pages =3D folio_nr_pages(new); unsigned long flags; =20 @@ -6943,30 +7065,33 @@ void mem_cgroup_migrate(struct folio *old, struct f= olio *new) return; =20 /* Page cache replacement: new folio already charged? */ - if (folio_memcg(new)) + if (folio_objcg(new)) return; =20 - memcg =3D get_mem_cgroup_from_folio(old); - VM_WARN_ON_ONCE_FOLIO(!memcg, old); - if (!memcg) + objcg =3D folio_objcg(old); + VM_WARN_ON_ONCE_FOLIO(!objcg, old); + if (!objcg) return; =20 + rcu_read_lock(); + memcg =3D obj_cgroup_memcg(objcg); + /* Force-charge the new page. The old one will be freed soon */ - if (!mem_cgroup_is_root(memcg)) { + if (!obj_cgroup_is_root(objcg)) { page_counter_charge(&memcg->memory, nr_pages); if (do_memsw_account()) page_counter_charge(&memcg->memsw, nr_pages); } =20 - css_get(&memcg->css); - commit_charge(new, memcg); + obj_cgroup_get(objcg); + commit_charge(new, objcg); =20 local_irq_save(flags); mem_cgroup_charge_statistics(memcg, nr_pages); memcg_check_events(memcg, folio_nid(new)); local_irq_restore(flags); =20 - css_put(&memcg->css); + rcu_read_unlock(); } =20 DEFINE_STATIC_KEY_FALSE(memcg_sockets_enabled_key); @@ -7141,6 +7266,7 @@ static struct mem_cgroup *mem_cgroup_id_get_online(st= ruct mem_cgroup *memcg) void mem_cgroup_swapout(struct folio *folio, swp_entry_t entry) { struct mem_cgroup *memcg, *swap_memcg; + struct obj_cgroup *objcg; unsigned int nr_entries; unsigned short oldid; =20 @@ -7153,15 +7279,16 @@ void mem_cgroup_swapout(struct folio *folio, swp_en= try_t entry) if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) return; =20 + objcg =3D folio_objcg(folio); + VM_WARN_ON_ONCE_FOLIO(!objcg, folio); + if (!objcg) + return; + /* * Interrupts should be disabled by the caller (see the comments below), * which can serve as RCU read-side critical sections. */ - memcg =3D folio_memcg(folio); - - VM_WARN_ON_ONCE_FOLIO(!memcg, folio); - if (!memcg) - return; + memcg =3D obj_cgroup_memcg(objcg); =20 /* * In case the memcg owning these pages has been offlined and doesn't @@ -7180,7 +7307,7 @@ void mem_cgroup_swapout(struct folio *folio, swp_entr= y_t entry) =20 folio->memcg_data =3D 0; =20 - if (!mem_cgroup_is_root(memcg)) + if (!obj_cgroup_is_root(objcg)) page_counter_uncharge(&memcg->memory, nr_entries); =20 if (!cgroup_memory_noswap && memcg !=3D swap_memcg) { @@ -7200,7 +7327,7 @@ void mem_cgroup_swapout(struct folio *folio, swp_entr= y_t entry) memcg_stats_unlock(); memcg_check_events(memcg, folio_nid(folio)); =20 - css_put(&memcg->css); + obj_cgroup_put(objcg); } =20 /** --=20 2.11.0 From nobody Sun May 5 20:57:33 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C885EC433EF for ; Tue, 24 May 2022 06:28:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235059AbiEXG2r (ORCPT ); Tue, 24 May 2022 02:28:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235048AbiEXG2f (ORCPT ); Tue, 24 May 2022 02:28:35 -0400 Received: from mail-pg1-x536.google.com (mail-pg1-x536.google.com [IPv6:2607:f8b0:4864:20::536]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 843BD72209 for ; Mon, 23 May 2022 23:28:29 -0700 (PDT) Received: by mail-pg1-x536.google.com with SMTP id t28so7936705pga.6 for ; Mon, 23 May 2022 23:28:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VRT3nBeIYnWORjFGgUrNJZAuxlpJgLGLO13gTpDFHrE=; b=8PujjlnVeI6J13WyXz+kTgVIeMvRKAqjfBM0f1CZmy36ow271MYI1koRfaVOmeJavo Bs6vuCtvQeR3d/rfyKlU46F2aCsbDIA1I/fsRTvx6nYln1rHXcV/5W0D3ELxK6f1Ecf0 CgSkMRA+a6v523xf+ZMs8mZOboyjugRwM4p+rSQQVV96/4eqCmeA4bx9GC2Ea4nfhfLA e4J7q7p8Sh/wemkRgQ8iepBIjupiT6ANaVxSGo8at6IjIg7l5+4pQW/g2oACvZBzt5nh mrC6iOGQTKu9NVxBoj3Ly9LRXo42V7ul8yyjUhEsvxaGZug3w3wgm7U09e2lU3e4CB9F HFUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VRT3nBeIYnWORjFGgUrNJZAuxlpJgLGLO13gTpDFHrE=; b=7mLoP8YRF349bao/KB6/anyiehm2tTkdshIkpE1KY0AcdHX0nsD1acbIv++ZT/LxRc 39u0zygJsZkmJW+b1XAG4Af6vTIIOZMTMSH/lBsnE6dBIV9mkkg6gBMBhWm67yvXAq4j f75+dAZT2BnUA20DCc1ETsiO9/O1gZqvfaUITFyh1ocfswwZ+NAoSx3RCdegYQUD0a/p 2JGcRvq3aH8/tj5D6AWYNczvY9s0wAUCXZdlSSheYgnYVNC9aMCh5bBiECWGnGtT+5kF CokZ2fDeXPGbNe3T0iGbCmBLPUA03ikG2o5mAbhrJnRteWA1pU//AUh+YJJLHZy2++iX VT2A== X-Gm-Message-State: AOAM532+VBdqQDcoGQmL9W5XQbxIElG7v1XR3u7bsdvamFkvlz4nNCF/ akE9XnvU7z1W3E/qMWr5PffyA523PN6mtQ== X-Google-Smtp-Source: ABdhPJw8XdwBF18E1nwEdwHGN4aTYH/lcpJm4fHP7nhdxHq1GZkMUFY/EvBUAVaAVU5iz54VOcQBOQ== X-Received: by 2002:a63:1d65:0:b0:3fa:7e32:8024 with SMTP id d37-20020a631d65000000b003fa7e328024mr5001251pgm.162.1653373709229; Mon, 23 May 2022 23:28:29 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([2408:8207:18da:2310:f940:af17:c2f5:8656]) by smtp.gmail.com with ESMTPSA id h5-20020a170902f54500b0016168e90f2dsm6254455plf.219.2022.05.23.23.28.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 May 2022 23:28:29 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, duanxiongchun@bytedance.com, longman@redhat.com, Muchun Song Subject: [PATCH v4 10/11] mm: lru: add VM_BUG_ON_FOLIO to lru maintenance function Date: Tue, 24 May 2022 14:05:50 +0800 Message-Id: <20220524060551.80037-11-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220524060551.80037-1-songmuchun@bytedance.com> References: <20220524060551.80037-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" We need to make sure that the page is deleted from or added to the correct lruvec list. So add a VM_BUG_ON_FOLIO() to catch invalid users. Signed-off-by: Muchun Song --- include/linux/mm_inline.h | 6 ++++++ mm/vmscan.c | 1 - 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index ac32125745ab..30d2393da613 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -97,6 +97,8 @@ void lruvec_add_folio(struct lruvec *lruvec, struct folio= *folio) { enum lru_list lru =3D folio_lru_list(folio); =20 + VM_BUG_ON_FOLIO(!folio_matches_lruvec(folio, lruvec), folio); + update_lru_size(lruvec, lru, folio_zonenum(folio), folio_nr_pages(folio)); if (lru !=3D LRU_UNEVICTABLE) @@ -114,6 +116,8 @@ void lruvec_add_folio_tail(struct lruvec *lruvec, struc= t folio *folio) { enum lru_list lru =3D folio_lru_list(folio); =20 + VM_BUG_ON_FOLIO(!folio_matches_lruvec(folio, lruvec), folio); + update_lru_size(lruvec, lru, folio_zonenum(folio), folio_nr_pages(folio)); /* This is not expected to be used on LRU_UNEVICTABLE */ @@ -131,6 +135,8 @@ void lruvec_del_folio(struct lruvec *lruvec, struct fol= io *folio) { enum lru_list lru =3D folio_lru_list(folio); =20 + VM_BUG_ON_FOLIO(!folio_matches_lruvec(folio, lruvec), folio); + if (lru !=3D LRU_UNEVICTABLE) list_del(&folio->lru); update_lru_size(lruvec, lru, folio_zonenum(folio), diff --git a/mm/vmscan.c b/mm/vmscan.c index 761d5e0dd78d..6c9e2eafc8f9 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2281,7 +2281,6 @@ static unsigned int move_pages_to_lru(struct list_hea= d *list) continue; } =20 - VM_BUG_ON_PAGE(!folio_matches_lruvec(folio, lruvec), page); add_page_to_lru_list(page, lruvec); nr_pages =3D thp_nr_pages(page); nr_moved +=3D nr_pages; --=20 2.11.0 From nobody Sun May 5 20:57:33 2024 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F81FC433F5 for ; Tue, 24 May 2022 06:28:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234297AbiEXG2z (ORCPT ); Tue, 24 May 2022 02:28:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49804 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235078AbiEXG2n (ORCPT ); Tue, 24 May 2022 02:28:43 -0400 Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1658895DCB for ; Mon, 23 May 2022 23:28:36 -0700 (PDT) Received: by mail-pj1-x102f.google.com with SMTP id nk9-20020a17090b194900b001df2fcdc165so1254618pjb.0 for ; Mon, 23 May 2022 23:28:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=2hYwGkjhBJyJx1MvqMBFvObYccg6CgYmwL5qHdR4ZiM=; b=GFMb6tk3YXF6rCtekI1Z960E9oVS5AdQHpWxVIVmq5MXcaqETUL8TVzU1uUNBFKyo/ /PfpUkia6P9oSanDlllyDyuTZtZfeHRXRPQweEyxDm/QpQlRpxxXslqEhTiCj+QvJ6tW lbKHpdbI5+Izjs/Lkfjl/AhUWX6g8UPf14bJps3dLRI2XDn499bnbeEz6NROkbiMSA+z N2rJHhmjTqd1HSnTND3D3F4eQWYTpm7WEb5KcGeoN1yG30KwPM+v8Nhl9NDVbh5R6Hzt hFhD4dv4h7vobCU/eAw/H7JNVRdgdBkPfyMFeHZIIAJOl5iqYlOSkrCt3LeNoerW6Zpr kpqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2hYwGkjhBJyJx1MvqMBFvObYccg6CgYmwL5qHdR4ZiM=; b=XhpVMwBRze7iKvVywxYnz959oiqQmroTM9gCeHw+hCNUM/zZNjYXo+h+YIJUVorgIl StgQUy8pT71jCLi91lovGZ6IjrnrbfCJD2L2N0c/97NsUMN/ylPAcRdF3UC4qr4ebDZU atL2Y7THBvuXIJDZlwhoMioKjhJy/leRiI2e9gb2+n5Q/aZrr+hBPSJ3ebDuwHUlPQUN hbDTeY3cqw6so+TqOJWGSbApdP3tO00PUpP02Byk6UMZrppA8/A7kHsINNG/upZbEyhZ sogVsuYbRbqvVKJOGaqQo6JYm+lbcF0Hvj54qM9H0U+5rPaNNH7IpZ2FND/TrquTx/6z vmdA== X-Gm-Message-State: AOAM532EIAdHmLsXn6+6TSpAcJBXs9iWlRWJ+RPI+K38u6fI6iT/6WGY ck0mrDncGNdrkASe6/Y2im6VuQ== X-Google-Smtp-Source: ABdhPJyNK37mlb293mGVMDlTbFm6VQ+rWibO81tqDEaLbEmDCctCXVD1f61mgaBCRozNBaSHZMSkfQ== X-Received: by 2002:a17:902:bb90:b0:158:a031:2ff2 with SMTP id m16-20020a170902bb9000b00158a0312ff2mr25960427pls.117.1653373715289; Mon, 23 May 2022 23:28:35 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([2408:8207:18da:2310:f940:af17:c2f5:8656]) by smtp.gmail.com with ESMTPSA id h5-20020a170902f54500b0016168e90f2dsm6254455plf.219.2022.05.23.23.28.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 23 May 2022 23:28:35 -0700 (PDT) From: Muchun Song To: hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com Cc: cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, duanxiongchun@bytedance.com, longman@redhat.com, Muchun Song Subject: [PATCH v4 11/11] mm: lru: use lruvec lock to serialize memcg changes Date: Tue, 24 May 2022 14:05:51 +0800 Message-Id: <20220524060551.80037-12-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220524060551.80037-1-songmuchun@bytedance.com> References: <20220524060551.80037-1-songmuchun@bytedance.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" As described by commit fc574c23558c ("mm/swap.c: serialize memcg changes in pagevec_lru_move_fn"), TestClearPageLRU() aims to serialize mem_cgroup_move_account() during pagevec_lru_move_fn(). Now folio_lruvec_lock*() has the ability to detect whether page memcg has been changed. So we can use lruvec lock to serialize mem_cgroup_move_account() during pagevec_lru_move_fn(). This change is a partial revert of the commit fc574c23558c ("mm/swap.c: serialize memcg changes in pagevec_lru_move_fn"). And pagevec_lru_move_fn() is more hot compare with mem_cgroup_move_account(), removing an atomic operation would be an optimization. Also this change would not dirty cacheline for a page which isn't on the LRU. Signed-off-by: Muchun Song --- mm/memcontrol.c | 31 +++++++++++++++++++++++++++++++ mm/swap.c | 45 ++++++++++++++------------------------------- mm/vmscan.c | 9 ++++----- 3 files changed, 49 insertions(+), 36 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1a35f7fde3ed..7b6d9c308d91 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1331,12 +1331,38 @@ struct lruvec *folio_lruvec_lock(struct folio *foli= o) lruvec =3D folio_lruvec(folio); spin_lock(&lruvec->lru_lock); =20 + /* + * The memcg of the page can be changed by any the following routines: + * + * 1) mem_cgroup_move_account() or + * 2) memcg_reparent_objcgs() + * + * The possible bad scenario would like: + * + * CPU0: CPU1: CPU2: + * lruvec =3D folio_lruvec() + * + * if (!isolate_lru_page()) + * mem_cgroup_move_account() + * + * memcg_reparent_objcgs() + * + * spin_lock(&lruvec->lru_lock) + * ^^^^^^ + * wrong lock + * + * Either CPU1 or CPU2 can change page memcg, so we need to check + * whether page memcg is changed, if so, we should reacquire the + * new lruvec lock. + */ if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) { spin_unlock(&lruvec->lru_lock); goto retry; } =20 /* + * When we reach here, it means that the folio_memcg(folio) is stable. + * * Preemption is disabled in the internal of spin_lock, which can serve * as RCU read-side critical sections. */ @@ -1367,6 +1393,7 @@ struct lruvec *folio_lruvec_lock_irq(struct folio *fo= lio) lruvec =3D folio_lruvec(folio); spin_lock_irq(&lruvec->lru_lock); =20 + /* See the comments in folio_lruvec_lock(). */ if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) { spin_unlock_irq(&lruvec->lru_lock); goto retry; @@ -1402,6 +1429,7 @@ struct lruvec *folio_lruvec_lock_irqsave(struct folio= *folio, lruvec =3D folio_lruvec(folio); spin_lock_irqsave(&lruvec->lru_lock, *flags); =20 + /* See the comments in folio_lruvec_lock(). */ if (unlikely(lruvec_memcg(lruvec) !=3D folio_memcg(folio))) { spin_unlock_irqrestore(&lruvec->lru_lock, *flags); goto retry; @@ -5751,7 +5779,10 @@ static int mem_cgroup_move_account(struct page *page, obj_cgroup_put(rcu_dereference(from->objcg)); rcu_read_unlock(); =20 + /* See the comments in folio_lruvec_lock(). */ + spin_lock(&from_vec->lru_lock); folio->memcg_data =3D (unsigned long)rcu_access_pointer(to->objcg); + spin_unlock(&from_vec->lru_lock); =20 __folio_memcg_unlock(from); =20 diff --git a/mm/swap.c b/mm/swap.c index 9680f2fc48b1..984b100e84e4 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -199,14 +199,8 @@ static void pagevec_lru_move_fn(struct pagevec *pvec, struct page *page =3D pvec->pages[i]; struct folio *folio =3D page_folio(page); =20 - /* block memcg migration during page moving between lru */ - if (!TestClearPageLRU(page)) - continue; - lruvec =3D folio_lruvec_relock_irqsave(folio, lruvec, &flags); (*move_fn)(page, lruvec); - - SetPageLRU(page); } if (lruvec) unlock_page_lruvec_irqrestore(lruvec, flags); @@ -218,7 +212,7 @@ static void pagevec_move_tail_fn(struct page *page, str= uct lruvec *lruvec) { struct folio *folio =3D page_folio(page); =20 - if (!folio_test_unevictable(folio)) { + if (folio_test_lru(folio) && !folio_test_unevictable(folio)) { lruvec_del_folio(lruvec, folio); folio_clear_active(folio); lruvec_add_folio_tail(lruvec, folio); @@ -313,7 +307,8 @@ void lru_note_cost_folio(struct folio *folio) =20 static void __folio_activate(struct folio *folio, struct lruvec *lruvec) { - if (!folio_test_active(folio) && !folio_test_unevictable(folio)) { + if (folio_test_lru(folio) && !folio_test_active(folio) && + !folio_test_unevictable(folio)) { long nr_pages =3D folio_nr_pages(folio); =20 lruvec_del_folio(lruvec, folio); @@ -370,12 +365,9 @@ static void folio_activate(struct folio *folio) { struct lruvec *lruvec; =20 - if (folio_test_clear_lru(folio)) { - lruvec =3D folio_lruvec_lock_irq(folio); - __folio_activate(folio, lruvec); - unlock_page_lruvec_irq(lruvec); - folio_set_lru(folio); - } + lruvec =3D folio_lruvec_lock_irq(folio); + __folio_activate(folio, lruvec); + unlock_page_lruvec_irq(lruvec); } #endif =20 @@ -518,6 +510,9 @@ static void lru_deactivate_file_fn(struct page *page, s= truct lruvec *lruvec) bool active =3D PageActive(page); int nr_pages =3D thp_nr_pages(page); =20 + if (!PageLRU(page)) + return; + if (PageUnevictable(page)) return; =20 @@ -555,7 +550,7 @@ static void lru_deactivate_file_fn(struct page *page, s= truct lruvec *lruvec) =20 static void lru_deactivate_fn(struct page *page, struct lruvec *lruvec) { - if (PageActive(page) && !PageUnevictable(page)) { + if (PageLRU(page) && PageActive(page) && !PageUnevictable(page)) { int nr_pages =3D thp_nr_pages(page); =20 del_page_from_lru_list(page, lruvec); @@ -571,7 +566,7 @@ static void lru_deactivate_fn(struct page *page, struct= lruvec *lruvec) =20 static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec) { - if (PageAnon(page) && PageSwapBacked(page) && + if (PageLRU(page) && PageAnon(page) && PageSwapBacked(page) && !PageSwapCache(page) && !PageUnevictable(page)) { int nr_pages =3D thp_nr_pages(page); =20 @@ -1006,8 +1001,9 @@ void __pagevec_release(struct pagevec *pvec) } EXPORT_SYMBOL(__pagevec_release); =20 -static void __pagevec_lru_add_fn(struct folio *folio, struct lruvec *lruve= c) +static void __pagevec_lru_add_fn(struct page *page, struct lruvec *lruvec) { + struct folio *folio =3D page_folio(page); int was_unevictable =3D folio_test_clear_unevictable(folio); long nr_pages =3D folio_nr_pages(folio); =20 @@ -1053,20 +1049,7 @@ static void __pagevec_lru_add_fn(struct folio *folio= , struct lruvec *lruvec) */ void __pagevec_lru_add(struct pagevec *pvec) { - int i; - struct lruvec *lruvec =3D NULL; - unsigned long flags =3D 0; - - for (i =3D 0; i < pagevec_count(pvec); i++) { - struct folio *folio =3D page_folio(pvec->pages[i]); - - lruvec =3D folio_lruvec_relock_irqsave(folio, lruvec, &flags); - __pagevec_lru_add_fn(folio, lruvec); - } - if (lruvec) - unlock_page_lruvec_irqrestore(lruvec, flags); - release_pages(pvec->pages, pvec->nr); - pagevec_reinit(pvec); + pagevec_lru_move_fn(pvec, __pagevec_lru_add_fn); } =20 /** diff --git a/mm/vmscan.c b/mm/vmscan.c index 6c9e2eafc8f9..ec1272ca5ead 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4793,18 +4793,17 @@ void check_move_unevictable_pages(struct pagevec *p= vec) nr_pages =3D thp_nr_pages(page); pgscanned +=3D nr_pages; =20 - /* block memcg migration during page moving between lru */ - if (!TestClearPageLRU(page)) + lruvec =3D folio_lruvec_relock_irq(folio, lruvec); + + if (!PageLRU(page) || !PageUnevictable(page)) continue; =20 - lruvec =3D folio_lruvec_relock_irq(folio, lruvec); - if (page_evictable(page) && PageUnevictable(page)) { + if (page_evictable(page)) { del_page_from_lru_list(page, lruvec); ClearPageUnevictable(page); add_page_to_lru_list(page, lruvec); pgrescued +=3D nr_pages; } - SetPageLRU(page); } =20 if (lruvec) { --=20 2.11.0