From nobody Thu Apr 9 04:42:38 2026 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7AE9C433FE for ; Wed, 2 Nov 2022 02:05:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230187AbiKBCFG (ORCPT ); Tue, 1 Nov 2022 22:05:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34600 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230122AbiKBCEy (ORCPT ); Tue, 1 Nov 2022 22:04:54 -0400 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1FD6E6351 for ; Tue, 1 Nov 2022 19:03:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667354631; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9awyJRy6zbIYnxU2Cyp8GDi5vivjY8I9wO3N/Lj5YCI=; b=FRgrnSxEw6MrXo8g7bVS2LwGLrnJ/DyTJ6aDxu52eO3jWNDdp75fa+yu8ZL3P+m1w9CZpy dbE8JoyWEcsN+OeM6DQjn6rAMJT7f9obDeka0Zm4eHx4UzWMSXwXq7uMJeI14JgVSBhrXZ OUGRv4w3Ih4I9vbH7D0ob5E4t7BHnuY= Received: from mail-oa1-f71.google.com (mail-oa1-f71.google.com [209.85.160.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-548-y8dEA0X8PuaogSBUFbIpNw-1; Tue, 01 Nov 2022 22:03:49 -0400 X-MC-Unique: y8dEA0X8PuaogSBUFbIpNw-1 Received: by mail-oa1-f71.google.com with SMTP id 586e51a60fabf-13bf576ffa6so8095986fac.9 for ; Tue, 01 Nov 2022 19:03:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9awyJRy6zbIYnxU2Cyp8GDi5vivjY8I9wO3N/Lj5YCI=; b=FKliEiPpmxf2y01QRbKdhGdThKDeBZ7qlxKCH3x/YfncLKdulq2v+LDkaHlHpea5x1 /WTxGPPI4kgROHTX+YtWVjLoXjFU1P5qamc1muVLlqPp5loHtwJhmHeDxbxlMvL1ukqk 5lY1+9Aut9IuAVRUVPrGibA6dv4UM8b/IxmSPw/T1Kt/oRM62pT7MzNHwefrHpMJhrdE 7G4iTS+RRAsa0Zo3mh+GwW4FX31FlLozniEHEz22UsxseNT4xxOHq1YbPWawSTei2Fna nMAr8zKHE14s+dIzR5YSi4v2REgdX/BpxKgWBYseTtWqRqM7AEF6s9uuWQyZB4PttDS+ 2onQ== X-Gm-Message-State: ACrzQf2489St25aQCBKbbIkegZAqV4ff3a2PwDIrMDPpNZB3TRQR4hIu 40vPHe0FNjaJSEZFwBz7KI5KQ+lKPNrScMIf3fHGp23GyZWQvDrNVFIX1uIDxYnrK49U3zwPyCV cruh1tuWUqdYbyRFF8IOj+UQS X-Received: by 2002:a05:6871:58b:b0:13c:be46:a02 with SMTP id u11-20020a056871058b00b0013cbe460a02mr11636093oan.8.1667354629098; Tue, 01 Nov 2022 19:03:49 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5ltoJltHI6Z/nEB+eev0WW7uJWB140pg3Bn+RcqlseVCQsPXYU6LZG5IjvE4MMVOkxV19MIA== X-Received: by 2002:a05:6871:58b:b0:13c:be46:a02 with SMTP id u11-20020a056871058b00b0013cbe460a02mr11636072oan.8.1667354628892; Tue, 01 Nov 2022 19:03:48 -0700 (PDT) Received: from LeoBras.redhat.com ([2804:1b3:a802:1099:7cb2:3a49:6197:5307]) by smtp.gmail.com with ESMTPSA id h15-20020a9d6f8f000000b00665919f7823sm4526624otq.8.2022.11.01.19.03.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 19:03:48 -0700 (PDT) From: Leonardo Bras To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Frederic Weisbecker , Leonardo Bras , Phil Auld , Marcelo Tosatti Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 2/3] mm/memcontrol: Change stock_lock type from local_lock_t to spinlock_t Date: Tue, 1 Nov 2022 23:02:42 -0300 Message-Id: <20221102020243.522358-3-leobras@redhat.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221102020243.522358-1-leobras@redhat.com> References: <20221102020243.522358-1-leobras@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" In this context, since it's using per-cpu variables, changing from local_lock to spinlock should not deal much impact in performance and can allow operations such as stock draining to happen in remote cpus. Why performance would probably not get impacted: 1 - Since the lock is in the same cache line as the information that is used next, there is no extra memory access for caching the lock. 2 - Since it's a percpu struct, there should be no other cpu sharing this cacheline, so there is no need for cacheline invalidation, and writing to the lock should be as fast as the next struct members. 3 - Even the write in (2) could be pipelined and batched with following writes to the cacheline (such as nr_pages member), further decreasing the impact of this change. Suggested-by: Marcelo Tosatti Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 2d8549ae1b300..add46da2e6df1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2167,7 +2167,7 @@ void unlock_page_memcg(struct page *page) } =20 struct memcg_stock_pcp { - local_lock_t stock_lock; + spinlock_t stock_lock; /* Protects the percpu struct */ struct mem_cgroup *cached; /* this never be root cgroup */ unsigned int nr_pages; =20 @@ -2184,7 +2184,7 @@ struct memcg_stock_pcp { #define FLUSHING_CACHED_CHARGE 0 }; static DEFINE_PER_CPU(struct memcg_stock_pcp, memcg_stock) =3D { - .stock_lock =3D INIT_LOCAL_LOCK(stock_lock), + .stock_lock =3D __SPIN_LOCK_UNLOCKED(stock_lock), }; static DEFINE_MUTEX(percpu_charge_mutex); =20 @@ -2229,15 +2229,15 @@ static bool consume_stock(struct mem_cgroup *memcg,= unsigned int nr_pages) if (nr_pages > MEMCG_CHARGE_BATCH) return ret; =20 - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock =3D this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + if (memcg =3D=3D stock->cached && stock->nr_pages >=3D nr_pages) { stock->nr_pages -=3D nr_pages; ret =3D true; } =20 - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); =20 return ret; } @@ -2274,14 +2274,14 @@ static void drain_local_stock(struct work_struct *d= ummy) * drain_stock races is that we always operate on local CPU stock * here with IRQ disabled */ - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock =3D this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + old =3D drain_obj_stock(stock); drain_stock(stock); clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); =20 - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) obj_cgroup_put(old); } @@ -2309,10 +2309,12 @@ static void __refill_stock(struct mem_cgroup *memcg= , unsigned int nr_pages) static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { unsigned long flags; + struct memcg_stock_pcp *stock; =20 - local_lock_irqsave(&memcg_stock.stock_lock, flags); + stock =3D this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); __refill_stock(memcg, nr_pages); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); } =20 /* @@ -3157,8 +3159,8 @@ void mod_objcg_state(struct obj_cgroup *objcg, struct= pglist_data *pgdat, unsigned long flags; int *bytes; =20 - local_lock_irqsave(&memcg_stock.stock_lock, flags); stock =3D this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); =20 /* * Save vmstat data in stock and skip vmstat array update unless @@ -3210,7 +3212,7 @@ void mod_objcg_state(struct obj_cgroup *objcg, struct= pglist_data *pgdat, if (nr) mod_objcg_mlstate(objcg, pgdat, idx, nr); =20 - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) obj_cgroup_put(old); } @@ -3221,15 +3223,15 @@ static bool consume_obj_stock(struct obj_cgroup *ob= jcg, unsigned int nr_bytes) unsigned long flags; bool ret =3D false; =20 - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock =3D this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + if (objcg =3D=3D stock->cached_objcg && stock->nr_bytes >=3D nr_bytes) { stock->nr_bytes -=3D nr_bytes; ret =3D true; } =20 - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); =20 return ret; } @@ -3319,9 +3321,9 @@ static void refill_obj_stock(struct obj_cgroup *objcg= , unsigned int nr_bytes, unsigned long flags; unsigned int nr_pages =3D 0; =20 - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock =3D this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + if (stock->cached_objcg !=3D objcg) { /* reset if necessary */ old =3D drain_obj_stock(stock); obj_cgroup_get(objcg); @@ -3337,7 +3339,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg= , unsigned int nr_bytes, stock->nr_bytes &=3D (PAGE_SIZE - 1); } =20 - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) obj_cgroup_put(old); =20 --=20 2.38.1