From nobody Sat Feb 7 12:05:34 2026 Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1D4951ACEAE for ; Wed, 18 Dec 2024 11:47:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734522423; cv=none; b=JmhQ/IhlVesWsUMHVQcdEL/dGxjuZKxPLI8N/rqicJpfbft9O5vVQ4YaFtWsn9DSGemfj2RuHhByj97TF1h/1S/NAkup5i2wMNrly1FUOxTq6/PlVZQ12nMxJfjLljuQ1sPa3oA5mcTIgNaVlNYBzx3UzywM1z31muXBH+u7ZpY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1734522423; c=relaxed/simple; bh=iO77dclC7icaLUgCLpicVvYq/fLvHBaS3sC4soxvpTs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iBTogdLhjvEjZ6yfmBGQxYzLkJdalecIMOdWj/wsp6uZJPfXOFYtWxk5aX0urk4+zYonY3RoyVK5YIdFh5U6u4tllRhvjGk/NO/me8eLZehSdQyd++PTsx9PAOZcyoqM9460ln10e5inEeQluUYTD+mMbq0XjwhXIq2Y/GINXTc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=OcdcnwHb; arc=none smtp.client-ip=209.85.216.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OcdcnwHb" Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-2f43da61ba9so152065a91.2 for ; Wed, 18 Dec 2024 03:47:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1734522421; x=1735127221; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=+Vzzy564Q7ltilyK4Qu+63SEqQV/iay9myd3eLrJl6Y=; b=OcdcnwHbCzyEpgCimBeYCfU2SS/eHKfPM4/CO85Oz9q6wc82OvjS0m5axor9EeO0Xq Ytt7VlFDlYPkn7E6KvMEdbXOyXGNAAt+P2IVwlCebVQ5PMcsbwW4eNlc/zkWGJAVv4gv 7obQD+O9oJrU6xqRrIX3jHGvDbUpY4MbQCNRY/V5MqC01xhwOSrWIJCZqduIdTJBbJdn m8a0SF1xrechO5ehSpz8sz8GRsejKW1+nBR2G8LSYTVsYslA9nvlMP7R7v8BmLv5es8a DOmTEAIrg81Gjnoklk92IwUAoJPTEWWu/eThpqMS8RxPvIVuj6FmeN2Tv9xnAPhkSX4y wrNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734522421; x=1735127221; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=+Vzzy564Q7ltilyK4Qu+63SEqQV/iay9myd3eLrJl6Y=; b=B7icpzLk+r1+Sxmu90ovDmP7gxxFSqKedVUncXzTsr+41zqQOvrpbNrEAuwJOcivQy WMfNNbeXUyJYqKuSER/fJXczclMohFiFMvtFdsoridRC0n4IoQXcNoeoT+QpvfF2PPXr IyjBIa4E0jCZuWTG4dTTRUUKNXM47woIyOjNfhOAi1lwUbI5KGrqqjEj6Uur2ZDhHNKu ge0sY4vxAvSbDw2TQ+t6UXZa7uFMcizoIk4x9a65Pbd/pkCeg5nvErCvCuf8JVZucdj6 6Wgb86rtbmNHVKqbjsBfRmBl4Kq9O3p2P7c/GVOEVtUGzWTuK1mUYRJQrCdnDkxcFAFf n1lg== X-Forwarded-Encrypted: i=1; AJvYcCU6vXqn6lr1UImXIjRBC2vd5szuivQlQprJCJ46wGO1mbP2DjJ75JKMljyWTZqezbN5Crt4Rzyqxk6LLLY=@vger.kernel.org X-Gm-Message-State: AOJu0Ywul7LlvZPQKay3IuYzna6uOhgFhW0Q0juCeUxaFWQS7CSyUIKu O5qKWSuiOZnL5EAnLMzyWOWovcVnPsvAFnvpnnGO4ub7jcV7lhHE X-Gm-Gg: ASbGncsBRTCm7oLGnHMGcNOFSE2JVXnqC87ErcmwjLN4ZpTPA2pW+uNe3rFwWXAAdDu VikEpI7cM71JFg40K25p9+i3XgGLTu0OhOFoe8tZA3WXjEWjVczc/N/aaDc9IgKZaQrfMMqRLJn 7MUFNX4/2Hi9/VfBqWLobM48H4iRfwE1MJ9fe+Y+K4iGlQQPNt0xKQ1n87a0RRNacu5Nvopvc1s It2IktyK2gP6MdSBPBmJyggskVNaBqHB+LfUb8Ck7dGdHpEtIHWAppxooWxktPbH1snpcLLAq8k TgTirY9wCc4= X-Google-Smtp-Source: AGHT+IEO+JWDMXQW6FsZJE3+1Wvh/7poieUhJ+CwA1N0IXPZ0ViHOqmG3ylHlvEdt5u9FJAhum58XA== X-Received: by 2002:a17:90b:5206:b0:2ee:c04a:4281 with SMTP id 98e67ed59e1d1-2f2e91a903cmr3823034a91.6.1734522421476; Wed, 18 Dec 2024 03:47:01 -0800 (PST) Received: from KASONG-MC4.tencent.com ([43.132.141.21]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db7f50sm74337285ad.39.2024.12.18.03.46.58 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Wed, 18 Dec 2024 03:47:01 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Hugh Dickins , "Huang, Ying" , Yosry Ahmed , Roman Gushchin , Shakeel Butt , Johannes Weiner , Barry Song , Michal Hocko , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v3 3/4] mm/swap_cgroup: remove global swap cgroup lock Date: Wed, 18 Dec 2024 19:46:32 +0800 Message-ID: <20241218114633.85196-4-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241218114633.85196-1-ryncsn@gmail.com> References: <20241218114633.85196-1-ryncsn@gmail.com> Reply-To: Kairui Song Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Kairui Song commit e9e58a4ec3b1 ("memcg: avoid use cmpxchg in swap cgroup maintainance") replaced the cmpxchg/xchg with a global irq spinlock because some archs doesn't support 2 bytes cmpxchg/xchg. Clearly this won't scale well. And as commented in swap_cgroup.c, this lock is not needed for map synchronization. Emulation of 2 bytes xchg with atomic cmpxchg isn't hard, so implement it to get rid of this lock. Introduced two helpers for doing so and they can be easily dropped if a generic 2 byte xchg is support. Testing using 64G brd and build with build kernel with make -j96 in 1.5G memory cgroup using 4k folios showed below improvement (6 test run): Before this series: Sys time: 10782.29 (stdev 42.353886) Real time: 171.49 (stdev 0.595541) After this commit: Sys time: 9617.23 (stdev 37.764062), -10.81% Real time: 159.65 (stdev 0.587388), -6.90% With 64k folios and 2G memcg: Before this series: Sys time: 8176.94 (stdev 26.414712) Real time: 141.98 (stdev 0.797382) After this commit: Sys time: 7358.98 (stdev 54.927593), -10.00% Real time: 134.07 (stdev 0.757463), -5.57% Sequential swapout of 8G 64k zero folios with madvise (24 test run): Before this series: 5461409.12 us (stdev 183957.827084) After this commit: 5420447.26 us (stdev 196419.240317) Sequential swapin of 8G 4k zero folios (24 test run): Before this series: 19736958.916667 us (stdev 189027.246676) After this commit: 19662182.629630 us (stdev 172717.640614) Performance is better or at least not worse for all tests above. Signed-off-by: Kairui Song Reviewed-by: Roman Gushchin --- mm/swap_cgroup.c | 77 ++++++++++++++++++++++++++++++------------------ 1 file changed, 49 insertions(+), 28 deletions(-) diff --git a/mm/swap_cgroup.c b/mm/swap_cgroup.c index 1770b076f6b7..cf0445cb35ed 100644 --- a/mm/swap_cgroup.c +++ b/mm/swap_cgroup.c @@ -7,19 +7,20 @@ =20 static DEFINE_MUTEX(swap_cgroup_mutex); =20 +/* Pack two cgroup id (short) of two entries in one swap_cgroup (atomic_t)= */ +#define ID_PER_SC (sizeof(struct swap_cgroup) / sizeof(unsigned short)) +#define ID_SHIFT (BITS_PER_TYPE(unsigned short)) +#define ID_MASK (BIT(ID_SHIFT) - 1) struct swap_cgroup { - unsigned short id; + atomic_t ids; }; =20 struct swap_cgroup_ctrl { struct swap_cgroup *map; - spinlock_t lock; }; =20 static struct swap_cgroup_ctrl swap_cgroup_ctrl[MAX_SWAPFILES]; =20 -#define SC_PER_PAGE (PAGE_SIZE/sizeof(struct swap_cgroup)) - /* * SwapCgroup implements "lookup" and "exchange" operations. * In typical usage, this swap_cgroup is accessed via memcg's charge/uncha= rge @@ -30,19 +31,35 @@ static struct swap_cgroup_ctrl swap_cgroup_ctrl[MAX_SWA= PFILES]; * SwapCache(and its swp_entry) is under lock. * - When called via swap_free(), there is no user of this entry and no r= ace. * Then, we don't need lock around "exchange". - * - * TODO: we can push these buffers out to HIGHMEM. */ -static struct swap_cgroup *lookup_swap_cgroup(swp_entry_t ent, - struct swap_cgroup_ctrl **ctrlp) +static unsigned short __swap_cgroup_id_lookup(struct swap_cgroup *map, + pgoff_t offset) { - pgoff_t offset =3D swp_offset(ent); - struct swap_cgroup_ctrl *ctrl; + unsigned int shift =3D (offset % ID_PER_SC) * ID_SHIFT; + unsigned int old_ids =3D atomic_read(&map[offset / ID_PER_SC].ids); =20 - ctrl =3D &swap_cgroup_ctrl[swp_type(ent)]; - if (ctrlp) - *ctrlp =3D ctrl; - return &ctrl->map[offset]; + BUILD_BUG_ON(!is_power_of_2(ID_PER_SC)); + BUILD_BUG_ON(sizeof(struct swap_cgroup) !=3D sizeof(atomic_t)); + + return (old_ids >> shift) & ID_MASK; +} + +static unsigned short __swap_cgroup_id_xchg(struct swap_cgroup *map, + pgoff_t offset, + unsigned short new_id) +{ + unsigned short old_id; + struct swap_cgroup *sc =3D &map[offset / ID_PER_SC]; + unsigned int shift =3D (offset % ID_PER_SC) * ID_SHIFT; + unsigned int new_ids, old_ids =3D atomic_read(&sc->ids); + + do { + old_id =3D (old_ids >> shift) & ID_MASK; + new_ids =3D (old_ids & ~(ID_MASK << shift)); + new_ids |=3D ((unsigned int)new_id) << shift; + } while (!atomic_try_cmpxchg(&sc->ids, &old_ids, new_ids)); + + return old_id; } =20 /** @@ -58,21 +75,19 @@ unsigned short swap_cgroup_record(swp_entry_t ent, unsi= gned short id, unsigned int nr_ents) { struct swap_cgroup_ctrl *ctrl; - struct swap_cgroup *sc; - unsigned short old; - unsigned long flags; pgoff_t offset =3D swp_offset(ent); pgoff_t end =3D offset + nr_ents; + unsigned short old, iter; + struct swap_cgroup *map; =20 - sc =3D lookup_swap_cgroup(ent, &ctrl); + ctrl =3D &swap_cgroup_ctrl[swp_type(ent)]; + map =3D ctrl->map; =20 - spin_lock_irqsave(&ctrl->lock, flags); - old =3D sc->id; - for (; offset < end; offset++, sc++) { - VM_BUG_ON(sc->id !=3D old); - sc->id =3D id; - } - spin_unlock_irqrestore(&ctrl->lock, flags); + old =3D __swap_cgroup_id_lookup(map, offset); + do { + iter =3D __swap_cgroup_id_xchg(map, offset, id); + VM_BUG_ON(iter !=3D old); + } while (++offset !=3D end); =20 return old; } @@ -85,9 +100,13 @@ unsigned short swap_cgroup_record(swp_entry_t ent, unsi= gned short id, */ unsigned short lookup_swap_cgroup_id(swp_entry_t ent) { + struct swap_cgroup_ctrl *ctrl; + if (mem_cgroup_disabled()) return 0; - return lookup_swap_cgroup(ent, NULL)->id; + + ctrl =3D &swap_cgroup_ctrl[swp_type(ent)]; + return __swap_cgroup_id_lookup(ctrl->map, swp_offset(ent)); } =20 int swap_cgroup_swapon(int type, unsigned long max_pages) @@ -98,14 +117,16 @@ int swap_cgroup_swapon(int type, unsigned long max_pag= es) if (mem_cgroup_disabled()) return 0; =20 - map =3D vcalloc(max_pages, sizeof(struct swap_cgroup)); + BUILD_BUG_ON(sizeof(unsigned short) * ID_PER_SC !=3D + sizeof(struct swap_cgroup)); + map =3D vcalloc(DIV_ROUND_UP(max_pages, ID_PER_SC), + sizeof(struct swap_cgroup)); if (!map) goto nomem; =20 ctrl =3D &swap_cgroup_ctrl[type]; mutex_lock(&swap_cgroup_mutex); ctrl->map =3D map; - spin_lock_init(&ctrl->lock); mutex_unlock(&swap_cgroup_mutex); =20 return 0; --=20 2.47.1