From nobody Sat Feb 7 18:00:49 2026 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 69B171A7047 for ; Tue, 13 Aug 2024 20:47:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723582078; cv=none; b=ILcFjeu47DyHFzQFXWjpW1WMZQyeDEAI4d0DdCG+AxFQ3BD91BYF39jsi1OormRVTZAjzYcaytwvFB4x0QhNJaonYoVp1qM8H3cyZPV7tX1ocEzzvcUwUlaQbFQllr8StKLysGA1j/o1Y7yiXBneQrPS71rwDx5ibX+CVWOAGsQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723582078; c=relaxed/simple; bh=aQJok3gSjvHmXRiPrNrzugTOpERQ4gz+fm2rPvjru2I=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=j+2QlE4/qdwr22YkBaoQ/SJqfnjxqWS2OsTW7e0/AAjahdH3J56KrWzulp1E02LTj1ykmnrw7LLM40vilEP2glNNB7MmNusBywSMT5u1omDLH9XCogmTi2anhV6te8kjLFqmqEdtT41sJmn0WfoIXH9L1zVablLtXt/Jadh3W7E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--kinseyho.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=vsOcKd0s; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--kinseyho.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="vsOcKd0s" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-6818fa37eecso5845658a12.1 for ; Tue, 13 Aug 2024 13:47:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723582075; x=1724186875; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=BYWJhwEE6NYCkgyDLBhCsOHKcazNm9LYuzYyiRCtiIM=; b=vsOcKd0sLbpv/ij/D4qpGkGeeJcPivdojRl/gizzBRZCMS5pJVL4G9XfiVuX6ThhFX PyjGOtPs+9LsYUIsjw2yCwnVg4MzWzjL0+sBmPlGkLgGdDBsB8iapH/7Q4Zpl+2BEJ0m 6tTJR6EA+1YuS7zP6HakykyqKAxOi9P3sMQqtp46A10WrWCU9lGITSR/oS23oXWbxVEx 5TBSJiz7MmXXRW6ZJXVoKZGWIONA3rOGQtGTyPnf9uDEYxcJVCDqW27YSlKLRoY7MfQ7 9n+Y1Qr//iZiQpvjsS1V9DR65nscfmw7tNbyDJfY8d7R9Ytq1cHKog3E3m5mhHFPrQbL nN/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723582075; x=1724186875; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BYWJhwEE6NYCkgyDLBhCsOHKcazNm9LYuzYyiRCtiIM=; b=NSjAaDXegJ7HbdrUkrdvZR554IDkHLuGnD1JOsuhpwklEpMktCu0loAXo9RNJuGL8l WlkeL9RnVLWcbdF9t4BxBdFHZwYNy4NLn0/o4ZNXkPqRzdCFJmyVPkGQWlPYxEplUQL9 DauYc03WTrhoO0MQjKqRFvW+snFTKw23FwPHI+J6mkGhzUbch4fvWefhiBqsXW3jCPHc 8bXVveqXA5W06Z8fqL4AWayae2ZqwUaZFKfH052LpjF2rLyBGw4xF7GvSzCKhlFSviSh XXeJRYisv4dckL93F1dHNC7mnW5NHqE1dcVGKUXWg7M7jN3Zz/E2C99NxgL2I/chiW4y 5hEQ== X-Forwarded-Encrypted: i=1; AJvYcCWoRLSJOw9jZZR4yf0sEF/lL3lSCbj3UVoKVJ6sVhpTKAVkwFdX2TLEpX3KgjjJJwIA+S89Ga27s17wBh+wrEslj+TYe0DtvcPCP+q1 X-Gm-Message-State: AOJu0YyIF+4xZZ6wN56l61gERs3KQFujGQtTVUsfI9W1PjwGlF4uGSW2 PwBzAsV6n+Qh4AzIwHSvlKhcDQqUxq3XfmiDmjeSaRsOgDc7yO8rxN1HPoG0Zmqck6OI7vyRl7w tRQXammyhbg== X-Google-Smtp-Source: AGHT+IHxMBB1WIpSpQKruSLzaZb5mFQ79yQ49km4F0bFBxJKf7BgUc4XswgMUAKJQxmwnNey7FUrSuzdOv6Aag== X-Received: from kinseyct.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:46b]) (user=kinseyho job=sendgmr) by 2002:a63:114a:0:b0:694:4311:6eb4 with SMTP id 41be03b00d2f7-7c6a587fd3fmr1059a12.8.1723582074503; Tue, 13 Aug 2024 13:47:54 -0700 (PDT) Date: Tue, 13 Aug 2024 20:47:11 +0000 In-Reply-To: <20240813204716.842811-1-kinseyho@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240813204716.842811-1-kinseyho@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240813204716.842811-2-kinseyho@google.com> Subject: [PATCH mm-unstable v2 1/5] cgroup: clarify css sibling linkage is protected by cgroup_mutex or RCU From: Kinsey Ho To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Yosry Ahmed , Roman Gushchin , Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Tejun Heo , Zefan Li , mkoutny@suse.com, Kinsey Ho Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Explicitly document that css sibling/descendant linkage is protected by cgroup_mutex or RCU. Also, document in css_next_descendant_pre() and similar functions that is isn't necessary to hold a ref on @pos. The following changes in this patchset rely on this clarification for simplification in memcg iteration code. Suggested-By: Yosry Ahmed Signed-Off-By: Kinsey Ho Reviewed-by: Michal Koutn=C3=BD --- include/linux/cgroup-defs.h | 6 +++++- kernel/cgroup/cgroup.c | 16 +++++++++------- 2 files changed, 14 insertions(+), 8 deletions(-) diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h index 7fc2d0195f56..6862243bd1c2 100644 --- a/include/linux/cgroup-defs.h +++ b/include/linux/cgroup-defs.h @@ -172,7 +172,11 @@ struct cgroup_subsys_state { /* reference count - access via css_[try]get() and css_put() */ struct percpu_ref refcnt; =20 - /* siblings list anchored at the parent's ->children */ + /* + * siblings list anchored at the parent's ->children + * + * linkage is protected by cgroup_mutex or RCU + */ struct list_head sibling; struct list_head children; =20 diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index 0a97cb2ef124..ece2316e2bca 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -4602,8 +4602,9 @@ struct cgroup_subsys_state *css_next_child(struct cgr= oup_subsys_state *pos, * * While this function requires cgroup_mutex or RCU read locking, it * doesn't require the whole traversal to be contained in a single critical - * section. This function will return the correct next descendant as long - * as both @pos and @root are accessible and @pos is a descendant of @root. + * section. Additionally, it isn't necessary to hold onto a reference to @= pos. + * This function will return the correct next descendant as long as both @= pos + * and @root are accessible and @pos is a descendant of @root. * * If a subsystem synchronizes ->css_online() and the start of iteration, a * css which finished ->css_online() is guaranteed to be visible in the @@ -4651,8 +4652,9 @@ EXPORT_SYMBOL_GPL(css_next_descendant_pre); * * While this function requires cgroup_mutex or RCU read locking, it * doesn't require the whole traversal to be contained in a single critical - * section. This function will return the correct rightmost descendant as - * long as @pos is accessible. + * section. Additionally, it isn't necessary to hold onto a reference to @= pos. + * This function will return the correct rightmost descendant as long as @= pos + * is accessible. */ struct cgroup_subsys_state * css_rightmost_descendant(struct cgroup_subsys_state *pos) @@ -4696,9 +4698,9 @@ css_leftmost_descendant(struct cgroup_subsys_state *p= os) * * While this function requires cgroup_mutex or RCU read locking, it * doesn't require the whole traversal to be contained in a single critical - * section. This function will return the correct next descendant as long - * as both @pos and @cgroup are accessible and @pos is a descendant of - * @cgroup. + * section. Additionally, it isn't necessary to hold onto a reference to @= pos. + * This function will return the correct next descendant as long as both @= pos + * and @cgroup are accessible and @pos is a descendant of @cgroup. * * If a subsystem synchronizes ->css_online() and the start of iteration, a * css which finished ->css_online() is guaranteed to be visible in the --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 18:00:49 2026 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7ADA71A76A5 for ; Tue, 13 Aug 2024 20:47:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723582078; cv=none; b=ABQOH6xHTJxLpV85/b6eiHKeksBk0VUZPcNEL6FSbdEYHo+zD9TPQBErobntAf6lGm8VU2esc+Sma2CYMhf0LYkTbZvGcGaw4U77KwX904YB1OKmhpCXU5l9m2F3GnOqvh97BQYefOkr2DJxom0i5uiDX4JJJPSfuxCwIk7x8Z4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723582078; c=relaxed/simple; bh=KPTZtUl0BijCL5/vrgLCA/Byfa3nBjCHG6N9mdvcBjM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=bhYRwMjXk/d12v4Oo540Y8NNDZ4xugiVDB94cMXg215237bgm9dEhjzuGHfIG0wojB5mCPd+cdly9M/XYBJ0nMm4CtljVpxurtPoE1iRGzgXSE0+OzxqRa1EjwSd1rIm1zbeq36gyjLxBTRxuXCpQaKJPr9w0WofKFByXhLhGIw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--kinseyho.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=TosVy3aD; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--kinseyho.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="TosVy3aD" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e0be7c74d79so8776303276.0 for ; Tue, 13 Aug 2024 13:47:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723582076; x=1724186876; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hmvEk3aWaELcFNHaifCCcbwnogY+IyR2g5JVb5or6kw=; b=TosVy3aDbg3ag/3xrFjPACUDxbfsynvJqjyHB1MSN1nTpA7bXECTlrAwFkdmBkAGUz Xql1idM8Vrtxwwd5O+ZP5QxCyhYZ2pDfjSX1YuYyQjO96lS8LetxGZ/xBPt8cPXUa/qB pjS40boryFdK5D5uq9spvw0fUIXWzpg7rBKbI9k83236C7KRBThnNg4M6QrUocdnzCSQ Lfpdovh5GA8ifXDNs3WE/A/iRnUu75rLD71ssoC/MqgcgDNaqHIGXlWdlVAVR9/FZ3Cv VQggOBRZUfZ57rmsFrH0uu+tC/ucAhZsFFDF4ETmXf+lyAJ3b+qbfw+zVBoc/7hh4H0R WQYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723582076; x=1724186876; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hmvEk3aWaELcFNHaifCCcbwnogY+IyR2g5JVb5or6kw=; b=wQwyqCVTwXzKSzDKoyNU5MuVFBKJXnnv6RCVf2A7NuyLhByp+rcaE/pF17tSml5iD0 ygVNc41Cptbr7x7D9vo2hDx7FQZIx4Ggw03kstD4GozopvYDQ4vraaZAG9/MYFeW1cCb fcfQK/AL6128R97+WH7xNjW1GuBwdY5MJ2weGUjSqEsfCBWViMPWcuL9yfBWbvqIH6FW riMRiW/4UwCKxw1Dt9tefEYzjhR4e2Dats80RmtixOuj2Qs8VS6ANG6CZrASiaBbN0LB TVfAQPURMcEcqAo0OqRCkq2wytmoELV4sOFUdaKIjDIjhDpc7GeY1neLnc1iDigEkK7O 7Syw== X-Forwarded-Encrypted: i=1; AJvYcCVN61SW+TAufpWdcCPkjqlO2q3tdgyqX6sV8+mEJILoG1VpWAuFAjV7N2SmiGPSQnKCtENenkgXSZhnVWOy2lOVNMkpUQVM2Ms8lmq6 X-Gm-Message-State: AOJu0Yybb19U0UJanTDcQHZ+WUzhIyv6wCViNTC+2luFaH2SyYBSMMZh 3m/tUEdkZ6tAwShmMSaPUkA3SORNiCjy6xNi6xu7NEKHpBbhuBPGTTJfuGWc7Bhx54blIjKG7fk rJV8rWIgfsA== X-Google-Smtp-Source: AGHT+IHiDHCjFRbPR5+mkcQhIrf9lDLp4M5EcHeImCSP6xD1TY/GSvpgSst1fxEwRAEkFihKnMepRjSKepm4bA== X-Received: from kinseyct.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:46b]) (user=kinseyho job=sendgmr) by 2002:a25:c788:0:b0:e03:62dc:63de with SMTP id 3f1490d57ef6-e1155ada202mr39205276.6.1723582076373; Tue, 13 Aug 2024 13:47:56 -0700 (PDT) Date: Tue, 13 Aug 2024 20:47:12 +0000 In-Reply-To: <20240813204716.842811-1-kinseyho@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240813204716.842811-1-kinseyho@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240813204716.842811-3-kinseyho@google.com> Subject: [PATCH mm-unstable v2 2/5] mm: don't hold css->refcnt during traversal From: Kinsey Ho To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Yosry Ahmed , Roman Gushchin , Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Tejun Heo , Zefan Li , mkoutny@suse.com, Kinsey Ho Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To obtain the pointer to the next memcg position, mem_cgroup_iter() currently holds css->refcnt during memcg traversal only to put css->refcnt at the end of the routine. This isn't necessary as an rcu_read_lock is already held throughout the function. The use of the RCU read lock with css_next_descendant_pre() guarantees that sibling linkage is safe without holding a ref on the passed-in @css. Remove css->refcnt usage during traversal by leveraging RCU. Signed-off-by: Kinsey Ho --- include/linux/memcontrol.h | 2 +- mm/memcontrol.c | 18 +----------------- 2 files changed, 2 insertions(+), 18 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 90ecd2dbca06..1aaed2f1f6ae 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -75,7 +75,7 @@ struct lruvec_stats_percpu; struct lruvec_stats; =20 struct mem_cgroup_reclaim_iter { - struct mem_cgroup *position; + struct mem_cgroup __rcu *position; /* scan generation, increased every round-trip */ unsigned int generation; }; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index dacf4fec4541..1688aae3b1b4 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1052,20 +1052,7 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup= *root, else if (reclaim->generation !=3D iter->generation) goto out_unlock; =20 - while (1) { - pos =3D READ_ONCE(iter->position); - if (!pos || css_tryget(&pos->css)) - break; - /* - * css reference reached zero, so iter->position will - * be cleared by ->css_released. However, we should not - * rely on this happening soon, because ->css_released - * is called from a work queue, and by busy-waiting we - * might block it. So we clear iter->position right - * away. - */ - (void)cmpxchg(&iter->position, pos, NULL); - } + pos =3D rcu_dereference(iter->position); } else if (prev) { pos =3D prev; } @@ -1106,9 +1093,6 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup = *root, */ (void)cmpxchg(&iter->position, pos, memcg); =20 - if (pos) - css_put(&pos->css); - if (!memcg) iter->generation++; } --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 18:00:49 2026 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 41B811A7069 for ; Tue, 13 Aug 2024 20:47:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723582080; cv=none; b=bx5Kaf8ra5K2v8/lbkFuL62uqgvNYRT/joOi8o1KA9EsjvZT0Zpmnqw0Zda6jiu8mdRMPRZ/uitGVrfcnGxw7XXDn8pkEY01QcqT3vZOObSuZmLl+N08K+RFBn6fN/qBtzIn88v3apzgQJXpPPQJjFaP5It4QmqhDbMRkfPsmdk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723582080; c=relaxed/simple; bh=jUZ3GBO033wal/6XTVBPTFuNeWp5VVLyaFBWVIzsCNA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=CK4fQmzzhOqH4aeVbw1JHLcBFnNK/BWI5FjCIIgby/nRnzufIEGE8IXTgrEWvfBNnHZP6HZTnQYZRfNymK8npQlWtdEifIHK3m1vbq8c/xurL3s+gkGh/lTLIlu+20702KS0n6h2qXKErzkmfYXUFx2vruzvrT7dNqjtfJuX1X8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--kinseyho.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=sOmAAx+1; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--kinseyho.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="sOmAAx+1" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-6c8f99fef10so6730395a12.3 for ; Tue, 13 Aug 2024 13:47:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723582078; x=1724186878; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=7PkbmuKxOROTjGI4E3wXgUmmHtkXl9USrDaeWbBH7v4=; b=sOmAAx+1JAaLfKbZj5vBhe91KEoAQqTsCJ9F7LJqF1OtzySYtV7OkdWWFV/wOiKZfC idlpsvmGal55I8X2qeX4wZKsWcKpVP/WD09eqZ3Jb73RLaQ7QSoGoCM7p9NItle2jtyD 9JIEoeD6wpFgLRRyRcIRaRVbFyCQgr10J8POseXlBPNZ8Pr1SYpGKEXN0q5UN6j0pZNK DPyv0VcvUMwBiz752PYmldKjiuyuHF+p9XFhkwTpRMNapqTDAl90JxQpbGzdz+Ni398q OLPv3UflzPi/0U3ZFbvx5YJG1cEjcAVjljUfuCPyEBk2wBxHuBb2I290mMaIDEyMHpuR egsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723582078; x=1724186878; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7PkbmuKxOROTjGI4E3wXgUmmHtkXl9USrDaeWbBH7v4=; b=sia6M8rrCP8cTa6Ktr59Q9geGo0cMW7D62j/5sOFAUao8Vhyq9CMhS+ODDVvvlUPIC N62o+bxHwBOHxignpjgFSDt19Plk5FJty1bXf51ZLR0vKVZZt8lKw59/1b0zGL/00kgw S0CEXuZARKjUq/E7ju1q76nqbC62wxwZf2AYwpN7Heb2lzGcjJsm0/+PmCUgwuR8feNv vrUAD9iO1sR3z/a/vZeZD2OjprqWSyMYGjBcK5xCoIFmtArENbE9TIlBCHJPum3+x0Uc VffGM244o7rP2DMXKfSBN+wPXvBJ/dlo9VZ4fMH5Nn9M4TBiVSZElcum06MzD9jtmdFQ Y+xA== X-Forwarded-Encrypted: i=1; AJvYcCUjlynVFhcrS2Yg999NBD6zw/WuMzBpM9hksI1WgThb2MfUsQvlNcg9Ahd/TlpgjyWv3A7yRPDVJqQOai9djt2TnpeJYE7S5Lr5tNXm X-Gm-Message-State: AOJu0YySpMuQ/1wz/abi4++HXEYpj/bGX1/BBTRtVNOAVJoDgV+TjhvM r2qGcTeHZa0QzmF+EwCk7G37pS7R9SI3WvePIdY0pkGPzCEBFX4kGj1br1BWuxkvLXvmo+IZzfK Bq5K11ABEdQ== X-Google-Smtp-Source: AGHT+IHAciPzqd2lPyRlBVSYjYEEfNedI8L6N+K0qntudc/ZA3AgPVxsZZlT71exmwJpvjWHZeV3opJ32aRJGw== X-Received: from kinseyct.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:46b]) (user=kinseyho job=sendgmr) by 2002:a05:6a02:58d:b0:6ea:a4f3:14a3 with SMTP id 41be03b00d2f7-7c6a56d0242mr754a12.4.1723582078235; Tue, 13 Aug 2024 13:47:58 -0700 (PDT) Date: Tue, 13 Aug 2024 20:47:13 +0000 In-Reply-To: <20240813204716.842811-1-kinseyho@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240813204716.842811-1-kinseyho@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240813204716.842811-4-kinseyho@google.com> Subject: [PATCH mm-unstable v2 3/5] mm: increment gen # before restarting traversal From: Kinsey Ho To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Yosry Ahmed , Roman Gushchin , Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Tejun Heo , Zefan Li , mkoutny@suse.com, Kinsey Ho Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The generation number in struct mem_cgroup_reclaim_iter should be incremented on every round-trip. Currently, it is possible for a concurrent reclaimer to jump in at the end of the hierarchy, causing a traversal restart (resetting the iteration position) without incrementing the generation number. By resetting the position without incrementing the generation, it's possible for another ongoing mem_cgroup_iter() thread to walk the tree twice. Move the traversal restart such that the generation number is incremented before the restart. Signed-off-by: Kinsey Ho --- mm/memcontrol.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1688aae3b1b4..937b7efc41ca 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1036,7 +1036,7 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup = *root, root =3D root_mem_cgroup; =20 rcu_read_lock(); - +restart: if (reclaim) { struct mem_cgroup_per_node *mz; =20 @@ -1063,14 +1063,6 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup= *root, for (;;) { css =3D css_next_descendant_pre(css, &root->css); if (!css) { - /* - * Reclaimers share the hierarchy walk, and a - * new one might jump in right at the end of - * the hierarchy - make sure they see at least - * one group and restart from the beginning. - */ - if (!prev) - continue; break; } =20 @@ -1093,8 +1085,18 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup= *root, */ (void)cmpxchg(&iter->position, pos, memcg); =20 - if (!memcg) + if (!memcg) { iter->generation++; + + /* + * Reclaimers share the hierarchy walk, and a + * new one might jump in right at the end of + * the hierarchy - make sure they see at least + * one group and restart from the beginning. + */ + if (!prev) + goto restart; + } } =20 out_unlock: --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 18:00:49 2026 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F41521A76D6 for ; Tue, 13 Aug 2024 20:48:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723582082; cv=none; b=TW34xhMKIPOyfykIbV5gtMqsjC53Gn3NsysgXFfiJA09cT+DiLhvqoy2yU08QbvIShzkK8MMoNU/rMaV3luofDA5TozRY5ksZuulHNxmBkQQVG/0npAvnMgoQzwlqZa4Krqav1ilWco5cffiWtfRUzxcRNz9eEgPCw6spcIqcSo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723582082; c=relaxed/simple; bh=C/e58YefSGLLp6zQAuNfJl8UoPSA34YdN5tmlENHTfc=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=mJmcZ+P+U5gwac5JZ+5khbITQ+a9EDnJX0cD9r2lPRfKgp9yJTlUVNpXGYudCm81+uaLaWDhlbl8/2QXrewJc2/8IZdnLJkWwzKquWZsO3+f7txWdxR9XEPziqYMmVZ9gc5MKBgxaVy+g2G1hkdSLtOao26lUDR1h8zcp7jNA5o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--kinseyho.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=nGWyvTN5; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--kinseyho.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="nGWyvTN5" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-721d20a0807so5906712a12.1 for ; Tue, 13 Aug 2024 13:48:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723582080; x=1724186880; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AL9cOogrL47hO93mBXXtaHX7JJ+ZXjoDFTtox3tfTp8=; b=nGWyvTN5wgmjv14RtGZlkBQhlYFXa7BoU7He3DSD1/Cmndzs88VamXDDt97z91vS3z ShszqkiPeFfMJkfSITMJFInj0T+bnujAB2w7t5v50vp4QMX068ymxrsyy6s6c8pf/2QO ZLD9HMUOje/V7bjdc6vFq3jV7sn0L/RCCYOAl53XeS5n0XFD/NCqPUWBDy7RecnRCmUS brcsXM+qMgaAywtBGDS2htBqh/SwVCEUAEa6JHGqUAwSez2UABOGxep/4LI+zf9dnKfh 22FCRBEfvdIAgrDXoO8E9js+taC7VNed6o0YNuiO6XmGh0lS+xuy4Wb8BduDiez7vgYO E+Gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723582080; x=1724186880; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AL9cOogrL47hO93mBXXtaHX7JJ+ZXjoDFTtox3tfTp8=; b=vVHGBIqYC58PRnYHi+z5h/WWGyFQC9vO1mnT7i3NGZ04DkE2BDE7+16jASg8jS/Xqa jB8C+5S/sxMyvk6Am03c42GrT97LoWZmbi/zCQsh9GB2GHc7caqLTdDSoF4ztfnXAAhw Ke0BAcl/8C9dYzjysRLW+N2HB7Bo+1ogEhl67vEAS4nuKa3jVJflqs5r1b8L/DvmUTXb QYvMl9Q3x/LIkuTZZPQg7LfC99phAkAF0yPLz9d2XwgxLBgxQMD24ULU/+mwcckmyLsA HMkYiYQkOQTtJbJ7WahXt8ljzr3t9h8tvguyJbfKsbnkJUdnFJk+xby6QZYf1gfnfd75 bGmw== X-Forwarded-Encrypted: i=1; AJvYcCVrmNpGLEVKOCeYYFAWh31nhEqrV+fiDs93jPPHTC8/Lv3ILS8DcRj+3btpMobZks6VQC2vPFmeYd0BDzrcNTfJ0qo+t2Zozoq54LGF X-Gm-Message-State: AOJu0YyMf/llgeWXdc/0m4SVcFKhpzUvcVAWF/aGTWxFIT3BJI+Kq34p soOmXt4HQ+BElws5YiorwvP6H57QIH/PJvVRx2djCIANMRJhbh2cPT+2YIuuAwplJRNPJLcO1xQ gq3dws3ksnA== X-Google-Smtp-Source: AGHT+IFYKAB9rOjPeQV1OvEWBj+/G0SON71UCsBpqeijGE33YYArHBwlllAogxZUus4RJ93ox/3eY4q8R0h4Eg== X-Received: from kinseyct.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:46b]) (user=kinseyho job=sendgmr) by 2002:a63:4045:0:b0:77d:336c:a857 with SMTP id 41be03b00d2f7-7c6a588a314mr766a12.11.1723582080284; Tue, 13 Aug 2024 13:48:00 -0700 (PDT) Date: Tue, 13 Aug 2024 20:47:14 +0000 In-Reply-To: <20240813204716.842811-1-kinseyho@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240813204716.842811-1-kinseyho@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240813204716.842811-5-kinseyho@google.com> Subject: [PATCH mm-unstable v2 4/5] mm: restart if multiple traversals raced From: Kinsey Ho To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Yosry Ahmed , Roman Gushchin , Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Tejun Heo , Zefan Li , mkoutny@suse.com, Kinsey Ho Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, if multiple reclaimers raced on the same position, the reclaimers which detect the race will still reclaim from the same memcg. Instead, the reclaimers which detect the race should move on to the next memcg in the hierarchy. So, in the case where multiple traversals race, jump back to the start of the mem_cgroup_iter() function to find the next memcg in the hierarchy to reclaim from. Signed-off-by: Kinsey Ho --- include/linux/memcontrol.h | 4 ++-- mm/memcontrol.c | 22 ++++++++++++++-------- 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 1aaed2f1f6ae..aada9ef3ca44 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -57,7 +57,7 @@ enum memcg_memory_event { =20 struct mem_cgroup_reclaim_cookie { pg_data_t *pgdat; - unsigned int generation; + int generation; }; =20 #ifdef CONFIG_MEMCG @@ -77,7 +77,7 @@ struct lruvec_stats; struct mem_cgroup_reclaim_iter { struct mem_cgroup __rcu *position; /* scan generation, increased every round-trip */ - unsigned int generation; + atomic_t generation; }; =20 /* diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 937b7efc41ca..84de46ece9a9 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1025,7 +1025,7 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup = *root, struct mem_cgroup_reclaim_cookie *reclaim) { struct mem_cgroup_reclaim_iter *iter; - struct cgroup_subsys_state *css =3D NULL; + struct cgroup_subsys_state *css; struct mem_cgroup *memcg =3D NULL; struct mem_cgroup *pos =3D NULL; =20 @@ -1038,18 +1038,20 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgrou= p *root, rcu_read_lock(); restart: if (reclaim) { + int gen; struct mem_cgroup_per_node *mz; =20 mz =3D root->nodeinfo[reclaim->pgdat->node_id]; iter =3D &mz->iter; + gen =3D atomic_read(&iter->generation); =20 /* * On start, join the current reclaim iteration cycle. * Exit when a concurrent walker completes it. */ if (!prev) - reclaim->generation =3D iter->generation; - else if (reclaim->generation !=3D iter->generation) + reclaim->generation =3D gen; + else if (reclaim->generation !=3D gen) goto out_unlock; =20 pos =3D rcu_dereference(iter->position); @@ -1057,8 +1059,7 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup = *root, pos =3D prev; } =20 - if (pos) - css =3D &pos->css; + css =3D pos ? &pos->css : NULL; =20 for (;;) { css =3D css_next_descendant_pre(css, &root->css); @@ -1072,21 +1073,26 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgrou= p *root, * and kicking, and don't take an extra reference. */ if (css =3D=3D &root->css || css_tryget(css)) { - memcg =3D mem_cgroup_from_css(css); break; } } =20 + memcg =3D mem_cgroup_from_css(css); + if (reclaim) { /* * The position could have already been updated by a competing * thread, so check that the value hasn't changed since we read * it to avoid reclaiming from the same cgroup twice. */ - (void)cmpxchg(&iter->position, pos, memcg); + if (cmpxchg(&iter->position, pos, memcg) !=3D pos) { + if (css && css !=3D &root->css) + css_put(css); + goto restart; + } =20 if (!memcg) { - iter->generation++; + atomic_inc(&iter->generation); =20 /* * Reclaimers share the hierarchy walk, and a --=20 2.46.0.76.ge559c4bf1a-goog From nobody Sat Feb 7 18:00:49 2026 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D0B701AAE1E for ; Tue, 13 Aug 2024 20:48:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723582084; cv=none; b=Im5COIjpjBKb2ER3B4wny2F9lqmy+vOuRiM+vr0gNElvZOYmO8WF3PM1Nma8/oPAVeaC+N7BhIGjIEOWWS7140pXcWHPU2gZmrPxjb2ES428tN0pwKzOKKI9w4F31gA0s4gYTYbAflmMLZGuQjB4HS5w5fQdDHwblhj+cP7ajXU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723582084; c=relaxed/simple; bh=4J0w/9yc2JVD8H+ljNSMIrZxux6P1Zepecw3a7ltVZI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=VDQG4E0r41RD4z15pVdBRI2132x7caIRs6aTIXLUcYL4iaUVPWGx0LTWEr9AXU+vJfcXqteBK+HduBmpzjE8ZCFpU/pJuYeKWqp8gvxPvZIdXkTAlPEt1zU+SUH6ZczG8TP+r64DU6Zp5w5vJkmuYrLkd5vjzjWfTinesaO0Wb4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--kinseyho.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ldb639jd; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--kinseyho.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ldb639jd" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-68d1d966ef7so128342247b3.1 for ; Tue, 13 Aug 2024 13:48:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723582082; x=1724186882; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IUGrqgCsQ93s1mn5jRCuzR3kYh6lZBiPS4o7+rn6SUc=; b=ldb639jdIF1THzAgJ3aocsbuVk63GxEKy0UfE698W9aiSV63f5kuyrR28V7bzJZPfB tM9GpAzn+RJKifC9nhVOHL0uuFTIkOWDyndBxdLrI1UqiuzfPdPb7v1SLawEvJ77LLU6 lt17lGXczQjErGVXCikm8/ahNs5c+7Mi2N45dXF2HzV5rxOvn6T/qTa1RRxPTix8rcLD ffTb+AvPvp111IfwdMHbqp1MleuIEVV/p1r9iEG8gdOYoxFUhEMbZOq0jUvDM4wpWPq7 wGMqhGo9dd40TqZBZrbaxG0YUIIa/3ARUZNW3WwXovLu718S0MxWdS8sn4Pb3QDf+Ims 8YlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723582082; x=1724186882; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IUGrqgCsQ93s1mn5jRCuzR3kYh6lZBiPS4o7+rn6SUc=; b=dYPx1GkLVSpyhPNAZ3PJNygmefc/HJBblVgy1FBaubhrgv1h2ewafag0LjefQ/tBNc YW29EpvciEOEu4A+k/c1cDiMpruXm3hPKqtCCbsE0tuDB61eB+1/eLoYp9NrTbCP7d8m FXtlUhHHrhMr3fIOBQHbG6CGRBiWU1J5QShz1NdqanL4EErqgRQGdlvkMLCOn0UO2N3U nfmezoHhoOhx9vmt1CjLNiewbUVdz6MYoBViTaO/8NvDRBppsixQePt1jTsbogJ4JqdQ 7hU3N7I5BbF37mJ3+rej75zRr9ieG1PL9yusctYIarzY3mGyNyw2Dx3/jkXe8JtLGs8q hhVw== X-Forwarded-Encrypted: i=1; AJvYcCX82sfAGBVqMx5LwZGSEbdRPQwS0ARfELa3+K/qq0XMUXGGXg+/pl9ZL+tjUeFO/76nuUgvu5XZgL4dWdDz2Ucd5s31aCkHA6AByAzr X-Gm-Message-State: AOJu0Ywjn3cYVzK44K1uR3dLTAJ9ihLBkLd1B1Tm63QhWhG7zCV4GDzR nBi6nc5OQ7hOdUohIH2A3YqQ43ySaVRqP1z2BiNBaV8Y3il/o/RRJH6YFEaFBwC8f3DldlEoVjB z3ytOqNdJFw== X-Google-Smtp-Source: AGHT+IFKq4mle76EHVxRvn3g1KvqjdbIlRdYAcckM9mm8INZMPhwqmIEIFFgUqblYHs8iR9k3Dh891L/+5f5YA== X-Received: from kinseyct.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:46b]) (user=kinseyho job=sendgmr) by 2002:a05:6902:18d:b0:e03:589b:cbe1 with SMTP id 3f1490d57ef6-e1155b67c75mr35668276.7.1723582081844; Tue, 13 Aug 2024 13:48:01 -0700 (PDT) Date: Tue, 13 Aug 2024 20:47:15 +0000 In-Reply-To: <20240813204716.842811-1-kinseyho@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240813204716.842811-1-kinseyho@google.com> X-Mailer: git-send-email 2.46.0.76.ge559c4bf1a-goog Message-ID: <20240813204716.842811-6-kinseyho@google.com> Subject: [PATCH mm-unstable v2 5/5] mm: clean up mem_cgroup_iter() From: Kinsey Ho To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, Yosry Ahmed , Roman Gushchin , Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Tejun Heo , Zefan Li , mkoutny@suse.com, Kinsey Ho Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A clean up to make variable names more clear and to improve code readabilit= y. No functional change. Signed-off-by: Kinsey Ho --- mm/memcontrol.c | 30 +++++++++++------------------- 1 file changed, 11 insertions(+), 19 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 84de46ece9a9..87a0dc9d779a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1026,8 +1026,8 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup = *root, { struct mem_cgroup_reclaim_iter *iter; struct cgroup_subsys_state *css; - struct mem_cgroup *memcg =3D NULL; - struct mem_cgroup *pos =3D NULL; + struct mem_cgroup *pos; + struct mem_cgroup *next =3D NULL; =20 if (mem_cgroup_disabled()) return NULL; @@ -1039,10 +1039,9 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup= *root, restart: if (reclaim) { int gen; - struct mem_cgroup_per_node *mz; + int nid =3D reclaim->pgdat->node_id; =20 - mz =3D root->nodeinfo[reclaim->pgdat->node_id]; - iter =3D &mz->iter; + iter =3D &root->nodeinfo[nid]->iter; gen =3D atomic_read(&iter->generation); =20 /* @@ -1055,29 +1054,22 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgrou= p *root, goto out_unlock; =20 pos =3D rcu_dereference(iter->position); - } else if (prev) { + } else pos =3D prev; - } =20 css =3D pos ? &pos->css : NULL; =20 - for (;;) { - css =3D css_next_descendant_pre(css, &root->css); - if (!css) { - break; - } - + while ((css =3D css_next_descendant_pre(css, &root->css))) { /* * Verify the css and acquire a reference. The root * is provided by the caller, so we know it's alive * and kicking, and don't take an extra reference. */ - if (css =3D=3D &root->css || css_tryget(css)) { + if (css =3D=3D &root->css || css_tryget(css)) break; - } } =20 - memcg =3D mem_cgroup_from_css(css); + next =3D mem_cgroup_from_css(css); =20 if (reclaim) { /* @@ -1085,13 +1077,13 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgrou= p *root, * thread, so check that the value hasn't changed since we read * it to avoid reclaiming from the same cgroup twice. */ - if (cmpxchg(&iter->position, pos, memcg) !=3D pos) { + if (cmpxchg(&iter->position, pos, next) !=3D pos) { if (css && css !=3D &root->css) css_put(css); goto restart; } =20 - if (!memcg) { + if (!next) { atomic_inc(&iter->generation); =20 /* @@ -1110,7 +1102,7 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup = *root, if (prev && prev !=3D root) css_put(&prev->css); =20 - return memcg; + return next; } =20 /** --=20 2.46.0.76.ge559c4bf1a-goog