From nobody Wed Dec 17 02:58:19 2025 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 66EAA16A95E for ; Wed, 24 Jul 2024 19:02:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721847742; cv=none; b=QY/tQXFFF07zrwKlhGJ7IWrRL0XdTggbd23eb1wAmEMuENsdKz9l4e3s5Sp+HyD4zVr9EAFzy1gekRQ3c7zN290m7bIlqjeuJhA09A5hajrHZgbfFqGSrZJK3tUwN4nnMrT6964SNcasXMOqOPzH15CcSk/iipt7Ci4V/SL7udw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721847742; c=relaxed/simple; bh=dnIpnYFNL7tcZIOMfeqJiwj7BfmA8twPI/chQqFF0z0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=bkEzVhtyTgTq3ymwlQyemQAs0KDZGyj+jJpl+oZUj65OSrxEluZqkKa9eo1q5yk7AEYrLD7A2P1vnxhc21RjW9v/3bHKHQ14Rqppy4i0WCR9muL9L6Fni1hly0I7vYEQWu3dicX1hi02DJLiPhBZiHKs1e/BJkcATGPaQYUlGh8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--kinseyho.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=qIXZx/sC; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--kinseyho.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="qIXZx/sC" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-7908626a4a6so1129814a12.1 for ; Wed, 24 Jul 2024 12:02:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1721847740; x=1722452540; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=iLDZhQ0id4uDXJarhBRTOmxWLKOk3mwU62F8HwEfdow=; b=qIXZx/sCrPq2bQDUPrsl7BdTS28VE3ZfmFF1KDt3gnTIg8EVVTy41oCGqKV93s1gqK M2DM/EJnsK4kU4rlq7qnkiywnl7l3xwnZTK7PgSkygXfEJSTME5FE+kCmwjkYwr81WbK aChG5D+R/ti4ajsha7PwlOn5r0qlQ0+PnMbCuVLAt5FZVLwI62AHcZUq3poob/nd5rqU 31wpGct5x8aWXT6+tfqdZQKwARa3LxmdI4fSMRzkA1jEEg3fouW3bBmYRZtP1dfpQ9Ue Kl9YnNf/RvN37rar8jFVBLuUNLZI+YzYP+8GEko9wlmGZnsQrTkHkif6aufEAubvEXq7 k7mA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721847740; x=1722452540; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iLDZhQ0id4uDXJarhBRTOmxWLKOk3mwU62F8HwEfdow=; b=nYxaQpdkkEBueROQfSLiNPAPs1fH0zvChnZyfk7Y5TDPpp3UtPYQVXupy8iFQ1ZFQM 5gVimbDxUJ2RpI+ZeqQpCv7ghn8opM63l59MBc01dnzG/1a0hMGPfyjYfAoePkg2q21w 91eLslnHCkaLNoasC8jexxk+5ss/AM3lnXBwGhGY1XTLCj91IrEkIm01IXeVn+me7/uI /j7KKvU9TlkM92vs6lwX5UcNoIZB0E65bEvUNdjRifYHTSkcD3PmE5WEhTB4Q1TvRC21 Wq4f4eLq7y+5NkvZNxisLIRpvuhFWFxk39ZGSkYJY7qaH/Ij5jLXbMLtflzZ1M38qJWE /g8Q== X-Forwarded-Encrypted: i=1; AJvYcCW1bCMKw61MSY52fUFye3bq6wyKnTK2L82ATuzmNmWTQYFQh2p/VGWBebv6oXYmIYe/m/mdl1fQRwUbl+/vHbG6Sp+VC7ba5RNGrRoU X-Gm-Message-State: AOJu0YzEwyoztnfsbOZ0nSJrZaOIj+qlKrk3p8KRbtpEMmM3VP5FncdF qZ3CRZT8X/dcLA3MXiFMt6m7z/fDMsQcCCtIUNNHI32u2ktZyPPG3QbOkim1N+l8CIJy8We6Gma vK7KVpBNNnQ== X-Google-Smtp-Source: AGHT+IE2wliyUgg36eRbrdQG0eW2t+LBhnewXuB2szeGvK9iDIHirGHs2LzRSu2WcnyoQhiliA2iavEfWfM/Kg== X-Received: from kinseyct.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:46b]) (user=kinseyho job=sendgmr) by 2002:a17:90a:6b0a:b0:2c8:632:7efe with SMTP id 98e67ed59e1d1-2cdb9661b65mr32367a91.4.1721847739514; Wed, 24 Jul 2024 12:02:19 -0700 (PDT) Date: Wed, 24 Jul 2024 19:02:11 +0000 In-Reply-To: <20240724190214.1108049-1-kinseyho@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240724190214.1108049-1-kinseyho@google.com> X-Mailer: git-send-email 2.45.2.1089.g2a221341d9-goog Message-ID: <20240724190214.1108049-2-kinseyho@google.com> Subject: [PATCH mm-unstable v1 1/4] mm: don't hold css->refcnt during traversal From: Kinsey Ho To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Roman Gushchin , Kinsey Ho Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" To obtain the pointer to the saved memcg position, mem_cgroup_iter() currently holds css->refcnt during memcg traversal only to put css->refcnt at the end of the routine. This isn't necessary as an rcu_read_lock is already held throughout the function. Remove css->refcnt usage during traversal by leveraging RCU. Signed-off-by: Kinsey Ho Acked-by: Roman Gushchin --- include/linux/memcontrol.h | 2 +- mm/memcontrol.c | 18 +----------------- 2 files changed, 2 insertions(+), 18 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 7e2eb091049a..4cbab85e2e56 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -75,7 +75,7 @@ struct lruvec_stats_percpu; struct lruvec_stats; =20 struct mem_cgroup_reclaim_iter { - struct mem_cgroup *position; + struct mem_cgroup __rcu *position; /* scan generation, increased every round-trip */ unsigned int generation; }; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 960371788687..062bfeee799c 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1019,20 +1019,7 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup= *root, else if (reclaim->generation !=3D iter->generation) goto out_unlock; =20 - while (1) { - pos =3D READ_ONCE(iter->position); - if (!pos || css_tryget(&pos->css)) - break; - /* - * css reference reached zero, so iter->position will - * be cleared by ->css_released. However, we should not - * rely on this happening soon, because ->css_released - * is called from a work queue, and by busy-waiting we - * might block it. So we clear iter->position right - * away. - */ - (void)cmpxchg(&iter->position, pos, NULL); - } + pos =3D rcu_dereference(iter->position); } else if (prev) { pos =3D prev; } @@ -1073,9 +1060,6 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup = *root, */ (void)cmpxchg(&iter->position, pos, memcg); =20 - if (pos) - css_put(&pos->css); - if (!memcg) iter->generation++; } --=20 2.45.2.1089.g2a221341d9-goog From nobody Wed Dec 17 02:58:19 2025 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7DA4D16C68C for ; Wed, 24 Jul 2024 19:02:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721847744; cv=none; b=ONRVtl4rKHv13PXOzp30QI15za4yZtNKRvvCj2d9sl+7QW+oKhOSAUNzdqpfAQ4TvBxDduykrfBE0ayoCuwBy8BJtKx7M4eogFmkwaeO+YvDdKberCbHpUiZ+n8X0vd4lIc0HT5BYRq25tlM6rrcGERTDgNUsIEIsPQR7hDqq/Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721847744; c=relaxed/simple; bh=OFk5wDl8xpbdEo44lplWr1p5F7ojENQPq4VGlH1jB9o=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=SVaL3DkLUQAeEJd5OyhbYQHXZSvDyqFQ6PhZsO9U4Y27/H1fhoi6TOKB7ctw470MTI46pOjTBvVBGEZPQJCNooisJ2JXTGpyI2ssjYhQcgCE4tIAquKO7fcmlXhG9GdE8ipI0p93rbRYfNfqvR0mRp/MXOkLGSm2ZQYgYpwk/aU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--kinseyho.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=XXcCH6fj; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--kinseyho.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="XXcCH6fj" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-72c1d0fafb3so105817a12.2 for ; Wed, 24 Jul 2024 12:02:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1721847743; x=1722452543; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=AmGz1OUSAYzLu3Mkr9WfTC/ktWFGonkMAhMj+nbyydo=; b=XXcCH6fjno4piANDvv4EwU8SJ6HnglWRFS/PtVQpCDpj6Zpld5fLafQYEpo1YroAnf pjCIr/iyqhFvwvzrR+y9iP0vYtyW/cTY3TYP5yDRnMLZgET0BzBTzTwL0VirxUByu6Y/ wf2lX/XMo/cOPDRXqxYTCw9KqAUmoK/Yu99UE5qkfojeyZ+1BtnYwzHtsnOvP/oCQiHa PRAss8hUyAyYEYArLE+9LfNhMBa67q/TT73UgtYh25nH5gqYf8pQIca3S2B/a4KnFSO3 8hSnnfYMelW6/bvIZcvDvwdiKORBkKppjmK4VwrA2BUpTG+edvB5DYJV+8pl5A7OWjOP WvDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721847743; x=1722452543; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=AmGz1OUSAYzLu3Mkr9WfTC/ktWFGonkMAhMj+nbyydo=; b=IgPXO+gKV9IWkP7QqECps/NuFoNQWBGkG3vJEn0B5zPY4GQ+frEvB36xUr4rS/BcIM eBirpuAFJDnlWgWnIqDQ8gfRl2jKUm0AnMw20CFkGglIz2GSs0O1rJzplPSrIq4uJFuv 9tHWIfIx+BO8UV32v3cjwgKUQaAVMjdUcsoMut/rwXQhc8czadth5mSVNei4X2uZ1aK2 Odgt+75WduFFZLeOa2rnv3LLrB+s18GzIQ3K2KwA/K/gJ5B+RC6jWwub3Syud6OOjJ3F rCP9VYtiQE8hFnX9PJlvMsX0mD3M2koV9tUN4dxxHNcYofv5PIHVyOSlf2CAT9XEZZWo BSwg== X-Forwarded-Encrypted: i=1; AJvYcCXdcJuF5BK87MY0PQSBIlS2shIBpzBBPwICeQaQ2LN0Amuw64c176OutiTXNW4GtephoYWbCo2hgaPSht71+hcE7BlEz0UtF8lSM7U6 X-Gm-Message-State: AOJu0YyirNxcO+jcTnXksSfI9drjwHl/kMELXjtAdehfwkLx9HGzy/3C ljfeeeml0S0TJ4gRxO0frZovieO0+IEYkTvALb0KhbNMq+AHi4xbVU55fWjYMA00Pqid5OHEFKy LKBA+plfQpw== X-Google-Smtp-Source: AGHT+IGLvyDW6X3i6jqfX6m7Tz2BzQtjRTiuAHBzI1w6qDTToawljkbGNllzBAuC3KsBrh2qsLZDEguaiP99GA== X-Received: from kinseyct.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:46b]) (user=kinseyho job=sendgmr) by 2002:a17:903:1105:b0:1fb:27bd:82e2 with SMTP id d9443c01a7336-1fed38ccc31mr9895ad.8.1721847742363; Wed, 24 Jul 2024 12:02:22 -0700 (PDT) Date: Wed, 24 Jul 2024 19:02:12 +0000 In-Reply-To: <20240724190214.1108049-1-kinseyho@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240724190214.1108049-1-kinseyho@google.com> X-Mailer: git-send-email 2.45.2.1089.g2a221341d9-goog Message-ID: <20240724190214.1108049-3-kinseyho@google.com> Subject: [PATCH mm-unstable v1 2/4] mm: increment gen # before restarting traversal From: Kinsey Ho To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Roman Gushchin , Kinsey Ho Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The generation number in struct mem_cgroup_reclaim_iter should be incremented on every round-trip. Currently, it is possible for a concurrent reclaimer to jump in at the end of the hierarchy, causing a traversal restart (resetting the iteration position) without incrementing the generation number. Move the traversal restart such that the generation number is incremented before the restart. Signed-off-by: Kinsey Ho Acked-by: Johannes Weiner Acked-by: Roman Gushchin --- mm/memcontrol.c | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 062bfeee799c..f672bc47c6b5 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1003,7 +1003,7 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup = *root, root =3D root_mem_cgroup; =20 rcu_read_lock(); - +restart: if (reclaim) { struct mem_cgroup_per_node *mz; =20 @@ -1030,14 +1030,6 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup= *root, for (;;) { css =3D css_next_descendant_pre(css, &root->css); if (!css) { - /* - * Reclaimers share the hierarchy walk, and a - * new one might jump in right at the end of - * the hierarchy - make sure they see at least - * one group and restart from the beginning. - */ - if (!prev) - continue; break; } =20 @@ -1060,8 +1052,18 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup= *root, */ (void)cmpxchg(&iter->position, pos, memcg); =20 - if (!memcg) + if (!memcg) { iter->generation++; + + /* + * Reclaimers share the hierarchy walk, and a + * new one might jump in right at the end of + * the hierarchy - make sure they see at least + * one group and restart from the beginning. + */ + if (!prev) + goto restart; + } } =20 out_unlock: --=20 2.45.2.1089.g2a221341d9-goog From nobody Wed Dec 17 02:58:19 2025 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24BC016C6AD for ; Wed, 24 Jul 2024 19:02:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721847746; cv=none; b=oMXoyZVnTGzOq6yA8NcrlXyCUX7txGBafnIm/IieOGuvKzRSF0QhorGA0+dYZatvDl0ojwrHBACo70hLlpPJkYu2LOMsSQB1I+vSfI9Tb5dp+a4doxkBkfa1VURm0Vf2X7MrtDapIbFT9uSuL1eGzPx2WTto2Jkldzq+kvJS6W8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721847746; c=relaxed/simple; bh=sTj77cJUVCVZkNczDF9RCrE7mv7tSFEGITNZCht7iBM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=H5wdlXrD9UdhGfENOyWo382DedvU3eqOdwHAC/5tFv0A7c/TG/PUSFu2T9ej9S1AG++7Xwr1yH+832rpAy5H4Y/ECJOArLlIB7JDVx4/x18DvQmBkPQPGLDQY/mfWBSXIQyVCrB7o7xOrt/NivjQ2M3geBpGs6dyQdPtHjk03b0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--kinseyho.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ao+bXdE4; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--kinseyho.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ao+bXdE4" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-1fd5fe96cfeso1011595ad.0 for ; Wed, 24 Jul 2024 12:02:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1721847744; x=1722452544; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/TQiG1UXUGUFW+lVBdf0Q/W1aoqYwGoLbWZS7cahHzw=; b=ao+bXdE4Gx09INNpC6JaNknFPgIcqgc+zUagBuZwHxHj94+0S4Pts0DI9iCVWSHs0w etgj/0V4vA6lTCJEU35kb1vzXV+12um1PsjlYipqLvZiVboOQz8UyyMX95XMM4WBIYV+ nmsxshAV69Gn5phyIGk24wqA/Bi4S/Ii0rYzs8oL/q72JvmSdAfrM7E53PVGKnrWS0vB +3wDrvWy83199Z1tIycSGF8/+tpSJkC4eeNtunLR8MQ0hNCoafLXDowqAkPK09W8XWDD pt+qDlOp/+F75mzCA1TkmPCUiZ4P1h/e9/tog6k4qNuYdIWbmYOOrVpIDq2a75tKCUfo FGQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721847744; x=1722452544; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/TQiG1UXUGUFW+lVBdf0Q/W1aoqYwGoLbWZS7cahHzw=; b=w2ZpmLcC93UanjzTs8R0cUgrhJY1NZhKLZoGOecpHSj/CjjkE8jsriuDtXjQBwDCeF +lKe0MR4NCqv9KqKh9qq4vpzr49J/1/nbyJ6TL6N5nvVPssQLXegYd4W9AJV05F2MT6O SQPlXLFIdTV3gJey4eYArKyS395n+1uRH+V/SDuHIEbjlKaPw5F2LGH6FSqOwcP2QDzH YFlVlrsFnjDbq4xM3qUwMV356XNkNpWl2QPEKN8xeFb8fCbn7SCd3sp5Qw+T4OyLrBEg CWcMcoYLkQ4P6Ww1gljwgX9Vd6DpTTipeHxjNAPtq0d6QngWFbZWkCmWZ4AnKrZfmJj1 d4vw== X-Forwarded-Encrypted: i=1; AJvYcCXe/PmY+htO+rP+g8FUZLuefyxtQ2cP+d0CRfj9nuC81d2oTnI/XCvYhm0Kan84ZS1IpBr5/b7gvpCIM1OK5iQSmtboG4dN2zXSfYzs X-Gm-Message-State: AOJu0YyOHsHolF/OYC6lQzu+W6THdv/YKNiSPdmLR/A+Vz3TB01BSbg/ nC1I8+QJVdN43WnAYTJQsehj0fJSuVEAf4mAnqocKqH+2x0RuZ1vynLTY/WpHN0ejtbDiNUzqtg Pw4+e8pP8IQ== X-Google-Smtp-Source: AGHT+IE+tnQU31Bbj4dJVy+3PQYbdu9GD4nWCJk68x0wq+h9pms1Ho5g0zih9cGcyJ9eyRiAWRuy7HbM8eN7gg== X-Received: from kinseyct.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:46b]) (user=kinseyho job=sendgmr) by 2002:a17:903:24d:b0:1fc:369b:c1b5 with SMTP id d9443c01a7336-1fed3531a98mr17415ad.5.1721847744302; Wed, 24 Jul 2024 12:02:24 -0700 (PDT) Date: Wed, 24 Jul 2024 19:02:13 +0000 In-Reply-To: <20240724190214.1108049-1-kinseyho@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240724190214.1108049-1-kinseyho@google.com> X-Mailer: git-send-email 2.45.2.1089.g2a221341d9-goog Message-ID: <20240724190214.1108049-4-kinseyho@google.com> Subject: [PATCH mm-unstable v1 3/4] mm: restart if multiple traversals raced From: Kinsey Ho To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Roman Gushchin , Kinsey Ho Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, if multiple reclaimers raced on the same position, the reclaimers which detect the race will still reclaim from the same memcg. Instead, the reclaimers which detect the race should move on to the next memcg in the hierarchy. So, in the case where multiple traversals race, jump back to the start of the mem_cgroup_iter() function to find the next memcg in the hierarchy to reclaim from. Signed-off-by: Kinsey Ho Acked-by: Roman Gushchin --- include/linux/memcontrol.h | 4 ++-- mm/memcontrol.c | 14 ++++++++++---- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 4cbab85e2e56..2b354abe6d48 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -57,7 +57,7 @@ enum memcg_memory_event { =20 struct mem_cgroup_reclaim_cookie { pg_data_t *pgdat; - unsigned int generation; + int generation; }; =20 #ifdef CONFIG_MEMCG @@ -77,7 +77,7 @@ struct lruvec_stats; struct mem_cgroup_reclaim_iter { struct mem_cgroup __rcu *position; /* scan generation, increased every round-trip */ - unsigned int generation; + atomic_t generation; }; =20 /* diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f672bc47c6b5..4314a2b8848d 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1005,18 +1005,20 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgrou= p *root, rcu_read_lock(); restart: if (reclaim) { + int gen; struct mem_cgroup_per_node *mz; =20 mz =3D root->nodeinfo[reclaim->pgdat->node_id]; iter =3D &mz->iter; + gen =3D atomic_read(&iter->generation); =20 /* * On start, join the current reclaim iteration cycle. * Exit when a concurrent walker completes it. */ if (!prev) - reclaim->generation =3D iter->generation; - else if (reclaim->generation !=3D iter->generation) + reclaim->generation =3D gen; + else if (reclaim->generation !=3D gen) goto out_unlock; =20 pos =3D rcu_dereference(iter->position); @@ -1050,10 +1052,14 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgrou= p *root, * thread, so check that the value hasn't changed since we read * it to avoid reclaiming from the same cgroup twice. */ - (void)cmpxchg(&iter->position, pos, memcg); + if (cmpxchg(&iter->position, pos, memcg) !=3D pos) { + if (css && css !=3D &root->css) + css_put(css); + goto restart; + } =20 if (!memcg) { - iter->generation++; + atomic_inc(&iter->generation); =20 /* * Reclaimers share the hierarchy walk, and a --=20 2.45.2.1089.g2a221341d9-goog From nobody Wed Dec 17 02:58:19 2025 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EE91216D33A for ; Wed, 24 Jul 2024 19:02:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721847748; cv=none; b=pWttAIBuFh7bCSzxvzKpD1AW8/zblG+9JfbKNbZsx1LEMczmg7o0sUfb22dg4XbE3jc6WByZOSJQgkFb/t3zOSyy+vvBSCEyvMfBpP0jR2P5pyGoPVi2Uh74QATBAZL5leHmKdpJWhAu69EvAKkhZsDvpcqfJlpkpVlcON33Pjk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721847748; c=relaxed/simple; bh=rGVLNhAjSzvTNlgJoN+vRdGIm9OMfTJHAiadJXPXIHw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=f0Qh531StBff9Md9gNftgcPrKCsEpox2lDSmPW/vYmHNyIVRafHafdMrUtyK2GQcQZtSXsdKLmvnYOVouIq2h7pKe2/tbFh1mX8qTxr3AHR9T4URJ2w8sl1VcHOMnLL1rzwSAjPEe9MvX1zCL/iia/eLqix8oGj64f2YqDIQuVk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--kinseyho.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=FtZe9R8b; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--kinseyho.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="FtZe9R8b" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2cb50fbebd9so193459a91.0 for ; Wed, 24 Jul 2024 12:02:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1721847746; x=1722452546; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Ys2craRirgSkjM83RG964eoUiETMfTD1cND24tGgc4M=; b=FtZe9R8bIj6nd+rousnMGDbxAFpGSxrXJPJfBohrTQIod4ljLpf6c+3MyOnXSGyaPl JH1QcYBgbjs8ROSGNbcjbbpTuCDmVMJgtrXr5iwTITW7hgDPBW/TkuQ7daUTA0uR0JRL js0TnIFDl99nAYkgdiTtKpL8hWBxw3vYyARVwSRQeQA5pyfNIp2iVAy5CSW1CXzCzixz 3LE0VwTsWEvUN/VnZtTObKVAOJdtwKDBwqwaOti2OJI4pm6G/cXdIYbhB3oiFEd+6Uyj hc7yW6s/Q6ntS+MnuMpBMy4GMVyMH4bCEL8RAjbYMPO0PP43pnNfJ9urayF6dRkQH3zq 8e5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721847746; x=1722452546; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Ys2craRirgSkjM83RG964eoUiETMfTD1cND24tGgc4M=; b=IZahAxm/U4MECPXvdENWgUrNU2oCf9OoQbGKrUp8zo0/9ErrlltcX6rz0uTe9hkrSt Z5BAs0S/GHclCTlDJ2hgXn/x3sV5yZPRpkLMAwYfed3XznEQQBYfnhJLQ+4xrPJsZP+t J/fbxMEGwK6WM2uAZJUKz0ugnjE9kZZQAIisBRnS8HAoUWpf1pyGNIJA7CMRpT1NlYaA yP0+Ev8CildZiH0P8QINvZruK52a5lYOr7P0jImIuZk20/axGdT5Hr8rE1aW/MHWUd6f xD0+iwlqTdVT3vLydOCHqjtCUxHFjS6Hl2t4x+3SXaCVb7tyLxF3KscX3NyBbxOUMBzx JXYw== X-Forwarded-Encrypted: i=1; AJvYcCVTarGA8Nc7eG62C/XoN9utbm1m8Fbzc4TgKw6BqVkkVB0Xl0E7NaFyV/iljaqtAB9EhR6819JeC2Uchq3XObZzAiDnEBTRHN5uH8qw X-Gm-Message-State: AOJu0Yzx3kpvHRaflIFW6YGh3G6GRKgPQGA0MkFa9EaiHcOlf691/iio RC1Q2ec2E+k3/5e1BxJ7RfTP2iunXToEIsLtrPBqM1/jXi82mdm+f3JpFfaPxqkR9SpqvQLulgc o89rw0J78iA== X-Google-Smtp-Source: AGHT+IGYuWhGCshfz/1OfOrlFgcnBDPUyftLtmcQ4FX3/fi/oAJTe3cbfgo0sG0FCKIUMQtWQ+YJ7xLut+dAxw== X-Received: from kinseyct.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:46b]) (user=kinseyho job=sendgmr) by 2002:a17:90a:17ec:b0:2c9:9c25:7584 with SMTP id 98e67ed59e1d1-2cf23e1b73bmr1597a91.5.1721847746082; Wed, 24 Jul 2024 12:02:26 -0700 (PDT) Date: Wed, 24 Jul 2024 19:02:14 +0000 In-Reply-To: <20240724190214.1108049-1-kinseyho@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240724190214.1108049-1-kinseyho@google.com> X-Mailer: git-send-email 2.45.2.1089.g2a221341d9-goog Message-ID: <20240724190214.1108049-5-kinseyho@google.com> Subject: [PATCH mm-unstable v1 4/4] mm: clean up mem_cgroup_iter() From: Kinsey Ho To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Roman Gushchin , Kinsey Ho Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A clean up to make variable names more clear and to improve code readability. No functional change. Signed-off-by: Kinsey Ho Acked-by: Roman Gushchin --- mm/memcontrol.c | 36 ++++++++++++++---------------------- 1 file changed, 14 insertions(+), 22 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4314a2b8848d..7e3e95c62122 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -992,9 +992,9 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup *r= oot, struct mem_cgroup_reclaim_cookie *reclaim) { struct mem_cgroup_reclaim_iter *iter; - struct cgroup_subsys_state *css =3D NULL; - struct mem_cgroup *memcg =3D NULL; - struct mem_cgroup *pos =3D NULL; + struct cgroup_subsys_state *css; + struct mem_cgroup *pos; + struct mem_cgroup *next =3D NULL; =20 if (mem_cgroup_disabled()) return NULL; @@ -1006,10 +1006,9 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup= *root, restart: if (reclaim) { int gen; - struct mem_cgroup_per_node *mz; + int nid =3D reclaim->pgdat->node_id; =20 - mz =3D root->nodeinfo[reclaim->pgdat->node_id]; - iter =3D &mz->iter; + iter =3D &root->nodeinfo[nid]->iter; gen =3D atomic_read(&iter->generation); =20 /* @@ -1022,43 +1021,36 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgrou= p *root, goto out_unlock; =20 pos =3D rcu_dereference(iter->position); - } else if (prev) { + } else pos =3D prev; - } =20 - if (pos) - css =3D &pos->css; - - for (;;) { - css =3D css_next_descendant_pre(css, &root->css); - if (!css) { - break; - } + css =3D pos ? &pos->css : NULL; =20 + while ((css =3D css_next_descendant_pre(css, &root->css))) { /* * Verify the css and acquire a reference. The root * is provided by the caller, so we know it's alive * and kicking, and don't take an extra reference. */ - if (css =3D=3D &root->css || css_tryget(css)) { - memcg =3D mem_cgroup_from_css(css); + if (css =3D=3D &root->css || css_tryget(css)) break; - } } =20 + next =3D mem_cgroup_from_css(css); + if (reclaim) { /* * The position could have already been updated by a competing * thread, so check that the value hasn't changed since we read * it to avoid reclaiming from the same cgroup twice. */ - if (cmpxchg(&iter->position, pos, memcg) !=3D pos) { + if (cmpxchg(&iter->position, pos, next) !=3D pos) { if (css && css !=3D &root->css) css_put(css); goto restart; } =20 - if (!memcg) { + if (!next) { atomic_inc(&iter->generation); =20 /* @@ -1077,7 +1069,7 @@ struct mem_cgroup *mem_cgroup_iter(struct mem_cgroup = *root, if (prev && prev !=3D root) css_put(&prev->css); =20 - return memcg; + return next; } =20 /** --=20 2.45.2.1089.g2a221341d9-goog