From nobody Sun Feb 8 12:31:36 2026 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DACD22C0277 for ; Mon, 3 Nov 2025 07:53:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762156384; cv=none; b=FQaBraYEfRsTwfReRtlioGAfo+xRluq7o4Voz8gxeMR1p57XsSbUTKW/pYRAs5I9krgVL5vPHg1S2GszITkhKzu9UKN5yIO49rcm4fE0x0MGRMwScAgYMryqbFhQ3h/4dN4SELXg4FNvItGx4gNyCieUjEEzf5LWLxhk8UWpP2Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762156384; c=relaxed/simple; bh=e7ed8ZmJa/kqjlXsP+aE5zpUgSJSjGTZ/K4MPhvN53A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ioMoqEpLKwn7bnac56werBs/3adL/DvomBflwAPT670zdScTi1KN3WpXYl/lhlJZ0xERsTgGm3E8QUabfd4rP2l7+wubkkfl9sNxZrmVyjPVNEPQsnzm6DSrvlNZ0m/gYPEm8w6fHlPRiIYq02RNBpwb0ImNAE3ygXx3Ph9vnSc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=shopee.com; spf=pass smtp.mailfrom=shopee.com; dkim=pass (2048-bit key) header.d=shopee.com header.i=@shopee.com header.b=Z2OMwlnr; arc=none smtp.client-ip=209.85.216.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=shopee.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shopee.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shopee.com header.i=@shopee.com header.b="Z2OMwlnr" Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-340e525487eso822279a91.3 for ; Sun, 02 Nov 2025 23:53:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shopee.com; s=shopee.com; t=1762156380; x=1762761180; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MBJR74pL2UFLaHc8oAOxpJBTbxd/tOOTfgjPB+YfL2o=; b=Z2OMwlnrFpZPu1TtMLiVLFV8hnFN2kG30WbYGRbsESze970Nj7BxW0bhRs6j+6j3dX s7SAHDc/dy4Rv727nu5GHILrJbG1vxqPBu9eDJgt0k9iRvA6g0Fi5Qyved3MrPZ2GiFM FDUQ/q6QGjvOnMnH9RPTLD/j1s31ksoe5JgoQj0gCw82Ar2o6WnGSyuHchQkJuuyoYZX FPZPW1onvsN0zgphUJJLTGk1bAtxoyGI72yZgX7Ym1yWB4f3Kg3OMLpdNWKpmufdTnOn xOlzz0zm7VvQu1BBuqM0dquF2gk9nNXLOBhg7S3jfSquQXey0S4vrWSgMo+9ZRnpCOF7 rd1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762156380; x=1762761180; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MBJR74pL2UFLaHc8oAOxpJBTbxd/tOOTfgjPB+YfL2o=; b=Xr05efuo1loQAh7L5NjGjYRLp/Mzx87lNDGZttFxJdbelgdbpPnLFPcOZvN/TI4IMb 3yfuJX0sFPinJxerZM6B0e5ZfhEyC28BomBPtPMFze04oEL3LfkcOC+8nVqp9sYtGd0b yqIbzGbNpx1WjNd30W8FYfqxfrm/gpLKKHgrxqXm9YcAdiqKuN1h+KQ1CcsLJ4MvfNrr exmtoVcmHKl65pYLQImoW48vnX0bu5+r5vI0fh7BTJ1Bz2lEZHXUgA9yWNrCrkOi5MXQ MaWcvcIqBA6mxtrX2Jv3OUXUoadKFAKGgkNPSKdJHHp7cTYQ1dtgdmq26FdEXpLbNDQo cEKA== X-Forwarded-Encrypted: i=1; AJvYcCVPx6uY+bbXJkIrgRdtVE0R3BwWQHi6pgw+b7L1Y/xVkwRmIggvJ2fB5DmlG9s4FipPycyK2tyL3ZZGTCU=@vger.kernel.org X-Gm-Message-State: AOJu0Yyd5pqYOqNzNviauurWCLZnKISQPiIrZ3C9wg8VNm8vprivvUi2 8r6LegFeXLxZPybjzlO2o+C5kErD3EKjGxc6QmNNhFsHVOTaavBj2rvYAFMA6fqkTNc= X-Gm-Gg: ASbGncuYn7dr9FHNtWBpjE0MklD00MlisNKCfVVYfr/XznHM5OA1/4aXT8DZLTAPaJW WFpAXH1xU7Eo38aEDDlg1QG6vp1DHAFdsiot0baA7EWk8TFidli/412s8a7L8aZEIYB/gjC2OyE CcgxoS2B8J/vRKlsXDd9ySQ0Kjh8ZDYVUKAYSH2moK6Qv5OZCO3HvFafrl4gcmrErCw+s3lc/i0 fNGJYM3X/KHmJor6EqeC+EA1+b/Zyl3wFH4LNNNDBxbkJUAZ51hzNkyRT6n+XlzOaXWsdMVHM+V N8hyZoJn3Smy7ygZ9ZdskQsWS9aQ6x3B49PadmTycb37NvwrFQByauoPIezRcPZpClGDdJ/L0Sr XeLLXI/rpnQXawIR0u1YQTQ1m2OfU9GgcdjYztDARTqwfCwDQTvoEIJI4Bv7WI+FBCawONwExkM O2esQ0u8Jp5pNlhGbP+W3hma7t X-Google-Smtp-Source: AGHT+IHCeuWblU4UX1TsHgam4EVLA+G+yYi1frynG5wZVaDx/qokn9yyEJcHgqKW4FWQHlMPmkrGgA== X-Received: by 2002:a17:90a:d450:b0:340:a961:80c5 with SMTP id 98e67ed59e1d1-340a961873emr10029797a91.32.1762156380212; Sun, 02 Nov 2025 23:53:00 -0800 (PST) Received: from .shopee.com ([122.11.166.8]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-34159a16652sm34552a91.20.2025.11.02.23.52.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 02 Nov 2025 23:52:59 -0800 (PST) From: Leon Huang Fu To: stable@vger.kernel.org, greg@kroah.com Cc: tj@kernel.org, lizefan.x@bytedance.com, hannes@cmpxchg.org, corbet@lwn.net, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, muchun.song@linux.dev, akpm@linux-foundation.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com, lance.yang@linux.dev, leon.huangfu@shopee.com, shy828301@gmail.com, yosryahmed@google.com, sashal@kernel.org, vishal.moola@gmail.com, cerasuolodomenico@gmail.com, nphamcs@gmail.com, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Xin Hao , Michal Hocko , Muchun Song Subject: [PATCH 6.6.y 1/7] mm: memcg: add THP swap out info for anonymous reclaim Date: Mon, 3 Nov 2025 15:51:29 +0800 Message-ID: <20251103075135.20254-2-leon.huangfu@shopee.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20251103075135.20254-1-leon.huangfu@shopee.com> References: <20251103075135.20254-1-leon.huangfu@shopee.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Xin Hao [ Upstream commit 811244a501b967b00fecb1ae906d5dc6329c91e0 ] At present, we support per-memcg reclaim strategy, however we do not know the number of transparent huge pages being reclaimed, as we know the transparent huge pages need to be splited before reclaim them, and they will bring some performance bottleneck effect. for example, when two memcg (A & B) are doing reclaim for anonymous pages at same time, and 'A' memcg is reclaiming a large number of transparent huge pages, we can better analyze that the performance bottleneck will be caused by 'A' memcg. therefore, in order to better analyze such problems, there add THP swap out info for per-memcg. [akpm@linux-foundation.orgL fix swap_writepage_fs(), per Johannes] Link: https://lkml.kernel.org/r/20230913213343.GB48476@cmpxchg.org Link: https://lkml.kernel.org/r/20230913164938.16918-1-vernhao@tencent.com Signed-off-by: Xin Hao Suggested-by: Johannes Weiner Acked-by: Johannes Weiner Cc: Michal Hocko Cc: Roman Gushchin Cc: Shakeel Butt Cc: Muchun Song Signed-off-by: Andrew Morton Signed-off-by: Leon Huang Fu --- Documentation/admin-guide/cgroup-v2.rst | 9 +++++++++ mm/memcontrol.c | 2 ++ mm/page_io.c | 8 ++++---- mm/vmscan.c | 1 + 4 files changed, 16 insertions(+), 4 deletions(-) diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-= guide/cgroup-v2.rst index b26b5274eaaf..622a7f28db1f 100644 --- a/Documentation/admin-guide/cgroup-v2.rst +++ b/Documentation/admin-guide/cgroup-v2.rst @@ -1532,6 +1532,15 @@ PAGE_SIZE multiple when read back. collapsing an existing range of pages. This counter is not present when CONFIG_TRANSPARENT_HUGEPAGE is not set. + thp_swpout (npn) + Number of transparent hugepages which are swapout in one piece + without splitting. + + thp_swpout_fallback (npn) + Number of transparent hugepages which were split before swapout. + Usually because failed to allocate some continuous swap space + for the huge page. + memory.numa_stat A read-only nested-keyed file which exists on non-root cgroups. diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 2d2cada8a8a4..c61c90ea72a4 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -704,6 +704,8 @@ static const unsigned int memcg_vm_event_stat[] =3D { #ifdef CONFIG_TRANSPARENT_HUGEPAGE THP_FAULT_ALLOC, THP_COLLAPSE_ALLOC, + THP_SWPOUT, + THP_SWPOUT_FALLBACK, #endif }; diff --git a/mm/page_io.c b/mm/page_io.c index fe4c21af23f2..cb559ae324c6 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -208,8 +208,10 @@ int swap_writepage(struct page *page, struct writeback= _control *wbc) static inline void count_swpout_vm_event(struct folio *folio) { #ifdef CONFIG_TRANSPARENT_HUGEPAGE - if (unlikely(folio_test_pmd_mappable(folio))) + if (unlikely(folio_test_pmd_mappable(folio))) { + count_memcg_folio_events(folio, THP_SWPOUT, 1); count_vm_event(THP_SWPOUT); + } #endif count_vm_events(PSWPOUT, folio_nr_pages(folio)); } @@ -278,9 +280,6 @@ static void sio_write_complete(struct kiocb *iocb, long= ret) set_page_dirty(page); ClearPageReclaim(page); } - } else { - for (p =3D 0; p < sio->pages; p++) - count_swpout_vm_event(page_folio(sio->bvec[p].bv_page)); } for (p =3D 0; p < sio->pages; p++) @@ -296,6 +295,7 @@ static void swap_writepage_fs(struct page *page, struct= writeback_control *wbc) struct file *swap_file =3D sis->swap_file; loff_t pos =3D page_file_offset(page); + count_swpout_vm_event(page_folio(page)); set_page_writeback(page); unlock_page(page); if (wbc->swap_plug) diff --git a/mm/vmscan.c b/mm/vmscan.c index 258f5472f1e9..774bae2f54d7 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1922,6 +1922,7 @@ static unsigned int shrink_folio_list(struct list_hea= d *folio_list, folio_list)) goto activate_locked; #ifdef CONFIG_TRANSPARENT_HUGEPAGE + count_memcg_folio_events(folio, THP_SWPOUT_FALLBACK, 1); count_vm_event(THP_SWPOUT_FALLBACK); #endif if (!add_to_swap(folio)) -- 2.50.1 From nobody Sun Feb 8 12:31:36 2026 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6CD3F2C11C8 for ; Mon, 3 Nov 2025 07:53:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762156389; cv=none; b=LezOl0JJWIkpZKtKbdxaSo9wtfrrSrlCTNQ9sp68C5XETpzCRqh03PAbS9piGh/MuG5fLwyIuwa1unwr/REv9D+o6isD1LyFECds1pREsDq94Fd6R4by/vSAjxIjav4EsJ9+x3ijq0gAkvVnTvs/TDOBYERqBzNFTZ/bfKM0OAQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762156389; c=relaxed/simple; bh=XmQDlthpnpMTvEzwkFjfGJ/iooKFkhBQ7geY68RrYh0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jXqyZL4w5iTCs6vtp6XtbtEaO7Hg6FNDZW88hM+rizN7CC9veF3svA58jehC50sLfDAjabPkTghfibRg2s9n45hX4GQf/wwp1by5M6jV+zyBjmpGeVv+0vh0zgmGwj4sSbl0E5qRgkUJFQsSLauduIa+lK0F0B7wQ6Dkn+wxKYU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=shopee.com; spf=pass smtp.mailfrom=shopee.com; dkim=pass (2048-bit key) header.d=shopee.com header.i=@shopee.com header.b=isrfGv9n; arc=none smtp.client-ip=209.85.216.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=shopee.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shopee.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shopee.com header.i=@shopee.com header.b="isrfGv9n" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-3408c9a8147so2699425a91.0 for ; Sun, 02 Nov 2025 23:53:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shopee.com; s=shopee.com; t=1762156387; x=1762761187; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ky+X/IaBSB0OFL72v+WlIKaOZyGxRoEyNHwdneD4KGU=; b=isrfGv9nyKlGgMjPIsRq5ngvRJI0AdihmR8Z290majg8XvmW57J2zNjBaLgiHuQWR+ 5I0vrpuvZA8Em84RYkijNaU1CspYuNSYrkUtWaGVPBAylPIHo6J/C38/CWw/j2H/vD2F pk6u0rmwtF2n9MJMNXNnYGgf8xT01N092W8/V4P62n3TYPBt1Dh6kK1Msq+NHEVhFLp3 fQvimB0k/NOdiB2IegczFMQTw/LlOct5+NBLz9NgDg1EJttxqJCN+2p1YdxTrsf2pY+5 8P79nQemwIlQdl+dmCy1QCO9+hT3mH2et6n3iCqJ9yO/xnO0rV6BGSAt64F9Fsum30XS R2CQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762156387; x=1762761187; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ky+X/IaBSB0OFL72v+WlIKaOZyGxRoEyNHwdneD4KGU=; b=jsq0yvOUxcUKRX9XW7MndX+eXAFK5dNmlAt1MbCpu+Qvj6HovJQWZ/QGw8UEeWjjQu TpdCpgvsexa/EDs0qRTzjICZTOqcvAnOnWzbwEfvoQuDD4gyHMzS2kGtd7us5eO80FCF NnSDVxYs3kL+HvPQ5XsZ8/vB7r5nKkBHgCgrk9KR0TEDC650/8gmZ6xy7VKY/v/mae4D JzVsm+5X8D5B2bWm4t52anqZ5Zshrdn0MjtKTSgic+vQMVP60mfpqUX/wZ026i4g72N1 C7JQoL0xyZkOnQlsyPPABWEeh3mC3NXOXVfVK8iccEaBbdjMX6AjCxw3mNeBJ/x9+/MY 1blw== X-Forwarded-Encrypted: i=1; AJvYcCXBWUfH63yoMhuVm2Uu9HoK0mIRqYLDU6hZJ/09w18eiYRC3dQIFC1mszsm8fa5gdqLVPeWXXJfmqRVT7c=@vger.kernel.org X-Gm-Message-State: AOJu0YwyqBIadCNOA9BLIPBI8tBWKAfRB5LTNFjdGtv9IKWsrgpNT2oY WsnCuZ15T3NSDa7UBsb/zbULZ8aJRts8tEva9S+gSms9TIGP0jZ3Tr/TGpsfOnfj/Z4= X-Gm-Gg: ASbGncv1XPbauu3cmXUTYhq41rIqfy7qcpZ2OiMinlPbCMpmpknUMXGidvdPFe85eIW V0dIlXsyeQ5CzbHTBisdMhpXbg9JMDq5UYpuIuXA7pobpydEciRGau2fbiCEN5jdoAaO6/sJq4n thOjPkCixyIasRUAfZYVdOzMGjo4+91Xxfy33Dt3erXwJjepvLExDqiNfQa3Q4Tc6UDFdh0wxk+ fpoRnKjehZTkEcWGj1Jzt1oUk59vzAr93IAXUeZul+y1Oq21/mBDeixbNJtkIjg4dCrRPJ7KgA+ uuFfvCqCojykY3ublKeuawyG22WCu0zcjOzXBM6a4FbwV6T81Y28UhjrlRR5vXzsm+kvaGtTHmZ LorzGBWs2hZAOBcCdEuCy02bkSrNM1ChWbMgAkvV6SMhLTRKihctUBu8UoSXWQAJKkMC/lRdbDj aYaidIqHN0ncA4B3qKfhC7S38NGzXWybMmrY0= X-Google-Smtp-Source: AGHT+IEyO5cEySxDDsa5P6gc4+o8FeVcq4k+ABlDqETwxZQ5qkG9gowGljWIIRKONKOHt2sI5vETgQ== X-Received: by 2002:a17:90a:d410:b0:340:b572:3b81 with SMTP id 98e67ed59e1d1-340b5723cf2mr10885892a91.11.1762156386624; Sun, 02 Nov 2025 23:53:06 -0800 (PST) Received: from .shopee.com ([122.11.166.8]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-34159a16652sm34552a91.20.2025.11.02.23.53.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 02 Nov 2025 23:53:06 -0800 (PST) From: Leon Huang Fu To: stable@vger.kernel.org, greg@kroah.com Cc: tj@kernel.org, lizefan.x@bytedance.com, hannes@cmpxchg.org, corbet@lwn.net, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, muchun.song@linux.dev, akpm@linux-foundation.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com, lance.yang@linux.dev, leon.huangfu@shopee.com, shy828301@gmail.com, yosryahmed@google.com, sashal@kernel.org, vishal.moola@gmail.com, cerasuolodomenico@gmail.com, nphamcs@gmail.com, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Bagas Sanjaya , Chris Li , Shuah Khan Subject: [PATCH 6.6.y 2/7] mm: memcg: add per-memcg zswap writeback stat Date: Mon, 3 Nov 2025 15:51:30 +0800 Message-ID: <20251103075135.20254-3-leon.huangfu@shopee.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20251103075135.20254-1-leon.huangfu@shopee.com> References: <20251103075135.20254-1-leon.huangfu@shopee.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Domenico Cerasuolo [ Upstream commit 7108cc3f765cafd48a6a35f8add140beaecfa75b ] Since zswap now writes back pages from memcg-specific LRUs, we now need a new stat to show writebacks count for each memcg. [nphamcs@gmail.com: rename ZSWP_WB to ZSWPWB] Link: https://lkml.kernel.org/r/20231205193307.2432803-1-nphamcs@gmail.com Link: https://lkml.kernel.org/r/20231130194023.4102148-5-nphamcs@gmail.com Suggested-by: Nhat Pham Signed-off-by: Domenico Cerasuolo Signed-off-by: Nhat Pham Tested-by: Bagas Sanjaya Reviewed-by: Yosry Ahmed Cc: Chris Li Cc: Dan Streetman Cc: Johannes Weiner Cc: Michal Hocko Cc: Muchun Song Cc: Roman Gushchin Cc: Seth Jennings Cc: Shakeel Butt Cc: Shuah Khan Cc: Vitaly Wool Signed-off-by: Andrew Morton Signed-off-by: Leon Huang Fu --- include/linux/vm_event_item.h | 1 + mm/memcontrol.c | 1 + mm/vmstat.c | 1 + mm/zswap.c | 4 ++++ 4 files changed, 7 insertions(+) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 8abfa1240040..b61796a35d2b 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -145,6 +145,7 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, #ifdef CONFIG_ZSWAP ZSWPIN, ZSWPOUT, + ZSWPWB, #endif #ifdef CONFIG_X86 DIRECT_MAP_LEVEL2_SPLIT, diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c61c90ea72a4..03a984287e5b 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -700,6 +700,7 @@ static const unsigned int memcg_vm_event_stat[] =3D { #if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP) ZSWPIN, ZSWPOUT, + ZSWPWB, #endif #ifdef CONFIG_TRANSPARENT_HUGEPAGE THP_FAULT_ALLOC, diff --git a/mm/vmstat.c b/mm/vmstat.c index 57891697846b..3630c6e2bb41 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1397,6 +1397,7 @@ const char * const vmstat_text[] =3D { #ifdef CONFIG_ZSWAP "zswpin", "zswpout", + "zswpwb", #endif #ifdef CONFIG_X86 "direct_map_level2_splits", diff --git a/mm/zswap.c b/mm/zswap.c index 69681b9173fd..a3459440fc31 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -674,6 +674,10 @@ static int zswap_reclaim_entry(struct zswap_pool *pool) goto put_unlock; } + if (entry->objcg) + count_objcg_event(entry->objcg, ZSWPWB); + + count_vm_event(ZSWPWB); /* * Writeback started successfully, the page now belongs to the * swapcache. Drop the entry from zswap - unless invalidate already -- 2.50.1 From nobody Sun Feb 8 12:31:36 2026 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 64EAA2BE64C for ; Mon, 3 Nov 2025 07:53:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762156395; cv=none; b=iTXik34KHkqnaQui0G3CqpKM1/c+fp2zcr8ovDHN1xk6cq+AHaWv2dUJ9v/klTDBmCUXVWZEV5ymsVmyEFicQ/e6KrhmuzFd2nLSMl/YI9CM7/PgudB3NhLA/bO+D6PiUMA4zaPWa7jbndEusdZ0KVTlLUOZHpqf0Ynz/4lDfFc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762156395; c=relaxed/simple; bh=YI6y0WtkAOSxi3fWHEa12lXBLbRmRWX/UrNbhODMw7c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XWPCgZJk4N4cJO6uCf+sKTTR73YShkyrunI9eniw9SB1HzsDf/Ck0jkvWjozZTQ/WAZXsHVRphuCK7ZzXQl8uwWOhYFfJjAaw/VA+koFC4/Y2Z8DVW8/GR+HhYf7XSf9PMGnu5KzgFGg6GZTiTRqf5I6IhIL/LTugiTrOvQMLyo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=shopee.com; spf=pass smtp.mailfrom=shopee.com; dkim=pass (2048-bit key) header.d=shopee.com header.i=@shopee.com header.b=fgQbC0R1; arc=none smtp.client-ip=209.85.216.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=shopee.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shopee.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shopee.com header.i=@shopee.com header.b="fgQbC0R1" Received: by mail-pj1-f41.google.com with SMTP id 98e67ed59e1d1-339d7c403b6so3732161a91.2 for ; Sun, 02 Nov 2025 23:53:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shopee.com; s=shopee.com; t=1762156394; x=1762761194; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6yMtctYgS2rV+P47tm9bNW95kd+lcuzHA0J2B8KxNkI=; b=fgQbC0R16BbBxVgperj2mMJBzRM3/5FZwDegF6tlNBkkz0Kxf0M03DPBrG5fJN7pn9 ulelxzJlKlMU/yVcDbQjNm03n4dfnwn/hHKmz50/W+SlxkRtHG8ZaCmCCbNoKmafKQRX GPKrPT8hwvRLwI8KDP/TxdQwsnvvHsI5gcAZWqZHzPO688g7/pF0d50PdBjGueNCV7Ii RIPDfuX77WT+9EBrNSngHWeijBkppvejfd5RFW4UbVcWIdnZYatehs+24HehDOcdYzp6 sKjthJxLWMfFB25P4CSmQry8OK80bqEBs5pXvVaw1TC5CNPRcDMYhqX3YuVLi+LQtOZa fC5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762156394; x=1762761194; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6yMtctYgS2rV+P47tm9bNW95kd+lcuzHA0J2B8KxNkI=; b=DYL+wRSY4NooOTB8uZQzP7duDxi641phnO0Q+YY3QcOP8OARAFjAFGqoU0AiSkqVCj BKaXHXDSsHJUzJ+Md4mMfSsvFY62Sp46+q3ist0qORHgYJKyqERfNkyN29XL7vz5ddVp 9StEvDYyhlDPOhjnhENNOmkb6+ZoAv3x6Fc3+33umSuvJoy8w46WAmzGSkVLv+dKE00Z VkNXMQQeeUo3z2z/CMbggYn9Lyl/RRLuSwbAE8IO6sZqXtCoCModN8eH6heqYqA1dcQQ 9P8hdZn3x1XDT0Fu4/P5e+BGG07hjokmyyBy9HJNP61hYwBW5YWhc5BbQbCs/XpQbKP7 LEKQ== X-Forwarded-Encrypted: i=1; AJvYcCU40NN+clN+pEsJVc7/4r/bd0Yfb1zs2pLc0egA430YSv19NskLakqWZVYNf5LJ1yKmKjfaW5xaoiQbA+Y=@vger.kernel.org X-Gm-Message-State: AOJu0YyEW0pupZGP8AoSdC+8b8t4w1YtdE6eFL0oMufiE1iYfKr3806B HeE/tPKHjeiF1+SxwabkEBUtOsyQzvoFV75MnVzA6XLCtYNDtwd6VK18FdZ0AXK2bwE= X-Gm-Gg: ASbGnctRceybtM1Um/VI6jaUmwiXeoNdRWbX6Lom5R9Pv6hWixbir3p7iGq7+CzjqtZ j5Aow6Kqr2alIRqlYEv4G/fG9iW3uAhC4E20tEimChglYLGpNS9EntZLw6Suz4hRII7f0qNYpGs 6rLVfKQJNwlVAFoymR2PQrrFRb+Kj2vncdCLJYve0wNs0txiaozZZBYqYZV46QpHoxuPRH+FuTG wqdjY4sKcjxJK3/NuVaGz3JagIEi2J/BXOZlqNR1aqvUJ6DkdtTTvh0OI6jqh/RoBIAkhZYQxtU bjycqmwr3xouhR83pgIcQy2YMQbhBFf8dvoHfoUFv34URZ/aqt9hizlHIM0GSAx9B3ED8pMrINM baKkbt+k6NlwmdV8KQiVargtiB4vk7fORs1FDAykgDVQaTYUsz8KoG/IB9CU8KUXY47NipYb3OZ 30YWROSuoFQb8eKpHUTvlpuE3D X-Google-Smtp-Source: AGHT+IELQOgDx8D/eQP4ToiYJS7h+qaVvXs4UNrgHqqZp+hj5K9vMTs1s/xhbdYPxDl7zK1DXp+1dQ== X-Received: by 2002:a17:90b:53c7:b0:339:eff5:ef26 with SMTP id 98e67ed59e1d1-3408307e71amr13784912a91.30.1762156393719; Sun, 02 Nov 2025 23:53:13 -0800 (PST) Received: from .shopee.com ([122.11.166.8]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-34159a16652sm34552a91.20.2025.11.02.23.53.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 02 Nov 2025 23:53:13 -0800 (PST) From: Leon Huang Fu To: stable@vger.kernel.org, greg@kroah.com Cc: tj@kernel.org, lizefan.x@bytedance.com, hannes@cmpxchg.org, corbet@lwn.net, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, muchun.song@linux.dev, akpm@linux-foundation.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com, lance.yang@linux.dev, leon.huangfu@shopee.com, shy828301@gmail.com, yosryahmed@google.com, sashal@kernel.org, vishal.moola@gmail.com, cerasuolodomenico@gmail.com, nphamcs@gmail.com, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Chris Li , Bagas Sanjaya , Greg Thelen , Ivan Babrou , Michal Koutny , Waiman Long , Wei Xu Subject: [PATCH 6.6.y 3/7] mm: memcg: change flush_next_time to flush_last_time Date: Mon, 3 Nov 2025 15:51:31 +0800 Message-ID: <20251103075135.20254-4-leon.huangfu@shopee.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20251103075135.20254-1-leon.huangfu@shopee.com> References: <20251103075135.20254-1-leon.huangfu@shopee.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Yosry Ahmed [ Upstream commit 508bed884767a8eb394640bae9edcdf082816c43 ] Patch series "mm: memcg: subtree stats flushing and thresholds", v4. This series attempts to address shortages in today's approach for memcg stats flushing, namely occasionally stale or expensive stat reads. The series does so by changing the threshold that we use to decide whether to trigger a flush to be per memcg instead of global (patch 3), and then changing flushing to be per memcg (i.e. subtree flushes) instead of global (patch 5). This patch (of 5): flush_next_time is an inaccurate name. It's not the next time that periodic flushing will happen, it's rather the next time that ratelimited flushing can happen if the periodic flusher is late. Simplify its semantics by just storing the timestamp of the last flush instead, flush_last_time. Move the 2*FLUSH_TIME addition to mem_cgroup_flush_stats_ratelimited(), and add a comment explaining it. This way, all the ratelimiting semantics live in one place. No functional change intended. Link: https://lkml.kernel.org/r/20231129032154.3710765-1-yosryahmed@google.= com Link: https://lkml.kernel.org/r/20231129032154.3710765-2-yosryahmed@google.= com Signed-off-by: Yosry Ahmed Tested-by: Domenico Cerasuolo Acked-by: Shakeel Butt Acked-by: Chris Li (Google) Tested-by: Bagas Sanjaya Cc: Greg Thelen Cc: Ivan Babrou Cc: Johannes Weiner Cc: Michal Hocko Cc: Michal Koutny Cc: Muchun Song Cc: Roman Gushchin Cc: Tejun Heo Cc: Waiman Long Cc: Wei Xu Signed-off-by: Andrew Morton Signed-off-by: Leon Huang Fu --- mm/memcontrol.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 03a984287e5b..433cd273006d 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -590,7 +590,7 @@ static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush= _memcg_stats_dwork); static DEFINE_PER_CPU(unsigned int, stats_updates); static atomic_t stats_flush_ongoing =3D ATOMIC_INIT(0); static atomic_t stats_flush_threshold =3D ATOMIC_INIT(0); -static u64 flush_next_time; +static u64 flush_last_time; #define FLUSH_TIME (2UL*HZ) @@ -650,7 +650,7 @@ static void do_flush_stats(void) atomic_xchg(&stats_flush_ongoing, 1)) return; - WRITE_ONCE(flush_next_time, jiffies_64 + 2*FLUSH_TIME); + WRITE_ONCE(flush_last_time, jiffies_64); cgroup_rstat_flush(root_mem_cgroup->css.cgroup); @@ -666,7 +666,8 @@ void mem_cgroup_flush_stats(void) void mem_cgroup_flush_stats_ratelimited(void) { - if (time_after64(jiffies_64, READ_ONCE(flush_next_time))) + /* Only flush if the periodic flusher is one full cycle late */ + if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME)) mem_cgroup_flush_stats(); } -- 2.50.1 From nobody Sun Feb 8 12:31:36 2026 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8C2742C158E for ; Mon, 3 Nov 2025 07:53:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762156403; cv=none; b=edjaA23KEiMg3QCgNvTovbW2QHibj1AgbMsYswoOSFAG3c3iUlQETa1qMqkd1PTSEe83EWjkaqOYdXD5x/RRKcU+SESxZin6kGpVbQQNRSRdLdWXZkcizWj507msZLPnTm+3G71Yfh8fM+ZiLNAZP8uThMEBAEsPH7oFs6115OI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762156403; c=relaxed/simple; bh=Lodb6rLROk4Q3CAebuv8vkqskVQL3XSySEbny+ty2Qs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=l5b1zIg46u7D4nlZ6HpHi1696skDC80Ms3q8OEgda6ns/LT6+4z/soXKRc6ANESP9upbEKy1E7ISZWlxpjwztGyrUmhF+EOPoncGSZ21hI0AlmjHr7/PhtiZP5VW6UkISjJ5vzJqXrK4CzcG23ct+QgyehYbiX/dmjP7ixKhoZo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=shopee.com; spf=pass smtp.mailfrom=shopee.com; dkim=pass (2048-bit key) header.d=shopee.com header.i=@shopee.com header.b=RNpBwAS4; arc=none smtp.client-ip=209.85.216.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=shopee.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shopee.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shopee.com header.i=@shopee.com header.b="RNpBwAS4" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-340e525487eso822477a91.3 for ; Sun, 02 Nov 2025 23:53:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shopee.com; s=shopee.com; t=1762156401; x=1762761201; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nbphvwlZY5Z8mMpRxJCeoPt3Sbe7fHbYhkrPsa2IkNg=; b=RNpBwAS4yESmrj9Weh9BEo1RboZ8PK06YEA0hxkj1PpR/zXz1nB8SdZa24ikNZFH8v I/+0hPUmOjIT1kmeg4JM9BUUR4MmAziopCMnJYfZJ96Yd8xjk+gUA8cR2zaKsRK8VBY4 G/3BUoKW2hn4oKK2mb5LFyWsn5FBoRX0b7mlczVWgwIdY8aDae/GxBJwaZAKk9v+KZLF rNNZVSEdPS6Mfb2TyY4r9lo3Y+c6GRWsYCoSm49TiTbz5y317bhRCtdtYYKqTFzL25JI hOA/uEprySpMADE+pi2j8LC4pQgOPpWQB+t1EFzDR5HUyUUxZSBP5BGva3g/SdGR2o6B MtZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762156401; x=1762761201; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nbphvwlZY5Z8mMpRxJCeoPt3Sbe7fHbYhkrPsa2IkNg=; b=ZBAaTZcDJEDB1jBEBf4+vUSP9jEQ1kFXs8BacbYJk/hXYeM8Yk+nTCzSE715NssE1W eRmVKFGF6NmV/nzYWhQTr4ijOlOLb7OI9ITcf1EMZXBmyhvePD2ME7XUrQF54UB1/rzv 9hEeYJF0FRf+R8GzvqFOKS2kAjx0D60hXcWyrthUibHo3F9Swc+OuDl4pI921jEVtZL3 fFKxe82SUJ6NqgT5Fcav5KrMV9f2eEY90thMtvZClGTnE0rnqd3v38wflo2ARzR9fz+E Mn0j+4yuaSDpPFtayNYp8mQ+y+tseZ9acSntsNyO1kC+/Zl9+2MLU728+qIjmVjgTY9D x07A== X-Forwarded-Encrypted: i=1; AJvYcCV0uemNetW14hwjyKgxTDHUIGaF7lCqcDaLgHXmcmXevO/OW5KpzWpeill/f2GwLP8FZKZFhWLoJ3rE6tY=@vger.kernel.org X-Gm-Message-State: AOJu0YzqCsn8GCGFYTLO/yXhJdE7ENfVT7U+DMHXkSyBj2+Pwlv/DH9E G+VClrapjj4hyg9QHa/CGm1UH1/Dpc9QjOZ7ujeJfuwwJ+1WIu7ApwfcUJfrd3AgxT8= X-Gm-Gg: ASbGnct50Se2PQhch/j1Y0LM4mhAqDRWbrNO1/KH/6WkiZzwRQ+qWg0dyGZcRsyU6FA 72/aYg2VSXYRVEQ4S8ZndhRbgSfd/jG0Uaq0VOJ23ChZF+ricu8nU511kfw1iAaFN1vPZNTPL4I CuCfKMonyOe5lfHS4UlRIy5Nflz8rvPEYlF2xxgaW8XY2FYmU7BfTI/UTeWdaI2mNHSlLNCGtox z/K2ocMB/bvuzp+fO23gsA6Meh5fh0D9/mydXxcHDJZf5nV6IT6fBb3ToMqmN8KrE1p3ZuDKkcQ ljO3ZGS6F329OM6q+7Si2wNfrIcNGQt7UsZSAU9yAzcQVlVO/YzyeGXijTgQaseIdoEN+E283CK g5ZfINJ6pLjhQhGTRaqLnrm6FDWZKAb7vKXyoz0jhxQDeXe91ZhSMWa1jWFTNprssRDBTVi+8nI K3Csd/V5+HiXo0n55DLDD6QIwvkRTc4VjlGsc= X-Google-Smtp-Source: AGHT+IESJtnYCC+6tKU/IMMlAFX3OXThvg7/IT5AO+qIlJZ6EL/qIntowvs3vVciOZabgFGZ7594Wg== X-Received: by 2002:a17:90b:39cc:b0:329:e4d1:c20f with SMTP id 98e67ed59e1d1-34082fcc5f5mr15216714a91.9.1762156400905; Sun, 02 Nov 2025 23:53:20 -0800 (PST) Received: from .shopee.com ([122.11.166.8]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-34159a16652sm34552a91.20.2025.11.02.23.53.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 02 Nov 2025 23:53:20 -0800 (PST) From: Leon Huang Fu To: stable@vger.kernel.org, greg@kroah.com Cc: tj@kernel.org, lizefan.x@bytedance.com, hannes@cmpxchg.org, corbet@lwn.net, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, muchun.song@linux.dev, akpm@linux-foundation.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com, lance.yang@linux.dev, leon.huangfu@shopee.com, shy828301@gmail.com, yosryahmed@google.com, sashal@kernel.org, vishal.moola@gmail.com, cerasuolodomenico@gmail.com, nphamcs@gmail.com, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Chris Li , Greg Thelen , Ivan Babrou , Michal Koutny , Waiman Long , Wei Xu Subject: [PATCH 6.6.y 4/7] mm: memcg: move vmstats structs definition above flushing code Date: Mon, 3 Nov 2025 15:51:32 +0800 Message-ID: <20251103075135.20254-5-leon.huangfu@shopee.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20251103075135.20254-1-leon.huangfu@shopee.com> References: <20251103075135.20254-1-leon.huangfu@shopee.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Yosry Ahmed [ Upstream commit e0bf1dc859fdd08ef738824710770a30a8069433 ] The following patch will make use of those structs in the flushing code, so move their definitions (and a few other dependencies) a little bit up to reduce the diff noise in the following patch. No functional change intended. Link: https://lkml.kernel.org/r/20231129032154.3710765-3-yosryahmed@google.= com Signed-off-by: Yosry Ahmed Tested-by: Domenico Cerasuolo Acked-by: Shakeel Butt Cc: Chris Li Cc: Greg Thelen Cc: Ivan Babrou Cc: Johannes Weiner Cc: Michal Hocko Cc: Michal Koutny Cc: Muchun Song Cc: Roman Gushchin Cc: Tejun Heo Cc: Waiman Long Cc: Wei Xu Signed-off-by: Andrew Morton Signed-off-by: Leon Huang Fu --- mm/memcontrol.c | 148 ++++++++++++++++++++++++------------------------ 1 file changed, 74 insertions(+), 74 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 433cd273006d..157be6820fd1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -570,6 +570,80 @@ mem_cgroup_largest_soft_limit_node(struct mem_cgroup_t= ree_per_node *mctz) return mz; } +/* Subset of vm_event_item to report for memcg event stats */ +static const unsigned int memcg_vm_event_stat[] =3D { + PGPGIN, + PGPGOUT, + PGSCAN_KSWAPD, + PGSCAN_DIRECT, + PGSCAN_KHUGEPAGED, + PGSTEAL_KSWAPD, + PGSTEAL_DIRECT, + PGSTEAL_KHUGEPAGED, + PGFAULT, + PGMAJFAULT, + PGREFILL, + PGACTIVATE, + PGDEACTIVATE, + PGLAZYFREE, + PGLAZYFREED, +#if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP) + ZSWPIN, + ZSWPOUT, + ZSWPWB, +#endif +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + THP_FAULT_ALLOC, + THP_COLLAPSE_ALLOC, + THP_SWPOUT, + THP_SWPOUT_FALLBACK, +#endif +}; + +#define NR_MEMCG_EVENTS ARRAY_SIZE(memcg_vm_event_stat) +static int mem_cgroup_events_index[NR_VM_EVENT_ITEMS] __read_mostly; + +static void init_memcg_events(void) +{ + int i; + + for (i =3D 0; i < NR_MEMCG_EVENTS; ++i) + mem_cgroup_events_index[memcg_vm_event_stat[i]] =3D i + 1; +} + +static inline int memcg_events_index(enum vm_event_item idx) +{ + return mem_cgroup_events_index[idx] - 1; +} + +struct memcg_vmstats_percpu { + /* Local (CPU and cgroup) page state & events */ + long state[MEMCG_NR_STAT]; + unsigned long events[NR_MEMCG_EVENTS]; + + /* Delta calculation for lockless upward propagation */ + long state_prev[MEMCG_NR_STAT]; + unsigned long events_prev[NR_MEMCG_EVENTS]; + + /* Cgroup1: threshold notifications & softlimit tree updates */ + unsigned long nr_page_events; + unsigned long targets[MEM_CGROUP_NTARGETS]; +}; + +struct memcg_vmstats { + /* Aggregated (CPU and subtree) page state & events */ + long state[MEMCG_NR_STAT]; + unsigned long events[NR_MEMCG_EVENTS]; + + /* Non-hierarchical (CPU aggregated) page state & events */ + long state_local[MEMCG_NR_STAT]; + unsigned long events_local[NR_MEMCG_EVENTS]; + + /* Pending child counts during tree propagation */ + long state_pending[MEMCG_NR_STAT]; + unsigned long events_pending[NR_MEMCG_EVENTS]; +}; + /* * memcg and lruvec stats flushing * @@ -681,80 +755,6 @@ static void flush_memcg_stats_dwork(struct work_struct= *w) queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); } -/* Subset of vm_event_item to report for memcg event stats */ -static const unsigned int memcg_vm_event_stat[] =3D { - PGPGIN, - PGPGOUT, - PGSCAN_KSWAPD, - PGSCAN_DIRECT, - PGSCAN_KHUGEPAGED, - PGSTEAL_KSWAPD, - PGSTEAL_DIRECT, - PGSTEAL_KHUGEPAGED, - PGFAULT, - PGMAJFAULT, - PGREFILL, - PGACTIVATE, - PGDEACTIVATE, - PGLAZYFREE, - PGLAZYFREED, -#if defined(CONFIG_MEMCG_KMEM) && defined(CONFIG_ZSWAP) - ZSWPIN, - ZSWPOUT, - ZSWPWB, -#endif -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - THP_FAULT_ALLOC, - THP_COLLAPSE_ALLOC, - THP_SWPOUT, - THP_SWPOUT_FALLBACK, -#endif -}; - -#define NR_MEMCG_EVENTS ARRAY_SIZE(memcg_vm_event_stat) -static int mem_cgroup_events_index[NR_VM_EVENT_ITEMS] __read_mostly; - -static void init_memcg_events(void) -{ - int i; - - for (i =3D 0; i < NR_MEMCG_EVENTS; ++i) - mem_cgroup_events_index[memcg_vm_event_stat[i]] =3D i + 1; -} - -static inline int memcg_events_index(enum vm_event_item idx) -{ - return mem_cgroup_events_index[idx] - 1; -} - -struct memcg_vmstats_percpu { - /* Local (CPU and cgroup) page state & events */ - long state[MEMCG_NR_STAT]; - unsigned long events[NR_MEMCG_EVENTS]; - - /* Delta calculation for lockless upward propagation */ - long state_prev[MEMCG_NR_STAT]; - unsigned long events_prev[NR_MEMCG_EVENTS]; - - /* Cgroup1: threshold notifications & softlimit tree updates */ - unsigned long nr_page_events; - unsigned long targets[MEM_CGROUP_NTARGETS]; -}; - -struct memcg_vmstats { - /* Aggregated (CPU and subtree) page state & events */ - long state[MEMCG_NR_STAT]; - unsigned long events[NR_MEMCG_EVENTS]; - - /* Non-hierarchical (CPU aggregated) page state & events */ - long state_local[MEMCG_NR_STAT]; - unsigned long events_local[NR_MEMCG_EVENTS]; - - /* Pending child counts during tree propagation */ - long state_pending[MEMCG_NR_STAT]; - unsigned long events_pending[NR_MEMCG_EVENTS]; -}; - unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx) { long x =3D READ_ONCE(memcg->vmstats->state[idx]); -- 2.50.1 From nobody Sun Feb 8 12:31:36 2026 Received: from mail-pf1-f175.google.com (mail-pf1-f175.google.com [209.85.210.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D1FDA2C158E for ; Mon, 3 Nov 2025 07:53:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762156413; cv=none; b=uKV+rfPQlSVVREDKwN59hsg3jhWH9Q927eKuts6rk95zh8bRqoTzSvxOrFRRaGNy357f+hfDpgKFmNMe2H86Cu4iwlVEjKIScOPp1THE81blk4p4zaeLkzOpqQ3isbp+9woKDnInXIG0iQzevRRozafVfolMLYyMtHunE3A4STo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762156413; c=relaxed/simple; bh=DAl6x1WW+sdgvs4yNNBh1LtvTW7dfzKVhNfdGn4NeFs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MKqnDPAK8lk17nmn17lC4m6gGkd1P85Sl1NMvi1kT07kFEhhHT+S5dYcdAWi96QRDEsLIppo7m9Wug6G8ekheJhNP2EJH0OxpTw1o8S4TQXkY38t8BC/e0j7Y5wvMcmW6mFTixYq0kmjrrNUINKLKFoXYEbMQ7E2uYimKV/ggVo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=shopee.com; spf=pass smtp.mailfrom=shopee.com; dkim=pass (2048-bit key) header.d=shopee.com header.i=@shopee.com header.b=avRt75Sr; arc=none smtp.client-ip=209.85.210.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=shopee.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shopee.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shopee.com header.i=@shopee.com header.b="avRt75Sr" Received: by mail-pf1-f175.google.com with SMTP id d2e1a72fcca58-7a4c202a30aso4101103b3a.2 for ; Sun, 02 Nov 2025 23:53:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shopee.com; s=shopee.com; t=1762156408; x=1762761208; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Ba4jPz4GzIpOX2boFuFrVZ/wgd/V0hgb12GzaY8s2GM=; b=avRt75SrlEMRN6VHUrUG4/hxcSr/9e7hT8Lm4RVgI12j9XLFLeyv5LOpocBcQXqoHk AazKna8UPo/iQ9pIXo1ShuuTadSBN52RkDlzC2kWfV2k7++otXY+dRpyTopzKvObGoU/ MhaiWdvXhB/sr4rEjHcy2IKPHBBYh9i52H4SE12JKUBui4IqiuPyKgVz0KkwI3qy0pcL UuTch5Gy0SG2WHkhXwFCrMqxkOWuYWrt9erQALgIviHqetle8/DSjEoGsfaPDLWFGgem 5CRE4yo4kncazmWhFKnKLFNjQZmpda5lvwjGC2mptl+l6Ut8CwW0A1iovY7fEWFbKQWC utRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762156408; x=1762761208; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Ba4jPz4GzIpOX2boFuFrVZ/wgd/V0hgb12GzaY8s2GM=; b=SJPpjXg3Uz0HYeP/m3gZhi1SyFsVw/vRoC9y/sUm/csDKCKQn347gt8F6Zx5cGzZhF q0KwZQnkYo/mRmAKFlxKrn8gRyq89wJM89fT4SGc1L+0OAdI8SfH4vktwyypz9PqjyUp EdYa7YItVDLtqJK+3eM7yOLUT983wBsrAELOg885zD1rMEvcZyJbB465z/qhVKhig477 s0AnZl/P8OEX5Q3TeM83nWg2hMKE/B80rgRb/I6tzFcAcQWGcCC9gSYUEbIZeSHC0Wok J2Nb5cw90dbcfcfsRlncTBnT4NqXLcFJtTciQpykkj3ES+sjTqnJ1PLizVToHVxB1EJP r9Vg== X-Forwarded-Encrypted: i=1; AJvYcCUzEb+uWsmz+Y8Rsm+8u79cJBZE20zMLo4zpndLhchyJGWN9+avy2y+tBHOcnxQFJg4KEoClgPA4jWMp1A=@vger.kernel.org X-Gm-Message-State: AOJu0YzBi4LZytZKc/maXTph3s2CLJFDOrXd6pLSgglNnXCsKYeQabKU ohmea7K4p2jEokcLAZB/5mWUGqkL3UviLhdu5FlrH6LrSiCJ0UwVzF2GDY5vHUjO9hA= X-Gm-Gg: ASbGncuX+Lg1FLc86VaQnHSdCe/gWmIB+8ic+UzmNvUbcTmCagiS2r+9KKoBJsXZhHZ 8y4u2ID5OssZNlJ9LQtIoeBeKyEWKpEDA7EhCYqlUoyWlOuPqZ6YeTXq+8AjaYI68Y9vkzgpG9T ouX8S1Qj0oIbX0iHGnlYXHz1Ap82XhoQv3i1nGB1K99UHbIB8W4g7wd83f8wF8DnoqijYnNcI8v vrFRi180vgWBNult3k+p/AH2RJMeOHH62D6mUU/okm0qACTmB5bfUtu+X1juQ795wS9NS9ueT93 z0VlQLYiGn4HCDENW9jb9Bs5Idqdvk01wfH7tOivoQEd7PuSKfD+JqzHlEvgv8QqOA0hP1nS2zx rQHoyMIJYG66lRxmLFoLn7pEeqPoTdAcAucWW8pPc+NivzVwfvB94ahxcUzHQF4mkrUHxQ8Abhl LwzhVozQTsCL4uzQ== X-Google-Smtp-Source: AGHT+IGWNo1R+qKvdYnQyYk2CzdvOoi6Va0W2v8DV/ljaASF7VURewKuzp1i6g4RA/RRpxV0UmwQ/w== X-Received: by 2002:a05:6a20:7344:b0:311:99:7524 with SMTP id adf61e73a8af0-348ca351af3mr15450451637.18.1762156407934; Sun, 02 Nov 2025 23:53:27 -0800 (PST) Received: from .shopee.com ([122.11.166.8]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-34159a16652sm34552a91.20.2025.11.02.23.53.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 02 Nov 2025 23:53:27 -0800 (PST) From: Leon Huang Fu To: stable@vger.kernel.org, greg@kroah.com Cc: tj@kernel.org, lizefan.x@bytedance.com, hannes@cmpxchg.org, corbet@lwn.net, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, muchun.song@linux.dev, akpm@linux-foundation.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com, lance.yang@linux.dev, leon.huangfu@shopee.com, shy828301@gmail.com, yosryahmed@google.com, sashal@kernel.org, vishal.moola@gmail.com, cerasuolodomenico@gmail.com, nphamcs@gmail.com, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Chris Li , Greg Thelen , Ivan Babrou , Michal Koutny , Waiman Long , Wei Xu Subject: [PATCH 6.6.y 5/7] mm: memcg: make stats flushing threshold per-memcg Date: Mon, 3 Nov 2025 15:51:33 +0800 Message-ID: <20251103075135.20254-6-leon.huangfu@shopee.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20251103075135.20254-1-leon.huangfu@shopee.com> References: <20251103075135.20254-1-leon.huangfu@shopee.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Yosry Ahmed [ Upstream commit 8d59d2214c2362e7a9d185d80b613e632581af7b ] A global counter for the magnitude of memcg stats update is maintained on the memcg side to avoid invoking rstat flushes when the pending updates are not significant. This avoids unnecessary flushes, which are not very cheap even if there isn't a lot of stats to flush. It also avoids unnecessary lock contention on the underlying global rstat lock. Make this threshold per-memcg. The scheme is followed where percpu (now also per-memcg) counters are incremented in the update path, and only propagated to per-memcg atomics when they exceed a certain threshold. This provides two benefits: (a) On large machines with a lot of memcgs, the global threshold can be reached relatively fast, so guarding the underlying lock becomes less effective. Making the threshold per-memcg avoids this. (b) Having a global threshold makes it hard to do subtree flushes, as we cannot reset the global counter except for a full flush. Per-memcg counters removes this as a blocker from doing subtree flushes, which helps avoid unnecessary work when the stats of a small subtree are needed. Nothing is free, of course. This comes at a cost: (a) A new per-cpu counter per memcg, consuming NR_CPUS * NR_MEMCGS * 4 bytes. The extra memory usage is insigificant. (b) More work on the update side, although in the common case it will only be percpu counter updates. The amount of work scales with the number of ancestors (i.e. tree depth). This is not a new concept, adding a cgroup to the rstat tree involves a parent loop, so is charging. Testing results below show no significant regressions. (c) The error margin in the stats for the system as a whole increases from NR_CPUS * MEMCG_CHARGE_BATCH to NR_CPUS * MEMCG_CHARGE_BATCH * NR_MEMCGS. This is probably fine because we have a similar per-memcg error in charges coming from percpu stocks, and we have a periodic flusher that makes sure we always flush all the stats every 2s anyway. This patch was tested to make sure no significant regressions are introduced on the update path as follows. The following benchmarks were ran in a cgroup that is 2 levels deep (/sys/fs/cgroup/a/b/): (1) Running 22 instances of netperf on a 44 cpu machine with hyperthreading disabled. All instances are run in a level 2 cgroup, as well as netserver: # netserver -6 # netperf -6 -H ::1 -l 60 -t TCP_SENDFILE -- -m 10K Averaging 20 runs, the numbers are as follows: Base: 40198.0 mbps Patched: 38629.7 mbps (-3.9%) The regression is minimal, especially for 22 instances in the same cgroup sharing all ancestors (so updating the same atomics). (2) will-it-scale page_fault tests. These tests (specifically per_process_ops in page_fault3 test) detected a 25.9% regression before for a change in the stats update path [1]. These are the numbers from 10 runs (+ is good) on a machine with 256 cpus: LABEL | MEAN | MEDIAN | STDDEV | ------------------------------+-------------+-------------+------------- page_fault1_per_process_ops | | | | (A) base | 270249.164 | 265437.000 | 13451.836 | (B) patched | 261368.709 | 255725.000 | 13394.767 | | -3.29% | -3.66% | | page_fault1_per_thread_ops | | | | (A) base | 242111.345 | 239737.000 | 10026.031 | (B) patched | 237057.109 | 235305.000 | 9769.687 | | -2.09% | -1.85% | | page_fault1_scalability | | | (A) base | 0.034387 | 0.035168 | 0.0018283 | (B) patched | 0.033988 | 0.034573 | 0.0018056 | | -1.16% | -1.69% | | page_fault2_per_process_ops | | | (A) base | 203561.836 | 203301.000 | 2550.764 | (B) patched | 197195.945 | 197746.000 | 2264.263 | | -3.13% | -2.73% | | page_fault2_per_thread_ops | | | (A) base | 171046.473 | 170776.000 | 1509.679 | (B) patched | 166626.327 | 166406.000 | 768.753 | | -2.58% | -2.56% | | page_fault2_scalability | | | (A) base | 0.054026 | 0.053821 | 0.00062121 | (B) patched | 0.053329 | 0.05306 | 0.00048394 | | -1.29% | -1.41% | | page_fault3_per_process_ops | | | (A) base | 1295807.782 | 1297550.000 | 5907.585 | (B) patched | 1275579.873 | 1273359.000 | 8759.160 | | -1.56% | -1.86% | | page_fault3_per_thread_ops | | | (A) base | 391234.164 | 390860.000 | 1760.720 | (B) patched | 377231.273 | 376369.000 | 1874.971 | | -3.58% | -3.71% | | page_fault3_scalability | | | (A) base | 0.60369 | 0.60072 | 0.0083029 | (B) patched | 0.61733 | 0.61544 | 0.009855 | | +2.26% | +2.45% | | All regressions seem to be minimal, and within the normal variance for the benchmark. The fix for [1] assumes that 3% is noise -- and there were no further practical complaints), so hopefully this means that such variations in these microbenchmarks do not reflect on practical workloads. (3) I also ran stress-ng in a nested cgroup and did not observe any obvious regressions. [1]https://lore.kernel.org/all/20190520063534.GB19312@shao2-debian/ Link: https://lkml.kernel.org/r/20231129032154.3710765-4-yosryahmed@google.= com Signed-off-by: Yosry Ahmed Suggested-by: Johannes Weiner Tested-by: Domenico Cerasuolo Acked-by: Shakeel Butt Cc: Chris Li Cc: Greg Thelen Cc: Ivan Babrou Cc: Michal Hocko Cc: Michal Koutny Cc: Muchun Song Cc: Roman Gushchin Cc: Tejun Heo Cc: Waiman Long Cc: Wei Xu Signed-off-by: Andrew Morton Signed-off-by: Leon Huang Fu --- mm/memcontrol.c | 50 +++++++++++++++++++++++++++++++++---------------- 1 file changed, 34 insertions(+), 16 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 157be6820fd1..c31a5364f325 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -628,6 +628,9 @@ struct memcg_vmstats_percpu { /* Cgroup1: threshold notifications & softlimit tree updates */ unsigned long nr_page_events; unsigned long targets[MEM_CGROUP_NTARGETS]; + + /* Stats updates since the last flush */ + unsigned int stats_updates; }; struct memcg_vmstats { @@ -642,6 +645,9 @@ struct memcg_vmstats { /* Pending child counts during tree propagation */ long state_pending[MEMCG_NR_STAT]; unsigned long events_pending[NR_MEMCG_EVENTS]; + + /* Stats updates since the last flush */ + atomic64_t stats_updates; }; /* @@ -661,9 +667,7 @@ struct memcg_vmstats { */ static void flush_memcg_stats_dwork(struct work_struct *w); static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork); -static DEFINE_PER_CPU(unsigned int, stats_updates); static atomic_t stats_flush_ongoing =3D ATOMIC_INIT(0); -static atomic_t stats_flush_threshold =3D ATOMIC_INIT(0); static u64 flush_last_time; #define FLUSH_TIME (2UL*HZ) @@ -690,26 +694,37 @@ static void memcg_stats_unlock(void) preempt_enable_nested(); } + +static bool memcg_should_flush_stats(struct mem_cgroup *memcg) +{ + return atomic64_read(&memcg->vmstats->stats_updates) > + MEMCG_CHARGE_BATCH * num_online_cpus(); +} + static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) { + int cpu =3D smp_processor_id(); unsigned int x; if (!val) return; - cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id()); + cgroup_rstat_updated(memcg->css.cgroup, cpu); + + for (; memcg; memcg =3D parent_mem_cgroup(memcg)) { + x =3D __this_cpu_add_return(memcg->vmstats_percpu->stats_updates, + abs(val)); + + if (x < MEMCG_CHARGE_BATCH) + continue; - x =3D __this_cpu_add_return(stats_updates, abs(val)); - if (x > MEMCG_CHARGE_BATCH) { /* - * If stats_flush_threshold exceeds the threshold - * (>num_online_cpus()), cgroup stats update will be triggered - * in __mem_cgroup_flush_stats(). Increasing this var further - * is redundant and simply adds overhead in atomic update. + * If @memcg is already flush-able, increasing stats_updates is + * redundant. Avoid the overhead of the atomic update. */ - if (atomic_read(&stats_flush_threshold) <=3D num_online_cpus()) - atomic_add(x / MEMCG_CHARGE_BATCH, &stats_flush_threshold); - __this_cpu_write(stats_updates, 0); + if (!memcg_should_flush_stats(memcg)) + atomic64_add(x, &memcg->vmstats->stats_updates); + __this_cpu_write(memcg->vmstats_percpu->stats_updates, 0); } } @@ -728,13 +743,12 @@ static void do_flush_stats(void) cgroup_rstat_flush(root_mem_cgroup->css.cgroup); - atomic_set(&stats_flush_threshold, 0); atomic_set(&stats_flush_ongoing, 0); } void mem_cgroup_flush_stats(void) { - if (atomic_read(&stats_flush_threshold) > num_online_cpus()) + if (memcg_should_flush_stats(root_mem_cgroup)) do_flush_stats(); } @@ -748,8 +762,8 @@ void mem_cgroup_flush_stats_ratelimited(void) static void flush_memcg_stats_dwork(struct work_struct *w) { /* - * Always flush here so that flushing in latency-sensitive paths is - * as cheap as possible. + * Deliberately ignore memcg_should_flush_stats() here so that flushing + * in latency-sensitive paths is as cheap as possible. */ do_flush_stats(); queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); @@ -5658,6 +5672,10 @@ static void mem_cgroup_css_rstat_flush(struct cgroup= _subsys_state *css, int cpu) } } } + statc->stats_updates =3D 0; + /* We are in a per-cpu loop here, only do the atomic write once */ + if (atomic64_read(&memcg->vmstats->stats_updates)) + atomic64_set(&memcg->vmstats->stats_updates, 0); } #ifdef CONFIG_MMU -- 2.50.1 From nobody Sun Feb 8 12:31:36 2026 Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8D9FF2C21CD for ; Mon, 3 Nov 2025 07:53:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762156417; cv=none; b=TQNcoKis7TiJVgk8CUM38qzLskTgzNaDmfDjYqoAOmUZRfAf1jtAaWOyTh6siSqgdbvM85RcAyKoF/IuNbiJ7V/hP/fQKSmwXc0B7Q2lnmA1wy/4CVtwx9iS3k91jhtqbwul42RiL5gBo9yvAI5LL3+iRTX6527MxMbMf4ATAh4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762156417; c=relaxed/simple; bh=11CarfubxZ0qqJeB0Vehw4TOzZTVQzHPXsWuRlypEew=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rkwh49GtkHdsdHVA7nuYg65DX7HSjnmzAxpOiql4vNYUcoWgwu5uLb3nbS6cFjD4NoX1CMzsDsEjBDuVdKLZ4bTDlRheMkFBtDNif84COK4hhSU24MTGxklWiouPCK4okheBmHBep4xHveVbf501rr72LBYR7/GV/u8S1KEfcNw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=shopee.com; spf=pass smtp.mailfrom=shopee.com; dkim=pass (2048-bit key) header.d=shopee.com header.i=@shopee.com header.b=aoxfi8Ra; arc=none smtp.client-ip=209.85.216.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=shopee.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shopee.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shopee.com header.i=@shopee.com header.b="aoxfi8Ra" Received: by mail-pj1-f48.google.com with SMTP id 98e67ed59e1d1-34101107cc8so914092a91.0 for ; Sun, 02 Nov 2025 23:53:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shopee.com; s=shopee.com; t=1762156415; x=1762761215; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=M049GtTJOIx1jAzuZfnGS5SMrlVlEyxlsar/o1b4yF0=; b=aoxfi8Ra1FzgbVw0i+4G8onXMzKRoV0KsDT9xfY4lD3m8AjDKrQnM4V9jfkB0bXp3U 5v4rfNYf8W7aDq+WPzUdJTCL8oEDX5UbwTZHiiyLBug2ODPpgqrOWumgTzsQ08tYMhxh Eagmcmb7nqAObF9R5BY5+UVhCWpRVwLNcKg7SpU8d7Rr5syovYj+ZWOoAQO1T+ov03ID zvdFxCHurOPAzAVXxT5DegQbSUQlMPr2Z8shr5Ucqu2JeWNTzc2qcuZsHtFGGRdc6xcb bV70Mlvt3NDdZKTel/+W+f2ipTMDuocX8yWgjqKObAfVgMBFgdPz06lQ2Ut68u6002nB QBeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762156415; x=1762761215; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=M049GtTJOIx1jAzuZfnGS5SMrlVlEyxlsar/o1b4yF0=; b=ZrwKzv1wNtpEvp0Vm31e0AkDriWZKh8qkoIbTIS+zqWn7lRbnczTQ+F7TqFOgm0Dbe kgT3dT8PPE+Xm/mYddXw1vo0C27yqwTrNNSxdy2CTzGlrfZBWU2TfV1LUH81UKmGYXJP 9WW5NMGW32QgI4olLtoxtzqBasSs70zQRUNhe+EwDgUeDWFCysL8vC4jucUi/HidqEkb hWwJyusb4KvsK6axJZdabF1HPk49p6a4fmuD8ujezsVAJOrKRpXESVCbum8V4Dgdqqf/ AiRrGpuT3SL1Qkg/h3S5YWGneniwJu5WtXKV1BGih0IcVYnfjs6i7XNi/jutF3xhVp3K xCQg== X-Forwarded-Encrypted: i=1; AJvYcCVXqvulDmwGp1l0e2B8ZQ5pKdjOqzk+ivjHH2kEm1mU+FCzmLmRIhZAXuYolw56UgYWapfM/mOGGrWGjR0=@vger.kernel.org X-Gm-Message-State: AOJu0YyeEXWocZE/IqKgnyReWhNZdih0HjkhipDAz2aCMMW4OKbajAuO 1Jd4HZ+xw7QOWfIqX6LZMMw7LOoWFtP7SUfwljcxvBxfTqbswS2sL9Lj1afRmev3yAo= X-Gm-Gg: ASbGncsU9LK1/0O/imRo+WXpaVPKrIZ4TmCQKK2bsVLoW0qO4w5F3Vp2/QdIkRR7Yo+ RV0uiyrK1KofFD6wF3QLh/xhHFWfycbhMR1v5eJhusyDkUZ6S2Vmo7kJcQXcMZrw6eK1Gga72In FobwmcU4flXi80XJaKOCC+6BJgYouPI1h4JYOARVFl3SCHGwOn8lnrqFnSwRCj3qYgzCZ5h+RwB sOOjW9Zq4QSaf2kEHLDYY+oDdmB0NqsfgxLVq4ipvE28lksJIoWziRvpMg9Liof7g+Xh+IC9BUM DHNf7rT5lV4+TiMmxNj3esM6DE1JL8p89mCQiAf+yrzldiGVTDeZojL/+IZ11dPNp/Gogn0yeEP 9msdk+tlgj4H9pnkq1vcxaJNU12sB/U/l8o9CaS9pGwUQzKagLrK1ZHbqzL1O+OwzsB681oODKN ztOJQxem9k0vHQqYxKeLFcz7k+ X-Google-Smtp-Source: AGHT+IFNIljmMinHPhv4QokcHvxrVgwW0bD5RaqFND+JFxMKndgBolIUy2lVbgcYBW1Dd94/zRpM1A== X-Received: by 2002:a17:90b:3fc3:b0:340:d1a1:af6d with SMTP id 98e67ed59e1d1-340d1a1b159mr8269845a91.36.1762156414854; Sun, 02 Nov 2025 23:53:34 -0800 (PST) Received: from .shopee.com ([122.11.166.8]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-34159a16652sm34552a91.20.2025.11.02.23.53.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 02 Nov 2025 23:53:34 -0800 (PST) From: Leon Huang Fu To: stable@vger.kernel.org, greg@kroah.com Cc: tj@kernel.org, lizefan.x@bytedance.com, hannes@cmpxchg.org, corbet@lwn.net, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, muchun.song@linux.dev, akpm@linux-foundation.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com, lance.yang@linux.dev, leon.huangfu@shopee.com, shy828301@gmail.com, yosryahmed@google.com, sashal@kernel.org, vishal.moola@gmail.com, cerasuolodomenico@gmail.com, nphamcs@gmail.com, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Chris Li , Greg Thelen , Ivan Babrou , Michal Koutny , Waiman Long , Wei Xu Subject: [PATCH 6.6.y 6/7] mm: workingset: move the stats flush into workingset_test_recent() Date: Mon, 3 Nov 2025 15:51:34 +0800 Message-ID: <20251103075135.20254-7-leon.huangfu@shopee.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20251103075135.20254-1-leon.huangfu@shopee.com> References: <20251103075135.20254-1-leon.huangfu@shopee.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Yosry Ahmed [ Upstream commit b006847222623ac3cda8589d15379eac86a2bcb7 ] The workingset code flushes the stats in workingset_refault() to get accurate stats of the eviction memcg. In preparation for more scoped flushed and passing the eviction memcg to the flush call, move the call to workingset_test_recent() where we have a pointer to the eviction memcg. The flush call is sleepable, and cannot be made in an rcu read section. Hence, minimize the rcu read section by also moving it into workingset_test_recent(). Furthermore, instead of holding the rcu read lock throughout workingset_test_recent(), only hold it briefly to get a ref on the eviction memcg. This allows us to make the flush call after we get the eviction memcg. As for workingset_refault(), nothing else there appears to be protected by rcu. The memcg of the faulted folio (which is not necessarily the same as the eviction memcg) is protected by the folio lock, which is held from all callsites. Add a VM_BUG_ON() to make sure this doesn't change from under us. No functional change intended. Link: https://lkml.kernel.org/r/20231129032154.3710765-5-yosryahmed@google.= com Signed-off-by: Yosry Ahmed Tested-by: Domenico Cerasuolo Acked-by: Shakeel Butt Cc: Chris Li Cc: Greg Thelen Cc: Ivan Babrou Cc: Johannes Weiner Cc: Michal Hocko Cc: Michal Koutny Cc: Muchun Song Cc: Roman Gushchin Cc: Tejun Heo Cc: Waiman Long Cc: Wei Xu Signed-off-by: Andrew Morton Signed-off-by: Leon Huang Fu --- mm/workingset.c | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/mm/workingset.c b/mm/workingset.c index 9110957bec5b..6e61ad08df75 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -425,8 +425,16 @@ bool workingset_test_recent(void *shadow, bool file, b= ool *workingset) struct pglist_data *pgdat; unsigned long eviction; - if (lru_gen_enabled()) - return lru_gen_test_recent(shadow, file, &eviction_lruvec, &eviction, wo= rkingset); + rcu_read_lock(); + + if (lru_gen_enabled()) { + bool recent =3D lru_gen_test_recent(shadow, file, + &eviction_lruvec, &eviction, workingset); + + rcu_read_unlock(); + return recent; + } + unpack_shadow(shadow, &memcgid, &pgdat, &eviction, workingset); eviction <<=3D bucket_order; @@ -448,8 +456,16 @@ bool workingset_test_recent(void *shadow, bool file, b= ool *workingset) * configurations instead. */ eviction_memcg =3D mem_cgroup_from_id(memcgid); - if (!mem_cgroup_disabled() && !eviction_memcg) + if (!mem_cgroup_disabled() && + (!eviction_memcg || !mem_cgroup_tryget(eviction_memcg))) { + rcu_read_unlock(); return false; + } + + rcu_read_unlock(); + + /* Flush stats (and potentially sleep) outside the RCU read section */ + mem_cgroup_flush_stats_ratelimited(); eviction_lruvec =3D mem_cgroup_lruvec(eviction_memcg, pgdat); refault =3D atomic_long_read(&eviction_lruvec->nonresident_age); @@ -493,6 +509,7 @@ bool workingset_test_recent(void *shadow, bool file, bo= ol *workingset) } } + mem_cgroup_put(eviction_memcg); return refault_distance <=3D workingset_size; } @@ -519,19 +536,16 @@ void workingset_refault(struct folio *folio, void *sh= adow) return; } - /* Flush stats (and potentially sleep) before holding RCU read lock */ - mem_cgroup_flush_stats_ratelimited(); - - rcu_read_lock(); - /* * The activation decision for this folio is made at the level * where the eviction occurred, as that is where the LRU order * during folio reclaim is being determined. * * However, the cgroup that will own the folio is the one that - * is actually experiencing the refault event. + * is actually experiencing the refault event. Make sure the folio is + * locked to guarantee folio_memcg() stability throughout. */ + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); nr =3D folio_nr_pages(folio); memcg =3D folio_memcg(folio); pgdat =3D folio_pgdat(folio); @@ -540,7 +554,7 @@ void workingset_refault(struct folio *folio, void *shad= ow) mod_lruvec_state(lruvec, WORKINGSET_REFAULT_BASE + file, nr); if (!workingset_test_recent(shadow, file, &workingset)) - goto out; + return; folio_set_active(folio); workingset_age_nonresident(lruvec, nr); @@ -556,8 +570,6 @@ void workingset_refault(struct folio *folio, void *shad= ow) lru_note_cost_refault(folio); mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + file, nr); } -out: - rcu_read_unlock(); } /** -- 2.50.1 From nobody Sun Feb 8 12:31:36 2026 Received: from mail-pg1-f172.google.com (mail-pg1-f172.google.com [209.85.215.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9AFA62C21FE for ; Mon, 3 Nov 2025 07:53:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762156424; cv=none; b=Z5Qqdmdj5gY7Hm75PlheJAQ+MWu8fM7rjrlovUo56/JigIFQgZzcqBKgsi5B/hVDwXYfyy0f6/GPp1lp/Z0OIMhjAs7cGebSCjsnEP19wrlrhQAA8LYLoWNYIpXwrXSrJb6bKsLnX/n1qHV2hA6/mMSjs7Xk6AVJCE6EthLXIfA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1762156424; c=relaxed/simple; bh=OmLwWXxI7QJzZiWhcnz2SV5WgtWUgdvsnEL6HRwukQI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TC6OW5vrHjMr7vZlFLlr+ShJUGpY9E+QP5B/+iKrzhmLMeG3a0fQ4NdCTJBg6BLFuVaHjF0SE41ZEl1On60PJjCf8BMyN+N8SPdOajrR9/ealB/eCHwwH1fDeWFQUeTVYRAsvisTnDOaGyJdzIg4pdrV8qTC0NfZgyYOtI98Too= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=shopee.com; spf=pass smtp.mailfrom=shopee.com; dkim=pass (2048-bit key) header.d=shopee.com header.i=@shopee.com header.b=goHUFCmT; arc=none smtp.client-ip=209.85.215.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=shopee.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=shopee.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=shopee.com header.i=@shopee.com header.b="goHUFCmT" Received: by mail-pg1-f172.google.com with SMTP id 41be03b00d2f7-b6ce6d1d3dcso2777165a12.3 for ; Sun, 02 Nov 2025 23:53:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shopee.com; s=shopee.com; t=1762156422; x=1762761222; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=u/3QOrXfaRRSLiQ4GSPcpWfsbztlf+4SFaNdg8ef6Oc=; b=goHUFCmT9TIem6fW5MbBAL4jeNK4tqdC3hrR3vlJaa4fl7PF2Z28m0hXfK99mo/Liz Ca8NtMagcfru49Vnu5p1v0rJtsfFt9anp0yDukC3a5aoHPFaggtkOGqC4XI+k/bVukuw O4hJ+/CLpu32tOEPYOaQbVYTEMalCduYk9l5EXIcrf1LapGDyBHJLn9Pql1v60LOO/yX hcBnlDrYYg33aOHowgJLbaZiuhl2VZBtVYfAxhCD/5masIVBMu1AD8EvbEClbpeEoxgZ nU89ROlE/XOdnKiCSYODkfmEFLGnApxnGh5MEpEobp1mooeF1nVln001+nRq/pVTVbrL T7FA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762156422; x=1762761222; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=u/3QOrXfaRRSLiQ4GSPcpWfsbztlf+4SFaNdg8ef6Oc=; b=hc0kqXOVbfLWEG1TP4aV123rdxNC4b2XcLdIS/i7u/H+ZQXzSCDiz2sipErbUtn4j5 h0QYbCy0OT9lZA8IE9AuUvof7ialqPxqjdg1RnK2caTUVVFT9FVrVJbxkYNQw/3FjwUF Bi+aCSpuYCekZaqL6/V2G2HWJz+V1PsX4kx+9H0IGY6O5nvx1+vSFkRcLTdMCr8tuzCw NL773OhfLjVdyfzTWyu2GSt8Qdc8LSq4ESqxM/SG9PoYYrv5TuvTkDaelNnrdQJH4reV Ln4+eMD9eu5ooWVwemKjfkHxiC6wy3qX4Wt9m+a/A1tIDtU0c2g1w6M6teV2/H4Zo9m1 ODgA== X-Forwarded-Encrypted: i=1; AJvYcCUffg5gCnuq/StqylqhfK48TDQRxKcJ9RPh2pN2E9pNjbQK0/REWd4+8EGohLUyc++KkPXnq05GdQ6hwr0=@vger.kernel.org X-Gm-Message-State: AOJu0Ywt2Gke9S8d+lCsLjh6rk6e5vdWVqJ+rQUbAlC5XA2aBYzBpkuL RAXTC30HrC8JVFt4iZqjbDM4Z4ZMUxpJBZSPbB9bPgbNPa2PQLcmVxDPMo6bfhWiGtQ= X-Gm-Gg: ASbGncvbGYan0ScVum0yG5Hf6GF2lMxPprNSVgXDn+LDChz3ogVhFwTbNToXzKYza7J XKaI1VBkg0PZwBNacLkvMy2ozMsIcyEJXEiLDgZLUkL95UfnviDTtCOMb12ubd/bn81FCX15Yen ba2JJcNeluyY4gTdXijZqWV/6AWODTEQqSpmdvRpT/q5M+t+j9ZmMdmWfly6/So0V83qLH81KQT 4cl7trWETNCiCRYsrvdjLC2BJDzh7TsHno/9tTvTBmAxSUfRHJAiAmCrvNSodno+YqXx+bvH1kT zkztpyK3Qt9fUbGH7eeCmnh5L91970JbKRrNNJw+pRxdkTb1SJ3YXR9Uo7QGSvrF/3T7g1B6Z1+ iEYZAJxWSBPIn+BMFKannDX8Lh9Miu0HdsYeDqVKIEdj6Da+50Cs25J4slZULAMxrJOCDXN5Q7o OkQvOGbvT5R6+wGw== X-Google-Smtp-Source: AGHT+IHOYxavcvNCjMo8ESpp671IIFf4aThj7PcelCJSkknoZhse0VlY/vuaeqRwnpTxjujOeJrL1g== X-Received: by 2002:a17:902:d50f:b0:267:f7bc:673c with SMTP id d9443c01a7336-2951a51e6bfmr165298515ad.44.1762156421802; Sun, 02 Nov 2025 23:53:41 -0800 (PST) Received: from .shopee.com ([122.11.166.8]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-34159a16652sm34552a91.20.2025.11.02.23.53.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 02 Nov 2025 23:53:41 -0800 (PST) From: Leon Huang Fu To: stable@vger.kernel.org, greg@kroah.com Cc: tj@kernel.org, lizefan.x@bytedance.com, hannes@cmpxchg.org, corbet@lwn.net, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, muchun.song@linux.dev, akpm@linux-foundation.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com, lance.yang@linux.dev, leon.huangfu@shopee.com, shy828301@gmail.com, yosryahmed@google.com, sashal@kernel.org, vishal.moola@gmail.com, cerasuolodomenico@gmail.com, nphamcs@gmail.com, cgroups@vger.kernel.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Chris Li , Greg Thelen , Ivan Babrou , Michal Koutny , Waiman Long , Wei Xu Subject: [PATCH 6.6.y 7/7] mm: memcg: restore subtree stats flushing Date: Mon, 3 Nov 2025 15:51:35 +0800 Message-ID: <20251103075135.20254-8-leon.huangfu@shopee.com> X-Mailer: git-send-email 2.50.1 In-Reply-To: <20251103075135.20254-1-leon.huangfu@shopee.com> References: <20251103075135.20254-1-leon.huangfu@shopee.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Yosry Ahmed [ Upstream commit 7d7ef0a4686abe43cd76a141b340a348f45ecdf2 ] Stats flushing for memcg currently follows the following rules: - Always flush the entire memcg hierarchy (i.e. flush the root). - Only one flusher is allowed at a time. If someone else tries to flush concurrently, they skip and return immediately. - A periodic flusher flushes all the stats every 2 seconds. The reason this approach is followed is because all flushes are serialized by a global rstat spinlock. On the memcg side, flushing is invoked from userspace reads as well as in-kernel flushers (e.g. reclaim, refault, etc). This approach aims to avoid serializing all flushers on the global lock, which can cause a significant performance hit under high concurrency. This approach has the following problems: - Occasionally a userspace read of the stats of a non-root cgroup will be too expensive as it has to flush the entire hierarchy [1]. - Sometimes the stats accuracy are compromised if there is an ongoing flush, and we skip and return before the subtree of interest is actually flushed, yielding stale stats (by up to 2s due to periodic flushing). This is more visible when reading stats from userspace, but can also affect in-kernel flushers. The latter problem is particulary a concern when userspace reads stats after an event occurs, but gets stats from before the event. Examples: - When memory usage / pressure spikes, a userspace OOM handler may look at the stats of different memcgs to select a victim based on various heuristics (e.g. how much private memory will be freed by killing this). Reading stale stats from before the usage spike in this case may cause a wrongful OOM kill. - A proactive reclaimer may read the stats after writing to memory.reclaim to measure the success of the reclaim operation. Stale stats from before reclaim may give a false negative. - Reading the stats of a parent and a child memcg may be inconsistent (child larger than parent), if the flush doesn't happen when the parent is read, but happens when the child is read. As for in-kernel flushers, they will occasionally get stale stats. No regressions are currently known from this, but if there are regressions, they would be very difficult to debug and link to the source of the problem. This patch aims to fix these problems by restoring subtree flushing, and removing the unified/coalesced flushing logic that skips flushing if there is an ongoing flush. This change would introduce a significant regression with global stats flushing thresholds. With per-memcg stats flushing thresholds, this seems to perform really well. The thresholds protect the underlying lock from unnecessary contention. This patch was tested in two ways to ensure the latency of flushing is up to par, on a machine with 384 cpus: - A synthetic test with 5000 concurrent workers in 500 cgroups doing allocations and reclaim, as well as 1000 readers for memory.stat (variation of [2]). No regressions were noticed in the total runtime. Note that significant regressions in this test are observed with global stats thresholds, but not with per-memcg thresholds. - A synthetic stress test for concurrently reading memcg stats while memory allocation/freeing workers are running in the background, provided by Wei Xu [3]. With 250k threads reading the stats every 100ms in 50k cgroups, 99.9% of reads take <=3D 50us. Less than 0.01% of reads take more than 1ms, and no reads take more than 100ms. [1] https://lore.kernel.org/lkml/CABWYdi0c6__rh-K7dcM_pkf9BJdTRtAU08M43KO9M= E4-dsgfoQ@mail.gmail.com/ [2] https://lore.kernel.org/lkml/CAJD7tka13M-zVZTyQJYL1iUAYvuQ1fcHbCjcOBZcz= 6POYTV-4g@mail.gmail.com/ [3] https://lore.kernel.org/lkml/CAAPL-u9D2b=3DiF5Lf_cRnKxUfkiEe0AMDTu6yhrU= AzX0b6a6rDg@mail.gmail.com/ [akpm@linux-foundation.org: fix mm/zswap.c] [yosryahmed@google.com: remove stats flushing mutex] Link: https://lkml.kernel.org/r/CAJD7tkZgP3m-VVPn+fF_YuvXeQYK=3DtZZjJHj= =3DdzD=3DCcSSpp2qg@mail.gmail.com Link: https://lkml.kernel.org/r/20231129032154.3710765-6-yosryahmed@google.= com Signed-off-by: Yosry Ahmed Tested-by: Domenico Cerasuolo Acked-by: Shakeel Butt Cc: Chris Li Cc: Greg Thelen Cc: Ivan Babrou Cc: Johannes Weiner Cc: Michal Hocko Cc: Michal Koutny Cc: Muchun Song Cc: Roman Gushchin Cc: Tejun Heo Cc: Waiman Long Cc: Wei Xu Signed-off-by: Andrew Morton Signed-off-by: Leon Huang Fu --- include/linux/memcontrol.h | 8 ++--- mm/memcontrol.c | 68 ++++++++++++++++++++++---------------- mm/vmscan.c | 2 +- mm/workingset.c | 10 ++++-- 4 files changed, 51 insertions(+), 37 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index b1fdb1554f2f..8aee8b75aad0 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1039,8 +1039,8 @@ static inline unsigned long lruvec_page_state_local(s= truct lruvec *lruvec, return x; } -void mem_cgroup_flush_stats(void); -void mem_cgroup_flush_stats_ratelimited(void); +void mem_cgroup_flush_stats(struct mem_cgroup *memcg); +void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg); void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item i= dx, int val); @@ -1515,11 +1515,11 @@ static inline unsigned long lruvec_page_state_local= (struct lruvec *lruvec, return node_page_state(lruvec_pgdat(lruvec), idx); } -static inline void mem_cgroup_flush_stats(void) +static inline void mem_cgroup_flush_stats(struct mem_cgroup *memcg) { } -static inline void mem_cgroup_flush_stats_ratelimited(void) +static inline void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *m= emcg) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index c31a5364f325..d3f4486a45d8 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -667,7 +667,6 @@ struct memcg_vmstats { */ static void flush_memcg_stats_dwork(struct work_struct *w); static DECLARE_DEFERRABLE_WORK(stats_flush_dwork, flush_memcg_stats_dwork); -static atomic_t stats_flush_ongoing =3D ATOMIC_INIT(0); static u64 flush_last_time; #define FLUSH_TIME (2UL*HZ) @@ -728,35 +727,40 @@ static inline void memcg_rstat_updated(struct mem_cgr= oup *memcg, int val) } } -static void do_flush_stats(void) +static void do_flush_stats(struct mem_cgroup *memcg) { - /* - * We always flush the entire tree, so concurrent flushers can just - * skip. This avoids a thundering herd problem on the rstat global lock - * from memcg flushers (e.g. reclaim, refault, etc). - */ - if (atomic_read(&stats_flush_ongoing) || - atomic_xchg(&stats_flush_ongoing, 1)) - return; - - WRITE_ONCE(flush_last_time, jiffies_64); - - cgroup_rstat_flush(root_mem_cgroup->css.cgroup); + if (mem_cgroup_is_root(memcg)) + WRITE_ONCE(flush_last_time, jiffies_64); - atomic_set(&stats_flush_ongoing, 0); + cgroup_rstat_flush(memcg->css.cgroup); } -void mem_cgroup_flush_stats(void) +/* + * mem_cgroup_flush_stats - flush the stats of a memory cgroup subtree + * @memcg: root of the subtree to flush + * + * Flushing is serialized by the underlying global rstat lock. There is al= so a + * minimum amount of work to be done even if there are no stat updates to = flush. + * Hence, we only flush the stats if the updates delta exceeds a threshold= . This + * avoids unnecessary work and contention on the underlying lock. + */ +void mem_cgroup_flush_stats(struct mem_cgroup *memcg) { - if (memcg_should_flush_stats(root_mem_cgroup)) - do_flush_stats(); + if (mem_cgroup_disabled()) + return; + + if (!memcg) + memcg =3D root_mem_cgroup; + + if (memcg_should_flush_stats(memcg)) + do_flush_stats(memcg); } -void mem_cgroup_flush_stats_ratelimited(void) +void mem_cgroup_flush_stats_ratelimited(struct mem_cgroup *memcg) { /* Only flush if the periodic flusher is one full cycle late */ if (time_after64(jiffies_64, READ_ONCE(flush_last_time) + 2*FLUSH_TIME)) - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(memcg); } static void flush_memcg_stats_dwork(struct work_struct *w) @@ -765,7 +769,7 @@ static void flush_memcg_stats_dwork(struct work_struct = *w) * Deliberately ignore memcg_should_flush_stats() here so that flushing * in latency-sensitive paths is as cheap as possible. */ - do_flush_stats(); + do_flush_stats(root_mem_cgroup); queue_delayed_work(system_unbound_wq, &stats_flush_dwork, FLUSH_TIME); } @@ -1597,7 +1601,7 @@ static void memcg_stat_format(struct mem_cgroup *memc= g, struct seq_buf *s) * * Current memory state: */ - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(memcg); for (i =3D 0; i < ARRAY_SIZE(memory_stats); i++) { u64 size; @@ -4047,7 +4051,7 @@ static int memcg_numa_stat_show(struct seq_file *m, v= oid *v) int nid; struct mem_cgroup *memcg =3D mem_cgroup_from_seq(m); - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(memcg); for (stat =3D stats; stat < stats + ARRAY_SIZE(stats); stat++) { seq_printf(m, "%s=3D%lu", stat->name, @@ -4122,7 +4126,7 @@ static void memcg1_stat_format(struct mem_cgroup *mem= cg, struct seq_buf *s) BUILD_BUG_ON(ARRAY_SIZE(memcg1_stat_names) !=3D ARRAY_SIZE(memcg1_stats)); - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(memcg); for (i =3D 0; i < ARRAY_SIZE(memcg1_stats); i++) { unsigned long nr; @@ -4624,7 +4628,7 @@ void mem_cgroup_wb_stats(struct bdi_writeback *wb, un= signed long *pfilepages, struct mem_cgroup *memcg =3D mem_cgroup_from_css(wb->memcg_css); struct mem_cgroup *parent; - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(memcg); *pdirty =3D memcg_page_state(memcg, NR_FILE_DIRTY); *pwriteback =3D memcg_page_state(memcg, NR_WRITEBACK); @@ -6704,7 +6708,7 @@ static int memory_numa_stat_show(struct seq_file *m, = void *v) int i; struct mem_cgroup *memcg =3D mem_cgroup_from_seq(m); - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(memcg); for (i =3D 0; i < ARRAY_SIZE(memory_stats); i++) { int nid; @@ -7868,7 +7872,11 @@ bool obj_cgroup_may_zswap(struct obj_cgroup *objcg) break; } - cgroup_rstat_flush(memcg->css.cgroup); + /* + * mem_cgroup_flush_stats() ignores small changes. Use + * do_flush_stats() directly to get accurate stats for charging. + */ + do_flush_stats(memcg); pages =3D memcg_page_state(memcg, MEMCG_ZSWAP_B) / PAGE_SIZE; if (pages < max) continue; @@ -7933,8 +7941,10 @@ void obj_cgroup_uncharge_zswap(struct obj_cgroup *ob= jcg, size_t size) static u64 zswap_current_read(struct cgroup_subsys_state *css, struct cftype *cft) { - cgroup_rstat_flush(css->cgroup); - return memcg_page_state(mem_cgroup_from_css(css), MEMCG_ZSWAP_B); + struct mem_cgroup *memcg =3D mem_cgroup_from_css(css); + + mem_cgroup_flush_stats(memcg); + return memcg_page_state(memcg, MEMCG_ZSWAP_B); } static int zswap_max_show(struct seq_file *m, void *v) diff --git a/mm/vmscan.c b/mm/vmscan.c index 774bae2f54d7..aba757e5c597 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2911,7 +2911,7 @@ static void prepare_scan_count(pg_data_t *pgdat, stru= ct scan_control *sc) * Flush the memory cgroup stats, so that we read accurate per-memcg * lruvec stats for heuristics. */ - mem_cgroup_flush_stats(); + mem_cgroup_flush_stats(sc->target_mem_cgroup); /* * Determine the scan balance between anon and file LRUs. diff --git a/mm/workingset.c b/mm/workingset.c index 6e61ad08df75..7bac9be1b87f 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -464,8 +464,12 @@ bool workingset_test_recent(void *shadow, bool file, b= ool *workingset) rcu_read_unlock(); - /* Flush stats (and potentially sleep) outside the RCU read section */ - mem_cgroup_flush_stats_ratelimited(); + /* + * Flush stats (and potentially sleep) outside the RCU read section. + * XXX: With per-memcg flushing and thresholding, is ratelimiting + * still needed here? + */ + mem_cgroup_flush_stats_ratelimited(eviction_memcg); eviction_lruvec =3D mem_cgroup_lruvec(eviction_memcg, pgdat); refault =3D atomic_long_read(&eviction_lruvec->nonresident_age); @@ -676,7 +680,7 @@ static unsigned long count_shadow_nodes(struct shrinker= *shrinker, struct lruvec *lruvec; int i; - mem_cgroup_flush_stats_ratelimited(); + mem_cgroup_flush_stats_ratelimited(sc->memcg); lruvec =3D mem_cgroup_lruvec(sc->memcg, NODE_DATA(sc->nid)); for (pages =3D 0, i =3D 0; i < NR_LRU_LISTS; i++) pages +=3D lruvec_page_state_local(lruvec, -- 2.50.1