From nobody Tue Mar 3 03:38:38 2026 Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A0001314B8C for ; Mon, 2 Mar 2026 19:53:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772481195; cv=none; b=jpPntzOp9sY+Q6JUIrYenRwR9rDRbi87yl4HJrdL0e/h406Q4y5x+ysLeBJUMqcbabNPtoncHmJb8opByrDhLEAqHnEJz6Clh1HlOoOJfaHtnWAd+IWFC6CvjVJLmTYO9TPxvCxcPSkE+sIRcYK9DCgmnFKy5q9ICRoSySWz06E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772481195; c=relaxed/simple; bh=I+F9Yqfxg7lyOH6Y6lDKdBT3X45kJ6gN2c857wyMTEg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SDnj3hevqapLKI9Cd8uxfck+sv7voQssVXOrfFLR++sHSxgoeArs5Kcn0+jEUOIsAMezqSRD+S4rW10jCcDJheMgxopfO3/xWfTLcM+dRAip6CuuVQAWKUNwok8rZav1jCxX80nUto0LozgXECxvgeqHvut56ITHE/vPjiM0+as= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=d/S0SdA2; arc=none smtp.client-ip=209.85.222.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="d/S0SdA2" Received: by mail-qk1-f180.google.com with SMTP id af79cd13be357-8cb40149037so503527485a.2 for ; Mon, 02 Mar 2026 11:53:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1772481192; x=1773085992; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5m4nFf6dF08qw2hzXIDF9i+2wZFYfD7fQfBYIdzHPvQ=; b=d/S0SdA218kpr3WEz6Jfp4A08F9qCQyZo6qWjiKGVb2TInbyyuzb3p7GjmGO/+ZzlA kdTQLFmDSA+z5xkDmOmcwSKXW/6ELLYBkOuwQCaT/kuTndGs/q/oRwPXbSrU6oLO0OUC OYS8S9CJc8s9Jq0ajufCdcuj6iR1H4vTvOZoQIrRh3cu+/vmwPzDff6AlulOnxv1ytQm VFVT70CH/ANernkwQMoUnmqGLqQaemujiMJXnO68s+KMCn9Sm/lD2wmcA/8hkPiYf4az /bfRf0gMK4hQJFkd0GqLul6A58TnQ7EY34xX4lzf+AHu69dG4fIYx/EyP8statVf+FsG F3/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772481192; x=1773085992; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=5m4nFf6dF08qw2hzXIDF9i+2wZFYfD7fQfBYIdzHPvQ=; b=ny9fCJA2HLydrl+YfBjKUER/5ozVH8ONvXQCxmdTZM7yaAEFtYSd35GUoHrwAmGhdm xU8BPYUQH5gIolacS1u2rJcfIPEd1RQnGE7sr5E3wbhKMsnQB6kdcfrDwEAHD+lgS9Wk iEjaIL6DcaYmh4YITib7w2PYKBJt+T/DqsGKmR8Tm0o+eWNn6Bj/+U5k4NKL1EAaYWR6 +x1gG3W3ffxzfJCe05L+glftNeu2LGeLrFYqUHcIlgeEOBo2nY17/+EeHv5DDXaoI6bh 37iTICMrheZsfwPogvJK49vWAimEb7CI4YQRS/5XWM4ZCuvLAuUOyIpNwg9ogT1WaOtp arDA== X-Forwarded-Encrypted: i=1; AJvYcCVPKImiRhHdaoEbJNSWm1ikmjVqDJHkamF14QseEkNXa80AVRTPT22ifQ0K2cLyiAzltjDDoMrI+94yH7U=@vger.kernel.org X-Gm-Message-State: AOJu0YwmZY5TqELMZCLpaujOf5VC3ouQBn+4Co4G52qVR7YU5k8v9536 mQLxoYiRVLCSLayEA/o82tN5+7+tqnrRjNMCoxaRVS3DfhwTth6rNrRTSCKX+X1vYbo= X-Gm-Gg: ATEYQzzlURkke/ngWldCcGRgaIAy+9O7VOavrBzX+elTwaq1OHiFE52WtdgN/DjMZ1p xEqvVD9I1CBFfOQDG3Qr+ylaxgUyfmCbw7bA1/hVMZGfeb/Vo94eTB3V7be8ln2mx2TKf5gDZl3 o4Jqspk4yuYX+rKhl7/oJO270561x/AhZkZrHDn1HZH11s/mOdbCg90j8f5tMXxOfR5E5gKSLQ/ I/werUjfNsasJ0udzJA7xAMN0x0keItumV49WEWtwtbtKhWTYQrYqUNSBTq13NAevyGzdO5YwcJ o7TB8B3G3o8cduUzLcjybrlcCIYw4G1HurbXManQhuQP93UTbHk9cVm880rw+/LxCzNDLIP22yL AOEjKz4XDU+PBenBra8sfkw2PYf2BTjxUcTxMQvmEnsqqTX8UklQGI1OynWPcTsDGjKkaU3NT4b Nnc1nQU9s3/kHwG6neBgVLOg== X-Received: by 2002:a05:620a:254b:b0:8a2:3be9:1d79 with SMTP id af79cd13be357-8cbc8d7a9camr1778134585a.18.1772481192477; Mon, 02 Mar 2026 11:53:12 -0800 (PST) Received: from localhost ([2603:7000:c00:3a00:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id af79cd13be357-8cbbf652bb6sm1307822585a.4.2026.03.02.11.53.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Mar 2026 11:53:11 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Hao Li , Michal Hocko , Roman Gushchin , Shakeel Butt , Vlastimil Babka , Harry Yoo , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Johannes Weiner Subject: [PATCH 1/5] mm: memcg: factor out trylock_stock() and unlock_stock() Date: Mon, 2 Mar 2026 14:50:14 -0500 Message-ID: <20260302195305.620713-2-hannes@cmpxchg.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260302195305.620713-1-hannes@cmpxchg.org> References: <20260302195305.620713-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Johannes Weiner Consolidate the local lock acquisition and the local stock lookup. This allows subsequent patches to use !!stock as an easy way to disambiguate the locked vs. contended cases through the callstack. Signed-off-by: Johannes Weiner Acked-by: Shakeel Butt --- mm/memcontrol.c | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 753d76e96cc6..a975ab3aee10 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3208,6 +3208,19 @@ void __memcg_kmem_uncharge_page(struct page *page, i= nt order) obj_cgroup_put(objcg); } =20 +static struct obj_stock_pcp *trylock_stock(void) +{ + if (local_trylock(&obj_stock.lock)) + return this_cpu_ptr(&obj_stock); + + return NULL; +} + +static void unlock_stock(struct obj_stock_pcp *stock) +{ + local_unlock(&obj_stock.lock); +} + static void __account_obj_stock(struct obj_cgroup *objcg, struct obj_stock_pcp *stock, int nr, struct pglist_data *pgdat, enum node_stat_item idx) @@ -3263,10 +3276,10 @@ static bool consume_obj_stock(struct obj_cgroup *ob= jcg, unsigned int nr_bytes, struct obj_stock_pcp *stock; bool ret =3D false; =20 - if (!local_trylock(&obj_stock.lock)) + stock =3D trylock_stock(); + if (!stock) return ret; =20 - stock =3D this_cpu_ptr(&obj_stock); if (objcg =3D=3D READ_ONCE(stock->cached_objcg) && stock->nr_bytes >=3D n= r_bytes) { stock->nr_bytes -=3D nr_bytes; ret =3D true; @@ -3275,7 +3288,7 @@ static bool consume_obj_stock(struct obj_cgroup *objc= g, unsigned int nr_bytes, __account_obj_stock(objcg, stock, nr_bytes, pgdat, idx); } =20 - local_unlock(&obj_stock.lock); + unlock_stock(stock); =20 return ret; } @@ -3366,7 +3379,8 @@ static void refill_obj_stock(struct obj_cgroup *objcg= , unsigned int nr_bytes, struct obj_stock_pcp *stock; unsigned int nr_pages =3D 0; =20 - if (!local_trylock(&obj_stock.lock)) { + stock =3D trylock_stock(); + if (!stock) { if (pgdat) mod_objcg_mlstate(objcg, pgdat, idx, nr_acct); nr_pages =3D nr_bytes >> PAGE_SHIFT; @@ -3375,7 +3389,6 @@ static void refill_obj_stock(struct obj_cgroup *objcg= , unsigned int nr_bytes, goto out; } =20 - stock =3D this_cpu_ptr(&obj_stock); if (READ_ONCE(stock->cached_objcg) !=3D objcg) { /* reset if necessary */ drain_obj_stock(stock); obj_cgroup_get(objcg); @@ -3395,7 +3408,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg= , unsigned int nr_bytes, stock->nr_bytes &=3D (PAGE_SIZE - 1); } =20 - local_unlock(&obj_stock.lock); + unlock_stock(stock); out: if (nr_pages) obj_cgroup_uncharge_pages(objcg, nr_pages); --=20 2.53.0 From nobody Tue Mar 3 03:38:38 2026 Received: from mail-qv1-f44.google.com (mail-qv1-f44.google.com [209.85.219.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 755F2315D5B for ; Mon, 2 Mar 2026 19:53:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772481196; cv=none; b=X91XUIS5jAIZoK+Im9uEyILfIFo4doC6RdzDRkdjRl0hl51u+JQw6FVHS8JZbXz403MZZlZ2F7SJD4yLEopeHqGbPsHcbWvmR5yP346AFhD5Nwx1pgkkeEFuPmrpzBBKAEE2HFIJhCjh+ozb20zDRMmAIFrRQTIpW7HW/0H+W6I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772481196; c=relaxed/simple; bh=xMOOY+JO0QZspES/MoGCVCDt+stnCC4+6jDUOXw23q8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZS7VJD9zGv4AQYWVl3Eff988H2WuC8u/FdyyBOOEMC975Uyo9C3ZdB0At1rZdoC7U4LNKDvoEhxDKpfBA8UYmc6a/1dxhWVyjoJnrfXz+SXFwcQ24erE0YuQxG3OaUzvu0EzzpSoQuS96QLD/+9STUdukhult+LXtI54P2laE2k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=gdaDoVZb; arc=none smtp.client-ip=209.85.219.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="gdaDoVZb" Received: by mail-qv1-f44.google.com with SMTP id 6a1803df08f44-896f4627dffso68352146d6.0 for ; Mon, 02 Mar 2026 11:53:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1772481194; x=1773085994; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MpE/M8Cc0DYeAdsX8W3VyabxBIuEQ5Zyn3nZkaA1QSY=; b=gdaDoVZb9pIB8DDyj2ANOlDdYNRy3Vd3zTkGDmF1KiR4/jJXxgC0lHuDfKOjo4TFyA pXnu7rzlpVfqilqxw1x1k0y58/Gma+E64DRUuwbWMGhXxR0kvYdQjHTp9sgLonbiV4Nn xkWd3X01YTQSyLcWp/of/ZWllniNv71xrxBXDRiCXUc7Uxx8a3T/S+/+Cn3B1mxoOftC jSKjM+LYrRaL5q2YOcnvv4KdezJfU21B8RB2/x4bSEOxXDEKKUzUXRkwF+lYOqpzIGbg uQxmPi8OXkuidyRvNNFP09wscQaFUv/hAbymk09pPAGv3HC8jb85hRryy2ZgdCup0ORi hl+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772481194; x=1773085994; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=MpE/M8Cc0DYeAdsX8W3VyabxBIuEQ5Zyn3nZkaA1QSY=; b=AbA92rDLYGAX5V5f9l/64p/qPxDnN9NvUJWE0tSmpAOEciTiQjEPJUloH9rQvfBWQk 7x9YCT2MF0hpm1GrmhUOj9CV8hwuZ7n2DPlY2aitU3St3WGZx22OnWWoOi2bcu9KXtUC v5IZKAePzSoMnC7QS23lnmoxIAmU1RUgDclV6MmWsv+18vdnGznyZwiOfPq93IFE1qcT ZtjjXMCvLb0SzmLQWSFuZ9Bc2FjHtlwS3Kr+UfDT57k2S+yleVhTgoAnTxa27pYFRnZI yi3sFHpUIvexx5nCcjQmoOgLe7U6taQd5RemmiKCGbQXkXxkVGkvmeTxWZ9OcWFh1Aro KqlA== X-Forwarded-Encrypted: i=1; AJvYcCVyNkAQKwK5RuRu9s4OQI0Kw5VyKLDLaE/w1fDwT/TQ6Vchu3bKK2IPHEeCGwipExrOjnCj4uovlfNrJ9o=@vger.kernel.org X-Gm-Message-State: AOJu0Yw4fB5v32H5kXVYZXew/lGhGdwL/Ci6FDfB9cBoOY/6uKfu4h5z 8nv4Y5aWsO2zTk7zbcY7DQxRpQHFjHESYgjsk6Ktx7S1G7dkT0bEYRGlEWHnRjCKmZc= X-Gm-Gg: ATEYQzxm1FOPxL85LB6ve1xxrZ9uepnwuYHMYu6VI9uo4IjrnJ9U4yH8n12eCJtjaWr +oTk1vdGMDBo8IXIBae3tSoES7r9gqO5PvWx0i+P4JNmvFCx3xap3JPf4ckGCozVRvf2lmJQKLW JWnBxK3e8rqYDIqgCYWuc6Pb+/6yz9gxLPlCK/r4FRwfF7SnlBIgK1/peA8Ill5gKxRWruFnKUe f5+Tk9YtYlQ6+9uhxzobEanL73AgXz7QU7HrULasVQKsPDu1MyxPoCDfGFIK5OWr5jDxsBlz3uA nrL3vNeulBdr2aYwULrs5vKO0Cmiq80dRTxAI/eWHM97AO4wOHsg4ibmltOW5HfC8BEI0rUoDWA 5l3SJkwjmaacNw1HXH6RDN/100glFM9LKcaY9lHaRSNXaUF22hls5sQFQP5FZQEPSkFwv/oVKkz 1KkdHyi/uJT7xARIaYQJ2BZQ== X-Received: by 2002:a05:6214:226c:b0:895:d97:a2ba with SMTP id 6a1803df08f44-899d1e9866cmr180883656d6.67.1772481194477; Mon, 02 Mar 2026 11:53:14 -0800 (PST) Received: from localhost ([2603:7000:c00:3a00:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-899e53a0d04sm58618466d6.1.2026.03.02.11.53.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Mar 2026 11:53:13 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Hao Li , Michal Hocko , Roman Gushchin , Shakeel Butt , Vlastimil Babka , Harry Yoo , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Johannes Weiner Subject: [PATCH 2/5] mm: memcg: simplify objcg charge size and stock remainder math Date: Mon, 2 Mar 2026 14:50:15 -0500 Message-ID: <20260302195305.620713-3-hannes@cmpxchg.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260302195305.620713-1-hannes@cmpxchg.org> References: <20260302195305.620713-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" From: Johannes Weiner Use PAGE_ALIGN() and a more natural cache remainder calculation. Signed-off-by: Johannes Weiner Acked-by: Shakeel Butt --- mm/memcontrol.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index a975ab3aee10..0d0a77fedb00 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3417,7 +3417,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg= , unsigned int nr_bytes, static int obj_cgroup_charge_account(struct obj_cgroup *objcg, gfp_t gfp, = size_t size, struct pglist_data *pgdat, enum node_stat_item idx) { - unsigned int nr_pages, nr_bytes; + size_t charge_size, remainder; int ret; =20 if (likely(consume_obj_stock(objcg, size, pgdat, idx))) @@ -3446,16 +3446,12 @@ static int obj_cgroup_charge_account(struct obj_cgr= oup *objcg, gfp_t gfp, size_t * bytes is (sizeof(object) + PAGE_SIZE - 2) if there is no data * race. */ - nr_pages =3D size >> PAGE_SHIFT; - nr_bytes =3D size & (PAGE_SIZE - 1); + charge_size =3D PAGE_ALIGN(size); + remainder =3D charge_size - size; =20 - if (nr_bytes) - nr_pages +=3D 1; - - ret =3D obj_cgroup_charge_pages(objcg, gfp, nr_pages); - if (!ret && (nr_bytes || pgdat)) - refill_obj_stock(objcg, nr_bytes ? PAGE_SIZE - nr_bytes : 0, - false, size, pgdat, idx); + ret =3D obj_cgroup_charge_pages(objcg, gfp, charge_size >> PAGE_SHIFT); + if (!ret && (remainder || pgdat)) + refill_obj_stock(objcg, remainder, false, size, pgdat, idx); =20 return ret; } --=20 2.53.0 From nobody Tue Mar 3 03:38:38 2026 Received: from mail-qk1-f179.google.com (mail-qk1-f179.google.com [209.85.222.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7C92B383C89 for ; Mon, 2 Mar 2026 19:53:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772481204; cv=none; b=lK8qhNuADmunbGyrpTeeFQOJ5DhKxGaOurFpvBmkUW7H+axnnPWlxp+02U5ehT456o5f5fUtaChKtO6jQEPkpopum4UlulzbnLwArEba2brNeCbOrE8yyF+zFrMr4LAt6FnnzsH/2zavKU0tt++u7y8h1dTxC7rducqSJROm5O0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772481204; c=relaxed/simple; bh=WXSj+9QejVVwAbxgamPZUuinqhgHMtGM4UaQbMwBBKI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gJ8Gr+T+C3OcgHDY0WvrMSGCcsvs1MMz6L08eHDUhBSl9m7zRPYDvwK4M0ECQqHV2Msv0JAKePWF8Hf7qK30/OrgmHkAo+DaxIGiusgCpXvRZaVmbagZ9heOYqTorunUnRxDkiGhI0nlch/S5HYUKC9Ax2Nkks6j4oYeLJuONNo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=BLk9JKh6; arc=none smtp.client-ip=209.85.222.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="BLk9JKh6" Received: by mail-qk1-f179.google.com with SMTP id af79cd13be357-8cb20bcff5aso474883285a.3 for ; Mon, 02 Mar 2026 11:53:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1772481196; x=1773085996; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9SxQ/eLtCsKDNFbOz1QdWcfhbyrBVWfPJbSjku1OsMI=; b=BLk9JKh6cH5ymvXZgtiwZtIaJiyPJ+bv2tvqf1uDd9Rm6h0AOMzjjMfHctF8d0bMzE ce7x9aeo5ebc1n9Gty19rxSf9aTsyFwy+o9lYAuKvYKSZJxxEbKeJuC+p/9WkWReftMs 2QcaFDMsM7FXJHiv6htlICaP0p8WkokiG3/5fH+G3r+mWvdKCkcg/UKdfTB8MgJUNPYS S8kQ0S6lNAgBst9sm/yXQRF500httNBcBzfpCTxe5DpVBsUGbELw4gg9egdfS3t4f+Gd xB70DJ90JjqlDS3nHfFxBK8FRl2mJjUT01PAsemzPe8cochrU6/NZDm4T/6weZ08yTmN 8Z6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772481196; x=1773085996; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=9SxQ/eLtCsKDNFbOz1QdWcfhbyrBVWfPJbSjku1OsMI=; b=KUJRctZsQEIkh1cazIIZavWc4sfVxr6E5nNHkqig3sAVIu/fs/1nG/+SmDHg2+FDnz JP7BzgrTl0Wq6GjQJZZsQodF8pA8fPSXw1omeEAvs4z6wmlHoO+lBdSZ1aNWACMHfEjd T0iY7yvCgFLKEIqN+m3lqfbTmPFxk594MLhUrTd5p036D2pyUDeCTlwPMkTRaCLDbmEy ey06ViMk7aa0aU6JyBpT4PLuW7COmG/Gas3CxHhc6uJxMF7YeIvVf7iQapyH8U/JC8a1 k5LL+2sG8PR3CXq54Pp/COP3yJls1qTOeeJ/Oq03VI0fo1AjXWYOEHYs4RSdq/wi99bF khwQ== X-Forwarded-Encrypted: i=1; AJvYcCUjdoQggKxb5nkpg44M5eEFdFvNPINmk+Wtpj+oFdiyFGRun43YIOK4/yCMGEzccYtUrzRl0ZCJ3ToipDA=@vger.kernel.org X-Gm-Message-State: AOJu0Yyz5Mgdl12qX0xn+bE9x8ZF1c7REL8xaaKM+mqbmW1v8C4iCGnG 42/ufk9a54+Adu+QcEwvyLH48wnu1HlNLF8KlGr3PUYGntOIjJ5oXXRrsU1Sm/SLFaw= X-Gm-Gg: ATEYQzyt0UkbKg9qk07+9MicELmUP0+hAhlK8fpUsK7Yq8p9Cpnmyhw+GyWBF1qp5UN OjGj9kUO8AxWMpuexj3m8rpGkVrt9h+UT7+k/HSsLGugjncnYnSxuiE7ASGpVHlrS3RXUn1ufaf 1BVmUvmwUjiHxFqKN7MsoAJm1VQfv3mWoPx9lZauE3A2oiJpV/ZzStCdDmjAv9zw1VTzx0PbNII mXXvFUCjWXqI8SaqN/jfNwidRWgBXugYQYrueE5TyEXtMpGgE+PdsMB1g44kiCu3Npd+XGJzBij nApk1moa7dwFGIsYYsgYNUSgL2oD5u94RT7aZNZnhZJxLF5GGOkLNQRpyCUP0NnoLvpmM8mIGT6 M5fpd6LtlOQUIwSYcF+SCqzQRE0zbPeNyDZHZKtuvPF3uHnYiH3RcxL8GTI4r3M3gRQrlrz0sG/ R9gKzDYGgb/f/QaFBt+LLO1w== X-Received: by 2002:a05:620a:400f:b0:8c6:ff02:d825 with SMTP id af79cd13be357-8cbc8e03048mr1741310485a.48.1772481196370; Mon, 02 Mar 2026 11:53:16 -0800 (PST) Received: from localhost ([2603:7000:c00:3a00:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id af79cd13be357-8cbbf6535b4sm1217544685a.9.2026.03.02.11.53.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Mar 2026 11:53:15 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Hao Li , Michal Hocko , Roman Gushchin , Shakeel Butt , Vlastimil Babka , Harry Yoo , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/5] mm: memcontrol: split out __obj_cgroup_charge() Date: Mon, 2 Mar 2026 14:50:16 -0500 Message-ID: <20260302195305.620713-4-hannes@cmpxchg.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260302195305.620713-1-hannes@cmpxchg.org> References: <20260302195305.620713-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move the page charge and remainder calculation into its own function. It will make the slab stat refactor easier to follow. Signed-off-by: Johannes Weiner Acked-by: Shakeel Butt --- mm/memcontrol.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0d0a77fedb00..32c09b4d520f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3414,10 +3414,24 @@ static void refill_obj_stock(struct obj_cgroup *obj= cg, unsigned int nr_bytes, obj_cgroup_uncharge_pages(objcg, nr_pages); } =20 +static int __obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, + size_t size, size_t *remainder) +{ + size_t charge_size; + int ret; + + charge_size =3D PAGE_ALIGN(size); + ret =3D obj_cgroup_charge_pages(objcg, gfp, charge_size >> PAGE_SHIFT); + if (!ret) + *remainder =3D charge_size - size; + + return ret; +} + static int obj_cgroup_charge_account(struct obj_cgroup *objcg, gfp_t gfp, = size_t size, struct pglist_data *pgdat, enum node_stat_item idx) { - size_t charge_size, remainder; + size_t remainder; int ret; =20 if (likely(consume_obj_stock(objcg, size, pgdat, idx))) @@ -3446,10 +3460,7 @@ static int obj_cgroup_charge_account(struct obj_cgro= up *objcg, gfp_t gfp, size_t * bytes is (sizeof(object) + PAGE_SIZE - 2) if there is no data * race. */ - charge_size =3D PAGE_ALIGN(size); - remainder =3D charge_size - size; - - ret =3D obj_cgroup_charge_pages(objcg, gfp, charge_size >> PAGE_SHIFT); + ret =3D __obj_cgroup_charge(objcg, gfp, size, &remainder); if (!ret && (remainder || pgdat)) refill_obj_stock(objcg, remainder, false, size, pgdat, idx); =20 --=20 2.53.0 From nobody Tue Mar 3 03:38:38 2026 Received: from mail-qv1-f45.google.com (mail-qv1-f45.google.com [209.85.219.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5105D317164 for ; Mon, 2 Mar 2026 19:53:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772481204; cv=none; b=dTeQfhwkYYZv/OyR+8qZRf2o/iX17F4ZB8ofxD1wNLOiJFnchKxV3FlMnXlzsD9xM+UL33iLqboxVBUstXya7wvF8KJubzVc7QWeddOIoSCIu6sU32RNlUUV6yL0uBkKpjcT74J4OzqQvBsfngX4SMUsP7hBrw6OSFELnSo8xPI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772481204; c=relaxed/simple; bh=cC80qRo7JR4gIfZy7Oohm8yvU+7yInxXX1FTNUyJ+DE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=N2rWLBHRxBMGy41PI2UitOK6LpaHBXBF/hLuYmyOKCCqALhQjH6iTdsNXkUn8f83NmeUC0uGI4afD+lNZFAN2wJZPTUPd7cQw9hl+VmQyjSVdYOzlp1KAlM5vEGUC7a/V7YXadsP9LuUahgE5IQfzLtGtfZaftWvemGE8T0canY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=bjSmNgXH; arc=none smtp.client-ip=209.85.219.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="bjSmNgXH" Received: by mail-qv1-f45.google.com with SMTP id 6a1803df08f44-899fa9610bbso21623616d6.0 for ; Mon, 02 Mar 2026 11:53:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1772481198; x=1773085998; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IA4kjBdpYkYKS/nfolmQRGvXpsjWiFeExJHfqapx5Xg=; b=bjSmNgXHF9jdG7y1MwjfLEMVc2T2l2ANlNOIVyDFNqI4XEeZcNgHrbrI5y6E82otJR d93wfhQvLJMzNyHr+/tPeHyrAvlRo7QGg+pBZHSehdk6WMUUlm+kJfUpom3RDsYeJjrM eWwctbX+Xg6UsC1BruAEUHgPfQPzqC9BRSZzV08OeD/wa0LRucJ+52I40qHUc3FLtzSC bjslffLMATeayzms2TY4ZVTJzhANUNRakjBxL91puuH7KXSiP/n8lBu+gmUhraUrImG6 u57WZN8ymqCN+RkEK1wYdsXvlNUtZPajURjm5iaMSmttd6Ebgkx8sTDOOG/+4KChQvGW iDBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772481198; x=1773085998; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=IA4kjBdpYkYKS/nfolmQRGvXpsjWiFeExJHfqapx5Xg=; b=nGeOzjyWu3C1SnFfhPmymkKcPiOusQ/JSflJg1pnnYmM+LXGH8oKRpB9wYsrbwQ3DM S2eFkYFPLc9Zo8/H3wFAAnZVTM9r7SK6VZVcfk3iEkyqh+wH/bLgsn0R03373/G06wAa HvvM4JJTu1oMPCS6IuqQppIk7NshoA6Mu+64QejRsaCnwaaVdXzUc4t1QazOZrKCiXbM jfAHuiq7yqvARfQLWyFumK1XD2J9/XLSbcvJyZ//bHmGCTNOEWJ2Wc+FrKRs7JJ3D0AS YEJ3wSyCY0+2/YI+J2LqCfZT+tJWQuNjkgWlo9kLvqSr/O2F5TtOChb0dWkuWUt/6pvS ojsg== X-Forwarded-Encrypted: i=1; AJvYcCWXSzAaG6U06Yg1JwNjz4tj2StwFpbPFuYFQ2h3BnKQjoLWAgGkU28Cl5ErgPp6thBzMiJtv4cbM0a3dJk=@vger.kernel.org X-Gm-Message-State: AOJu0YyuBF1cc7PAyuS1nd/1AtFB1PQ5SJ0kanU9UgjHjDNrwjs27/4w iU8/yHmyLKQ8SGtEtX8h3p6uYEUGPYRoq18R52ud7irqjmJH6sisKDeUzRDH5dJwwg0= X-Gm-Gg: ATEYQzxmDFyuhvOpa2saTGQLgtdUPmV0j1ZTVwapqytOYIvD2sUvByuZdAgMLCROVYN 13ck+mRenJVk//7OdwmGSlz85LSVFBTPk9OU/dG0DG6Q/gaXewoS6nCea9DnqT4nQw2/nG6xqJ4 nP6DvutTokUvgsFmJKKtSAGvwloPFkA0XV4LjlWw5Mqc+hhRookg5kcY7t8WOMGMegZc69BNrRf UxWzG2slICn3KLdy6ouD0PZffIyLW4rziHrJp4Ky2DjUOjgma6yCIIngJJ9NN8UBtlb0uW3zB+c gTMD/7+YS4yhF33rHmAUF5A3GT1FXg6ZRIdTRnT60P5vEVxYkT29CZUwEuxXlUwanb2c0akm1MA eRV1FLO3lr7lLggjlMiIyysrcmE/8fvLg2DEPGZFV+kaoYnqM/jYPoijmrs7iJ9aR4E9WNGbZYE p+5UC2rgCcbntXQOMme6IBB5IFANJ/V+Hp X-Received: by 2002:ad4:5ded:0:b0:89a:4f:1172 with SMTP id 6a1803df08f44-89a004f1317mr42550936d6.50.1772481198200; Mon, 02 Mar 2026 11:53:18 -0800 (PST) Received: from localhost ([2603:7000:c00:3a00:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-899f4fb208dsm44178686d6.28.2026.03.02.11.53.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Mar 2026 11:53:17 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Hao Li , Michal Hocko , Roman Gushchin , Shakeel Butt , Vlastimil Babka , Harry Yoo , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 4/5] mm: memcontrol: use __account_obj_stock() in the !locked path Date: Mon, 2 Mar 2026 14:50:17 -0500 Message-ID: <20260302195305.620713-5-hannes@cmpxchg.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260302195305.620713-1-hannes@cmpxchg.org> References: <20260302195305.620713-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make __account_obj_stock() usable for the case where the local trylock failed. Then switch refill_obj_stock() over to it. This consolidates the mod_objcg_mlstate() call into one place and will make the next patch easier to follow. Signed-off-by: Johannes Weiner Acked-by: Shakeel Butt --- mm/memcontrol.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 32c09b4d520f..4f12b75743d4 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3227,6 +3227,9 @@ static void __account_obj_stock(struct obj_cgroup *ob= jcg, { int *bytes; =20 + if (!stock) + goto direct; + /* * Save vmstat data in stock and skip vmstat array update unless * accumulating over a page of vmstat data or when pgdat changes. @@ -3266,6 +3269,7 @@ static void __account_obj_stock(struct obj_cgroup *ob= jcg, nr =3D 0; } } +direct: if (nr) mod_objcg_mlstate(objcg, pgdat, idx, nr); } @@ -3382,7 +3386,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg= , unsigned int nr_bytes, stock =3D trylock_stock(); if (!stock) { if (pgdat) - mod_objcg_mlstate(objcg, pgdat, idx, nr_acct); + __account_obj_stock(objcg, NULL, nr_acct, pgdat, idx); nr_pages =3D nr_bytes >> PAGE_SHIFT; nr_bytes =3D nr_bytes & (PAGE_SIZE - 1); atomic_add(nr_bytes, &objcg->nr_charged_bytes); --=20 2.53.0 From nobody Tue Mar 3 03:38:38 2026 Received: from mail-qv1-f41.google.com (mail-qv1-f41.google.com [209.85.219.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 59676383C88 for ; Mon, 2 Mar 2026 19:53:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772481207; cv=none; b=e4wOwJAiGLZX9TPKb7+aP3N5tQMStpGtg6OtTeDGhekK/q+hlSOsvDNZFW/I/yZiODg8wEB12MGaNIO2thb2v2Wja17LFoELuNdOWxVcyq9furFffSHNxWeK6B4Kb+faZ4Shc15C9OQHl99ulXrNo3WtU+KEbShkrgJooRI361c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1772481207; c=relaxed/simple; bh=HEeg9n9CQQLgAaFgl++GPRpIQzNa5AIDCSTnURtsnD4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KpOBGERXz3VknIgu3YyVMP1KX2WSodqdmzQ8Mg4YZ2IYUWX0mgXkWxtjMoLE9wCebqsc1j4VccrvwJjxAtXBmdrP0JobHun+NuF5B8hduJzoRD0wfJ1bCBAO/96uoLWmyJFMYDe0/sBRASgKRg2RlPQxQyKCkDP4uP53SLqfESs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org; spf=pass smtp.mailfrom=cmpxchg.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b=OPvJFXz4; arc=none smtp.client-ip=209.85.219.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=cmpxchg.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg.org header.i=@cmpxchg.org header.b="OPvJFXz4" Received: by mail-qv1-f41.google.com with SMTP id 6a1803df08f44-899e87b04d8so36962396d6.3 for ; Mon, 02 Mar 2026 11:53:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg.org; s=google; t=1772481200; x=1773086000; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=g7vZGkE+1JaDLDSfa23XDw27l8nhvvwnMmFZbgJqVLA=; b=OPvJFXz4MGknnu7B55WQ/LCFfJdk8PdeRLmB19v+PwYpxz2tbjrjqnWcD9ryAm3GJ8 btLD+CUSXx6kKCpUqbcCSqhhiWHhDXzcjlvHp3nAjictLPvRdg3SXKKF9rTllbEqybbU Hk9HU9fFSIeH6BsvCvzhQTi9gD0bj3q2qWDkVAJJwlH0CcA2kTi7vNS9F3YfbZBPBXdu 6sFgct7wiD1XxWF9ARxVs4g70GpMoLDtTOewr2dNUbDi1NFC3fFaiMkyVj1mfW+NvJ4K VrhugsDdmcLP1DsbS/UvFa53W2OeiRjTT5uNfU+Jf7chmxmobbqkbGvPlh6trNNVfQrn GC5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1772481200; x=1773086000; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=g7vZGkE+1JaDLDSfa23XDw27l8nhvvwnMmFZbgJqVLA=; b=mMHjmckD43G+aymHP4dI2r47Dv8ZzA3qVnB6DEr9NaPa1PDle8t2YkRhv0BWFhTqM0 ZNm++zgkwIctqhO7kjzcp42597yOPwb59C5nAxZdQQffrsqBqbRN7mvKWZXzjxObGEkZ 0I/8Vc7WiVyGN0uQlyv7m7QXy9Sff951x7au8fJEjVB+rXf0SaAfyV1Oz11lk3QblF3R Y8V74Ws8sOepOUJt3hENz12WnwyB0Pq3XnM/vyMrAcEdq13RnW2xl3unVJi4DCQtbtoe T1ExjKdq+5/l5Zhqchf8bgxW7TlZs59vWHo94O1VlneAgE9GD1MkGGrPcNibySwGoNnz Km+A== X-Forwarded-Encrypted: i=1; AJvYcCUL3fZoHjfcJqBjhHtiojMyWZZzy8RF1+rsoD6Q+yFCsUjnpQFWIcNcvofHW4R1AXV6RquZA2v85AC3kgE=@vger.kernel.org X-Gm-Message-State: AOJu0YxLqu76ph6The1aVSScG3nLDJAlg3LgcfHDzwBGFyFnAO2Cc9nz VcRlObjaXodLEecgywjXHsuCoX1xZg1mITbovBoBmpi6Z45WpB3/V85DQeVuBkBWeEQ= X-Gm-Gg: ATEYQzzLW2baMXB9I2Twqz5RxzODa0EridMOWs00Q/Rr7UqqvX+9Bh/eIiCisED0Mkt VBM9YldgKpyYM4GepF8MlEccfHvbldfJjy9MhKLZTVl1L/isBTxrePuAnGU1SSqM5wTPwl8c5lS mjOeUZUktcmVa5ZtomTYsnzjWcdRKaesoVrfegu3BzGalo09z94X3YCD+mmus4/Y6ilAEHDv3kg +LieHuJkNBzQ6LOylzk0mfUp6O9MOMwmpFspCSAtOge/IL7gksBvGzY2X89fqVE7TY/np1zSVl+ wRja/9lMpQ8/g9bps5rQLmT+QmjS8we3iHBCm8DFEyFHiNyi1ZQ1PHyZl/sM/m89jvW3qyIA8sQ K1PxGWlCSgjQ0MfBAaA/wAH1JHX1h1fJJhBkCPogCAg3aRcthTgD1ZslBQtJin8+j4MWGT1Vjwg lJ84J5brawUOP7hVqL/9EQ689C0lonQgH7 X-Received: by 2002:a05:6214:d4a:b0:899:fd80:f79c with SMTP id 6a1803df08f44-899fd8106b5mr59620616d6.22.1772481200266; Mon, 02 Mar 2026 11:53:20 -0800 (PST) Received: from localhost ([2603:7000:c00:3a00:365a:60ff:fe62:ff29]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-899f4fb208dsm44179416d6.28.2026.03.02.11.53.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 02 Mar 2026 11:53:19 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Hao Li , Michal Hocko , Roman Gushchin , Shakeel Butt , Vlastimil Babka , Harry Yoo , linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 5/5] mm: memcg: separate slab stat accounting from objcg charge cache Date: Mon, 2 Mar 2026 14:50:18 -0500 Message-ID: <20260302195305.620713-6-hannes@cmpxchg.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260302195305.620713-1-hannes@cmpxchg.org> References: <20260302195305.620713-1-hannes@cmpxchg.org> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Cgroup slab metrics are cached per-cpu the same way as the sub-page charge cache. However, the intertwined code to manage those dependent caches right now is quite difficult to follow. Specifically, cached slab stat updates occur in consume() if there was enough charge cache to satisfy the new object. If that fails, whole pages are reserved, and slab stats are updated when the remainder of those pages, after subtracting the size of the new slab object, are put into the charge cache. This already juggles a delicate mix of the object size, the page charge size, and the remainder to put into the byte cache. Doing slab accounting in this path as well is fragile, and has recently caused a bug where the input parameters between the two caches were mixed up. Refactor the consume() and refill() paths into unlocked and locked variants that only do charge caching. Then let the slab path manage its own lock section and open-code charging and accounting. This makes the slab stat cache subordinate to the charge cache: __refill_obj_stock() is called first to prepare it; __account_obj_stock() follows to hitch a ride. This results in a minor behavioral change: previously, a mismatching percpu stock would always be drained for the purpose of setting up slab account caching, even if there was no byte remainder to put into the charge cache. Now, the stock is left alone, and slab accounting takes the uncached path if there is a mismatch. This is exceedingly rare, and it was probably never worth draining the whole stock just to cache the slab stat update. Signed-off-by: Johannes Weiner Acked-by: Shakeel Butt --- mm/memcontrol.c | 100 +++++++++++++++++++++++++++++------------------- 1 file changed, 61 insertions(+), 39 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4f12b75743d4..9c6f9849b717 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3218,16 +3218,18 @@ static struct obj_stock_pcp *trylock_stock(void) =20 static void unlock_stock(struct obj_stock_pcp *stock) { - local_unlock(&obj_stock.lock); + if (stock) + local_unlock(&obj_stock.lock); } =20 +/* Call after __refill_obj_stock() to ensure stock->cached_objg =3D=3D obj= cg */ static void __account_obj_stock(struct obj_cgroup *objcg, struct obj_stock_pcp *stock, int nr, struct pglist_data *pgdat, enum node_stat_item idx) { int *bytes; =20 - if (!stock) + if (!stock || READ_ONCE(stock->cached_objcg) !=3D objcg) goto direct; =20 /* @@ -3274,8 +3276,20 @@ static void __account_obj_stock(struct obj_cgroup *o= bjcg, mod_objcg_mlstate(objcg, pgdat, idx, nr); } =20 -static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_by= tes, - struct pglist_data *pgdat, enum node_stat_item idx) +static bool __consume_obj_stock(struct obj_cgroup *objcg, + struct obj_stock_pcp *stock, + unsigned int nr_bytes) +{ + if (objcg =3D=3D READ_ONCE(stock->cached_objcg) && + stock->nr_bytes >=3D nr_bytes) { + stock->nr_bytes -=3D nr_bytes; + return true; + } + + return false; +} + +static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_by= tes) { struct obj_stock_pcp *stock; bool ret =3D false; @@ -3284,14 +3298,7 @@ static bool consume_obj_stock(struct obj_cgroup *obj= cg, unsigned int nr_bytes, if (!stock) return ret; =20 - if (objcg =3D=3D READ_ONCE(stock->cached_objcg) && stock->nr_bytes >=3D n= r_bytes) { - stock->nr_bytes -=3D nr_bytes; - ret =3D true; - - if (pgdat) - __account_obj_stock(objcg, stock, nr_bytes, pgdat, idx); - } - + ret =3D __consume_obj_stock(objcg, stock, nr_bytes); unlock_stock(stock); =20 return ret; @@ -3376,17 +3383,14 @@ static bool obj_stock_flush_required(struct obj_sto= ck_pcp *stock, return flush; } =20 -static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_byt= es, - bool allow_uncharge, int nr_acct, struct pglist_data *pgdat, - enum node_stat_item idx) +static void __refill_obj_stock(struct obj_cgroup *objcg, + struct obj_stock_pcp *stock, + unsigned int nr_bytes, + bool allow_uncharge) { - struct obj_stock_pcp *stock; unsigned int nr_pages =3D 0; =20 - stock =3D trylock_stock(); if (!stock) { - if (pgdat) - __account_obj_stock(objcg, NULL, nr_acct, pgdat, idx); nr_pages =3D nr_bytes >> PAGE_SHIFT; nr_bytes =3D nr_bytes & (PAGE_SIZE - 1); atomic_add(nr_bytes, &objcg->nr_charged_bytes); @@ -3404,20 +3408,25 @@ static void refill_obj_stock(struct obj_cgroup *obj= cg, unsigned int nr_bytes, } stock->nr_bytes +=3D nr_bytes; =20 - if (pgdat) - __account_obj_stock(objcg, stock, nr_acct, pgdat, idx); - if (allow_uncharge && (stock->nr_bytes > PAGE_SIZE)) { nr_pages =3D stock->nr_bytes >> PAGE_SHIFT; stock->nr_bytes &=3D (PAGE_SIZE - 1); } =20 - unlock_stock(stock); out: if (nr_pages) obj_cgroup_uncharge_pages(objcg, nr_pages); } =20 +static void refill_obj_stock(struct obj_cgroup *objcg, + unsigned int nr_bytes, + bool allow_uncharge) +{ + struct obj_stock_pcp *stock =3D trylock_stock(); + __refill_obj_stock(objcg, stock, nr_bytes, allow_uncharge); + unlock_stock(stock); +} + static int __obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size, size_t *remainder) { @@ -3432,13 +3441,12 @@ static int __obj_cgroup_charge(struct obj_cgroup *o= bjcg, gfp_t gfp, return ret; } =20 -static int obj_cgroup_charge_account(struct obj_cgroup *objcg, gfp_t gfp, = size_t size, - struct pglist_data *pgdat, enum node_stat_item idx) +int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size) { size_t remainder; int ret; =20 - if (likely(consume_obj_stock(objcg, size, pgdat, idx))) + if (likely(consume_obj_stock(objcg, size))) return 0; =20 /* @@ -3465,20 +3473,15 @@ static int obj_cgroup_charge_account(struct obj_cgr= oup *objcg, gfp_t gfp, size_t * race. */ ret =3D __obj_cgroup_charge(objcg, gfp, size, &remainder); - if (!ret && (remainder || pgdat)) - refill_obj_stock(objcg, remainder, false, size, pgdat, idx); + if (!ret && remainder) + refill_obj_stock(objcg, remainder, false); =20 return ret; } =20 -int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size) -{ - return obj_cgroup_charge_account(objcg, gfp, size, NULL, 0); -} - void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size) { - refill_obj_stock(objcg, size, true, 0, NULL, 0); + refill_obj_stock(objcg, size, true); } =20 static inline size_t obj_full_size(struct kmem_cache *s) @@ -3493,6 +3496,7 @@ static inline size_t obj_full_size(struct kmem_cache = *s) bool __memcg_slab_post_alloc_hook(struct kmem_cache *s, struct list_lru *l= ru, gfp_t flags, size_t size, void **p) { + size_t obj_size =3D obj_full_size(s); struct obj_cgroup *objcg; struct slab *slab; unsigned long off; @@ -3533,6 +3537,7 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache *= s, struct list_lru *lru, for (i =3D 0; i < size; i++) { unsigned long obj_exts; struct slabobj_ext *obj_ext; + struct obj_stock_pcp *stock; =20 slab =3D virt_to_slab(p[i]); =20 @@ -3552,9 +3557,20 @@ bool __memcg_slab_post_alloc_hook(struct kmem_cache = *s, struct list_lru *lru, * TODO: we could batch this until slab_pgdat(slab) changes * between iterations, with a more complicated undo */ - if (obj_cgroup_charge_account(objcg, flags, obj_full_size(s), - slab_pgdat(slab), cache_vmstat_idx(s))) - return false; + stock =3D trylock_stock(); + if (!stock || !__consume_obj_stock(objcg, stock, obj_size)) { + size_t remainder; + + unlock_stock(stock); + if (__obj_cgroup_charge(objcg, flags, obj_size, &remainder)) + return false; + stock =3D trylock_stock(); + if (remainder) + __refill_obj_stock(objcg, stock, remainder, false); + } + __account_obj_stock(objcg, stock, obj_size, + slab_pgdat(slab), cache_vmstat_idx(s)); + unlock_stock(stock); =20 obj_exts =3D slab_obj_exts(slab); get_slab_obj_exts(obj_exts); @@ -3576,6 +3592,7 @@ void __memcg_slab_free_hook(struct kmem_cache *s, str= uct slab *slab, for (int i =3D 0; i < objects; i++) { struct obj_cgroup *objcg; struct slabobj_ext *obj_ext; + struct obj_stock_pcp *stock; unsigned int off; =20 off =3D obj_to_index(s, slab, p[i]); @@ -3585,8 +3602,13 @@ void __memcg_slab_free_hook(struct kmem_cache *s, st= ruct slab *slab, continue; =20 obj_ext->objcg =3D NULL; - refill_obj_stock(objcg, obj_size, true, -obj_size, - slab_pgdat(slab), cache_vmstat_idx(s)); + + stock =3D trylock_stock(); + __refill_obj_stock(objcg, stock, obj_size, true); + __account_obj_stock(objcg, stock, -obj_size, + slab_pgdat(slab), cache_vmstat_idx(s)); + unlock_stock(stock); + obj_cgroup_put(objcg); } } --=20 2.53.0