From nobody Sun Feb 8 10:33:05 2026 Received: from mail-yw1-f171.google.com (mail-yw1-f171.google.com [209.85.128.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E140B7261E for ; Thu, 2 Oct 2025 20:46:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759438002; cv=none; b=iFH3JgIeMty3BJqg/alThvm2tKjX1DqOmUZ4X9ac1hZoho8SlozKBJD4qHMgpNMNhF0ZvNFrENMzbr1E9bw6yYlWLrXD01PTmOgtjtHhL5K9s4do37o76rc4ntFXmtYejdeWtyrHmGoRxp7d52mfDnkf+tj33y3wVL7XvR1Xeig= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759438002; c=relaxed/simple; bh=W9YCvKP7YXSdEluBH6DcHRoGcPV0kHwJhzHayQHvHbc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ttyt/sbCMp49mqp8N1Q8LcVYqO/jrML6yeo5HXvbenvRZ00+B63T5J6ln12NYLSUAixBL/Cw5dCiScq7aqbwaspcEued1GTCotcvRjYzA/GSuqcL2MCJOY7yJcyjY6fOh7qLsj4S820xjJHcFe4kEDzXD2E0Ai9e04L8v968H1E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=WKZcpUuI; arc=none smtp.client-ip=209.85.128.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WKZcpUuI" Received: by mail-yw1-f171.google.com with SMTP id 00721157ae682-71d60110772so16013747b3.0 for ; Thu, 02 Oct 2025 13:46:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1759438000; x=1760042800; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=2jwps1lz6jxYNn7Dnn5Z09tOaOeYs2HTQuW80UbgHqQ=; b=WKZcpUuIPT7AZBoCpxHltcYUL3w80yVQZBNSshyJS2OZgpZVrX7V/kLX7ZXEj61f5e L/fV/EgGjF8UQRAkfJF1dVgJuOVRMYj9Q9KROjsaikAimE7jgQm+8leEpxgeQidN8Hfb HQPZrYJRsVplBe6c6Eeo0Hih1EoFgG+o9tFGsvewjHDGFqWoqqqSNv3LWEYh93qqV9Nz iQzcMxDwRcVMoLt3JJf0vthwy+bKZq5HqPbs4f6m7hNvzftMtMFQJUa2YzjXxuF0MiOs u8jyZc+nChDTcm4LirtXltWHghQuub9ZS7k2q1oOh4k1rmR5xJb4+5et188dr+HkYBwa ikNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759438000; x=1760042800; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2jwps1lz6jxYNn7Dnn5Z09tOaOeYs2HTQuW80UbgHqQ=; b=khYgaBfRX4KqboSYE4Ex0fOMIh60Ul/NpwENvyTugTr05bU2NloRKVxxpy/fLicrZ9 8ucYKnS9hX/mXRxpBp++O2lieuxKg36SCfT9vg5T1XI6XE/vHVfZRUkTpPni4aCDBicO l7rDiPoASYlXakXmYpwEEG+jlWsjLGpbk7KrKsYMZdcGDRsBn19C2S8+ZQcNM+19IJbI Cxy1Ua2ujovuTwmGQhycSoiuJTf8S80rsfdEbFZmOMoBudksfLoTFzP6d+bkNYTh783c INXC4ivAalLeCfFdil8K8fbZnme0KeoEXDFY8XU8sNnPPVMawQdckvI1LWd9G78ABs1I ndIg== X-Forwarded-Encrypted: i=1; AJvYcCVCU6UwKINTSEqz6nxJyvXQsvzE1YqOOlmF8zceq/eEgflMxDU42z4wyi5ubWOm5WIEpSpAZ+iLadthLq0=@vger.kernel.org X-Gm-Message-State: AOJu0YytInoCnz8kgkgzfTG5FkGdTEy9sjvQsU1lLLKXnN+uW4YLZB8B E5h1TRFPIODOQKqqN188PX914JFpOmj5gPT7jJsNe1oq7UMYpk4Te8hH X-Gm-Gg: ASbGncs82xLwNjt9zh4cJqOn1P+7a2sMGC4VX6Ddc8Kxeivm7PNUY7OW0/QRmzT63lQ JW5Y2IjYOGQjYD94IRpmY59obCe7RF0bMM0MAzohHdHDaom40SGINiUjFbG99Mbzhu7jUdCBeQq wews23uXHyJ/am8XevVifq9aIkvr5fWkwPEIQ4I4ApzxaxJ0fU2eXYPyf2sWTKXRHvQG8c41jbA 16xhx8iSa6FookYcrtpv93nXYSYPQJZ1PetrjmduIbhm663AhJPCfaa/5BNE8WsiARRYq2EWqma 2FSXLw2NDr0/996lLIr+pnvOWJLjXuOdlQQEIHhq3aWaspxzMgUiPiXiKKetGEodXOmOY4pXv18 GUCu5DswqoUKL+D/phUU4c29/GxlA68RyU6ZS4wpwYFqKO5i6mGH+F8RpQOErwShFCfPQpuZuv7 8lBw== X-Google-Smtp-Source: AGHT+IHzWFTSbAUSJS7YF3yHmxIfnxGDKF6Uqmg2CyEoFyCqQ6BTPK8MNk89Gdh1FJmam2OVjyxg6Q== X-Received: by 2002:a05:690c:6e8e:b0:772:3578:2ec5 with SMTP id 00721157ae682-77f942c2c16mr14178777b3.0.1759437999539; Thu, 02 Oct 2025 13:46:39 -0700 (PDT) Received: from localhost ([2a03:2880:25ff:5b::]) by smtp.gmail.com with ESMTPSA id 00721157ae682-77f81e370c7sm11291757b3.55.2025.10.02.13.46.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Oct 2025 13:46:39 -0700 (PDT) From: Joshua Hahn To: Andrew Morton Cc: Chris Mason , Kiryl Shutsemau , "Liam R. Howlett" , Brendan Jackman , David Hildenbrand , Johannes Weiner , Lorenzo Stoakes , Michal Hocko , Mike Rapoport , Suren Baghdasaryan , Vlastimil Babka , Zi Yan , linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel-team@meta.com Subject: [PATCH v3 1/3] mm/page_alloc/vmstat: Simplify refresh_cpu_vm_stats change detection Date: Thu, 2 Oct 2025 13:46:31 -0700 Message-ID: <20251002204636.4016712-2-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20251002204636.4016712-1-joshua.hahnjy@gmail.com> References: <20251002204636.4016712-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Currently, refresh_cpu_vm_stats returns an int, indicating how many changes were made during its updates. Using this information, callers like vmstat_update can heuristically determine if more work will be done in the future. However, all of refresh_cpu_vm_stats's callers either (a) ignore the result, only caring about performing the updates, or (b) only care about whether changes were made, but not *how many* changes were made. Simplify the code by returning a bool instead to indicate if updates were made. In addition, simplify fold_diff and decay_pcp_high to return a bool for the same reason. Signed-off-by: Joshua Hahn Reviewed-by: SeongJae Park Reviewed-by: Vlastimil Babka --- include/linux/gfp.h | 2 +- mm/page_alloc.c | 8 ++++---- mm/vmstat.c | 28 +++++++++++++++------------- 3 files changed, 20 insertions(+), 18 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 5ebf26fcdcfa..63c72cb1d117 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -386,7 +386,7 @@ extern void free_pages(unsigned long addr, unsigned int= order); #define free_page(addr) free_pages((addr), 0) =20 void page_alloc_init_cpuhp(void); -int decay_pcp_high(struct zone *zone, struct per_cpu_pages *pcp); +bool decay_pcp_high(struct zone *zone, struct per_cpu_pages *pcp); void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp); void drain_all_pages(struct zone *zone); void drain_local_pages(struct zone *zone); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 304e12bf2e4e..b9fc357d2d48 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2557,10 +2557,10 @@ static int rmqueue_bulk(struct zone *zone, unsigned= int order, * Called from the vmstat counter updater to decay the PCP high. * Return whether there are addition works to do. */ -int decay_pcp_high(struct zone *zone, struct per_cpu_pages *pcp) +bool decay_pcp_high(struct zone *zone, struct per_cpu_pages *pcp) { int high_min, to_drain, batch; - int todo =3D 0; + bool todo =3D false; =20 high_min =3D READ_ONCE(pcp->high_min); batch =3D READ_ONCE(pcp->batch); @@ -2573,7 +2573,7 @@ int decay_pcp_high(struct zone *zone, struct per_cpu_= pages *pcp) pcp->high =3D max3(pcp->count - (batch << CONFIG_PCP_BATCH_SCALE_MAX), pcp->high - (pcp->high >> 3), high_min); if (pcp->high > high_min) - todo++; + todo =3D true; } =20 to_drain =3D pcp->count - pcp->high; @@ -2581,7 +2581,7 @@ int decay_pcp_high(struct zone *zone, struct per_cpu_= pages *pcp) spin_lock(&pcp->lock); free_pcppages_bulk(zone, to_drain, pcp, 0); spin_unlock(&pcp->lock); - todo++; + todo =3D true; } =20 return todo; diff --git a/mm/vmstat.c b/mm/vmstat.c index bb09c032eecf..98855f31294d 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -771,25 +771,25 @@ EXPORT_SYMBOL(dec_node_page_state); =20 /* * Fold a differential into the global counters. - * Returns the number of counters updated. + * Returns whether counters were updated. */ static int fold_diff(int *zone_diff, int *node_diff) { int i; - int changes =3D 0; + bool changed =3D false; =20 for (i =3D 0; i < NR_VM_ZONE_STAT_ITEMS; i++) if (zone_diff[i]) { atomic_long_add(zone_diff[i], &vm_zone_stat[i]); - changes++; + changed =3D true; } =20 for (i =3D 0; i < NR_VM_NODE_STAT_ITEMS; i++) if (node_diff[i]) { atomic_long_add(node_diff[i], &vm_node_stat[i]); - changes++; + changed =3D true; } - return changes; + return changed; } =20 /* @@ -806,16 +806,16 @@ static int fold_diff(int *zone_diff, int *node_diff) * with the global counters. These could cause remote node cache line * bouncing and will have to be only done when necessary. * - * The function returns the number of global counters updated. + * The function returns whether global counters were updated. */ -static int refresh_cpu_vm_stats(bool do_pagesets) +static bool refresh_cpu_vm_stats(bool do_pagesets) { struct pglist_data *pgdat; struct zone *zone; int i; int global_zone_diff[NR_VM_ZONE_STAT_ITEMS] =3D { 0, }; int global_node_diff[NR_VM_NODE_STAT_ITEMS] =3D { 0, }; - int changes =3D 0; + bool changed =3D false; =20 for_each_populated_zone(zone) { struct per_cpu_zonestat __percpu *pzstats =3D zone->per_cpu_zonestats; @@ -839,7 +839,8 @@ static int refresh_cpu_vm_stats(bool do_pagesets) if (do_pagesets) { cond_resched(); =20 - changes +=3D decay_pcp_high(zone, this_cpu_ptr(pcp)); + if (decay_pcp_high(zone, this_cpu_ptr(pcp))) + changed =3D true; #ifdef CONFIG_NUMA /* * Deal with draining the remote pageset of this @@ -861,13 +862,13 @@ static int refresh_cpu_vm_stats(bool do_pagesets) } =20 if (__this_cpu_dec_return(pcp->expire)) { - changes++; + changed =3D true; continue; } =20 if (__this_cpu_read(pcp->count)) { drain_zone_pages(zone, this_cpu_ptr(pcp)); - changes++; + changed =3D true; } #endif } @@ -887,8 +888,9 @@ static int refresh_cpu_vm_stats(bool do_pagesets) } } =20 - changes +=3D fold_diff(global_zone_diff, global_node_diff); - return changes; + if (fold_diff(global_zone_diff, global_node_diff)) + changed =3D true; + return changed; } =20 /* --=20 2.47.3 From nobody Sun Feb 8 10:33:06 2026 Received: from mail-yw1-f170.google.com (mail-yw1-f170.google.com [209.85.128.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 28FBB1C4A10 for ; Thu, 2 Oct 2025 20:46:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759438003; cv=none; b=dS4VjP7+FqJe6L/PGsWOle2wk3jzVzEzqC4Jra3bZp3cbv11KQQUbYyoGF3etp3NMZdmR5E6BspwRbhQFzGVIuCqNa4y3vm/OECNcRH/rGnzHCK7fAwEamBqYnHm+ujdgwgSODEc8ngD9rZIB/4bErn5pZq+WknSZa0H5FjHIMw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759438003; c=relaxed/simple; bh=LpQAUeGmcEFMDxio1VSUeBPLxCuWe9iLxX++cXo0OC8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kBRFQtZ6iPLQhbzqVhchHw0PawSWNb1QQgaoubgErzh2Vg5rxiG0lkXYeSqWj24JnZ0NF45lOnByOhf1KnV2sP/gIGCu8WrmAgL5l09eIw4KOSADhvn+FjsELgoi7VyGVFStqSNphqcsH7gXFfoe0iTL5ez6l3p2bUJ/aYcdKJQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Um8Uxy3J; arc=none smtp.client-ip=209.85.128.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Um8Uxy3J" Received: by mail-yw1-f170.google.com with SMTP id 00721157ae682-71d6051aeafso16483957b3.2 for ; Thu, 02 Oct 2025 13:46:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1759438001; x=1760042801; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Mtg5K7mFEUovxKHIKqKvsg9K7Q6jEbo/PlbyX0ggh/g=; b=Um8Uxy3JxqRfdTagbLel+7wHMclWZlYyAMXR/P0nUe5FESqPJjRJiveQvb5K6s5hUl 0bKKH6RI6TKhJc/lZderkOExbMdaJ3hL1sgWtxrjTPSq+pujwxLtarn6+UOy7bRi9xnx 0qyW52IdOFXmcoPz/E/20OnLnDaZjQ2S4w4+VXBpE8Lz1Kajj/rZYKZiBQmmq9jPm63Q KKZBvuug5H4aFBa6Tt56Jof5UbDwhLFcaNbdlF/5R50Nj2VHWQjJE64xfJilyz+XPVgh Vbgcn+zy9NgcUb+k6xHRiUySzkfzxMbGQbmoVUCYeIe6ryopH+ZWzms7WvluLlyYWIT+ rY7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759438001; x=1760042801; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Mtg5K7mFEUovxKHIKqKvsg9K7Q6jEbo/PlbyX0ggh/g=; b=rJ6huRzBucO2gpYJp+bqc7/lnXYLqAUt2FpRk/CFBxeON8cJcOzgIYqCycF4wv546+ yqoIgJlh3H7XLyHRdrHTKkx1ar83sdGUnieMnA9em+VDkzUxZZFGIq0+IIhLfwmw5emo RgDUCuILtE0g4ubhv+RwQbHoh2W7i9hQjZMrhK/+wG2FUUBF4uuTMwlYcQ0EjQbRBLv4 kgau2GgzDXlp7xVGNvU0c712xQDO+3SlY26PP7agaE3z1/LvxFS+jNvB7ViT7etqybkE inXQVnmy3YshYW1m1PdNSsdCq2iMds+DtDqaHPH8Xokwa0JSyqZEuTDeisba5cINi7BZ MrqA== X-Forwarded-Encrypted: i=1; AJvYcCX+wKXFCEuWDHH0YHLh0ujGGBAdGvK7P8f9cyJH4BVspO82mrCMFnQi0q07QfEppsOU1ea8EBw6gLWsTLg=@vger.kernel.org X-Gm-Message-State: AOJu0YwOB1pNcsNwp23MF6wsWBu7faxcZEPmYz9kF/VWQZ1eY77ZdZSK ww7Z2VGPkMVNkozlsbeVbFIjvmnuxXSvDEhdU7MHMRJD2IkcZVZq+1yG X-Gm-Gg: ASbGncv8++qKLHBLTe09rGCgBmDmPD6UaNDJLID8hZS5hnGcpd70l2CQSyAELkQ7UlW 1F2qjwsQRgDU4K5maLDjR4NEx0x7cVuhicNASvrouXrUoTrzcEi43dsY4yhfNrDfAlU4NLV+td6 9ii+meCxB2VNarbYWeT1UVa/hCgWdYMVL1FCjNkoMSl/V8WBHU3R1WAW9tg1lJWQkIEHvtZM1oo rhI3D9IkfS1QfE2/aPGrojFxYewRjvYEmegzhG2jBHH5qSmqqdkqSrAJBAPGyCHUl58JWuDhslP 4KH/jL62tSMKH8lPs2WVdDZ7hu6T+oHIRmRbUg0F2RQ0ug+XO1vyEacCSwkKR3/dRLC0ApEAsOe HU+rt/aski9axHbqAu94cGuQA+faY8KJOxyB6gyPRQ+Obf0UELpz4+pJzffs9LrUZC1G+z6TbX0 YLHAYhE8IPh5+gLa+l/bd2IFoC X-Google-Smtp-Source: AGHT+IH1dK4l8EVgnO4Il3oHYv3U0XOv5jcIZNNZM8+Kj7vgvveFLxrmq74ZKJzTaehh0HFqLOPNkQ== X-Received: by 2002:a05:690c:6085:b0:772:2b5e:1f10 with SMTP id 00721157ae682-77f9456fdf6mr14998337b3.1.1759438001032; Thu, 02 Oct 2025 13:46:41 -0700 (PDT) Received: from localhost ([2a03:2880:25ff:74::]) by smtp.gmail.com with ESMTPSA id 00721157ae682-77f81e63e8bsm11224447b3.67.2025.10.02.13.46.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Oct 2025 13:46:40 -0700 (PDT) From: Joshua Hahn To: Andrew Morton Cc: Chris Mason , Kiryl Shutsemau , Brendan Jackman , Johannes Weiner , Michal Hocko , Suren Baghdasaryan , Vlastimil Babka , Zi Yan , linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel-team@meta.com Subject: [PATCH v3 2/3] mm/page_alloc: Batch page freeing in decay_pcp_high Date: Thu, 2 Oct 2025 13:46:32 -0700 Message-ID: <20251002204636.4016712-3-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20251002204636.4016712-1-joshua.hahnjy@gmail.com> References: <20251002204636.4016712-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" It is possible for pcp->count - pcp->high to exceed pcp->batch by a lot. When this happens, we should perform batching to ensure that free_pcppages_bulk isn't called with too many pages to free at once and starve out other threads that need the pcp lock. Since we are still only freeing the difference between the initial pcp->count and pcp->high values, there should be no change to how many pages are freed. Suggested-by: Chris Mason Suggested-by: Andrew Morton Co-developed-by: Johannes Weiner Signed-off-by: Joshua Hahn Reviewed-by: Vlastimil Babka --- mm/page_alloc.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index b9fc357d2d48..f525f197c5fd 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2559,7 +2559,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned i= nt order, */ bool decay_pcp_high(struct zone *zone, struct per_cpu_pages *pcp) { - int high_min, to_drain, batch; + int high_min, to_drain, to_drain_batched, batch; bool todo =3D false; =20 high_min =3D READ_ONCE(pcp->high_min); @@ -2577,11 +2577,14 @@ bool decay_pcp_high(struct zone *zone, struct per_c= pu_pages *pcp) } =20 to_drain =3D pcp->count - pcp->high; - if (to_drain > 0) { + while (to_drain > 0) { + to_drain_batched =3D min(to_drain, batch); spin_lock(&pcp->lock); - free_pcppages_bulk(zone, to_drain, pcp, 0); + free_pcppages_bulk(zone, to_drain_batched, pcp, 0); spin_unlock(&pcp->lock); todo =3D true; + + to_drain -=3D to_drain_batched; } =20 return todo; --=20 2.47.3 From nobody Sun Feb 8 10:33:06 2026 Received: from mail-yw1-f178.google.com (mail-yw1-f178.google.com [209.85.128.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B8D381EEA55 for ; Thu, 2 Oct 2025 20:46:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759438005; cv=none; b=ZKiyLk0uW+kCFZh41tOSnIguj3Gb/6l1XFejHIFB7NWSQs6DfvOAp8BSbKXiLFeI4UmlBwQhsIZMwDj4tkFEQ/Vq87Cenyfu88PpQKadO8FB5QRJmkA46427ZUtR1Y7YYfFDLA1CwaitK8PEdvDcep0N+UVGkw4KNWMPveeZM6U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1759438005; c=relaxed/simple; bh=JggwyU8+falGG8LJc+r4zOOOtJrz8MlfWHTD353hWew=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=E72z5sVXOd96rS6lgoasUqA/Sa3WhW7Hgq/pETP9cAdNY20nOHKATD+z+I1n1HlJW28KVN2kwdl/e27GXtBW+dFzETvXgXTss9efRskGQSy1j7RdAY6+dEIanOip2fdRgeyK2F8R/Iln3V9ybmjHz/x6zqf2cWWPYDkKofV3m5U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=DF4AxCyh; arc=none smtp.client-ip=209.85.128.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DF4AxCyh" Received: by mail-yw1-f178.google.com with SMTP id 00721157ae682-71d5fe46572so20425767b3.1 for ; Thu, 02 Oct 2025 13:46:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1759438003; x=1760042803; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=OBa7sIAa7lZB7AE0W4U5YndRgjFk5P+mjzsHi3y1lyA=; b=DF4AxCyhf2OEw+cY3IkDe9IGfe7P2UFQWFF68CCSB7SXUim7fDGbUHKf88bDPchVq+ i1uUYQgT+jvM50yxdMqxd5UhvlLoRYQ99fmnESQG+MUoOFCLlq1xONVkkLh1DRorHV6L J3CVzaK5Z+/PGJkyC3Ix3Z6enGY0Pt87jMhVRUfBGkWKjB8epagwnpLYdfW8rGWXnIbq 3qAx2wN7v6+I7lP7DsJvX1D8ez/Q164KlRlK+NXiOQvycOJfjFYyzNjv2F8oBT+2X+yz IWT67aqwhfLYF/6g89Nnww6L8EzkCITCLQXjV7P+diN6QiU+SFcJeACumaUE7jmlSEBj wKtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1759438003; x=1760042803; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=OBa7sIAa7lZB7AE0W4U5YndRgjFk5P+mjzsHi3y1lyA=; b=ZCNeYGwZP7nfV1/2iY9JW6PtpXyUy/7F2UQWME+5FHDnidYxQgoWYAWDy+eMfgFSOx 6lekcbr7c8w08wGND0hq7eddaiEr7C8iPgI7g/zjNWXkdUUsmOvO+reQEZl/pMnKjRyS buJh6GAQc/BfJXeH5vwSEqhhSVyvxKsmzFKlvVRAfYxIuH4TxiHSNgHES5aelKjOfZGu jPrNJoFCuSwvXMeJ61egJ+bg+WUEBACL1DdTeW8Z1L0UngbOBrX0YwDU/05jQTc9mnUi 1XkAiFBpFJ8eWVSOuidf6fnVaAsozDjkFSK9s3oyUGJroaRCxgMvmrNgJgnbpOmAwzXU Ouig== X-Forwarded-Encrypted: i=1; AJvYcCXD3Z57do0mQareRLOXZ/4+qNYfMocxX4dz7IYbf24z4ayDQo2KQUdhknNuSk+PzCO5xEAS3lr++GqzDEk=@vger.kernel.org X-Gm-Message-State: AOJu0YzStx9q9s+dinXbWm+8fDh9enBm5UiwsM9Wtzwwn/PcXOU9i9lN a7uM2inCgZ52vi0rn4aII4siB+T+BAJWAKPH0k7TPqvFh+X/8Jz3ISJS X-Gm-Gg: ASbGncufkntqePjHkgyQ34E8HBy1avmxGH8YiMqAKFRLst+qJ0tw1CdLUJ5tdpXofLz dauHxC0JSuBB0AutoGrvYO456mPp5VAihyRPJdhnj6ofbU2rpEMyiSNHQBs1f5arlrTXJYrmSPR budQ6oP1Eo2HIBAkh0PWvN3jBd+bqIwDKtZo+gcE2D5jfv9xR7XTX/ONqWNZc0YXsqwxnziKhPL CTFBN5h7r4njP/E9W1dE9c/en/gJuslMA7Yx2ORA3iJtYiVjj5pySdVMFjQgS3B82807bkrPSUa O2GPWtTKCVsoauivsMAAY7dTX9BqEK9LqVJdyvDHnQXp95qAe+0g1KCu4ydaEChihmy6PPoHDCR XOpx3yHfB3/CnFF9R5+3baZ9geAv67jm6w+03o3ZmrJZaFg6IbFsypbPl3C+hZTL/8hs/OpkvmL oN0wC4NIb4GIE= X-Google-Smtp-Source: AGHT+IENI2go9fghJpen+tUDrCCnqIsPZ10tsTLEeBnhI7cBoG/y7/sR3qgtxNTvBex2gqOGrkvgtA== X-Received: by 2002:a05:690c:a4c8:b0:77f:771e:fb78 with SMTP id 00721157ae682-77f81b6c90amr49034987b3.18.1759438002398; Thu, 02 Oct 2025 13:46:42 -0700 (PDT) Received: from localhost ([2a03:2880:25ff:2::]) by smtp.gmail.com with ESMTPSA id 00721157ae682-77f81d1444asm11412977b3.42.2025.10.02.13.46.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 Oct 2025 13:46:41 -0700 (PDT) From: Joshua Hahn To: Andrew Morton Cc: Chris Mason , Kiryl Shutsemau , Brendan Jackman , Johannes Weiner , Michal Hocko , Suren Baghdasaryan , Vlastimil Babka , Zi Yan , linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel-team@meta.com Subject: [PATCH v3 3/3] mm/page_alloc: Batch page freeing in free_frozen_page_commit Date: Thu, 2 Oct 2025 13:46:33 -0700 Message-ID: <20251002204636.4016712-4-joshua.hahnjy@gmail.com> X-Mailer: git-send-email 2.47.3 In-Reply-To: <20251002204636.4016712-1-joshua.hahnjy@gmail.com> References: <20251002204636.4016712-1-joshua.hahnjy@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Before returning, free_frozen_page_commit calls free_pcppages_bulk using nr_pcp_free to determine how many pages can appropritately be freed, based on the tunable parameters stored in pcp. While this number is an accurate representation of how many pages should be freed in total, it is not an appropriate number of pages to free at once using free_pcppages_bulk, since we have seen the value consistently go above 2000 in the Meta fleet on larger machines. As such, perform batched page freeing in free_pcppages_bulk by using pcp->batch member. In order to ensure that other processes are not starved of the zone lock, free both the zone lock and pcp lock to yield to other threads. Note that because free_frozen_page_commit now performs a spinlock inside the function (and can fail), the function may now return with a freed pcp. To handle this, return true if the pcp is locked on exit and false otherwis= e. In addition, since free_frozen_page_commit must now be aware of what UP flags were stored at the time of the spin lock, and because we must be able to report new UP flags to the callers, add a new unsigned long* parameter UP_flags to keep track of this. Suggested-by: Chris Mason Co-developed-by: Johannes Weiner Signed-off-by: Joshua Hahn --- mm/page_alloc.c | 66 ++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 57 insertions(+), 9 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index f525f197c5fd..9b9f5a44496c 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2818,12 +2818,21 @@ static int nr_pcp_high(struct per_cpu_pages *pcp, s= truct zone *zone, return high; } =20 -static void free_frozen_page_commit(struct zone *zone, +/* + * Tune pcp alloc factor and adjust count & free_count. Free pages to brin= g the + * pcp's watermarks below high. + * + * May return a freed pcp, if during page freeing the pcp spinlock cannot = be + * reacquired. Return true if pcp is locked, false otherwise. + */ +static bool free_frozen_page_commit(struct zone *zone, struct per_cpu_pages *pcp, struct page *page, int migratetype, - unsigned int order, fpi_t fpi_flags) + unsigned int order, fpi_t fpi_flags, unsigned long *UP_flags) { int high, batch; + int to_free, to_free_batched; int pindex; + int cpu =3D smp_processor_id(); bool free_high =3D false; =20 /* @@ -2861,15 +2870,20 @@ static void free_frozen_page_commit(struct zone *zo= ne, * Do not attempt to take a zone lock. Let pcp->count get * over high mark temporarily. */ - return; + return true; } =20 high =3D nr_pcp_high(pcp, zone, batch, free_high); if (pcp->count < high) - return; + return true; + + to_free =3D nr_pcp_free(pcp, batch, high, free_high); + if (to_free =3D=3D 0) + return true; =20 - free_pcppages_bulk(zone, nr_pcp_free(pcp, batch, high, free_high), - pcp, pindex); +free_batch: + to_free_batched =3D min(to_free, batch); + free_pcppages_bulk(zone, to_free_batched, pcp, pindex); if (test_bit(ZONE_BELOW_HIGH, &zone->flags) && zone_watermark_ok(zone, 0, high_wmark_pages(zone), ZONE_MOVABLE, 0)) { @@ -2887,6 +2901,35 @@ static void free_frozen_page_commit(struct zone *zon= e, next_memory_node(pgdat->node_id) < MAX_NUMNODES) atomic_set(&pgdat->kswapd_failures, 0); } + high =3D nr_pcp_high(pcp, zone, batch, free_high); + to_free -=3D to_free_batched; + if (pcp->count >=3D high) { + pcp_spin_unlock(pcp); + pcp_trylock_finish(*UP_flags); + + pcp_trylock_prepare(*UP_flags); + pcp =3D pcp_spin_trylock(zone->per_cpu_pageset); + if (!pcp) { + pcp_trylock_finish(*UP_flags); + return false; + } + + /* + * Check if this thread has been migrated to a different + * CPU. If that is the case, give up and indicate that + * the pcp is returned in an unlocked state. + */ + if (smp_processor_id() !=3D cpu) { + pcp_spin_unlock(pcp); + pcp_trylock_finish(*UP_flags); + return false; + } + } + + if (to_free > 0 && pcp->count >=3D high) + goto free_batch; + + return true; } =20 /* @@ -2934,7 +2977,9 @@ static void __free_frozen_pages(struct page *page, un= signed int order, pcp_trylock_prepare(UP_flags); pcp =3D pcp_spin_trylock(zone->per_cpu_pageset); if (pcp) { - free_frozen_page_commit(zone, pcp, page, migratetype, order, fpi_flags); + if (!free_frozen_page_commit(zone, pcp, page, migratetype, + order, fpi_flags, &UP_flags)) + return; pcp_spin_unlock(pcp); } else { free_one_page(zone, page, pfn, order, fpi_flags); @@ -3034,8 +3079,11 @@ void free_unref_folios(struct folio_batch *folios) migratetype =3D MIGRATE_MOVABLE; =20 trace_mm_page_free_batched(&folio->page); - free_frozen_page_commit(zone, pcp, &folio->page, migratetype, - order, FPI_NONE); + if (!free_frozen_page_commit(zone, pcp, &folio->page, + migratetype, order, FPI_NONE, &UP_flags)) { + pcp =3D NULL; + locked_zone =3D NULL; + } } =20 if (pcp) { --=20 2.47.3