From nobody Tue Dec 2 02:18:56 2025 Received: from mail-wm1-f51.google.com (mail-wm1-f51.google.com [209.85.128.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 47AD5285CAA for ; Wed, 19 Nov 2025 21:08:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763586516; cv=none; b=sTnl764WBsq7ZczAEdM7+5hbrUlws2VcDfeY47wwOV3czaa3GgPxowtNoyDo6GRZHeh2GWZOGglCQpTjrftJ7n7iB0PVfBeHgq7EsbdO9OpPSAaFMXW/TwAUQy8t9gUz1GcT+2gS6xXKgt/jxtnfAHr9NgPVquKsQWwoHhF3Iv0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1763586516; c=relaxed/simple; bh=440SnwIdg1HEk2eIUgXjLOOMaUMWM/e3pqkoAkB7unc=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=ZlU7sC5Hf7pjhGpfHC6dQfRkObaM9QA/i0qRapV8zizRZLOfYCHuZazY8aafcGmKbUvEyJ7m/LG/SSk/y9mpK2btNrR54D/vh6gMFJav4XIxelY7fxygaIlFExrS5MF+6W2/J80RYusnChN3Cp40UJDpDN2CdoS9NaJ+jB39lpI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JXXIcZyp; arc=none smtp.client-ip=209.85.128.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JXXIcZyp" Received: by mail-wm1-f51.google.com with SMTP id 5b1f17b1804b1-47775fb6cb4so1581485e9.0 for ; Wed, 19 Nov 2025 13:08:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1763586512; x=1764191312; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=/x+2YkUzIKw5LUN7QUbaCv0UtaIYtv82+zCXS2emyik=; b=JXXIcZypMRG+1d4RzrsHRNXMMMwSToZWVxcCN6/gI9RU0m+a1kq7dwXwj2M9XYvQrj QjWHL1k8wcOaklspo+K/5DdsJoMuYyavxcjqLM6tX+y/dHC1Cqy7Fo0bI265Nt/vcRqG Cc4yVQoAg20MJaf016FGes1Vo5KLsBH99cKOu8W1jw7KmtVTZ489ffJvxIjWLN2mSvXl fIri5x1Xrsy3FU6XylKnBrn5xIWMpHd+0JBaNj8avTa2I0sQnl/gjd+OorW5PqHQZYac 66rk87NBpQZTPnDpLbC50XobToJgC1BNUnxPRocab9LbwnWnw2E0DIQnxQtLDsIz7595 yklA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1763586512; x=1764191312; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=/x+2YkUzIKw5LUN7QUbaCv0UtaIYtv82+zCXS2emyik=; b=TtNk2GarmaMidZoEejNjtPtdhoN1k1+RPxVQ6jhSDijMeWxn+z34B/2Ow9Sfj3jPAM hJu2kR/UVOz5RRosLjSjGZFjMHuH3xKbFfKFcU8wSZfUdvR6WtSSV1MLDc3M6USF1dV/ ITtCkNIUeW2gZpJasU5LMC4iklPit4MJ0LFibU3RH17QT0bFlnvuqXf7ASsvyDkRUTXv frif292qP0a0nAQnRbSwt21BSNn4jvuXGBbPe4SbJ/EPufoNq7PWNTcLDde96UktmvML 6X72Be0cMoAFMe3S0KuEF/EUoaOsn0GEC/FRYw7HefI6p3x90EZISvG2VRHPjNm96ON9 Hmpw== X-Forwarded-Encrypted: i=1; AJvYcCWuAkzryfeZo5yj6iEnh8zU6C7+Wpiw/u+HSZtiM+ntYqPMjcP8tYpF+deVhfhTGapXlUwt41eerB4x3CA=@vger.kernel.org X-Gm-Message-State: AOJu0Yyt3W+y0oKNEoZGSoNGQBIB5856/S3T9rTnL7NpMt1bzGY4hTfD 8ZQ+DN6WpdatiQczNhQAzC7MYqYqv5Rdd/ioomqlec6CPfZ5NTyqD0j1 X-Gm-Gg: ASbGncur+nnhMDPjWXj9jaM1vyP15ASnbuSkk5KylgOxHmRvcTf+YyXFpQ0TzVz0212 HpU/vOAJMLgpWlCO4Q2lIRljzGhLmQDlc3pCpKk4oDrqK9Dpu3L0Pqv4QfG/xFXqVerCITo2C20 cRQFWxGR0h7A8vI3AVRofJYRwyJHApTMKv+7+YGlwfDJ+SzHsChfNGMTdCKNvFNZjsrA84BWKWz cDTCKK8/2V6zK3mW6Wr6H8B+n/wsnVvXXVLp4JaU8zwmTKPQjQE07aZsP8G+HYAcrv2b6hm3i37 Ym4ivPeSKEnQL+roGEiDUuZmq+pUESjqpijYQeUL3nMDNsUy0uhgMZGUiozvH08N5Hk+SwyMuXf eWjy8lZdoJKOBLNjadd85b+6e1/qPjap75XyjkW2t9zp5adj9oa5zcCRUnEGxTPR1ZsoFS7gAHh qCxWu3mf4DDwYa0FQjQplpEyGU++hqZ6lIP6jNoIOATJYwxwEQqhEyQa/HwmI= X-Google-Smtp-Source: AGHT+IGu/JcMLbFe1ns1xbHqyUMg1819xYufBVBDVVdgsDa4/PT7CZf0ZP5BjX/6EqeQTZop9RePOw== X-Received: by 2002:a05:600c:19d3:b0:477:b0b8:4dd0 with SMTP id 5b1f17b1804b1-477b8a8f296mr6261345e9.17.1763586512466; Wed, 19 Nov 2025 13:08:32 -0800 (PST) Received: from f.. (cst-prg-14-82.cust.vodafone.cz. [46.135.14.82]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-477b831421fsm9558365e9.10.2025.11.19.13.08.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 19 Nov 2025 13:08:31 -0800 (PST) From: Mateusz Guzik To: dennis@kernel.org Cc: akpm@linux-foundation.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Mateusz Guzik Subject: [PATCH] percpu_counter: reduce i-cache footprint of percpu_counter_add_batch() fast path Date: Wed, 19 Nov 2025 22:08:20 +0100 Message-ID: <20251119210820.2959128-1-mjguzik@gmail.com> X-Mailer: git-send-email 2.43.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When compiled with gcc 14.2 for the x86-64 architecture with ORC frame unwinder the fast path still has the most unfortunate size of 66 bytes, in part from register spilling to falicitate the fallback. Moving it out solves the problem by keeping it just below 64 bytes. Signed-off-by: Mateusz Guzik --- lib/percpu_counter.c | 30 ++++++++++++++++++++---------- 1 file changed, 20 insertions(+), 10 deletions(-) diff --git a/lib/percpu_counter.c b/lib/percpu_counter.c index 2891f94a11c6..0cf6f1101903 100644 --- a/lib/percpu_counter.c +++ b/lib/percpu_counter.c @@ -89,24 +89,34 @@ EXPORT_SYMBOL(percpu_counter_set); * Safety against interrupts is achieved in 2 ways: * 1. the fast path uses local cmpxchg (note: no lock prefix) * 2. the slow path operates with interrupts disabled + * + * Slowpath is implemented as a separate routine to reduce register spilla= ge by gcc. */ -void percpu_counter_add_batch(struct percpu_counter *fbc, s64 amount, s32 = batch) +static void noinline percpu_counter_add_batch_slowpath(struct percpu_count= er *fbc, + s64 amount, s32 batch) { s64 count; unsigned long flags; =20 + raw_spin_lock_irqsave(&fbc->lock, flags); + /* + * Note: by now we might have migrated to another CPU or the value + * might have changed. + */ + count =3D __this_cpu_read(*fbc->counters); + fbc->count +=3D count + amount; + __this_cpu_sub(*fbc->counters, count); + raw_spin_unlock_irqrestore(&fbc->lock, flags); +} + +void percpu_counter_add_batch(struct percpu_counter *fbc, s64 amount, s32 = batch) +{ + s64 count; + count =3D this_cpu_read(*fbc->counters); do { if (unlikely(abs(count + amount) >=3D batch)) { - raw_spin_lock_irqsave(&fbc->lock, flags); - /* - * Note: by now we might have migrated to another CPU - * or the value might have changed. - */ - count =3D __this_cpu_read(*fbc->counters); - fbc->count +=3D count + amount; - __this_cpu_sub(*fbc->counters, count); - raw_spin_unlock_irqrestore(&fbc->lock, flags); + percpu_counter_add_batch_slowpath(fbc, amount, batch); return; } } while (!this_cpu_try_cmpxchg(*fbc->counters, &count, count + amount)); --=20 2.48.1