From nobody Fri Jun 12 05:52:00 2026 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46D793358CA for ; Wed, 8 Apr 2026 10:06:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775642814; cv=none; b=tdT5ndFeYYyttvWQKXwfDobx/clcNohxZhabsiLaHgYomxX969rHFCQaFfRhM7QXp4WEUstgZaq5o+hd9qvcLlptuaEFvWhjNHNRzwPk1Ov5BMQXEKKZoSYWbG8W7qRcLsEORVNwAJG+eLyKsf9U+pRbQBrOaETMYJqHJHLrPhM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1775642814; c=relaxed/simple; bh=BQD0rQ4+SuGgZDdFZXw6+M4nfywYpPxeOB4QvGXvn14=; h=Date:Mime-Version:Message-ID:Subject:From:To:Cc:Content-Type; b=FjefI4wivleInfsX5Uzq8tlwU1KKOFfFax5j4+dTN1JijILSsMGpY50xp5Zbcrtprs8Ap8181/cng/mg0RNZrUKcyUX8Yna0T3Lf2twHOmKwuz060v5+2yq8D0g305KPGDsKVVJwZ6pd2a3FbaBk2JVbJUMy9skfvBp1P/pz1Lc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--joonwonkang.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=QzLxbgcd; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--joonwonkang.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="QzLxbgcd" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2b241be0126so162592725ad.3 for ; Wed, 08 Apr 2026 03:06:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1775642806; x=1776247606; darn=vger.kernel.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=2qPBZiQ6FrhRd8WblhKjUR/UPSv9+M4JMXeTTgxKUw8=; b=QzLxbgcdwvrmoVwRKz7vkOmuzqsbi0Y8+xTepU+CXTrf9DR1ZN9yw5HvoKN3oksWQQ MdFSZikzh7Z32CjNlpkt9G7zS5UqGac1SPKoJSdzbf0RyFeZNrwHxQMj26yEfml5uPy7 2FUilnx6dF03i8CmhJ36zt2iM2BvoLffHltlWwyMfv12acjG6r1KDtl/5YpzCpnaMZp3 522N4YE1Pt57oHvOvzxDXzCE7E2ZvIMsQanDS1QtB5PEzEL+Lgc3+qYHUqcbLpaplKr1 nOvQ91HbcA62kf+2zHWfM2GLjiRp+KOehKQS9VYIdGw19kYS40UHT7953iyEHkGOg6yW cvKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1775642806; x=1776247606; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=2qPBZiQ6FrhRd8WblhKjUR/UPSv9+M4JMXeTTgxKUw8=; b=iJI9jc24AWgNMEEl1uli3PKYYG+xzEkrbuWZuMxq4nI3Ve3Zrwz23WOhme2cklMIE3 Wr8rSBjFXicNwx385NL01krrbeE4WVJGrUAVCOp7bdMmKUqT33xAIWo/28AHclkLPA2o DMA/ZYF4mxt5q+liOkgyDcB3E2yso74ekgvTcOURqUpJlUwzl6HKlGuHWa4qjPcOgQ79 KiKl+yJTbb4r0EGb//VOaCSMMT8w/zh+ASXr74ZXrkBDHWmvfGJ9gbrejB251M5b1aBw 7LZpZD3AjjMHp5SMIpMtqF47ZnfczU+e8QZ3xu67pqWB+LfEE1dyilAQDYxubtPBZV4S mwpQ== X-Forwarded-Encrypted: i=1; AJvYcCVKybVEl8VRDhGCJvMSgoxU7OGwwMUL/PCWdKMlpatgv8V3dVvc7co5gGEaXkZxUQjtgNdzfoE3XGZY0V0=@vger.kernel.org X-Gm-Message-State: AOJu0YyPA1u9zpJmwOhiwWH/2T9uhG0W7KtFtIBCFVBLm7SUcEgG17Pz GxVgDcFkvz1LsyAe1idREl0/rxT+/1AHpFmBTQFHl5r7UxhdLhW/n1VMBZHQs2sS6caysUbCLRf 1Jtgeejzhd1X9F7d1HtvsFY15bg== X-Received: from plsp8.prod.google.com ([2002:a17:902:bd08:b0:2b0:af76:bcc2]) (user=joonwonkang job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:17cb:b0:2b2:58c7:2cd6 with SMTP id d9443c01a7336-2b28167569cmr243068675ad.5.1775642805735; Wed, 08 Apr 2026 03:06:45 -0700 (PDT) Date: Wed, 8 Apr 2026 10:06:42 +0000 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Mailer: git-send-email 2.53.0.1213.gd9a14994de-goog Message-ID: <20260408100642.83919-1-joonwonkang@google.com> Subject: [PATCH v2] percpu: Fix hint invariant breakage From: Joonwon Kang To: dennis@kernel.org, tj@kernel.org, cl@gentwo.org Cc: akpm@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, dodam@google.com, Joonwon Kang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The invariant "scan_hint_start > contig_hint_start if and only if scan_hint =3D=3D contig_hint" should be kept for hint management. However, it could be broken in some cases: - if (new contig =3D=3D contig_hint =3D=3D scan_hint) && (contig_hint_sta= rt < scan_hint_start < new contig start) && the new contig is to become a new contig_hint due to its better alignment, then scan_hint should be invalidated instead of keeping the old value. - if (new contig =3D=3D contig_hint > scan_hint) && (new contig start < contig_hint_start) && the new contig is not to become a new contig_hint, then scan_hint should be not updated to the new contig. This commit refactors the percpu block update code to make it more visible on what to consider, e.g. when the new contig overlaps with the old contig_hint or scan_hint, fixes the invariant breakage and also optimizes scan_hint further. Some of the optimization cases when no overlap occurs are: - if (new contig > contig_hint > scan_hint) && (scan_hint_start < new contig start < contig_hint_start), then keep scan_hint instead of invalidating it. - if (new contig > contig_hint =3D=3D scan_hint) && (contig_hint_start < new contig start < scan_hint_start), then update scan_hint to the old contig_hint instead of invalidating it. - if (new contig =3D=3D contig_hint > scan_hint) && (new contig start < contig_hint_start) && the new contig is to become a new contig_hint due to its better alignment, then update scan_hint to the old contig_hint instead of invalidating or keeping it. Signed-off-by: Joonwon Kang --- v1 -> v2: Consider cases where the new contig overlaps with the existing contig_hint or scan_hint. mm/percpu.c | 124 +++++++++++++++++++++++++++++++++++----------------- 1 file changed, 85 insertions(+), 39 deletions(-) diff --git a/mm/percpu.c b/mm/percpu.c index 81462ce5866e..57fa67f3726d 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -630,6 +630,11 @@ static inline bool pcpu_region_overlap(int a, int b, i= nt x, int y) static void pcpu_block_update(struct pcpu_block_md *block, int start, int = end) { int contig =3D end - start; + int scan_hint_cand_1 =3D 0; + int scan_hint_cand_1_start =3D 0; + int scan_hint_cand_2 =3D 0; + int scan_hint_cand_2_start =3D 0; + bool overlap_with_contig_hint; =20 block->first_free =3D min(block->first_free, start); if (start =3D=3D 0) @@ -638,57 +643,98 @@ static void pcpu_block_update(struct pcpu_block_md *b= lock, int start, int end) if (end =3D=3D block->nr_bits) block->right_free =3D contig; =20 + if (block->contig_hint =3D=3D 0) { + block->contig_hint =3D contig; + block->contig_hint_start =3D start; + block->scan_hint =3D 0; + return; + } + + overlap_with_contig_hint =3D pcpu_region_overlap( + start, end, block->contig_hint_start, + block->contig_hint_start + block->contig_hint); + if (contig > block->contig_hint) { - /* promote the old contig_hint to be the new scan_hint */ - if (start > block->contig_hint_start) { - if (block->contig_hint > block->scan_hint) { - block->scan_hint_start =3D - block->contig_hint_start; - block->scan_hint =3D block->contig_hint; - } else if (start < block->scan_hint_start) { - /* - * The old contig_hint =3D=3D scan_hint. But, the - * new contig is larger so hold the invariant - * scan_hint_start < contig_hint_start. - */ - block->scan_hint =3D 0; - } - } else { - block->scan_hint =3D 0; + if (!overlap_with_contig_hint) { + scan_hint_cand_1 =3D block->contig_hint; + scan_hint_cand_1_start =3D block->contig_hint_start; } - block->contig_hint_start =3D start; + block->contig_hint =3D contig; + block->contig_hint_start =3D start; } else if (contig =3D=3D block->contig_hint) { if (block->contig_hint_start && (!start || __ffs(start) > __ffs(block->contig_hint_start))) { - /* start has a better alignment so use it */ + scan_hint_cand_1 =3D block->contig_hint; + scan_hint_cand_1_start =3D block->contig_hint_start; + + /* Start has a better alignment so use it. */ block->contig_hint_start =3D start; - if (start < block->scan_hint_start && - block->contig_hint > block->scan_hint) - block->scan_hint =3D 0; - } else if (start > block->scan_hint_start || - block->contig_hint > block->scan_hint) { - /* - * Knowing contig =3D=3D contig_hint, update the scan_hint - * if it is farther than or larger than the current - * scan_hint. - */ - block->scan_hint_start =3D start; - block->scan_hint =3D contig; + } else { + if (!overlap_with_contig_hint) { + scan_hint_cand_1 =3D contig; + scan_hint_cand_1_start =3D start; + } } } else { /* - * The region is smaller than the contig_hint. So only update - * the scan_hint if it is larger than or equal and farther than - * the current scan_hint. + * Consider only when the new contig is larger than or equal to + * the old scan hint. */ - if ((start < block->contig_hint_start && - (contig > block->scan_hint || - (contig =3D=3D block->scan_hint && - start > block->scan_hint_start)))) { - block->scan_hint_start =3D start; - block->scan_hint =3D contig; + if (contig >=3D block->scan_hint) { + scan_hint_cand_1 =3D contig; + scan_hint_cand_1_start =3D start; + } + } + + if (block->scan_hint && + !pcpu_region_overlap(start, end, block->scan_hint_start, + block->scan_hint_start + block->scan_hint)) { + scan_hint_cand_2 =3D block->scan_hint; + scan_hint_cand_2_start =3D block->scan_hint_start; + } + + /* Make scan_hint_cand_1 be the best candidate for the new scan hint. */ + if ((scan_hint_cand_2 > scan_hint_cand_1) || + (scan_hint_cand_2 =3D=3D scan_hint_cand_1 && + scan_hint_cand_2_start > scan_hint_cand_1_start)) { + int tmp_hint =3D scan_hint_cand_1; + int tmp_hint_start =3D scan_hint_cand_1_start; + + scan_hint_cand_1 =3D scan_hint_cand_2; + scan_hint_cand_1_start =3D scan_hint_cand_2_start; + scan_hint_cand_2 =3D tmp_hint; + scan_hint_cand_2_start =3D tmp_hint_start; + } + + /* + * At this point, it is guaranteed that none of the scan hint + * candidates overlaps with the new contig hint while they may overlap + * with the old scan hint, and that the first candidate is larger in + * size or, it equal, farther than the second one. + */ + + if (block->contig_hint > scan_hint_cand_1) { + if (scan_hint_cand_1_start < block->contig_hint_start) { + block->scan_hint =3D scan_hint_cand_1; + block->scan_hint_start =3D scan_hint_cand_1_start; + } else if (scan_hint_cand_2_start < block->contig_hint_start) { + block->scan_hint =3D scan_hint_cand_2; + block->scan_hint_start =3D scan_hint_cand_2_start; + } else { + block->scan_hint =3D 0; + } + } else if (block->contig_hint =3D=3D scan_hint_cand_1) { + if (scan_hint_cand_1_start > block->contig_hint_start) { + block->scan_hint =3D scan_hint_cand_1; + block->scan_hint_start =3D scan_hint_cand_1_start; + } else if (scan_hint_cand_2 < block->contig_hint && + scan_hint_cand_2_start < scan_hint_cand_1_start) { + block->scan_hint =3D scan_hint_cand_2; + block->scan_hint_start =3D scan_hint_cand_2_start; + } else { + block->scan_hint =3D 0; } } } --=20 2.53.0.1213.gd9a14994de-goog