From nobody Sat Apr 4 07:47:59 2026 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 56C8436EA99 for ; Fri, 20 Mar 2026 11:52:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774007539; cv=none; b=rQn1wzd8HCSsFi/Mz9SuRkYI0pC1S73mb9BS74seDEsnjxrKw0HUJEvvtLN5a2TjIJT4kb99mM+soOk9VGWlfjUYaW8m57h32CNanWK0AmUN+r71fQ4orotxgI27q1t59vUJlKK6m6qF7JmGbUtZXo8w7AnlH+GwwT0kdOSNvrU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1774007539; c=relaxed/simple; bh=mTqM57n50if47fYiCbU6m2ylueCsiD+9fLSKGIQ5vTA=; h=Date:Mime-Version:Message-ID:Subject:From:To:Cc:Content-Type; b=uxHA3aPfvrYXlsG9aYGdjrGoSrRmHlsl37hpqDf1VMNj+IoVks/nkXqHkNRqeqIJenQ2M52arRhuuOufNJj8ySknbkxYdPEq6hNkOfFPAEo1PUNvffw2aGjYLcBYD/YmQIMB/ElutGy9a5sPFAkmSzxxrS9cYj2AYHegf3JRJ70= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--joonwonkang.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Bhu1QFwb; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--joonwonkang.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Bhu1QFwb" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-3594620fe97so12717223a91.1 for ; Fri, 20 Mar 2026 04:52:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1774007538; x=1774612338; darn=vger.kernel.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=gU69MYAltTpsVwDKd9QeUs/oT95TyhbpJovXUD5z/eo=; b=Bhu1QFwbeMmGpMB543pahj+xCFUFKkm1sVem24ABuE8r7gufOzJmyCWY/sPflkL/QC 6NFvSJX9yBLaJAk+Ni8ONOwTrSOyc2hq+gqMpy0U8iiFZgR4epbHXMNpOTwNN80ud7VV fkf/mNhC2NEnD9xiVByUEMtnqly0GN955Rvxdb/kYfzIpTD8LFN0knOhME0mWptkEQt3 GuF1E8ffTK6/OU8yZhBZI+trV7Ju5Mi9kf7j3eiPpP1VVJP9yDMtEsMovdS1DkFEbY3w QInqiT7MXZz0lFXVyWdhL1y34wY/NcWIPef5eY32CfyWtbwH1QSADIKmpnbSz8JN99jY lU8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1774007538; x=1774612338; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=gU69MYAltTpsVwDKd9QeUs/oT95TyhbpJovXUD5z/eo=; b=scBvIFr8wBMEIZfJBu8Z6C4CTVXzskNdXssd2kpl0jsZNQ5FM0GP9P0exFyld4kFUS feCtvo1CrwWuLxwPD7erwO0C2iZyvS2+dyBThbDLPYQb2Z5eKvtjxSd6rvobdH+mxIbQ b7am8mwQA3qDkKwpu0/ylOCHXXh1gJY2ufDDfuNqitGUV2uE2a30d7XV7nvpAuKGn4/G JO6yt5xuy4NQ6ksjH6svP1T858/igQQWMGCmlCvHj2goMaCUGdOXxAj8+XufuP4l0kgq lULWXlNNs5czPAW5GF6YfZpp+OI2OP5+8oUgSjIxx4wEI+3B5l3tNKbqaUGkqESMVpb3 i6sA== X-Forwarded-Encrypted: i=1; AJvYcCVPGO+QXyA6Y6KtHedGxUmcC0B8GCtNcyJY/an5vDuhsgPUN/O+Bhz+BR5vWfL+fiHe+WJq2uY9xorkkWQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yyq3SBPBaYLBsfThViZdkUPzopNoythVgnnEc46q9qLctzHTeCf dW6eueLz4fuZ/XgHF1FloKMUtsTzHkk7iYyi+oT4Wwcfee0InBFgMtd0LCOVoLIuCEsr4rR1x0m 2/vtNPoVKncEMivHoUGsGX3UjZA== X-Received: from pjzd9.prod.google.com ([2002:a17:90a:e289:b0:356:1f2b:7ea1]) (user=joonwonkang job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:4c8e:b0:35b:93d8:6aaa with SMTP id 98e67ed59e1d1-35bd2cf8065mr2240997a91.19.1774007537556; Fri, 20 Mar 2026 04:52:17 -0700 (PDT) Date: Fri, 20 Mar 2026 11:52:14 +0000 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 X-Mailer: git-send-email 2.53.0.1018.g2bb0e51243-goog Message-ID: <20260320115214.143933-1-joonwonkang@google.com> Subject: [PATCH] percpu: Fix hint invariant breakage From: Joonwon Kang To: dennis@kernel.org, tj@kernel.org, cl@gentwo.org Cc: akpm@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Joonwon Kang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The invariant "scan_hint_start > contig_hint_start if and only if scan_hint =3D=3D contig_hint" should be kept for hint management. However, it could be broken in some cases: - if (new contig =3D=3D contig_hint =3D=3D scan_hint) && (contig_hint_sta= rt < scan_hint_start < new contig start) && the new contig is to become a new contig_hint due to its better alignment, then scan_hint should be invalidated instead of keeping it. - if (new contig =3D=3D contig_hint > scan_hint) && (start < contig_hint_start) && the new contig is not to become a new contig_hint, then scan_hint should be invalidated instead of being updated to the new contig. This commit fixes this invariant breakage and also optimizes scan_hint by keeping it or updating it when acceptable: - if (new contig > contig_hint > scan_hint) && (scan_hint_start < new contig start < contig_hint_start), then keep scan_hint instead of invalidating it. - if (new contig > contig_hint =3D=3D scan_hint) && (contig_hint_start < new contig start < scan_hint_start), then update scan_hint to the old contig_hint instead of invalidating it. - if (new contig =3D=3D contig_hint > scan_hint) && (new contig start < contig_hint_start) && the new contig is to become a new contig_hint due to its better alignment, then update scan_hint to the old contig_hint instead of invalidating or keeping it. Signed-off-by: Joonwon Kang --- mm/percpu.c | 60 ++++++++++++++++++++++++++++++++++------------------- 1 file changed, 39 insertions(+), 21 deletions(-) diff --git a/mm/percpu.c b/mm/percpu.c index 81462ce5866e..a0e4f8acb7c2 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -641,19 +641,13 @@ static void pcpu_block_update(struct pcpu_block_md *b= lock, int start, int end) if (contig > block->contig_hint) { /* promote the old contig_hint to be the new scan_hint */ if (start > block->contig_hint_start) { - if (block->contig_hint > block->scan_hint) { + if (block->contig_hint > block->scan_hint || + start < block->scan_hint_start) { block->scan_hint_start =3D block->contig_hint_start; block->scan_hint =3D block->contig_hint; - } else if (start < block->scan_hint_start) { - /* - * The old contig_hint =3D=3D scan_hint. But, the - * new contig is larger so hold the invariant - * scan_hint_start < contig_hint_start. - */ - block->scan_hint =3D 0; } - } else { + } else if (start < block->scan_hint_start) { block->scan_hint =3D 0; } block->contig_hint_start =3D start; @@ -662,20 +656,44 @@ static void pcpu_block_update(struct pcpu_block_md *b= lock, int start, int end) if (block->contig_hint_start && (!start || __ffs(start) > __ffs(block->contig_hint_start))) { + if (block->contig_hint > block->scan_hint) { + if (start < block->contig_hint_start) { + block->scan_hint =3D block->contig_hint; + block->scan_hint_start =3D block->contig_hint_start; + } + } else if (start > block->scan_hint_start) { + /* + * old contig_hint =3D=3D old scan_hint =3D=3D contig. + * But, the new contig is farther than the old + * scan_hint so hold the invariant + * scan_hint_start > contig_hint_start iff + * scan_hint =3D=3D contig_hint. + */ + block->scan_hint =3D 0; + } + /* start has a better alignment so use it */ block->contig_hint_start =3D start; - if (start < block->scan_hint_start && - block->contig_hint > block->scan_hint) - block->scan_hint =3D 0; - } else if (start > block->scan_hint_start || - block->contig_hint > block->scan_hint) { - /* - * Knowing contig =3D=3D contig_hint, update the scan_hint - * if it is farther than or larger than the current - * scan_hint. - */ - block->scan_hint_start =3D start; - block->scan_hint =3D contig; + } else { + if (block->contig_hint > block->scan_hint) { + if (start < block->contig_hint_start) { + /* + * old scan_hint < contig =3D=3D old + * contig_hint. But, the new contig is + * before the old contig_hint so hold + * the invariant + * scan_hint_start > contig_hint_start + * iff scan_hint =3D=3D contig_hint. + */ + block->scan_hint =3D 0; + } else { + block->scan_hint_start =3D start; + block->scan_hint =3D contig; + } + } else if (start > block->scan_hint_start) { + block->scan_hint_start =3D start; + block->scan_hint =3D contig; + } } } else { /* --=20 2.53.0.1018.g2bb0e51243-goog