From nobody Thu Sep 24 14:26:50 2026 Received: from mail-qk2-f12.google.com (mail-qk2-f12.google.com [74.125.230.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1ED2B538D8A for ; Tue, 22 Sep 2026 23:58:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.204 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121523; cv=none; b=UYwnTFwoXtVsKKZFWYmwKuvUNsLjUEW94gTO4MQfifjnrcGirBrKLis/FRYL2cAZ92q5npXcfcGfyBrslb+oSLaWdmNlPMHqyUbV60GvH9kyVVBsP7S5haqO0iHCXHAKs9Lre64FUQdWTdAJPOUND9bUf2uAhLCejGpOCgOrgxM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121523; c=relaxed/simple; bh=cQdFuMK1oYjSd02zbj70fvBguBZGQxgaHFdk28ZprJQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=XbLa8QukhFlmsON9Yz6JmmRwamybbHGph5Ofpuxis/L7PZjOVfqV4N5zQD84Y28ZHQOgMko9qRTOiYFWCqm4M2LFqPWvVrItL2RugvGC2j2Kx8ecndBCaeEtEeQbbWHOfNokxorcLEDzOqvkMUJyr6Zw7AMDjajfWU8s3K5ZG+E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=jUwumqMQ; arc=none smtp.client-ip=74.125.230.204 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="jUwumqMQ" Received: by mail-qk2-f12.google.com with SMTP id d75a77b69052e-530d0d5ae11so4681151cf.0 for ; Tue, 22 Sep 2026 16:58:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1790121515; x=1790726315; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=hpRsm9zRoWaAs7SHkz+XL+jFcM2tMaqBrqTdDN1OoLg=; b=jUwumqMQfM9+mSxHk2HpZtxAcrOI9ZeRwiHFlSrbUmMz/tliq1saBKuvzmZ0W8kBEi vS6T+0QZKLOnxnjxZtpbt++MR9XAByplSYfbfJXYdASVsoj/QbZ1NpzN3SRTGOdfVV26 Q9BdXuOg/fvrr1Ein5Ku4fx0+xlHKtSBl2m2tpI8Gch9psxmBsgbwoVjOKOJtcTLNYmT vvNJdALOK/93yuKhaUKBZ4JhTvzANLbrWz6MnQcTUEJ/Zz5AtpJv+RacKbbyGcZF9niA 1pUohX2h+25L7vTcOqsIM0wmKcWLWTFBp+G4VlXx5NmKto4E1kIM6VixWjF+41bVszPk uOMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790121515; x=1790726315; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=hpRsm9zRoWaAs7SHkz+XL+jFcM2tMaqBrqTdDN1OoLg=; b=vsC28jzqectBGnHgqoKFOEnro2VZHTy/zujohxMFoFoKXesrahkAume8nSo0ZZNKXQ nRlK5+oTbc+ugAp3DY7BMWORoNOTq6YPoePKSJL7ZTSRzjw7g1H1AWGkNY67K6jR/Imq XlWmmXn86TYqsm0P3LstPKCccmhg4FZ03LSqv3oBX0yScpURHPPkoDMPUDqO5rKrZ4gg dqe3GgNqFtCgU+0WHUSdQMvahhACAUsgTqm+Op+di7DoOtT5HhK11ONBuf4/QPQ2DsPv 8dE2d5mxcrrGN7OCu9yxjfomrcnnxdjQRqZbCs2nbfrOXQ56gCGQ617AP2w01500DA5G hkWg== X-Gm-Message-State: AFuF++kb0KXjdNr9h39QBe92ZjY8UM35tnBnyPjLDkhl5XDBlauoyD1Y 8fit3VEe11KhFIwD6TvTY5n92etrrp5hkcC1ooXS3kq3DkvTyQisTal1HdAXc6ncGKc= X-Gm-Gg: AYBFou1+xjri/dJaPPh85i4hNqS5m9jPq+LgFw9uTOne/XYEBs22+bxs3Bf2LnQSCm8 ak90bLWijWjPEwxblbauj/Yab/QkoOU3+w9Mz9eByCOwDnxWuQ2OCptIV/TrgwMvfNtNKipMPWj sXHnZpWRuBZdzRJZ1nU+iiIkeHFMZbZa6oOjw6BcckwfqYmS979TLUTuF9uxbEgkPZSIeqy+rV/ MNcbRNEeClckzR74/rZ98kRTMe9uc53GAFPzrfzDy5iMhMmcP+tOBev+6C+57qq2u0dwAi2zTI4 vvzSBFARkZY9lT+cTMWuCkO/xl6SpkF5FpP7dIU8BuqxQ8goNiaW8PMHgJr3GKFxehyZa/b4sAU c+z3SqYbshwf2L8AiUANQqo5lG2W5pI/G/eaz6sdkres9hENR0W1iijt3xbsZlYNsW3sSLZVZmk zdO15UIhMGq/TN/u44UTQIac5oFpoMOPXDPmWpo2yH3sOnpLcsxDCPhKn43hUBIuHuD5866LQi0 BX0tHgPLus5bbLjptV54Qn6j7Sx6dxEJWVpJWkeQHb1mg//MEiyijX13+Ou X-Received: by 2002:a05:622a:4ce:b0:530:e343:b664 with SMTP id d75a77b69052e-532ead57eb8mr17951371cf.58.1790121515108; Tue, 22 Sep 2026 16:58:35 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-60-52.washdc.fios.verizon.net. [173.79.60.52]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-532eb3c6715sm8323261cf.30.2026.09.22.16.58.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 16:58:34 -0700 (PDT) From: Gregory Price To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-team@meta.com, akpm@linux-foundation.org, liam@infradead.org, ljs@kernel.org, david@kernel.org, vbabka@kernel.org, jannh@google.com, rppt@kernel.org, surenb@google.com, mhocko@suse.com, shuah@kernel.org, "Gregory Price (Meta)" Subject: [PATCH 01/10] selftests/mm: exercise MADV_COLD and MADV_PAGEOUT Date: Tue, 22 Sep 2026 19:58:21 -0400 Message-ID: <20260922235830.2350770-2-gourry@gourry.net> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922235830.2350770-1-gourry@gourry.net> References: <20260922235830.2350770-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" MADV_COLD and MADV_PAGEOUT share a page-table walker covering ordinary PTEs, PTE-mapped large folios and huge PMDs. Existing selftests provide little coverage of its range, folio-state and permission decisions. Add 28 cases ahead of a planned walker refactor. All cases check syscall results and mapping contents, but they deliberately have different review contracts. This commit was originally 28 individual commits, but was squashed for the initial RFC to limit noise during initial review. The question is what set of self-tests should retain. The eight user-visible contract tests verify: - MADV_COLD does not populate a PTE hole; - COLD and PAGEOUT reject a locked VMA; - full-range PAGEOUT swaps a PMD-mapped THP without corrupting it; - MADV_COLD does not fault in a swapped PTE; - PAGEOUT preserves an unevictable base page; - authorized file PAGEOUT evicts a clean file page; - unauthorized shared-file PAGEOUT leaves the page resident; and - private-file PAGEOUT reclaims a COW page but preserves a file page. The eighteen implementation-detail tests record the current folio and page-table behavior: - full and partial MADV_COLD on a PTE-mapped THP; - full, repeated and partial MADV_COLD on PMD-mapped THPs; - MADV_COLD on the shared zero page and huge zero page; - partial MADV_COLD on a shared PMD-mapped THP; - partial MADV_COLD on pinned PMD- and PTE-mapped THPs; - parallel partial MADV_COLD split attempts; - PAGEOUT on an unevictable THP; - MADV_COLD on active base-page and huge-PMD folios; - partial PAGEOUT on PMD- and PTE-mapped THPs; - partial and full MADV_COLD on shared PTE-mapped THPs; and - PAGEOUT filtering of a PMD-sized protected file folio. These tests intentionally assert splitting, PMD/PTE mapping shape, folio flags or folio sharing. They may need adjustment after a valid kernel implementation change. Their purpose here is to detect unintended changes during the refactor. Two additional stress tests use implementation-specific setup while keeping contract-level assertions: - MADV_COLD on a folio still queued in another CPU's LRU batch - MADV_PAGEOUT during NUMA migration of unevictable shmem THPs. The stress tests require only successful calls and preserved contents. They do not assert which internal path won. Tests requiring root, swap, THP allocation, multiple CPUs or NUMA nodes report a skip when their prerequisites are unavailable. Assisted-by: LLM Signed-off-by: Gregory Price (Meta) --- tools/testing/selftests/mm/Makefile | 2 + tools/testing/selftests/mm/ksft_madvise.sh | 4 + tools/testing/selftests/mm/madvise.c | 1673 ++++++++++++++++++++ tools/testing/selftests/mm/run_vmtests.sh | 5 + tools/testing/selftests/mm/vm_util.c | 4 +- tools/testing/selftests/mm/vm_util.h | 6 + 6 files changed, 1692 insertions(+), 2 deletions(-) create mode 100755 tools/testing/selftests/mm/ksft_madvise.sh create mode 100644 tools/testing/selftests/mm/madvise.c diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/= mm/Makefile index beacc0f873049..8912aa85ca273 100644 --- a/tools/testing/selftests/mm/Makefile +++ b/tools/testing/selftests/mm/Makefile @@ -69,6 +69,7 @@ TEST_GEN_FILES +=3D hugetlb-shm TEST_GEN_FILES +=3D hugetlb-soft-offline TEST_GEN_FILES +=3D khugepaged TEST_GEN_FILES +=3D madv_populate +TEST_GEN_FILES +=3D madvise TEST_GEN_FILES +=3D map_fixed_noreplace TEST_GEN_FILES +=3D map_populate ifneq (,$(filter $(ARCH),arm64 riscv riscv64 x86 x86_64 loongarch32 loonga= rch64)) @@ -155,6 +156,7 @@ TEST_PROGS +=3D ksft_kmemleak_dedup.sh TEST_PROGS +=3D ksft_ksm.sh TEST_PROGS +=3D ksft_ksm_numa.sh TEST_PROGS +=3D ksft_madv_guard.sh +TEST_PROGS +=3D ksft_madvise.sh TEST_PROGS +=3D ksft_madv_populate.sh TEST_PROGS +=3D ksft_memfd_secret.sh TEST_PROGS +=3D ksft_memory_failure.sh diff --git a/tools/testing/selftests/mm/ksft_madvise.sh b/tools/testing/sel= ftests/mm/ksft_madvise.sh new file mode 100755 index 0000000000000..29cbbca800f2c --- /dev/null +++ b/tools/testing/selftests/mm/ksft_madvise.sh @@ -0,0 +1,4 @@ +#!/bin/sh -e +# SPDX-License-Identifier: GPL-2.0 + +./run_vmtests.sh -t madvise diff --git a/tools/testing/selftests/mm/madvise.c b/tools/testing/selftests= /mm/madvise.c new file mode 100644 index 0000000000000..abde0b1c8a42e --- /dev/null +++ b/tools/testing/selftests/mm/madvise.c @@ -0,0 +1,1673 @@ +// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "kselftest.h" +#include "vm_util.h" + +#ifndef MADV_COLLAPSE +#define MADV_COLLAPSE 25 +#endif + +#define NR_CONCURRENT_THREADS 8 +#define NR_CONCURRENT_THPS 8 +#define NR_CONCURRENT_ROUNDS 400 +#define NR_MIGRATION_RACE_THPS 16 +#define NR_MIGRATION_RACE_ROUNDS 20 +#define NUMA_MASK_BITS 1024 +#define NUMA_MASK_LONGS (NUMA_MASK_BITS / (8 * sizeof(unsigned long))) + +static pthread_barrier_t concurrent_start_barrier; +static pthread_barrier_t concurrent_done_barrier; +static atomic_int concurrent_worker_errno; +static char *concurrent_area; +static size_t pmd_size; + +#if defined(SYS_get_mempolicy) && defined(SYS_mbind) && \ + defined(SYS_migrate_pages) +struct migration_pageout_data { + pthread_barrier_t start_barrier; + atomic_bool stop; + atomic_int calls; + atomic_int error; + char *mapping; + size_t size; +}; + +static bool find_two_memory_nodes(unsigned long *mask, int *node1, int *no= de2) +{ + int node; + + if (syscall(SYS_get_mempolicy, NULL, mask, NUMA_MASK_BITS + 1, NULL, + MPOL_F_MEMS_ALLOWED)) + return false; + + *node1 =3D *node2 =3D -1; + for (node =3D 0; node < NUMA_MASK_BITS; node++) { + if (!(mask[node / (8 * sizeof(*mask))] & + (1UL << (node % (8 * sizeof(*mask)))))) + continue; + if (*node1 < 0) { + *node1 =3D node; + } else { + *node2 =3D node; + return true; + } + } + + return false; +} + +static void numa_mask_set(unsigned long *mask, int node) +{ + mask[node / (8 * sizeof(*mask))] |=3D + 1UL << (node % (8 * sizeof(*mask))); +} + +static void *madvise_pageout_worker(void *arg) +{ + struct migration_pageout_data *data =3D arg; + + pthread_barrier_wait(&data->start_barrier); + while (!atomic_load_explicit(&data->stop, memory_order_relaxed)) { + if (madvise(data->mapping, data->size, MADV_PAGEOUT) && + errno !=3D EAGAIN) { + atomic_store(&data->error, errno); + break; + } + atomic_fetch_add(&data->calls, 1); + } + + return NULL; +} + +static int migrate_between_nodes(unsigned long *old_nodes, + unsigned long *new_nodes) +{ + int i; + + for (i =3D 0; i < NR_MIGRATION_RACE_ROUNDS; i++) { + unsigned long *tmp; + + if (syscall(SYS_migrate_pages, 0, NUMA_MASK_BITS + 1, + old_nodes, new_nodes) < 0) + return errno; + tmp =3D old_nodes; + old_nodes =3D new_nodes; + new_nodes =3D tmp; + } + + return 0; +} + +static int race_pageout_with_migration(char *mapping, size_t size, + unsigned long *mask1, + unsigned long *mask2) +{ + struct migration_pageout_data data =3D { + .mapping =3D mapping, + .size =3D size, + }; + pthread_t thread; + int ret; + + ret =3D pthread_barrier_init(&data.start_barrier, NULL, 2); + if (ret) + return ret; + ret =3D pthread_create(&thread, NULL, madvise_pageout_worker, &data); + if (ret) { + pthread_barrier_destroy(&data.start_barrier); + return ret; + } + pthread_barrier_wait(&data.start_barrier); + + ret =3D migrate_between_nodes(mask1, mask2); + atomic_store(&data.stop, true); + pthread_join(thread, NULL); + pthread_barrier_destroy(&data.start_barrier); + + if (ret) + return ret; + if (atomic_load(&data.error)) + return atomic_load(&data.error); + return atomic_load(&data.calls) ? 0 : EIO; +} +#endif + +static char *map_aligned_pages(size_t size) +{ + char *mapping, *aligned; + + mapping =3D mmap(NULL, size + pmd_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (mapping =3D=3D MAP_FAILED) + ksft_exit_fail_perror("mmap"); + + aligned =3D (char *)(((uintptr_t)mapping + pmd_size - 1) & + ~(pmd_size - 1)); + if (aligned !=3D mapping) + munmap(mapping, aligned - mapping); + if (aligned + size < mapping + size + pmd_size) + munmap(aligned + size, + mapping + size + pmd_size - (aligned + size)); + + memset(aligned, 1, size); + return aligned; +} + +/* MADV_COLLAPSE may fail transiently with EAGAIN. */ +static bool collapse_all(char *mapping, size_t size, int nr_hpages) +{ + int ret, retry; + + for (retry =3D 0; retry < 10; retry++) { + ret =3D madvise(mapping, size, MADV_COLLAPSE); + if (!ret) { + if (check_huge_anon(mapping, size, nr_hpages, pmd_size)) + return true; + } else if (errno !=3D EAGAIN) { + return false; + } + usleep(10000); + } + + return false; +} + +static bool collapse_shmem(char *mapping, size_t size, int nr_hpages) +{ + int ret, retry; + + for (retry =3D 0; retry < 10; retry++) { + ret =3D madvise(mapping, size, MADV_COLLAPSE); + if (!ret) { + if (check_huge_shmem(mapping, size, nr_hpages, pmd_size)) + return true; + } else if (errno !=3D EAGAIN) { + return false; + } + usleep(10000); + } + + return false; +} + +static char *map_unevictable_shmem(size_t size, int *shmid) +{ + char *reservation, *mapping, *aligned; + + reservation =3D mmap(NULL, size + pmd_size, PROT_NONE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (reservation =3D=3D MAP_FAILED) + ksft_exit_fail_perror("mmap reservation"); + aligned =3D (char *)(((uintptr_t)reservation + pmd_size - 1) & + ~(pmd_size - 1)); + munmap(reservation, size + pmd_size); + + *shmid =3D shmget(IPC_PRIVATE, size, IPC_CREAT | 0600); + if (*shmid < 0) + ksft_exit_fail_perror("shmget"); + if (shmctl(*shmid, SHM_LOCK, NULL)) { + shmctl(*shmid, IPC_RMID, NULL); + ksft_test_result_skip("could not lock a shmem segment\n"); + return MAP_FAILED; + } + mapping =3D shmat(*shmid, aligned, 0); + if (mapping =3D=3D (void *)-1) + ksft_exit_fail_perror("shmat"); + if (mapping !=3D aligned) + ksft_exit_fail_msg("shmat did not honor the aligned address\n"); + + return mapping; +} + +static void unmap_unevictable_shmem(char *mapping, int shmid) +{ + shmctl(shmid, SHM_UNLOCK, NULL); + shmdt(mapping); + shmctl(shmid, IPC_RMID, NULL); +} + +static char *map_aligned_file(int fd, size_t size, int flags) +{ + char *mapping, *aligned; + + mapping =3D mmap(NULL, size + pmd_size, PROT_NONE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (mapping =3D=3D MAP_FAILED) + ksft_exit_fail_perror("mmap reservation"); + aligned =3D (char *)(((uintptr_t)mapping + pmd_size - 1) & + ~(pmd_size - 1)); + if (mmap(aligned, size, PROT_READ | PROT_WRITE, flags | MAP_FIXED, + fd, 0) =3D=3D MAP_FAILED) + ksft_exit_fail_perror("mmap file"); + if (aligned !=3D mapping) + munmap(mapping, aligned - mapping); + if (aligned + size < mapping + size + pmd_size) + munmap(aligned + size, + mapping + size + pmd_size - (aligned + size)); + + return aligned; +} + +static void split_pmd_mapping(char *mapping) +{ + const size_t page_size =3D getpagesize(); + + if (mprotect(mapping + page_size, page_size, PROT_READ) || + mprotect(mapping + page_size, page_size, PROT_READ | PROT_WRITE)) + ksft_exit_fail_perror("mprotect"); + if (!check_large_folios(mapping, pmd_size, 1, pmd_size)) + ksft_exit_fail_msg("mprotect split the physical THP\n"); +} + +static void check_memory(char *mapping, size_t size) +{ + size_t offset; + + for (offset =3D 0; offset < size; offset +=3D getpagesize()) + if (mapping[offset] !=3D 1) + ksft_exit_fail_msg("memory changed at offset %zu\n", offset); +} + +/* + * MADV_COLD on a full PTE-mapped THP must preserve the folio, while an + * operation on only half of it must split the folio. Neither operation may + * alter the mapping contents. + */ +static void test_pte_mapped_madvise_cold(void) +{ + char *mapping =3D map_aligned_pages(pmd_size); + + if (!collapse_all(mapping, pmd_size, 1)) { + munmap(mapping, pmd_size); + ksft_test_result_skip("could not allocate a PMD-sized THP\n"); + return; + } + + split_pmd_mapping(mapping); + if (madvise(mapping, pmd_size, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + if (!check_large_folios(mapping, pmd_size, 1, pmd_size)) + ksft_exit_fail_msg("full MADV_COLD split a large folio\n"); + + if (madvise(mapping, pmd_size / 2, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + if (!check_large_folios(mapping, pmd_size, 0, pmd_size)) + ksft_exit_fail_msg("partial MADV_COLD left a large folio\n"); + check_memory(mapping, pmd_size); + + munmap(mapping, pmd_size); + ksft_test_result_pass("MADV_COLD handles a PTE-mapped THP\n"); +} + +/* + * A PTE walk must skip holes without populating them or overlooking the + * present pages on either side. + */ +static void test_madvise_cold_pte_hole(void) +{ + const size_t page_size =3D getpagesize(); + const size_t size =3D 3 * page_size; + char *mapping; + int pagemap_fd; + + mapping =3D mmap(NULL, size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (mapping =3D=3D MAP_FAILED) + ksft_exit_fail_perror("mmap"); + mapping[0] =3D 1; + mapping[2 * page_size] =3D 1; + + pagemap_fd =3D open("/proc/self/pagemap", O_RDONLY); + if (pagemap_fd < 0) + ksft_exit_fail_perror("open pagemap"); + if (pagemap_is_populated(pagemap_fd, mapping + page_size)) + ksft_exit_fail_msg("PTE hole was populated before MADV_COLD\n"); + + if (madvise(mapping, size, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + if (mapping[0] !=3D 1 || mapping[2 * page_size] !=3D 1) + ksft_exit_fail_msg("MADV_COLD changed populated pages\n"); + if (pagemap_is_populated(pagemap_fd, mapping + page_size)) + ksft_exit_fail_msg("MADV_COLD populated a PTE hole\n"); + + close(pagemap_fd); + munmap(mapping, size); + ksft_test_result_pass("MADV_COLD skips PTE holes\n"); +} + +/* + * A read fault on private anonymous memory may install the shared zero pa= ge. + * MADV_COLD must ignore that special PTE without replacing it or changing= the + * mapping contents. + */ +static void test_madvise_cold_zero_page(void) +{ + const size_t page_size =3D getpagesize(); + char *mapping; + int pagemap_fd; + + mapping =3D mmap(NULL, page_size, PROT_READ, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (mapping =3D=3D MAP_FAILED) + ksft_exit_fail_perror("mmap"); + if (mapping[0]) + ksft_exit_fail_msg("anonymous mapping is not zero-filled\n"); + + pagemap_fd =3D open("/proc/self/pagemap", O_RDONLY); + if (pagemap_fd < 0) + ksft_exit_fail_perror("open pagemap"); + if (!pagemap_is_populated(pagemap_fd, mapping)) + ksft_exit_fail_msg("zero page is not populated\n"); + + if (madvise(mapping, page_size, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + if (mapping[0] || !pagemap_is_populated(pagemap_fd, mapping)) + ksft_exit_fail_msg("MADV_COLD changed the zero-page mapping\n"); + + close(pagemap_fd); + munmap(mapping, page_size); + ksft_test_result_pass("MADV_COLD skips the shared zero page\n"); +} + +/* + * A read fault may map the shared huge zero page with a PMD. MADV_COLD mu= st + * ignore that special PMD without replacing it or changing the mapping. + */ +static void test_madvise_cold_huge_zero_page(void) +{ + char *reservation, *mapping; + unsigned long pfn; + uint64_t flags; + int pagemap_fd, kpageflags_fd; + + if (geteuid()) { + ksft_test_result_skip("requires root to read page flags\n"); + return; + } + reservation =3D mmap(NULL, 2 * pmd_size, PROT_NONE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (reservation =3D=3D MAP_FAILED) + ksft_exit_fail_perror("mmap reservation"); + mapping =3D (char *)(((uintptr_t)reservation + pmd_size - 1) & + ~(pmd_size - 1)); + if (mmap(mapping, pmd_size, PROT_READ, + MAP_PRIVATE | MAP_ANONYMOUS | MAP_FIXED, -1, 0) =3D=3D MAP_FAILED) + ksft_exit_fail_perror("mmap huge zero page"); + if (mapping !=3D reservation) + munmap(reservation, mapping - reservation); + if (mapping + pmd_size < reservation + 2 * pmd_size) + munmap(mapping + pmd_size, + reservation + 2 * pmd_size - (mapping + pmd_size)); + if (madvise(mapping, pmd_size, MADV_HUGEPAGE)) + ksft_exit_fail_perror("MADV_HUGEPAGE"); + if (mapping[0]) + ksft_exit_fail_msg("anonymous mapping is not zero-filled\n"); + + pagemap_fd =3D open("/proc/self/pagemap", O_RDONLY); + kpageflags_fd =3D open("/proc/kpageflags", O_RDONLY); + if (pagemap_fd < 0 || kpageflags_fd < 0) + ksft_exit_fail_perror("open page flags"); + pfn =3D pagemap_get_pfn(pagemap_fd, mapping); + if (pfn =3D=3D -1ul || pageflags_get(pfn, kpageflags_fd, &flags) || + !(flags & KPF_ZERO_PAGE)) { + close(kpageflags_fd); + close(pagemap_fd); + munmap(mapping, pmd_size); + ksft_test_result_skip("huge zero page is not available\n"); + return; + } + + if (madvise(mapping, pmd_size, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + if (mapping[0] || pagemap_get_pfn(pagemap_fd, mapping) !=3D pfn) + ksft_exit_fail_msg("MADV_COLD changed the huge-zero-page mapping\n"); + + close(kpageflags_fd); + close(pagemap_fd); + munmap(mapping, pmd_size); + ksft_test_result_pass("MADV_COLD skips the huge zero page\n"); +} + +/* + * Neither hint is valid for a locked VMA: reclaiming it would violate the + * mlock contract, and merely aging it would serve no purpose. + */ +static void test_madvise_lru_locked_vma(void) +{ + const size_t page_size =3D getpagesize(); + char *mapping; + + mapping =3D mmap(NULL, page_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (mapping =3D=3D MAP_FAILED) + ksft_exit_fail_perror("mmap"); + mapping[0] =3D 1; + if (mlock(mapping, page_size)) { + munmap(mapping, page_size); + ksft_test_result_skip("could not lock a page\n"); + return; + } + + errno =3D 0; + if (!madvise(mapping, page_size, MADV_COLD) || errno !=3D EINVAL) + ksft_exit_fail_msg("MADV_COLD accepted a locked VMA\n"); + errno =3D 0; + if (!madvise(mapping, page_size, MADV_PAGEOUT) || errno !=3D EINVAL) + ksft_exit_fail_msg("MADV_PAGEOUT accepted a locked VMA\n"); + if (mapping[0] !=3D 1) + ksft_exit_fail_msg("madvise changed locked memory\n"); + + munlock(mapping, page_size); + munmap(mapping, page_size); + ksft_test_result_pass("COLD and PAGEOUT reject a locked VMA\n"); +} + +/* + * A full-range MADV_COLD operates directly on a huge PMD. Assert that agi= ng + * the mapping preserves both the physical THP and its contents. + */ +static void test_full_pmd_madvise_cold(void) +{ + char *mapping =3D map_aligned_pages(pmd_size); + + if (!collapse_all(mapping, pmd_size, 1)) { + munmap(mapping, pmd_size); + ksft_test_result_skip("could not allocate a PMD-sized THP\n"); + return; + } + if (madvise(mapping, pmd_size, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + if (!check_large_folios(mapping, pmd_size, 1, pmd_size)) + ksft_exit_fail_msg("full MADV_COLD split a PMD-mapped THP\n"); + check_memory(mapping, pmd_size); + + munmap(mapping, pmd_size); + ksft_test_result_pass("full MADV_COLD preserves a PMD-mapped THP\n"); +} + +/* + * MADV_COLD is idempotent. A second request must handle an already-old hu= ge + * PMD without splitting the THP or changing its contents. + */ +static void test_repeated_pmd_madvise_cold(void) +{ + char *mapping =3D map_aligned_pages(pmd_size); + + if (!collapse_all(mapping, pmd_size, 1)) { + munmap(mapping, pmd_size); + ksft_test_result_skip("could not allocate a PMD-sized THP\n"); + return; + } + if (madvise(mapping, pmd_size, MADV_COLD) || + madvise(mapping, pmd_size, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + if (!check_large_folios(mapping, pmd_size, 1, pmd_size)) + ksft_exit_fail_msg("repeated MADV_COLD split a PMD-mapped THP\n"); + check_memory(mapping, pmd_size); + + munmap(mapping, pmd_size); + ksft_test_result_pass("repeated MADV_COLD preserves a PMD-mapped THP\n"); +} + +/* + * MADV_COLD on half of a PMD-mapped THP must split the folio so the + * unadvised half is not aged as part of the THP. + */ +static void test_partial_pmd_madvise_cold(void) +{ + char *mapping =3D map_aligned_pages(pmd_size); + + if (!collapse_all(mapping, pmd_size, 1)) { + munmap(mapping, pmd_size); + ksft_test_result_skip("could not allocate a PMD-sized THP\n"); + return; + } + if (madvise(mapping, pmd_size / 2, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + if (!check_large_folios(mapping, pmd_size, 0, pmd_size)) + ksft_exit_fail_msg("partial MADV_COLD left a PMD-mapped THP\n"); + check_memory(mapping, pmd_size); + + munmap(mapping, pmd_size); + ksft_test_result_pass("partial MADV_COLD splits a PMD-mapped THP\n"); +} + +static void pageout_half_thp(char *mapping) +{ + if (madvise(mapping, pmd_size / 2, MADV_PAGEOUT)) + ksft_exit_fail_perror("MADV_PAGEOUT"); + if (!check_large_folios(mapping, pmd_size, 0, pmd_size)) + ksft_exit_fail_msg("partial MADV_PAGEOUT left a large folio\n"); + check_memory(mapping, pmd_size); +} + +/* + * Full-range MADV_PAGEOUT can reclaim a PMD-mapped THP directly. Assert t= hat + * the mapping is swapped without corrupting the folio contents. + */ +static void test_full_pmd_madvise_pageout(void) +{ + char *mapping =3D map_aligned_pages(pmd_size); + int pagemap_fd, retry; + bool swapped =3D false; + + if (!collapse_all(mapping, pmd_size, 1)) { + munmap(mapping, pmd_size); + ksft_test_result_skip("could not allocate a PMD-sized THP\n"); + return; + } + pagemap_fd =3D open("/proc/self/pagemap", O_RDONLY); + if (pagemap_fd < 0) + ksft_exit_fail_perror("open pagemap"); + + for (retry =3D 0; retry < 100; retry++) { + if (madvise(mapping, pmd_size, MADV_PAGEOUT)) + ksft_exit_fail_perror("MADV_PAGEOUT"); + if (pagemap_is_swapped(pagemap_fd, mapping)) { + swapped =3D true; + break; + } + usleep(10000); + } + if (swapped) + check_memory(mapping, pmd_size); + + close(pagemap_fd); + munmap(mapping, pmd_size); + if (!swapped) { + ksft_test_result_skip("MADV_PAGEOUT did not swap the THP\n"); + return; + } + ksft_test_result_pass("full MADV_PAGEOUT swaps a PMD-mapped THP\n"); +} + +/* + * A swapped PTE is non-present but not empty. MADV_COLD must skip the ent= ry + * without faulting the page back in or changing its contents. + */ +static void test_madvise_cold_swapped_pte(void) +{ + const size_t page_size =3D getpagesize(); + char *mapping =3D map_aligned_pages(page_size); + int pagemap_fd, retry; + bool swapped =3D false; + + pagemap_fd =3D open("/proc/self/pagemap", O_RDONLY); + if (pagemap_fd < 0) + ksft_exit_fail_perror("open pagemap"); + for (retry =3D 0; retry < 100; retry++) { + if (madvise(mapping, page_size, MADV_PAGEOUT)) + ksft_exit_fail_perror("MADV_PAGEOUT"); + if (pagemap_is_swapped(pagemap_fd, mapping)) { + swapped =3D true; + break; + } + usleep(10000); + } + if (!swapped) { + close(pagemap_fd); + munmap(mapping, page_size); + ksft_test_result_skip("MADV_PAGEOUT did not swap the page\n"); + return; + } + + if (madvise(mapping, page_size, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + if (!pagemap_is_swapped(pagemap_fd, mapping)) + ksft_exit_fail_msg("MADV_COLD faulted in a swapped PTE\n"); + if (mapping[0] !=3D 1) + ksft_exit_fail_msg("swapped page contents changed\n"); + + close(pagemap_fd); + munmap(mapping, page_size); + ksft_test_result_pass("MADV_COLD skips a swapped PTE\n"); +} + +/* + * vmsplice() retains a reference to a THP in a pipe, preventing a partial + * MADV_COLD from splitting it. The advice must leave the folio intact whi= le + * pinned, then split it normally after the pipe releases the reference. + */ +static void test_partial_pinned_madvise_cold(bool pte_mapped) +{ + const size_t page_size =3D getpagesize(); + char *mapping =3D map_aligned_pages(pmd_size); + struct iovec iov =3D { + .iov_base =3D mapping, + .iov_len =3D page_size, + }; + char *buffer; + int pipefd[2]; + int retry; + + if (!collapse_all(mapping, pmd_size, 1)) { + munmap(mapping, pmd_size); + ksft_test_result_skip("could not allocate a PMD-sized THP\n"); + return; + } + if (pte_mapped) + split_pmd_mapping(mapping); + if (pipe(pipefd)) + ksft_exit_fail_perror("pipe"); + if (vmsplice(pipefd[1], &iov, 1, SPLICE_F_GIFT) !=3D page_size) { + close(pipefd[0]); + close(pipefd[1]); + munmap(mapping, pmd_size); + ksft_test_result_skip("vmsplice could not retain a THP page\n"); + return; + } + + if (madvise(mapping, pmd_size / 2, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + if (!check_large_folios(mapping, pmd_size, 1, pmd_size)) + ksft_exit_fail_msg("MADV_COLD split a pinned THP\n"); + + buffer =3D malloc(page_size); + if (!buffer) + ksft_exit_fail_perror("malloc"); + if (read(pipefd[0], buffer, page_size) !=3D page_size) + ksft_exit_fail_perror("read pipe"); + free(buffer); + close(pipefd[0]); + close(pipefd[1]); + + for (retry =3D 0; retry < 10; retry++) { + if (madvise(mapping, pmd_size / 2, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + if (check_large_folios(mapping, pmd_size, 0, pmd_size)) + break; + usleep(10000); + } + if (retry =3D=3D 10) + ksft_exit_fail_msg("MADV_COLD did not split an unpinned THP\n"); + check_memory(mapping, pmd_size); + + munmap(mapping, pmd_size); + ksft_test_result_pass("partial MADV_COLD skips a pinned %s THP\n", + pte_mapped ? "PTE-mapped" : "PMD-mapped"); +} + +static void test_partial_pinned_pmd_madvise_cold(void) +{ + test_partial_pinned_madvise_cold(false); +} + +/* + * Pinning a PTE-mapped THP also prevents a partial MADV_COLD from splitti= ng + * it. Once the pipe releases the reference, a serial retry must split it. + */ +static void test_partial_pinned_pte_madvise_cold(void) +{ + test_partial_pinned_madvise_cold(true); +} + +static void *madvise_cold_worker(void *unused) +{ + int round, i; + + (void)unused; + for (round =3D 0; round < NR_CONCURRENT_ROUNDS; round++) { + pthread_barrier_wait(&concurrent_start_barrier); + for (i =3D 0; i < NR_CONCURRENT_THPS; i++) { + if (madvise(concurrent_area + i * pmd_size, + pmd_size / 2, MADV_COLD)) + atomic_store(&concurrent_worker_errno, errno); + } + pthread_barrier_wait(&concurrent_done_barrier); + } + + return NULL; +} + +/* + * Concurrent partial MADV_COLD calls deliberately contend for each THP lo= ck. + * Alternate PMD- and PTE-mapped rounds to exercise both split paths. A ca= ller + * that loses the trylock race may skip that folio, so retry serially befo= re + * asserting the stable interface: all calls succeed, the THPs remain + * splittable, and their contents are unchanged. + */ +static void test_concurrent_partial_madvise_cold(void) +{ + const size_t size =3D NR_CONCURRENT_THPS * pmd_size; + pthread_t threads[NR_CONCURRENT_THREADS]; + int round, i; + + concurrent_area =3D map_aligned_pages(size); + if (!collapse_all(concurrent_area, size, NR_CONCURRENT_THPS)) { + munmap(concurrent_area, size); + ksft_test_result_skip("could not allocate PMD-sized THPs\n"); + return; + } + if (pthread_barrier_init(&concurrent_start_barrier, NULL, + NR_CONCURRENT_THREADS + 1) || + pthread_barrier_init(&concurrent_done_barrier, NULL, + NR_CONCURRENT_THREADS + 1)) + ksft_exit_fail_msg("pthread_barrier_init failed\n"); + for (i =3D 0; i < NR_CONCURRENT_THREADS; i++) + if (pthread_create(&threads[i], NULL, madvise_cold_worker, NULL)) + ksft_exit_fail_msg("pthread_create failed\n"); + + for (round =3D 0; round < NR_CONCURRENT_ROUNDS; round++) { + if (!collapse_all(concurrent_area, size, NR_CONCURRENT_THPS)) + ksft_exit_fail_msg("round %d: failed to form PMD THPs\n", + round); + if (round & 1) + for (i =3D 0; i < NR_CONCURRENT_THPS; i++) + split_pmd_mapping(concurrent_area + i * pmd_size); + pthread_barrier_wait(&concurrent_start_barrier); + pthread_barrier_wait(&concurrent_done_barrier); + + if (atomic_load(&concurrent_worker_errno)) { + errno =3D atomic_load(&concurrent_worker_errno); + ksft_exit_fail_perror("MADV_COLD"); + } + for (i =3D 0; i < NR_CONCURRENT_THPS; i++) { + if (madvise(concurrent_area + i * pmd_size, + pmd_size / 2, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD retry"); + } + if (!check_large_folios(concurrent_area, size, 0, pmd_size)) + ksft_exit_fail_msg("round %d: PMD THP remained\n", round); + } + check_memory(concurrent_area, size); + + for (i =3D 0; i < NR_CONCURRENT_THREADS; i++) + pthread_join(threads[i], NULL); + pthread_barrier_destroy(&concurrent_done_barrier); + pthread_barrier_destroy(&concurrent_start_barrier); + munmap(concurrent_area, size); + ksft_test_result_pass("concurrent partial MADV_COLD preserves memory\n"); +} + +/* + * SHM_LOCK makes a shmem folio unevictable without setting VM_LOCKED on t= his + * VMA. MADV_PAGEOUT must put an isolated folio back on its LRU rather than + * reclaiming it. + */ +static void test_madvise_pageout_unevictable(void) +{ + const size_t page_size =3D getpagesize(); + unsigned long pfn; + uint64_t flags; + char *mapping; + int pagemap_fd, kpageflags_fd; + int shmid; + + if (geteuid()) { + ksft_test_result_skip("requires root to read page flags\n"); + return; + } + shmid =3D shmget(IPC_PRIVATE, page_size, IPC_CREAT | 0600); + if (shmid < 0) + ksft_exit_fail_perror("shmget"); + if (shmctl(shmid, SHM_LOCK, NULL)) { + shmctl(shmid, IPC_RMID, NULL); + ksft_test_result_skip("could not lock a shmem segment\n"); + return; + } + mapping =3D shmat(shmid, NULL, 0); + if (mapping =3D=3D (void *)-1) + ksft_exit_fail_perror("shmat"); + mapping[0] =3D 1; + + pagemap_fd =3D open("/proc/self/pagemap", O_RDONLY); + kpageflags_fd =3D open("/proc/kpageflags", O_RDONLY); + if (pagemap_fd < 0 || kpageflags_fd < 0) + ksft_exit_fail_perror("open page flags"); + + /* Drain the LRU add batch so the locked mapping becomes unevictable. */ + if (madvise(mapping, page_size, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + pfn =3D pagemap_get_pfn(pagemap_fd, mapping); + if (pfn =3D=3D -1ul || pageflags_get(pfn, kpageflags_fd, &flags) || + !(flags & KPF_UNEVICTABLE)) + ksft_exit_fail_msg("SHM_LOCK page is not unevictable\n"); + + if (madvise(mapping, page_size, MADV_PAGEOUT)) + ksft_exit_fail_perror("MADV_PAGEOUT"); + if (!pagemap_is_populated(pagemap_fd, mapping) || mapping[0] !=3D 1) + ksft_exit_fail_msg("MADV_PAGEOUT reclaimed an unevictable page\n"); + + close(kpageflags_fd); + close(pagemap_fd); + shmctl(shmid, SHM_UNLOCK, NULL); + shmdt(mapping); + shmctl(shmid, IPC_RMID, NULL); + ksft_test_result_pass("MADV_PAGEOUT preserves an unevictable page\n"); +} + +/* + * Exercise the unevictable PAGEOUT path at PMD granularity. SHM_LOCK keeps + * the shmem THP resident without marking its VMA VM_LOCKED. + */ +static void test_madvise_pageout_unevictable_thp(void) +{ + char *mapping; + int pagemap_fd, shmid; + + if (geteuid()) { + ksft_test_result_skip("requires root to lock a shmem segment\n"); + return; + } + mapping =3D map_unevictable_shmem(pmd_size, &shmid); + if (mapping =3D=3D MAP_FAILED) + return; + memset(mapping, 1, pmd_size); + if (!collapse_shmem(mapping, pmd_size, 1)) { + unmap_unevictable_shmem(mapping, shmid); + ksft_test_result_skip("could not allocate an unevictable THP\n"); + return; + } + pagemap_fd =3D open("/proc/self/pagemap", O_RDONLY); + if (pagemap_fd < 0) + ksft_exit_fail_perror("open pagemap"); + + if (madvise(mapping, pmd_size, MADV_PAGEOUT)) + ksft_exit_fail_perror("MADV_PAGEOUT"); + if (!pagemap_is_populated(pagemap_fd, mapping) || + !check_huge_shmem(mapping, pmd_size, 1, pmd_size)) + ksft_exit_fail_msg("MADV_PAGEOUT reclaimed an unevictable THP\n"); + check_memory(mapping, pmd_size); + + close(pagemap_fd); + unmap_unevictable_shmem(mapping, shmid); + ksft_test_result_pass("MADV_PAGEOUT preserves an unevictable THP\n"); +} + +/* + * migrate_pages() isolates folios from the LRU before replacing their page + * table entries. Race that interval against MADV_PAGEOUT on PMD-mapped TH= Ps. + * Either operation may skip a folio owned by the other, but repeated calls + * must not report an error or corrupt the mapping. + */ +static void test_madvise_pageout_migration(void) +{ +#if defined(SYS_get_mempolicy) && defined(SYS_mbind) && \ + defined(SYS_migrate_pages) + unsigned long allowed[NUMA_MASK_LONGS] =3D {}; + unsigned long mask1[NUMA_MASK_LONGS] =3D {}; + unsigned long mask2[NUMA_MASK_LONGS] =3D {}; + const size_t size =3D NR_MIGRATION_RACE_THPS * pmd_size; + char *mapping; + int node1, node2; + int error; + int shmid; + + if (geteuid()) { + ksft_test_result_skip("requires root to lock a shmem segment\n"); + return; + } + if (!find_two_memory_nodes(allowed, &node1, &node2)) { + ksft_test_result_skip("requires two allowed NUMA memory nodes\n"); + return; + } + + numa_mask_set(mask1, node1); + numa_mask_set(mask2, node2); + mapping =3D map_unevictable_shmem(size, &shmid); + if (mapping =3D=3D MAP_FAILED) + return; + if (syscall(SYS_mbind, mapping, size, MPOL_BIND, mask1, + NUMA_MASK_BITS + 1, 0)) { + unmap_unevictable_shmem(mapping, shmid); + ksft_test_result_skip("could not bind shmem to a NUMA node\n"); + return; + } + memset(mapping, 1, size); + if (!collapse_shmem(mapping, size, NR_MIGRATION_RACE_THPS)) { + unmap_unevictable_shmem(mapping, shmid); + ksft_test_result_skip("could not allocate unevictable shmem THPs\n"); + return; + } + + error =3D race_pageout_with_migration(mapping, size, mask1, mask2); + if (error =3D=3D ENOSYS) { + unmap_unevictable_shmem(mapping, shmid); + ksft_test_result_skip("NUMA migration is unavailable\n"); + return; + } + if (error) { + errno =3D error; + ksft_exit_fail_perror("MADV_PAGEOUT/migrate_pages race"); + } + check_memory(mapping, size); + + unmap_unevictable_shmem(mapping, shmid); + ksft_test_result_pass("MADV_PAGEOUT races NUMA migration safely\n"); +#else + ksft_test_result_skip("NUMA migration system calls are unavailable\n"); +#endif +} + +static bool activate_folio(char *mapping, char *drain, unsigned long pfn, + int kpageflags_fd) +{ + struct iovec local =3D { .iov_base =3D drain, .iov_len =3D 1 }; + struct iovec remote =3D { .iov_base =3D mapping, .iov_len =3D 1 }; + uint64_t flags; + int retry; + + for (retry =3D 0; retry < 10; retry++) { + if (process_vm_readv(getpid(), &local, 1, &remote, 1, 0) !=3D 1) + ksft_exit_fail_perror("process_vm_readv"); + /* Drain the activation batch without advising the target folio. */ + if (madvise(drain, getpagesize(), MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + if (pageflags_get(pfn, kpageflags_fd, &flags)) + ksft_exit_fail_perror("read kpageflags"); + if (flags & KPF_ACTIVE) + return true; + } + + return false; +} + +/* + * Repeated GUP accesses promote an inactive folio on the traditional LRU. + * MADV_COLD must deactivate that folio while preserving its contents. + */ +static void test_madvise_cold_active_folio(void) +{ + const size_t page_size =3D getpagesize(); + char *mapping =3D map_aligned_pages(page_size); + char *drain =3D map_aligned_pages(page_size); + cpu_set_t old_mask, mask; + int pagemap_fd, kpageflags_fd; + unsigned long pfn; + uint64_t flags; + int cpu; + + if (geteuid()) { + munmap(drain, page_size); + munmap(mapping, page_size); + ksft_test_result_skip("requires root to read page flags\n"); + return; + } + if (sched_getaffinity(0, sizeof(old_mask), &old_mask)) + ksft_exit_fail_perror("sched_getaffinity"); + cpu =3D sched_getcpu(); + if (cpu < 0) + ksft_exit_fail_perror("sched_getcpu"); + CPU_ZERO(&mask); + CPU_SET(cpu, &mask); + if (sched_setaffinity(0, sizeof(mask), &mask)) + ksft_exit_fail_perror("sched_setaffinity"); + + pagemap_fd =3D open("/proc/self/pagemap", O_RDONLY); + kpageflags_fd =3D open("/proc/kpageflags", O_RDONLY); + if (pagemap_fd < 0 || kpageflags_fd < 0) + ksft_exit_fail_perror("open page flags"); + pfn =3D pagemap_get_pfn(pagemap_fd, mapping); + if (pfn =3D=3D -1ul) + ksft_exit_fail_msg("could not read page PFN\n"); + + if (!activate_folio(mapping, drain, pfn, kpageflags_fd)) { + close(kpageflags_fd); + close(pagemap_fd); + sched_setaffinity(0, sizeof(old_mask), &old_mask); + munmap(drain, page_size); + munmap(mapping, page_size); + ksft_test_result_skip("could not activate a folio\n"); + return; + } + + if (madvise(mapping, page_size, MADV_COLD) || + madvise(drain, page_size, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + if (pageflags_get(pfn, kpageflags_fd, &flags)) + ksft_exit_fail_perror("read kpageflags"); + if ((flags & KPF_ACTIVE) || mapping[0] !=3D 1) + ksft_exit_fail_msg("MADV_COLD did not deactivate the folio\n"); + + close(kpageflags_fd); + close(pagemap_fd); + if (sched_setaffinity(0, sizeof(old_mask), &old_mask)) + ksft_exit_fail_perror("restore affinity"); + munmap(drain, page_size); + munmap(mapping, page_size); + ksft_test_result_pass("MADV_COLD deactivates an active folio\n"); +} + +struct remote_fault_data { + char *mapping; + pthread_barrier_t barrier; + int cpu; + int error; +}; + +static void *remote_fault_worker(void *arg) +{ + struct remote_fault_data *data =3D arg; + cpu_set_t mask; + + CPU_ZERO(&mask); + CPU_SET(data->cpu, &mask); + if (sched_setaffinity(0, sizeof(mask), &mask)) + data->error =3D errno; + else + data->mapping[0] =3D 1; + pthread_barrier_wait(&data->barrier); + pthread_barrier_wait(&data->barrier); + return NULL; +} + +/* + * A folio faulted on another CPU can remain in that CPU's pending LRU bat= ch. + * MADV_COLD drains only the calling CPU and must safely skip the non-LRU + * folio without changing the mapping. + */ +static void test_madvise_cold_remote_lru_batch(void) +{ + const size_t page_size =3D getpagesize(); + struct remote_fault_data data =3D { .cpu =3D -1 }; + char *mapping; + cpu_set_t old_mask, mask; + pthread_t thread; + unsigned long pfn; + uint64_t flags; + int pagemap_fd, kpageflags_fd; + int cpu, main_cpu =3D -1; + + if (geteuid()) { + ksft_test_result_skip("requires root to read page flags\n"); + return; + } + if (sched_getaffinity(0, sizeof(old_mask), &old_mask)) + ksft_exit_fail_perror("sched_getaffinity"); + for (cpu =3D 0; cpu < CPU_SETSIZE; cpu++) { + if (!CPU_ISSET(cpu, &old_mask)) + continue; + if (main_cpu < 0) { + main_cpu =3D cpu; + } else { + data.cpu =3D cpu; + break; + } + } + if (main_cpu < 0 || data.cpu < 0) { + ksft_test_result_skip("requires two CPUs\n"); + return; + } + CPU_ZERO(&mask); + CPU_SET(main_cpu, &mask); + if (sched_setaffinity(0, sizeof(mask), &mask)) + ksft_exit_fail_perror("sched_setaffinity"); + + mapping =3D mmap(NULL, page_size, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + if (mapping =3D=3D MAP_FAILED) + ksft_exit_fail_perror("mmap"); + data.mapping =3D mapping; + if (pthread_barrier_init(&data.barrier, NULL, 2) || + pthread_create(&thread, NULL, remote_fault_worker, &data)) + ksft_exit_fail_msg("could not start fault worker\n"); + pthread_barrier_wait(&data.barrier); + if (data.error) { + errno =3D data.error; + ksft_exit_fail_perror("worker sched_setaffinity"); + } + + pagemap_fd =3D open("/proc/self/pagemap", O_RDONLY); + kpageflags_fd =3D open("/proc/kpageflags", O_RDONLY); + if (pagemap_fd < 0 || kpageflags_fd < 0) + ksft_exit_fail_perror("open page flags"); + pfn =3D pagemap_get_pfn(pagemap_fd, mapping); + if (pfn =3D=3D -1ul || pageflags_get(pfn, kpageflags_fd, &flags)) + ksft_exit_fail_msg("could not read page flags\n"); + if (flags & KPF_LRU) { + pthread_barrier_wait(&data.barrier); + pthread_join(thread, NULL); + pthread_barrier_destroy(&data.barrier); + close(kpageflags_fd); + close(pagemap_fd); + sched_setaffinity(0, sizeof(old_mask), &old_mask); + munmap(mapping, page_size); + ksft_test_result_skip("remote LRU batch was already drained\n"); + return; + } + + if (madvise(mapping, page_size, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + if (mapping[0] !=3D 1 || !pagemap_is_populated(pagemap_fd, mapping)) + ksft_exit_fail_msg("MADV_COLD changed a pending-LRU page\n"); + + pthread_barrier_wait(&data.barrier); + pthread_join(thread, NULL); + pthread_barrier_destroy(&data.barrier); + close(kpageflags_fd); + close(pagemap_fd); + if (sched_setaffinity(0, sizeof(old_mask), &old_mask)) + ksft_exit_fail_perror("restore affinity"); + munmap(mapping, page_size); + ksft_test_result_pass("MADV_COLD skips a remote pending-LRU page\n"); +} + +/* + * Exercise the same active-folio transition through a huge PMD, where the + * page-table aging operation differs from the PTE implementation. + */ +static void test_pmd_madvise_cold_active_folio(void) +{ + const size_t page_size =3D getpagesize(); + char *mapping; + char *drain =3D map_aligned_pages(page_size); + cpu_set_t old_mask, mask; + int pagemap_fd, kpageflags_fd; + unsigned long pfn; + uint64_t flags; + int cpu, fd, retry; + bool active =3D false; + + if (geteuid()) { + munmap(drain, page_size); + ksft_test_result_skip("requires root to read page flags\n"); + return; + } + fd =3D memfd_create("madvise-active-thp", 0); + if (fd < 0) + ksft_exit_fail_perror("memfd_create"); + if (ftruncate(fd, pmd_size)) + ksft_exit_fail_perror("ftruncate"); + mapping =3D map_aligned_file(fd, pmd_size, MAP_SHARED); + memset(mapping, 1, pmd_size); + if (msync(mapping, pmd_size, MS_SYNC) || + !collapse_shmem(mapping, pmd_size, 1)) { + munmap(drain, page_size); + munmap(mapping, pmd_size); + close(fd); + ksft_test_result_skip("could not allocate a PMD-sized shmem THP\n"); + return; + } + if (sched_getaffinity(0, sizeof(old_mask), &old_mask)) + ksft_exit_fail_perror("sched_getaffinity"); + cpu =3D sched_getcpu(); + if (cpu < 0) + ksft_exit_fail_perror("sched_getcpu"); + CPU_ZERO(&mask); + CPU_SET(cpu, &mask); + if (sched_setaffinity(0, sizeof(mask), &mask)) + ksft_exit_fail_perror("sched_setaffinity"); + + pagemap_fd =3D open("/proc/self/pagemap", O_RDONLY); + kpageflags_fd =3D open("/proc/kpageflags", O_RDONLY); + if (pagemap_fd < 0 || kpageflags_fd < 0) + ksft_exit_fail_perror("open page flags"); + pfn =3D pagemap_get_pfn(pagemap_fd, mapping); + if (pfn =3D=3D -1ul) + ksft_exit_fail_msg("could not read THP PFN\n"); + + for (retry =3D 0; retry < 10; retry++) { + if (pread(fd, drain, 1, 0) !=3D 1) + ksft_exit_fail_perror("pread"); + if (madvise(drain, page_size, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + if (pageflags_get(pfn, kpageflags_fd, &flags)) + ksft_exit_fail_perror("read kpageflags"); + if (flags & KPF_ACTIVE) { + active =3D true; + break; + } + } + if (!active) { + close(kpageflags_fd); + close(pagemap_fd); + sched_setaffinity(0, sizeof(old_mask), &old_mask); + munmap(drain, page_size); + munmap(mapping, pmd_size); + close(fd); + ksft_test_result_skip("could not activate a THP\n"); + return; + } + + if (madvise(mapping, pmd_size, MADV_COLD) || + madvise(drain, page_size, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + if (pageflags_get(pfn, kpageflags_fd, &flags)) + ksft_exit_fail_perror("read kpageflags"); + if (flags & KPF_ACTIVE) + ksft_exit_fail_msg("MADV_COLD did not deactivate the THP\n"); + if (!check_huge_shmem(mapping, pmd_size, 1, pmd_size)) + ksft_exit_fail_msg("MADV_COLD split an active THP\n"); + check_memory(mapping, pmd_size); + + close(kpageflags_fd); + close(pagemap_fd); + if (sched_setaffinity(0, sizeof(old_mask), &old_mask)) + ksft_exit_fail_perror("restore affinity"); + munmap(drain, page_size); + munmap(mapping, pmd_size); + close(fd); + ksft_test_result_pass("MADV_COLD deactivates an active THP\n"); +} + +/* + * MADV_PAGEOUT on half of a PMD-mapped THP must split the folio before + * reclaim so the unadvised half is not reclaimed as part of the THP. + */ +static void test_partial_pmd_madvise_pageout(void) +{ + char *mapping =3D map_aligned_pages(pmd_size); + + if (!collapse_all(mapping, pmd_size, 1)) { + munmap(mapping, pmd_size); + ksft_test_result_skip("could not allocate a PMD-sized THP\n"); + return; + } + pageout_half_thp(mapping); + + munmap(mapping, pmd_size); + ksft_test_result_pass("partial MADV_PAGEOUT splits a PMD-mapped THP\n"); +} + +/* + * mprotect() replaces the huge PMD with PTEs without splitting the physic= al + * THP. MADV_PAGEOUT on half of that PTE-mapped THP must split the folio. + */ +static void test_partial_pte_madvise_pageout(void) +{ + char *mapping =3D map_aligned_pages(pmd_size); + + if (!collapse_all(mapping, pmd_size, 1)) { + munmap(mapping, pmd_size); + ksft_test_result_skip("could not allocate a PMD-sized THP\n"); + return; + } + split_pmd_mapping(mapping); + pageout_half_thp(mapping); + + munmap(mapping, pmd_size); + ksft_test_result_pass("partial MADV_PAGEOUT splits a PTE-mapped THP\n"); +} + +static pid_t fork_waiting_child(int pipefd[2]) +{ + pid_t pid; + + if (pipe(pipefd)) + ksft_exit_fail_perror("pipe"); + pid =3D fork(); + if (pid < 0) + ksft_exit_fail_perror("fork"); + if (!pid) { + char byte; + + close(pipefd[1]); + while (read(pipefd[0], &byte, 1) < 0 && errno =3D=3D EINTR) + ; + _exit(0); + } + close(pipefd[0]); + return pid; +} + +/* + * fork() gives the THP another mapping, then mprotect() PTE-maps it in the + * parent. Assert that partial MADV_COLD leaves the shared physical THP and + * its contents intact. + */ +static void test_shared_pte_mapped_madvise_cold(void) +{ + char *mapping =3D map_aligned_pages(pmd_size); + int pipefd[2], status; + pid_t pid; + + if (!collapse_all(mapping, pmd_size, 1)) { + munmap(mapping, pmd_size); + ksft_test_result_skip("could not allocate a PMD-sized THP\n"); + return; + } + pid =3D fork_waiting_child(pipefd); + + split_pmd_mapping(mapping); + if (madvise(mapping, pmd_size / 2, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + if (!check_large_folios(mapping, pmd_size, 1, pmd_size)) + ksft_exit_fail_msg("MADV_COLD split a shared large folio\n"); + check_memory(mapping, pmd_size); + + close(pipefd[1]); + if (waitpid(pid, &status, 0) !=3D pid || !WIFEXITED(status) || + WEXITSTATUS(status)) + ksft_exit_fail_msg("child process failed\n"); + munmap(mapping, pmd_size); + ksft_test_result_pass("MADV_COLD skips a shared PTE-mapped THP\n"); +} + +/* + * A full-range PTE walk reaches the mapcount check rather than the partial + * folio split check. A second mapping must still prevent MADV_COLD from + * operating on the shared physical THP. + */ +static void test_full_shared_pte_mapped_madvise_cold(void) +{ + char *mapping =3D map_aligned_pages(pmd_size); + int pipefd[2], status; + pid_t pid; + + if (!collapse_all(mapping, pmd_size, 1)) { + munmap(mapping, pmd_size); + ksft_test_result_skip("could not allocate a PMD-sized THP\n"); + return; + } + pid =3D fork_waiting_child(pipefd); + + split_pmd_mapping(mapping); + if (madvise(mapping, pmd_size, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + if (!check_large_folios(mapping, pmd_size, 1, pmd_size)) + ksft_exit_fail_msg("MADV_COLD split a shared large folio\n"); + check_memory(mapping, pmd_size); + + close(pipefd[1]); + if (waitpid(pid, &status, 0) !=3D pid || !WIFEXITED(status) || + WEXITSTATUS(status)) + ksft_exit_fail_msg("child process failed\n"); + munmap(mapping, pmd_size); + ksft_test_result_pass("full MADV_COLD skips a shared PTE-mapped THP\n"); +} + +/* + * fork() gives a PMD-mapped THP another mapping. Assert that partial + * MADV_COLD leaves the shared physical THP and its contents intact. + */ +static void test_shared_pmd_madvise_cold(void) +{ + char *mapping =3D map_aligned_pages(pmd_size); + int pipefd[2], status; + pid_t pid; + + if (!collapse_all(mapping, pmd_size, 1)) { + munmap(mapping, pmd_size); + ksft_test_result_skip("could not allocate a PMD-sized THP\n"); + return; + } + pid =3D fork_waiting_child(pipefd); + + if (madvise(mapping, pmd_size / 2, MADV_COLD)) + ksft_exit_fail_perror("MADV_COLD"); + if (!check_large_folios(mapping, pmd_size, 1, pmd_size)) + ksft_exit_fail_msg("MADV_COLD split a shared PMD-mapped THP\n"); + check_memory(mapping, pmd_size); + + close(pipefd[1]); + if (waitpid(pid, &status, 0) !=3D pid || !WIFEXITED(status) || + WEXITSTATUS(status)) + ksft_exit_fail_msg("child process failed\n"); + munmap(mapping, pmd_size); + ksft_test_result_pass("MADV_COLD skips a shared PMD-mapped THP\n"); +} + +static char *map_readonly_file(size_t size, int flags, int *fd) +{ + char template[] =3D "/tmp/madvise-pageout-XXXXXX"; + char *mapping; + + *fd =3D mkstemp(template); + if (*fd < 0) + ksft_exit_fail_perror("mkstemp"); + if (unlink(template) || ftruncate(*fd, size) || fchmod(*fd, 0400)) + ksft_exit_fail_perror("prepare file"); + mapping =3D mmap(NULL, size, PROT_READ | PROT_WRITE, flags, *fd, 0); + if (mapping =3D=3D MAP_FAILED) + ksft_exit_fail_perror("mmap"); + + return mapping; +} + +static bool pageout_until_evicted(char *mapping, size_t size, int pagemap_= fd) +{ + int retry; + + for (retry =3D 0; retry < 100; retry++) { + if (madvise(mapping, size, MADV_PAGEOUT)) + ksft_exit_fail_perror("MADV_PAGEOUT"); + if (!pagemap_is_populated(pagemap_fd, mapping)) + return true; + usleep(10000); + } + return false; +} + +/* + * The owner of a file may page out its clean page-cache pages. Assert that + * MADV_PAGEOUT removes the populated PTE and preserves the file contents. + */ +static void test_pageout_file(void) +{ + const size_t page_size =3D getpagesize(); + char *mapping; + int pagemap_fd; + bool evicted; + int fd; + + mapping =3D map_readonly_file(page_size, MAP_PRIVATE, &fd); + pagemap_fd =3D open("/proc/self/pagemap", O_RDONLY); + if (pagemap_fd < 0) + ksft_exit_fail_perror("open pagemap"); + if (mapping[0]) + ksft_exit_fail_msg("new file is not zero-filled\n"); + + evicted =3D pageout_until_evicted(mapping, page_size, pagemap_fd); + if (mapping[0]) + ksft_exit_fail_msg("file mapping contents changed\n"); + + close(pagemap_fd); + munmap(mapping, page_size); + close(fd); + if (!evicted) { + ksft_test_result_skip("MADV_PAGEOUT did not evict the file page\n"); + return; + } + ksft_test_result_pass("MADV_PAGEOUT evicts an authorized file page\n"); +} + +static int pageout_shared_file_without_permission(char *mapping, + size_t page_size) +{ + int pagemap_fd; + + pagemap_fd =3D open("/proc/self/pagemap", O_RDONLY); + if (pagemap_fd < 0) + return KSFT_FAIL; + if (setgid(65534) || setuid(65534)) + return KSFT_FAIL; + if (mapping[0] || !pagemap_is_populated(pagemap_fd, mapping)) + return KSFT_FAIL; + if (madvise(mapping, page_size, MADV_PAGEOUT)) + return KSFT_FAIL; + if (!pagemap_is_populated(pagemap_fd, mapping) || mapping[0]) + return KSFT_FAIL; + return KSFT_PASS; +} + +/* + * A caller without file write permission may not page out a shared file + * mapping. Assert that MADV_PAGEOUT succeeds without evicting its file pa= ge. + */ +static void test_pageout_unauthorized_shared_file(void) +{ + const size_t page_size =3D getpagesize(); + char *mapping; + int fd, status; + pid_t pid; + + if (geteuid()) { + ksft_test_result_skip("requires root to change credentials\n"); + return; + } + + mapping =3D map_readonly_file(page_size, MAP_SHARED, &fd); + + pid =3D fork(); + if (pid < 0) + ksft_exit_fail_perror("fork"); + if (!pid) + _exit(pageout_shared_file_without_permission(mapping, page_size)); + + if (waitpid(pid, &status, 0) !=3D pid || !WIFEXITED(status) || + WEXITSTATUS(status) !=3D KSFT_PASS) + ksft_exit_fail_msg("unauthorized child evicted a shared file page\n"); + munmap(mapping, page_size); + close(fd); + ksft_test_result_pass("MADV_PAGEOUT skips an unauthorized shared file\n"); +} + +/* + * An unauthorized private mapping may page out anonymous COW pages, but n= ot + * its file folios. Assert that partial MADV_PAGEOUT filters a large shmem + * folio before attempting to split or reclaim it. + */ +static void test_pageout_anon_only_large_folio(void) +{ + char *mapping, *shared; + gid_t old_egid; + int fd; + + if (geteuid()) { + ksft_test_result_skip("requires root to change credentials\n"); + return; + } + fd =3D memfd_create("madvise-pageout-large", 0); + if (fd < 0) + ksft_exit_fail_perror("memfd_create"); + if (ftruncate(fd, pmd_size)) + ksft_exit_fail_perror("ftruncate"); + shared =3D map_aligned_file(fd, pmd_size, MAP_SHARED); + memset(shared, 1, pmd_size); + if (msync(shared, pmd_size, MS_SYNC)) + ksft_exit_fail_perror("msync"); + munmap(shared, pmd_size); + + mapping =3D map_aligned_file(fd, pmd_size, MAP_PRIVATE); + if (!collapse_shmem(mapping, pmd_size, 1)) { + munmap(mapping, pmd_size); + close(fd); + ksft_test_result_skip("could not allocate a PMD-sized shmem THP\n"); + return; + } + if (fchmod(fd, 0400)) + ksft_exit_fail_perror("fchmod"); + + /* Retain saved uid 0 so this process can restore its credentials. */ + old_egid =3D getegid(); + if (setegid(65534) || seteuid(65534)) + ksft_exit_fail_perror("drop privileges"); + if (madvise(mapping, pmd_size / 2, MADV_PAGEOUT)) + ksft_exit_fail_perror("MADV_PAGEOUT"); + if (seteuid(0) || setegid(old_egid)) + ksft_exit_fail_perror("restore privileges"); + + if (!check_huge_shmem(mapping, pmd_size, 1, pmd_size) || + mapping[0] !=3D 1 || mapping[pmd_size - 1] !=3D 1) + ksft_exit_fail_msg("MADV_PAGEOUT changed a protected large folio\n"); + munmap(mapping, pmd_size); + close(fd); + ksft_test_result_pass("MADV_PAGEOUT filters a protected large folio\n"); +} + +/* + * A private file mapping can contain both anonymous COW and file-backed + * pages. As a caller without file write permission, assert that PAGEOUT s= waps + * the COW page but leaves the file-backed page resident. + */ +static void test_pageout_anon_only(void) +{ + const size_t page_size =3D getpagesize(); + char *mapping; + int pagemap_fd; + bool swapped; + int fd; + + if (geteuid()) { + ksft_test_result_skip("requires root to change credentials\n"); + return; + } + + mapping =3D map_readonly_file(2 * page_size, MAP_PRIVATE, &fd); + pagemap_fd =3D open("/proc/self/pagemap", O_RDONLY); + if (pagemap_fd < 0) + ksft_exit_fail_perror("open pagemap"); + + if (setgid(65534) || setuid(65534)) + ksft_exit_fail_perror("drop privileges"); + if (mapping[0] || mapping[page_size]) + ksft_exit_fail_msg("new file is not zero-filled\n"); + mapping[0] =3D 1; + if (madvise(mapping, 2 * page_size, MADV_PAGEOUT)) + ksft_exit_fail_perror("MADV_PAGEOUT"); + if (!pagemap_is_populated(pagemap_fd, mapping + page_size)) + ksft_exit_fail_msg("MADV_PAGEOUT evicted a protected file page\n"); + swapped =3D pagemap_is_swapped(pagemap_fd, mapping); + if (mapping[0] !=3D 1 || mapping[page_size]) + ksft_exit_fail_msg("private file mapping contents changed\n"); + + close(pagemap_fd); + munmap(mapping, 2 * page_size); + close(fd); + if (!swapped) { + ksft_test_result_skip("MADV_PAGEOUT did not swap the COW page\n"); + return; + } + ksft_test_result_pass("MADV_PAGEOUT filters private file pages\n"); +} + +int main(void) +{ + pmd_size =3D read_pmd_pagesize(); + + ksft_print_header(); + ksft_set_plan(28); + if (!pmd_size) + ksft_exit_skip("PMD-sized THPs are not supported\n"); + + test_full_pmd_madvise_cold(); + test_repeated_pmd_madvise_cold(); + test_partial_pmd_madvise_cold(); + test_pte_mapped_madvise_cold(); + test_madvise_cold_pte_hole(); + test_madvise_cold_zero_page(); + test_madvise_cold_huge_zero_page(); + test_madvise_lru_locked_vma(); + test_shared_pmd_madvise_cold(); + test_full_pmd_madvise_pageout(); + test_madvise_cold_swapped_pte(); + test_partial_pinned_pmd_madvise_cold(); + test_partial_pinned_pte_madvise_cold(); + test_concurrent_partial_madvise_cold(); + test_madvise_pageout_unevictable(); + test_madvise_pageout_unevictable_thp(); + test_madvise_pageout_migration(); + test_madvise_cold_active_folio(); + test_madvise_cold_remote_lru_batch(); + test_pmd_madvise_cold_active_folio(); + test_partial_pmd_madvise_pageout(); + test_partial_pte_madvise_pageout(); + test_shared_pte_mapped_madvise_cold(); + test_full_shared_pte_mapped_madvise_cold(); + test_pageout_file(); + test_pageout_unauthorized_shared_file(); + test_pageout_anon_only_large_folio(); + test_pageout_anon_only(); + ksft_finished(); +} diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/self= tests/mm/run_vmtests.sh index 6990485b1a9da..bc136ead10f91 100755 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -51,6 +51,8 @@ separated by spaces: hmm smoke tests - madv_guard test madvise(2) MADV_GUARD_INSTALL and MADV_GUARD_REMOVE options +- madvise + test MADV_COLD and MADV_PAGEOUT - madv_populate test memadvise(2) MADV_POPULATE_{READ,WRITE} options - memfd_secret @@ -327,6 +329,9 @@ CATEGORY=3D"hmm" run_test bash ./test_hmm.sh smoke # MADV_GUARD_INSTALL and MADV_GUARD_REMOVE tests CATEGORY=3D"madv_guard" run_test ./guard-regions =20 +# MADV_COLD and MADV_PAGEOUT tests +CATEGORY=3D"madvise" run_test ./madvise + # MADV_POPULATE_READ and MADV_POPULATE_WRITE tests CATEGORY=3D"madv_populate" run_test ./madv_populate =20 diff --git a/tools/testing/selftests/mm/vm_util.c b/tools/testing/selftests= /mm/vm_util.c index 31d331c1c4521..af0a7f65bfd9f 100644 --- a/tools/testing/selftests/mm/vm_util.c +++ b/tools/testing/selftests/mm/vm_util.c @@ -373,8 +373,8 @@ static bool __check_pmd_huge(void *addr, char *pattern,= int nr_hpages, return thp =3D=3D (nr_hpages * (hpage_size >> 10)); } =20 -static bool check_large_folios(void *addr, size_t len, int nr_hpages, - uint64_t hpage_size) +bool check_large_folios(void *addr, size_t len, int nr_hpages, + uint64_t hpage_size) { int order =3D 0, pagesize =3D getpagesize(); unsigned int nr_pages =3D hpage_size / pagesize; diff --git a/tools/testing/selftests/mm/vm_util.h b/tools/testing/selftests= /mm/vm_util.h index 072a6c756c517..33ae68d721df9 100644 --- a/tools/testing/selftests/mm/vm_util.h +++ b/tools/testing/selftests/mm/vm_util.h @@ -22,10 +22,14 @@ #define PM_SWAP BIT_ULL(62) #define PM_PRESENT BIT_ULL(63) =20 +#define KPF_LRU BIT_ULL(5) +#define KPF_ACTIVE BIT_ULL(6) #define KPF_COMPOUND_HEAD BIT_ULL(15) #define KPF_COMPOUND_TAIL BIT_ULL(16) +#define KPF_UNEVICTABLE BIT_ULL(18) #define KPF_HWPOISON BIT_ULL(19) #define KPF_THP BIT_ULL(22) +#define KPF_ZERO_PAGE BIT_ULL(24) /* * Ignore the checkpatch warning, we must read from x but don't want to do * anything with it in order to trigger a read page fault. We therefore mu= st use @@ -97,6 +101,8 @@ unsigned long rss_anon(void); bool check_huge_anon(void *addr, size_t len, int nr_hpages, uint64_t hpage= _size); bool check_huge_file(void *addr, size_t len, int nr_hpages, uint64_t hpage= _size); bool check_huge_shmem(void *addr, size_t len, int nr_hpages, uint64_t hpag= e_size); +bool check_large_folios(void *addr, size_t len, int nr_hpages, + uint64_t hpage_size); int64_t allocate_transhuge(void *ptr, int pagemap_fd); int pageflags_get(unsigned long pfn, int kpageflags_fd, uint64_t *flags); int gather_folio_orders(char *vaddr_start, size_t len, --=20 2.53.0-Meta From nobody Thu Sep 24 14:26:50 2026 Received: from mail-qk2-f12.google.com (mail-qk2-f12.google.com [74.125.230.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 44E184FC8D1 for ; Tue, 22 Sep 2026 23:58:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.204 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121521; cv=none; b=CpHIncA+3qPYdj/K0JZS2ghiy3jFQ2G19l9Av4qOO8H7sbf1rIlFzQRZZqSjtxpsVBuY3RozeWytuM0e5ZxUCML6PdFfRmicgoF+eKbwOrEJRUoovPOKf/sNkv47Mo4XSXtDZlGJU/uOzAdVo1v2KobT3iLelRLsmUWJnJ2XmO8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121521; c=relaxed/simple; bh=EK7vI9KcRnwekK8Ynl1zk9PVNmfiyEBbTFnZNQww8Vk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CPlgCHd1m5v8Rxe9FLjgiBYLRXmpItHnaq/giAS3T1mh1Hr70B27rwAjre7I7OZQvqArmvEw7zXOoXCJZ4HJwpPtiMB+3+UZtvwq2AxAP9pcCZ4VmKGFvOZXu/XUkGgGcqvDAMo8hovCipxWPwXJJ7KYa4k/lVy04HHv/luEVxU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=kuchFlsf; arc=none smtp.client-ip=74.125.230.204 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="kuchFlsf" Received: by mail-qk2-f12.google.com with SMTP id d75a77b69052e-52fb76ec395so4199211cf.2 for ; Tue, 22 Sep 2026 16:58:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1790121518; x=1790726318; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Cv67mS8beABpbmQAZ394RKqHc9TebilH66SXRMDKuYM=; b=kuchFlsfo8e/wdQdQUZBvc3/ALkKndlwQhowxd4RKdio+DBMgYD3jA/8H9dEnv+Woj E+77CsEAjJ1TMzuUwX+9BAsx0pMgkpTXufqGa1e0s6XwZVEGpdmmO4RHudb9M47V+cAe CtujIHn1BZ/Jlz5ISZB1ZPwEWLvg1xz7bxBJ16CjRIcwvOD82AtqGKuAjgc+lmgOWKtl PiM7gvCHrbAnqaTx5e+UUjmqyjDKsUyTMgy1ckmlNhJDQaUpbeMGpXle8+jdS1q7UN1Z tH5I6dpQ4Ayw/OPelgRMxfaZSk3Qv3nfUBPE+hoPv61xcSonF+ZV/ssQRb2H4AKiMYqE p1Jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790121518; x=1790726318; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Cv67mS8beABpbmQAZ394RKqHc9TebilH66SXRMDKuYM=; b=IHgYwMmvNUrfv4iOBW2nZXhGJxOsWw2jMtj9tyNugjZnjpIvlgVJJM36B1i2UK/Jza y9ejDSIvYyhP+K3XAKA9O3iZtU7ST6LZGW5JOpHObED38cZmgfGUI3muHbgJlvvGt7/o Gh8EAmLJ8LMG9c82b3S+9wTE20vaTOwy7yJxEjLTcI3MIyYDBWL8fkZxWd4OXWk+W3km dl9mMBN6b/cjWb7W222IEeJiPehLBsT04hn+Wk2vRPgiYGr2b9d9P2NjI7REGoP+nn+G xmTU6oLpSUvN4a3zVsWwg27RuLaVEPI/mNdZkqy/kLgeOktQCE+g25xzA4Jhq8pqvHl9 v5Kg== X-Gm-Message-State: AFuF++kecb73NCA/m8ttpClvH8lF+UmyqJM5LjVvl8HdoWbVscAlblwC oiq6WVWlTdveapR4WqOIV/xZqSbXfvxtbTSmp+ArBptfQzIHyS9kHnd09JrdyceCFfQ= X-Gm-Gg: AYBFou20rjLXEWIOnYhRs77qRAmygBWhSe3I/TYViKHVmhJQZC6GjPfavO7PvXbtk/3 osM+fTCABIoXmvTyHYuakrImKqau4u7X1IqHctuIqh189XUnH3PgU3a5hsGqnfU42in2h/gDAOi 81m2kTjBZYmK3n1OmqEQ93GxBvJ5HeYykKoFV0DyGlwHn6XuLPZQNMw15z5GqvscHso78mrrPik Zody717i/+zVhEoibBbHSPfgvnZ20g0TKFNac5TpGhlX9MW8Ie6ImhHKe2KW/n6cNx5rfGy1Ju7 pdnK+4AzM/98qBHWAT2tjgJA+0pwRnthaGRySXMqiY+L3kmoCHgm4ykl5chbR8G8lmje3KiwE8c EkJBMUVRwLv8QQRue4mHL9hrCCScx9sx2Qs8hYPuR+EkwyAklb8qz9ZDlvhxSy30QnfF1YgUyfJ E8W3HT3pHH2Cni2W5MDWpj6d4q/s2hQzUCGHssU+whJo22sJB+ULEY5X8J+Lq8xshRnZ2I9BtPe qilPHgFmu2icNZ3H5LdqmHm7HKEqCz463S4CiHqj6LeMT7xwDYwNGNIXy2f X-Received: by 2002:a05:622a:6101:b0:532:b08b:45cd with SMTP id d75a77b69052e-532eacd610fmr16957421cf.31.1790121517717; Tue, 22 Sep 2026 16:58:37 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-60-52.washdc.fios.verizon.net. [173.79.60.52]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-532eb3c6715sm8323261cf.30.2026.09.22.16.58.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 16:58:36 -0700 (PDT) From: Gregory Price To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-team@meta.com, akpm@linux-foundation.org, liam@infradead.org, ljs@kernel.org, david@kernel.org, vbabka@kernel.org, jannh@google.com, rppt@kernel.org, surenb@google.com, mhocko@suse.com, shuah@kernel.org, "Gregory Price (Meta)" Subject: [PATCH 02/10] mm/madvise: name the shared LRU PMD callback Date: Tue, 22 Sep 2026 19:58:22 -0400 Message-ID: <20260922235830.2350770-3-gourry@gourry.net> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922235830.2350770-1-gourry@gourry.net> References: <20260922235830.2350770-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The MADV_COLD and MADV_PAGEOUT page-walk callback is named after its PTE implementation even though it is registered as a PMD callback and serves both LRU operations. Rename it to madvise_lru_pmd_entry() before separating the PMD and PTE paths. No functional change intended. Assisted-by: LLM Signed-off-by: Gregory Price (Meta) Reviewed-by: Lorenzo Stoakes (ARM) --- mm/madvise.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 00b1be655a8b5..83d54ab385da8 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -361,9 +361,8 @@ static inline int madvise_folio_pte_batch(unsigned long= addr, unsigned long end, FPB_MERGE_YOUNG_DIRTY); } =20 -static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, - unsigned long addr, unsigned long end, - struct mm_walk *walk) +static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr, + unsigned long end, struct mm_walk *walk) { struct madvise_walk_private *private =3D walk->private; struct mmu_gather *tlb =3D private->tlb; @@ -581,7 +580,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, } =20 static const struct mm_walk_ops cold_walk_ops =3D { - .pmd_entry =3D madvise_cold_or_pageout_pte_range, + .pmd_entry =3D madvise_lru_pmd_entry, .walk_lock =3D PGWALK_RDLOCK, }; =20 --=20 2.53.0-Meta From nobody Thu Sep 24 14:26:50 2026 Received: from mail-qk2-f13.google.com (mail-qk2-f13.google.com [74.125.230.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8FC42530DEA for ; Tue, 22 Sep 2026 23:58:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.205 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121525; cv=none; b=nrcABKULKJtJsMKOQZ2BctO6IfBL/1KLvvC5vov0EHsJInEVmWGG+jxOx7qJfzXSpIBU7p+uvBAhm9nOXrP/CDjorbnamYBP2JtulOe/aQnXPeuk3WnMMNudyEwCuzy8WHqUApKyA6qR7Gu3VsWv6W8VgPiNVWUlLDegq/xL4TI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121525; c=relaxed/simple; bh=Z+b3w7hN29ny0n+lgoibjklp6h6u1gN7hC/nkJ+vNU8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=is12o8ZmmqU1MLjbO5de5SQ9Xsimy+KFy2OCOhh3QC33VErx+fk14HM8vLYioKg32HTHGxFRfTIoHdKkHZW9NiSBJNF56Cn63+2H4p7sfNPtblPAReBS86s1sRwTfT3ktIzwYqBTIuTFQw+j+DKCK4yxehYg0zo3Lv/v+QCOdak= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=h4a7kRms; arc=none smtp.client-ip=74.125.230.205 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="h4a7kRms" Received: by mail-qk2-f13.google.com with SMTP id d75a77b69052e-530e2f50d01so4353641cf.0 for ; Tue, 22 Sep 2026 16:58:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1790121519; x=1790726319; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=/oNw9RPMr3V+/qUjJEYCXBc3lvf+tyCPjegYTQeEM3I=; b=h4a7kRms1pVwjzo/b9JB0U2KBFZevFmriKnNg9xWLUdLS4HzGrr7Hgq5DEYfM0CBtt ja1SI+7RiMU4DIxB9uFwo/rMyh78MRAwQlyFo2daQwwO+qsGgvYTrmplIRg2ogP6/oWh haOg5p534kDxTnhIiO0k+u37KE6dRuj31PJ5zwsEnsfz1+QTqX7S1H6pL+eDqKE8fqms eQHjfkpN2/C9RPQKwPAzzyKlEQ5F/Azhn9PgtX6UBmlaU0NJXQCqjLuzPcgJZtLbOebY SyDZdD6mfvMjznc5VvZxNrYF3adzn56iSpuTK6+dijswNE5ggT5Zv3lbONkUntKrvaI3 yVAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790121519; x=1790726319; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=/oNw9RPMr3V+/qUjJEYCXBc3lvf+tyCPjegYTQeEM3I=; b=PrjmpupJKxoFyG0p7JdftTWIftgxQTZ3YTMgTixvNiufPriDBGmL6tlYjZ+0T8BSZO UJBG5pIWeO3GB3KjoDP+2uuqgaqJw4X9cwvfdSSqQF+sM/sUPn7A/OvihlcXv3KRbubY zRbgMbdQq+fyviaFL6yxRkAZMF8MgUTdpJQWy30m/FlghSKSiWYZHGgVuyzW/0Ef1aoS KejBGVY5WJzSATqsD2euqeQ1IcDPunIiDuTazTTX2H14m5E9bdWQJOgq3D1Wp5IJjuur D28+acE3uBhyMUQTIwXorLQzYpJ8cicxeZfdeqFAFAYpFHyQHTZb++hd3GwG2hPbyi8+ NElg== X-Gm-Message-State: AFuF++mim/fRaQsdnq2IwVMjdAy2/xgwPGdPTKXDPpLkK2sReKiCgzER VGYXzh2vfXqVyIqJbsCioFb6/nd3Jb3lOE2dDpL+GkeR2eb6RixfHLYV/ql6Gn0naeA= X-Gm-Gg: AYBFou0bC8kP9HIXlBdjjb+9yE+NZp+XGl6RXwirEuwU6085UUycIedGGK67jPmV2ly t8W6vL5JsOBsa1kL7fFY2KO98UlEhFQv1KU7iACy+4HmgI297uf9qQD3YhKZRo/ULdW92DHdqtz 3HSanN3Vhj3+qI/OAU+tD4t074i8yk2pUO8wS9BJ+qeaL/iDk0/iRjbPxkYW90je/NzcmC0dNRO UtwPPh6iGVnMk25G4YvRhY5HBCLQ9tN9eVYzYxUbOq0CcGwGsuE4QTpqWSL1bGpykvXWeCgNNeb /2Tzde66fxZ9ugJY484pXZFaIZZUQTh/BonhozK7wDEUYoxoYZgNyH1oimvkar+mgfdwqS4mUnX IexGPMXosvXknnPBzfF7roB0tUXVD8OtDefILocFN6WHOjdSrsEa8DBJr9fQb3Lfr174WsiG8Ra uojyHFeL5u9Dg2LdEsYPeuesF0XimfX5WAFWKNOa70aKz1LfLGroNO2YyjBRzjaztNYgLkatxxN Q/kZF5soAgWnufJCyjDIOfHtha2454cl7vGsWRP9AY9iPcxoWpqEwjet+u80Dz355Im7+0= X-Received: by 2002:a05:622a:305:b0:530:f77a:89aa with SMTP id d75a77b69052e-532eacb0064mr19428811cf.30.1790121519361; Tue, 22 Sep 2026 16:58:39 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-60-52.washdc.fios.verizon.net. [173.79.60.52]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-532eb3c6715sm8323261cf.30.2026.09.22.16.58.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 16:58:39 -0700 (PDT) From: Gregory Price To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-team@meta.com, akpm@linux-foundation.org, liam@infradead.org, ljs@kernel.org, david@kernel.org, vbabka@kernel.org, jannh@google.com, rppt@kernel.org, surenb@google.com, mhocko@suse.com, shuah@kernel.org, "Gregory Price (Meta)" Subject: [PATCH 03/10] mm/madvise: factor shared LRU folio handling Date: Tue, 22 Sep 2026 19:58:23 -0400 Message-ID: <20260922235830.2350770-4-gourry@gourry.net> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922235830.2350770-1-gourry@gourry.net> References: <20260922235830.2350770-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The huge-PMD and PTE paths duplicate folio filtering, reference clearing, deactivation and pageout isolation. Keeping both copies synchronized obscures the page-table-specific control flow. Factor the common filtering and LRU operation into helpers. Keep the lock-before-reference sequence visible at each split site because an early reference can prevent split_folio() from succeeding. No functional change intended. Assisted-by: LLM Signed-off-by: Gregory Price (Meta) Reviewed-by: Lorenzo Stoakes (ARM) --- mm/madvise.c | 90 +++++++++++++++++++++++++--------------------------- 1 file changed, 43 insertions(+), 47 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 83d54ab385da8..c345fef23f15d 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -361,6 +361,39 @@ static inline int madvise_folio_pte_batch(unsigned lon= g addr, unsigned long end, FPB_MERGE_YOUNG_DIRTY); } =20 +static inline void +madvise_lru_folio(struct folio *folio, bool pageout, + struct list_head *folio_list) +{ + /* + * Clear references before deactivating or reclaiming the folio. This can + * make idle-page tracking miss recent accesses. + */ + folio_clear_referenced(folio); + folio_test_clear_young(folio); + if (folio_test_active(folio)) + folio_set_workingset(folio); + + if (!pageout) { + folio_deactivate(folio); + return; + } + + if (!folio_isolate_lru(folio)) + return; + if (folio_test_unevictable(folio)) + folio_putback_lru(folio); + else + list_add(&folio->lru, folio_list); +} + +static bool madvise_lru_folio_is_filtered(struct folio *folio, + bool pageout_anon_only) +{ + return folio_maybe_mapped_shared(folio) || + (pageout_anon_only && !folio_test_anon(folio)); +} + static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) { @@ -373,15 +406,14 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned= long addr, spinlock_t *ptl; struct folio *folio =3D NULL; LIST_HEAD(folio_list); - bool pageout_anon_only_filter; unsigned int batch_count =3D 0; + bool pageout_anon_only; int nr; =20 if (fatal_signal_pending(current)) return -EINTR; - - pageout_anon_only_filter =3D pageout && !vma_is_anonymous(vma) && - !can_do_file_pageout(vma); + pageout_anon_only =3D pageout && !vma_is_anonymous(vma) && + !can_do_file_pageout(vma); =20 #ifdef CONFIG_TRANSPARENT_HUGEPAGE if (pmd_trans_huge(*pmd)) { @@ -407,11 +439,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned = long addr, if (folio_is_zone_device(folio)) goto huge_unlock; =20 - /* Do not interfere with other mappings of this folio */ - if (folio_maybe_mapped_shared(folio)) - goto huge_unlock; - - if (pageout_anon_only_filter && !folio_test_anon(folio)) + if (madvise_lru_folio_is_filtered(folio, pageout_anon_only)) goto huge_unlock; =20 if (next - addr !=3D HPAGE_PMD_SIZE) { @@ -437,19 +465,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned = long addr, tlb_remove_pmd_tlb_entry(tlb, pmd, addr); } =20 - folio_clear_referenced(folio); - folio_test_clear_young(folio); - if (folio_test_active(folio)) - folio_set_workingset(folio); - if (pageout) { - if (folio_isolate_lru(folio)) { - if (folio_test_unevictable(folio)) - folio_putback_lru(folio); - else - list_add(&folio->lru, &folio_list); - } - } else - folio_deactivate(folio); + madvise_lru_folio(folio, pageout, &folio_list); huge_unlock: spin_unlock(ptl); if (pageout) @@ -502,9 +518,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned l= ong addr, if (nr < folio_nr_pages(folio)) { int err; =20 - if (folio_maybe_mapped_shared(folio)) - continue; - if (pageout_anon_only_filter && !folio_test_anon(folio)) + if (madvise_lru_folio_is_filtered(folio, pageout_anon_only)) continue; if (!folio_trylock(folio)) continue; @@ -537,7 +551,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned l= ong addr, folio_mapcount(folio) !=3D folio_nr_pages(folio)) continue; =20 - if (pageout_anon_only_filter && !folio_test_anon(folio)) + if (pageout_anon_only && !folio_test_anon(folio)) continue; =20 if (!pageout && pte_young(ptent)) { @@ -546,25 +560,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned = long addr, tlb_remove_tlb_entries(tlb, pte, nr, addr); } =20 - /* - * We are deactivating a folio for accelerating reclaiming. - * VM couldn't reclaim the folio unless we clear PG_young. - * As a side effect, it makes confuse idle-page tracking - * because they will miss recent referenced history. - */ - folio_clear_referenced(folio); - folio_test_clear_young(folio); - if (folio_test_active(folio)) - folio_set_workingset(folio); - if (pageout) { - if (folio_isolate_lru(folio)) { - if (folio_test_unevictable(folio)) - folio_putback_lru(folio); - else - list_add(&folio->lru, &folio_list); - } - } else - folio_deactivate(folio); + madvise_lru_folio(folio, pageout, &folio_list); } =20 out: @@ -651,8 +647,8 @@ static long madvise_pageout(struct madvise_behavior *ma= dv_behavior) * owner nor write capable of the file. We allow private file mappings * further to pageout dirty anon pages. */ - if (!vma_is_anonymous(vma) && (!can_do_file_pageout(vma) && - (vma->vm_flags & VM_MAYSHARE))) + if (!vma_is_anonymous(vma) && !can_do_file_pageout(vma) && + (vma->vm_flags & VM_MAYSHARE)) return 0; =20 lru_add_drain(); --=20 2.53.0-Meta From nobody Thu Sep 24 14:26:50 2026 Received: from mail-qk2-f13.google.com (mail-qk2-f13.google.com [74.125.230.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E65AA13AF2 for ; Tue, 22 Sep 2026 23:58:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.205 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121525; cv=none; b=NIb6uSA6ac8VM5iJj9Af+Xcd7w/JHO0Xgy2JmGe+81CAJpiZnp/zBLj5uAql45bwR/dh8vsQZML9cC23isSWmbT7T3o42EGfHvt9menMffU4QhyBdvKElsJS8wrmPGEgdocniU3oXiL0gc2q+MoILmW5+8pbNW7eH1Nh4t9c9so= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121525; c=relaxed/simple; bh=zMZuj4OSysiY14sv8fTeBIk97LrMvKFmXN9V3kYNFd8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=E6NbH/zxiVBY30lR3lDh6BywvMUaW6ojrFvO7bbmeMZkzXao5ClFdKT/MQi9KBolYv1CFBt+EYBSeVyYdHraoy2m3jeUx5g3QH9ByuN8McBk8SkeQZPfAiXTP4W3gfbaacqNUKXrwLYyfW4kNAZx+9x6TqTnK8n1MPc+QMI8V0I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=ZQlPmVvq; arc=none smtp.client-ip=74.125.230.205 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="ZQlPmVvq" Received: by mail-qk2-f13.google.com with SMTP id d75a77b69052e-52fb76c9deeso3986421cf.1 for ; Tue, 22 Sep 2026 16:58:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1790121521; x=1790726321; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=+hLggoqTstCCp/LK3cQnyxDjfuxVFKuLURbcW1u38EU=; b=ZQlPmVvqnrKTa/s7wZXOFwt/c41ay6syY03TW+bkacaUMnzOBh2O4gIkr5lK71nepG lP5a7YPcrMmn1KP1BSXTBioNccWJCOWBrjo7kgblEgBa4tuxD7KiLUEC50Tl1y35e9Nj pJryVWIR0xi0HBesbyainB71Ho0tAqPElanRPh5BB6nbm2Gr7HHbGnhGFdTs2ffOnp9+ U+bFboRvOHCjjlgip3QfeLSIKHA+D+Dvex7lWXE7/3oEL0yoQk1traeOx80cV1Ifj8O+ +vGD74gp+URIAh49cWni2XmFUb4fyp1KT5xBb+v+OqC3sbJYerFtT6gZAybz4JmCNr6a DBmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790121521; x=1790726321; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=+hLggoqTstCCp/LK3cQnyxDjfuxVFKuLURbcW1u38EU=; b=2U9l2/YXYIhUwVuZSJzZBBeir1a5yo5hG6qt239i1+nu0krF1HYYb277lVLDPCnYao cXPRp92lKEVsFwxP1yu8R7lQB/9arvd9qoupnb5SRkP1rCDXiEqn2grclv8vIpwGBa22 pA4xMLEGFYBGoYO/TZiNuwAI0fIqAR8jV/Sc/Sakx2uZwKRB4OhJNcLAWa/iU+MPMJ3S 6tsPcI3URwTojegeVhWwJXjZkFqDVhtRFbQySVI/np7Oec89tBtdu2+BnE/LoXJhH+x8 zn+q/GuMd2jyAWxsKe4gwTBthqkKOBDphMBPZ8t87NGug1utgz74itnjWBvxV4M0yQJe aYJw== X-Gm-Message-State: AFuF++kF/KImeDXDKd45dCmLSblII+Tmm9RAHy/TNEEkKOKBwr1uiW0n lpyhhhY/H9H3aN6gTmmkLX/yHHhhQpxkjPOm0KYwFe4CAJq/cpOtNai6+5/dlR/T1nYYD3og5fq nLeqjyMg= X-Gm-Gg: AYBFou1S5AhJo0UxJr3ra8oXHdohFrcr54HOuBdNGt4fMedB20SwxweHs6YXxb5RLLp uSGQeDNxYuOZgdXxhGqx4wBJcjaG+IHeF2KknBefftPGVo++gD9QisHMmfQq1MctUumvyEfSTew 27Uq78eXpZj46Lq3KdHS2Iu41yXKSj/j+Vh7VYNXoyaBeZXXHivHwRbVHQq8SA5h2eAk+i0E0x2 gT22ge7JICg+NFtGCBGjfgUM16U2woe1EXJMdf821LzVIy5wEaRzekZaXl1e8JKJeaZuT+hA4sL OYdlRPp3sxj6UFNwPYgN10FzJ6uFppkM9m6Juy3G4rdZG3wfuNG5LK7+nu/+UPTWxipUY4oA8TY 3IriBfWlDQn4EAJ6/MKObwpygmotbsMMXU5GEC8PU7pGyuAy4YMF037SUe/QbToKlvzabqOxwAB j5+KNJQvhq30cU1RnB6ZUH1VIsUfeclNLqrq9Y2hRyrRMUAoxof7e8PbYqwPG8kQxd6MtVEaNxw 8UHqUrE80fDaJWfomJoO0fNRfY6iDAUdDYqQkQURI219hqpsjy+0GmsFid5 X-Received: by 2002:ac8:6f1a:0:b0:532:c161:b968 with SMTP id d75a77b69052e-532eac32322mr21316581cf.1.1790121520905; Tue, 22 Sep 2026 16:58:40 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-60-52.washdc.fios.verizon.net. [173.79.60.52]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-532eb3c6715sm8323261cf.30.2026.09.22.16.58.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 16:58:40 -0700 (PDT) From: Gregory Price To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-team@meta.com, akpm@linux-foundation.org, liam@infradead.org, ljs@kernel.org, david@kernel.org, vbabka@kernel.org, jannh@google.com, rppt@kernel.org, surenb@google.com, mhocko@suse.com, shuah@kernel.org, "Gregory Price (Meta)" Subject: [PATCH 04/10] mm/madvise: use the PMD softleaf validity helper Date: Tue, 22 Sep 2026 19:58:24 -0400 Message-ID: <20260922235830.2350770-5-gourry@gourry.net> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922235830.2350770-1-gourry@gourry.net> References: <20260922235830.2350770-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" A non-present huge PMD must contain a software leaf type supported at PMD level. The open-coded check names the currently supported migration and device-private entries instead of expressing that invariant. Use pmd_is_valid_softleaf() so the validation follows the central definition of valid PMD softleaf entries. No functional change intended. Assisted-by: LLM Signed-off-by: Gregory Price (Meta) Reviewed-by: Lorenzo Stoakes (ARM) --- mm/madvise.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index c345fef23f15d..b31b877c2c130 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -427,8 +427,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned l= ong addr, =20 orig_pmd =3D *pmd; if (unlikely(!pmd_present(orig_pmd))) { - VM_WARN_ON_ONCE(!pmd_is_migration_entry(orig_pmd) && - !pmd_is_device_private_entry(orig_pmd)); + VM_WARN_ON_ONCE(!pmd_is_valid_softleaf(orig_pmd)); goto huge_unlock; } =20 --=20 2.53.0-Meta From nobody Thu Sep 24 14:26:50 2026 Received: from mail-qk2-f42.google.com (mail-qk2-f42.google.com [74.125.230.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C11B35275B0 for ; Tue, 22 Sep 2026 23:58:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.234 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121528; cv=none; b=JFQ9QZyqeqsZtB4QtFk9wwHtPYOVlW/n+y0SG55IJYJVkZ5c3byodmh0doGWJSov+lIfVWKETzKlR1Z9PdA6pMjLA87eyb8rzRK7HRZUnCG+BjupWvNpNj793jwYE7ephFN60BR66lxhoA0WK83zaVQq7lox1hURXHUxJfcPHDs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121528; c=relaxed/simple; bh=8CDetGO+wf6wYqZIC5YUwDIjZnqoRkQGXiEHkxcWWi0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cEPaOY/Ki18GBB9p84xWFxGOcsqqcxkEhi+eG9D1b9X2ciCMF6zfhHY1ny+beMFytsZBi3gc1UTkiNLyI6smswKfCWNKUI3hmc6sh6NBzxc9R+kTm2k/cP2f5xn+82aBP7CTiNiGJKbXl0eN99UK15bH5MnGhyYU+Z7wWZEnTJg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=MHkzANJS; arc=none smtp.client-ip=74.125.230.234 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="MHkzANJS" Received: by mail-qk2-f42.google.com with SMTP id d75a77b69052e-5329fc7b325so3130951cf.0 for ; Tue, 22 Sep 2026 16:58:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1790121523; x=1790726323; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=c5oF3nurCQ+QFDROgbfsMr6dUR727+FSjg6LodV8bBM=; b=MHkzANJS62tbMEU9gnOYp3Cqnankq2bCuDQIzaiuPLZuVSeUm/+4A+kMSj9MG+PUYF a251iGzm+Idj1Qaj6h/3U+sWu27m787YrrLoJYnhIFdTU5E9ojNG9HrrigSxEA7izkFG iVfEYAzOpO40e2FT12LxaBr9bKBBZH9YxMZH2D8zOarMdRpjjeNB4MHOamEFUymQHCWj Fuu8o339joB6Jj8DjeOqb/y0mfRq7CAMXXxSKXb9sJlylKUjC82NpmsilsyXPf1Y1ZCt QrKH5thuEZWfDoVXSBgYhTyq2BEBoYrBMH8cAudrELJCYybNK/+0tMEjpWb302L5JIJJ q5pQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790121523; x=1790726323; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=c5oF3nurCQ+QFDROgbfsMr6dUR727+FSjg6LodV8bBM=; b=gZw52Y85nzlVUjsoyUI8+EhJAWIB+wH5asfW3Z7D4raKpBfOI+4Ay8/ByZvFrMJw1D JXHWRJq4fcokP3aLXvJaPu8OnO77gNWO8Yi9paPlHXYvGSBQ7qVd13yfhDMFsFF6B6gb mql0O8EVEkrd51K2qKC/fo/ZNumkpUb0pO/TsPEYg3G2wSY0xKLVSoVKEx4TF/CSOW7Y nWMKRaq9FKO7Ehv8oBCCChC7N/ykA2CexQhGlR+ZseA7CwSZ4IkUb0icgbqOliMfJGK6 49um4ncK/1gx8DSh/OPPZgzFuldjPNviSzsmFGLwkS0SVavgRVWp+ch7CFYFHe3qJ4Yn Xr/w== X-Gm-Message-State: AFuF++leUBPKlnOfqZc9OtGVmeS8/4eA5vuKkd4imA++ckulNco65aFp cocwW4Vr3sTsZJrEP2H3ydFlMHDpJCGVX3AfYTAHd+BKhQsjSJ6dJFwxstq+CLj94qI= X-Gm-Gg: AYBFou301mUJ4P/dPmlvnicmO6N1jSlh8QMv7UwjF63sZBdBvTWNupRY4IYcadXmf5W lgi/JgLhVw5uy3265pPS5HMSZqs52/6iTv3ycEd67asV68ePbjQT9oB5Y9CoGVyrYQEIZKk3uTw Asm69TguzMBlZ5PNpVyOXGmwmTu9+SocGbsrTd4mtu6c6nsKks0MCiEuUkZ/tRE67NhddEOF/mE WRRAlzzr58WeffsSdCby+BJWsiJV2CwUtPCIJTxY5Fi73d/DYLzCrLp9EIr3/4FxSCQxRv+f4ZQ BaupGeeL+u3z18FO7jO/oruDbQ8k6AAyDcnA1HdIVrZaMibj3RArRyI6w37BeDNWzf6LI69BfVV 0OHV4Zg76wkSLAUbJRYORomdP6GuEld1J2BmXPBiDDYYDwe1+HAYXJFSEOBSETSNx5ufMSPNrXP XQvaoL1p0gBMI0Zw6M6/xQR5vAUab7F9c85B7Wi+s2A/jeBnFjmQ2RoyzCLFHtBXo2K4XiOr51o WWI/Sh0f7LUHSDBe89U7N0lhy8v0uRM9UR11SaRFMlfMUk0z2k+iaURANP5HTj8pchEHDOVt2Gi 6dGbqg== X-Received: by 2002:ac8:7fc1:0:b0:51c:555:7dea with SMTP id d75a77b69052e-532eac6984emr21152101cf.30.1790121523008; Tue, 22 Sep 2026 16:58:43 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-60-52.washdc.fios.verizon.net. [173.79.60.52]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-532eb3c6715sm8323261cf.30.2026.09.22.16.58.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 16:58:42 -0700 (PDT) From: Gregory Price To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-team@meta.com, akpm@linux-foundation.org, liam@infradead.org, ljs@kernel.org, david@kernel.org, vbabka@kernel.org, jannh@google.com, rppt@kernel.org, surenb@google.com, mhocko@suse.com, shuah@kernel.org, "Gregory Price (Meta)" Subject: [PATCH 05/10] mm/madvise: factor huge-PMD folio processing Date: Tue, 22 Sep 2026 19:58:25 -0400 Message-ID: <20260922235830.2350770-6-gourry@gourry.net> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922235830.2350770-1-gourry@gourry.net> References: <20260922235830.2350770-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The huge-PMD branch combines PMD validation and lock ownership with folio filtering, splitting, aging and isolation. Move the folio-specific work into a helper called with the PMD lock held. Return a locked and referenced folio only when the caller must drop the PMD lock and split it. No functional change intended. Assisted-by: LLM Signed-off-by: Gregory Price (Meta) --- mm/madvise.c | 70 ++++++++++++++++++++++++++++++++++------------------ 1 file changed, 46 insertions(+), 24 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index b31b877c2c130..6b518f7f73651 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -394,6 +394,49 @@ static bool madvise_lru_folio_is_filtered(struct folio= *folio, (pageout_anon_only && !folio_test_anon(folio)); } =20 +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static void madvise_cold_pmd(struct mmu_gather *tlb, struct vm_area_struct= *vma, + pmd_t *pmd, unsigned long addr, pmd_t orig_pmd) +{ + if (!pmd_young(orig_pmd)) + return; + + pmdp_invalidate(vma, addr, pmd); + orig_pmd =3D pmd_mkold(orig_pmd); + set_pmd_at(tlb->mm, addr, pmd, orig_pmd); + tlb_remove_pmd_tlb_entry(tlb, pmd, addr); +} + +/* Return a locked, referenced folio only when it must be split. */ +static struct folio * +madvise_lru_huge_pmd_locked(pmd_t *pmd, pmd_t orig_pmd, + unsigned long addr, unsigned long next, struct mm_walk *walk, + struct list_head *folio_list, bool pageout_anon_only) +{ + const struct madvise_walk_private *private =3D walk->private; + struct vm_area_struct *vma =3D walk->vma; + struct folio *folio; + + folio =3D vm_normal_folio_pmd(vma, addr, orig_pmd); + if (!folio || folio_is_zone_device(folio)) + return NULL; + if (madvise_lru_folio_is_filtered(folio, pageout_anon_only)) + return NULL; + + if (next - addr !=3D HPAGE_PMD_SIZE) { + if (!folio_trylock(folio)) + return NULL; + folio_get(folio); + return folio; + } + + if (!private->pageout) + madvise_cold_pmd(private->tlb, vma, pmd, addr, orig_pmd); + madvise_lru_folio(folio, private->pageout, folio_list); + return NULL; +} +#endif + static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) { @@ -431,22 +474,11 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned= long addr, goto huge_unlock; } =20 - folio =3D vm_normal_folio_pmd(vma, addr, orig_pmd); - if (!folio) - goto huge_unlock; - - if (folio_is_zone_device(folio)) - goto huge_unlock; - - if (madvise_lru_folio_is_filtered(folio, pageout_anon_only)) - goto huge_unlock; - - if (next - addr !=3D HPAGE_PMD_SIZE) { + folio =3D madvise_lru_huge_pmd_locked(pmd, orig_pmd, addr, next, + walk, &folio_list, pageout_anon_only); + if (folio) { int err; =20 - if (!folio_trylock(folio)) - goto huge_unlock; - folio_get(folio); spin_unlock(ptl); err =3D split_folio(folio); folio_unlock(folio); @@ -455,16 +487,6 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned = long addr, goto regular_folio; return 0; } - - if (!pageout && pmd_young(orig_pmd)) { - pmdp_invalidate(vma, addr, pmd); - orig_pmd =3D pmd_mkold(orig_pmd); - - set_pmd_at(mm, addr, pmd, orig_pmd); - tlb_remove_pmd_tlb_entry(tlb, pmd, addr); - } - - madvise_lru_folio(folio, pageout, &folio_list); huge_unlock: spin_unlock(ptl); if (pageout) --=20 2.53.0-Meta From nobody Thu Sep 24 14:26:50 2026 Received: from mail-qk2-f13.google.com (mail-qk2-f13.google.com [74.125.230.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 990C953A896 for ; Tue, 22 Sep 2026 23:58:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.205 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121529; cv=none; b=Kgr3PfSgEVQE5CeHcfV1uX0UI97k4RG6/Y9mkgkkP/h0CxQ+JfEFOsPJzSW7Zw6T44dTdqQ4xpFzLfAi//DKL3ZxxVLks46FNHx/B58vGVCTIJ/4tHGdQtAMUyH1IlKrKG2ag4arHrnPPLxBvU42aDV6BuEInbdcK56r+PSVb+o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121529; c=relaxed/simple; bh=D1jZ5yEljPoeJJ0ckwdQ1BT6N6s+ZZzFqlaHb94bagg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aCncKcJER+1ZbAh7KHwpvwzE0Qa1ueDvLDQhawlWvpBB0GI/R61Hhc6WJxAMJnQ64JMz7Xte4rLQL7rfvVl5+jcAWWL0fziDsjukNyl/CQWf6wB6Ru1a1cC/4VP7J/Zt5DvUdne+jyn8AZIOS0UUkjgnX6btpQAniOVJJ9B6gO4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=iryWxFMR; arc=none smtp.client-ip=74.125.230.205 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="iryWxFMR" Received: by mail-qk2-f13.google.com with SMTP id d75a77b69052e-52fb766bfd8so3948401cf.3 for ; Tue, 22 Sep 2026 16:58:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1790121525; x=1790726325; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=Obo7lHtS5J/HNUJlSapmyf6Vr5VJeymYNsQD5zlwwVs=; b=iryWxFMRMIDM1eYpZoWC4OmqlBlL98gpUES4nnPKD7lF05+2E6OAODPIOdnp6+8Baj /1Xzq9/Pu2JTL7jNkx+puo7upwImP0nEKlfeb8mRFWczyLlJfvzeTgeyiNWQUeG5MU6h pKY1wkS/mHhh2qjf5o2ZUlAoaw9h3lkjr2oCbY3A44WAF3KnY5VkGEpFscLacIi0zMF4 bGk/tycgZQuVY312Yl9MreEE1rOywZi7AEDxSfv2cfQIPeWLtKBe4VgL6oEWYpG7raEB zk5uqVcSyj9OZ+P7gro8VSPDwqjkIbKJBz4cNtQqmyqbUcHCHedwSYxN7YtL5sdIsewd K4Kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790121525; x=1790726325; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=Obo7lHtS5J/HNUJlSapmyf6Vr5VJeymYNsQD5zlwwVs=; b=sxW7RBzj1N/9915ky5HBkf6oPlkP4RuYK9M/LsLSXZZ9DoDDpUanS/5FuG6DR07lTE U9XSMsNSqbREV0tOgZYt76tqG6BUcif7QKAc6JHBEuRKqTEcNj1SQZcXjfA9LkKM18hz wkfQUKqtv+lNXGbaVKV403FV6r/kUq+4vmlz4C3mHRtDsPjmQILfb7b41weZ3zjgoTnQ 6dWXaPmwc7FotO/oU7JRMRH+YeU1+UfGAgLJjaOmnoNB37ItXFOB7UCLxxzO09ugCIpL Pu33ZTppsrZc5Ipo3qSVlsb+zGmgGHuQdMZ6gK8PbI0JEdClxk960Z4EE0281z9O8sLl JwyA== X-Gm-Message-State: AFuF++kUwGJdn3Dikvt7R5P2BbUAl8QEVMWakA4rQiUfPJ8yABYqcZ+s MYOO+ZIs9fPJ1o85UAINyGk4uOYpe4Itk52hC2Lyga1gbANoPMmQg7UFNPLUGCC1hsA= X-Gm-Gg: AYBFou1ixZNplAK4sRMzsGPkUTTuUzumDdIUT44LagdH3iFqNQGE2LDvkR6PWiTiu5L avjRx3rMqL/5Sj8XNyTDqQs+YyxAmjiv6/i6PsITusbfnvOUs2cY3Bc3VkEfnpCf13PjM9pnfbU cYsRYOxTf3orT4pTRNxc9/ZTVl2q5ngXEWPMo+s2VrtUbTnUe55DxCmPWWKlJSb6ZCiYdehHF/e 9ubg3OohGJpvQPh9HO3AIZZJO3U//fO/B4XGp7I0LApZOX2FRlK/HRPhshpYRbDVpBeRiC1Wu+X mdUmjX5JJUzXrM3H+TIQNQN/RNuXjHgMxjwU7t4cIZzmK6CLP7fZM6cC3bpF24VKFF6uHn97xzk r+RgXb6nXp/4MKd62bEUtmQDbN1FFr4QFKq55hMWuciaPkc4ZQzdq7DXgoGvXGUuqvmD83JqSEo 7y+HTaM3aXxCoOC5wRH+bbMxkl5xLbZ51R2ojhmuGyg21NaRDrxZFIM0ASIQMtLRCpBgZKpNbLn E/yr0BZGueV4CTx7u/gVZ5nOGKKkWuJiXkVkrfb8Xb+TbBWvDFMc6b+4hlgKV7mRV49zZo= X-Received: by 2002:a05:622a:2b09:b0:531:1e27:90fd with SMTP id d75a77b69052e-532eac081e7mr17011201cf.30.1790121525263; Tue, 22 Sep 2026 16:58:45 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-60-52.washdc.fios.verizon.net. [173.79.60.52]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-532eb3c6715sm8323261cf.30.2026.09.22.16.58.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 16:58:44 -0700 (PDT) From: Gregory Price To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-team@meta.com, akpm@linux-foundation.org, liam@infradead.org, ljs@kernel.org, david@kernel.org, vbabka@kernel.org, jannh@google.com, rppt@kernel.org, surenb@google.com, mhocko@suse.com, shuah@kernel.org, "Gregory Price (Meta)" Subject: [PATCH 06/10] mm/madvise: separate huge PMDs from the PTE walk Date: Tue, 22 Sep 2026 19:58:26 -0400 Message-ID: <20260922235830.2350770-7-gourry@gourry.net> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922235830.2350770-1-gourry@gourry.net> References: <20260922235830.2350770-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The PMD callback still owns huge-PMD locking, splitting and reclaim along with its PTE-table loop. This obscures the transition between the two paths. Move huge-PMD lock ownership into a dedicated helper. Return false after a requested PMD split succeeds so the callback continues at PTE level. Use the PMD boundary already supplied by walk_pmd_range(). No functional change intended. Assisted-by: LLM Signed-off-by: Gregory Price (Meta) --- mm/madvise.c | 85 +++++++++++++++++++++++++++++----------------------- 1 file changed, 48 insertions(+), 37 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 6b518f7f73651..83b27258c9673 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -435,6 +435,52 @@ madvise_lru_huge_pmd_locked(pmd_t *pmd, pmd_t orig_pmd, madvise_lru_folio(folio, private->pageout, folio_list); return NULL; } + +/* Return false when a requested split requires a PTE walk. */ +static bool madvise_lru_huge_pmd(pmd_t *pmd, unsigned long addr, + unsigned long next, struct mm_walk *walk, + bool pageout_anon_only) +{ + const struct madvise_walk_private *private =3D walk->private; + struct mmu_gather *tlb =3D private->tlb; + bool pageout =3D private->pageout; + LIST_HEAD(folio_list); + struct folio *folio =3D NULL; + spinlock_t *ptl; + pmd_t orig_pmd; + + tlb_change_page_size(tlb, HPAGE_PMD_SIZE); + ptl =3D pmd_trans_huge_lock(pmd, walk->vma); + if (!ptl) + return true; + + orig_pmd =3D *pmd; + if (unlikely(!pmd_present(orig_pmd))) { + VM_WARN_ON_ONCE(!pmd_is_valid_softleaf(orig_pmd)); + } else { + folio =3D madvise_lru_huge_pmd_locked(pmd, orig_pmd, addr, next, + walk, &folio_list, pageout_anon_only); + } + spin_unlock(ptl); + + if (folio) { + int err =3D split_folio(folio); + + folio_unlock(folio); + folio_put(folio); + return err !=3D 0; + } + if (pageout) + reclaim_pages(&folio_list); + return true; +} +#else +static bool madvise_lru_huge_pmd(pmd_t *pmd, unsigned long addr, + unsigned long next, struct mm_walk *walk, + bool pageout_anon_only) +{ + return false; +} #endif =20 static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr, @@ -458,44 +504,9 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned = long addr, pageout_anon_only =3D pageout && !vma_is_anonymous(vma) && !can_do_file_pageout(vma); =20 -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - if (pmd_trans_huge(*pmd)) { - pmd_t orig_pmd; - unsigned long next =3D pmd_addr_end(addr, end); - - tlb_change_page_size(tlb, HPAGE_PMD_SIZE); - ptl =3D pmd_trans_huge_lock(pmd, vma); - if (!ptl) - return 0; - - orig_pmd =3D *pmd; - if (unlikely(!pmd_present(orig_pmd))) { - VM_WARN_ON_ONCE(!pmd_is_valid_softleaf(orig_pmd)); - goto huge_unlock; - } - - folio =3D madvise_lru_huge_pmd_locked(pmd, orig_pmd, addr, next, - walk, &folio_list, pageout_anon_only); - if (folio) { - int err; - - spin_unlock(ptl); - err =3D split_folio(folio); - folio_unlock(folio); - folio_put(folio); - if (!err) - goto regular_folio; - return 0; - } -huge_unlock: - spin_unlock(ptl); - if (pageout) - reclaim_pages(&folio_list); + if (pmd_trans_huge(*pmd) && + madvise_lru_huge_pmd(pmd, addr, end, walk, pageout_anon_only)) return 0; - } - -regular_folio: -#endif tlb_change_page_size(tlb, PAGE_SIZE); restart: start_pte =3D pte =3D pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); --=20 2.53.0-Meta From nobody Thu Sep 24 14:26:50 2026 Received: from mail-qk2-f43.google.com (mail-qk2-f43.google.com [74.125.230.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ED96453B32F for ; Tue, 22 Sep 2026 23:58:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.235 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121536; cv=none; b=pNzIlxeEyDOEE4oHugjv2FjLBLawxAmqKuWeI2xSlUb9ql+Cu0ryUb3S2IPo1YoCQg2rpO5fhmv9VRITGxMwWljSGk1hyAK+wG6hDYXwxyfV7sx9tDE4hkURcbkcvPbaTQ6K5vwXKovjyjVXXWLNqTdVAbxBfA//L3Po/OSrDnw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121536; c=relaxed/simple; bh=pC7CcSkiGbHApKl+B9rATKci8vcVThz5BEzv7IQiVAA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gv32xM6JIqc99u2tm2SwBFpJf9IVTNNOFglc6qC7kEsvpp0E2zX0A8O3/KQN9cYLpjA5gQo8NBmVQaBsCP1ZW68da1Je8mxF3yhR3fLeKeiI2J34V9V4a8oENDyCGbzMvngGO+vGjBQnMRwd7xxvIqdzbRPzSX0mZTjXQshf6kw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=XoJmyt1z; arc=none smtp.client-ip=74.125.230.235 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="XoJmyt1z" Received: by mail-qk2-f43.google.com with SMTP id d75a77b69052e-52fb766bfd6so4082111cf.1 for ; Tue, 22 Sep 2026 16:58:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1790121527; x=1790726327; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=LeVENJt4LIsrgXUQ5UaykSbIE82DoLdeqUsQ6JLwyy0=; b=XoJmyt1zSoWwMe3FHt67O76tRyM3njd+eHjRaTWzLXpCQQ6bSHsp3ns/5kqnHChFQK ctVZdJgE3YxWl/d1luRFIGr7SFvB2xhPAQK9Iikclz/S3OhroWINIqhNkjb9jxsB3kwI 8Erhi39x3Gu9L/cgeQOWlvNyDY8t2jm0Nf238iThmVxREPrU8RAne2Fmf5V8oTR7iRQ0 zTK+yWGufbxTUz06KAgffqvFOsflOzOXoH6LjZ3hU3U1diN6632VOx97GF8B8fGAv2Cm 9pgsEBZK/MQ1ZXdG0VJH2FUsTMLPyS00xyoG352uHr9YQFpdwJyhTautA2n0Zvv4becL a8iQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790121527; x=1790726327; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=LeVENJt4LIsrgXUQ5UaykSbIE82DoLdeqUsQ6JLwyy0=; b=lXqNw41Jar8Ki6Dv4LV9pF1qkMD408kArWoj4L+necxYVq5/osriE0hwc2casOuU4P YC6AZWznsXJj7UMU0zLI+iLD2J/7vLPc7GBdac2BbQ23cvTG+dBJb2r/tD8JqGq1F9nV l5CIqPqYZ2GB4u2Tu3mz6SBYm/KfSd2ympM9murFI6WS0c3N5WcH65UER/VIZfwqGxAA JyMN2UsSiQENdrFmpKlYDS4dRrrFjrFmeqCb4QOdeclHAorg9v/G9hl0O+RCcpfOt206 tFsDwgNRwRlV9fiF8M12mIAMJm1FiGC6w4BSg+Gw/oHhfoa9+SKkTZhxqiXs0+9fnM+j OSKQ== X-Gm-Message-State: AFuF++ns6Thx67PFfEs5q0kRhIsGpWV4QGUCzAdi1ZLAJwa006217iH8 TMaHKpHjRY7czSIshIAtYVowb5nMOlVtDQlu9+bSSeV27+h9/rxkaXCqCqj1AZAHRJQ= X-Gm-Gg: AYBFou1n0ngYvo3b5blD/iY8V1taoy1bNjdnL5FzfIE40wui0AdcM/qb4K4Ej3UQ8pz Lez18bUNp+1XHaFWaz/6zEvvikGvnm4FKn0+KxfQHtCsV749Cy3nT5kvxjkT3NuxwyOlw9qGalx 34INs9e84o/zdR1eRey9EYAhl6yFYuLVdtWu7mfXGLde9dneaftazgU5WPOPDtoo94xue2S5btf Co5Atl9n702a2Hjq8ECV4RdSivEOzEyMm6W/GwwzC22kpcS9xbuZz4ozvcjgZF/QDVgCvqcoK2F rOqC020Rs2dIYqzYsFw0wUUBntOaYlJ+7GqsdMBRbMquAmsTQipk6sTIlU8xrysIoV6xLbAzJl4 aNCzjtUdR7t87zKEcpxxrxAIcDEeoMIvN5vHWrNE123TYa0PJS7SOCyqOU69om4LVkJH+3YgCEz tJSI0qyZrmrA4SOCbylCvbSfIQvEOgTvLmtp62zyL6jTjhpk0wEIIadw0yVx7KB7vSiL69KGQR7 irUPlIXzU+LHMSaBxpzx+GE4Z4jTbq7G7+U+4wTO7n2xeYCkB2x1+AI3D5SwuPiNsqv2Ws= X-Received: by 2002:ac8:5986:0:b0:532:9adc:639d with SMTP id d75a77b69052e-532ead489d2mr18461111cf.65.1790121527246; Tue, 22 Sep 2026 16:58:47 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-60-52.washdc.fios.verizon.net. [173.79.60.52]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-532eb3c6715sm8323261cf.30.2026.09.22.16.58.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 16:58:46 -0700 (PDT) From: Gregory Price To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-team@meta.com, akpm@linux-foundation.org, liam@infradead.org, ljs@kernel.org, david@kernel.org, vbabka@kernel.org, jannh@google.com, rppt@kernel.org, surenb@google.com, mhocko@suse.com, shuah@kernel.org, "Gregory Price (Meta)" Subject: [PATCH 07/10] mm/madvise: separate PTE-batch folio processing Date: Tue, 22 Sep 2026 19:58:27 -0400 Message-ID: <20260922235830.2350770-8-gourry@gourry.net> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922235830.2350770-1-gourry@gourry.net> References: <20260922235830.2350770-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The PTE loop combines page-table iteration with validation and processing of each folio batch. This hides the batching rule for large folios and the ownership transferred when a partial folio must be split. Move one batch into a PTL-locked helper. A split candidate is returned locked and referenced so the caller can release the PTE mapping before calling split_folio(). No functional change intended. Assisted-by: LLM Signed-off-by: Gregory Price (Meta) --- mm/madvise.c | 132 +++++++++++++++++++++++++-------------------------- 1 file changed, 65 insertions(+), 67 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 83b27258c9673..350b854ccc197 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -394,6 +394,54 @@ static bool madvise_lru_folio_is_filtered(struct folio= *folio, (pageout_anon_only && !folio_test_anon(folio)); } =20 +/* Return a split candidate locked and referenced for use after the PTL dr= op. */ +static struct folio * +madvise_lru_pte_batch_locked(pte_t *pte, unsigned long addr, + unsigned long end, struct mm_walk *walk, + struct list_head *folio_list, bool pageout_anon_only, int *nr) +{ + const struct madvise_walk_private *private =3D walk->private; + struct vm_area_struct *vma =3D walk->vma; + struct folio *folio; + pte_t ptent; + + *nr =3D 1; + ptent =3D ptep_get(pte); + if (pte_none(ptent) || !pte_present(ptent)) + return NULL; + + folio =3D vm_normal_folio(vma, addr, ptent); + if (!folio || folio_is_zone_device(folio)) + return NULL; + + /* Split PTE-mapped large folios before advising only part of them. */ + if (folio_test_large(folio)) { + *nr =3D madvise_folio_pte_batch(addr, end, folio, pte, &ptent); + if (*nr < folio_nr_pages(folio)) { + if (madvise_lru_folio_is_filtered(folio, pageout_anon_only)) + return NULL; + if (!folio_trylock(folio)) + return NULL; + folio_get(folio); + return folio; + } + } + + if (!folio_test_lru(folio) || + folio_mapcount(folio) !=3D folio_nr_pages(folio)) + return NULL; + if (pageout_anon_only && !folio_test_anon(folio)) + return NULL; + + if (!private->pageout && pte_young(ptent)) { + clear_young_dirty_ptes(vma, addr, pte, *nr, CYDP_CLEAR_YOUNG); + tlb_remove_tlb_entries(private->tlb, pte, *nr, addr); + } + + madvise_lru_folio(folio, private->pageout, folio_list); + return NULL; +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE static void madvise_cold_pmd(struct mmu_gather *tlb, struct vm_area_struct= *vma, pmd_t *pmd, unsigned long addr, pmd_t orig_pmd) @@ -491,7 +539,7 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned l= ong addr, bool pageout =3D private->pageout; struct mm_struct *mm =3D tlb->mm; struct vm_area_struct *vma =3D walk->vma; - pte_t *start_pte, *pte, ptent; + pte_t *start_pte, *pte; spinlock_t *ptl; struct folio *folio =3D NULL; LIST_HEAD(folio_list); @@ -515,9 +563,6 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned l= ong addr, flush_tlb_batched_pending(mm); lazy_mmu_mode_enable(); for (; addr < end; pte +=3D nr, addr +=3D nr * PAGE_SIZE) { - nr =3D 1; - ptent =3D ptep_get(pte); - if (++batch_count =3D=3D SWAP_CLUSTER_MAX) { batch_count =3D 0; if (need_resched()) { @@ -528,71 +573,24 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned= long addr, } } =20 - if (pte_none(ptent)) - continue; - - if (!pte_present(ptent)) - continue; - - folio =3D vm_normal_folio(vma, addr, ptent); - if (!folio || folio_is_zone_device(folio)) - continue; - - /* - * If we encounter a large folio, only split it if it is not - * fully mapped within the range we are operating on. Otherwise - * leave it as is so that it can be swapped out whole. If we - * fail to split a folio, leave it in place and advance to the - * next pte in the range. - */ - if (folio_test_large(folio)) { - nr =3D madvise_folio_pte_batch(addr, end, folio, pte, &ptent); - if (nr < folio_nr_pages(folio)) { - int err; - - if (madvise_lru_folio_is_filtered(folio, pageout_anon_only)) - continue; - if (!folio_trylock(folio)) - continue; - folio_get(folio); - lazy_mmu_mode_disable(); - pte_unmap_unlock(start_pte, ptl); - start_pte =3D NULL; - err =3D split_folio(folio); - folio_unlock(folio); - folio_put(folio); - start_pte =3D pte =3D - pte_offset_map_lock(mm, pmd, addr, &ptl); - if (!start_pte) - break; - flush_tlb_batched_pending(mm); - lazy_mmu_mode_enable(); - if (!err) - nr =3D 0; - continue; - } - } - - /* - * Do not interfere with other mappings of this folio and - * non-LRU folio. If we have a large folio at this point, we - * know it is fully mapped so if its mapcount is the same as its - * number of pages, it must be exclusive. - */ - if (!folio_test_lru(folio) || - folio_mapcount(folio) !=3D folio_nr_pages(folio)) - continue; - - if (pageout_anon_only && !folio_test_anon(folio)) + folio =3D madvise_lru_pte_batch_locked(pte, addr, end, walk, + &folio_list, pageout_anon_only, &nr); + if (!folio) continue; =20 - if (!pageout && pte_young(ptent)) { - clear_young_dirty_ptes(vma, addr, pte, nr, - CYDP_CLEAR_YOUNG); - tlb_remove_tlb_entries(tlb, pte, nr, addr); - } - - madvise_lru_folio(folio, pageout, &folio_list); + lazy_mmu_mode_disable(); + pte_unmap_unlock(start_pte, ptl); + start_pte =3D NULL; + if (!split_folio(folio)) + nr =3D 0; + folio_unlock(folio); + folio_put(folio); + start_pte =3D pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!start_pte) + break; + pte =3D start_pte; + flush_tlb_batched_pending(mm); + lazy_mmu_mode_enable(); } =20 out: --=20 2.53.0-Meta From nobody Thu Sep 24 14:26:50 2026 Received: from mail-qk2-f42.google.com (mail-qk2-f42.google.com [74.125.230.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5B88A53A3B0 for ; Tue, 22 Sep 2026 23:58:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.234 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121535; cv=none; b=aFcRrXvLsC7uBRqbtwZbipkm/+/STW6IIJ22VvB2TMCry1RWL/uIToB7pCrxKXnAWBbMpnbTv4pNXGD1P4qfrVxNXFq940v65+umpq4SaZt8+ToIMSAkun27oElTONa/paxtactZonxdidGNALcEX3K06QFHCB+2NTdIZlLVi5E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121535; c=relaxed/simple; bh=mdmK36Yhyq2qbhMWTlH9S7tlgVHXySzUY+72VZaX/r4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=i81iAKOpBZzX3SSsd6bg5kU1ZI1rzry6BKFwmnGTo8toPkU1Hvzd6IslXhV7FPYcYnDDINxzwU/COMCjQfojr8b8+F2GYFTYDViqtF3Ku0eIR7Z8BpMrod3eof2fGILjQ6FF7CLxIVzHvxN33Gg7Y71vBGXqhqIZpX4hIXImt2c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=hRBeDa3s; arc=none smtp.client-ip=74.125.230.234 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="hRBeDa3s" Received: by mail-qk2-f42.google.com with SMTP id d75a77b69052e-5329fc7b325so3131281cf.0 for ; Tue, 22 Sep 2026 16:58:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1790121529; x=1790726329; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=UWLbjJuQ41hy5QZZXmUfMBTlvhCx61z13QEH9bWMQto=; b=hRBeDa3spPlForJHccFY2fZPouvuZwfEN8x/pEG8YONm7/r8x2KiLMPDn2zicof1Rm PymObrlPR/jkVQbvns/da6s0cNuYn0HuuVodBkslc6nfRxOmVpE3HOmmShEE4eytA/kC HmH9gUZnESZy0SqSWaOBIqkzvl0ejYPzDNs5XMZ4LNVeLfHyYCUx9iNPDDyOhquBf0Wd retmUcR+sM5O6FruUUnLybZhuTOfnvykGv+Op0/a1bX8mUgxlreXJK2eYDbg61mWwMFx HsrSDtGB9HPVXntZDTXv24VK+PhfNgc/bo2qSnMYqTW2f4IOJ1/k06n3fB4vd6UDnoom /j/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790121529; x=1790726329; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=UWLbjJuQ41hy5QZZXmUfMBTlvhCx61z13QEH9bWMQto=; b=dDXP72GxgpMXvCzwAwJV2AWTYRsc5n+xCiYtLlSx3AapoyFclsWCHXhsaZ4rCD0spe 5RkAM7Ku7Wk+WKNM3orK7VFUUyv/jZBn3tMZEBufJVEBcOWHQx8nVy3R+oVVb2sauZhU x1Mb0p1qREqPsn8L6wr8EdivW5U7+z1O1VltRWVWE/ZkT50VDR4WyiUaq4jjPliizjbw tIbOWHm2g9xLbFUtZ+TIOPSGFeNl6OrpodYwVEToqsHXxHqmPMYiwOweg5NmMS5uFjwa 7Cwp/EyGzYxUPnGWMGK5ObBFMjOFBPz35mpmWUf1ncRIlnFSYiUETHroBKePg8t/01rj MGQw== X-Gm-Message-State: AFuF++nY3OyPUwtUOzEA9wicicwqWDJsf9+WV7D3FleIHbYs/bJphvG4 BLsygKj5wUM5c1wDtqN2yQ7n0VHIi8VjGkEMRMWprHwCfxES3uwWkB15wM2tlcQ/jLI= X-Gm-Gg: AYBFou05yY/7/RwFQ5vOJDwqqV+W+7xY3X22W87U274YIuFgl0tLt/r2XUd3xHwzQMj CQcPtUTVTVZwXWi9dhw0sIv2Qrb16Pe8mKTmz427hhZAaXaTiOA7bmnHOc0XJ9PvwnPhF803Px7 o5HTvHJgQBxzsEHxL8FGPZgwfgeg0nfalHGvEXXSe6tzpjhfCzV7Qr+f0seN600/kj3HTXppG8f lnpd3CGghY0bAg2Ni/HvaEt5bMmepZt/3x0ClvhiT8t8zZ4RPRDrCqwjwv1B9udq54Sk/KZF3Wi 5q9/+0iII4+ZlVQMvQ7BJsxn7Ufsl/O2F1CdJyhr7KN9dvP7AgtinptRZ0/9t0/F6odqWl4pz+t tT/nHWlCE3o5E6YQRtYJKvmc5ti+YXzXZuTZwd2cuV4GnE9W3dg9hjmSiZrDfrFLOUIdjPv5w7R GcJ8ryb2dtlY2TzJ7HvcvGVyg/SFA0fCXiqWFXW9lSpBTZtcA8TE2Va4YIyZZVMZhXw/tzekcKI sfG0Fp6/LO+sllSsoHU8ysR78G3EYFe+DDPnvE0PtDNKXgOm0Y0xjZ1obmarEaL7l9tv6E= X-Received: by 2002:a05:622a:1345:b0:532:d202:328e with SMTP id d75a77b69052e-532ead6aaffmr19546621cf.55.1790121529209; Tue, 22 Sep 2026 16:58:49 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-60-52.washdc.fios.verizon.net. [173.79.60.52]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-532eb3c6715sm8323261cf.30.2026.09.22.16.58.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 16:58:48 -0700 (PDT) From: Gregory Price To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-team@meta.com, akpm@linux-foundation.org, liam@infradead.org, ljs@kernel.org, david@kernel.org, vbabka@kernel.org, jannh@google.com, rppt@kernel.org, surenb@google.com, mhocko@suse.com, shuah@kernel.org, "Gregory Price (Meta)" Subject: [PATCH 08/10] mm/madvise: separate the PTL-held PTE scan Date: Tue, 22 Sep 2026 19:58:28 -0400 Message-ID: <20260922235830.2350770-9-gourry@gourry.net> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922235830.2350770-1-gourry@gourry.net> References: <20260922235830.2350770-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The PTE loop mixes its PTL-protected scan with the surrounding lock drops for rescheduling and large-folio splitting. Move the scan into a helper that runs entirely under the PTE lock. It returns a locked and referenced split candidate, or stops at the existing scheduling boundary so the caller can drop the lock before yielding. No functional change intended. Assisted-by: LLM Signed-off-by: Gregory Price (Meta) --- mm/madvise.c | 66 +++++++++++++++++++++++++++++++--------------------- 1 file changed, 39 insertions(+), 27 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 350b854ccc197..e3c3acfcd9b66 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -531,6 +531,28 @@ static bool madvise_lru_huge_pmd(pmd_t *pmd, unsigned = long addr, } #endif =20 +static struct folio * +madvise_lru_pte_range_locked(pte_t *pte, unsigned long *addr, + unsigned long end, struct mm_walk *walk, + struct list_head *folio_list, bool pageout_anon_only, int *nr, + unsigned int *batch_count) +{ + struct folio *folio; + + for (; *addr < end; pte +=3D *nr, *addr +=3D *nr * PAGE_SIZE) { + if (++(*batch_count) =3D=3D SWAP_CLUSTER_MAX) { + *batch_count =3D 0; + if (need_resched()) + return NULL; + } + folio =3D madvise_lru_pte_batch_locked(pte, *addr, end, walk, + folio_list, pageout_anon_only, nr); + if (folio) + return folio; + } + return NULL; +} + static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) { @@ -562,36 +584,26 @@ static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned= long addr, goto out; flush_tlb_batched_pending(mm); lazy_mmu_mode_enable(); - for (; addr < end; pte +=3D nr, addr +=3D nr * PAGE_SIZE) { - if (++batch_count =3D=3D SWAP_CLUSTER_MAX) { - batch_count =3D 0; - if (need_resched()) { - lazy_mmu_mode_disable(); - pte_unmap_unlock(start_pte, ptl); - cond_resched(); - goto restart; - } - } - - folio =3D madvise_lru_pte_batch_locked(pte, addr, end, walk, - &folio_list, pageout_anon_only, &nr); - if (!folio) - continue; - + folio =3D madvise_lru_pte_range_locked(pte, &addr, end, walk, + &folio_list, pageout_anon_only, &nr, &batch_count); + if (!folio && addr < end) { lazy_mmu_mode_disable(); pte_unmap_unlock(start_pte, ptl); - start_pte =3D NULL; - if (!split_folio(folio)) - nr =3D 0; - folio_unlock(folio); - folio_put(folio); - start_pte =3D pte_offset_map_lock(mm, pmd, addr, &ptl); - if (!start_pte) - break; - pte =3D start_pte; - flush_tlb_batched_pending(mm); - lazy_mmu_mode_enable(); + cond_resched(); + goto restart; } + if (!folio) + goto out; + + lazy_mmu_mode_disable(); + pte_unmap_unlock(start_pte, ptl); + start_pte =3D NULL; + if (!split_folio(folio)) + nr =3D 0; + folio_unlock(folio); + folio_put(folio); + addr +=3D nr * PAGE_SIZE; + goto restart; =20 out: if (start_pte) { --=20 2.53.0-Meta From nobody Thu Sep 24 14:26:50 2026 Received: from mail-qk2-f13.google.com (mail-qk2-f13.google.com [74.125.230.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6990B53B345 for ; Tue, 22 Sep 2026 23:58:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.205 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121536; cv=none; b=GFWqdbM7w58bAy2RJcHmVKsojAt4HnVHwh7d3/CKKMevRv6VnVN0DdBnQ6GuczVBn4BQVppMNBFeiQx57grgkve0HbjxwiEKgCmczz7vevgDKacaoE5NwVYCMDLDtfjbIdj+za0ZCk8yjLM5SUNqmHStVSu0PBIyBk4ASlLp8rA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121536; c=relaxed/simple; bh=4tfxaowEwyLcwym1ubVvN+8S1jtOYuzn6GPPLRK9Vaw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=idDHIOo8g/Zflkoqy4G24E9xRWSqGIzqnEzlx43auKKhFKRPukXXyl3Pm+Jpio+A1sdw0l9xb0TDcX4MGzgKTYLuku8jBnx0mpq6hz+DmqFhKib7BEoabl7wFH7H0jERlyj6VJW3G2617Uv6noR8fwb+zcRzdg62ciLhnJ1tNys= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=PLnFCRyG; arc=none smtp.client-ip=74.125.230.205 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="PLnFCRyG" Received: by mail-qk2-f13.google.com with SMTP id d75a77b69052e-530e2f50d01so4354461cf.0 for ; Tue, 22 Sep 2026 16:58:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1790121530; x=1790726330; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=l3xhM/OIBt/SuFbuoCQCsVkZeh3BpJl7fkoqP+MrWAM=; b=PLnFCRyG9XwSahqQaJvq0Fu5ThIzedEFNpa+UfG2Fw7NAJH7p9BJSOpShVDtFNPcr2 BloihYSiRgceklrQsI2tRTTre+lky/GxWZVqCoIAaelx/YllZkk4o2NCerlzqW/WuERA uLcWdNMqwe5iGZMb7dUOrgn6CTcBSZVMeAwkImYOTiD9linvBba2Xze1C18omyWYo5jb m5ueaGqyI3gdE++U/lHQqf4KJcQCFmYHa8WS7Jl3g+FMTvjZvOR52lPd7LglYdmnF6gI maiHjTOMzyZSW8YOlG9nHr2BKYahUzEh1VaE6Y2RTF4F1Nd9c1aebWcjQXlgGv00b+4U s4nA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790121530; x=1790726330; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=l3xhM/OIBt/SuFbuoCQCsVkZeh3BpJl7fkoqP+MrWAM=; b=kLAdXhRH5ilQhN3JWKWw9Dpy2mXOQYg3vJ9WjMKxTCIhCrkE6dA11F7capkJEDVtHa L/xeEkx7pz+XxNzuseuW9c70IcPhAoOotvYxnGMjABqYyHuJjtkW9XDngqGkdTGhq3jM 7/NlOFBG61y8exC4SH+wPBP3BWebxVsg0GDfC+Bee5oZUUiqZPoG80mY8bLe+RHknajy sauy7nbwSaCNtRr3g96M+qRitk3JACsVHIyULYkEBPNsrI9XFTB07LRofYdUR5lVJXS2 cnA7MN1eS0IO1ml5DDEdholYD2QDna2wRmj7vQDL/2tqPcPtlwEqm8FtmFgBY7K05a+g NbIg== X-Gm-Message-State: AFuF++mCnqHX/Woy+vYwMlLyNX7++ySQ67W35oKfgohnc9govCxu0fkJ JLDzACx5SdULETYJvF5CBIuBm6IXNNXpf/gB1ak0/gva46mF07/bGcu1uaeF9NVaiTg= X-Gm-Gg: AYBFou29gciLQhMI2JOacMj1TP+ZWUHBDP3jmf7/TXhczOZopZ294YcQzOc6Uz/o7z8 jSDPs9p+HMpWxMrgJ9HiTq7wAlaex4C7hsrNpi17nskuDg6UFoSPUKnMDbS/mf0IBFsWQhNuBmA Wt07suW4fOhUPg6npHY9o2vaFFGSnEvMNDEw9N0Lcs+FYRtcoJqXKJk8wjCHSxrSYkQGfuypYKQ 8fV/ZkD7V1iKUkqSg27n4U91RvH1FokEgF58rd1vTjg8dYIoWwmDkwoD7whXKIAvJ6+WLw3v4wk jraHqhjdXc7FjCvf4VESG1cX//1e0r/hYrsE5ZRn347BjqxfbqIWLTJW0EZ+YAQi+K9+lTn6ZmC lKgabrzRRtcW/t1jopyWSBUyIsaoHbJsH/HL0t162G8jnbOChgIkOqI6KrI+gCAxVl03i98sAfZ BcufoBm7K9ysz5IxfjXsF65AkB3F7eeProrbrzYQQdwYExBfB8xi4uxCDcok3zmmCL1sqaaN0nr Iq5w0yKZogWjb3ubQVqrfZGWA5jsku795dGu+uzurYnUiiTye8CD02HSBziEe8LGcXLTfI= X-Received: by 2002:a05:622a:411a:b0:52f:f73c:3269 with SMTP id d75a77b69052e-532ead1e0c5mr18771951cf.36.1790121530629; Tue, 22 Sep 2026 16:58:50 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-60-52.washdc.fios.verizon.net. [173.79.60.52]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-532eb3c6715sm8323261cf.30.2026.09.22.16.58.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 16:58:50 -0700 (PDT) From: Gregory Price To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-team@meta.com, akpm@linux-foundation.org, liam@infradead.org, ljs@kernel.org, david@kernel.org, vbabka@kernel.org, jannh@google.com, rppt@kernel.org, surenb@google.com, mhocko@suse.com, shuah@kernel.org, "Gregory Price (Meta)" Subject: [PATCH 09/10] mm/madvise: make cold and pageout PTE lock ownership explicit Date: Tue, 22 Sep 2026 19:58:29 -0400 Message-ID: <20260922235830.2350770-10-gourry@gourry.net> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922235830.2350770-1-gourry@gourry.net> References: <20260922235830.2350770-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The PMD callback still maps and unmaps the PTE table, drives split retries and handles rescheduling after dispatching huge PMDs. This leaves PTE lock ownership mixed with the page-walk callback. Move that lifecycle into madvise_lru_pte_range(). Its outer loop drops the lock before splitting or yielding and resumes at the current address, leaving the PMD callback to select the huge-PMD or PTE path. No functional change intended. Assisted-by: LLM Signed-off-by: Gregory Price (Meta) --- mm/madvise.c | 94 ++++++++++++++++++++++++++-------------------------- 1 file changed, 47 insertions(+), 47 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index e3c3acfcd9b66..88a4a03dee04c 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -553,66 +553,66 @@ madvise_lru_pte_range_locked(pte_t *pte, unsigned lon= g *addr, return NULL; } =20 -static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr, - unsigned long end, struct mm_walk *walk) +static void madvise_lru_pte_range(pmd_t *pmd, unsigned long addr, + unsigned long end, struct mm_walk *walk, + bool pageout_anon_only) { - struct madvise_walk_private *private =3D walk->private; - struct mmu_gather *tlb =3D private->tlb; - bool pageout =3D private->pageout; - struct mm_struct *mm =3D tlb->mm; - struct vm_area_struct *vma =3D walk->vma; + const struct madvise_walk_private *private =3D walk->private; + struct mm_struct *mm =3D private->tlb->mm; + LIST_HEAD(folio_list); pte_t *start_pte, *pte; spinlock_t *ptl; - struct folio *folio =3D NULL; - LIST_HEAD(folio_list); + struct folio *folio; unsigned int batch_count =3D 0; - bool pageout_anon_only; int nr; =20 - if (fatal_signal_pending(current)) - return -EINTR; - pageout_anon_only =3D pageout && !vma_is_anonymous(vma) && - !can_do_file_pageout(vma); + tlb_change_page_size(private->tlb, PAGE_SIZE); + while (addr < end) { + start_pte =3D pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!start_pte) + break; + pte =3D start_pte; + flush_tlb_batched_pending(mm); + lazy_mmu_mode_enable(); + folio =3D madvise_lru_pte_range_locked(pte, &addr, end, walk, + &folio_list, pageout_anon_only, &nr, &batch_count); =20 - if (pmd_trans_huge(*pmd) && - madvise_lru_huge_pmd(pmd, addr, end, walk, pageout_anon_only)) - return 0; - tlb_change_page_size(tlb, PAGE_SIZE); -restart: - start_pte =3D pte =3D pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); - if (!start_pte) - goto out; - flush_tlb_batched_pending(mm); - lazy_mmu_mode_enable(); - folio =3D madvise_lru_pte_range_locked(pte, &addr, end, walk, - &folio_list, pageout_anon_only, &nr, &batch_count); - if (!folio && addr < end) { lazy_mmu_mode_disable(); pte_unmap_unlock(start_pte, ptl); - cond_resched(); - goto restart; - } - if (!folio) - goto out; =20 - lazy_mmu_mode_disable(); - pte_unmap_unlock(start_pte, ptl); - start_pte =3D NULL; - if (!split_folio(folio)) - nr =3D 0; - folio_unlock(folio); - folio_put(folio); - addr +=3D nr * PAGE_SIZE; - goto restart; - -out: - if (start_pte) { - lazy_mmu_mode_disable(); - pte_unmap_unlock(start_pte, ptl); + if (!folio && addr < end) { + cond_resched(); + continue; + } + if (folio) { + if (!split_folio(folio)) + nr =3D 0; + folio_unlock(folio); + folio_put(folio); + addr +=3D nr * PAGE_SIZE; + } } - if (pageout) + if (private->pageout) reclaim_pages(&folio_list); cond_resched(); +} + +static int madvise_lru_pmd_entry(pmd_t *pmd, unsigned long addr, + unsigned long next, struct mm_walk *walk) +{ + const struct madvise_walk_private *private =3D walk->private; + struct vm_area_struct *vma =3D walk->vma; + bool pageout_anon_only; + + if (fatal_signal_pending(current)) + return -EINTR; + pageout_anon_only =3D private->pageout && !vma_is_anonymous(vma) && + !can_do_file_pageout(vma); + + if (pmd_trans_huge(*pmd) && + madvise_lru_huge_pmd(pmd, addr, next, walk, pageout_anon_only)) + return 0; + madvise_lru_pte_range(pmd, addr, next, walk, pageout_anon_only); =20 return 0; } --=20 2.53.0-Meta From nobody Thu Sep 24 14:26:50 2026 Received: from mail-qk2-f12.google.com (mail-qk2-f12.google.com [74.125.230.204]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3C061535FC3 for ; Tue, 22 Sep 2026 23:58:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=74.125.230.204 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121537; cv=none; b=P+Oj5HR6MgNh06Eyyoq9yYBKJZjSZKMso+q/YuaDQGZKMJ/9v2TVXi5nCabw1BFn1DnhmgGeeiEB8Ac3ShvlqXIKOGFWCD1qFiqCGB9xgkdd6APhO/upd4tyMif2dj61JzZ0uVh1C63yu/yu55bRiLk9cY8zpNKEjBB0juL7uWU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1790121537; c=relaxed/simple; bh=CXRsgxYB/5yutOnrrQ4y8UUCmA9/hvuTy++ixQDxir4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=aSxhv3X/PgRoJq3d/7WmvsaLR/7mzpaS5bjNmb9pLawxtouqWY6t6leKPES5mlV/qFhUWuLVPVooBblJQhLNgyqsaHXSfw6A0DicYzrTcy5mfV4u3v3U30em+3Hi26Fy9CLoxbzXy2Hlk31XjwEWFh2ZrQY+eE9lxqj5p15GGxc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net; spf=pass smtp.mailfrom=gourry.net; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b=j1dHDMeR; arc=none smtp.client-ip=74.125.230.204 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=gourry.net Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gourry.net Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gourry.net header.i=@gourry.net header.b="j1dHDMeR" Received: by mail-qk2-f12.google.com with SMTP id d75a77b69052e-530d0d5ae11so4682801cf.0 for ; Tue, 22 Sep 2026 16:58:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gourry.net; s=google; t=1790121532; x=1790726332; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ndNxVkCk5WMhyvlnx4WQtybJ1XuSLrgG5RrCZvpyS/4=; b=j1dHDMeRzTL7SoTAwFc6Z9lbnP0//XP+NF4ttqyc7E5YzHOMhWoid90cUvnRdtneqY HfrqDs4aYGdO6fZevuovqeLK1IuYLvocXBYZaX4k0MmgOhPLFKMbupb2j6UKrAUGdVo3 L1jf2rWH0DjWWNUWWV2EJyp5bZrVqo1OZWV7obFgGTcC4how8zP/5t0D7Nz1AvDsa1cZ qeWDV++W+h6WBGaHSb0C7OIjQByHEXjBHyljQErpR8/VYAxRGw1b51RqdF9mfoZsoSLG XkIGVKWgyJyrJkPquky1oMN41rThNkD3Xp0Wz0I+hacXrzy04lvYTXwduiIdvxiLroSb fdfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790121532; x=1790726332; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=ndNxVkCk5WMhyvlnx4WQtybJ1XuSLrgG5RrCZvpyS/4=; b=ZUzeo2KxG6Bz+tZkQNXHsPOhESb5/+GLD64SnTDT2MZCxHEu+YB925FYydsETHlN4r EP22ICi8HSh5mAaCLMvFcSz6BUD5pUEjKUA5Xk/uFWunJXu3c3vQYTqgjtD8CKhoG1an FN6Kz/o9T1gK7BXRD6jK5m1/rTJhlrUOl8a4SXseBbiJQrlJDS25yNqwk5+l4hakT7T3 z7AMwTcvJHiBq6YDlxHxhTawsjbQOEpn3YqS9WHqJ0QWCWKWj4znDRr7Abjwrz8MOfZb oZ2vvZeBDDxnulZx2+vzkmczdogP4e+WAL8e1h/z0qTKVDblMCRwJnkwbcNcB5dz7wLx tyPw== X-Gm-Message-State: AFuF++ldaZ6bniiLIKunE8pwdvMWC42Cl82zTOWxudeZp8HXuxYx8Q3M zBGRF7SqQ874smhA5xbQoFhMg/8ictIvdbY6YWvqBDxkaNvSbl3u5NMg8dpIiDe2mZE= X-Gm-Gg: AYBFou1DcGyBfrEiVkEOBcRxBxOlB8duDfDOxSc0rnxFujonWdyPPh/dLTU+/C33KP5 1lUiXy6syQzHCjCWsxZjIQPP88Ao4SSmD1n/jn6vDLwTtmhiPtsRs1rWFlVM4utuwt3g3gxYjb8 oqNotUaKN4TrRAXg09363O/iu+BNutc3TVlP+O8uiN9H993y7SPmtGDr7FntriEk0ISbydmMfKa MST7SME8dtxUJWlcTAXzonJ9X69a+EQCviHuZoMgDi2gpLW4E4WSKxowi6YkA2yqPvcyHtFUDWS TKaoyL7Ln1YgfHnEAlZ2N4WrkojqjZTZAwCz0wXtoeYOv6Woghmorad4qs4G3No1wUCwHHF+Tjl 5N+jb42fTtz+YV/35c/h9MYn8T18oA80h73QJ+Ey7HxqSQnu3pirL6gakGYKBWQtyl2h8+C8xl2 eRyunvbW8WkzwniO+LpWxtrUqmIRYKAMP/88xiZK2cwA2mnHT4H+bJsnBEk1Z4SfjeOT8szXMAQ kFiTFcHCYlPxUCN/RSfOek933RbkedpA1ROJmnSG9arTcB0+tZCLyoeiZ2C X-Received: by 2002:a05:622a:1e06:b0:532:9ade:b6ef with SMTP id d75a77b69052e-532eadc3523mr18790541cf.69.1790121532255; Tue, 22 Sep 2026 16:58:52 -0700 (PDT) Received: from gourry-fedora-PF4VCD3F.lan (pool-173-79-60-52.washdc.fios.verizon.net. [173.79.60.52]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-532eb3c6715sm8323261cf.30.2026.09.22.16.58.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2026 16:58:51 -0700 (PDT) From: Gregory Price To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, kernel-team@meta.com, akpm@linux-foundation.org, liam@infradead.org, ljs@kernel.org, david@kernel.org, vbabka@kernel.org, jannh@google.com, rppt@kernel.org, surenb@google.com, mhocko@suse.com, shuah@kernel.org, "Gregory Price (Meta)" Subject: [PATCH 10/10] mm/madvise: share cold and pageout walk setup Date: Tue, 22 Sep 2026 19:58:30 -0400 Message-ID: <20260922235830.2350770-11-gourry@gourry.net> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260922235830.2350770-1-gourry@gourry.net> References: <20260922235830.2350770-1-gourry@gourry.net> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The MADV_COLD and MADV_PAGEOUT entry points duplicate the TLB and page-walk setup while initializing equivalent private state. Use one madvise_lru_vma_range() helper for the shared TLB lifecycle. Keep operation-specific validation and policy in the two entry points. No functional change intended. Assisted-by: LLM Signed-off-by: Gregory Price (Meta) --- mm/madvise.c | 49 +++++++++++++++++-------------------------------- 1 file changed, 17 insertions(+), 32 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 88a4a03dee04c..37e3d3a205a76 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -622,21 +622,19 @@ static const struct mm_walk_ops cold_walk_ops =3D { .walk_lock =3D PGWALK_RDLOCK, }; =20 -static void madvise_cold_page_range(struct mmu_gather *tlb, - struct madvise_behavior *madv_behavior) - +static void +madvise_lru_vma_range(struct madvise_behavior *madv_behavior, + struct madvise_walk_private *walk_private) { struct vm_area_struct *vma =3D madv_behavior->vma; struct madvise_behavior_range *range =3D &madv_behavior->range; - struct madvise_walk_private walk_private =3D { - .pageout =3D false, - .tlb =3D tlb, - }; =20 - tlb_start_vma(tlb, vma); + tlb_gather_mmu(walk_private->tlb, madv_behavior->mm); + tlb_start_vma(walk_private->tlb, vma); walk_page_range_vma(vma, range->start, range->end, &cold_walk_ops, - &walk_private); - tlb_end_vma(tlb, vma); + walk_private); + tlb_end_vma(walk_private->tlb, vma); + tlb_finish_mmu(walk_private->tlb); } =20 static inline bool can_madv_lru_vma(struct vm_area_struct *vma) @@ -647,39 +645,28 @@ static inline bool can_madv_lru_vma(struct vm_area_st= ruct *vma) static long madvise_cold(struct madvise_behavior *madv_behavior) { struct vm_area_struct *vma =3D madv_behavior->vma; - struct mmu_gather tlb; + struct madvise_walk_private walk_private =3D { + .tlb =3D madv_behavior->tlb, + .pageout =3D false, + }; =20 if (!can_madv_lru_vma(vma)) return -EINVAL; =20 lru_add_drain(); - tlb_gather_mmu(&tlb, madv_behavior->mm); - madvise_cold_page_range(&tlb, madv_behavior); - tlb_finish_mmu(&tlb); + madvise_lru_vma_range(madv_behavior, &walk_private); =20 return 0; } =20 -static void madvise_pageout_page_range(struct mmu_gather *tlb, - struct vm_area_struct *vma, - struct madvise_behavior_range *range) +static long madvise_pageout(struct madvise_behavior *madv_behavior) { + struct vm_area_struct *vma =3D madv_behavior->vma; struct madvise_walk_private walk_private =3D { + .tlb =3D madv_behavior->tlb, .pageout =3D true, - .tlb =3D tlb, }; =20 - tlb_start_vma(tlb, vma); - walk_page_range_vma(vma, range->start, range->end, &cold_walk_ops, - &walk_private); - tlb_end_vma(tlb, vma); -} - -static long madvise_pageout(struct madvise_behavior *madv_behavior) -{ - struct mmu_gather tlb; - struct vm_area_struct *vma =3D madv_behavior->vma; - if (!can_madv_lru_vma(vma)) return -EINVAL; =20 @@ -694,9 +681,7 @@ static long madvise_pageout(struct madvise_behavior *ma= dv_behavior) return 0; =20 lru_add_drain(); - tlb_gather_mmu(&tlb, madv_behavior->mm); - madvise_pageout_page_range(&tlb, vma, &madv_behavior->range); - tlb_finish_mmu(&tlb); + madvise_lru_vma_range(madv_behavior, &walk_private); =20 return 0; } --=20 2.53.0-Meta