From nobody Mon Feb 9 17:32:23 2026 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D4CAD194AD1 for ; Mon, 20 Jan 2025 22:47:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737413263; cv=none; b=K7TKKNXrjsynKiGemIW5lItKPCM8gDYMwpKxvJtDPh07vPIW+5SO8361nEWhKmEOagEhlJgmAdwkwM3kPqTgQn9PScg0buyLRZ/eJGA1zKBlmQlPRSD0Jo8wSQCZOcN179hjg+PP08apiRzLBzNgto9iAqCBWYqmtwnAiVs7TsU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737413263; c=relaxed/simple; bh=6yRYdhB36jPY4pi6JDQDXDdlcHNAwhXvAVGVoQsuzrA=; h=MIME-Version:From:Date:Message-ID:Subject:To:Cc:Content-Type; b=NGTyJ5byJGrjISBdoLRzqgIDhZT4XrloixNzEn1MJITMBlJi7ykGW8d4ZEgPlFtnzmhZVznOlRjjDkDtehv9nX/+FKKc7dg5TU3/VHo7u7EEM3E8SoOobgX+/lDy/0dsix2EBUM1jLmCnG6/4H8VcvXqfAT1BHSUDAA4uwm/yNE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=3Crho2t6; arc=none smtp.client-ip=209.85.216.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="3Crho2t6" Received: by mail-pj1-f49.google.com with SMTP id 98e67ed59e1d1-2ee76befe58so8457410a91.2 for ; Mon, 20 Jan 2025 14:47:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1737413261; x=1738018061; darn=vger.kernel.org; h=cc:to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=9lZqlsInkbs+VoXx/qwEPKi8//IGtA1d4kj7PVcfNO4=; b=3Crho2t6mIUB/6YkgoGFOGidy5mSjJsOlWzEYedgNxIqUlKZi5V1QLkQfKhO6bPKoI XesWxvtGCq1vnr/eLOsV4gSTWCAMYzsX0jetKOh0/oTeGcUEIFmXkzDLxfusFz+LffrJ w+yzr8XlxHSkN9cpaRW5WRj0Z988xOmgHG3LTiP8IUHlHBoL8EiNZS4UzwT9mH3JwSRE hjlpxC+EwYU+VjsO4hLDwjJjePo5/iqKzsPJqm8RvAQY9YV8guwV5xnKpvIQp720hPJk 6b9zRtu7OBiQ823p9Q+atZ3YlW8dfqu/S/84/dahgRz5YjUdJ60wCycxgOsd4OeBHc1q Haaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737413261; x=1738018061; h=cc:to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=9lZqlsInkbs+VoXx/qwEPKi8//IGtA1d4kj7PVcfNO4=; b=B2XVmuN8kikknQMd0SyvP8czlv3tgajLXjG31UzBtrd1t5SrsMgmc0zMel0QZWL1tP 1wOLd7/Wjk3R7DrMV78ZwXYWV+vg3c62erzOAkV8Vfy9zEiO6nqxhNlhxSIfEx44EWsD DCFrwQKx6U7MKTrY+i2HwagRediQGr0l7TAde+nhWOa0Nh80x9XUQHrHUiBZe7pw2zN4 y99EpsbVNeRU4pf/kB0I8J3CMncMcoKrQX11IggxLf+6pwVPaXqo4w+cbfbbnM19didJ dUjC6wdAdz9ETDqzSNdR1ZUnp4C0Ljibss3f+Rzd7WmvWIZl5GWZhUNvA+HwNj0Jl4E1 W3oA== X-Forwarded-Encrypted: i=1; AJvYcCWyLIeq3r+cga+CV7kIXvdRpaVmqKa12md7W5GlqHTky/unvCa025m4s5OupDECoFMHZ5sF001dWbOp3JU=@vger.kernel.org X-Gm-Message-State: AOJu0YyaSAutWD3WEPEe8c8he/e7baZenm+iSKVRj9Jn9qYryBIT55PS W9aZ0auBwqYH+4xryCxsoXFk/3KEUThLnn21Z6abB1j6fnMVJOJQB/u53nGBm+CO8VxU//KpLLw 1FHL6XExN5N4nZ1OrHTrBSQ46X5Yar2xmzfTN X-Gm-Gg: ASbGnctJmafOnz7e4o28hcBCidvR3wFumQgNKrJiqglm9MJIkSpJ/UNsenn0jmrvkVJ MF+7uy72EqcnOWkIGTleTcQ9fQCdrOfg6z/9OmxhFeZ1TUGzPfMw= X-Google-Smtp-Source: AGHT+IHhxuz+A74LuQdLWm/u83rIBnpUH6By64pXZNa+qPds8fXdA4RdP+mNcN+Wt+pjFnoiajPqrKWnVWXrLxhfvUU= X-Received: by 2002:a17:90b:38d0:b0:2ee:d35c:3996 with SMTP id 98e67ed59e1d1-2f782d972f5mr21138021a91.31.1737413260868; Mon, 20 Jan 2025 14:47:40 -0800 (PST) Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Vinay Banakar Date: Mon, 20 Jan 2025 16:47:29 -0600 X-Gm-Features: AbW1kvZZQxAZWdnvhEoMNBWPMVHeOp6Nkjvi_VNCLn_UqeLbc7ODri1JXcT4lF8 Message-ID: Subject: [PATCH] mm: Optimize TLB flushes during page reclaim To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: akpm@linux-foundation.org, willy@infradead.org, mgorman@suse.de, Wei Xu , Greg Thelen Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The current implementation in shrink_folio_list() performs full TLB flushes and issues IPIs for each individual page being reclaimed. This causes unnecessary overhead during memory reclaim, whether triggered by madvise(MADV_PAGEOUT) or kswapd, especially in scenarios where applications are actively moving cold pages to swap while maintaining high performance requirements for hot pages. The current code: 1. Clears PTE and unmaps each page individually 2. Performs a full TLB flush on all cores using the VMA (via CR3 write) or issues individual TLB shootdowns (invlpg+invlpcid) for single-core usage 3. Submits each page individually to BIO This approach results in: - Excessive full TLB flushes across all cores - Unnecessary IPI storms when processing multiple pages - Suboptimal I/O submission patterns I initially tried using selective TLB shootdowns (invlpg) instead of full TLB flushes per each page to avoid interference with other threads. However, this approach still required sending IPIs to all cores for each page, which did not significantly improve application throughput. This patch instead optimizes the process by batching operations, issuing one IPI per PMD instead of per page. This reduces interrupts by a factor of 512 and enables batching page submissions to BIO. The new approach: 1. Collect dirty pages that need to be written back 2. Issue a single TLB flush for all dirty pages in the batch 3. Process the collected pages for writebacks (submit to BIO) Testing shows significant reduction in application throughput impact during page-out operations. Applications maintain better performance during memory reclaim, when triggered by explicit madvise(MADV_PAGEOUT) calls. I'd appreciate your feedback on this approach, especially on the correctness of batched BIO submissions. Looking forward to your comments. Signed-off-by: Vinay Banakar --- mm/vmscan.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-= -------------------------------- 1 file changed, 74 insertions(+), 33 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index bd489c1af..1bd510622 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1035,6 +1035,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, struct folio_batch free_folios; LIST_HEAD(ret_folios); LIST_HEAD(demote_folios); + LIST_HEAD(pageout_list); unsigned int nr_reclaimed =3D 0; unsigned int pgactivate =3D 0; bool do_demote_pass; @@ -1351,39 +1352,9 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, if (!sc->may_writepage) goto keep_locked; - /* - * Folio is dirty. Flush the TLB if a writable entry - * potentially exists to avoid CPU writes after I/O - * starts and then write it out here. - */ - try_to_unmap_flush_dirty(); - switch (pageout(folio, mapping, &plug)) { - case PAGE_KEEP: - goto keep_locked; - case PAGE_ACTIVATE: - goto activate_locked; - case PAGE_SUCCESS: - stat->nr_pageout +=3D nr_pages; - - if (folio_test_writeback(folio)) - goto keep; - if (folio_test_dirty(folio)) - goto keep; - - /* - * A synchronous write - probably a ramdisk. Go - * ahead and try to reclaim the folio. - */ - if (!folio_trylock(folio)) - goto keep; - if (folio_test_dirty(folio) || - folio_test_writeback(folio)) - goto keep_locked; - mapping =3D folio_mapping(folio); - fallthrough; - case PAGE_CLEAN: - ; /* try to free the folio below */ - } + /* Add to pageout list for defered bio submissions */ + list_add(&folio->lru, &pageout_list); + continue; } /* @@ -1494,6 +1465,76 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, } /* 'folio_list' is always empty here */ + if (!list_empty(&pageout_list)) { + /* + * Batch TLB flushes by flushing once before processing all dirty pages. + * Since we operate on one PMD at a time, this batches TLB flushes at + * PMD granularity rather than per-page, reducing IPIs. + */ + struct address_space *mapping; + try_to_unmap_flush_dirty(); + + while (!list_empty(&pageout_list)) { + struct folio *folio =3D lru_to_folio(&pageout_list); + list_del(&folio->lru); + + /* Recheck if page got reactivated */ + if (folio_test_active(folio) || + (folio_mapped(folio) && folio_test_young(folio))) + goto skip_pageout_locked; + + mapping =3D folio_mapping(folio); + pageout_t pageout_res =3D pageout(folio, mapping, &plug); + switch (pageout_res) { + case PAGE_KEEP: + goto skip_pageout_locked; + case PAGE_ACTIVATE: + goto skip_pageout_locked; + case PAGE_SUCCESS: + stat->nr_pageout +=3D folio_nr_pages(folio); + + if (folio_test_writeback(folio) || + folio_test_dirty(folio)) + goto skip_pageout; + + /* + * A synchronous write - probably a ramdisk. Go + * ahead and try to reclaim the folio. + */ + if (!folio_trylock(folio)) + goto skip_pageout; + if (folio_test_dirty(folio) || + folio_test_writeback(folio)) + goto skip_pageout_locked; + + // Try to free the page + if (!mapping || + !__remove_mapping(mapping, folio, true, + sc->target_mem_cgroup)) + goto skip_pageout_locked; + + nr_reclaimed +=3D folio_nr_pages(folio); + folio_unlock(folio); + continue; + + case PAGE_CLEAN: + if (!mapping || + !__remove_mapping(mapping, folio, true, + sc->target_mem_cgroup)) + goto skip_pageout_locked; + + nr_reclaimed +=3D folio_nr_pages(folio); + folio_unlock(folio); + continue; + } + +skip_pageout_locked: + folio_unlock(folio); +skip_pageout: + list_add(&folio->lru, &ret_folios); + } + } + /* Migrate folios selected for demotion */ nr_reclaimed +=3D demote_folio_list(&demote_folios, pgdat); /* Folios that could not be demoted are still in @demote_folios */