From nobody Sat Jul 25 02:41:19 2026 Received: from stravinsky.debian.org (stravinsky.debian.org [82.195.75.108]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9FFED3DB999; Mon, 20 Jul 2026 10:43:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=82.195.75.108 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784544228; cv=none; b=HDmSl8Jw6TO6VkT9ul5FpY4IVvuFwhylxRjoxHNyfo19YQ+2Tm8pYYLq7AaaUvjcMhak5rf8LqUrz+AjpP69AFVCx1B/pqUr1b2ZV89nYV923JNsedMx4UvcmTJPBe8pZLV77Qe3ABQx0en3TgwO5ZijvaCxwGn+Ql2Z9MID7Lo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784544228; c=relaxed/simple; bh=OSjY/yBLoOPouuxkDklnm8NP7swE9/aGE7jyD806zbQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:To:Cc; b=fUYVCXAmvPPQ/tR3MPGLPz5fS/gYVZWYy/UiqeJ0PFhPYKPn2uXvw/XT4iUBr9wHMFYyKiEi37rh56j8R73QKgTnrKUfRWHgxBGM+8wp53gkWECJXqt2+dC9iQmKVKqWrCBVNqb1LztDW9fIcuGbYijbKjwdRmeJ2buJfVX3kcM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org; spf=pass smtp.mailfrom=debian.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b=TDgTMASG; arc=none smtp.client-ip=82.195.75.108 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=debian.org Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=debian.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=debian.org header.i=@debian.org header.b="TDgTMASG" DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=debian.org; s=smtpauto.stravinsky; h=X-Debian-User:Cc:To:Message-Id: Content-Transfer-Encoding:Content-Type:MIME-Version:Subject:Date:From: Reply-To:Content-ID:Content-Description:In-Reply-To:References; bh=0KnocTSeBHw5oYENnuQRyIIKYC+sMkJg/4yZ6eXlvk8=; b=TDgTMASG6vPzN7ISP2BLrJ4l2N 1r4/iRBDZGZBsBqvHOFEcO1mhbYbAw3ERSd3Wr/azLPPN4ILbuJd954KGp4EjcZxyPR+XYimdMbPi DHTSyc3P1t6FfSp3WvFfoKYgZplSqvldXvbk8JkN6z6rigQxSHrYXQDBlYNQC4Cu9Qad2di+RvAY5 3hqo4n+2jwl7aN9OPNzj1CH11Px3QtcT+55Mb9PbzHk/w70J1L2JaAUG9aR1S1DWSg0KufdIXYFGR wAbW/5BNyhYwJrFGSXix8oIXswUVj9nruQyPq0itzJ0Vre4Pff5J5KnyZjyTj5AAKYSsG2ueMeCAs /y1KhiMg==; Received: from authenticated-user by stravinsky.debian.org with esmtpsa (TLS1.3:ECDHE_X25519__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim 4.96) (envelope-from ) id 1wllT8-001Nbl-2f; Mon, 20 Jul 2026 10:43:32 +0000 From: Breno Leitao Date: Mon, 20 Jul 2026 03:39:33 -0700 Subject: [PATCH v5] fs/pipe: unify the page pools into a single per-pipe pool Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Message-Id: <20260720-b4-pipe-unification-v5-1-9002a3fe5e6d@debian.org> X-B4-Tracking: v=1; b=H4sIAOT6XWoC/23Ny2rDMBBA0V8Rs/YUPSzJ9qr/UbrQY5TMRjZyY lqC/70kUJoSb+/i3Bus1JhWmMQNGm288lxhErYTkM6hngg5wyRAS+2k0xZjjwsvhNfKhVO48Fw xxODjYLXN5KETsDQq/PVQPz47AWdeL3P7fkw2da+/njv0NoUSszYluKiItH/PFDnUt7md4A5u+ g/x0h8jGiVStDrmTEQlvyDmGRmPEYMKB5lCITl6N6gXpH9ClDxGelRYilHJjEUl5f4h+77/AMo r6ZmGAQAA X-Change-ID: 20260625-b4-pipe-unification-aba7b8525de7 To: Alexander Viro , Christian Brauner , oleg@redhat.com, mjguzik@gmail.com, josh@joshtriplett.org, Jan Kara , jlayton@kernel.org Cc: axboe@kernel.dk, shakeel.butt@linux.dev, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, kernel-team@meta.com, Breno Leitao X-Mailer: b4 0.16-dev-d5d98 X-Developer-Signature: v=1; a=openpgp-sha256; l=12477; i=leitao@debian.org; h=from:subject:message-id; bh=OSjY/yBLoOPouuxkDklnm8NP7swE9/aGE7jyD806zbQ=; b=owEBbQKS/ZANAwAIATWjk5/8eHdtAcsmYgBqXfvPzorQaaNo3Dg5pMPkIbIpQcfqwNPlfHRNW BdyB40g0pCJAjMEAAEIAB0WIQSshTmm6PRnAspKQ5s1o5Of/Hh3bQUCal37zwAKCRA1o5Of/Hh3 bUcaD/9/zzHTWqXaCUr9MIRYE2eS4A5EU3s138TVsEr5rukfQzDWhTB7uXZqpdKpvWXoAKlcTzK fpE9SgywEsRH5tCUT63bD2wZO/AXytnFVmFj14+zzds6jLU8CLrohYEOI1sUcpWaUpES039InXJ 9RBebETNeKV+GKWusXbRmYOC7KVvEkXUCdHfKcvgJ7xVKxqN8KQFC/Zck05OUynk8KX5ZRbiJBw mo1+mLKLy3LY0gw927tnKMik4s8zCCEQREK44L8KQdwxAN1Rf5IMomDm68GjS6n34YmuljOpcFB Jkktvx0ksVkdMe0+PEohtsmtcVd5njV3xITFv+JfwsIIwYJpy51OaunYWqfsOMAgQoJq8wqmBwv 92OQDWQamWu0XcpcKv1mbivlPisWLq/9WPj7ekuzDOF8SGRuYbvlGy7RDaTJWVRws71YOUeYCKB pgYxtvhPzdtg7DHMRyBFzww94Mqy4BY1gZsRQIhLKXgHl4AFt2zhUSvzXaCfftlnhaYyO6BFUVv LYq+bYjeEFg1bsI1h3NWRJHee/MPTQP0Yzogj5wCfKUKC3fqmKDapJxyB3rg6D5HchKamBiRPkk Y5jZBzemIKQ4UweUaO8ErRCnTETcLcuUjBzvrGKkb1IJ/vJArDzTP5abVOwErTj4WIGxCqSXe1U zZvE07edBCogBvw== X-Developer-Key: i=leitao@debian.org; a=openpgp; fpr=AC8539A6E8F46702CA4A439B35A3939FFC78776D X-Debian-User: leitao Pipes keep two separate page caches: a) The per-pipe, lock-protected tmp_page[2] b) An on-stack anon_pipe_prealloc burst pool of up to eight pages filled before the lock Converge them into a single per-pipe pool (struct anon_pipe_prealloc embedded in pipe_inode_info) with the same budget as before: up to PIPE_PREALLOC_MAX (8) pages, trimmed back to PIPE_PREALLOC_KEEP (2) after each operation. tmp_page[2] is removed. Pages are still allocated and freed outside pipe->mutex; only the assignment into the pool is done under it. The pool count is also read locklessly in the prefill path, so it is annotated __data_racy. anon_pipe_prefill_and_lock() tops the pool up to the write's page count -- and returns with pipe->mutex held, so a write acquires the lock only once. anon_pipe_trim_and_unlock() trims the pool under that same lock before dropping it, then frees the excess. Signed-off-by: Breno Leitao Reviewed-by: Mateusz Guzik Reviewed-by: Oleg Nesterov --- Changes in v5: - mark count as racy (Oleg) - Link to v4: https://patch.msgid.link/20260710-b4-pipe-unification-v4-1-ff= 31c39f1c16@debian.org Changes in v4: - Rename anon_pipe_trim_pool_and_unlock() to anon_pipe_trim_and_unlock() for naming symmetry (Guzik) - Link to v3: https://patch.msgid.link/20260709-b4-pipe-unification-v3-1-80= cafe097681@debian.org Changes in v3: - Squashes them into a single patch (Guzik) - Fold prefill and trim into one mutex acquire per write. (Guzik) - Link to v2: https://patch.msgid.link/20260707-b4-pipe-unification-v2-0-eb= 52bddeeefd@debian.org Changes in v2: - User READ_ONCE to read prealloc.count - Trim the pool at the reader side - Link to v1: https://lore.kernel.org/r/20260626-b4-pipe-unification-v1-0-d= 23fa6b1ee27@debian.org --- fs/pipe.c | 171 ++++++++++++++++++++----------------------= ---- include/linux/pipe_fs_i.h | 22 +++++- 2 files changed, 94 insertions(+), 99 deletions(-) diff --git a/fs/pipe.c b/fs/pipe.c index 429b0714ec575..3c6061cefe791 100644 --- a/fs/pipe.c +++ b/fs/pipe.c @@ -111,75 +111,95 @@ void pipe_double_lock(struct pipe_inode_info *pipe1, pipe_lock(pipe2); } =20 -#define PIPE_PREALLOC_MAX 8 +static struct page *anon_pipe_prealloc_pop(struct anon_pipe_prealloc *prea= lloc) +{ + if (!prealloc->count) + return NULL; =20 -struct anon_pipe_prealloc { - struct page *pages[PIPE_PREALLOC_MAX]; - unsigned int count; -}; + prealloc->count--; + + return prealloc->pages[prealloc->count]; +} + +/* Push a page to the prealloc pool. Returns true if added, false if full.= */ +static bool anon_pipe_prealloc_push(struct anon_pipe_prealloc *prealloc, + struct page *page) +{ + if (prealloc->count >=3D PIPE_PREALLOC_MAX) + return false; + prealloc->pages[prealloc->count++] =3D page; + return true; +} =20 /* - * Pre-allocate pages outside pipe->mutex for multi-page writes. - * alloc_page() with GFP_HIGHUSER can sleep in reclaim and runs memcg - * charging; doing it under the mutex stalls a concurrent reader. - * - * Loop alloc_page() instead of alloc_pages_bulk_*(): the bulk path refuses - * __GFP_ACCOUNT under memcg (see commit 8dcb3060d81d "memcg: page_alloc: - * skip bulk allocator for __GFP_ACCOUNT") and silently degrades to a sing= le - * page. A per-page loop keeps memcg accounting and the task NUMA mempolicy - * honoured for every page; the per-call overhead is small compared to the - * pipe->mutex hold-time being shrunk. Any shortfall is covered by the - * in-lock alloc_page() fallback in anon_pipe_get_page(). + * Top up the pipe's own pool, then take pipe->mutex and return with it he= ld. + * The shortfall is allocated outside the lock; the push and the caller's = write + * then run under a single lock acquisition, avoiding a separate prefill + * lock/unlock cycle. anon_pipe_get_page() drains the pool instead of allo= cating + * under the lock. */ -static void anon_pipe_get_page_prealloc(struct anon_pipe_prealloc *preallo= c, - size_t total_len) +static void anon_pipe_prefill_and_lock(struct pipe_inode_info *pipe, size_= t total_len) { - unsigned int want, i; - struct page *page; - - prealloc->count =3D 0; - if (total_len <=3D PAGE_SIZE) - return; + struct page *pages[PIPE_PREALLOC_MAX]; + unsigned int want, have, need, n =3D 0; =20 want =3D min_t(unsigned int, DIV_ROUND_UP(total_len, PAGE_SIZE), PIPE_PREALLOC_MAX); + /* Unlocked read; the pool is refilled under the lock below. */ + have =3D min_t(unsigned int, READ_ONCE(pipe->prealloc.count), want); + need =3D want - have; + + if (!need) { + mutex_lock(&pipe->mutex); + return; + } + + while (n < need) { + struct page *page =3D alloc_page(GFP_HIGHUSER | __GFP_ACCOUNT); =20 - for (i =3D 0; i < want; i++) { - page =3D alloc_page(GFP_HIGHUSER | __GFP_ACCOUNT); if (!page) break; - prealloc->pages[prealloc->count++] =3D page; + pages[n++] =3D page; } + + mutex_lock(&pipe->mutex); + while (n && anon_pipe_prealloc_push(&pipe->prealloc, pages[n - 1])) + n--; + + /* + * Just flush any extra page that got affected by the TOCTOU + * effect + */ + while (n) + put_page(pages[--n]); } =20 -static struct page *anon_pipe_prealloc_pop(struct anon_pipe_prealloc *prea= lloc) +/* + * Called with pipe->mutex held. Trim the pool down to PIPE_PREALLOC_KEEP = under + * the lock, drop it, then free the excess outside the critical section. + */ +static void anon_pipe_trim_and_unlock(struct pipe_inode_info *pipe) { - if (!prealloc->count) - return NULL; + struct page *excess[PIPE_PREALLOC_MAX]; + unsigned int nexcess =3D 0; =20 - prealloc->count--; + while (pipe->prealloc.count > PIPE_PREALLOC_KEEP) + excess[nexcess++] =3D anon_pipe_prealloc_pop(&pipe->prealloc); + mutex_unlock(&pipe->mutex); =20 - return prealloc->pages[prealloc->count]; + while (nexcess) + put_page(excess[--nexcess]); } =20 -static struct page *anon_pipe_get_page(struct pipe_inode_info *pipe, - struct anon_pipe_prealloc *prealloc) +static struct page *anon_pipe_get_page(struct pipe_inode_info *pipe) { struct page *page; =20 - /* Drain prealloc first to keep tmp_page[] hot for later small writes. */ - page =3D anon_pipe_prealloc_pop(prealloc); + /* Drain the prealloc pool before allocating. Called with mutex held. */ + page =3D anon_pipe_prealloc_pop(&pipe->prealloc); if (page) return page; =20 - for (int i =3D 0; i < ARRAY_SIZE(pipe->tmp_page); i++) { - if (pipe->tmp_page[i]) { - page =3D pipe->tmp_page[i]; - pipe->tmp_page[i] =3D NULL; - return page; - } - } - /* FWIW: This is called with pipe->mutex held */ return alloc_page(GFP_HIGHUSER | __GFP_ACCOUNT); } @@ -187,48 +207,11 @@ static struct page *anon_pipe_get_page(struct pipe_in= ode_info *pipe, static void anon_pipe_put_page(struct pipe_inode_info *pipe, struct page *page) { - if (page_count(page) =3D=3D 1) { - for (int i =3D 0; i < ARRAY_SIZE(pipe->tmp_page); i++) { - if (!pipe->tmp_page[i]) { - pipe->tmp_page[i] =3D page; - return; - } - } - } - - put_page(page); -} - -/* - * Stash leftover prealloc pages in tmp_page[] so the next write to this - * pipe gets a hot page without entering the allocator. - */ -static void anon_pipe_refill_tmp_pages(struct pipe_inode_info *pipe, - struct anon_pipe_prealloc *prealloc) -{ - int i, idx; - - if (!prealloc->count) + if (page_count(page) =3D=3D 1 && + anon_pipe_prealloc_push(&pipe->prealloc, page)) return; =20 - for (i =3D 0; i < ARRAY_SIZE(pipe->tmp_page); i++) { - if (pipe->tmp_page[i]) - continue; - if (!prealloc->count) - return; - idx =3D --prealloc->count; - pipe->tmp_page[i] =3D prealloc->pages[idx]; - prealloc->pages[idx] =3D NULL; - } -} - -/* Runs after mutex_unlock() to keep put_page() out of the critical sectio= n. */ -static void anon_pipe_free_pages(struct anon_pipe_prealloc *prealloc) -{ - while (prealloc->count) { - prealloc->count--; - put_page(prealloc->pages[prealloc->count]); - } + put_page(page); } =20 static void anon_pipe_buf_release(struct pipe_inode_info *pipe, @@ -485,7 +468,8 @@ anon_pipe_read(struct kiocb *iocb, struct iov_iter *to) } if (pipe_is_empty(pipe)) wake_next_reader =3D false; - mutex_unlock(&pipe->mutex); + /* Consumed buffers may have refilled the pool; trim it and unlock. */ + anon_pipe_trim_and_unlock(pipe); =20 if (wake_writer) wake_up_interruptible_sync_poll(&pipe->wr_wait, EPOLLOUT | EPOLLWRNORM); @@ -524,7 +508,6 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *fr= om) { struct file *filp =3D iocb->ki_filp; struct pipe_inode_info *pipe =3D filp->private_data; - struct anon_pipe_prealloc prealloc; unsigned int head; ssize_t ret =3D 0; size_t total_len =3D iov_iter_count(from); @@ -548,9 +531,7 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *fr= om) if (unlikely(total_len =3D=3D 0)) return 0; =20 - anon_pipe_get_page_prealloc(&prealloc, total_len); - - mutex_lock(&pipe->mutex); + anon_pipe_prefill_and_lock(pipe, total_len); =20 if (!pipe->readers) { if ((iocb->ki_flags & IOCB_NOSIGNAL) =3D=3D 0) @@ -607,7 +588,7 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *fr= om) struct page *page; int copied; =20 - page =3D anon_pipe_get_page(pipe, &prealloc); + page =3D anon_pipe_get_page(pipe); if (unlikely(!page)) { if (!ret) ret =3D -ENOMEM; @@ -671,11 +652,9 @@ anon_pipe_write(struct kiocb *iocb, struct iov_iter *f= rom) wake_next_writer =3D true; } out: - anon_pipe_refill_tmp_pages(pipe, &prealloc); if (pipe_is_full(pipe)) wake_next_writer =3D false; - mutex_unlock(&pipe->mutex); - anon_pipe_free_pages(&prealloc); + anon_pipe_trim_and_unlock(pipe); =20 /* * If we do do a wakeup event, we do a 'sync' wakeup, because we @@ -956,10 +935,8 @@ void free_pipe_info(struct pipe_inode_info *pipe) if (pipe->watch_queue) put_watch_queue(pipe->watch_queue); #endif - for (i =3D 0; i < ARRAY_SIZE(pipe->tmp_page); i++) { - if (pipe->tmp_page[i]) - __free_page(pipe->tmp_page[i]); - } + for (i =3D 0; i < pipe->prealloc.count; i++) + __free_page(pipe->prealloc.pages[i]); kfree(pipe->bufs); kfree(pipe); } diff --git a/include/linux/pipe_fs_i.h b/include/linux/pipe_fs_i.h index 7f6a92ac97047..9e8b60ad806a9 100644 --- a/include/linux/pipe_fs_i.h +++ b/include/linux/pipe_fs_i.h @@ -14,6 +14,9 @@ #define PIPE_BUF_FLAG_LOSS 0x40 /* Message loss happened after this buffer= */ #endif =20 +#define PIPE_PREALLOC_MAX 8 /* max pages in prealloc pool */ +#define PIPE_PREALLOC_KEEP 2 /* keep at least this many after trim */ + /** * struct pipe_buffer - a linux kernel pipe buffer * @page: the page containing the data for the pipe buffer @@ -58,6 +61,21 @@ union pipe_index { }; }; =20 +/** + * struct anon_pipe_prealloc - per-pipe page preallocation pool + * @pages: array of cached pages (pool) + * @count: number of pages currently in the pool + * + * Each pipe keeps a small bounded pool of preallocated pages to reduce + * allocation overhead during writes. The pool is bounded at PIPE_PREALLOC= _MAX + * and trimmed down to PIPE_PREALLOC_KEEP after a write completes. + */ +struct anon_pipe_prealloc { + struct page *pages[PIPE_PREALLOC_MAX]; + + unsigned int __data_racy count; +}; + /** * struct pipe_inode_info - a linux kernel pipe * @mutex: mutex protecting the whole thing @@ -68,7 +86,7 @@ union pipe_index { * @max_usage: The maximum number of slots that may be used in the ring * @ring_size: total number of buffers (should be a power of 2) * @nr_accounted: The amount this pipe accounts for in user->pipe_bufs - * @tmp_page: cached released page + * @prealloc: per-pipe page preallocation pool * @readers: number of current readers of this pipe * @writers: number of current writers of this pipe * @files: number of struct file referring this pipe (protected by ->i_loc= k) @@ -99,7 +117,7 @@ struct pipe_inode_info { #ifdef CONFIG_WATCH_QUEUE bool note_loss; #endif - struct page *tmp_page[2]; + struct anon_pipe_prealloc prealloc; struct fasync_struct *fasync_readers; struct fasync_struct *fasync_writers; struct pipe_buffer *bufs; --- base-commit: b9810cd75b9fb56a3425d391cba3f608502bd474 change-id: 20260625-b4-pipe-unification-aba7b8525de7 Best regards, -- =20 Breno Leitao