From nobody Mon Sep 29 22:43:17 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9ED2BC00140 for ; Mon, 15 Aug 2022 22:32:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1348697AbiHOWaq (ORCPT ); Mon, 15 Aug 2022 18:30:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38724 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1348990AbiHOWZz (ORCPT ); Mon, 15 Aug 2022 18:25:55 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 043F81272C3; Mon, 15 Aug 2022 12:44:35 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8854A610A5; Mon, 15 Aug 2022 19:44:34 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4E306C433C1; Mon, 15 Aug 2022 19:44:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1660592674; bh=uLFx4RHYD59OS7NFdOPVWAzPXWiJpunBElMg6vgcyso=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=REBZeN5+7WP713Xz3RBUwz+ktmRqrmxHddPrBM5pKqBfmBVYIUmk47dwHh8IPPMpx IP1MYnHOILcmpZFMXsCgMOiEMRRXoZ+maa1uw6D4PkrVJwmq9dBrtkI6DZCmKhUNYr jD32JkxKK+AmcKmb48SPnPzD53eapsVG+qmDlxs8= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org Cc: Greg Kroah-Hartman , stable@vger.kernel.org, Ben Segall , Shakeel Butt , Alexander Viro , Linus Torvalds , Eric Dumazet , Roman Penyaev , Jason Baron , Khazhismel Kumykov , Heiher , stable@kernel.org, Andrew Morton Subject: [PATCH 5.19 0146/1157] epoll: autoremove wakers even more aggressively Date: Mon, 15 Aug 2022 19:51:42 +0200 Message-Id: <20220815180445.482120649@linuxfoundation.org> X-Mailer: git-send-email 2.37.2 In-Reply-To: <20220815180439.416659447@linuxfoundation.org> References: <20220815180439.416659447@linuxfoundation.org> User-Agent: quilt/0.67 MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Type: text/plain; charset="utf-8" From: Benjamin Segall commit a16ceb13961068f7209e34d7984f8e42d2c06159 upstream. If a process is killed or otherwise exits while having active network connections and many threads waiting on epoll_wait, the threads will all be woken immediately, but not removed from ep->wq. Then when network traffic scans ep->wq in wake_up, every wakeup attempt will fail, and will not remove the entries from the list. This means that the cost of the wakeup attempt is far higher than usual, does not decrease, and this also competes with the dying threads trying to actually make progress and remove themselves from the wq. Handle this by removing visited epoll wq entries unconditionally, rather than only when the wakeup succeeds - the structure of ep_poll means that the only potential loss is the timed_out->eavail heuristic, which now can race and result in a redundant ep_send_events attempt. (But only when incoming data and a timeout actually race, not on every timeout) Shakeel added: : We are seeing this issue in production with real workloads and it has : caused hard lockups. Particularly network heavy workloads with a lot : of threads in epoll_wait() can easily trigger this issue if they get : killed (oom-killed in our case). Link: https://lkml.kernel.org/r/xm26fsjotqda.fsf@google.com Signed-off-by: Ben Segall Tested-by: Shakeel Butt Cc: Alexander Viro Cc: Linus Torvalds Cc: Shakeel Butt Cc: Eric Dumazet Cc: Roman Penyaev Cc: Jason Baron Cc: Khazhismel Kumykov Cc: Heiher Cc: Signed-off-by: Andrew Morton Signed-off-by: Greg Kroah-Hartman --- fs/eventpoll.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) --- a/fs/eventpoll.c +++ b/fs/eventpoll.c @@ -1747,6 +1747,21 @@ static struct timespec64 *ep_timeout_to_ return to; } =20 +/* + * autoremove_wake_function, but remove even on failure to wake up, becaus= e we + * know that default_wake_function/ttwu will only fail if the thread is al= ready + * woken, and in that case the ep_poll loop will remove the entry anyways,= not + * try to reuse it. + */ +static int ep_autoremove_wake_function(struct wait_queue_entry *wq_entry, + unsigned int mode, int sync, void *key) +{ + int ret =3D default_wake_function(wq_entry, mode, sync, key); + + list_del_init(&wq_entry->entry); + return ret; +} + /** * ep_poll - Retrieves ready events, and delivers them to the caller-suppl= ied * event buffer. @@ -1828,8 +1843,15 @@ static int ep_poll(struct eventpoll *ep, * normal wakeup path no need to call __remove_wait_queue() * explicitly, thus ep->lock is not taken, which halts the * event delivery. + * + * In fact, we now use an even more aggressive function that + * unconditionally removes, because we don't reuse the wait + * entry between loop iterations. This lets us also avoid the + * performance issue if a process is killed, causing all of its + * threads to wake up without being removed normally. */ init_wait(&wait); + wait.func =3D ep_autoremove_wake_function; =20 write_lock_irq(&ep->lock); /*