From nobody Sat Sep 26 11:01:57 2026 Received: from m16.mail.163.com (m16.mail.163.com [220.197.31.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1295C426432; Wed, 2 Sep 2026 09:38:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=220.197.31.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788341925; cv=none; b=gFklZ4W90sQ0h8OQk5S16B2d/bCRiT1xqE6KQsocPlE8FE3kfcU4IHPmIgjrHUlFLF6/4QHWmFXmHS3PdjJzntG425MZ70Vtxm6CAIBZF5klnZeO47Y3T/AjOeymZbiGreWQKzgW25geO+khJSHHH2G6blWYX8sttxHmL/Oohzs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788341925; c=relaxed/simple; bh=NiyyWUH1prkVse1OkE9B2qMJ6JZL7ZE6Xekm99UEJDo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CE1nlrqLdM8ZaRfouTFvtTKoLI5VWT952daKz3w2SdpIx4L8pMvHAIOFs02L9d28nmaXba2ag53rc9JVEas7zRQ3Yyb7uZjZ/YYHZ3yajSs/+w+BknC3Y27LrTisxTZo0Xc3kO/3FiByr3PBt+VSfbzktwCcAzOHrh8mHdxkq3I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=dC1wHpmN; arc=none smtp.client-ip=220.197.31.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="dC1wHpmN" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=+J HuhUJjxcmnFhFTzIqRT3sPERPM9fOZMeWKzbtsUd0=; b=dC1wHpmN31QT46fFrg 8WVPCopGREbkKxn8Uw3IEVLU7lnqT/Dyw1PnX9JIQYOTJZHkaAvv32XbqiUkla3T t2CmntK3SexCfSfIo91YLBfLBg8qpU1g0u1ATS8+plxvXDD6Y7xd/O8iXiDhLaF/ MvSEj3k3csjQVQyiQIyk7pnxc= Received: from nec8-i7 (unknown []) by gzga-smtp-mtada-g0-0 (Coremail) with SMTP id _____wD3F7Vu7pdqogtpAA--.21222S3; Wed, 02 Sep 2026 17:37:53 +0800 (CST) From: chenyuan_fl@163.com To: bpf@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Alexei Starovoitov , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Martin KaFai Lau , Song Liu , Yonghong Song , Jiri Olsa , Emil Tsalapatis , Ihor Solodrai , Yuan Chen Subject: [PATCH bpf-next v5 1/3] bpf, arena: fix range_tree_clear inconsistency on kmalloc_nolock failure Date: Wed, 2 Sep 2026 17:37:38 +0800 Message-ID: <20260902093740.2338724-2-chenyuan_fl@163.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260902093740.2338724-1-chenyuan_fl@163.com> References: <20260902093740.2338724-1-chenyuan_fl@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: _____wD3F7Vu7pdqogtpAA--.21222S3 X-Coremail-Antispam: 1Uf129KBjvJXoW7Cw48Xw45trWrXFyUuFW8tFb_yoW8GF1fpw 4UK34fAws0q3yUWrZa9F4vkr98Can3Xr48K343tw4kAr15A3Z7urnY9r4293ZrAFZFy3Wr tF1jva9rK3WUuFDanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x07UCXd8UUUUU= X-CM-SenderInfo: xfkh05pxdqswro6rljoofrz/xtbDAhFYFmqX7nEOsgAA3D Content-Type: text/plain; charset="utf-8" From: Yuan Chen range_tree_clear() pre-allocates the right-half node before modifying the tree, so an allocation failure returns -ENOMEM without altering the range tree. Signed-off-by: Yuan Chen --- kernel/bpf/range_tree.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/kernel/bpf/range_tree.c b/kernel/bpf/range_tree.c index 2f28886f3ff7..15b588377a76 100644 --- a/kernel/bpf/range_tree.c +++ b/kernel/bpf/range_tree.c @@ -143,16 +143,22 @@ int range_tree_clear(struct range_tree *rt, u32 start= , u32 len) if (rn->rn_start < start && rn->rn_last > last) { u32 old_last =3D rn->rn_last; =20 + /* + * Pre-allocate the right-half node before modifying + * the tree. If allocation fails we return -ENOMEM + * without altering the range tree. + */ + new_rn =3D kmalloc_nolock(sizeof(struct range_node), + __GFP_ACCOUNT, NUMA_NO_NODE); + if (!new_rn) + return -ENOMEM; + /* Overlaps with the entire clearing range */ range_it_remove(rn, rt); rn->rn_last =3D start - 1; range_it_insert(rn, rt); =20 - /* Add a range */ - new_rn =3D kmalloc_nolock(sizeof(struct range_node), __GFP_ACCOUNT, - NUMA_NO_NODE); - if (!new_rn) - return -ENOMEM; + /* Add right-half range */ new_rn->rn_start =3D last + 1; new_rn->rn_last =3D old_last; range_it_insert(new_rn, rt); --=20 2.54.0 From nobody Sat Sep 26 11:01:57 2026 Received: from m16.mail.163.com (m16.mail.163.com [220.197.31.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D8634229A6; Wed, 2 Sep 2026 09:38:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=220.197.31.3 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788341920; cv=none; b=bdjOZYqj8rBSUT7OHwzNh9E83GGN4h+pReZao7MVYSSK6kjX0Bbw4WmTdmAM7ZI38KmuIbiEgLkKp77WdkK1gf6yprkWUWIC+iXbQNM0w7+RX3lhPou+PbS5Wnr/6npwXYAdGp2VOqLb+3MTczG7A2tZzdCgdal+Nilp2OgVQMM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788341920; c=relaxed/simple; bh=IwROk44CZaQG+siuubT32QXiRJHbp5/a/TuPA93r2pQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LtaS34C2odJRT+5uZcU9hVRe9l+UZ9/tIICC0dtuPp8Y5wRXPy9ZnOb4eVjiwNVlAVpr/xVjyGlLLtllGlfnxLBfYChmJ8FNOep+Vf9BI1Evttp/M95oUtydRSSUfIg/x8AvtA8IiqzME7bbx95lIGX3vEfYC17eYAy+KA6e2Bw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=PFufibr8; arc=none smtp.client-ip=220.197.31.3 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="PFufibr8" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=zr VKOIfZyNfdBaup7kUIwUMlr9Bdz0dloClPNSFyXyY=; b=PFufibr8UkN7dLNEP4 e3w84zL+/Fj0jHTso8J/7oZSBacGgvQ93plbH0KoHMsyEwuJrp/p7TlOnmdbhU0M qgeACe4oAcustGRdyRkVzkwY9zkVoGDHKHtkjpG0fvlRyf1A6jlCz7nsjZtTR5xB adOWQC51HXVoYshU7dMxl8Ijs= Received: from nec8-i7 (unknown []) by gzga-smtp-mtada-g0-0 (Coremail) with SMTP id _____wD3F7Vu7pdqogtpAA--.21222S4; Wed, 02 Sep 2026 17:37:54 +0800 (CST) From: chenyuan_fl@163.com To: bpf@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Alexei Starovoitov , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Martin KaFai Lau , Song Liu , Yonghong Song , Jiri Olsa , Emil Tsalapatis , Ihor Solodrai , Yuan Chen Subject: [PATCH bpf-next v5 2/3] bpf, arena: fix range_tree_set inconsistency on kmalloc_nolock failure Date: Wed, 2 Sep 2026 17:37:39 +0800 Message-ID: <20260902093740.2338724-3-chenyuan_fl@163.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260902093740.2338724-1-chenyuan_fl@163.com> References: <20260902093740.2338724-1-chenyuan_fl@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: _____wD3F7Vu7pdqogtpAA--.21222S4 X-Coremail-Antispam: 1Uf129KBjvJXoWxCw13AF1kWr48GF4xAFy8Grg_yoW5CFWrpw 15KrWrJrsxJ3y7Ww1Fvr48uFyrCw1fXw4ktrZxGw4kA3W5Ar97AFn0kF4jkFWDAFy8Cr15 tF1jv3y7GF4UuFDanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x07jbo7NUUUUU= X-CM-SenderInfo: xfkh05pxdqswro6rljoofrz/xtbDAhJYFmqX7nIOyAAA35 Content-Type: text/plain; charset="utf-8" From: Yuan Chen range_tree_set() pre-allocates the node needed for a brand-new range before calling range_tree_clear(), so an allocation failure returns -ENOMEM without having modified the tree (previously the overlapping nodes were already removed by range_tree_clear() before the allocation was attempted, permanently losing the cleared sub-ranges). Signed-off-by: Yuan Chen --- kernel/bpf/range_tree.c | 45 +++++++++++++++++++++++++++++++---------- 1 file changed, 34 insertions(+), 11 deletions(-) diff --git a/kernel/bpf/range_tree.c b/kernel/bpf/range_tree.c index 15b588377a76..0420ab715f20 100644 --- a/kernel/bpf/range_tree.c +++ b/kernel/bpf/range_tree.c @@ -199,6 +199,7 @@ int is_range_tree_set(struct range_tree *rt, u32 start,= u32 len) int range_tree_set(struct range_tree *rt, u32 start, u32 len) { u32 last =3D start + len - 1; + struct range_node *new_rn =3D NULL; struct range_node *right; struct range_node *left; int err; @@ -208,20 +209,40 @@ int range_tree_set(struct range_tree *rt, u32 start, = u32 len) if (left && left->rn_start <=3D start && left->rn_last >=3D last) return 0; =20 + /* + * The pre-clear probe only decides whether a fresh node is needed; + * adjacency on either side is invariant across the clear, so it can + * run before the tree is modified and -ENOMEM leaves it untouched. + * The merge below re-fetches its handles after the clear instead of + * relying on the overlapping nodes being updated in place. + */ + left =3D range_it_iter_first(rt, start - 1, start - 1); + right =3D range_it_iter_first(rt, last + 1, last + 1); + if (!left && !right) { + new_rn =3D kmalloc_nolock(sizeof(struct range_node), + __GFP_ACCOUNT, NUMA_NO_NODE); + if (!new_rn) + return -ENOMEM; + } + /* Clear out everything in the range we want to set. */ err =3D range_tree_clear(rt, start, len); if (err) - return err; + goto out_free_new; =20 /* Do we have a left-adjacent range ? */ left =3D range_it_iter_first(rt, start - 1, start - 1); - if (left && left->rn_last + 1 !=3D start) - return -EFAULT; + if (left && left->rn_last + 1 !=3D start) { + err =3D -EFAULT; + goto out_free_new; + } =20 /* Do we have a right-adjacent range ? */ right =3D range_it_iter_first(rt, last + 1, last + 1); - if (right && right->rn_start !=3D last + 1) - return -EFAULT; + if (right && right->rn_start !=3D last + 1) { + err =3D -EFAULT; + goto out_free_new; + } =20 if (left && right) { /* Combine left and right adjacent ranges */ @@ -241,14 +262,16 @@ int range_tree_set(struct range_tree *rt, u32 start, = u32 len) right->rn_start =3D start; range_it_insert(right, rt); } else { - left =3D kmalloc_nolock(sizeof(struct range_node), __GFP_ACCOUNT, NUMA_N= O_NODE); - if (!left) - return -ENOMEM; - left->rn_start =3D start; - left->rn_last =3D last; - range_it_insert(left, rt); + /* No adjacent ranges; use the pre-allocated node */ + new_rn->rn_start =3D start; + new_rn->rn_last =3D last; + range_it_insert(new_rn, rt); } return 0; + +out_free_new: + kfree_nolock(new_rn); + return err; } =20 void range_tree_destroy(struct range_tree *rt) --=20 2.54.0 From nobody Sat Sep 26 11:01:57 2026 Received: from m16.mail.163.com (m16.mail.163.com [117.135.210.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 721FB4218B6; Wed, 2 Sep 2026 09:38:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=117.135.210.3 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788341919; cv=none; b=ryoh1ti1hSekrAL0RzdfhDWbXpkSL5IkuhJACyh8vKFNF8w5bC0c7LRNSKnwinbVAaQBus8auN/u6Fx9psaiJk8ntQImup6/h6t+F/SAX9MbYmrupgsbp1nbKyny6MXRTNtqPo+bBCsDwL1JCWEbXoG+xE5iEH379BPy/dB60pg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788341919; c=relaxed/simple; bh=8NRZLSZC/WH0iPpJIOU+E5GNNTl2iXvwpUd2oPyB8RM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=I2oejMp3Flx9PBbuakOwPIXxL8Y7mrZtAH9gl2u9e0q1fsppRUQlToRt81Lh4tZU0L8QZ/VUR98H6r9vWAhBbv2MuqqLXT5tERHJ+R5/6MdZrhTMVpVg2W48j/IdNavJ4ZJQNGHWLXPrWVQ8y2cXeOn5M0kFK2ZDq4zs6M09B5Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=infkKRYu; arc=none smtp.client-ip=117.135.210.3 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="infkKRYu" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=2q 9utxZ+9B0IUlUY+Tcgn2k/NPUl5FtceBfR+r6rKN8=; b=infkKRYuj0shgaFgMW I7sh7iw22hQIqi+YiNOMcecMOS0IbZZXa8SIy0xDTDFKCWohMsU9Uvcn+A73mUif 6b2qk/Y5KG8hmugVSfTDTp1GqR0XGEK6MLcU6B4YXvQJS6dCRbt8BNB+Z+LaPVdz 9oH6YHGPzRfTtPCQIPo9mFvh8= Received: from nec8-i7 (unknown []) by gzga-smtp-mtada-g0-0 (Coremail) with SMTP id _____wD3F7Vu7pdqogtpAA--.21222S5; Wed, 02 Sep 2026 17:37:55 +0800 (CST) From: chenyuan_fl@163.com To: bpf@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Alexei Starovoitov , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Martin KaFai Lau , Song Liu , Yonghong Song , Jiri Olsa , Emil Tsalapatis , Ihor Solodrai , Yuan Chen Subject: [PATCH bpf-next v5 3/3] bpf, arena: handle range_tree_set failures in alloc/free paths Date: Wed, 2 Sep 2026 17:37:40 +0800 Message-ID: <20260902093740.2338724-4-chenyuan_fl@163.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260902093740.2338724-1-chenyuan_fl@163.com> References: <20260902093740.2338724-1-chenyuan_fl@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: _____wD3F7Vu7pdqogtpAA--.21222S5 X-Coremail-Antispam: 1Uf129KBjvJXoW3Jw43ZF1UXrWkWr15AFW7XFb_yoWfWw4fpF s8G3s8trs5J3yI9r43Zr1v9r13KwsYqw4UGFWIka4rAry5Zr9IyFWxCF1UuFy5CrWkXr12 kr4jq34rKrWUZFDanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x07j9ku7UUUUU= X-CM-SenderInfo: xfkh05pxdqswro6rljoofrz/xtbDARRZF2qX7nQNAwAA3y Content-Type: text/plain; charset="utf-8" From: Yuan Chen arena_alloc_pages(), arena_free_pages() and arena_free_worker() now handle range_tree_set() errors. arena_free_pages() aborts the free on error, and arena_free_worker() moves range_tree_set() before PTE clearing so that a failed tree update leaves the PTEs intact instead of freeing pages that the arena free tree does not track. Also check the range_tree_set() return value in arena_alloc_pages()'s error path, which restores the unpopulated tail of a partially allocated range; log a warning instead of silently leaking the virtual range when the tree update fails. range_tree_set() is failure-atomic (it pre-allocates the node before touching the tree), so on -ENOMEM the range stays tracked as allocated and the pages remain mapped and accessible. A failed free is therefore retryable, and arena_map_free() reclaims any retained pages at map destruction; aborting the free avoids clearing PTEs for pages the arena free tree does not track. In arena_free_worker() a failed tree update used to leave the span in the drained list, where the second loop would still flush TLB entries, zap user VMAs, and free the span itself: the free request was dropped, user mappings were destroyed for a free that never happened, and the pages stayed mapped until map destruction. Keep failed spans on arena->free_spans instead and retry them on a later worker run; only spans whose PTE clearing actually ran are flushed, zapped, and released. The retry queues arena->free_irq while the map can concurrently be freed. arena_map_free() relied on irq_work_sync() + flush_work(), which miss an irq_work queued by the running worker between the two calls: the irq_work can fire after the arena is freed and its callback schedules free_work on freed memory. Set arena->dying under the arena spinlock before draining, so the worker stops requeueing, steal the orphaned spans (their pages are reclaimed by existing_page_cb()), and drain with flush_work() + irq_work_sync() + flush_work(). Setting @dying requires the arena spinlock. raw_res_spin_lock_irqsave() can fail (-EDEADLK on a proven deadlock cycle, -ETIMEDOUT after a long hold), and proceeding without the lock would race the worker. Retry a bounded number of times for a long but finite hold and do not retry -EDEADLK; on exhaustion leak the arena with a WARN carrying the error code rather than hang map free. Suggested-by: Emil Tsalapatis Signed-off-by: Yuan Chen --- kernel/bpf/arena.c | 95 ++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 87 insertions(+), 8 deletions(-) diff --git a/kernel/bpf/arena.c b/kernel/bpf/arena.c index 7b6847200b43..b0d1f0facfb2 100644 --- a/kernel/bpf/arena.c +++ b/kernel/bpf/arena.c @@ -5,6 +5,7 @@ #include #include #include +#include #include "linux/filter.h" #include #include @@ -67,6 +68,8 @@ struct bpf_arena { struct irq_work free_irq; struct work_struct free_work; struct llist_head free_spans; + /* set under spinlock during map free; stops the worker retry loop */ + bool dying; }; =20 static void arena_free_worker(struct work_struct *work); @@ -370,6 +373,9 @@ static int existing_page_cb(pte_t *ptep, unsigned long = addr, void *data) static void arena_map_free(struct bpf_map *map) { struct bpf_arena *arena =3D container_of(map, struct bpf_arena, map); + struct llist_node *list, *pos, *t; + unsigned long flags; + int ret, i; =20 /* * Check that user vma-s are not around when bpf map is freed. @@ -380,7 +386,40 @@ static void arena_map_free(struct bpf_map *map) if (WARN_ON_ONCE(!list_empty(&arena->vma_list))) return; =20 - /* Ensure no pending deferred frees */ + /* + * No fallback if this fails, so retry a few times for a long but + * finite hold; -EDEADLK can't be waited out. Cap the retries: + * leaking the arena is better than hanging map free. + */ + for (i =3D 0; i < 10; i++) { + ret =3D raw_res_spin_lock_irqsave(&arena->spinlock, flags); + if (!ret || ret =3D=3D -EDEADLK) + break; + msleep(1); + } + if (ret) { + WARN_ONCE(1, "bpf_arena: spinlock acquire failed %d\n", ret); + return; + } + /* + * Set @dying before draining: the worker checks it under this + * spinlock before requeueing, so a failed span is either stolen + * here or dropped by the worker. + */ + arena->dying =3D true; + list =3D llist_del_all(&arena->free_spans); + raw_res_spin_unlock_irqrestore(&arena->spinlock, flags); + + llist_for_each_safe(pos, t, list) + kfree_nolock(llist_entry(pos, struct arena_free_span, node)); + + /* + * flush_work() lets the running worker observe @dying so it stops + * requeueing; irq_work_sync() retires anything queued before that; + * the final flush_work() runs the instance which the retired + * irq_work's callback may have scheduled. + */ + flush_work(&arena->free_work); irq_work_sync(&arena->free_irq); flush_work(&arena->free_work); =20 @@ -766,7 +805,9 @@ static long arena_alloc_pages(struct bpf_arena *arena, = long uaddr, long page_cnt bpf_map_memcg_exit(old_memcg, new_memcg); return clear_lo32(arena->user_vm_start) + uaddr32; out: - range_tree_set(&arena->rt, pgoff + mapped, page_cnt - mapped); + if (range_tree_set(&arena->rt, pgoff + mapped, page_cnt - mapped)) + pr_warn_ratelimited("bpf_arena: leak range %ld+%ld on failed alloc\n", + pgoff + mapped, page_cnt - mapped); raw_res_spin_unlock_irqrestore(&arena->spinlock, flags); if (mapped) { flush_vmap_cache(kern_vm_start + uaddr32, mapped << PAGE_SHIFT); @@ -881,7 +922,20 @@ static void arena_free_pages(struct bpf_arena *arena, = long uaddr, long page_cnt, if (ret) goto defer; =20 - range_tree_set(&arena->rt, pgoff, page_cnt); + ret =3D range_tree_set(&arena->rt, pgoff, page_cnt); + if (ret) { + /* + * range_tree_set() is failure-atomic, so -ENOMEM leaves the + * range allocated and the pages mapped; abort the free rather + * than release pages the tree does not track. Nothing retries + * the free; the program can free the range again. + */ + pr_warn_ratelimited("bpf_arena: free of %lx+%ld failed\n", + uaddr, page_cnt); + raw_res_spin_unlock_irqrestore(&arena->spinlock, flags); + bpf_map_memcg_exit(old_memcg, new_memcg); + return; + } =20 init_llist_head(&free_pages); cdata.arena =3D arena; @@ -977,12 +1031,13 @@ static void arena_free_worker(struct work_struct *wo= rk) struct llist_node *list, *pos, *t; struct arena_free_span *s; u64 arena_vm_start, user_vm_start; - struct llist_head free_pages; + struct llist_head free_pages, cleared; struct clear_range_data cdata; struct page *page; unsigned long full_uaddr; long kaddr, page_cnt, pgoff; unsigned long flags; + bool retry =3D false; =20 if (raw_res_spin_lock_irqsave(&arena->spinlock, flags)) { schedule_work(work); @@ -992,28 +1047,52 @@ static void arena_free_worker(struct work_struct *wo= rk) bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg); =20 init_llist_head(&free_pages); + init_llist_head(&cleared); cdata.arena =3D arena; cdata.free_pages =3D &free_pages; arena_vm_start =3D bpf_arena_get_kern_vm_start(arena); user_vm_start =3D bpf_arena_get_user_vm_start(arena); =20 list =3D llist_del_all(&arena->free_spans); - llist_for_each(pos, list) { + llist_for_each_safe(pos, t, list) { s =3D llist_entry(pos, struct arena_free_span, node); page_cnt =3D s->page_cnt; kaddr =3D arena_vm_start + s->uaddr; pgoff =3D compute_pgoff(arena, s->uaddr); =20 + /* + * Set the range free before clearing PTEs, and requeue the + * span on failure: the PTEs stay intact and the free is + * retried later. Only spans moved to @cleared (PTE clearing + * actually ran) reach the flush/zap/release loop below. + */ + if (range_tree_set(&arena->rt, pgoff, page_cnt)) { + if (arena->dying) { + /* + * The map is being freed. PTEs stay intact + * and the pages are reclaimed by + * arena_map_free() via existing_page_cb(). + */ + kfree_nolock(s); + continue; + } + llist_add(&s->node, &arena->free_spans); + retry =3D true; + continue; + } + /* clear ptes and collect pages in free_pages llist */ apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT, apply_range_clear_cb, &cdata); - - range_tree_set(&arena->rt, pgoff, page_cnt); + llist_add(&s->node, &cleared); } raw_res_spin_unlock_irqrestore(&arena->spinlock, flags); =20 + if (retry) + irq_work_queue(&arena->free_irq); + /* Iterate the list again without holding spinlock to do the tlb flush an= d zap_pages */ - llist_for_each_safe(pos, t, list) { + llist_for_each_safe(pos, t, cleared.first) { s =3D llist_entry(pos, struct arena_free_span, node); page_cnt =3D s->page_cnt; full_uaddr =3D clear_lo32(user_vm_start) + s->uaddr; --=20 2.54.0