kernel/bpf/memalloc.c | 51 ++++++++++++++++++++++++------------------- 1 file changed, 29 insertions(+), 22 deletions(-)
From: Pu Lehui <pulehui@huawei.com>
Syzkaller repeatedly triggered UAF splats related to nodes in
waiting_for_gp_ttrace within the bpf memalloc:
BUG: KASAN: slab-use-after-free in llist_del_first+0x85/0x110 lib/llist.c:61
Read of size 8 at addr ffff8881572cd080 by task syz.4.470/5112
...
llist_del_first+0x85/0x110 lib/llist.c:61
alloc_bulk+0x193/0x460 kernel/bpf/memalloc.c:229
bpf_mem_refill+0x386/0x560 kernel/bpf/memalloc.c:436
Freed by task 14:
...
__free_rcu kernel/bpf/memalloc.c:281 [inline]
__free_rcu_tasks_trace+0x48/0xd0 kernel/bpf/memalloc.c:291
rcu_tasks_invoke_cbs+0x1ec/0x3e0 kernel/rcu/tasks.h:571
rcu_tasks_one_gp+0x13d/0x220 kernel/rcu/tasks.h:621
rcu_tasks_kthread+0xf3/0x120 kernel/rcu/tasks.h:651
Initially, we suspected that alloc_bulk() lacked RCU Tasks Trace
protection when accessing waiting_for_gp_ttrace. However, explicitly
adding rcu_read_lock_trace() did not help.
This is expected because, as noted in commit 57b23c0f612d ("bpf: Retire
rcu_trace_implies_rcu_gp()"), an RCU Tasks Trace GP currently implies
(and will continue to imply in the future) a normal RCU GP. Since
alloc_bulk() runs in an RCU read-side CS (!PREEMPT_RT runs in IRQ
context, PREEMPT_RT runs with guard(rcu)), an RCU Tasks Trace GP cannot
complete while alloc_bulk() is accessing the list.
Instead, the UAF occurs after the GP expires: when the __free_rcu()
callback runs, there is no synchronization protecting llist_del_all()
against concurrent alloc_bulk() operates on waiting_for_gp_ttrace,
leading to the race condition below:
CPU0 CPU1
__free_rcu (RCU Tasks Trace callback)
alloc_bulk
llist_del_first(&c->waiting_for_gp_ttrace)
entry = smp_load_acquire(&head->first);
do {
if (entry == NULL)
return NULL;
free_all(llist_del_all(&c->waiting_for_gp_ttrace))
llist_for_each_safe(pos, t, llnode)
free_one(pos);
next = READ_ONCE(entry->next); <-- trigger UAF
} while (!try_cmpxchg(&head->first, &entry, next));
In addition, there is also a theoretical race condition on the
free_by_rcu_ttrace list. This race requires two preconditions: an
in-flight Tasks Trace GP keeping c->call_rcu_ttrace_in_progress == 1,
and concurrent cross-CPU frees repopulating c->free_by_rcu_ttrace with
new nodes. Under these conditions, the following scenario triggers UAF:
// CPU0
// irq work is still busy (on PREEMPT_RT)
alloc_bulk()
llist_del_first(&c->free_by_rcu_ttrace)
entry = smp_load_acquire(&head->first);
do {
if (entry == NULL)
return NULL;
// CPU1
bpf_mem_alloc_destroy()
WRITE_ONCE(c->draining, true)
// wait for CPU0
irq_work_sync()
// CPU2
do_call_rcu_ttrace(tgt(CPU0))
if (c->draining) {
llist_del_all(&c->free_by_rcu_ttrace)
free_all()
}
// CPU0 continue
next = READ_ONCE(entry->next); <-- trigger UAF
while (!try_cmpxchg(&head->first, &entry, next));
Fix this by introducing a raw spinlock to synchronize the concurrent
consumption on waiting_for_gp_ttrace and free_by_rcu_ttrace.
Fixes: 04fabf00b4d3 ("bpf: Allow reuse from waiting_for_gp_ttrace list.")
Suggested-by: Alexei Starovoitov <ast@kernel.org>
Suggested-by: Hou Tao <houtao1@huawei.com>
Signed-off-by: Pu Lehui <pulehui@huawei.com>
---
v3:
- Fix concurrent issue also for free_by_rcu_ttrace. (bpfci and Hou Tao)
- Use scoped_guard. (Leon)
v2: https://lore.kernel.org/bpf/20260827084013.1062816-1-pulehui@huaweicloud.com
- Use raw spinlock to fix concurrent alloc_bulk and __free_rcu on
waiting_for_gp_ttrace after GP. (Hou Tao)
v1: https://lore.kernel.org/bpf/20260826103615.932094-1-pulehui@huaweicloud.com
kernel/bpf/memalloc.c | 51 ++++++++++++++++++++++++-------------------
1 file changed, 29 insertions(+), 22 deletions(-)
diff --git a/kernel/bpf/memalloc.c b/kernel/bpf/memalloc.c
index e9662db7198f..5927943c765a 100644
--- a/kernel/bpf/memalloc.c
+++ b/kernel/bpf/memalloc.c
@@ -119,6 +119,7 @@ struct bpf_mem_cache {
struct llist_head waiting_for_gp_ttrace;
struct rcu_head rcu_ttrace;
atomic_t call_rcu_ttrace_in_progress;
+ raw_spinlock_t lock;
};
struct bpf_mem_caches {
@@ -214,25 +215,25 @@ static void alloc_bulk(struct bpf_mem_cache *c, int cnt, int node, bool atomic)
gfp = __GFP_NOWARN | __GFP_ACCOUNT;
gfp |= atomic ? GFP_NOWAIT : GFP_KERNEL;
- for (i = 0; i < cnt; i++) {
- /*
- * For every 'c' llist_del_first(&c->free_by_rcu_ttrace); is
- * done only by one CPU == current CPU. Other CPUs might
- * llist_add() and llist_del_all() in parallel.
- */
- obj = llist_del_first(&c->free_by_rcu_ttrace);
- if (!obj)
- break;
- add_obj_to_free_list(c, obj);
- }
- if (i >= cnt)
- return;
+ scoped_guard(raw_spinlock_irqsave, &c->lock) {
+ for (i = 0; i < cnt; i++) {
+ /*
+ * For every 'c' llist_del_first(&c->free_by_rcu_ttrace); is
+ * done only by one CPU == current CPU. Other CPUs might
+ * llist_add() and llist_del_all() in parallel.
+ */
+ obj = llist_del_first(&c->free_by_rcu_ttrace);
+ if (!obj)
+ break;
+ add_obj_to_free_list(c, obj);
+ }
- for (; i < cnt; i++) {
- obj = llist_del_first(&c->waiting_for_gp_ttrace);
- if (!obj)
- break;
- add_obj_to_free_list(c, obj);
+ for (; i < cnt; i++) {
+ obj = llist_del_first(&c->waiting_for_gp_ttrace);
+ if (!obj)
+ break;
+ add_obj_to_free_list(c, obj);
+ }
}
if (i >= cnt)
return;
@@ -279,8 +280,12 @@ static int free_all(struct bpf_mem_cache *c, struct llist_node *llnode, bool per
static void __free_rcu(struct rcu_head *head)
{
struct bpf_mem_cache *c = container_of(head, struct bpf_mem_cache, rcu_ttrace);
+ struct llist_node *llnode;
+
+ scoped_guard(raw_spinlock_irqsave, &c->lock)
+ llnode = llist_del_all(&c->waiting_for_gp_ttrace);
- free_all(c, llist_del_all(&c->waiting_for_gp_ttrace), !!c->percpu_size);
+ free_all(c, llnode, !!c->percpu_size);
atomic_set(&c->call_rcu_ttrace_in_progress, 0);
}
@@ -300,7 +305,8 @@ static void do_call_rcu_ttrace(struct bpf_mem_cache *c)
if (atomic_xchg(&c->call_rcu_ttrace_in_progress, 1)) {
if (unlikely(READ_ONCE(c->draining))) {
- llnode = llist_del_all(&c->free_by_rcu_ttrace);
+ scoped_guard(raw_spinlock_irqsave, &c->lock)
+ llnode = llist_del_all(&c->free_by_rcu_ttrace);
free_all(c, llnode, !!c->percpu_size);
}
return;
@@ -535,6 +541,7 @@ int bpf_mem_alloc_init(struct bpf_mem_alloc *ma, int size, bool percpu)
c->objcg = objcg;
c->percpu_size = percpu_size;
c->tgt = c;
+ raw_spin_lock_init(&c->lock);
init_refill_work(c);
prefill_mem_cache(c, cpu);
}
@@ -557,7 +564,7 @@ int bpf_mem_alloc_init(struct bpf_mem_alloc *ma, int size, bool percpu)
c->objcg = objcg;
c->percpu_size = percpu_size;
c->tgt = c;
-
+ raw_spin_lock_init(&c->lock);
init_refill_work(c);
prefill_mem_cache(c, cpu);
}
@@ -609,7 +616,7 @@ int bpf_mem_alloc_percpu_unit_init(struct bpf_mem_alloc *ma, int size)
c->objcg = objcg;
c->percpu_size = percpu_size;
c->tgt = c;
-
+ raw_spin_lock_init(&c->lock);
init_refill_work(c);
prefill_mem_cache(c, cpu);
}
--
2.34.1> This is expected because, as noted in commit 57b23c0f612d ("bpf: Retire
> rcu_trace_implies_rcu_gp()"), an RCU Tasks Trace GP currently implies
> (and will continue to imply in the future) a normal RCU GP. Since
> alloc_bulk() runs in an RCU read-side CS (!PREEMPT_RT runs in IRQ
> context, PREEMPT_RT runs with guard(rcu)), an RCU Tasks Trace GP cannot
> complete while alloc_bulk() is accessing the list.
Is this claim about the RCU Tasks Trace GP accurate? Looking at the SRCU
implementation, synchronize_rcu() sits in the middle of the SRCU scan
(kernel/rcu/srcutree.c srcu_readers_active_idx_check()), not at the end.
An RCU GP is only required to wait for read-side critical sections that
were already in progress when the GP started. Once a CPU reports its
quiescent state, a new hardirq handler on that CPU does not delay the GP.
So an alloc_bulk() that starts after the synchronize_rcu() call cannot
hold off the RCU Tasks Trace GP at all.
Even if the GP could not complete during alloc_bulk(), that would not
prevent the callback from running during alloc_bulk(). Callbacks are
invoked asynchronously after the GP ends (srcu_invoke_callbacks workqueue
in current kernels, or rcu_tasks_invoke_cbs in the 6.6 kernel that
produced the splat).
The race diagram three paragraphs later shows __free_rcu executing
concurrently with alloc_bulk's llist_del_first, which would not be
possible if the statement above were true.
Would it be more accurate to say that RCU read-side protection is not the
missing ingredient here, but rather mutual exclusion between the two
llist consumers?
> diff --git a/kernel/bpf/memalloc.c b/kernel/bpf/memalloc.c
> index e9662db7198f..5927943c765a 100644
> --- a/kernel/bpf/memalloc.c
> +++ b/kernel/bpf/memalloc.c
[ ... ]
> @@ -214,25 +215,25 @@ static void alloc_bulk(struct bpf_mem_cache *c, int cnt, int node, bool atomic)
> gfp = __GFP_NOWARN | __GFP_ACCOUNT;
> gfp |= atomic ? GFP_NOWAIT : GFP_KERNEL;
>
> - for (i = 0; i < cnt; i++) {
> - /*
> - * For every 'c' llist_del_first(&c->free_by_rcu_ttrace); is
> - * done only by one CPU == current CPU. Other CPUs might
> - * llist_add() and llist_del_all() in parallel.
> - */
> - obj = llist_del_first(&c->free_by_rcu_ttrace);
> - if (!obj)
> - break;
> - add_obj_to_free_list(c, obj);
> - }
> - if (i >= cnt)
> - return;
> + scoped_guard(raw_spinlock_irqsave, &c->lock) {
> + for (i = 0; i < cnt; i++) {
> + /*
> + * For every 'c' llist_del_first(&c->free_by_rcu_ttrace); is
> + * done only by one CPU == current CPU. Other CPUs might
> + * llist_add() and llist_del_all() in parallel.
> + */
> + obj = llist_del_first(&c->free_by_rcu_ttrace);
> + if (!obj)
> + break;
> + add_obj_to_free_list(c, obj);
> + }
>
> - for (; i < cnt; i++) {
> - obj = llist_del_first(&c->waiting_for_gp_ttrace);
> - if (!obj)
> - break;
> - add_obj_to_free_list(c, obj);
> + for (; i < cnt; i++) {
> + obj = llist_del_first(&c->waiting_for_gp_ttrace);
> + if (!obj)
> + break;
> + add_obj_to_free_list(c, obj);
> + }
> }
> if (i >= cnt)
> return;
The comment moved inside the new critical section still states the
lockless rationale:
/*
* For every 'c' llist_del_first(&c->free_by_rcu_ttrace); is
* done only by one CPU == current CPU. Other CPUs might
* llist_add() and llist_del_all() in parallel.
*/
The last sentence asserts that concurrent llist_del_all() is fine.
According to include/linux/llist.h, the del_first x del_all combination
requires a lock, and the commit message's first race diagram shows exactly
this combination producing the KASAN splat.
Should this comment be updated to reflect that c->lock now serializes
against __free_rcu() and the draining branch of do_call_rcu_ttrace(),
rather than asserting that lockless operation is safe?
The related comments in enque_to_free() ("Nothing races to add to
free_by_rcu_ttrace list.") and drain_mem_cache() ("Except for
waiting_for_gp_ttrace list, there are no concurrent operations on these
lists") may also need updates, though they are outside this diff.
---
AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
CI run summary: https://github.com/kernel-patches/bpf/actions/runs/33525009657
On 2026/9/1 23:55, bot+bpf-ci@kernel.org wrote:
>> This is expected because, as noted in commit 57b23c0f612d ("bpf: Retire
>> rcu_trace_implies_rcu_gp()"), an RCU Tasks Trace GP currently implies
>> (and will continue to imply in the future) a normal RCU GP. Since
>> alloc_bulk() runs in an RCU read-side CS (!PREEMPT_RT runs in IRQ
>> context, PREEMPT_RT runs with guard(rcu)), an RCU Tasks Trace GP cannot
>> complete while alloc_bulk() is accessing the list.
>
> Is this claim about the RCU Tasks Trace GP accurate? Looking at the SRCU
> implementation, synchronize_rcu() sits in the middle of the SRCU scan
> (kernel/rcu/srcutree.c srcu_readers_active_idx_check()), not at the end.
>
> An RCU GP is only required to wait for read-side critical sections that
> were already in progress when the GP started. Once a CPU reports its
> quiescent state, a new hardirq handler on that CPU does not delay the GP.
> So an alloc_bulk() that starts after the synchronize_rcu() call cannot
> hold off the RCU Tasks Trace GP at all.
>
> Even if the GP could not complete during alloc_bulk(), that would not
> prevent the callback from running during alloc_bulk(). Callbacks are
> invoked asynchronously after the GP ends (srcu_invoke_callbacks workqueue
> in current kernels, or rcu_tasks_invoke_cbs in the 6.6 kernel that
> produced the splat).
>
> The race diagram three paragraphs later shows __free_rcu executing
> concurrently with alloc_bulk's llist_del_first, which would not be
> possible if the statement above were true.
>
> Would it be more accurate to say that RCU read-side protection is not the
> missing ingredient here, but rather mutual exclusion between the two
> llist consumers?
Regarding the commit message: there is no real contradiction. Those
paragraphs were merely explaining our initial troubleshooting hypothesis
and addressing earlier discussions where adding rcu_read_lock_trace()
was suggested [0], rather than claiming that RCU Tasks Trace prevents
the race.
However, to keep the commit log focused and avoid any confusion, I will
simply drop that part in the next
[0]
https://lore.kernel.org/bpf/3f72c4e7-340f-4374-9ebe-f9bffd08c755@paulmck-laptop/
>
>> diff --git a/kernel/bpf/memalloc.c b/kernel/bpf/memalloc.c
>> index e9662db7198f..5927943c765a 100644
>> --- a/kernel/bpf/memalloc.c
>> +++ b/kernel/bpf/memalloc.c
>
> [ ... ]
>
>> @@ -214,25 +215,25 @@ static void alloc_bulk(struct bpf_mem_cache *c, int cnt, int node, bool atomic)
>> gfp = __GFP_NOWARN | __GFP_ACCOUNT;
>> gfp |= atomic ? GFP_NOWAIT : GFP_KERNEL;
>>
>> - for (i = 0; i < cnt; i++) {
>> - /*
>> - * For every 'c' llist_del_first(&c->free_by_rcu_ttrace); is
>> - * done only by one CPU == current CPU. Other CPUs might
>> - * llist_add() and llist_del_all() in parallel.
>> - */
>> - obj = llist_del_first(&c->free_by_rcu_ttrace);
>> - if (!obj)
>> - break;
>> - add_obj_to_free_list(c, obj);
>> - }
>> - if (i >= cnt)
>> - return;
>> + scoped_guard(raw_spinlock_irqsave, &c->lock) {
>> + for (i = 0; i < cnt; i++) {
>> + /*
>> + * For every 'c' llist_del_first(&c->free_by_rcu_ttrace); is
>> + * done only by one CPU == current CPU. Other CPUs might
>> + * llist_add() and llist_del_all() in parallel.
>> + */
>> + obj = llist_del_first(&c->free_by_rcu_ttrace);
>> + if (!obj)
>> + break;
>> + add_obj_to_free_list(c, obj);
>> + }
>>
>> - for (; i < cnt; i++) {
>> - obj = llist_del_first(&c->waiting_for_gp_ttrace);
>> - if (!obj)
>> - break;
>> - add_obj_to_free_list(c, obj);
>> + for (; i < cnt; i++) {
>> + obj = llist_del_first(&c->waiting_for_gp_ttrace);
>> + if (!obj)
>> + break;
>> + add_obj_to_free_list(c, obj);
>> + }
>> }
>> if (i >= cnt)
>> return;
>
> The comment moved inside the new critical section still states the
> lockless rationale:
>
> /*
> * For every 'c' llist_del_first(&c->free_by_rcu_ttrace); is
> * done only by one CPU == current CPU. Other CPUs might
> * llist_add() and llist_del_all() in parallel.
> */
>
> The last sentence asserts that concurrent llist_del_all() is fine.
> According to include/linux/llist.h, the del_first x del_all combination
> requires a lock, and the commit message's first race diagram shows exactly
> this combination producing the KASAN splat.
>
> Should this comment be updated to reflect that c->lock now serializes
> against __free_rcu() and the draining branch of do_call_rcu_ttrace(),
> rather than asserting that lockless operation is safe?
make sense to me.
>
> The related comments in enque_to_free() ("Nothing races to add to
The comments in enque_to_free looks fine, no need to modify.
> free_by_rcu_ttrace list.") and drain_mem_cache() ("Except for
> waiting_for_gp_ttrace list, there are no concurrent operations on these
> lists") may also need updates, though they are outside this diff.
The comments in drain_mem_cache are slightly incorrect, but the impact
is not significant. In addition, they are outside this diff.
>
>
> ---
> AI reviewed your patch. Please fix the bug or email reply why it's not a bug.
> See: https://github.com/kernel-patches/vmtest/blob/master/ci/claude/README.md
>
> CI run summary: https://github.com/kernel-patches/bpf/actions/runs/33525009657
© 2016 - 2026 Red Hat, Inc.