From nobody Sat Sep 26 13:09:08 2026 Received: from m16.mail.163.com (m16.mail.163.com [220.197.31.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 153933815D9; Tue, 1 Sep 2026 06:30:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=220.197.31.2 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788244229; cv=none; b=kiDrZDfeBCtXrwxdU6vCxA3neXWw27NKGPdJUNlDqeRnqv1Fy5rhnoMs5BuloIIUy0Lhs9NSaaWlX4R0G1rbZA9oh4S6U4C9nqNuBztH26hQJuYvfmhHq6tSX3X5sicd5E63SG4pznF0jCkRBI5G9KUDjW3egSI3PM6/DIroj8Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788244229; c=relaxed/simple; bh=S4eGY462Yepo4JD995kGaKRbsAUhv8M/wPLctXTvQJk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=MSCnIk4U7PdpLjCRF1WFmWj7GIpwvI1+DNrhlr6H5MAdEWo6cYbil7WlEbsDiAa2BJRn3ku8Oh5joFFtomGLLcL1P7yc+KPkDCkdymjLYSouIAwI8SpeJS7oTU0G8p9S/UH/as0nFHPQDdGoKfzxcKGTo/J0nqJYt/fDEnxheqo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=lGFgQvJc; arc=none smtp.client-ip=220.197.31.2 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="lGFgQvJc" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=1r TN9M+Bqw98xM48PiM/4jWqJzlG1hXlHCckkiXqcmU=; b=lGFgQvJcnobvoVXrO+ 6nWKbDbApHmMBNtI5g1KzvmsvzqK2I9+ESikHP778QlRwGLmyTRmkGrSxL22vxl0 /rKqnIGCgmPyhQfHgTvIhY+4LkzuiLlxhAnWJNNi2DPQECAZj53oOz7CicmEL9b8 HZkGO/ZHP2IzbNq8T1ReJrBHo= Received: from nec8-i7 (unknown []) by gzsmtp1 (Coremail) with SMTP id PCgvCgD3n5+mcJZqA6mBNA--.25535S3; Tue, 01 Sep 2026 14:28:59 +0800 (CST) From: chenyuan_fl@163.com To: bpf@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Mykyta Yatsenko , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Martin KaFai Lau , Emil Tsalapatis , Ihor Solodrai , Shuah Khan , Nuoqi Gui , Yuan Chen Subject: [PATCH bpf-next v3 1/4] bpf: Cancel special fields in resizable hashtab on recycle Date: Tue, 1 Sep 2026 14:28:42 +0800 Message-ID: <20260901062845.1379760-2-chenyuan_fl@163.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260901062845.1379760-1-chenyuan_fl@163.com> References: <0560a24d-2cf9-4e5b-aa61-580af1e56de1@gmail.com> <20260901062845.1379760-1-chenyuan_fl@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: PCgvCgD3n5+mcJZqA6mBNA--.25535S3 X-Coremail-Antispam: 1Uf129KBjvJXoWxCr1fZF13AF1UGw1rtF1rtFb_yoW7Gr17pF Z3Wr1akr1kJrn0qFZ0yw4vk3yrX3s5t3yakFZ8GryFvw1rXF97Jr1rJayxuFyYyF1vvrsa qFn2vrWava15C37anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x07jrMa8UUUUU= X-CM-SenderInfo: xfkh05pxdqswro6rljoofrz/xtbDAQyHRWqWcKx2DwAA3W Content-Type: text/plain; charset="utf-8" From: Yuan Chen rhtab_delete_elem() and rhtab_map_update_existing() eagerly call bpf_obj_free_fields() when an element is deleted or its value is replaced. This runs kptr destructors in the caller's execution context, which is unsafe for BPF programs running in NMI context (e.g. perf_event programs attached to hardware PMU overflows): referenced kptr destructors may take locks or otherwise cannot run in NMI. Commit a3a81d247651 ("bpf: Cancel special fields on map value recycle") switched the hash map and array recycle paths to bpf_obj_cancel_fields(), which only cancels NMI-safe fields (timer, workqueue, task_work), but it missed the resizable hashtab. rhtab_map_update_existing() even documents the intended "cancel" semantics while still calling bpf_obj_free_fields(). Fix the resizable hashtab the same way: call bpf_obj_cancel_fields() on delete and in-place update, matching htab. Referenced kptrs stay attached to the recycled element and are destroyed by rhtab_mem_dtor() once the element is eventually freed, keeping the reference accounting balanced. No special-field initialization is added to the element alloc path: fresh elements come zeroed from the bpf mem allocator, recycled elements already had their timer/workqueue/task_work slots reset by bpf_obj_cancel_fields(), and check_and_init_map_value() would zero the kptr slot of a recycled element, dropping the reference without releasing it. Verified with a selftest: a perf_event (NMI) program overwrites a rhtab element that holds a referenced task kptr, and a second phase deletes and re-inserts the element to exercise the recycle path. Before the patch the NMI update eagerly released the kptr and the recycle path zeroed the inherited slot; after the patch the kptr is inherited on both paths and the probe observes it non-NULL. Fixes: a3a81d247651 ("bpf: Cancel special fields on map value recycle") Suggested-by: Mykyta Yatsenko Signed-off-by: Yuan Chen --- kernel/bpf/hashtab.c | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index d40cb5dd446c..aaedda3730f3 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -2864,16 +2864,6 @@ static int rhtab_map_alloc_check(union bpf_attr *att= r) return htab_map_alloc_check(attr); } =20 -static void rhtab_check_and_free_fields(struct bpf_rhtab *rhtab, - struct rhtab_elem *elem) -{ - if (IS_ERR_OR_NULL(rhtab->map.record)) - return; - - bpf_obj_free_fields(rhtab->map.record, - rhtab_elem_value(elem, rhtab->map.key_size)); -} - static void rhtab_mem_dtor(void *obj, void *ctx) { struct htab_btf_record *hrec =3D ctx; @@ -2963,8 +2953,9 @@ static int rhtab_delete_elem(struct bpf_rhtab *rhtab,= struct rhtab_elem *elem, v rhtab_read_elem_value(&rhtab->map, copy, elem, flags); check_and_init_map_value(&rhtab->map, copy); } - /* Release internal structs: kptr, bpf_timer, task_work, wq */ - rhtab_check_and_free_fields(rhtab, elem); + /* Cancel NMI-safe fields; full destruction happens in rhtab_mem_dtor */ + bpf_obj_cancel_fields(&rhtab->map, + rhtab_elem_value(elem, rhtab->map.key_size)); bpf_mem_cache_free_rcu(&rhtab->ma, elem); return 0; } @@ -3022,10 +3013,12 @@ static long rhtab_map_update_existing(struct bpf_ma= p *map, struct rhtab_elem *el * BPF_F_LOCK, matching arraymap semantics. * * copy_map_value() skips special-field offsets, so old timers/ - * kptrs/etc. still sit in the slot. Cancel them after the copy - * to match arraymap's update semantics. + * kptrs/etc. still sit in the slot. Cancel the NMI-safe ones after + * the copy to match arraymap's update semantics; referenced kptrs + * stay attached and are destroyed by rhtab_mem_dtor(). */ - rhtab_check_and_free_fields(rhtab, elem); + bpf_obj_cancel_fields(&rhtab->map, + rhtab_elem_value(elem, rhtab->map.key_size)); return 0; } =20 @@ -3066,7 +3059,14 @@ static long rhtab_map_update_elem(struct bpf_map *ma= p, void *key, void *value, u =20 memcpy(elem->data, key, map->key_size); copy_map_value(map, rhtab_elem_value(elem, map->key_size), value); - check_and_init_map_value(map, rhtab_elem_value(elem, map->key_size)); + /* + * No explicit special-field initialization, matching the hash map's + * non-prealloc path: fresh elements come zeroed from the bpf mem + * allocator, and recycled elements had their timer/workqueue/task_work + * slots reset by bpf_obj_cancel_fields() on delete. kptr slots are + * left untouched so a recycled element keeps owning its reference + * until rhtab_mem_dtor() releases it. + */ =20 /* Prevent deadlock for NMI programs attempting to take bucket lock */ bpf_disable_instrumentation(); --=20 2.54.0 From nobody Sat Sep 26 13:09:08 2026 Received: from m16.mail.163.com (m16.mail.163.com [220.197.31.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 826DE3812C7; Tue, 1 Sep 2026 06:30:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=220.197.31.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788244225; cv=none; b=MB0KnCbeAfzXHHkoa9ST5kNIBaLUWFGBN4xxX8d+VpKGcJc9imtv3Es915j5JCj1d1sTI6Ma0nLEatME1KojnOp0xc+io0R/sG9C+9gh3aZ5jr8egzmISd1TFIaoIktnAxdwIg+iGOfdijf0w0ESHRNh12IC1PzgU9NwPdLd8YI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788244225; c=relaxed/simple; bh=+iYCkLJBD1CTXM4wo0FayQCQ8EHm1fbEwSf2jgfMvqU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pHGqLqB+UXWev5hQyAq2uNqWywzqJ1CfYIrsjm0gsHe9rjGt9zVKuJF9CRy+/KykZ+Isls4Q+2VcMrQN5RuTUXAXjqXcI/lu+Hrohhxo4BL83Qwkg1gtEUj0+9OPXfJF400mTCm6RwTPHv2qfB4iy+Y4B6Oi8fCBRFcGtuyasyQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=DfYyUX8X; arc=none smtp.client-ip=220.197.31.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="DfYyUX8X" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=XR UR5zZRZ7TV9XYlyi/hU2YGzIrVLJbcihPkGSjgxmk=; b=DfYyUX8XqcSZc4CgdX 7pkQMtc1S6Y0CS+wnoOG9tAqV9Kz5lvisF2jngC1Oog1dlmjbSweRw58y9bA7T7t TsLY505lqnTTvXomHcrvN3mE2W2aR1lEBpTk/wKg1kbLzCyGEUba181vpiacQ7Fd pMqm1V08FGNNhO/lb4lDbB2rY= Received: from nec8-i7 (unknown []) by gzsmtp1 (Coremail) with SMTP id PCgvCgD3n5+mcJZqA6mBNA--.25535S4; Tue, 01 Sep 2026 14:29:01 +0800 (CST) From: chenyuan_fl@163.com To: bpf@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Mykyta Yatsenko , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Martin KaFai Lau , Emil Tsalapatis , Ihor Solodrai , Shuah Khan , Nuoqi Gui , Yuan Chen Subject: [PATCH bpf-next v3 2/4] bpf: Fix use-after-free of program BTF in mem-alloc destructor Date: Tue, 1 Sep 2026 14:28:43 +0800 Message-ID: <20260901062845.1379760-3-chenyuan_fl@163.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260901062845.1379760-1-chenyuan_fl@163.com> References: <0560a24d-2cf9-4e5b-aa61-580af1e56de1@gmail.com> <20260901062845.1379760-1-chenyuan_fl@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: PCgvCgD3n5+mcJZqA6mBNA--.25535S4 X-Coremail-Antispam: 1Uf129KBjvJXoWxWw1UGw15Jw4rXF17Zr15twb_yoW5KF1rpF 4xCFW3Kr4ktFn2kFZ3Xw4DAFy5Za97ZF15uFy3Ww1Y93yFqr1DAa1IgFWrZFy5CrWkKwnY 9r1jga9xGrWjyrDanT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDUYxBIdaVFxhVjvjDU0xZFpf9x07jbnYwUUUUU= X-CM-SenderInfo: xfkh05pxdqswro6rljoofrz/xtbC5Q2HRWqWcK2xmAAA3i Content-Type: text/plain; charset="utf-8" From: Yuan Chen bpf_ma_set_dtor() duplicates the map's btf_record for the bpf_mem_alloc destructor. btf_record_dup() only borrows the BTF references held by the fields: kptrs and list_head/rb_root fields point at the program BTF, whose lifetime is independent of the map. The duplicated record is used from the deferred mem-alloc destructor workqueue, which can run after the program BTF is gone (bpf_map_free() drops the map's reference, and the RCU callback may run first). Reading the borrowed descriptors there is a use-after-free, detected by KASAN as "slab-use-after-free in btf_is_kernel". Keep a reference on the borrowed program BTFs for the lifetime of the duplicated record. The record is freed from a preemptible worker, so the last btf_put() (which only schedules RCU destruction) does not make the field descriptors safe to read; snapshot the borrowed BTFs, free the record, then drop the references. The rhtab kptr selftests exercise this path on every map teardown and triggered the bug under KASAN; with this fix they pass cleanly. Fixes: 1df97a7453ee ("bpf: Register dtor for freeing special fields") Signed-off-by: Yuan Chen --- kernel/bpf/hashtab.c | 66 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index aaedda3730f3..30bcc573772e 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -493,11 +493,76 @@ static void htab_pcpu_mem_dtor(void *obj, void *ctx) bpf_obj_free_fields(hrec->record, per_cpu_ptr(pptr, cpu)); } =20 +/* + * The duplicated record borrows program BTFs, but is freed from a deferred + * workqueue that may run after the program BTF is gone. Hold a reference = for + * the record's lifetime and snapshot the borrowed BTFs, since + * btf_record_free() frees the record. + */ + +/* Program BTF borrowed by the field, or NULL. */ +static struct btf *htab_field_borrowed_btf(const struct btf_field *field) +{ + struct btf *btf =3D NULL; + + switch (field->type) { + case BPF_KPTR_UNREF: + case BPF_KPTR_REF: + case BPF_KPTR_PERCPU: + case BPF_UPTR: + btf =3D field->kptr.btf; + break; + case BPF_LIST_HEAD: + case BPF_RB_ROOT: + btf =3D field->graph_root.btf; + break; + default: + break; + } + + if (btf && !btf_is_kernel(btf)) + return btf; + return NULL; +} + +/* Snapshot the program BTFs @rec borrows into @btfs; returns their count.= */ +static int htab_record_prog_btfs_snapshot(struct btf_record *rec, + struct btf **btfs) +{ + int i, n =3D 0; + + if (IS_ERR_OR_NULL(rec)) + return 0; + + for (i =3D 0; i < rec->cnt; i++) { + struct btf *btf =3D htab_field_borrowed_btf(&rec->fields[i]); + + if (btf) + btfs[n++] =3D btf; + } + return n; +} + +static void htab_record_prog_btf_get(struct btf_record *rec) +{ + struct btf *btfs[BTF_FIELDS_MAX]; + int i, n; + + n =3D htab_record_prog_btfs_snapshot(rec, btfs); + for (i =3D 0; i < n; i++) + btf_get(btfs[i]); +} + static void htab_dtor_ctx_free(void *ctx) { struct htab_btf_record *hrec =3D ctx; + struct btf *btfs[BTF_FIELDS_MAX]; + int i, n; =20 + n =3D htab_record_prog_btfs_snapshot(hrec->record, btfs); btf_record_free(hrec->record); + for (i =3D 0; i < n; i++) + btf_put(btfs[i]); kfree(ctx); } =20 @@ -521,6 +586,7 @@ static int bpf_ma_set_dtor(struct bpf_map *map, struct = bpf_mem_alloc *ma, kfree(hrec); return err; } + htab_record_prog_btf_get(hrec->record); bpf_mem_alloc_set_dtor(ma, dtor, htab_dtor_ctx_free, hrec); return 0; } --=20 2.54.0 From nobody Sat Sep 26 13:09:08 2026 Received: from m16.mail.163.com (m16.mail.163.com [220.197.31.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 939C32F616B; Tue, 1 Sep 2026 06:30:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=220.197.31.3 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788244223; cv=none; b=Ctx1oT40XtaEycXbK+UWk2LqPTs9wn3J6U4hvEL5LDKBBQ2GThfWRWkpHQKinYCDcMK6cubay1WepsOSLiAA/zCgT5dO84bxc8lUm49Mn1u39mUYIyJsrvt0xxjr//+ZffDM7BpatcPFvCw4UHa6tfNJXOpbS3EU6oCAl4U8K4k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788244223; c=relaxed/simple; bh=k/u9HlMctB2v28FpIueJbxVyDxdLj/IQFRyP4kvf0fc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bxL50PFBmK2eN7MA9nOf91MJKNqGshh4j3Sm276Bf4vsBTccebQ6whOL6+E1E+VEcrNs/X9aegWIMWuCCQl8sDBa/daqWmZu9TpIMWxX6xXZftYt/983ZSM8xNx90duLJ5p/wIXn2smrusaxvDH+XKcDnjajOMtrA0KIesD6Cfk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=gK62Wvtj; arc=none smtp.client-ip=220.197.31.3 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="gK62Wvtj" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=lv bxUgIVISfLN5BI0oqpLaNIawZZ/6r+osaCkDJVSMo=; b=gK62WvtjyLPonYLRJ8 3Z+iarOKtuWwNKjCkWFoolb5JS31LWJYgVtZ50IUH6VMM/oQFOycdiCJvnl7uuXZ hgzNfzWKYiWMGoouLvCCobwpK+fjjqedz5hbh9auRMQBobr1snqoyC27AJWmKGc/ IaidITkowUvg0EFqVYf5svGh4= Received: from nec8-i7 (unknown []) by gzsmtp1 (Coremail) with SMTP id PCgvCgD3n5+mcJZqA6mBNA--.25535S5; Tue, 01 Sep 2026 14:29:02 +0800 (CST) From: chenyuan_fl@163.com To: bpf@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Mykyta Yatsenko , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Martin KaFai Lau , Emil Tsalapatis , Ihor Solodrai , Shuah Khan , Nuoqi Gui , Yuan Chen Subject: [PATCH bpf-next v3 3/4] selftests/bpf: Test rhtab kptr recycle from NMI context Date: Tue, 1 Sep 2026 14:28:44 +0800 Message-ID: <20260901062845.1379760-4-chenyuan_fl@163.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260901062845.1379760-1-chenyuan_fl@163.com> References: <0560a24d-2cf9-4e5b-aa61-580af1e56de1@gmail.com> <20260901062845.1379760-1-chenyuan_fl@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: PCgvCgD3n5+mcJZqA6mBNA--.25535S5 X-Coremail-Antispam: 1Uf129KBjvAXoW3CryDKw1xKFyrGF1ktrykGrg_yoW8JryUAo Z3Way5Zw18Gw1DKr95uFn7Gr1rW3yvga4fXw4rK3y5Xa48tay7CF98Cw4xX3W29F10qrWU ua4qv34fZr4fJFn3n29KB7ZKAUJUUUU8529EdanIXcx71UUUUU7v73VFW2AGmfu7bjvjm3 AaLaJ3UbIYCTnIWIevJa73UjIFyTuYvjxUamFAUUUUU X-CM-SenderInfo: xfkh05pxdqswro6rljoofrz/xtbDAg+HRWqWcK+FpgAA3P Content-Type: text/plain; charset="utf-8" From: Yuan Chen A perf_event program running in NMI context overwrites a rhtab element whose value holds a referenced task kptr. The old kptr must stay attached to the element (cancel semantics, matching hash maps); before the rhtab recycle fix the NMI update eagerly released it and the probe observed NULL. The test asserts the NMI program actually ran, so the probe result is meaningful. A second phase deletes and re-inserts the element 2000 times. The re-insertion may recycle the freed element, which still owns the kptr; before the fix the alloc path zeroed the inherited slot via check_and_init_map_value(), leaking the reference, and the probe never observed a non-NULL pointer. The test requires at least one recycle to inherit the kptr, and also verifies that plain (non-special) value bytes still round-trip through the recycled element on every iteration. The NMI phase is skipped when no hardware PMU is available. Signed-off-by: Yuan Chen --- .../selftests/bpf/prog_tests/rhtab_kptr.c | 184 ++++++++++++++++++ .../testing/selftests/bpf/progs/rhtab_kptr.c | 146 ++++++++++++++ .../testing/selftests/bpf/rhtab_kptr_common.h | 6 + 3 files changed, 336 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/rhtab_kptr.c create mode 100644 tools/testing/selftests/bpf/progs/rhtab_kptr.c create mode 100644 tools/testing/selftests/bpf/rhtab_kptr_common.h diff --git a/tools/testing/selftests/bpf/prog_tests/rhtab_kptr.c b/tools/te= sting/selftests/bpf/prog_tests/rhtab_kptr.c new file mode 100644 index 000000000000..4bdcc9ce5500 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/rhtab_kptr.c @@ -0,0 +1,184 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 KylinSoft Co., Ltd. */ + +#include +#include +#include +#include +#include +#include "rhtab_kptr.skel.h" + +/* Userspace mirror of the BPF-side struct val_t (progs/rhtab_kptr.c). The + * update syscall copies map->value_size bytes from the buffer, so it must + * be at least that large; special fields are skipped by the value copy but + * the kernel still reads the full value_size from userspace. + */ +struct val_t_user { + __u64 tsk; + __u32 magic; + __u32 pad; +}; + +_Static_assert(sizeof(struct val_t_user) =3D=3D 16, "val_t layout drift"); +_Static_assert(offsetof(struct val_t_user, magic) =3D=3D 8, "val_t magic o= ffset drift"); + +/* Zeroed value for creating/recreating elements; BSS is zero-filled. */ +static struct val_t_user zero; + +/* Cached CPU count and scratch buffer for percpu counter summation. */ +static __u64 *cpu_vals; +static int ncpu =3D -1; + +static __u64 read_counter(struct rhtab_kptr *skel, u32 idx) +{ + __u64 sum =3D 0; + int i, err; + + if (!cpu_vals) + return 0; + err =3D bpf_map_lookup_elem(bpf_map__fd(skel->maps.counters), &idx, + cpu_vals); + if (!ASSERT_OK(err, "lookup_counter")) + return 0; + for (i =3D 0; i < ncpu; i++) + sum +=3D cpu_vals[i]; + return sum; +} + +/* Run @name via BPF_PROG_TEST_RUN, asserting both the syscall status and + * that the program exited 0. Returns 0 on success. + */ +static int run_prog_ok(struct rhtab_kptr *skel, const char *name) +{ + LIBBPF_OPTS(bpf_test_run_opts, topts); + struct bpf_program *prog; + int err; + + prog =3D bpf_object__find_program_by_name(skel->obj, name); + if (!ASSERT_OK_PTR(prog, name)) + return -1; + err =3D bpf_prog_test_run_opts(bpf_program__fd(prog), &topts); + if (!ASSERT_OK(err, name)) + return -1; + if (!ASSERT_EQ(topts.retval, 0, name)) + return -1; + return 0; +} + +void test_rhtab_kptr(void) +{ + struct perf_event_attr attr =3D { + .type =3D PERF_TYPE_HARDWARE, + .config =3D PERF_COUNT_HW_CPU_CYCLES, + .freq =3D 1, + .sample_freq =3D read_perf_max_sample_freq(), + .size =3D sizeof(struct perf_event_attr), + }; + struct rhtab_kptr *skel; + __u64 init_before, nonnull_before; + __u32 key =3D 0; + int pmu_fd, i, retries =3D 0; + + ncpu =3D libbpf_num_possible_cpus(); + if (!ASSERT_GT(ncpu, 0, "num_possible_cpus")) + return; + cpu_vals =3D calloc(ncpu, sizeof(*cpu_vals)); + if (!ASSERT_OK_PTR(cpu_vals, "calloc_cpu_vals")) + return; + + skel =3D rhtab_kptr__open_and_load(); + if (!ASSERT_OK_PTR(skel, "open_and_load")) + goto out_free; + + /* Create the element and stash a referenced task kptr in it. */ + if (!ASSERT_OK(bpf_map_update_elem(bpf_map__fd(skel->maps.rhtab), + &key, &zero, BPF_ANY), "create_elem")) + goto out; + if (run_prog_ok(skel, "init_elem") !=3D 0) + goto out; + + pmu_fd =3D syscall(__NR_perf_event_open, &attr, -1, 0, -1, 0); + if (pmu_fd >=3D 0) { + skel->links.nmi_update =3D bpf_program__attach_perf_event(skel->progs.nm= i_update, + pmu_fd); + if (!ASSERT_OK_PTR(skel->links.nmi_update, "attach_perf_event")) { + close(pmu_fd); + goto out; + } + + /* Let the NMI handler overwrite the element, and make sure it + * actually ran before probing (otherwise the probe would pass + * vacuously even on an unfixed kernel). + */ + for (i =3D 0; i < 20 && read_counter(skel, 1) =3D=3D 0; i++) + usleep(100000); + ASSERT_GT(read_counter(skel, 1), 0, "nmi_update_ran"); + + bpf_link__destroy(skel->links.nmi_update); + skel->links.nmi_update =3D NULL; + close(pmu_fd); + + /* + * The old kptr must still be attached to the element: the + * NMI update path only cancels NMI-safe fields, mirroring + * hash map semantics. Before the fix the kptr was released + * from the NMI context and the probe below would see NULL. + */ + if (run_prog_ok(skel, "probe_elem") !=3D 0) + goto out; + + ASSERT_EQ(read_counter(skel, 2), 1, "xchg_non_null"); + ASSERT_EQ(read_counter(skel, 3), 0, "xchg_null"); + } else { + test__skip(); + } + + /* + * Now exercise the delete/re-insert recycle path. The delete only + * cancels NMI-safe fields, so the freed element still owns the kptr. + * If the re-insertion recycles that element, the kptr must be + * inherited; zeroing it (as check_and_init_map_value() did before + * the fix) leaks the reference and probe_elem() observes NULL. + * Fresh memory handed out by the allocator is zeroed, so NULL probes + * are expected too; only require that the inherited kptr survives at + * least one recycle. Every iteration runs exactly one probe, so the + * counters must add up to the loop count. + */ + init_before =3D read_counter(skel, 0); + nonnull_before =3D read_counter(skel, 2); + for (i =3D 0; i < 2000; i++) { + if (run_prog_ok(skel, "init_elem") !=3D 0) { + /* init_elem fails only if the element is missing, + * which must not happen in this single-threaded + * loop; count it so a rhtab bug cannot be absorbed + * silently. + */ + retries++; + if (!ASSERT_OK(bpf_map_update_elem(bpf_map__fd(skel->maps.rhtab), + &key, &zero, BPF_ANY), + "recreate_elem")) + goto out; + if (run_prog_ok(skel, "init_elem") !=3D 0) + goto out; + } + if (run_prog_ok(skel, "del_elem") !=3D 0 || + run_prog_ok(skel, "upd_elem") !=3D 0 || + run_prog_ok(skel, "probe_elem") !=3D 0) + goto out; + } + + /* + * Plain (non-special) value bytes must survive the recycle path: + * every probe must observe the magic value written by upd_elem() in + * the same iteration, regardless of whether the element memory was + * recycled or freshly allocated. + */ + ASSERT_EQ(retries, 0, "no_unexpected_recreate"); + ASSERT_EQ(read_counter(skel, 0) - init_before, 2000, "init_loop_count"); + ASSERT_EQ(read_counter(skel, 4), 2000, "recycle_magic_roundtrip"); + ASSERT_GT(read_counter(skel, 2), nonnull_before, "recycle_xchg_non_null"); +out: + rhtab_kptr__destroy(skel); +out_free: + free(cpu_vals); +} diff --git a/tools/testing/selftests/bpf/progs/rhtab_kptr.c b/tools/testing= /selftests/bpf/progs/rhtab_kptr.c new file mode 100644 index 000000000000..c96cf7f2d799 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/rhtab_kptr.c @@ -0,0 +1,146 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 KylinSoft Co., Ltd. */ + +/* + * Verify that the rhtab update/delete recycle paths do not eagerly destroy + * referenced kptrs. rhtab must match the hash map semantics introduced by + * commit a3a81d247651 ("bpf: Cancel special fields on map value recycle"): + * only NMI-safe fields (timer, workqueue, task_work) are cancelled on + * update/delete, while kptrs stay attached to the recycled element until = it + * is eventually freed. + * + * Two paths are exercised: + * 1. a perf_event (NMI) program overwrites an existing element; without = the + * fix the NMI update releases the old kptr and probe_elem() observes + * NULL; + * 2. the element is deleted and re-inserted; the re-insertion may recycle + * the freed element, and zeroing the inherited kptr slot (as + * check_and_init_map_value() did before the fix) would drop the + * reference without releasing it. probe_elem() must observe the + * inherited non-NULL pointer, and plain (non-special) value bytes must + * still round-trip through the recycled element. + * + * The delete program checks that the element really disappeared, otherwise + * the following update would be an in-place update whose value copy skips + * the special fields, and the surviving kptr would prove nothing about the + * recycle path. + */ +#include +#include +#include "rhtab_kptr_common.h" + +char LICENSE[] SEC("license") =3D "GPL"; + +struct val_t { + struct task_struct __kptr *tsk; + __u32 magic; +}; + +struct { + __uint(type, BPF_MAP_TYPE_RHASH); + __uint(max_entries, 16); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, __u32); + __type(value, struct val_t); +} rhtab SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __uint(max_entries, 5); + __type(key, __u32); + __type(value, __u64); +} counters SEC(".maps"); + +/* 0: init ok, 1: nmi update ok, 2: probe xchg non-NULL, 3: probe xchg NUL= L, + * 4: probe saw expected magic value + */ +static __always_inline void bump(u32 idx) +{ + u64 *v =3D bpf_map_lookup_elem(&counters, &idx); + + if (v) + (*v)++; +} + +extern struct task_struct *bpf_task_acquire(struct task_struct *p) __ksym; +extern void bpf_task_release(struct task_struct *p) __ksym; + +SEC("perf_event") +int nmi_update(struct bpf_perf_event_data *ctx) +{ + struct val_t val =3D {}; + u32 key =3D 0; + + if (bpf_map_update_elem(&rhtab, &key, &val, BPF_ANY) =3D=3D 0) + bump(1); + return 0; +} + +SEC("syscall") +int init_elem(void *ctx) +{ + struct val_t *val; + struct task_struct *task, *old; + u32 key =3D 0; + + val =3D bpf_map_lookup_elem(&rhtab, &key); + if (!val) + return 1; + task =3D bpf_task_acquire(bpf_get_current_task_btf()); + if (!task) + return 2; + old =3D bpf_kptr_xchg(&val->tsk, task); + if (old) + bpf_task_release(old); + bump(0); + return 0; +} + +SEC("syscall") +int del_elem(void *ctx) +{ + u32 key =3D 0; + + if (bpf_map_delete_elem(&rhtab, &key)) + return 1; + /* The element must really be gone: otherwise the following upd_elem() + * is an in-place update on the surviving element and the kptr that + * probe_elem() observes never went through a recycle. + */ + if (bpf_map_lookup_elem(&rhtab, &key)) + return 2; + return 0; +} + +SEC("syscall") +int upd_elem(void *ctx) +{ + struct val_t val =3D { .magic =3D RHTAB_MAGIC }; + u32 key =3D 0; + + if (bpf_map_update_elem(&rhtab, &key, &val, BPF_ANY)) + return 1; + return 0; +} + +SEC("syscall") +int probe_elem(void *ctx) +{ + struct val_t *val; + struct task_struct *old; + u32 key =3D 0; + + val =3D bpf_map_lookup_elem(&rhtab, &key); + if (!val) + return 1; + old =3D bpf_kptr_xchg(&val->tsk, NULL); + if (old) { + bpf_task_release(old); + bump(2); + } else { + bump(3); + } + if (val->magic =3D=3D RHTAB_MAGIC) + bump(4); + return 0; +} diff --git a/tools/testing/selftests/bpf/rhtab_kptr_common.h b/tools/testin= g/selftests/bpf/rhtab_kptr_common.h new file mode 100644 index 000000000000..1b629fcb6742 --- /dev/null +++ b/tools/testing/selftests/bpf/rhtab_kptr_common.h @@ -0,0 +1,6 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2026 KylinSoft Co., Ltd. */ +#pragma once + +/* Magic value stored in the plain bytes of rhtab values ("RHAS"). */ +#define RHTAB_MAGIC 0x52484153 --=20 2.54.0 From nobody Sat Sep 26 13:09:08 2026 Received: from m16.mail.163.com (m16.mail.163.com [220.197.31.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1DC673750DB; Tue, 1 Sep 2026 06:30:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=220.197.31.4 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788244221; cv=none; b=WAeaGG56YpUw/qf+qF22v+fI0P2zp1EfgiMV3jLniiIu/kjWhdDGCPhJS+wVXEqr/hc+F7YYCczASenguMNcs64qhOdR5WYPpUYgAnOQI/Fn9/Z+V9X+gc7j20JORFWFfhRrBnzidfMsoQDFRbqxwbbjv0Kfi0lOlvmoDHWogSI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1788244221; c=relaxed/simple; bh=awYWRwZTpc803kJeb1VD2Jniuc1ElIjgAZiHBIqFLss=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Xmmj+B8H8LhcAmKMVKjCI7JVdnnfKyVJMujD4BHojQsKxGT4zeoak15a1vovHuQicJZpkaF59xuUJIJO85BT2vJB7OUuMCwihFYWZbLDcirL68pOQByncVPA3da6PqCl5a1FF6iSVLXUcrqrv+ah1wGo6Go9yqy7P9Wf8278040= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com; spf=pass smtp.mailfrom=163.com; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b=mJw7WJfc; arc=none smtp.client-ip=220.197.31.4 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=163.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=163.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=163.com header.i=@163.com header.b="mJw7WJfc" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=163.com; s=s110527; h=From:To:Subject:Date:Message-ID:MIME-Version; bh=A0 wRnOe1OptnJji3/EBk4wcEzLHeWJp2ZtxElyEj4Xk=; b=mJw7WJfc3hZcdE2IfG QZhv6tMuoiLriEEtRlX3UGdJIgwjUghqWvyAwxBzaupaSNdO3NtQvPwHCuzZdHDF rWgw38/+Azc55h26xLzaaLGqSBOLTmcLVh/9NZVJSG/2OD+BzFCq1qMjTHEbKQXi 2LfjyL5WI/BgHrbYSGLzn9Xdw= Received: from nec8-i7 (unknown []) by gzsmtp1 (Coremail) with SMTP id PCgvCgD3n5+mcJZqA6mBNA--.25535S6; Tue, 01 Sep 2026 14:29:04 +0800 (CST) From: chenyuan_fl@163.com To: bpf@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, Mykyta Yatsenko , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Eduard Zingerman , Kumar Kartikeya Dwivedi , Martin KaFai Lau , Emil Tsalapatis , Ihor Solodrai , Shuah Khan , Nuoqi Gui , Yuan Chen Subject: [PATCH bpf-next v3 4/4] selftests/bpf: Test rhtab special-field combinations Date: Tue, 1 Sep 2026 14:28:45 +0800 Message-ID: <20260901062845.1379760-5-chenyuan_fl@163.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260901062845.1379760-1-chenyuan_fl@163.com> References: <0560a24d-2cf9-4e5b-aa61-580af1e56de1@gmail.com> <20260901062845.1379760-1-chenyuan_fl@163.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-CM-TRANSID: PCgvCgD3n5+mcJZqA6mBNA--.25535S6 X-Coremail-Antispam: 1Uf129KBjvAXoWfCFWDZFyktFyruFy8tF4Utwb_yoW8tFW8to Z3Ww45Za18GryvgFyvgFyvkF1rWayvg3srXF1Yg3y7Xa40kFy5ur9rCw47Xw17u3W09rWU ua4jq34fZF1fJFWrn29KB7ZKAUJUUUU8529EdanIXcx71UUUUU7v73VFW2AGmfu7bjvjm3 AaLaJ3UbIYCTnIWIevJa73UjIFyTuYvjxUFeHqDUUUU X-CM-SenderInfo: xfkh05pxdqswro6rljoofrz/xtbDARCIRmqWcLB2YwAA32 Content-Type: text/plain; charset="utf-8" From: Yuan Chen BPF_MAP_TYPE_RHASH allows spin locks, timers, workqueues, task_work, kptrs (referenced, untrusted, per-cpu) and refcounts in map values. The recycle fix only changes kptr slot handling, so verify each field combination end to end: * lock_kptr: bpf_spin_lock + referenced kptr + plain data in one value. BPF_F_LOCK syscall updates/lookups must work before and after many delete/re-insert recycle cycles, the referenced kptr must be inherited on recycled elements (zeroing it would leak the reference), and the plain bytes must round-trip every iteration. * timer: arm a bpf_timer and verify it fires, delete the element and verify the timer is cancelled, then re-insert (possibly recycling the freed element) and arm a fresh timer again. * kptr_untrusted: the untrusted kptr must survive the recycle like a referenced one. * kptr_percpu: the per-cpu kptr reference must survive the recycle (zeroing it would leak the reference). On the unfixed kernel the three kptr subtests fail at the recycle assertions while the lock and timer paths still pass, isolating the behavior change to kptr slots only. Signed-off-by: Yuan Chen --- .../selftests/bpf/prog_tests/rhtab_fields.c | 337 ++++++++++++++++ .../selftests/bpf/progs/rhtab_fields.c | 378 ++++++++++++++++++ .../selftests/bpf/rhtab_fields_common.h | 19 + 3 files changed, 734 insertions(+) create mode 100644 tools/testing/selftests/bpf/prog_tests/rhtab_fields.c create mode 100644 tools/testing/selftests/bpf/progs/rhtab_fields.c create mode 100644 tools/testing/selftests/bpf/rhtab_fields_common.h diff --git a/tools/testing/selftests/bpf/prog_tests/rhtab_fields.c b/tools/= testing/selftests/bpf/prog_tests/rhtab_fields.c new file mode 100644 index 000000000000..93e8cbacca65 --- /dev/null +++ b/tools/testing/selftests/bpf/prog_tests/rhtab_fields.c @@ -0,0 +1,337 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 KylinSoft Co., Ltd. */ + +#include +#include +#include +#include +#include +#include "rhtab_fields.skel.h" +#include "rhtab_fields_common.h" + +#define RECYCLE_LOOPS 2000 + +/* Userspace view of the lkmap value. The BPF side owns the real layout; + * the spin lock and the kptr are special fields that value copies skip, + * so only the plain bytes actually matter here. + */ +struct lock_kptr_val_user { + __u32 lock; + __u32 pad; + __u64 tsk; + __u32 magic; + __u32 pad2; +}; + +_Static_assert(sizeof(struct lock_kptr_val_user) =3D=3D 24, "lkmap layout = drift"); +_Static_assert(offsetof(struct lock_kptr_val_user, magic) =3D=3D 16, + "lkmap magic offset drift"); + +/* + * Zeroed value buffer shared by every create/update issued from userspace. + * The update syscall copies map->value_size bytes from this buffer (speci= al + * fields among them), so a short stack variable would be read past its en= d; + * BSS is zero-filled and 64 bytes cover every map in the skeleton. Each + * caller re-checks the size to keep that true as maps are added. + */ +static __u8 zero_val[64]; + +/* Cached CPU count and scratch buffer for percpu counter summation. */ +static __u64 *cpu_vals; +static int ncpu; + +static __u64 read_counter(struct rhtab_fields *skel, __u32 idx) +{ + __u64 sum =3D 0; + int i, err; + + err =3D bpf_map_lookup_elem(bpf_map__fd(skel->maps.counters), &idx, + cpu_vals); + if (!ASSERT_OK(err, "lookup_counter")) + return 0; + for (i =3D 0; i < ncpu; i++) + sum +=3D cpu_vals[i]; + return sum; +} + +/* + * Run @name through BPF_PROG_TEST_RUN. Returns 0 on success and reports t= he + * program retval through @retval, so callers can tell a syscall failure, + * a prog that exited nonzero, and a prog that exited 0 apart. + */ +static int run_prog(struct rhtab_fields *skel, const char *name, int *retv= al) +{ + LIBBPF_OPTS(bpf_test_run_opts, topts); + struct bpf_program *prog; + int err; + + prog =3D bpf_object__find_program_by_name(skel->obj, name); + if (!ASSERT_OK_PTR(prog, name)) + return -1; + err =3D bpf_prog_test_run_opts(bpf_program__fd(prog), &topts); + if (!ASSERT_OK(err, name)) + return -1; + if (retval) + *retval =3D topts.retval; + return 0; +} + +/* run_prog() plus the assertion that the program exited 0. */ +static int run_prog_ok(struct rhtab_fields *skel, const char *name) +{ + int retval =3D -1; + + if (!ASSERT_OK(run_prog(skel, name, &retval), name)) + return -1; + if (!ASSERT_EQ(retval, 0, name)) + return -1; + return 0; +} + +/* Create one element filled with zero_val in @map. */ +static int create_zero_elem(struct bpf_map *map, const char *name) +{ + __u32 key =3D 0; + int fd; + + if (!ASSERT_LE(bpf_map__value_size(map), sizeof(zero_val), + "value_size_fits")) + return -1; + fd =3D bpf_map__fd(map); + return bpf_map_update_elem(fd, &key, zero_val, BPF_ANY); +} + +static void recycle_loop(struct rhtab_fields *skel, int map_fd, + const char *init, const char *del, + const char *upd, const char *probe, + int *retries) +{ + __u32 key =3D 0; + int i; + + for (i =3D 0; i < RECYCLE_LOOPS; i++) { + if (run_prog_ok(skel, init) !=3D 0) { + /* init fails only if the element is missing, which + * must not happen in this single-threaded loop. Count + * it so a rhtab bug cannot be absorbed silently; the + * caller asserts the count is zero. + */ + (*retries)++; + if (!ASSERT_OK(bpf_map_update_elem(map_fd, &key, + zero_val, BPF_ANY), + "recreate_elem")) + return; + if (run_prog_ok(skel, init) !=3D 0) + return; + } + if (run_prog_ok(skel, del) !=3D 0) + return; + if (run_prog_ok(skel, upd) !=3D 0) + return; + if (run_prog_ok(skel, probe) !=3D 0) + return; + } +} + +static void subtest_lock_kptr(struct rhtab_fields *skel) +{ + struct lock_kptr_val_user val =3D {}; + struct lock_kptr_val_user out =3D {}; + __u64 nonnull_before, total; + int map_fd, retries =3D 0; + __u32 key =3D 0; + + map_fd =3D bpf_map__fd(skel->maps.lkmap); + + if (!ASSERT_OK(create_zero_elem(skel->maps.lkmap, "create_elem"), + "create_elem")) + return; + + /* The spin lock must be usable from the syscall path (BPF_F_LOCK). */ + val.magic =3D LK_MAGIC; + if (!ASSERT_OK(bpf_map_update_elem(map_fd, &key, &val, BPF_F_LOCK), + "locked_update")) + return; + if (!ASSERT_OK(bpf_map_lookup_elem_flags(map_fd, &key, &out, BPF_F_LOCK), + "locked_lookup")) + return; + ASSERT_EQ(out.magic, LK_MAGIC, "locked_lookup_magic"); + + /* + * Delete/re-insert recycle cycles: the referenced kptr must be + * inherited on recycled elements (zeroing it would leak the + * reference) and the plain magic bytes must round-trip every time. + * Every iteration runs exactly one probe, so the two probe counters + * must add up to the loop count; the magic check must hold on every + * single probe. + */ + nonnull_before =3D read_counter(skel, 1); + recycle_loop(skel, map_fd, "lk_init", "lk_del", "lk_upd", "lk_probe", + &retries); + ASSERT_EQ(retries, 0, "no_unexpected_recreate"); + ASSERT_EQ(read_counter(skel, 0), RECYCLE_LOOPS, "lk_init_count"); + total =3D read_counter(skel, 1) + read_counter(skel, 2); + ASSERT_EQ(total, RECYCLE_LOOPS, "lk_probe_count"); + ASSERT_EQ(read_counter(skel, 3), RECYCLE_LOOPS, + "recycle_magic_roundtrip"); + ASSERT_GT(read_counter(skel, 1), nonnull_before, + "recycle_xchg_non_null"); + + /* The spin lock must still work after many recycles. */ + val.magic =3D LK_MAGIC + 1; + if (!ASSERT_OK(bpf_map_update_elem(map_fd, &key, &val, BPF_F_LOCK), + "post_recycle_locked_update")) + return; + memset(&out, 0, sizeof(out)); + if (!ASSERT_OK(bpf_map_lookup_elem_flags(map_fd, &key, &out, BPF_F_LOCK), + "post_recycle_locked_lookup")) + return; + ASSERT_EQ(out.magic, LK_MAGIC + 1, "post_recycle_locked_magic"); +} + +static void subtest_timer(struct rhtab_fields *skel) +{ + int fired, map_fd; + __u32 key =3D 0; + + map_fd =3D bpf_map__fd(skel->maps.tmap); + + if (!ASSERT_OK(create_zero_elem(skel->maps.tmap, "create_elem"), + "create_elem")) + return; + + /* 1. A short-delay timer must fire. */ + skel->data->timer_delay_ns =3D 50000; + if (run_prog_ok(skel, "arm_timer") !=3D 0) + return; + usleep(300000); + if (!ASSERT_GT(skel->bss->timer_fired, 0, "timer_fired_first")) + return; + + /* + * 2. The real cancellation test: arm a long-delay timer on a freshly + * recycled element and delete the element while the timer is still + * pending. If the delete failed to cancel it, the callback would run + * before the sleep below ends. The element must be deleted and + * recreated between arms: bpf_timer_init() returns -EBUSY on an + * element whose timer has not been cancelled and freed yet. + */ + fired =3D skel->bss->timer_fired; + if (!ASSERT_OK(bpf_map_delete_elem(map_fd, &key), + "delete_before_rearm")) + return; + if (!ASSERT_OK(create_zero_elem(skel->maps.tmap, "recreate_elem"), + "recreate_before_rearm")) + return; + skel->data->timer_delay_ns =3D 200000000; + if (run_prog_ok(skel, "arm_timer") !=3D 0) + return; + if (!ASSERT_OK(bpf_map_delete_elem(map_fd, &key), + "delete_cancels_timer")) + return; + usleep(300000); + ASSERT_EQ(skel->bss->timer_fired, fired, "timer_cancelled_after_delete"); + + /* 3. A recycled element can arm and fire a fresh timer again. */ + skel->data->timer_delay_ns =3D 50000; + if (!ASSERT_OK(create_zero_elem(skel->maps.tmap, "recreate_elem"), + "recreate_second")) + return; + if (run_prog_ok(skel, "arm_timer") !=3D 0) + return; + usleep(300000); + ASSERT_GT(skel->bss->timer_fired, fired, "timer_fired_second"); +} + +struct kptr_recycle_case { + const char *name; + struct bpf_map **map; + const char *init; + const char *del; + const char *upd; + const char *probe; + __u32 init_idx; /* counter bumped on every successful init */ + __u32 nonnull_idx; /* counter bumped when the kptr was inherited */ + __u32 null_idx; /* counter bumped when the kptr was not inherited */ + __u32 marker_idx; /* percpu data roundtrip counter, 0 if none */ +}; + +static void subtest_kptr_recycle(struct rhtab_fields *skel, + const struct kptr_recycle_case *c) +{ + __u64 nonnull_before, total; + struct bpf_map *map; + int map_fd, retries =3D 0; + + map =3D *c->map; + map_fd =3D bpf_map__fd(map); + + if (!ASSERT_OK(create_zero_elem(map, "create_elem"), "create_elem")) + return; + + /* The kptr must survive the recycle without leaking its reference, + * exactly like the referenced kptr in the lkmap subtest. + */ + nonnull_before =3D read_counter(skel, c->nonnull_idx); + recycle_loop(skel, map_fd, c->init, c->del, c->upd, c->probe, + &retries); + ASSERT_EQ(retries, 0, "no_unexpected_recreate"); + ASSERT_EQ(read_counter(skel, c->init_idx), RECYCLE_LOOPS, + "init_count"); + total =3D read_counter(skel, c->nonnull_idx) + + read_counter(skel, c->null_idx); + ASSERT_EQ(total, RECYCLE_LOOPS, "probe_count"); + ASSERT_GT(read_counter(skel, c->nonnull_idx), nonnull_before, + "recycle_non_null"); + /* The marker read is CPU-local, so assert at least one hit. */ + if (c->marker_idx) + ASSERT_GT(read_counter(skel, c->marker_idx), 0, + "recycle_data_roundtrip"); +} + +static void subtest_kptr_recycles(struct rhtab_fields *skel) +{ + const struct kptr_recycle_case cases[] =3D { + { .name =3D "kptr_untrusted", .map =3D &skel->maps.umap, + .init =3D "u_init", .del =3D "u_del", .upd =3D "u_upd", + .probe =3D "u_probe", .init_idx =3D 4, .nonnull_idx =3D 5, + .null_idx =3D 6 }, + { .name =3D "kptr_percpu", .map =3D &skel->maps.pcmap, + .init =3D "pc_init", .del =3D "pc_del", .upd =3D "pc_upd", + .probe =3D "pc_probe", .init_idx =3D 7, .nonnull_idx =3D 8, + .null_idx =3D 9, .marker_idx =3D 10 }, + }; + int i; + + for (i =3D 0; i < ARRAY_SIZE(cases); i++) { + if (test__start_subtest(cases[i].name)) + subtest_kptr_recycle(skel, &cases[i]); + } +} + +void test_rhtab_fields(void) +{ + struct rhtab_fields *skel; + + ncpu =3D libbpf_num_possible_cpus(); + if (!ASSERT_GT(ncpu, 0, "num_possible_cpus")) + return; + cpu_vals =3D calloc(ncpu, sizeof(*cpu_vals)); + if (!ASSERT_OK_PTR(cpu_vals, "calloc_cpu_vals")) + return; + + skel =3D rhtab_fields__open_and_load(); + if (!ASSERT_OK_PTR(skel, "open_and_load")) { + free(cpu_vals); + return; + } + + if (test__start_subtest("lock_kptr")) + subtest_lock_kptr(skel); + if (test__start_subtest("timer")) + subtest_timer(skel); + subtest_kptr_recycles(skel); + + rhtab_fields__destroy(skel); + free(cpu_vals); +} diff --git a/tools/testing/selftests/bpf/progs/rhtab_fields.c b/tools/testi= ng/selftests/bpf/progs/rhtab_fields.c new file mode 100644 index 000000000000..f8bcd88b7f34 --- /dev/null +++ b/tools/testing/selftests/bpf/progs/rhtab_fields.c @@ -0,0 +1,378 @@ +// SPDX-License-Identifier: GPL-2.0 +/* Copyright (c) 2026 KylinSoft Co., Ltd. */ + +/* + * Combined special-field tests for BPF_MAP_TYPE_RHASH. Each map carries a + * different field combination and is exercised through delete/re-insert + * cycles so the bpf memory allocator recycles element memory: + * + * 1. lkmap: bpf_spin_lock + referenced kptr + plain data in one value. + * After every recycle the spin lock must still be usable (initialized= by + * the alloc path), the referenced kptr must be inherited instead of + * zeroed (zeroing would leak the reference), and the plain bytes must + * round-trip. + * 2. tmap: bpf_timer. The delete path must cancel a timer that is still + * pending (the delay is tunable through timer_delay_ns so userspace c= an + * arm a long timer and delete the element before it fires), and a + * recycled element must be able to arm a fresh timer again. + * 3. umap: untrusted (unreferenced) kptr. The inherited pointer must be + * preserved on recycle, matching hash map behavior. + * 4. pcmap: per-cpu kptr. Like the referenced kptr, the per-cpu reference + * must not be dropped on recycle, and the object's contents must + * survive the recycle round-trip. + * + * The delete programs check that the element really disappeared, otherwise + * the following update would be an in-place update whose value copy skips + * the special fields, and the surviving kptr would prove nothing about the + * recycle path. + */ + +#include +#include +#include "bpf_experimental.h" +#include "rhtab_fields_common.h" + +char LICENSE[] SEC("license") =3D "GPL"; + +struct lock_kptr_val { + struct bpf_spin_lock lock; + struct task_struct __kptr *tsk; + __u32 magic; +}; + +struct { + __uint(type, BPF_MAP_TYPE_RHASH); + __uint(max_entries, 16); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, __u32); + __type(value, struct lock_kptr_val); +} lkmap SEC(".maps"); + +struct timer_val { + struct bpf_timer timer; +}; + +struct { + __uint(type, BPF_MAP_TYPE_RHASH); + __uint(max_entries, 16); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, __u32); + __type(value, struct timer_val); +} tmap SEC(".maps"); + +struct unref_val { + struct task_struct __kptr_untrusted *tsk; +}; + +struct { + __uint(type, BPF_MAP_TYPE_RHASH); + __uint(max_entries, 16); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, __u32); + __type(value, struct unref_val); +} umap SEC(".maps"); + +struct pcval { + __u64 v; +}; + +struct pcpu_val { + struct pcval __percpu_kptr *pc; +}; + +struct { + __uint(type, BPF_MAP_TYPE_RHASH); + __uint(max_entries, 16); + __uint(map_flags, BPF_F_NO_PREALLOC); + __type(key, __u32); + __type(value, struct pcpu_val); +} pcmap SEC(".maps"); + +struct { + __uint(type, BPF_MAP_TYPE_PERCPU_ARRAY); + __uint(max_entries, 11); + __type(key, __u32); + __type(value, __u64); +} counters SEC(".maps"); + +/* 0: lk init ok, 1: lk probe xchg non-NULL, 2: lk probe xchg NULL, + * 3: lk probe magic ok, + * 4: u init ok, 5: u probe ptr non-NULL, 6: u probe ptr NULL, + * 7: pc init ok, 8: pc probe xchg non-NULL, 9: pc probe xchg NULL, + * 10: pc probe data roundtrip + */ +static __always_inline void bump(u32 idx) +{ + u64 *v =3D bpf_map_lookup_elem(&counters, &idx); + + if (v) + (*v)++; +} + +extern struct task_struct *bpf_task_acquire(struct task_struct *p) __ksym; +extern void bpf_task_release(struct task_struct *p) __ksym; +extern void bpf_rcu_read_lock(void) __ksym; +extern void bpf_rcu_read_unlock(void) __ksym; + +/* Tunable from userspace (an initialized global lands in .data, so the + * driver writes it through skel->data, not skel->bss). + */ +int timer_delay_ns =3D 50000; +int timer_fired; + +/* Map 1: spin lock + referenced kptr + plain data. */ + +SEC("syscall") +int lk_init(void *ctx) +{ + struct lock_kptr_val *val; + struct task_struct *task, *old; + u32 key =3D 0; + + val =3D bpf_map_lookup_elem(&lkmap, &key); + if (!val) + return 1; + task =3D bpf_task_acquire(bpf_get_current_task_btf()); + if (!task) + return 2; + old =3D bpf_kptr_xchg(&val->tsk, task); + if (old) + bpf_task_release(old); + bump(0); + return 0; +} + +SEC("syscall") +int lk_del(void *ctx) +{ + u32 key =3D 0; + + if (bpf_map_delete_elem(&lkmap, &key)) + return 1; + /* The element must really be gone: otherwise the following lk_upd() + * is an in-place update on the surviving element and the kptr that + * lk_probe() then observes never went through a recycle. + */ + if (bpf_map_lookup_elem(&lkmap, &key)) + return 2; + return 0; +} + +SEC("syscall") +int lk_upd(void *ctx) +{ + struct lock_kptr_val val =3D { .magic =3D LK_MAGIC }; + u32 key =3D 0; + + /* BPF_ANY is safe even though the value holds a spin lock: value + * copies skip special fields, so the lock word is never written and + * the prog side does not need to take the lock for an update. + */ + bpf_map_update_elem(&lkmap, &key, &val, BPF_ANY); + return 0; +} + +SEC("syscall") +int lk_probe(void *ctx) +{ + struct lock_kptr_val *val; + struct task_struct *old; + __u32 magic; + u32 key =3D 0; + + val =3D bpf_map_lookup_elem(&lkmap, &key); + if (!val) + return 1; + /* Take the lock directly: a recycled element that came back with a + * corrupted lock word deadlocks here instead of passing. Helpers are + * forbidden while the lock is held, so the xchg stays outside. + */ + bpf_spin_lock(&val->lock); + magic =3D val->magic; + bpf_spin_unlock(&val->lock); + old =3D bpf_kptr_xchg(&val->tsk, NULL); + if (old) { + bpf_task_release(old); + bump(1); + } else { + bump(2); + } + if (magic =3D=3D LK_MAGIC) + bump(3); + return 0; +} + +/* Map 2: bpf_timer. */ + +static int timer_cb(void *map, void *key, struct timer_val *value) +{ + timer_fired++; + return 0; +} + +SEC("syscall") +int arm_timer(void *ctx) +{ + struct timer_val *val; + u32 key =3D 0; + + val =3D bpf_map_lookup_elem(&tmap, &key); + if (!val) + return 1; + /* 1 =3D=3D CLOCK_MONOTONIC */ + if (bpf_timer_init(&val->timer, &tmap, 1)) + return 2; + bpf_timer_set_callback(&val->timer, timer_cb); + if (bpf_timer_start(&val->timer, timer_delay_ns, 0)) + return 3; + return 0; +} + +/* Map 3: untrusted kptr. */ + +SEC("syscall") +int u_init(void *ctx) +{ + struct unref_val *val; + u32 key =3D 0; + + val =3D bpf_map_lookup_elem(&umap, &key); + if (!val) + return 1; + val->tsk =3D bpf_get_current_task_btf(); + bump(4); + return 0; +} + +SEC("syscall") +int u_del(void *ctx) +{ + u32 key =3D 0; + + if (bpf_map_delete_elem(&umap, &key)) + return 1; + if (bpf_map_lookup_elem(&umap, &key)) + return 2; + return 0; +} + +SEC("syscall") +int u_upd(void *ctx) +{ + struct unref_val val =3D {}; + u32 key =3D 0; + + bpf_map_update_elem(&umap, &key, &val, BPF_ANY); + return 0; +} + +SEC("syscall") +int u_probe(void *ctx) +{ + struct unref_val *val; + u32 key =3D 0; + + val =3D bpf_map_lookup_elem(&umap, &key); + if (!val) + return 1; + if (val->tsk) + bump(5); + else + bump(6); + val->tsk =3D NULL; + return 0; +} + +/* Map 4: per-cpu kptr. */ + +SEC("syscall") +int pc_init(void *ctx) +{ + struct pcpu_val *val; + struct pcval *p, *cp, *q, *old; + u32 key =3D 0; + + val =3D bpf_map_lookup_elem(&pcmap, &key); + if (!val) + return 1; + p =3D bpf_percpu_obj_new(struct pcval); + if (!p) + return 2; + old =3D bpf_kptr_xchg(&val->pc, p); + if (old) + bpf_percpu_obj_drop(old); + /* After the xchg the slot holds p, so q aliases p. Syscall progs are + * sleepable, so the kptr field load only yields a trusted per-cpu + * view under an explicit RCU read lock. + */ + bpf_rcu_read_lock(); + q =3D val->pc; + if (q) { + cp =3D bpf_this_cpu_ptr(q); + cp->v =3D PC_MAGIC; + } + bpf_rcu_read_unlock(); + bump(7); + return 0; +} + +SEC("syscall") +int pc_del(void *ctx) +{ + u32 key =3D 0; + + if (bpf_map_delete_elem(&pcmap, &key)) + return 1; + if (bpf_map_lookup_elem(&pcmap, &key)) + return 2; + return 0; +} + +SEC("syscall") +int pc_upd(void *ctx) +{ + struct pcpu_val val =3D {}; + u32 key =3D 0; + + bpf_map_update_elem(&pcmap, &key, &val, BPF_ANY); + return 0; +} + +SEC("syscall") +int pc_probe(void *ctx) +{ + struct pcpu_val *val; + struct pcval *cp, *q, *old; + u32 key =3D 0; + int marker =3D 0; + + val =3D bpf_map_lookup_elem(&pcmap, &key); + if (!val) + return 1; + /* The object was freshly allocated and marked by pc_init() within + * the same iteration, so an inherited object must still carry the + * marker; q aliases old until the drop. Read the marker under the + * RCU read lock (which makes the field load trusted) and before the + * xchg NULLs the field. The read is CPU-local and the loop may + * migrate between CPUs, so userspace only asserts that this fired + * at least once. + */ + bpf_rcu_read_lock(); + q =3D val->pc; + if (q) { + cp =3D bpf_this_cpu_ptr(q); + if (cp->v =3D=3D PC_MAGIC) + marker =3D 1; + } + bpf_rcu_read_unlock(); + old =3D bpf_kptr_xchg(&val->pc, NULL); + if (old) { + if (marker) + bump(10); + bpf_percpu_obj_drop(old); + bump(8); + } else { + bump(9); + } + return 0; +} diff --git a/tools/testing/selftests/bpf/rhtab_fields_common.h b/tools/test= ing/selftests/bpf/rhtab_fields_common.h new file mode 100644 index 000000000000..b52fb5e62807 --- /dev/null +++ b/tools/testing/selftests/bpf/rhtab_fields_common.h @@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* Copyright (c) 2026 KylinSoft Co., Ltd. */ +#pragma once + +/* + * Constants shared between the rhtab_fields BPF program (progs/) and its + * userspace driver (prog_tests/) so the two sides cannot drift apart + * silently. + */ + +/* Magic value stored in the plain bytes of lkmap values ("RHAB"). */ +#define LK_MAGIC 0x52484142 + +/* + * Marker written into freshly allocated per-cpu objects of pcmap. Distinct + * from LK_MAGIC on purpose: a hit proves the data came from our own + * pc_init(), not from anything else. + */ +#define PC_MAGIC 0x43504d47 --=20 2.54.0