From nobody Tue Sep 29 04:11:28 2026 Received: from mail-pj1-f72.google.com (mail-pj1-f72.google.com [209.85.216.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6893742C502 for ; Wed, 12 Aug 2026 23:19:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.72 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786576758; cv=none; b=pHMPR2dEagOGFp9ahovNX57HiRw1+AgbvNWQU8QIHWB5z0UG7GiQ8FVY5KNycLhldu3HyfMjBtizeQKmLs2hx68xR3k+xLtQxBCveMTN/yDqIivRmEV0vaI+Gdtxkd+0H+pU47lOu2tUueIkN6ynLdVnNljCMLQiHtIky4Mt+sc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786576758; c=relaxed/simple; bh=DjY7J8c/6/IjNjdfWdKHbpiLn2QZM5cbxrpCjkPxUf0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=QlqRX6ioN+A/S3N2YPk8V/IxhXzQ2lE/hm33vExAcFXpLVLIj4lVGDHhXcATYZmmpQsT8r9LgMYX3+f0Jmb5YHEIMYdfACfVO2AZ0g21CJRBcoywWmPmqPQWtJ2QO+fOt3POMdqcte52xY5reC7c1Z+64+xLwt4TTvNQyseXK4o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--tjmercier.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=HfAlhAQT; arc=none smtp.client-ip=209.85.216.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--tjmercier.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="HfAlhAQT" Received: by mail-pj1-f72.google.com with SMTP id 98e67ed59e1d1-38e8fee6af3so1888960a91.1 for ; Wed, 12 Aug 2026 16:19:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1786576755; x=1787181555; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=75rhLWRcSidlUgdQX62fzK5vbrG66n2hkP7GIeooNvs=; b=HfAlhAQTNjReVkhMI7qlSVnQkfk5bbKJbQrWepXb82JAQhjiPIeiF7tpcv8dt3TW5d srpqpCxRBjO4byc+6uNXukPW9xzbdJVl+FLqrvUnKXh8up2ylaSMwA55PWFxp4QpiphT o7g+qF2s4nOsMDF5/yqfnWYCJi3kBkVVsCm6cuMF+tynVA9RPhKs3yaU3+xQ9Yh4ZEpP 1oxye9hYRjoBgMVwy+l1GdZezgtpTxAcnR2K9UR9/aumWLnoDYQYtV/rpOeCgr3wx25b C2dT/OGU6pkoUtM4XPo3Vb5MhdVAIg4ixNhHeIutLSjBuTQyABNenahPi4QfD1i/GXx2 +hhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786576755; x=1787181555; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=75rhLWRcSidlUgdQX62fzK5vbrG66n2hkP7GIeooNvs=; b=h2hwJZEz2WSOvYAwcbyWaLueszHyrIW+qO58Flz3yaMkMHHjVlMlgqtAYqanlUj7Ll AUN+gRXrrs4QRgcRGohudZNsSeaaQuV26XCL3ubKMp59s0s/6AAD++gdLpABaPC+Hzjl SsGV/JnQVwEgo4WKNqgcyKXXvr7c4SY8E0wedIuCfOjPD2QoyHbV94WlJ+mxmB3pNeZH s51CvN42hKitOWOfEqb1xszmTtU8EIpbTL7HoPx4RZ4UxkfE90SMl3LVpTSqI0vhd1+H +JENWnybSM7Bp/BAzeFWCZDZe4O+WqPoUZZCFenNtvELAVqxBqSu8uBQQXwuC9vhbMLa jrUA== X-Forwarded-Encrypted: i=1; AHgh+Rq2xOAiKBsN+8e7Y6XKO7fF/PGnJ4dXSrstW4VgDVmyJ1VZuR+sf708+gTED1f6l3Zq2wi/gUjR0IWMk+I=@vger.kernel.org X-Gm-Message-State: AOJu0YwdC3y6/YLwEthKfL4I6rqxzyS7B6fAHHfcKjmwVjYnuZcuxHwT w+NdZ+I3FOjuW84yx2FpcDcaX0DX8l6IgS4p5igx5CuDQDk7/cpQxyreSREMr5v2rPaXLYNJUkJ 92Ub1Fb8m2ZeBLu3w0A== X-Received: from pjes1.prod.google.com ([2002:a17:90a:741:b0:38e:ba48:b0cc]) (user=tjmercier job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:4b92:b0:37f:e1b6:4c7d with SMTP id 98e67ed59e1d1-3931e079c22mr1555677a91.6.1786576754178; Wed, 12 Aug 2026 16:19:14 -0700 (PDT) Date: Wed, 12 Aug 2026 16:19:04 -0700 In-Reply-To: <20260812231905.2956588-1-tjmercier@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260812231905.2956588-1-tjmercier@google.com> X-Mailer: git-send-email 2.55.0.691.gc56d675ccc-goog Message-ID: <20260812231905.2956588-2-tjmercier@google.com> Subject: [PATCH bpf-next v4 1/2] bpf: htab: Split htab_elem_lru and htab_elem_pcpu off of htab_elem From: "T.J. Mercier" To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, eddyz87@gmail.com, memxor@gmail.com, martin.lau@linux.dev, song@kernel.org, yonghong.song@linux.dev, jolsa@kernel.org, emil@etsalapatis.com, mykyta.yatsenko5@gmail.com Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, "T.J. Mercier" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The htab_elem struct is used as the per-element type for all BPF hash map types and includes bpf_lru_node in a union with a ptr_to_pptr pointer. For standard (non-LRU, non-PCPU) hash maps, the 24 byte union allocated for every element is entirely unused. For non-preallocated PCPU maps, ptr_to_pptr only requires 8 bytes, leaving 16 bytes of unused overhead in the union. For preallocated PCPU maps ptr_to_pptr is unused since elements are freed to the PCPU freelist. Eliminate this per-element memory overhead by splitting htab_elem into dedicated structures for each map type: - struct htab_elem: Minimal structure for standard hash maps and preallocated PCPU maps (saves 24 bytes per element). - struct htab_elem_pcpu: Structure for non-preallocated PCPU maps containing ptr_to_pptr (saves 16 bytes per element). - struct htab_elem_lru: Retains struct bpf_lru_node for LRU maps. Because element sizes now vary by map type, add key_offset to struct bpf_htab to track the dynamic key offset. Update helper accessors and lookups to compute key and value offsets using htab->key_offset. Pointers to struct htab_elem in the existing code (e.g. htab_elem_hash) serve as generic base element pointers. This is possible because htab_elem, htab_elem_pcpu, and htab_elem_lru share a common initial sequence, making pointer casts safe. Signed-off-by: T.J. Mercier --- kernel/bpf/hashtab.c | 354 +++++++++++++++++++++++++--------------- kernel/bpf/map_in_map.c | 13 ++ kernel/bpf/map_in_map.h | 2 + 3 files changed, 233 insertions(+), 136 deletions(-) diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index 9f394e1aa2e8..fed4b1db8b78 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -102,11 +102,13 @@ struct bpf_htab { bool use_percpu_counter; u32 n_buckets; /* number of hash buckets */ u32 elem_size; /* size of each element in bytes */ + u32 key_offset; /* offset of key in bytes */ u32 hashrnd; }; =20 /* each htab element is struct htab_elem + key + value */ -struct htab_elem { +struct htab_elem; +struct htab_node { union { struct hlist_nulls_node hash_node; struct { @@ -117,12 +119,28 @@ struct htab_elem { }; }; }; - union { - /* pointer to per-cpu pointer */ - void *ptr_to_pptr; - struct bpf_lru_node lru_node; - }; - u32 hash; +}; + +struct htab_elem { + struct htab_node node; + u32 hash __aligned(8); + char key[] __aligned(8); +}; + +struct htab_elem_lru { + struct htab_node node; + struct bpf_lru_node lru_node; + u32 hash __aligned(8); + char key[] __aligned(8); +}; + +/* Only for non-preallocated PCPU maps. Preallocated PCPU maps don't need + * ptr_to_pptr, and use htab_elem. + */ +struct htab_elem_pcpu { + struct htab_node node; + void *ptr_to_pptr; + u32 hash __aligned(8); char key[] __aligned(8); }; =20 @@ -136,6 +154,21 @@ static inline bool htab_is_prealloc(const struct bpf_h= tab *htab) return !(htab->map.map_flags & BPF_F_NO_PREALLOC); } =20 +static struct bpf_lru_node *htab_elem_lru_node(struct htab_elem *l) +{ + return &((struct htab_elem_lru *)l)->lru_node; +} + +static void *htab_elem_get_ptr_to_pptr(struct htab_elem *l) +{ + return ((struct htab_elem_pcpu *)l)->ptr_to_pptr; +} + +static void htab_elem_set_ptr_to_pptr(struct htab_elem *l, void *ptr) +{ + ((struct htab_elem_pcpu *)l)->ptr_to_pptr =3D ptr; +} + static void htab_init_buckets(struct bpf_htab *htab) { unsigned int i; @@ -183,25 +216,30 @@ static inline bool is_fd_htab(const struct bpf_htab *= htab) return htab->map.map_type =3D=3D BPF_MAP_TYPE_HASH_OF_MAPS; } =20 -static inline void *htab_elem_value(struct htab_elem *l, u32 key_size) +static void *htab_elem_key(struct bpf_htab *htab, struct htab_elem *l) +{ + return (void *)l + htab->key_offset; +} + +static inline void *htab_elem_value(struct bpf_htab *htab, struct htab_ele= m *l) { - return l->key + round_up(key_size, 8); + return htab_elem_key(htab, l) + round_up(htab->map.key_size, 8); } =20 -static inline void htab_elem_set_ptr(struct htab_elem *l, u32 key_size, +static inline void htab_elem_set_ptr(struct bpf_htab *htab, struct htab_el= em *l, void __percpu *pptr) { - *(void __percpu **)htab_elem_value(l, key_size) =3D pptr; + *(void __percpu **)htab_elem_value(htab, l) =3D pptr; } =20 -static inline void __percpu *htab_elem_get_ptr(struct htab_elem *l, u32 ke= y_size) +static inline void __percpu *htab_elem_get_ptr(struct bpf_htab *htab, stru= ct htab_elem *l) { - return *(void __percpu **)htab_elem_value(l, key_size); + return *(void __percpu **)htab_elem_value(htab, l); } =20 -static void *fd_htab_map_get_ptr(const struct bpf_map *map, struct htab_el= em *l) +static void *fd_htab_map_get_ptr(struct bpf_htab *htab, struct htab_elem *= l) { - return *(void **)htab_elem_value(l, map->key_size); + return *(void **)htab_elem_value(htab, l); } =20 static struct htab_elem *get_htab_elem(struct bpf_htab *htab, int i) @@ -209,6 +247,16 @@ static struct htab_elem *get_htab_elem(struct bpf_htab= *htab, int i) return (struct htab_elem *) (htab->elems + i * (u64)htab->elem_size); } =20 +static u32 htab_elem_hash(struct bpf_htab *htab, struct htab_elem *l) +{ + return *(u32 *)(htab_elem_key(htab, l) - 8); +} + +static void htab_elem_set_hash(struct bpf_htab *htab, struct htab_elem *l,= u32 hash) +{ + *(u32 *)(htab_elem_key(htab, l) - 8) =3D hash; +} + /* Both percpu and fd htab support in-place update, so no need for * extra elem. LRU itself can remove the least used element, so * there is no need for an extra elem during map_update. @@ -231,7 +279,7 @@ static void htab_free_prealloced_internal_structs(struc= t bpf_htab *htab) =20 elem =3D get_htab_elem(htab, i); bpf_map_free_internal_structs(&htab->map, - htab_elem_value(elem, htab->map.key_size)); + htab_elem_value(htab, elem)); cond_resched(); } } @@ -254,7 +302,7 @@ static void htab_free_prealloced_fields(struct bpf_htab= *htab) =20 elem =3D get_htab_elem(htab, i); if (htab_is_percpu(htab)) { - void __percpu *pptr =3D htab_elem_get_ptr(elem, htab->map.key_size); + void __percpu *pptr =3D htab_elem_get_ptr(htab, elem); int cpu; =20 for_each_possible_cpu(cpu) { @@ -263,7 +311,7 @@ static void htab_free_prealloced_fields(struct bpf_htab= *htab) } } else { bpf_obj_free_fields(htab->map.record, - htab_elem_value(elem, htab->map.key_size)); + htab_elem_value(htab, elem)); cond_resched(); } cond_resched(); @@ -280,8 +328,7 @@ static void htab_free_elems(struct bpf_htab *htab) for (i =3D 0; i < htab->map.max_entries; i++) { void __percpu *pptr; =20 - pptr =3D htab_elem_get_ptr(get_htab_elem(htab, i), - htab->map.key_size); + pptr =3D htab_elem_get_ptr(htab, get_htab_elem(htab, i)); free_percpu(pptr); cond_resched(); } @@ -308,8 +355,8 @@ static struct htab_elem *prealloc_lru_pop(struct bpf_ht= ab *htab, void *key, =20 if (node) { bpf_map_inc_elem_count(&htab->map); - l =3D container_of(node, struct htab_elem, lru_node); - memcpy(l->key, key, htab->map.key_size); + l =3D (struct htab_elem *)container_of(node, struct htab_elem_lru, lru_n= ode); + memcpy(htab_elem_key(htab, l), key, htab->map.key_size); return l; } =20 @@ -340,8 +387,7 @@ static int prealloc_init(struct bpf_htab *htab) GFP_USER | __GFP_NOWARN); if (!pptr) goto free_elems; - htab_elem_set_ptr(get_htab_elem(htab, i), htab->map.key_size, - pptr); + htab_elem_set_ptr(htab, get_htab_elem(htab, i), pptr); cond_resched(); } =20 @@ -349,8 +395,8 @@ static int prealloc_init(struct bpf_htab *htab) if (htab_is_lru(htab)) err =3D bpf_lru_init(&htab->lru, htab->map.map_flags & BPF_F_NO_COMMON_LRU, - offsetof(struct htab_elem, hash) - - offsetof(struct htab_elem, lru_node), + offsetof(struct htab_elem_lru, hash) - + offsetof(struct htab_elem_lru, lru_node), htab_lru_map_delete_node, htab); else @@ -361,11 +407,11 @@ static int prealloc_init(struct bpf_htab *htab) =20 if (htab_is_lru(htab)) bpf_lru_populate(&htab->lru, htab->elems, - offsetof(struct htab_elem, lru_node), + offsetof(struct htab_elem_lru, lru_node), htab->elem_size, num_entries); else pcpu_freelist_populate(&htab->freelist, - htab->elems + offsetof(struct htab_elem, fnode), + htab->elems + offsetof(struct htab_elem, node.fnode), htab->elem_size, num_entries); =20 return 0; @@ -401,7 +447,7 @@ static int alloc_extra_elems(struct bpf_htab *htab) /* pop will succeed, since prealloc_init() * preallocated extra num_possible_cpus elements */ - l_new =3D container_of(l, struct htab_elem, fnode); + l_new =3D container_of(l, struct htab_elem, node.fnode); *per_cpu_ptr(pptr, cpu) =3D l_new; } htab->extra_elems =3D pptr; @@ -425,8 +471,8 @@ static int htab_map_alloc_check(union bpf_attr *attr) bool zero_seed =3D (attr->map_flags & BPF_F_ZERO_SEED); int numa_node =3D bpf_map_attr_numa_node(attr); =20 - BUILD_BUG_ON(offsetof(struct htab_elem, fnode.next) !=3D - offsetof(struct htab_elem, hash_node.pprev)); + BUILD_BUG_ON(offsetof(struct htab_node, fnode.next) !=3D + offsetof(struct htab_node, hash_node.pprev)); =20 if (zero_seed && !capable(CAP_SYS_ADMIN)) /* Guard against local DoS, and discourage production use. */ @@ -476,7 +522,7 @@ static void htab_mem_dtor(void *obj, void *ctx) if (IS_ERR_OR_NULL(hrec->record)) return; =20 - map_value =3D htab_elem_value(elem, hrec->key_size); + map_value =3D (void *)elem + sizeof(struct htab_elem) + round_up(hrec->ke= y_size, 8); bpf_obj_free_fields(hrec->record, map_value); } =20 @@ -583,8 +629,14 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *= attr) =20 htab->n_buckets =3D roundup_pow_of_two(htab->map.max_entries); =20 - htab->elem_size =3D sizeof(struct htab_elem) + - round_up(htab->map.key_size, 8); + if (htab_is_lru(htab)) + htab->key_offset =3D offsetof(struct htab_elem_lru, key); + else if (percpu && !prealloc) + htab->key_offset =3D offsetof(struct htab_elem_pcpu, key); + else + htab->key_offset =3D offsetof(struct htab_elem, key); + + htab->elem_size =3D htab->key_offset + round_up(htab->map.key_size, 8); if (percpu) htab->elem_size +=3D sizeof(void *); else @@ -692,14 +744,16 @@ static inline struct hlist_nulls_head *select_bucket(= struct bpf_htab *htab, u32 } =20 /* this lookup function can only be called with bucket lock taken */ -static struct htab_elem *lookup_elem_raw(struct hlist_nulls_head *head, u3= 2 hash, +static struct htab_elem *lookup_elem_raw(struct bpf_htab *htab, + struct hlist_nulls_head *head, u32 hash, void *key, u32 key_size) { struct hlist_nulls_node *n; struct htab_elem *l; =20 - hlist_nulls_for_each_entry_rcu(l, n, head, hash_node) - if (l->hash =3D=3D hash && !memcmp(&l->key, key, key_size)) + hlist_nulls_for_each_entry_rcu(l, n, head, node.hash_node) + if (htab_elem_hash(htab, l) =3D=3D hash && + !memcmp(htab_elem_key(htab, l), key, key_size)) return l; =20 return NULL; @@ -709,7 +763,8 @@ static struct htab_elem *lookup_elem_raw(struct hlist_n= ulls_head *head, u32 hash * the unlikely event when elements moved from one bucket into another * while link list is being walked */ -static struct htab_elem *lookup_nulls_elem_raw(struct hlist_nulls_head *he= ad, +static struct htab_elem *lookup_nulls_elem_raw(struct bpf_htab *htab, + struct hlist_nulls_head *head, u32 hash, void *key, u32 key_size, u32 n_buckets) { @@ -717,8 +772,9 @@ static struct htab_elem *lookup_nulls_elem_raw(struct h= list_nulls_head *head, struct htab_elem *l; =20 again: - hlist_nulls_for_each_entry_rcu(l, n, head, hash_node) - if (l->hash =3D=3D hash && !memcmp(&l->key, key, key_size)) + hlist_nulls_for_each_entry_rcu(l, n, head, node.hash_node) + if (htab_elem_hash(htab, l) =3D=3D hash && + !memcmp(htab_elem_key(htab, l), key, key_size)) return l; =20 if (unlikely(get_nulls_value(n) !=3D (hash & (n_buckets - 1)))) @@ -747,17 +803,18 @@ static void *__htab_map_lookup_elem(struct bpf_map *m= ap, void *key) =20 head =3D select_bucket(htab, hash); =20 - l =3D lookup_nulls_elem_raw(head, hash, key, key_size, htab->n_buckets); + l =3D lookup_nulls_elem_raw(htab, head, hash, key, key_size, htab->n_buck= ets); =20 return l; } =20 static void *htab_map_lookup_elem(struct bpf_map *map, void *key) { + struct bpf_htab *htab =3D container_of(map, struct bpf_htab, map); struct htab_elem *l =3D __htab_map_lookup_elem(map, key); =20 if (l) - return htab_elem_value(l, map->key_size); + return htab_elem_value(htab, l); =20 return NULL; } @@ -775,6 +832,7 @@ static void *htab_map_lookup_elem(struct bpf_map *map, = void *key) */ static int htab_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_= buf) { + struct bpf_htab *htab =3D container_of(map, struct bpf_htab, map); struct bpf_insn *insn =3D insn_buf; const int ret =3D BPF_REG_0; =20 @@ -783,7 +841,7 @@ static int htab_map_gen_lookup(struct bpf_map *map, str= uct bpf_insn *insn_buf) *insn++ =3D BPF_EMIT_CALL(__htab_map_lookup_elem); *insn++ =3D BPF_JMP_IMM(BPF_JEQ, ret, 0, 1); *insn++ =3D BPF_ALU64_IMM(BPF_ADD, ret, - offsetof(struct htab_elem, key) + + htab->key_offset + round_up(map->key_size, 8)); return insn - insn_buf; } @@ -791,12 +849,13 @@ static int htab_map_gen_lookup(struct bpf_map *map, s= truct bpf_insn *insn_buf) static __always_inline void *__htab_lru_map_lookup_elem(struct bpf_map *ma= p, void *key, const bool mark) { + struct bpf_htab *htab =3D container_of(map, struct bpf_htab, map); struct htab_elem *l =3D __htab_map_lookup_elem(map, key); =20 if (l) { if (mark) - bpf_lru_node_set_ref(&l->lru_node); - return htab_elem_value(l, map->key_size); + bpf_lru_node_set_ref(htab_elem_lru_node(l)); + return htab_elem_value(htab, l); } =20 return NULL; @@ -815,6 +874,7 @@ static void *htab_lru_map_lookup_elem_sys(struct bpf_ma= p *map, void *key) static int htab_lru_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf) { + struct bpf_htab *htab =3D container_of(map, struct bpf_htab, map); struct bpf_insn *insn =3D insn_buf; const int ret =3D BPF_REG_0; const int ref_reg =3D BPF_REG_1; @@ -824,15 +884,15 @@ static int htab_lru_map_gen_lookup(struct bpf_map *ma= p, *insn++ =3D BPF_EMIT_CALL(__htab_map_lookup_elem); *insn++ =3D BPF_JMP_IMM(BPF_JEQ, ret, 0, 4); *insn++ =3D BPF_LDX_MEM(BPF_B, ref_reg, ret, - offsetof(struct htab_elem, lru_node) + + offsetof(struct htab_elem_lru, lru_node) + offsetof(struct bpf_lru_node, ref)); *insn++ =3D BPF_JMP_IMM(BPF_JNE, ref_reg, 0, 1); *insn++ =3D BPF_ST_MEM(BPF_B, ret, - offsetof(struct htab_elem, lru_node) + + offsetof(struct htab_elem_lru, lru_node) + offsetof(struct bpf_lru_node, ref), 1); *insn++ =3D BPF_ALU64_IMM(BPF_ADD, ret, - offsetof(struct htab_elem, key) + + htab->key_offset + round_up(map->key_size, 8)); return insn - insn_buf; } @@ -844,13 +904,13 @@ static void check_and_cancel_fields(struct bpf_htab *= htab, return; =20 if (htab_is_percpu(htab)) { - void __percpu *pptr =3D htab_elem_get_ptr(elem, htab->map.key_size); + void __percpu *pptr =3D htab_elem_get_ptr(htab, elem); int cpu; =20 for_each_possible_cpu(cpu) bpf_obj_cancel_fields(&htab->map, per_cpu_ptr(pptr, cpu)); } else { - void *map_value =3D htab_elem_value(elem, htab->map.key_size); + void *map_value =3D htab_elem_value(htab, elem); =20 bpf_obj_cancel_fields(&htab->map, map_value); } @@ -869,17 +929,17 @@ static bool htab_lru_map_delete_node(void *arg, struc= t bpf_lru_node *node) struct bucket *b; int ret; =20 - tgt_l =3D container_of(node, struct htab_elem, lru_node); - b =3D __select_bucket(htab, tgt_l->hash); + tgt_l =3D (struct htab_elem *)container_of(node, struct htab_elem_lru, lr= u_node); + b =3D __select_bucket(htab, htab_elem_hash(htab, tgt_l)); head =3D &b->head; =20 ret =3D htab_lock_bucket(b, &flags); if (ret) return false; =20 - hlist_nulls_for_each_entry_rcu(l, n, head, hash_node) + hlist_nulls_for_each_entry_rcu(l, n, head, node.hash_node) if (l =3D=3D tgt_l) { - hlist_nulls_del_rcu(&l->hash_node); + hlist_nulls_del_rcu(&l->node.hash_node); bpf_map_dec_elem_count(&htab->map); break; } @@ -912,18 +972,19 @@ static int htab_map_get_next_key(struct bpf_map *map,= void *key, void *next_key) head =3D select_bucket(htab, hash); =20 /* lookup the key */ - l =3D lookup_nulls_elem_raw(head, hash, key, key_size, htab->n_buckets); + l =3D lookup_nulls_elem_raw(htab, head, hash, key, key_size, htab->n_buck= ets); =20 if (!l) goto find_first_elem; =20 /* key was found, get next key in the same bucket */ - next_l =3D hlist_nulls_entry_safe(rcu_dereference_raw(hlist_nulls_next_rc= u(&l->hash_node)), - struct htab_elem, hash_node); + next_l =3D hlist_nulls_entry_safe( + rcu_dereference_raw(hlist_nulls_next_rcu(&l->node.hash_node)), + struct htab_elem, node.hash_node); =20 if (next_l) { /* if next elem in this hash list is non-zero, just return it */ - memcpy(next_key, next_l->key, key_size); + memcpy(next_key, htab_elem_key(htab, next_l), key_size); return 0; } =20 @@ -938,10 +999,10 @@ static int htab_map_get_next_key(struct bpf_map *map,= void *key, void *next_key) =20 /* pick first element in the bucket */ next_l =3D hlist_nulls_entry_safe(rcu_dereference_raw(hlist_nulls_first_= rcu(head)), - struct htab_elem, hash_node); + struct htab_elem, node.hash_node); if (next_l) { /* if it's not empty, just return it */ - memcpy(next_key, next_l->key, key_size); + memcpy(next_key, htab_elem_key(htab, next_l), key_size); return 0; } } @@ -955,7 +1016,7 @@ static void htab_elem_free(struct bpf_htab *htab, stru= ct htab_elem *l) check_and_cancel_fields(htab, l); =20 if (htab->map.map_type =3D=3D BPF_MAP_TYPE_PERCPU_HASH) - bpf_mem_cache_free(&htab->pcpu_ma, l->ptr_to_pptr); + bpf_mem_cache_free(&htab->pcpu_ma, htab_elem_get_ptr_to_pptr(l)); bpf_mem_cache_free(&htab->ma, l); } =20 @@ -965,7 +1026,7 @@ static void htab_put_fd_value(struct bpf_htab *htab, s= truct htab_elem *l) void *ptr; =20 if (map->ops->map_fd_put_ptr) { - ptr =3D fd_htab_map_get_ptr(map, l); + ptr =3D fd_htab_map_get_ptr(htab, l); map->ops->map_fd_put_ptr(map, ptr, true); } } @@ -1006,7 +1067,7 @@ static void free_htab_elem(struct bpf_htab *htab, str= uct htab_elem *l) if (htab_is_prealloc(htab)) { bpf_map_dec_elem_count(&htab->map); check_and_cancel_fields(htab, l); - pcpu_freelist_push(&htab->freelist, &l->fnode); + pcpu_freelist_push(&htab->freelist, &l->node.fnode); } else { dec_elem_count(htab); htab_elem_free(htab, l); @@ -1097,7 +1158,7 @@ static struct htab_elem *alloc_htab_elem(struct bpf_h= tab *htab, void *key, l =3D __pcpu_freelist_pop(&htab->freelist); if (!l) return ERR_PTR(-E2BIG); - l_new =3D container_of(l, struct htab_elem, fnode); + l_new =3D container_of(l, struct htab_elem, node.fnode); bpf_map_inc_elem_count(&htab->map); } } else { @@ -1117,10 +1178,10 @@ static struct htab_elem *alloc_htab_elem(struct bpf= _htab *htab, void *key, } } =20 - memcpy(l_new->key, key, key_size); + memcpy(htab_elem_key(htab, l_new), key, key_size); if (percpu) { if (prealloc) { - pptr =3D htab_elem_get_ptr(l_new, key_size); + pptr =3D htab_elem_get_ptr(htab, l_new); } else { /* alloc_percpu zero-fills */ void *ptr =3D bpf_mem_cache_alloc(&htab->pcpu_ma); @@ -1130,26 +1191,26 @@ static struct htab_elem *alloc_htab_elem(struct bpf= _htab *htab, void *key, l_new =3D ERR_PTR(-ENOMEM); goto dec_count; } - l_new->ptr_to_pptr =3D ptr; + htab_elem_set_ptr_to_pptr(l_new, ptr); pptr =3D *(void __percpu **)ptr; } =20 pcpu_init_value(htab, pptr, value, onallcpus, map_flags); =20 if (!prealloc) - htab_elem_set_ptr(l_new, key_size, pptr); + htab_elem_set_ptr(htab, l_new, pptr); } else if (fd_htab_map_needs_adjust(htab)) { size =3D round_up(size, 8); - memcpy(htab_elem_value(l_new, key_size), value, size); + memcpy(htab_elem_value(htab, l_new), value, size); } else if (map_flags & BPF_F_LOCK) { copy_map_value_locked(&htab->map, - htab_elem_value(l_new, key_size), + htab_elem_value(htab, l_new), value, false); } else { - copy_map_value(&htab->map, htab_elem_value(l_new, key_size), value); + copy_map_value(&htab->map, htab_elem_value(htab, l_new), value); } =20 - l_new->hash =3D hash; + htab_elem_set_hash(htab, l_new, hash); return l_new; dec_count: dec_elem_count(htab); @@ -1199,7 +1260,7 @@ static long htab_map_update_elem(struct bpf_map *map,= void *key, void *value, if (unlikely(!btf_record_has_field(map->record, BPF_SPIN_LOCK))) return -EINVAL; /* find an element without taking the bucket lock */ - l_old =3D lookup_nulls_elem_raw(head, hash, key, key_size, + l_old =3D lookup_nulls_elem_raw(htab, head, hash, key, key_size, htab->n_buckets); ret =3D check_flags(htab, l_old, map_flags); if (ret) @@ -1207,7 +1268,7 @@ static long htab_map_update_elem(struct bpf_map *map,= void *key, void *value, if (l_old) { /* grab the element lock and update value in place */ copy_map_value_locked(map, - htab_elem_value(l_old, key_size), + htab_elem_value(htab, l_old), value, false); return 0; } @@ -1221,7 +1282,7 @@ static long htab_map_update_elem(struct bpf_map *map,= void *key, void *value, if (ret) return ret; =20 - l_old =3D lookup_elem_raw(head, hash, key, key_size); + l_old =3D lookup_elem_raw(htab, head, hash, key, key_size); =20 ret =3D check_flags(htab, l_old, map_flags); if (ret) @@ -1235,7 +1296,7 @@ static long htab_map_update_elem(struct bpf_map *map,= void *key, void *value, * and update element in place */ copy_map_value_locked(map, - htab_elem_value(l_old, key_size), + htab_elem_value(htab, l_old), value, false); ret =3D 0; goto err; @@ -1252,9 +1313,9 @@ static long htab_map_update_elem(struct bpf_map *map,= void *key, void *value, /* add new element to the head of the list, so that * concurrent search will find it before old elem */ - hlist_nulls_add_head_rcu(&l_new->hash_node, head); + hlist_nulls_add_head_rcu(&l_new->node.hash_node, head); if (l_old) { - hlist_nulls_del_rcu(&l_old->hash_node); + hlist_nulls_del_rcu(&l_old->node.hash_node); =20 /* l_old has already been stashed in htab->extra_elems, cancel * its reusable special fields before it is available for reuse. @@ -1275,7 +1336,7 @@ static void htab_lru_push_free(struct bpf_htab *htab,= struct htab_elem *elem) { check_and_cancel_fields(htab, elem); bpf_map_dec_elem_count(&htab->map); - bpf_lru_push_free(&htab->lru, &elem->lru_node); + bpf_lru_push_free(&htab->lru, htab_elem_lru_node(elem)); } =20 static long htab_lru_map_update_elem(struct bpf_map *map, void *key, void = *value, @@ -1310,13 +1371,13 @@ static long htab_lru_map_update_elem(struct bpf_map= *map, void *key, void *value l_new =3D prealloc_lru_pop(htab, key, hash); if (!l_new) return -ENOMEM; - copy_map_value(&htab->map, htab_elem_value(l_new, map->key_size), value); + copy_map_value(&htab->map, htab_elem_value(htab, l_new), value); =20 ret =3D htab_lock_bucket(b, &flags); if (ret) goto err_lock_bucket; =20 - l_old =3D lookup_elem_raw(head, hash, key, key_size); + l_old =3D lookup_elem_raw(htab, head, hash, key, key_size); =20 ret =3D check_flags(htab, l_old, map_flags); if (ret) @@ -1325,10 +1386,10 @@ static long htab_lru_map_update_elem(struct bpf_map= *map, void *key, void *value /* add new element to the head of the list, so that * concurrent search will find it before old elem */ - hlist_nulls_add_head_rcu(&l_new->hash_node, head); + hlist_nulls_add_head_rcu(&l_new->node.hash_node, head); if (l_old) { - bpf_lru_node_set_ref(&l_new->lru_node); - hlist_nulls_del_rcu(&l_old->hash_node); + bpf_lru_node_set_ref(htab_elem_lru_node(l_new)); + hlist_nulls_del_rcu(&l_old->node.hash_node); } ret =3D 0; =20 @@ -1383,7 +1444,7 @@ static long htab_map_update_elem_in_place(struct bpf_= map *map, void *key, if (ret) return ret; =20 - l_old =3D lookup_elem_raw(head, hash, key, key_size); + l_old =3D lookup_elem_raw(htab, head, hash, key, key_size); =20 ret =3D check_flags(htab, l_old, map_flags); if (ret) @@ -1392,10 +1453,10 @@ static long htab_map_update_elem_in_place(struct bp= f_map *map, void *key, if (l_old) { /* Update value in-place */ if (percpu) { - pcpu_copy_value(htab, htab_elem_get_ptr(l_old, key_size), + pcpu_copy_value(htab, htab_elem_get_ptr(htab, l_old), value, onallcpus, map_flags); } else { - void **inner_map_pptr =3D htab_elem_value(l_old, key_size); + void **inner_map_pptr =3D htab_elem_value(htab, l_old); =20 old_map_ptr =3D *inner_map_pptr; WRITE_ONCE(*inner_map_pptr, *(void **)value); @@ -1407,7 +1468,7 @@ static long htab_map_update_elem_in_place(struct bpf_= map *map, void *key, ret =3D PTR_ERR(l_new); goto err; } - hlist_nulls_add_head_rcu(&l_new->hash_node, head); + hlist_nulls_add_head_rcu(&l_new->node.hash_node, head); } err: htab_unlock_bucket(b, flags); @@ -1456,22 +1517,22 @@ static long __htab_lru_percpu_map_update_elem(struc= t bpf_map *map, void *key, if (ret) goto err_lock_bucket; =20 - l_old =3D lookup_elem_raw(head, hash, key, key_size); + l_old =3D lookup_elem_raw(htab, head, hash, key, key_size); =20 ret =3D check_flags(htab, l_old, map_flags); if (ret) goto err; =20 if (l_old) { - bpf_lru_node_set_ref(&l_old->lru_node); + bpf_lru_node_set_ref(htab_elem_lru_node(l_old)); =20 /* per-cpu hash map can update value in-place */ - pcpu_copy_value(htab, htab_elem_get_ptr(l_old, key_size), + pcpu_copy_value(htab, htab_elem_get_ptr(htab, l_old), value, onallcpus, map_flags); } else { - pcpu_init_value(htab, htab_elem_get_ptr(l_new, key_size), + pcpu_init_value(htab, htab_elem_get_ptr(htab, l_new), value, onallcpus, map_flags); - hlist_nulls_add_head_rcu(&l_new->hash_node, head); + hlist_nulls_add_head_rcu(&l_new->node.hash_node, head); l_new =3D NULL; } ret =3D 0; @@ -1480,7 +1541,7 @@ static long __htab_lru_percpu_map_update_elem(struct = bpf_map *map, void *key, err_lock_bucket: if (l_new) { bpf_map_dec_elem_count(&htab->map); - bpf_lru_push_free(&htab->lru, &l_new->lru_node); + bpf_lru_push_free(&htab->lru, htab_elem_lru_node(l_new)); } return ret; } @@ -1521,9 +1582,9 @@ static long htab_map_delete_elem(struct bpf_map *map,= void *key) if (ret) return ret; =20 - l =3D lookup_elem_raw(head, hash, key, key_size); + l =3D lookup_elem_raw(htab, head, hash, key, key_size); if (l) - hlist_nulls_del_rcu(&l->hash_node); + hlist_nulls_del_rcu(&l->node.hash_node); else ret =3D -ENOENT; =20 @@ -1556,10 +1617,10 @@ static long htab_lru_map_delete_elem(struct bpf_map= *map, void *key) if (ret) return ret; =20 - l =3D lookup_elem_raw(head, hash, key, key_size); + l =3D lookup_elem_raw(htab, head, hash, key, key_size); =20 if (l) - hlist_nulls_del_rcu(&l->hash_node); + hlist_nulls_del_rcu(&l->node.hash_node); else ret =3D -ENOENT; =20 @@ -1581,8 +1642,8 @@ static void delete_all_elements(struct bpf_htab *htab) struct hlist_nulls_node *n; struct htab_elem *l; =20 - hlist_nulls_for_each_entry_safe(l, n, head, hash_node) { - hlist_nulls_del_rcu(&l->hash_node); + hlist_nulls_for_each_entry_safe(l, n, head, node.hash_node) { + hlist_nulls_del_rcu(&l->node.hash_node); htab_elem_free(htab, l); } cond_resched(); @@ -1599,10 +1660,10 @@ static void htab_free_malloced_internal_structs(str= uct bpf_htab *htab) struct hlist_nulls_node *n; struct htab_elem *l; =20 - hlist_nulls_for_each_entry(l, n, head, hash_node) { + hlist_nulls_for_each_entry(l, n, head, node.hash_node) { /* We only free internal structs on uref dropping to zero */ bpf_map_free_internal_structs(&htab->map, - htab_elem_value(l, htab->map.key_size)); + htab_elem_value(htab, l)); } cond_resched_rcu(); } @@ -1697,7 +1758,7 @@ static int __htab_map_lookup_and_delete_elem(struct b= pf_map *map, void *key, if (ret) return ret; =20 - l =3D lookup_elem_raw(head, hash, key, key_size); + l =3D lookup_elem_raw(htab, head, hash, key, key_size); if (!l) { ret =3D -ENOENT; goto out_unlock; @@ -1708,14 +1769,14 @@ static int __htab_map_lookup_and_delete_elem(struct= bpf_map *map, void *key, void __percpu *pptr; int off =3D 0, cpu; =20 - pptr =3D htab_elem_get_ptr(l, key_size); + pptr =3D htab_elem_get_ptr(htab, l); for_each_possible_cpu(cpu) { copy_map_value_long(&htab->map, value + off, per_cpu_ptr(pptr, cpu)); check_and_init_map_value(&htab->map, value + off); off +=3D roundup_value_size; } } else { - void *src =3D htab_elem_value(l, map->key_size); + void *src =3D htab_elem_value(htab, l); =20 if (flags & BPF_F_LOCK) copy_map_value_locked(map, value, src, true); @@ -1724,7 +1785,7 @@ static int __htab_map_lookup_and_delete_elem(struct b= pf_map *map, void *key, /* Zeroing special fields in the temp buffer */ check_and_init_map_value(map, value); } - hlist_nulls_del_rcu(&l->hash_node); + hlist_nulls_del_rcu(&l->node.hash_node); =20 out_unlock: htab_unlock_bucket(b, bflags); @@ -1860,7 +1921,7 @@ __htab_map_lookup_and_delete_batch(struct bpf_map *ma= p, } =20 bucket_cnt =3D 0; - hlist_nulls_for_each_entry_rcu(l, n, head, hash_node) + hlist_nulls_for_each_entry_rcu(l, n, head, node.hash_node) bucket_cnt++; =20 if (bucket_cnt && !locked) { @@ -1897,14 +1958,14 @@ __htab_map_lookup_and_delete_batch(struct bpf_map *= map, if (!locked) goto next_batch; =20 - hlist_nulls_for_each_entry_safe(l, n, head, hash_node) { - memcpy(dst_key, l->key, key_size); + hlist_nulls_for_each_entry_safe(l, n, head, node.hash_node) { + memcpy(dst_key, htab_elem_key(htab, l), key_size); =20 if (is_percpu) { int off =3D 0, cpu; void __percpu *pptr; =20 - pptr =3D htab_elem_get_ptr(l, map->key_size); + pptr =3D htab_elem_get_ptr(htab, l); if (elem_map_flags & BPF_F_CPU) { cpu =3D elem_map_flags >> 32; copy_map_value(&htab->map, dst_val, per_cpu_ptr(pptr, cpu)); @@ -1918,7 +1979,7 @@ __htab_map_lookup_and_delete_batch(struct bpf_map *ma= p, } } } else { - value =3D htab_elem_value(l, key_size); + value =3D htab_elem_value(htab, l); if (is_fd_htab(htab)) { struct bpf_map **inner_map =3D value; =20 @@ -1936,7 +1997,7 @@ __htab_map_lookup_and_delete_batch(struct bpf_map *ma= p, check_and_init_map_value(map, dst_val); } if (do_delete) { - hlist_nulls_del_rcu(&l->hash_node); + hlist_nulls_del_rcu(&l->node.hash_node); =20 /* bpf_lru_push_free() will acquire lru_lock, which * may cause deadlock. See comments in function @@ -1948,7 +2009,7 @@ __htab_map_lookup_and_delete_batch(struct bpf_map *ma= p, * lock being held and it violates the lock rule, so * invoke free_htab_elem() after unlock as well. */ - l->batch_flink =3D node_to_free; + l->node.batch_flink =3D node_to_free; node_to_free =3D l; } dst_key +=3D key_size; @@ -1960,7 +2021,7 @@ __htab_map_lookup_and_delete_batch(struct bpf_map *ma= p, =20 while (node_to_free) { l =3D node_to_free; - node_to_free =3D node_to_free->batch_flink; + node_to_free =3D node_to_free->node.batch_flink; if (is_lru_map) htab_lru_push_free(htab, l); else @@ -2108,8 +2169,8 @@ bpf_hash_map_seq_find_next(struct bpf_iter_seq_hash_m= ap_info *info, /* no update/deletion on this bucket, prev_elem should be still valid * and we won't skip elements. */ - n =3D rcu_dereference_raw(hlist_nulls_next_rcu(&prev_elem->hash_node)); - elem =3D hlist_nulls_entry_safe(n, struct htab_elem, hash_node); + n =3D rcu_dereference_raw(hlist_nulls_next_rcu(&prev_elem->node.hash_nod= e)); + elem =3D hlist_nulls_entry_safe(n, struct htab_elem, node.hash_node); if (elem) return elem; =20 @@ -2125,7 +2186,7 @@ bpf_hash_map_seq_find_next(struct bpf_iter_seq_hash_m= ap_info *info, =20 count =3D 0; head =3D &b->head; - hlist_nulls_for_each_entry_rcu(elem, n, head, hash_node) { + hlist_nulls_for_each_entry_rcu(elem, n, head, node.hash_node) { if (count >=3D skip_elems) { info->bucket_id =3D i; info->skip_elems =3D count; @@ -2183,12 +2244,12 @@ static int __bpf_hash_map_seq_show(struct seq_file = *seq, struct htab_elem *elem) ctx.meta =3D &meta; ctx.map =3D info->map; if (elem) { - ctx.key =3D elem->key; + ctx.key =3D htab_elem_key(info->htab, elem); if (!info->percpu_value_buf) { - ctx.value =3D htab_elem_value(elem, map->key_size); + ctx.value =3D htab_elem_value(info->htab, elem); } else { roundup_value_size =3D round_up(map->value_size, 8); - pptr =3D htab_elem_get_ptr(elem, map->key_size); + pptr =3D htab_elem_get_ptr(info->htab, elem); for_each_possible_cpu(cpu) { copy_map_value_long(map, info->percpu_value_buf + off, per_cpu_ptr(pptr, cpu)); @@ -2292,14 +2353,14 @@ static long bpf_for_each_hash_elem(struct bpf_map *= map, bpf_callback_t callback_ b =3D &htab->buckets[i]; rcu_read_lock(); head =3D &b->head; - hlist_nulls_for_each_entry_safe(elem, n, head, hash_node) { - key =3D elem->key; + hlist_nulls_for_each_entry_safe(elem, n, head, node.hash_node) { + key =3D htab_elem_key(htab, elem); if (is_percpu) { /* current cpu value for percpu map */ - pptr =3D htab_elem_get_ptr(elem, map->key_size); + pptr =3D htab_elem_get_ptr(htab, elem); val =3D this_cpu_ptr(pptr); } else { - val =3D htab_elem_value(elem, map->key_size); + val =3D htab_elem_value(htab, elem); } num_elems++; ret =3D callback_fn((u64)(long)map, (u64)(long)key, @@ -2403,10 +2464,11 @@ const struct bpf_map_ops htab_lru_map_ops =3D { /* Called from eBPF program */ static void *htab_percpu_map_lookup_elem(struct bpf_map *map, void *key) { + struct bpf_htab *htab =3D container_of(map, struct bpf_htab, map); struct htab_elem *l =3D __htab_map_lookup_elem(map, key); =20 if (l) - return this_cpu_ptr(htab_elem_get_ptr(l, map->key_size)); + return this_cpu_ptr(htab_elem_get_ptr(htab, l)); else return NULL; } @@ -2414,6 +2476,7 @@ static void *htab_percpu_map_lookup_elem(struct bpf_m= ap *map, void *key) /* inline bpf_map_lookup_elem() call for per-CPU hashmap */ static int htab_percpu_map_gen_lookup(struct bpf_map *map, struct bpf_insn= *insn_buf) { + struct bpf_htab *htab =3D container_of(map, struct bpf_htab, map); struct bpf_insn *insn =3D insn_buf; =20 if (!bpf_jit_supports_percpu_insn()) @@ -2424,7 +2487,7 @@ static int htab_percpu_map_gen_lookup(struct bpf_map = *map, struct bpf_insn *insn *insn++ =3D BPF_EMIT_CALL(__htab_map_lookup_elem); *insn++ =3D BPF_JMP_IMM(BPF_JEQ, BPF_REG_0, 0, 3); *insn++ =3D BPF_ALU64_IMM(BPF_ADD, BPF_REG_0, - offsetof(struct htab_elem, key) + roundup(map->key_size, 8)); + htab->key_offset + roundup(map->key_size, 8)); *insn++ =3D BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_0, 0); *insn++ =3D BPF_MOV64_PERCPU_REG(BPF_REG_0, BPF_REG_0); =20 @@ -2433,6 +2496,7 @@ static int htab_percpu_map_gen_lookup(struct bpf_map = *map, struct bpf_insn *insn =20 static void *htab_percpu_map_lookup_percpu_elem(struct bpf_map *map, void = *key, u32 cpu) { + struct bpf_htab *htab =3D container_of(map, struct bpf_htab, map); struct htab_elem *l; =20 if (cpu >=3D nr_cpu_ids) @@ -2440,18 +2504,19 @@ static void *htab_percpu_map_lookup_percpu_elem(str= uct bpf_map *map, void *key, =20 l =3D __htab_map_lookup_elem(map, key); if (l) - return per_cpu_ptr(htab_elem_get_ptr(l, map->key_size), cpu); + return per_cpu_ptr(htab_elem_get_ptr(htab, l), cpu); else return NULL; } =20 static void *htab_lru_percpu_map_lookup_elem(struct bpf_map *map, void *ke= y) { + struct bpf_htab *htab =3D container_of(map, struct bpf_htab, map); struct htab_elem *l =3D __htab_map_lookup_elem(map, key); =20 if (l) { - bpf_lru_node_set_ref(&l->lru_node); - return this_cpu_ptr(htab_elem_get_ptr(l, map->key_size)); + bpf_lru_node_set_ref(htab_elem_lru_node(l)); + return this_cpu_ptr(htab_elem_get_ptr(htab, l)); } =20 return NULL; @@ -2459,6 +2524,7 @@ static void *htab_lru_percpu_map_lookup_elem(struct b= pf_map *map, void *key) =20 static void *htab_lru_percpu_map_lookup_percpu_elem(struct bpf_map *map, v= oid *key, u32 cpu) { + struct bpf_htab *htab =3D container_of(map, struct bpf_htab, map); struct htab_elem *l; =20 if (cpu >=3D nr_cpu_ids) @@ -2466,8 +2532,8 @@ static void *htab_lru_percpu_map_lookup_percpu_elem(s= truct bpf_map *map, void *k =20 l =3D __htab_map_lookup_elem(map, key); if (l) { - bpf_lru_node_set_ref(&l->lru_node); - return per_cpu_ptr(htab_elem_get_ptr(l, map->key_size), cpu); + bpf_lru_node_set_ref(htab_elem_lru_node(l)); + return per_cpu_ptr(htab_elem_get_ptr(htab, l), cpu); } =20 return NULL; @@ -2475,6 +2541,7 @@ static void *htab_lru_percpu_map_lookup_percpu_elem(s= truct bpf_map *map, void *k =20 int bpf_percpu_hash_copy(struct bpf_map *map, void *key, void *value, u64 = map_flags) { + struct bpf_htab *htab =3D container_of(map, struct bpf_htab, map); struct htab_elem *l; void __percpu *pptr; int ret =3D -ENOENT; @@ -2494,7 +2561,7 @@ int bpf_percpu_hash_copy(struct bpf_map *map, void *k= ey, void *value, u64 map_fl /* We do not mark LRU map element here in order to not mess up * eviction heuristics when user space does a map walk. */ - pptr =3D htab_elem_get_ptr(l, map->key_size); + pptr =3D htab_elem_get_ptr(htab, l); if (map_flags & BPF_F_CPU) { cpu =3D map_flags >> 32; copy_map_value(map, value, per_cpu_ptr(pptr, cpu)); @@ -2532,6 +2599,7 @@ int bpf_percpu_hash_update(struct bpf_map *map, void = *key, void *value, static void htab_percpu_map_seq_show_elem(struct bpf_map *map, void *key, struct seq_file *m) { + struct bpf_htab *htab =3D container_of(map, struct bpf_htab, map); struct htab_elem *l; void __percpu *pptr; int cpu; @@ -2546,7 +2614,7 @@ static void htab_percpu_map_seq_show_elem(struct bpf_= map *map, void *key, =20 btf_type_seq_show(map->btf, map->btf_key_type_id, key, m); seq_puts(m, ": {\n"); - pptr =3D htab_elem_get_ptr(l, map->key_size); + pptr =3D htab_elem_get_ptr(htab, l); for_each_possible_cpu(cpu) { seq_printf(m, "\tcpu%d: ", cpu); btf_type_seq_show(map->btf, map->btf_value_type_id, @@ -2619,8 +2687,8 @@ static void fd_htab_map_free(struct bpf_map *map) for (i =3D 0; i < htab->n_buckets; i++) { head =3D select_bucket(htab, i); =20 - hlist_nulls_for_each_entry_safe(l, n, head, hash_node) { - void *ptr =3D fd_htab_map_get_ptr(map, l); + hlist_nulls_for_each_entry_safe(l, n, head, node.hash_node) { + void *ptr =3D fd_htab_map_get_ptr(htab, l); =20 map->ops->map_fd_put_ptr(map, ptr, false); } @@ -2705,6 +2773,7 @@ static void *htab_of_map_lookup_elem(struct bpf_map *= map, void *key) static int htab_of_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf) { + struct bpf_htab *htab =3D container_of(map, struct bpf_htab, map); struct bpf_insn *insn =3D insn_buf; const int ret =3D BPF_REG_0; =20 @@ -2713,7 +2782,7 @@ static int htab_of_map_gen_lookup(struct bpf_map *map, *insn++ =3D BPF_EMIT_CALL(__htab_map_lookup_elem); *insn++ =3D BPF_JMP_IMM(BPF_JEQ, ret, 0, 2); *insn++ =3D BPF_ALU64_IMM(BPF_ADD, ret, - offsetof(struct htab_elem, key) + + htab->key_offset + round_up(map->key_size, 8)); *insn++ =3D BPF_LDX_MEM(BPF_DW, ret, ret, 0); =20 @@ -3533,3 +3602,16 @@ const struct bpf_map_ops rhtab_map_ops =3D { .map_btf_id =3D &rhtab_map_btf_ids[0], .iter_seq_info =3D &rhash_iter_seq_info, }; + +size_t bpf_htab_map_meta_size(void) +{ + return sizeof(struct bpf_htab); +} + +void bpf_htab_map_meta_init(struct bpf_map *inner_map_meta, struct bpf_map= *inner_map) +{ + struct bpf_htab *inner_htab_meta =3D container_of(inner_map_meta, struct = bpf_htab, map); + struct bpf_htab *inner_htab =3D container_of(inner_map, struct bpf_htab, = map); + + inner_htab_meta->key_offset =3D inner_htab->key_offset; +} diff --git a/kernel/bpf/map_in_map.c b/kernel/bpf/map_in_map.c index d2cbab4bdf64..b2db5d0c98be 100644 --- a/kernel/bpf/map_in_map.c +++ b/kernel/bpf/map_in_map.c @@ -7,6 +7,15 @@ =20 #include "map_in_map.h" =20 +static bool bpf_map_type_is_htab(enum bpf_map_type type) +{ + return type =3D=3D BPF_MAP_TYPE_HASH || + type =3D=3D BPF_MAP_TYPE_PERCPU_HASH || + type =3D=3D BPF_MAP_TYPE_LRU_HASH || + type =3D=3D BPF_MAP_TYPE_LRU_PERCPU_HASH || + type =3D=3D BPF_MAP_TYPE_HASH_OF_MAPS; +} + struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd) { struct bpf_map *inner_map, *inner_map_meta; @@ -29,6 +38,8 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd) /* In some cases verifier needs to access beyond just base map. */ if (inner_map->ops =3D=3D &array_map_ops || inner_map->ops =3D=3D &percpu= _array_map_ops) inner_map_meta_size =3D sizeof(struct bpf_array); + else if (bpf_map_type_is_htab(inner_map->map_type)) + inner_map_meta_size =3D bpf_htab_map_meta_size(); =20 inner_map_meta =3D kzalloc(inner_map_meta_size, GFP_USER); if (!inner_map_meta) @@ -70,6 +81,8 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd) inner_array_meta->index_mask =3D inner_array->index_mask; inner_array_meta->elem_size =3D inner_array->elem_size; inner_map_meta->bypass_spec_v1 =3D inner_map->bypass_spec_v1; + } else if (bpf_map_type_is_htab(inner_map->map_type)) { + bpf_htab_map_meta_init(inner_map_meta, inner_map); } return inner_map_meta; } diff --git a/kernel/bpf/map_in_map.h b/kernel/bpf/map_in_map.h index 7d61602354de..1edc205772d8 100644 --- a/kernel/bpf/map_in_map.h +++ b/kernel/bpf/map_in_map.h @@ -15,5 +15,7 @@ void *bpf_map_fd_get_ptr(struct bpf_map *map, struct file= *map_file, int ufd); void bpf_map_fd_put_ptr(struct bpf_map *map, void *ptr, bool need_defer); u32 bpf_map_fd_sys_lookup_elem(void *ptr); +size_t bpf_htab_map_meta_size(void); +void bpf_htab_map_meta_init(struct bpf_map *inner_map_meta, struct bpf_map= *inner_map); =20 #endif --=20 2.55.0.691.gc56d675ccc-goog From nobody Tue Sep 29 04:11:28 2026 Received: from mail-pf1-f200.google.com (mail-pf1-f200.google.com [209.85.210.200]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 89BF133936B for ; Wed, 12 Aug 2026 23:19:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.200 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786576759; cv=none; b=l37au43qdOxe0N9VBdCqI2j4He+QO9au7McLJqtSmWXh+UIgGxzmNT7A/Q91eNEELTBE+qWXNNAOQhx3RkvsSu0IOeHindR7X+L0WA632FvfbWuv1EJY+PhYfQrNHTzG5xCezWzFReEq7/iggwL//U+dovPbM4EyX7c6vm9FAvA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1786576759; c=relaxed/simple; bh=mVTz+gLZlC8oAO3HfC02JAP/xHhCOTZ+M3hJiiDmago=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=tFk1GTorRyFOMtH6/+dsXOu9/L0Ko2LY75HDtfNHk9Vg4V5Wg+S6clSsg5uPEJ9bKRLORuuitjYmQykxtkKwpuRn9PafKyAke6GJTdNTySS9fsoPjV8nVatOreL1a9hs7ST2v524p1xrOYnqKw0H98e/2spDElEiIUKLLpkPi9Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--tjmercier.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Vj/YzMQ8; arc=none smtp.client-ip=209.85.210.200 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--tjmercier.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Vj/YzMQ8" Received: by mail-pf1-f200.google.com with SMTP id d2e1a72fcca58-84f7002f9afso1525439b3a.3 for ; Wed, 12 Aug 2026 16:19:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1786576757; x=1787181557; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=OvDfjx6OQ5cKkEqIXyG7fMSI0eq9VGCdFM9eAmAkcCc=; b=Vj/YzMQ8PQuorif4+7rHGLdpTYYzBu6GsNixUaKsBl+QOfX/E2ydpcim5eEzb0jTLP rRyKAWpxkjaQ3wKdScOdGC3/8TPWbxYFcz+Rz2VUxkZwzJ55n5oBMe94MTkfnKgnoEGr Wsupm+K94+lwnLC68SXQth1pjC2JY83CD+lCGKrANtYjpgl6WUkfK6tX+TIUOdVVULOX hcE/YT3BTExTVjR2z4prC1d4CikocoKavf3oaAhuTu118nmpQmTTqtPJpdOsh1+nu9tH TIsSeJy8O8/xhowgPs4dWXFWd4gD60bZsh4jpyA5KLMbAkNjcvGLiCVlO3tSfafuUysC TWHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1786576757; x=1787181557; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=OvDfjx6OQ5cKkEqIXyG7fMSI0eq9VGCdFM9eAmAkcCc=; b=GGzM7z37/hZXkthKjhJyTGnNzWuyT7ywYrhIeH5czHOzXrGiXzmfw8iBHZ5VP2Bd7z YjTTx1/BP0JPxCU1NLEE8XUXWAOJIUfMBDs9bsK4RzJfzWHb5Avo2NHubeyho5Tw9d04 Tcpu0QfsyYqBnoChCJxk+ezSapL/Omz7vID4xXSPPF2H4oi3Z9u6YBtwH4l9O6TxAdLL TvO+YULQIyODc+vG5jkFthFwS0rknUj7KaltpC/oRfLp2or+1GZKVQDyCOvmkRCTBzEF PoR1+y5wr90aCJ1gKTsuQHK9S8+2cutH1Ui9wA4q1ic6NrCUXxuXZJCk0e5WkFKJQlem XyGg== X-Forwarded-Encrypted: i=1; AHgh+RrPDaDlzV9NeS21ZahteDKo4tIxirfpfkDXDkJ6plL/g7Krth35s3UM2IMVNfsKw5GabVjy1UtYKqjzh/E=@vger.kernel.org X-Gm-Message-State: AOJu0Yz+DV2c3PldfYiFQCTMKziRzR2xxYsuXog12+tOGsUfQbnuhQn4 3qQJFrpZwLs5PQFtU9dEutDH2QkNHhQl0cZRWtJs8sTsPkiaofXWcRUAwJzeWhAugu1cNu2bAYW cqs62kcYveVNQUc82ZA== X-Received: from pfbhy7.prod.google.com ([2002:a05:6a00:6a07:b0:848:41c3:4371]) (user=tjmercier job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:9a9:b0:848:2e53:3f63 with SMTP id d2e1a72fcca58-84fc7514f14mr1307649b3a.21.1786576756803; Wed, 12 Aug 2026 16:19:16 -0700 (PDT) Date: Wed, 12 Aug 2026 16:19:05 -0700 In-Reply-To: <20260812231905.2956588-1-tjmercier@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260812231905.2956588-1-tjmercier@google.com> X-Mailer: git-send-email 2.55.0.691.gc56d675ccc-goog Message-ID: <20260812231905.2956588-3-tjmercier@google.com> Subject: [PATCH bpf-next v4 2/2] bpf: htab: Reduce elem_size by 8 bytes for small key sizes From: "T.J. Mercier" To: ast@kernel.org, daniel@iogearbox.net, andrii@kernel.org, eddyz87@gmail.com, memxor@gmail.com, martin.lau@linux.dev, song@kernel.org, yonghong.song@linux.dev, jolsa@kernel.org, emil@etsalapatis.com, mykyta.yatsenko5@gmail.com Cc: bpf@vger.kernel.org, linux-kernel@vger.kernel.org, "T.J. Mercier" Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" For standard and PCPU (non-LRU) hash maps with small key sizes (less than or equal to the word size), comparing keys requires only a single instruction. Storing a cached 32-bit hash value to shortcut full key comparisons provides no performance advantage for small keys, and consumes memory for every element. This memory can be saved by eliminating hash along with its associated 4 byte padding before the key, reducing the elem_size (and key_offset) by 8 bytes for standard and PCPU maps. Introduce htab_has_hash() to check whether a map requires a cached hash field. Update htab_elem_set_hash(), lookup_elem_raw(), and lookup_nulls_elem_raw() to conditionally bypass hash checking and storage when htab_has_hash() is false. Together with the previous patch, this reduces the minimum standard and preallocated hash map element size from 64 bytes down to 32 bytes, and non-preallocated per-CPU element size from 64 bytes down to 40 bytes. Signed-off-by: T.J. Mercier --- kernel/bpf/hashtab.c | 104 +++++++++++++----- .../selftests/bpf/progs/map_ptr_kern.c | 2 +- 2 files changed, 76 insertions(+), 30 deletions(-) diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index fed4b1db8b78..720766121ea2 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -100,6 +100,7 @@ struct bpf_htab { struct percpu_counter pcount; atomic_t count; bool use_percpu_counter; + bool has_hash; u32 n_buckets; /* number of hash buckets */ u32 elem_size; /* size of each element in bytes */ u32 key_offset; /* offset of key in bytes */ @@ -123,15 +124,13 @@ struct htab_node { =20 struct htab_elem { struct htab_node node; - u32 hash __aligned(8); - char key[] __aligned(8); + u8 data[] __aligned(8); }; =20 struct htab_elem_lru { struct htab_node node; struct bpf_lru_node lru_node; - u32 hash __aligned(8); - char key[] __aligned(8); + u8 data[] __aligned(8); }; =20 /* Only for non-preallocated PCPU maps. Preallocated PCPU maps don't need @@ -140,13 +139,13 @@ struct htab_elem_lru { struct htab_elem_pcpu { struct htab_node node; void *ptr_to_pptr; - u32 hash __aligned(8); - char key[] __aligned(8); + u8 data[] __aligned(8); }; =20 struct htab_btf_record { struct btf_record *record; u32 key_size; + u32 key_offset; }; =20 static inline bool htab_is_prealloc(const struct bpf_htab *htab) @@ -247,6 +246,11 @@ static struct htab_elem *get_htab_elem(struct bpf_htab= *htab, int i) return (struct htab_elem *) (htab->elems + i * (u64)htab->elem_size); } =20 +static bool htab_has_hash(const struct bpf_htab *htab) +{ + return htab->has_hash; +} + static u32 htab_elem_hash(struct bpf_htab *htab, struct htab_elem *l) { return *(u32 *)(htab_elem_key(htab, l) - 8); @@ -254,7 +258,8 @@ static u32 htab_elem_hash(struct bpf_htab *htab, struct= htab_elem *l) =20 static void htab_elem_set_hash(struct bpf_htab *htab, struct htab_elem *l,= u32 hash) { - *(u32 *)(htab_elem_key(htab, l) - 8) =3D hash; + if (htab_has_hash(htab)) + *(u32 *)(htab_elem_key(htab, l) - 8) =3D hash; } =20 /* Both percpu and fd htab support in-place update, so no need for @@ -395,7 +400,7 @@ static int prealloc_init(struct bpf_htab *htab) if (htab_is_lru(htab)) err =3D bpf_lru_init(&htab->lru, htab->map.map_flags & BPF_F_NO_COMMON_LRU, - offsetof(struct htab_elem_lru, hash) - + offsetof(struct htab_elem_lru, data) - offsetof(struct htab_elem_lru, lru_node), htab_lru_map_delete_node, htab); @@ -522,7 +527,7 @@ static void htab_mem_dtor(void *obj, void *ctx) if (IS_ERR_OR_NULL(hrec->record)) return; =20 - map_value =3D (void *)elem + sizeof(struct htab_elem) + round_up(hrec->ke= y_size, 8); + map_value =3D (void *)elem + hrec->key_offset + round_up(hrec->key_size, = 8); bpf_obj_free_fields(hrec->record, map_value); } =20 @@ -548,7 +553,7 @@ static void htab_dtor_ctx_free(void *ctx) } =20 static int bpf_ma_set_dtor(struct bpf_map *map, struct bpf_mem_alloc *ma, - void (*dtor)(void *, void *)) + void (*dtor)(void *, void *), u32 key_offset) { struct htab_btf_record *hrec; int err; @@ -561,6 +566,7 @@ static int bpf_ma_set_dtor(struct bpf_map *map, struct = bpf_mem_alloc *ma, if (!hrec) return -ENOMEM; hrec->key_size =3D map->key_size; + hrec->key_offset =3D key_offset; hrec->record =3D btf_record_dup(map->record); if (IS_ERR(hrec->record)) { err =3D PTR_ERR(hrec->record); @@ -583,9 +589,9 @@ static int htab_map_check_btf(struct bpf_map *map, cons= t struct btf *btf, * populated in htab_map_alloc(), so it will always appear as NULL. */ if (htab_is_percpu(htab)) - return bpf_ma_set_dtor(map, &htab->pcpu_ma, htab_pcpu_mem_dtor); + return bpf_ma_set_dtor(map, &htab->pcpu_ma, htab_pcpu_mem_dtor, htab->ke= y_offset); else - return bpf_ma_set_dtor(map, &htab->ma, htab_mem_dtor); + return bpf_ma_set_dtor(map, &htab->ma, htab_mem_dtor, htab->key_offset); } =20 static struct bpf_map *htab_map_alloc(union bpf_attr *attr) @@ -608,6 +614,13 @@ static struct bpf_map *htab_map_alloc(union bpf_attr *= attr) =20 bpf_map_init_from_attr(&htab->map, attr); =20 + /* Avoid hash memory use and comparisons where unnecessary. + * u32 hash reads are always atomic. If we elide them, key comparisons mu= st also be atomic + * to avoid false positive key matches due to torn key reads / writes. Th= is is only possible + * when the key fits within a word, so check key_size. + */ + htab->has_hash =3D htab_is_lru(htab) || htab->map.key_size > sizeof(unsig= ned long); + if (percpu_lru) { /* ensure each CPU's lru list has >=3D1 elements. * since we are at it, make each lru list has the same @@ -630,11 +643,13 @@ static struct bpf_map *htab_map_alloc(union bpf_attr = *attr) htab->n_buckets =3D roundup_pow_of_two(htab->map.max_entries); =20 if (htab_is_lru(htab)) - htab->key_offset =3D offsetof(struct htab_elem_lru, key); + htab->key_offset =3D offsetof(struct htab_elem_lru, data) + 8; else if (percpu && !prealloc) - htab->key_offset =3D offsetof(struct htab_elem_pcpu, key); + htab->key_offset =3D offsetof(struct htab_elem_pcpu, data) + + (htab_has_hash(htab) ? 8 : 0); else - htab->key_offset =3D offsetof(struct htab_elem, key); + htab->key_offset =3D offsetof(struct htab_elem, data) + + (htab_has_hash(htab) ? 8 : 0); =20 htab->elem_size =3D htab->key_offset + round_up(htab->map.key_size, 8); if (percpu) @@ -743,22 +758,44 @@ static inline struct hlist_nulls_head *select_bucket(= struct bpf_htab *htab, u32 return &__select_bucket(htab, hash)->head; } =20 -/* this lookup function can only be called with bucket lock taken */ -static struct htab_elem *lookup_elem_raw(struct bpf_htab *htab, - struct hlist_nulls_head *head, u32 hash, - void *key, u32 key_size) +static struct htab_elem *__lookup_elem_raw(struct bpf_htab *htab, + struct hlist_nulls_head *head, + u32 hash, void *key, u32 key_size, + struct hlist_nulls_node **out_n) { struct hlist_nulls_node *n; struct htab_elem *l; =20 - hlist_nulls_for_each_entry_rcu(l, n, head, node.hash_node) - if (htab_elem_hash(htab, l) =3D=3D hash && - !memcmp(htab_elem_key(htab, l), key, key_size)) - return l; + if (htab_has_hash(htab)) { + hlist_nulls_for_each_entry_rcu(l, n, head, node.hash_node) + if (htab_elem_hash(htab, l) =3D=3D hash && + !memcmp(htab_elem_key(htab, l), key, key_size)) + return l; + } else { + /* When hash is omitted, key comparisons must be atomic. Zero extend + * the caller's key to the word size to support an atomic compare. + */ + unsigned long k =3D 0; =20 + memcpy(&k, key, key_size); + hlist_nulls_for_each_entry_rcu(l, n, head, node.hash_node) + if (READ_ONCE(*(unsigned long *)htab_elem_key(htab, l)) =3D=3D k) + return l; + } + + if (out_n) + *out_n =3D n; return NULL; } =20 +/* this lookup function can only be called with bucket lock taken */ +static struct htab_elem *lookup_elem_raw(struct bpf_htab *htab, + struct hlist_nulls_head *head, u32 hash, + void *key, u32 key_size) +{ + return __lookup_elem_raw(htab, head, hash, key, key_size, NULL); +} + /* can be called without bucket lock. it will repeat the loop in * the unlikely event when elements moved from one bucket into another * while link list is being walked @@ -772,10 +809,9 @@ static struct htab_elem *lookup_nulls_elem_raw(struct = bpf_htab *htab, struct htab_elem *l; =20 again: - hlist_nulls_for_each_entry_rcu(l, n, head, node.hash_node) - if (htab_elem_hash(htab, l) =3D=3D hash && - !memcmp(htab_elem_key(htab, l), key, key_size)) - return l; + l =3D __lookup_elem_raw(htab, head, hash, key, key_size, &n); + if (l) + return l; =20 if (unlikely(get_nulls_value(n) !=3D (hash & (n_buckets - 1)))) goto again; @@ -1178,7 +1214,17 @@ static struct htab_elem *alloc_htab_elem(struct bpf_= htab *htab, void *key, } } =20 - memcpy(htab_elem_key(htab, l_new), key, key_size); + if (htab_has_hash(htab)) { + memcpy(htab_elem_key(htab, l_new), key, key_size); + } else { + /* Zero-extend key into k for an atomic write to support + * lockless RCU readers. + */ + unsigned long k =3D 0; + + memcpy(&k, key, key_size); + WRITE_ONCE(*(unsigned long *)htab_elem_key(htab, l_new), k); + } if (percpu) { if (prealloc) { pptr =3D htab_elem_get_ptr(htab, l_new); @@ -3181,7 +3227,7 @@ static int rhtab_map_check_btf(struct bpf_map *map, c= onst struct btf *btf, { struct bpf_rhtab *rhtab =3D container_of(map, struct bpf_rhtab, map); =20 - return bpf_ma_set_dtor(map, &rhtab->ma, rhtab_mem_dtor); + return bpf_ma_set_dtor(map, &rhtab->ma, rhtab_mem_dtor, offsetof(struct r= htab_elem, data)); } =20 static void rhtab_map_free_internal_structs(struct bpf_map *map) diff --git a/tools/testing/selftests/bpf/progs/map_ptr_kern.c b/tools/testi= ng/selftests/bpf/progs/map_ptr_kern.c index 373c8d17ea55..6bd4cb68c20c 100644 --- a/tools/testing/selftests/bpf/progs/map_ptr_kern.c +++ b/tools/testing/selftests/bpf/progs/map_ptr_kern.c @@ -114,7 +114,7 @@ static inline int check_hash(void) VERIFY(check_default_noinline(&hash->map, map)); =20 VERIFY(hash->n_buckets =3D=3D MAX_ENTRIES); - VERIFY(hash->elem_size =3D=3D 64); + VERIFY(hash->elem_size =3D=3D 32); =20 VERIFY(hash->count.counter =3D=3D 0); VERIFY(bpf_map_sum_elem_count(map) =3D=3D 0); --=20 2.55.0.691.gc56d675ccc-goog