From nobody Sat Jul 25 16:44:51 2026 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 14AFA2FB084 for ; Thu, 16 Jul 2026 00:44:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162643; cv=none; b=t/PEgbQLf05i96/rj4wneQ41cgm5kJzrUXKNSqqENLc1noBURLl03OyM8WIDoQ0rhRs+G4ihml2o5RCswnbbCyd2uyGI/9uKnBVv4pPlM9LwFkqAaB6ZSQW9naKUFh7v0YLprC2wsE5wL+XCNZ/Mn6gIpczwVMw2IO9/CPvtxvc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162643; c=relaxed/simple; bh=brLHTe2cJZX9XkyLF/aYl8/54ZJ9DmYkqSeWd6NM5g4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NkeeQBhY8LHJEJeo84X0jC8rJbMaqMcI12p9zXfOdZSvHjsFohcaUhC7REhTKQXehZSAK/VSqiM2Mwvf3nhZlYPJj+ekJ42h0knOcqij95dV0hMFIcjj3mX24OErMEHm/5AyGGtJhQumhW34R24GOXXyBth190AjJAHVoQl9Fo4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=dDMWsQvj; arc=none smtp.client-ip=209.85.215.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dDMWsQvj" Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-c96b08cdd1cso4092874a12.0 for ; Wed, 15 Jul 2026 17:44:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784162641; x=1784767441; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=1o+KziW6b/GVugCs/B2MAgSMCWqeQ8TakkxiOE3NICQ=; b=dDMWsQvjVwXce8kGohzHSzQewAopEHHb93ifkFkqSTGBYf08MOUUfQAm2Lmd8PHeWs sRe4DgYXMFHu+mmDsoucgpiWeQhOlInleCpZHoZHf/6HcxTSoQotzMGOSyb+24tXhP6/ Fb9tD9HKm8HyNCjc7LJrkvIlLuXKpVwEPMrGt9MCC62tsWq/48tMmLrAS6MsdWvnyTi6 2orsKzYiLtihAP8/fA8777nBy8PO+zbeWz9g+bT4NejucKiQOdaSECevADg6qESBYKqj uh50yxkleH6M3EqIrPIXrZ0YILWP7tP8eLrQTYm+JDWbVOVpg4QwToWKXVYnO4cMv/8/ C8TQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784162641; x=1784767441; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=1o+KziW6b/GVugCs/B2MAgSMCWqeQ8TakkxiOE3NICQ=; b=ZzGsHtX16nR3G0vdxJc/T8lZU/htpEGnkXoL/LGOLppmtzQ47MRS5vEbdAi8wP40Op iWkJUHqBp9bYTwxxpi3c/6N4bfZboArxK9VQGH3bHp4G+mk6xtzBqUbhEWe5GD0L9UTq 6waXOvo92b6XqUv1xF+gAoV8hSRttN66MtJYjM8HMh3JMgWKFj8bURsStH9+JZSzvEhy 71qaxDCv9DjqwYO/2bhC+WjjhjycSAHi8Aqq7c8oOkktGTpJOwVHx6GppBPlRV1ofVx0 aTck8oX9SPlQDhAM7tJ2+Dn2em93P0xt3AUdgSIcITKhcmpSaDdEoPJVRDyCqwiy4nXt MzkA== X-Forwarded-Encrypted: i=1; AHgh+RoFhw4doyS7s18bZqg+1zXZvzpksadzVApbjJQw68j3S4XBwQYOmS+sVX+VQ3mEKfKGdjJbDufoet0C6rU=@vger.kernel.org X-Gm-Message-State: AOJu0Yz+YR+Uk4Td/2eVP28hDFS74+e2ZMCK1YLlIqNRQZMWKgSkuJzZ YWmsA9IFb0phvwxM/3huIuf/VHWBkRP38sUhheoF9eKivjPuYIC4gzMD X-Gm-Gg: AfdE7cl7ho1ilAdVvE7bQVTLzN3l6JmBfMdeOvEzH1rHe3hVWSfHVfoecQ6cjxVzRnZ T/W5Kgo+V5rUdMn9aHOCdjOuSVIuwOOKdk0TVOLtvUk2WH1klCRsOyfvKZC+B8WPIMic0yaavDe O9cUpbgVOi2B8uUVurMqz2MDPECe5T1po81HCCa65Yqz6Hvcu8QZFYgWeOOQjj3qpGZuSTxuR8D sT4rdwmoQ9/wT5BvNLR0NZr7H4RaWXKW5ssT5MtL6ZkoxGjnLhj4Fnp/kBoPFjsiS65nYEX9Iqc e8V4WAIDFzD6plsPMxKg7b5sYF9Y6+JOAXnV+RVeB3DFVDwMs/tuuzRfGEdguHYLb7nzLylRn9G cVO9Sle70yjNUqK0Uhh4B/EqHMhnsr92JIKF8utsOLCMEuifJLdvDeTg+1rulAuFDqdJ13BReLI CQQFFcsMI39st48Rl0jGrl4LmXXVOYTnBCN6goobvrlNMtr0DqXTs8wIi2huDKVrRxyM3CNAXlk 7G4DM1cGg== X-Received: by 2002:a05:6a20:6a09:b0:3c1:b9c:4890 with SMTP id adf61e73a8af0-3c36c1957b7mr5208428637.22.1784162641314; Wed, 15 Jul 2026 17:44:01 -0700 (PDT) Received: from l.localdomain ([76.166.220.66]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-13cd4284a92sm4107620c88.4.2026.07.15.17.43.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 17:44:00 -0700 (PDT) From: Dave Seddon To: netdev@vger.kernel.org Cc: "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Stanislav Fomichev , Tom Herbert , Willem de Bruijn , linux-kernel@vger.kernel.org, Dave Seddon Subject: [PATCH net-next v1 01/11] net: flow_dissector: gate BPF program lookup behind a static key Date: Wed, 15 Jul 2026 17:43:47 -0700 Message-ID: <20260716004357.3652679-2-dave.seddon.ca@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> References: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" __skb_flow_dissect() takes rcu_read_lock(), resolves the netns from skb->dev/skb->sk, and does two dependent rcu_dereference() loads of the netns-BPF run_array (init_net's first - a shared cacheline - then the packet netns') on every dissect, even though the overwhelmingly common case is that no BPF flow dissector program is attached anywhere on the system. Add a static key that counts attached flow dissector programs and skip the whole lookup block when it is zero, exactly as bpf_sk_lookup_enabled does for the sibling NETNS_BPF_SK_LOOKUP attach type (also see cgroup_bpf_enabled_key[] for the same pattern per cgroup-BPF attach type). The sk_lookup type is link-only, so its inc/dec sit only in netns_bpf_attach_type_need()/unneed(), which the link attach/release paths already call. The flow dissector type also supports legacy BPF_PROG_ATTACH, so additionally inc on a fresh (non-replacement) prog attach, dec in __netns_bpf_prog_detach(), and dec for a remaining legacy prog in netns_bpf_pernet_pre_exit() (links are already handled by the existing per-link unneed loop there). All these paths run under netns_bpf_mutex. As with sk_lookup, there is a brief window where a just-attached program can miss a concurrent dissect (key flips after rcu_assign_pointer) or a just-detached one can cause a wasted lookup; both are benign and match the existing semantics of the sibling keys. DEBUG_NET_WARN_ON_ONCE(!net) moves inside the gated block: it exists to catch a missing netns for the BPF hook, which is only meaningful when a program can actually be attached. With the key disabled this removes the rcu_read_lock/unlock pair, the netns resolution, and both run_array loads from the per-packet path. Assisted-by: Claude:claude-fable-5 sparse smatch Signed-off-by: Dave Seddon --- include/linux/skbuff.h | 1 + kernel/bpf/net_namespace.c | 17 +++++- net/core/flow_dissector.c | 104 +++++++++++++++++++++---------------- 3 files changed, 75 insertions(+), 47 deletions(-) diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 22eda1d54a0e..6448c59dc807 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -1588,6 +1588,7 @@ void skb_flow_dissector_init(struct flow_dissector *f= low_dissector, unsigned int key_count); =20 struct bpf_flow_dissector; +extern struct static_key_false netns_bpf_flow_dissector_enabled; u32 bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx, __be16 proto, int nhoff, int hlen, unsigned int flags); =20 diff --git a/kernel/bpf/net_namespace.c b/kernel/bpf/net_namespace.c index 25f30f9edaef..0d48ea65697e 100644 --- a/kernel/bpf/net_namespace.c +++ b/kernel/bpf/net_namespace.c @@ -28,6 +28,9 @@ DEFINE_MUTEX(netns_bpf_mutex); static void netns_bpf_attach_type_unneed(enum netns_bpf_attach_type type) { switch (type) { + case NETNS_BPF_FLOW_DISSECTOR: + static_branch_dec(&netns_bpf_flow_dissector_enabled); + break; #ifdef CONFIG_INET case NETNS_BPF_SK_LOOKUP: static_branch_dec(&bpf_sk_lookup_enabled); @@ -41,6 +44,9 @@ static void netns_bpf_attach_type_unneed(enum netns_bpf_a= ttach_type type) static void netns_bpf_attach_type_need(enum netns_bpf_attach_type type) { switch (type) { + case NETNS_BPF_FLOW_DISSECTOR: + static_branch_inc(&netns_bpf_flow_dissector_enabled); + break; #ifdef CONFIG_INET case NETNS_BPF_SK_LOOKUP: static_branch_inc(&bpf_sk_lookup_enabled); @@ -352,6 +358,11 @@ int netns_bpf_prog_attach(const union bpf_attr *attr, = struct bpf_prog *prog) net->bpf.progs[type] =3D prog; if (attached) bpf_prog_put(attached); + else + /* Mark attach point as used on a fresh attach; a + * replacement keeps the existing count. + */ + netns_bpf_attach_type_need(type); =20 out_unlock: mutex_unlock(&netns_bpf_mutex); @@ -373,6 +384,8 @@ static int __netns_bpf_prog_detach(struct net *net, attached =3D net->bpf.progs[type]; if (!attached || attached !=3D old) return -ENOENT; + /* Mark attach point as unused */ + netns_bpf_attach_type_unneed(type); netns_bpf_run_array_detach(net, type); net->bpf.progs[type] =3D NULL; bpf_prog_put(attached); @@ -546,8 +559,10 @@ static void __net_exit netns_bpf_pernet_pre_exit(struc= t net *net) net_link->net =3D NULL; /* auto-detach link */ netns_bpf_attach_type_unneed(type); } - if (net->bpf.progs[type]) + if (net->bpf.progs[type]) { + netns_bpf_attach_type_unneed(type); bpf_prog_put(net->bpf.progs[type]); + } } mutex_unlock(&netns_bpf_mutex); } diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c index 8aa4f9b4df81..b8c59209b460 100644 --- a/net/core/flow_dissector.c +++ b/net/core/flow_dissector.c @@ -73,6 +73,14 @@ void skb_flow_dissector_init(struct flow_dissector *flow= _dissector, } EXPORT_SYMBOL(skb_flow_dissector_init); =20 +/* Enabled while any netns has a BPF flow dissector program attached; + * lets __skb_flow_dissect() skip the program lookup entirely in the + * common no-program case. Maintained by the netns-BPF attach/detach + * paths in kernel/bpf/net_namespace.c. + */ +DEFINE_STATIC_KEY_FALSE(netns_bpf_flow_dissector_enabled); +EXPORT_SYMBOL(netns_bpf_flow_dissector_enabled); + #ifdef CONFIG_BPF_SYSCALL int flow_dissector_bpf_prog_attach_check(struct net *net, struct bpf_prog *prog) @@ -1117,59 +1125,63 @@ bool __skb_flow_dissect(const struct net *net, FLOW_DISSECTOR_KEY_BASIC, target_container); =20 - rcu_read_lock(); + if (static_branch_unlikely(&netns_bpf_flow_dissector_enabled)) { + rcu_read_lock(); =20 - if (skb) { - if (!net) { - if (skb->dev) - net =3D dev_net_rcu(skb->dev); - else if (skb->sk) - net =3D sock_net(skb->sk); + if (skb) { + if (!net) { + if (skb->dev) + net =3D dev_net_rcu(skb->dev); + else if (skb->sk) + net =3D sock_net(skb->sk); + } } - } =20 - DEBUG_NET_WARN_ON_ONCE(!net); - if (net) { - enum netns_bpf_attach_type type =3D NETNS_BPF_FLOW_DISSECTOR; - struct bpf_prog_array *run_array; - - run_array =3D rcu_dereference(init_net.bpf.run_array[type]); - if (!run_array) - run_array =3D rcu_dereference(net->bpf.run_array[type]); - - if (run_array) { - struct bpf_flow_keys flow_keys; - struct bpf_flow_dissector ctx =3D { - .flow_keys =3D &flow_keys, - .data =3D data, - .data_end =3D data + hlen, - }; - __be16 n_proto =3D proto; - struct bpf_prog *prog; - u32 result; - - if (skb) { - ctx.skb =3D skb; - /* we can't use 'proto' in the skb case - * because it might be set to skb->vlan_proto - * which has been pulled from the data - */ - n_proto =3D skb->protocol; - } + DEBUG_NET_WARN_ON_ONCE(!net); + if (net) { + enum netns_bpf_attach_type type =3D NETNS_BPF_FLOW_DISSECTOR; + struct bpf_prog_array *run_array; + + run_array =3D rcu_dereference(init_net.bpf.run_array[type]); + if (!run_array) + run_array =3D rcu_dereference(net->bpf.run_array[type]); + + if (run_array) { + struct bpf_flow_keys flow_keys; + struct bpf_flow_dissector ctx =3D { + .flow_keys =3D &flow_keys, + .data =3D data, + .data_end =3D data + hlen, + }; + __be16 n_proto =3D proto; + struct bpf_prog *prog; + u32 result; + + if (skb) { + ctx.skb =3D skb; + /* we can't use 'proto' in the skb case + * because it might be set to + * skb->vlan_proto which has been pulled + * from the data + */ + n_proto =3D skb->protocol; + } =20 - prog =3D READ_ONCE(run_array->items[0].prog); - result =3D bpf_flow_dissect(prog, &ctx, n_proto, nhoff, - hlen, flags); - if (result !=3D BPF_FLOW_DISSECTOR_CONTINUE) { - __skb_flow_bpf_to_target(&flow_keys, flow_dissector, - target_container); - rcu_read_unlock(); - return result =3D=3D BPF_OK; + prog =3D READ_ONCE(run_array->items[0].prog); + result =3D bpf_flow_dissect(prog, &ctx, n_proto, + nhoff, hlen, flags); + if (result !=3D BPF_FLOW_DISSECTOR_CONTINUE) { + __skb_flow_bpf_to_target(&flow_keys, + flow_dissector, + target_container); + rcu_read_unlock(); + return result =3D=3D BPF_OK; + } } } - } =20 - rcu_read_unlock(); + rcu_read_unlock(); + } =20 if (dissector_uses_key(flow_dissector, FLOW_DISSECTOR_KEY_ETH_ADDRS)) { --=20 2.54.0 From nobody Sat Jul 25 16:44:51 2026 Received: from mail-pg1-f177.google.com (mail-pg1-f177.google.com [209.85.215.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 76A6C30B509 for ; Thu, 16 Jul 2026 00:44:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162645; cv=none; b=mDa8LQoTeCE5W/l1zCuEh4yz7Wplal/N9ZLfeds6ajCafW4FjC4PqFo5e2yLAoNQjA90UPm++jzl67Oc9srKb4JXUtKKX6XBdK5O+ajt+14/o2Z4ov+9vxK9OD/qK4diq++/fsOBm/s00DY56XYyD88iq2PLTs0HJ2AoQnrZRvM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162645; c=relaxed/simple; bh=2G6jX/fDUGhuh78H4k/QnWnytFyS22b5IyQpplABToo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=QCaSiFyho5dE81k1Qu5ZRyfXH8JqfMhc+D2WHZBkowVuscCON0H1Ms71zjReKCGWXR2pp3MqN9NZP5zlaIV1RNlGiAtNMHdV3BhJRIMxuQP8E9EpF2DmqNl+sTbWv3E/BCuf6gXKXCGu7rILUgvtMxZ+XVLyM6PnjA1++f5lCPg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=nAnVNOM+; arc=none smtp.client-ip=209.85.215.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nAnVNOM+" Received: by mail-pg1-f177.google.com with SMTP id 41be03b00d2f7-c9e30214d8fso4304315a12.3 for ; Wed, 15 Jul 2026 17:44:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784162643; x=1784767443; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=ielH78VxMdnXNQvVLt3KPq+JAgT1fEDU0jLTYAml4gA=; b=nAnVNOM+RvQVdrqqethTeWkRUg8OBrmX+mapucP2tgEM15iDH7MMMXadR0O9iYLKmS 6S8QmgOdIH+u2jfw1NSRvgJjRXXfDKyqraIGf3yD+SYJeomhuxRGKf0Ho5A3Qd7u36Y5 yZBz5WyiOoT5QHx7kIU7NgepaarzdB8WPPaSIFgBIW449EBrOrsjlna7mFR/zzI6ezqF bRe19Zt3iA1j7TamGuAtJmrWxWE2ig1MGhMP4N1bU0o4xnfPdng4XicZhGKygLzicDe+ I1fhjPm+JMqSBj7tJRLvjiuhmrLnVqZrHXw1rP3//rT94gfGu3B4yjxC7piJucs2PH3V NOmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784162643; x=1784767443; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=ielH78VxMdnXNQvVLt3KPq+JAgT1fEDU0jLTYAml4gA=; b=FVFQRk7RYYWiSpEiSkRO6A8W1pkAMW/Cg+HduF/qWEbuhKKcDIOAgSmK+voq8A1JPo ZRv1E2Q9mVmnJZwmYslbLRmzv6SDVZmj6vNaZDUY+ROGm1UEumghzVdpz9ZmWM5hbF0D eNnFZOQ6sBDEVQ8fy3h02+N5f7rF6U0t1Xgck4Elx7AmsfDLgPIIT4QnpuaH7tE/Ytnk hdMGA1YgKgCrwDucKTbr3+F0ySSH7OlMs2MgayDPDuQ5EruXOovSatvbcBYiCtgXNnrC yeobeVkDaAqFhY+ByTDneCCRU8QdUfPIlPM6yzF272scgJ7vulLtyfPBiFEbnJoG1zFE tAtg== X-Forwarded-Encrypted: i=1; AHgh+RoofYOviic7jXBnNahGjto3N3/dUkINuDj4rD5RmtMlIfVBrKNDnUdr1PRRomlYfBIN/RirzG4FXw+U25I=@vger.kernel.org X-Gm-Message-State: AOJu0YzCbnmrEUFBWdAoFWH9zoncLRMylzW2X8XZUn4AF8ok1ZuBX92C r9EnYFD6630Qy5pUiGtNyG7pA4EuXPlrRtKzAJmSUO1SuABP0Qmj6a/k X-Gm-Gg: AfdE7cn3ekmLbEQ2vksr42GCZydxBpYN5ZZ4nZB2J46aRGMj1NwSTxdAfmI81xIvch2 BSE4xKNr4npCVGGTaykjUQaDCiymIYUvC1XfEzNavkrEL821/c+PqzOZYkrvNcp+fYjvpIUmZbN etulwnMKP8obJR2lB079RiHPRKEDzq4WydN2ZfnhbS7abB9KoNy28XY4bTnxSI6vl2cWSSjLnBf jNlzW2qXha79ozZfWb/um0V5/DzPvrGwfPEn77gVLpvhypyxieJ0OfiHG7DbO/R8D6j/9QFyJY8 yWWka7gaode0UQCjwwOsQ+OfrmVh9gsRg1S80+U4hXf1hX/PC9sgqOcKkXSgNLG277TunwtXxln GOLnlYpbF33/h8aTnGu066S07pydVho5wYrpjKnLEWNKg/ZpUdcZvpUz9Az3T4lX1Z2kpYYAkhr siKBYDdOLqxag+NHwNGqnRyef77b7+M3emHD4YPghyTbGkoFrzJH3wqt49PGBF442SIBYUMcNCN gx1XV5hFA== X-Received: by 2002:a05:6a21:3a42:b0:3c3:8c91:5da0 with SMTP id adf61e73a8af0-3c38c916004mr628308637.15.1784162642618; Wed, 15 Jul 2026 17:44:02 -0700 (PDT) Received: from l.localdomain ([76.166.220.66]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-13cd4284a92sm4107620c88.4.2026.07.15.17.44.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 17:44:02 -0700 (PDT) From: Dave Seddon To: netdev@vger.kernel.org Cc: "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Stanislav Fomichev , Tom Herbert , Willem de Bruijn , linux-kernel@vger.kernel.org, Dave Seddon Subject: [PATCH net-next v1 02/11] net: flow_dissector: opt-in fast-path for eth + IPv{4,6} + {TCP,UDP} Date: Wed, 15 Jul 2026 17:43:48 -0700 Message-ID: <20260716004357.3652679-3-dave.seddon.ca@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> References: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable A small targeted optimisation to __skb_flow_dissect(). A new flow_dissect_fast() dispatcher routes the standard flow_keys dissector calls through per-shape straight-line extractors for the common eth + IPv4/IPv6 + TCP/UDP packet shapes. On any miss, falls back to the existing graph walk. Output is byte-identical to the slow path for the shapes handled. This is the parent of the fast-path series. Subsequent patches add per-shape extensions for single VLAN, QinQ, PPPoE, MPLS, IP-in-IP and GRE. Each patch ships its own per-shape sysctl in the new ``/proc/sys/net/flow_dissector/`` subtree; this patch establishes the subtree and registers the first entry, ``eth_ip``. The fast path is invoked strictly after the netns BPF flow dissector hook: if a BPF program is attached and returns a verdict other than BPF_FLOW_DISSECTOR_CONTINUE, __skb_flow_dissect() returns before flow_dissect_fast() is reached. Attached BPF dissectors therefore always take precedence and observe no behaviour change from this series. - What this does. Adds an opt-in fast-path for eth + IPv4 (IHL=3D5, not fragmented) + TCP/UDP and eth + IPv6 (no extension headers) + TCP/UDP. When the per-shape sysctl is enabled and a packet matches, the helper fills struct flow_keys byte-identically to the slow path and returns. On any miss it defers to the existing graph walk. Only the two standard dissectors are eligible (flow_keys_dissector and flow_keys_dissector_symmetric); custom dissectors and slow-path-only flags defer unconditionally. - How it gates. A new DEFINE_STATIC_KEY_FALSE(flow_dissector_eth_ip_key), externed in , toggled by /proc/sys/net/flow_dissector/eth_ip with proc_do_static_key. When the sysctl is 0 the dispatcher's per-shape case is a forward not- taken JMP =E2=80=94 same slow-path cost as today. When 1, the helper ru= ns inline. - Namespace. The sysctl subtree is /proc/sys/net/flow_dissector/ (registered by a new late_initcall in net/core/flow_dissector.c via register_net_sysctl(&init_net, "net/flow_dissector", tbl) =E2=80=94 same shape as net/mpls, net/bridge). - Forward declarations. flow_dissect_fast_ipv4 / _ipv6 are forward- declared at the top of the file (after includes). Subsequent patches add flow_dissect_fast_vlan etc. that call these helpers; anchoring the declarations at top of file makes the patch insertion order robust against kernel-version line-number drift. - Cost model. On a miss, dispatcher overhead is one ethertype switch (~5 cycles) plus the per-case static_branch test (~not-taken JMP when 0). On a hit, the straight-line body saves ~30 ns/skb of graph-walk cost on the common shape (Zen 2 / Cortex-A76; see the cover letter for the cross-uarch microbench table). - Output contract. Byte-identical with the slow path for the eligible shape. The IPv6 fast-path defers on any non-zero flow label: callers passing FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL -- skb_get_hash(), i.e. RPS/RFS, fq, fq_codel, cake -- make the slow path stop at a non-zero label, and dissectors requesting the label key write it; deferring on the label itself keeps byte-identity for every combination without threading flags into the helpers, and zero-label IPv6 stays on the fast path. Fragmented IPv4 packets defer. Custom dissectors defer, as do FLOW_DISSECTOR_F_* flags other than PARSE_1ST_FRAG and STOP_AT_FLOW_LABEL. Admitting STOP_AT_FLOW_LABEL is what makes the fast path reachable from skb_get_hash() at all -- the kernel's main dissection consumer. Assisted-by: Claude:claude-fable-5 sparse smatch Signed-off-by: Dave Seddon --- Documentation/admin-guide/sysctl/net.rst | 27 ++- include/net/flow_dissector.h | 5 + net/core/flow_dissector.c | 273 ++++++++++++++++++++++- 3 files changed, 302 insertions(+), 3 deletions(-) diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin= -guide/sysctl/net.rst index e586e17fc7a5..aac82169e019 100644 --- a/Documentation/admin-guide/sysctl/net.rst +++ b/Documentation/admin-guide/sysctl/net.rst @@ -460,7 +460,32 @@ historical importance. =20 Default: 0 =20 -2. /proc/sys/net/unix - Parameters for Unix domain sockets +2. /proc/sys/net/flow_dissector - Flow dissector fast-path per-shape sysct= ls +--------------------------------------------------------------------------- + +The flow dissector is invoked on every packet that needs a flow hash +(RPS, RFS, ECMP, sch_cake, cls_flower, ...). For common shapes +(Eth+IPv4/IPv6+TCP/UDP, plus per-shape extensions for VLAN/QinQ, +PPPoE, MPLS, IP-in-IP and GRE added by this series), an opt-in +fast-path bypasses the slow-path graph walk and writes +``struct flow_keys`` directly. Each shape is gated by its own static +key + sysctl so operators enable only what their deployment uses. + +All defaults are 0 (off). Flipping any of these on takes effect +immediately; no kernel rebuild needed. + +eth_ip +~~~~~~ + +Eth + IPv4 (IHL=3D5, no fragmentation) + TCP/UDP, and Eth + IPv6 (no +extension headers) + TCP/UDP. Output is byte-identical to the slow +path on the eligible shape; non-matching packets fall through to the +slow path unchanged. The dispatcher costs one not-taken JMP per call +when this is 0. + +Default: 0 + +3. /proc/sys/net/unix - Parameters for Unix domain sockets ---------------------------------------------------------- =20 There is only one file in this directory. diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h index ced79dc8e856..7d513a6cbb55 100644 --- a/include/net/flow_dissector.h +++ b/include/net/flow_dissector.h @@ -425,6 +425,11 @@ __be32 flow_get_u32_dst(const struct flow_keys *flow); extern struct flow_dissector flow_keys_dissector; extern struct flow_dissector flow_keys_basic_dissector; =20 +/* Per-shape fast-path static-branch keys, toggled via + * /proc/sys/net/flow_dissector/. All default off. + */ +extern struct static_key_false flow_dissector_eth_ip_key; + /* struct flow_keys_digest: * * This structure is used to hold a digest of the full flow keys. This is a diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c index b8c59209b460..88b1bd1cc4ca 100644 --- a/net/core/flow_dissector.c +++ b/net/core/flow_dissector.c @@ -37,6 +37,33 @@ #endif #include =20 +/* Per-shape fast-path gates, one static_branch per shape so operators + * enable only what their deployment carries. All default off. + */ +DEFINE_STATIC_KEY_FALSE(flow_dissector_eth_ip_key); +EXPORT_SYMBOL(flow_dissector_eth_ip_key); + +/* IPv4 version/IHL byte of an option-less header: version 4, IHL 5. */ +#define FLOW_DIS_IPV4_VIHL_NOOPT 0x45 + +/* Fast-path helper forward declarations. */ +static bool flow_dissect_fast_ipv4(const struct sk_buff *skb, + struct flow_dissector *flow_dissector, + void *target_container, + const void *data, + int nhoff, int hlen); +static bool flow_dissect_fast_ipv6(const struct sk_buff *skb, + struct flow_dissector *flow_dissector, + void *target_container, + const void *data, + int nhoff, int hlen); + +/* One of the two dissectors the fast-path eligibility check admits; + * defined here so flow_dissect_fast() below can reference it (its keys + * and init live near the bottom of the file). + */ +static struct flow_dissector flow_keys_dissector_symmetric __read_mostly; + static void dissector_set_key(struct flow_dissector *flow_dissector, enum flow_dissector_key_id key_id) { @@ -1043,6 +1070,223 @@ static bool is_pppoe_ses_hdr_valid(const struct ppp= oe_hdr *hdr) return hdr->ver =3D=3D 1 && hdr->type =3D=3D 1 && hdr->code =3D=3D 0; } =20 +/* The IPv4 fast-path assumes a 20-byte IPv4 header (IHL =3D=3D 5). The + * runtime IHL check below is the dynamic invariant; this static + * assertion is the compile-time one, in the unlikely case struct + * iphdr ever grows. + */ +static_assert(sizeof(struct iphdr) =3D=3D 20); + +/* Straight-line eth + IPv4 (IHL=3D5, not fragmented) + TCP|UDP. Returns + * true when @target_container has been filled byte-identically to the + * slow path; false on any miss (caller falls back to the slow path). + */ +static bool flow_dissect_fast_ipv4(const struct sk_buff *skb, + struct flow_dissector *flow_dissector, + void *target_container, + const void *data, + int nhoff, int hlen) +{ + struct flow_dissector_key_control *key_control; + struct flow_dissector_key_addrs *key_addrs; + struct flow_dissector_key_basic *key_basic; + struct flow_dissector_key_ports *key_ports; + const struct iphdr *iph; + int thoff; + + if (unlikely(hlen - nhoff < (int)sizeof(*iph) + 4)) + return false; + + iph =3D (const struct iphdr *)((const u8 *)data + nhoff); + + if (unlikely(*(const u8 *)iph !=3D FLOW_DIS_IPV4_VIHL_NOOPT)) + return false; + + if (unlikely(iph->frag_off & htons(IP_MF | IP_OFFSET))) + return false; + + if (unlikely(iph->protocol !=3D IPPROTO_TCP && + iph->protocol !=3D IPPROTO_UDP)) + return false; + + thoff =3D nhoff + (int)sizeof(*iph); + + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_CONTROL)) { + key_control =3D skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_CONTROL, + target_container); + key_control->addr_type =3D FLOW_DISSECTOR_KEY_IPV4_ADDRS; + key_control->thoff =3D min_t(u16, thoff, + skb ? skb->len : hlen); + key_control->flags =3D 0; + } + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_BASIC)) { + key_basic =3D skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_BASIC, + target_container); + key_basic->n_proto =3D htons(ETH_P_IP); + key_basic->ip_proto =3D iph->protocol; + } + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_IPV4_ADDRS)) { + key_addrs =3D skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_IPV4_ADDRS, + target_container); + memcpy(&key_addrs->v4addrs.src, &iph->saddr, + sizeof(key_addrs->v4addrs.src)); + memcpy(&key_addrs->v4addrs.dst, &iph->daddr, + sizeof(key_addrs->v4addrs.dst)); + } + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_PORTS)) { + const __be32 *ports =3D (const __be32 *) + ((const u8 *)data + thoff); + + key_ports =3D skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_PORTS, + target_container); + key_ports->ports =3D *ports; + } + + return true; +} + +/* Same as flow_dissect_fast_ipv4 but for a fixed 40-byte IPv6 header + * (no extension headers). The nexthdr check below holds the run-time + * invariant; the static_assert holds the compile-time one. + */ +static_assert(sizeof(struct ipv6hdr) =3D=3D 40); + +static bool flow_dissect_fast_ipv6(const struct sk_buff *skb, + struct flow_dissector *flow_dissector, + void *target_container, + const void *data, + int nhoff, int hlen) +{ + struct flow_dissector_key_control *key_control; + struct flow_dissector_key_addrs *key_addrs; + struct flow_dissector_key_basic *key_basic; + struct flow_dissector_key_ports *key_ports; + const struct ipv6hdr *iph; + int thoff; + + if (unlikely(hlen - nhoff < (int)sizeof(*iph) + 4)) + return false; + + iph =3D (const struct ipv6hdr *)((const u8 *)data + nhoff); + + if (unlikely((*(const u8 *)iph >> 4) !=3D 6)) + return false; + + /* Any non-zero flow label defers, checked before everything else + * (including the tunnel descents below). Callers passing + * FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL (skb_get_hash() -- so + * RPS/RFS, fq, cake) make the slow path stop at a non-zero label + * even on a tunnel outer, and dissectors requesting the label key + * write it; both diverge from the plain parse. Deferring on the + * label itself covers every combination without threading flags + * into the helpers, and zero-label IPv6 (the overwhelmingly + * common case) stays on the fast path. + */ + if (unlikely((iph->flow_lbl[0] & 0x0f) | + iph->flow_lbl[1] | iph->flow_lbl[2])) + return false; + + if (unlikely(iph->nexthdr !=3D IPPROTO_TCP && + iph->nexthdr !=3D IPPROTO_UDP)) + return false; + + thoff =3D nhoff + (int)sizeof(*iph); + + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_CONTROL)) { + key_control =3D skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_CONTROL, + target_container); + key_control->addr_type =3D FLOW_DISSECTOR_KEY_IPV6_ADDRS; + key_control->thoff =3D min_t(u16, thoff, + skb ? skb->len : hlen); + key_control->flags =3D 0; + } + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_BASIC)) { + key_basic =3D skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_BASIC, + target_container); + key_basic->n_proto =3D htons(ETH_P_IPV6); + key_basic->ip_proto =3D iph->nexthdr; + } + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_IPV6_ADDRS)) { + key_addrs =3D skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_IPV6_ADDRS, + target_container); + memcpy(&key_addrs->v6addrs.src, &iph->saddr, + sizeof(key_addrs->v6addrs.src)); + memcpy(&key_addrs->v6addrs.dst, &iph->daddr, + sizeof(key_addrs->v6addrs.dst)); + } + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_PORTS)) { + const __be32 *ports =3D (const __be32 *) + ((const u8 *)data + thoff); + + key_ports =3D skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_PORTS, + target_container); + key_ports->ports =3D *ports; + } + + return true; +} + +/* Top-level dispatcher: eligibility check (only the two standard + * dissectors and flag subset) + per-proto switch with per-shape + * static_branch gating. Each case's branch is a forward not-taken JMP + * when its sysctl is 0 =E2=80=94 matches the slow-path cost. + */ +static bool flow_dissect_fast(const struct sk_buff *skb, + struct flow_dissector *flow_dissector, + void *target_container, + const void *data, + __be16 proto, int nhoff, int hlen, + unsigned int flags) +{ + if (flow_dissector !=3D &flow_keys_dissector && + flow_dissector !=3D &flow_keys_dissector_symmetric) + return false; + + /* skb_get_hash() -- behind RPS/RFS, fq, fq_codel, cake -- passes + * FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL on every dissect, so the + * flag must be admitted or the fast path never runs for the + * kernel's main consumers. Its only semantic (stop early on a + * non-zero IPv6 flow label) is subsumed by the IPv6 fast path + * deferring on any non-zero label. Any other flag defers. + */ + if (flags & ~(unsigned int)(FLOW_DISSECTOR_F_PARSE_1ST_FRAG | + FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL)) + return false; + + switch (proto) { + case htons(ETH_P_IP): + if (!static_branch_unlikely(&flow_dissector_eth_ip_key)) + return false; + return flow_dissect_fast_ipv4(skb, flow_dissector, + target_container, data, + nhoff, hlen); + case htons(ETH_P_IPV6): + if (!static_branch_unlikely(&flow_dissector_eth_ip_key)) + return false; + return flow_dissect_fast_ipv6(skb, flow_dissector, + target_container, data, + nhoff, hlen); + default: + return false; + } +} + /** * __skb_flow_dissect - extract the flow_keys struct and return it * @net: associated network namespace, derived from @skb if NULL @@ -1183,6 +1427,13 @@ bool __skb_flow_dissect(const struct net *net, rcu_read_unlock(); } =20 + /* Opt-in fast-path; flow_dissect_fast() does the eligibility check + * and per-shape gating. + */ + if (flow_dissect_fast(skb, flow_dissector, target_container, + data, proto, nhoff, hlen, flags)) + return true; + if (dissector_uses_key(flow_dissector, FLOW_DISSECTOR_KEY_ETH_ADDRS)) { struct flow_dissector_key_eth_addrs *key_eth_addrs; @@ -1879,8 +2130,6 @@ void make_flow_keys_digest(struct flow_keys_digest *d= igest, } EXPORT_SYMBOL(make_flow_keys_digest); =20 -static struct flow_dissector flow_keys_dissector_symmetric __read_mostly; - u32 __skb_get_hash_symmetric_net(const struct net *net, const struct sk_bu= ff *skb) { struct flow_keys keys; @@ -2116,3 +2365,23 @@ static int __init init_default_flow_dissectors(void) return 0; } core_initcall(init_default_flow_dissectors); + +/* Per-shape fast-path sysctls under /proc/sys/net/flow_dissector/. */ +static struct ctl_table flow_dissector_sysctl_table[] =3D { + { + .procname =3D "eth_ip", + .data =3D &flow_dissector_eth_ip_key.key, + .maxlen =3D sizeof(flow_dissector_eth_ip_key), + .mode =3D 0644, + .proc_handler =3D proc_do_static_key, + }, +}; + +static int __init flow_dissector_sysctl_init(void) +{ + if (!register_net_sysctl(&init_net, "net/flow_dissector", + flow_dissector_sysctl_table)) + return -ENOMEM; + return 0; +} +late_initcall(flow_dissector_sysctl_init); --=20 2.54.0 From nobody Sat Jul 25 16:44:51 2026 Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D408D30677D for ; Thu, 16 Jul 2026 00:44:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162649; cv=none; b=FrGrNapKi/NrYHqeB+hn8RbdyqF2ds0/dBIOYQmEf3XBfHc5TOr8GrQr3iHkxH3tcdN+pH8m6+qUYAou6bFc0bvXrmWCtwy5o9cGlohPC1L8tWBfkb/ZcaJH01TRuSj6VMaVfYDgNZBEYBX/Hon4DePWYP+iQyDpDshG96GhtWY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162649; c=relaxed/simple; bh=LkzKdUO+JmG0HF2usl69sh+fsYe38FuCXJBrcY3/aIM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=LQFeip7rq+SBtCugwFA0eZ9Oq/Sr2Z0uqZBrrkxh25fqoyoG1AoQQgnp++ZsLQYEMcu21zjeyBZ3v9Y+TF/uLyPEAZ1EYXJF7bYXsaIMBT6ttD8qPZDkiRQpJoxol+6e3Oo13ENSQO4U/pPWigRJ8j8ijYmLg+yI1Rx3egPISlo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=N/gteU5L; arc=none smtp.client-ip=209.85.215.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="N/gteU5L" Received: by mail-pg1-f170.google.com with SMTP id 41be03b00d2f7-ca88130e09aso4019246a12.3 for ; Wed, 15 Jul 2026 17:44:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784162644; x=1784767444; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=oURpgbk9qcfxkTyiXEXzSzC9IFYy3cXT62egJ45Q2qY=; b=N/gteU5Lrf3Krb9xet+Gq4QYxFc2PYezqcnSa4OPDHBB0hLA751f+ajI0TtytaJNVz Iim62SNDb/f79+kfGOxAfpy7Zle+NcI9IHk0Q1dTmkvwcWVwnJBkdn9xf8x6ph225LRL enu+1kXarLHkjlCX+sgB5PF7m5pqDbX+XclG3QosD8LMJboF3KgADSpZ9kxI9pp1z3Al 8vnUbHdP0XwKr/TaSooO+3vhNH/FvS9v8I3oid7LhDQvN/Zs98EAyVAxxgdbJtSbKwaF kvDNPCpvAO8o5W1oOXQ3w+GV/pqN0YVbYZlp9Tiigxr3kWKWOSRglpX8/GK3Z/mJAyPv RH9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784162644; x=1784767444; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=oURpgbk9qcfxkTyiXEXzSzC9IFYy3cXT62egJ45Q2qY=; b=SDnyV5bArLWcR6sxqJUYNGufdkkZewWWnbDv1795cpwBcMaR46GzCTUsOW/9TvzqXr 44BZtjphKERuXhl/R3JV7fj8YCORQYm41jbX0MX365PVkgMKewYI595ksyi1kxtiNBec nzVhUxQniD38R9XybqcmrQicQddlFMfyQ2kt7PxtVgc3NgLQ6hZerqvkEOorP/bFsDIH tMhL6zAEmg+w2WlD1XdUf/dUmiXDa1cNHHPPSNHUrKZ4Tt4D4F2ucGsMPvC+DWXbcQl0 4FaSCxwcoLLUBRQbi+Emr6bCujsEoxSz+0Nk4RDS+XmDaCHICLuvQKjY+k7+JI3pgwOs sbIw== X-Forwarded-Encrypted: i=1; AHgh+RonTRWRL2El89cMiHeITDUExNH8z7JSjQz80MLtzkEN/XhPop1O82T3R4Pu+yR4bO1ukdEeJhVUG4Kc2c8=@vger.kernel.org X-Gm-Message-State: AOJu0YwjLlL6mpb9x9DEs2gTFKMlUr55BFJAyzrCY1aChywT3lIi6Kgf /fOkz5xtczpTnIl2gxm9d+X1ppzMMHZ+8m+mDd5URj1BJll1wN6gNxA8 X-Gm-Gg: AfdE7cmtuMogQTZzv9mpod0YA6HjDB38m7Gr2bKWG5Su9k8gHMuSI8LrGEaABGcH2KK LxF5jALYdiFHgaiV0dTDEBnb5QwITJpfyGOcefMWxK2CfqYnDASMW7OV0oYXMc4AaBLVNDf788I A+aBBwegBKrjPggN75oWMuQ9ue50Ls0SSHtn/t8yt+89jZl4FFG3NAqM9CnA+Em/WpwcJ2Olao+ 3UydK9X8dIH2BiGe6HY1L0LtfyUJsnVjOZHbzT7M4Ax8eCHNkYN+ufIDYrRe8KmK4fPAB3veOa1 WCSKW1Y+2RW5frAovczCwJsQjwGJ54CgwiLH62B/h9XPphbSgRAzCyPxMnSVXhIQ1JzoKpuh1Bg waEQw2F68pWzZnu2sCAEkVWs9vBsp6ug4jB1DsCImx8FbRLA35uIYq0p65ivOUNz5Yf0zdD3VdM pU+60j+W0NravFRq5FqRsRkUr05vIteF1SNr9l4/TsTULs7dUuFA/4EdDJohQwHGNT01cCT95+T kX4jIklDdJS2NwQOM09 X-Received: by 2002:a05:6a20:6a22:b0:3c3:8041:89e3 with SMTP id adf61e73a8af0-3c380418eedmr2728200637.63.1784162644049; Wed, 15 Jul 2026 17:44:04 -0700 (PDT) Received: from l.localdomain ([76.166.220.66]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-13cd4284a92sm4107620c88.4.2026.07.15.17.44.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 17:44:03 -0700 (PDT) From: Dave Seddon To: netdev@vger.kernel.org Cc: "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Stanislav Fomichev , Tom Herbert , Willem de Bruijn , linux-kernel@vger.kernel.org, Dave Seddon Subject: [PATCH net-next v1 03/11] net: flow_dissector: add fast-path for VLAN and QinQ + IP + TCP/UDP Date: Wed, 15 Jul 2026 17:43:49 -0700 Message-ID: <20260716004357.3652679-4-dave.seddon.ca@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> References: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Extension of the parent series covering the VLAN-tagged shapes: a single ETH_P_8021Q or ETH_P_8021AD tag, or two stacked tags (the canonical QinQ S-tag + C-tag), in front of IPv4/IPv6 + TCP/UDP. Adds flow_dissect_fast_vlan() that consumes the tag(s) and dispatches to the existing flow_dissect_fast_ipv4 / _ipv6 helpers on the inner ethertype. - What this covers. VLAN-tagged Eth (8021Q or 8021AD, depth 1 or 2) + IPv4 or IPv6 (per parent's shape constraints) + TCP/UDP. Both hardware-stripped (TCI in skb metadata, inner ethertype in skb->protocol; only ever the outermost tag) and in-band (4-byte vlan_hdr at data + nhoff) tag forms. - Keys written. Outer tag: FLOW_DISSECTOR_KEY_VLAN (vlan_id / vlan_priority / vlan_tpid / vlan_eth_type); inner tag (depth 1): FLOW_DISSECTOR_KEY_CVLAN, mirroring the slow-path's MAX -> VLAN -> CVLAN state machine. Bumps FLOW_DISSECTOR_KEY_NUM_OF_VLANS when requested by the dissector. Byte-identical with the slow path. Past depth 2 the helper returns false (defers to the slow path). - Gates. Two static keys, each with its own sysctl: DEFINE_STATIC_KEY_FALSE(flow_dissector_vlan_key) (/proc/sys/net/flow_dissector/vlan) gates entry at depth 0, and DEFINE_STATIC_KEY_FALSE(flow_dissector_qinq_key) (/proc/sys/net/flow_dissector/qinq) gates the depth-1 tag. Defaults 0; off has no behaviour change beyond one extra not-taken JMP in the dispatcher switch. - Sibling-key handling. The two gates are coupled in their proc handlers: writing qinq=3D1 auto-enables the vlan key (QinQ extends VLAN; depth-0 entry is gated by vlan), and turning vlan off clears the qinq key (depth-2 cannot fire when the outer entry is off). Reads always reflect the underlying key state. - Cost. On a miss, one ethertype switch + the static_branch test. On a hit, ~3-5 ns added to the IPv4/IPv6 fast-path body per tag consumed, while the slow-path's per-tag handling (state-machine transition + proto_again rewind) is skipped -- net saving ~30 ns/skb on the single-VLAN shape and ~50 ns/skb on QinQ. Verified byte-identical by the KUnit equivalence suite. Isolated dissector cost on the allshapes microbench (all fast paths compiled in, range across 7 measured microarchitectures): vlan -18.1%..-38.5%, qinq -16.1%..-39.8% -- see the cover letter's performance summary. Assisted-by: Claude:claude-fable-5 sparse smatch Signed-off-by: Dave Seddon --- Documentation/admin-guide/sysctl/net.rst | 31 +++++ include/net/flow_dissector.h | 2 + net/core/flow_dissector.c | 147 +++++++++++++++++++++++ 3 files changed, 180 insertions(+) diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin= -guide/sysctl/net.rst index aac82169e019..824f4b735caa 100644 --- a/Documentation/admin-guide/sysctl/net.rst +++ b/Documentation/admin-guide/sysctl/net.rst @@ -485,6 +485,37 @@ when this is 0. =20 Default: 0 =20 +vlan +~~~~ + +Single 802.1Q or 802.1AD tagged frames over Eth + IPv4/IPv6 + +TCP/UDP. Handles both the hardware-stripped form (tag in skb +metadata, inner ethertype in skb->protocol) and the in-band form +(4-byte vlan_hdr at data + nhoff). Writes FLOW_DISSECTOR_KEY_VLAN +and bumps FLOW_DISSECTOR_KEY_NUM_OF_VLANS when those keys are +requested by the dissector. Byte-identical with the slow path for +the eligible shape; defers on any miss (QinQ defers; see the qinq +sysctl). + +Default: 0 + +qinq +~~~~ + +Lifts the vlan fast-path's depth limit from 1 to 2 so two stacked +802.1AD / 802.1Q tags (canonical QinQ S-tag + C-tag, or two stacked +802.1Q tags) are also fast-pathed. Outer tag is written to +FLOW_DISSECTOR_KEY_VLAN, inner to FLOW_DISSECTOR_KEY_CVLAN, mirroring +the slow-path's MAX =E2=86=92 VLAN =E2=86=92 CVLAN state machine. Byte-ide= ntical +with the slow path for the eligible shape. + +Auto-toggle: writing qinq=3D1 also enables vlan (QinQ extends VLAN; +the depth-0 entry of the helper is gated by the vlan key). Turning +vlan off (writing 0) also clears qinq (depth-2 cannot fire when +the outer entry is off). + +Default: 0 + 3. /proc/sys/net/unix - Parameters for Unix domain sockets ---------------------------------------------------------- =20 diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h index 7d513a6cbb55..4f36f3889a28 100644 --- a/include/net/flow_dissector.h +++ b/include/net/flow_dissector.h @@ -429,6 +429,8 @@ extern struct flow_dissector flow_keys_basic_dissector; * /proc/sys/net/flow_dissector/. All default off. */ extern struct static_key_false flow_dissector_eth_ip_key; +extern struct static_key_false flow_dissector_vlan_key; +extern struct static_key_false flow_dissector_qinq_key; =20 /* struct flow_keys_digest: * diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c index 88b1bd1cc4ca..826314e28db0 100644 --- a/net/core/flow_dissector.c +++ b/net/core/flow_dissector.c @@ -42,6 +42,10 @@ */ DEFINE_STATIC_KEY_FALSE(flow_dissector_eth_ip_key); EXPORT_SYMBOL(flow_dissector_eth_ip_key); +DEFINE_STATIC_KEY_FALSE(flow_dissector_vlan_key); +EXPORT_SYMBOL(flow_dissector_vlan_key); +DEFINE_STATIC_KEY_FALSE(flow_dissector_qinq_key); +EXPORT_SYMBOL(flow_dissector_qinq_key); =20 /* IPv4 version/IHL byte of an option-less header: version 4, IHL 5. */ #define FLOW_DIS_IPV4_VIHL_NOOPT 0x45 @@ -1242,6 +1246,93 @@ static bool flow_dissect_fast_ipv6(const struct sk_b= uff *skb, return true; } =20 +/* Strip up to two 802.1Q/802.1AD tags: outer tag -> + * FLOW_DISSECTOR_KEY_VLAN, inner -> _CVLAN, mirroring the slow path's + * MAX -> VLAN -> CVLAN state machine; more than two tags defer. + * Depth 0 is gated by the vlan key, depth >=3D 1 by the qinq key (the + * qinq sysctl also flips vlan on -- see the proc_handlers). + */ +static bool flow_dissect_fast_vlan(const struct sk_buff *skb, + struct flow_dissector *flow_dissector, + void *target_container, + const void *data, + __be16 proto, int nhoff, int hlen, + int vlan_depth) +{ + struct flow_dissector_key_num_of_vlans *key_nvs; + struct flow_dissector_key_vlan *key_vlan; + enum flow_dissector_key_id vlan_key; + __be16 inner_proto; + __be16 saved_tpid; + u16 tci_prio; + u16 tci_id; + + if (vlan_depth >=3D 1 && + !static_branch_unlikely(&flow_dissector_qinq_key)) + return false; + if (vlan_depth >=3D 2) + return false; + + vlan_key =3D vlan_depth =3D=3D 0 ? + FLOW_DISSECTOR_KEY_VLAN : + FLOW_DISSECTOR_KEY_CVLAN; + saved_tpid =3D proto; + + /* HW-stripped tag is only ever the outermost. */ + if (vlan_depth =3D=3D 0 && skb && skb_vlan_tag_present(skb)) { + tci_id =3D skb_vlan_tag_get_id(skb); + tci_prio =3D skb_vlan_tag_get_prio(skb); + inner_proto =3D skb->protocol; + } else { + const struct vlan_hdr *vlan; + + if (unlikely(hlen - nhoff < (int)sizeof(*vlan))) + return false; + vlan =3D (const struct vlan_hdr *)((const u8 *)data + nhoff); + tci_id =3D ntohs(vlan->h_vlan_TCI) & VLAN_VID_MASK; + tci_prio =3D (ntohs(vlan->h_vlan_TCI) & + VLAN_PRIO_MASK) >> VLAN_PRIO_SHIFT; + inner_proto =3D vlan->h_vlan_encapsulated_proto; + nhoff +=3D (int)sizeof(*vlan); + } + + if (dissector_uses_key(flow_dissector, vlan_key)) { + key_vlan =3D skb_flow_dissector_target(flow_dissector, + vlan_key, + target_container); + key_vlan->vlan_id =3D tci_id; + key_vlan->vlan_priority =3D tci_prio; + key_vlan->vlan_tpid =3D saved_tpid; + key_vlan->vlan_eth_type =3D inner_proto; + } + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_NUM_OF_VLANS)) { + key_nvs =3D skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_NUM_OF_VLANS, + target_container); + key_nvs->num_of_vlans++; + } + + switch (inner_proto) { + case htons(ETH_P_IP): + return flow_dissect_fast_ipv4(skb, flow_dissector, + target_container, data, + nhoff, hlen); + case htons(ETH_P_IPV6): + return flow_dissect_fast_ipv6(skb, flow_dissector, + target_container, data, + nhoff, hlen); + case htons(ETH_P_8021Q): + case htons(ETH_P_8021AD): + return flow_dissect_fast_vlan(skb, flow_dissector, + target_container, data, + inner_proto, nhoff, hlen, + vlan_depth + 1); + default: + return false; + } +} + /* Top-level dispatcher: eligibility check (only the two standard * dissectors and flag subset) + per-proto switch with per-shape * static_branch gating. Each case's branch is a forward not-taken JMP @@ -1270,6 +1361,13 @@ static bool flow_dissect_fast(const struct sk_buff *= skb, return false; =20 switch (proto) { + case htons(ETH_P_8021Q): + case htons(ETH_P_8021AD): + if (!static_branch_unlikely(&flow_dissector_vlan_key)) + return false; + return flow_dissect_fast_vlan(skb, flow_dissector, + target_container, data, + proto, nhoff, hlen, 0); case htons(ETH_P_IP): if (!static_branch_unlikely(&flow_dissector_eth_ip_key)) return false; @@ -2367,6 +2465,41 @@ static int __init init_default_flow_dissectors(void) core_initcall(init_default_flow_dissectors); =20 /* Per-shape fast-path sysctls under /proc/sys/net/flow_dissector/. */ +/* proc_handler for net.flow_dissector.vlan. Writes of 0 also clear + * flow_dissector_qinq_key =E2=80=94 QinQ depth-2 is meaningless when the + * outer single-VLAN gate is off, so the slow path would have to + * handle every VLAN-tagged packet anyway. + */ +static int proc_set_vlan_key(const struct ctl_table *table, int write, + void *buffer, size_t *lenp, loff_t *ppos) +{ + int ret; + + ret =3D proc_do_static_key(table, write, buffer, lenp, ppos); + if (ret =3D=3D 0 && write && + !static_branch_unlikely(&flow_dissector_vlan_key) && + static_branch_unlikely(&flow_dissector_qinq_key)) + static_branch_disable(&flow_dissector_qinq_key); + return ret; +} + +/* proc_handler for net.flow_dissector.qinq. Writes of 1 also enable + * flow_dissector_vlan_key =E2=80=94 QinQ extends the VLAN fast-path to de= pth + * 2; it cannot fire when the depth-0 entry is gated off. + */ +static int proc_set_qinq_key(const struct ctl_table *table, int write, + void *buffer, size_t *lenp, loff_t *ppos) +{ + int ret; + + ret =3D proc_do_static_key(table, write, buffer, lenp, ppos); + if (ret =3D=3D 0 && write && + static_branch_unlikely(&flow_dissector_qinq_key) && + !static_branch_unlikely(&flow_dissector_vlan_key)) + static_branch_enable(&flow_dissector_vlan_key); + return ret; +} + static struct ctl_table flow_dissector_sysctl_table[] =3D { { .procname =3D "eth_ip", @@ -2375,6 +2508,20 @@ static struct ctl_table flow_dissector_sysctl_table[= ] =3D { .mode =3D 0644, .proc_handler =3D proc_do_static_key, }, + { + .procname =3D "vlan", + .data =3D &flow_dissector_vlan_key.key, + .maxlen =3D sizeof(flow_dissector_vlan_key), + .mode =3D 0644, + .proc_handler =3D proc_set_vlan_key, + }, + { + .procname =3D "qinq", + .data =3D &flow_dissector_qinq_key.key, + .maxlen =3D sizeof(flow_dissector_qinq_key), + .mode =3D 0644, + .proc_handler =3D proc_set_qinq_key, + }, }; =20 static int __init flow_dissector_sysctl_init(void) --=20 2.54.0 From nobody Sat Jul 25 16:44:51 2026 Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0C039405F7 for ; Thu, 16 Jul 2026 00:44:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162654; cv=none; b=CG1kGwqF34RQqpOH110Xzi/Kux1QInef7EvGLSu+OUKTeZOYofa+N3FtH0FO3g3ayqatVimj+zOdblorsUm7CL2rkGoJjdktRzXMoipgDoo0sFt2KG7YDZtCAmp8ul8VTDdRQqKVVjqg9f37LVcx7al9f05OTTk5pIadmORsu90= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162654; c=relaxed/simple; bh=/mWKmgPPP3TeTo5213fkgzBCXwFcfeZrukW48TpwrP4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=CLnCAPxg5ovahkzQe0THGxmk5wvlVT0xEWXCjf3LOH46JcIsYfz+7EUCULGOATC7OX4LiyUYwAX9E8H+CjocabeqmsslumkVoJmSbLspAVXjz9IlpWGjF23+sraK5//WcFgiL6GnNN8pJ5Ob3/L6XZCdeR3Cr7Nzm8jI4QWFYPs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=hmQ7wj3N; arc=none smtp.client-ip=209.85.210.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hmQ7wj3N" Received: by mail-pf1-f179.google.com with SMTP id d2e1a72fcca58-8486ac3f347so51363b3a.1 for ; Wed, 15 Jul 2026 17:44:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784162645; x=1784767445; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=4KYs6bF4G5Lma9vSvx/+axD0veNKsucRJ9eKrQH9KOc=; b=hmQ7wj3NWek2tmasKCWePocY+59m7HFcM0+AEmb3tY9H5yahHIR85V6GVN8PMeDXFE vdm4e/sdZLHdoKV5folV8HpLUsLe/5NEAPoVpA4BamyVXDoZ8B99DZxv/fxrmo9U8b5y 26B5x5NjO5IP3RuMeYqdZ9lB987tnlncxsVv+7btJlrOwtH1h7JGfbXqIGsXRKnccZDg RjISeEo2rEDCNlqwA0wttmPNTWtXMMydUnl1I1usYEFHCmrbpWs6DG+T/Ai2UbyPUQuu UEDIkhV4lZNZ0ExAm6CmR5zShKWUvfaA53Gio1KAL5Ynk9COZGkJzr4hX4QCqpOmMb4B 8NSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784162645; x=1784767445; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=4KYs6bF4G5Lma9vSvx/+axD0veNKsucRJ9eKrQH9KOc=; b=egy0aKbMr8HYj0BhspSMW2XOtcxLF2k3KusIviIOZhkVuvqKUF1251onBC7+aTqCeo 35ht8noYHxzIWfoh3aR9QKP28ysg79rF5X2k8JSqaFoseCtl8LpzYG9vOx92Mkh2SnLW x0g4TGHK/mvxKNBkhBUuFkdYI1i0A7Z3llDZtISLWcQj08d+9hYO1M39vAlAC6ugJSgg lShzcLdvnk5BanEPMCzIjVRu5QWrCnzmAFc5an0ieEqKzHmFEZmmY9RMk8bVdEg3f6f6 HypNgSksHm7+SbLIl4LhFDxY78fj6xhO/k/q+ViKjJNf2pLwwxXzrGNyKaKYzhV5iXCA 5Pvw== X-Forwarded-Encrypted: i=1; AHgh+RrOpHRblFC9CrP6A5n9LSiHuQKGYOEdHfG1dS4Q4bfrNjpJzMmlU2uNU6f1kqhdkvWK5SERvrg3+m19Wo4=@vger.kernel.org X-Gm-Message-State: AOJu0YxIwknA4w472gjDB/17HYpMeKPrlfrmRnv32a6ZCeeU2DIgs1E9 +dnHMlIcduncyYo7CSXTa1XmgKu5LtiQRKK+uW+5h1LE5U5bpoIxZm6x X-Gm-Gg: AfdE7cmLSIQ3ss8TlbAD8q5XUvoCxK5SJkiC/yIWCdN8qbpsaMPR8ZmT80WMOXJPIwq AYINxMHDKq8Ua5bptCBjuMLtfuqL4RuqJ7gLg+KFgQ7nxT5ZvD8pdfJjVfmqwtktBIRE+hmTbav d/y2tJj44yzW7jMXQQs3YvOYwb8dyRzU/9lwlKNJ+tvnTR9CJt7y6/YjwGzbpuaD/GsmCPicKV4 96hmSmCs8tUlDnlwOtX2LPXMnhoWEIBPDoA33bqGP6Quw1wNU4Tkob8UP3iM2RzXdQw83CjxeT0 abYF/Z+i6h6KfqXs3Hc9Ns+ODECI131vuMPYVGnx9moL3VG7yAXtJ38zfnVSvJvnpuPnRkIEEro yFNkwTBF0W6JEf0rr+bF+nrnfJno0LCZ/gDpamMgtBrO/uR7Nq7qN75FpnOmKWZY1uIu+f0PnmD 5igQf4BSqeErGkjz+YwPeMxjCGevr8BeOI4t6uPtIyugyebv8OVlPdv8UqISNvGeM52e4orNm+1 NE+ltULaA== X-Received: by 2002:a05:6a20:244c:b0:3bf:a8fa:a7b1 with SMTP id adf61e73a8af0-3c38dc1bbd0mr378449637.37.1784162645332; Wed, 15 Jul 2026 17:44:05 -0700 (PDT) Received: from l.localdomain ([76.166.220.66]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-13cd4284a92sm4107620c88.4.2026.07.15.17.44.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 17:44:04 -0700 (PDT) From: Dave Seddon To: netdev@vger.kernel.org Cc: "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Stanislav Fomichev , Tom Herbert , Willem de Bruijn , linux-kernel@vger.kernel.org, Dave Seddon Subject: [PATCH net-next v1 04/11] net: flow_dissector: add fast-path for PPPoE session + IPv{4,6} + TCP/UDP Date: Wed, 15 Jul 2026 17:43:50 -0700 Message-ID: <20260716004357.3652679-5-dave.seddon.ca@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> References: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Adds flow_dissect_fast_pppoe() =E2=80=94 a straight-line extractor for PPPoE session frames (ETH_P_PPP_SES, EtherType 0x8864 per RFC 2516) carrying IPv4 or IPv6 inside PPP. Strips the 6-byte pppoe_hdr + 2-byte PPP protocol field, writes FLOW_DISSECTOR_KEY_PPPOE (session_id, ppp_proto, type) when requested, and tail-calls into the eth_ip fast-path for the inner IPv4 or IPv6 + TCP/UDP. Byte-identical with the slow path's `case htons(ETH_P_PPP_SES)` block for the eligible shape (RFC-2516-valid header + PPP_IP / PPP_IPV6 + straight-line IP). On any miss =E2=80=94 invalid header, PFC-compressed PPP proto, PPP_MPLS_UC / PPP_MPLS_MC (defer until MPLS fast-path lands), LCP / IPCP / other control frames =E2=80=94 returns false so the dispatcher falls through to the slow path. Output is exactly what the slow path would produce. Why this matters: PPPoE is the dominant L2 over millions of ISP last- mile DSL/FTTH links, and the existing slow-path PPPoE block does a __skb_header_pointer + per-field switch on every packet at the graph-walk entry. The fast-path saves the walk overhead for the common IPv4 / IPv6 inner case while keeping the slow path as the escape valve for control / less-common inner protos. Gated by: - new DEFINE_STATIC_KEY_FALSE(flow_dissector_pppoe_key) - new sysctl /proc/sys/net/flow_dissector/pppoe (default 0) - dispatcher case `htons(ETH_P_PPP_SES)` with `static_branch_unlikely(&flow_dissector_pppoe_key)` guard When the sysctl is 0, the dispatcher hits a forward not-taken JMP =E2=80=94 same per-call cost as the vlan / qinq cases. The new dispatcher case also doesn't widen the switch table (it adds one case to a switch the compiler can already lower to a jump table on the ETH_P_* constants). Documentation/admin-guide/sysctl/net.rst grows a `pppoe` subsection under the existing `/proc/sys/net/flow_dissector/` chapter, between qinq and the next chapter, matching the existing per-shape layout. Assisted-by: Claude:claude-fable-5 sparse smatch Signed-off-by: Dave Seddon --- Documentation/admin-guide/sysctl/net.rst | 18 ++++++ include/net/flow_dissector.h | 1 + net/core/flow_dissector.c | 76 ++++++++++++++++++++++++ 3 files changed, 95 insertions(+) diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin= -guide/sysctl/net.rst index 824f4b735caa..1096e6ddca39 100644 --- a/Documentation/admin-guide/sysctl/net.rst +++ b/Documentation/admin-guide/sysctl/net.rst @@ -516,6 +516,24 @@ the outer entry is off). =20 Default: 0 =20 +pppoe +~~~~~ + +PPPoE session frames (RFC 2516, EtherType ``ETH_P_PPP_SES`` 0x8864) +carrying IPv4 or IPv6 inside PPP. The fast-path strips the 6-byte +PPPoE session header + 2-byte PPP protocol field, writes +FLOW_DISSECTOR_KEY_PPPOE (session_id, ppp_proto, type) when the +dissector requests it, and tail-calls into the eth_ip fast-path for +the inner IPv4 / IPv6 + TCP/UDP. Byte-identical with the slow path +for the eligible shape. + +Defers to slow path for: PPP_MPLS_UC / PPP_MPLS_MC (until an MPLS +fast-path lands), PFC-compressed PPP protocol fields, LCP/IPCP/etc. +control frames, and any malformed PPPoE header (ver !=3D 1, type !=3D 1, +code !=3D 0). + +Default: 0 + 3. /proc/sys/net/unix - Parameters for Unix domain sockets ---------------------------------------------------------- =20 diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h index 4f36f3889a28..df85162959cc 100644 --- a/include/net/flow_dissector.h +++ b/include/net/flow_dissector.h @@ -431,6 +431,7 @@ extern struct flow_dissector flow_keys_basic_dissector; extern struct static_key_false flow_dissector_eth_ip_key; extern struct static_key_false flow_dissector_vlan_key; extern struct static_key_false flow_dissector_qinq_key; +extern struct static_key_false flow_dissector_pppoe_key; =20 /* struct flow_keys_digest: * diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c index 826314e28db0..21a3f5e0ea05 100644 --- a/net/core/flow_dissector.c +++ b/net/core/flow_dissector.c @@ -46,6 +46,8 @@ DEFINE_STATIC_KEY_FALSE(flow_dissector_vlan_key); EXPORT_SYMBOL(flow_dissector_vlan_key); DEFINE_STATIC_KEY_FALSE(flow_dissector_qinq_key); EXPORT_SYMBOL(flow_dissector_qinq_key); +DEFINE_STATIC_KEY_FALSE(flow_dissector_pppoe_key); +EXPORT_SYMBOL(flow_dissector_pppoe_key); =20 /* IPv4 version/IHL byte of an option-less header: version 4, IHL 5. */ #define FLOW_DIS_IPV4_VIHL_NOOPT 0x45 @@ -1333,6 +1335,67 @@ static bool flow_dissect_fast_vlan(const struct sk_b= uff *skb, } } =20 +/* PPPoE session (RFC 2516) + 2-byte PPP protocol: write + * FLOW_DISSECTOR_KEY_PPPOE if requested, then tail-call the eth_ip + * fast-path for PPP_IP / PPP_IPV6. Anything else (PFC-compressed + * protocol, LCP, MPLS) defers. + */ +static bool flow_dissect_fast_pppoe(const struct sk_buff *skb, + struct flow_dissector *flow_dissector, + void *target_container, + const void *data, + int nhoff, int hlen) +{ + struct flow_dissector_key_pppoe *key_pppoe; + const struct { + struct pppoe_hdr hdr; + __be16 proto; + } *hdr; + __be16 inner_eth_proto; + u16 ppp_proto; + + if (unlikely(hlen - nhoff < (int)sizeof(*hdr))) + return false; + hdr =3D (const void *)((const u8 *)data + nhoff); + if (!is_pppoe_ses_hdr_valid(&hdr->hdr)) + return false; + + ppp_proto =3D ntohs(hdr->proto); + switch (ppp_proto) { + case PPP_IP: + inner_eth_proto =3D htons(ETH_P_IP); + break; + case PPP_IPV6: + inner_eth_proto =3D htons(ETH_P_IPV6); + break; + default: + /* MPLS unicast/multicast + everything else (LCP, IPCP, + * compressed PFC, ...) defers to the slow path. + */ + return false; + } + + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_PPPOE)) { + key_pppoe =3D skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_PPPOE, + target_container); + key_pppoe->session_id =3D hdr->hdr.sid; + key_pppoe->ppp_proto =3D htons(ppp_proto); + key_pppoe->type =3D htons(ETH_P_PPP_SES); + } + + nhoff +=3D PPPOE_SES_HLEN; + + if (inner_eth_proto =3D=3D htons(ETH_P_IP)) + return flow_dissect_fast_ipv4(skb, flow_dissector, + target_container, data, + nhoff, hlen); + return flow_dissect_fast_ipv6(skb, flow_dissector, + target_container, data, + nhoff, hlen); +} + /* Top-level dispatcher: eligibility check (only the two standard * dissectors and flag subset) + per-proto switch with per-shape * static_branch gating. Each case's branch is a forward not-taken JMP @@ -1380,6 +1443,12 @@ static bool flow_dissect_fast(const struct sk_buff *= skb, return flow_dissect_fast_ipv6(skb, flow_dissector, target_container, data, nhoff, hlen); + case htons(ETH_P_PPP_SES): + if (!static_branch_unlikely(&flow_dissector_pppoe_key)) + return false; + return flow_dissect_fast_pppoe(skb, flow_dissector, + target_container, data, + nhoff, hlen); default: return false; } @@ -2522,6 +2591,13 @@ static struct ctl_table flow_dissector_sysctl_table[= ] =3D { .mode =3D 0644, .proc_handler =3D proc_set_qinq_key, }, + { + .procname =3D "pppoe", + .data =3D &flow_dissector_pppoe_key.key, + .maxlen =3D sizeof(flow_dissector_pppoe_key), + .mode =3D 0644, + .proc_handler =3D proc_do_static_key, + }, }; =20 static int __init flow_dissector_sysctl_init(void) --=20 2.54.0 From nobody Sat Jul 25 16:44:51 2026 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B2C2C30B51E for ; Thu, 16 Jul 2026 00:44:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162652; cv=none; b=Cuwh69F3czNpwGeshYSnXxTyfcXALpTfqODKtacE3s3ASffPBOn/0nh2n5TEyKwTgPhLLAOy39mJKgRayKLFaj6YX03eQq7+SZJ2OxSv4ZD/5PIL883Qp+mT97vmCjYePEilk3DjnD8QD1NQo1KUyBGTR7kpXjz3La4kZJWQq3E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162652; c=relaxed/simple; bh=WhaPoRb0QmwTgOHptlhNgtNX0s0Rws9PcuRWlOBdLN8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=S/DgEohSz/XbZxnjBwPA1f2z0xlxO1P4QVagBYTZMMYexZ86TQqmK/wXgmic+5m5IVQ448hLKZ5q77q5vbeqV3fVAaMmvX2lApZjfXiSNECvu0ZK/tieSvmWRS7C+zDLvXBfh9AD9h+aY39cGVT3o6XrckQlHykq+Z4Z7q5Gj8Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=J3Xbxs28; arc=none smtp.client-ip=209.85.214.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="J3Xbxs28" Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-2caf228a910so46666005ad.2 for ; Wed, 15 Jul 2026 17:44:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784162647; x=1784767447; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=Wj3qOVD1qVOw8XUvfGDy0ElOIdjPdoQAPsV1zxudQMA=; b=J3Xbxs28jmmW4cEXtT6nmgy8HKTR0w677P7JwwvAmcm4YHf/R6EFVF1lTWeV448h9O OJvCudHaPnxB/QgYlmqEeWv+o75iYJGqA6gqSTWP9vqCPrSp2pgTQpJVU01Wv/iIPiOg RuqZmbo2VG0xMyUi4CJMGvYwUQJ9MI0J5YQLk5vHQvFzXuEraoLgm0MVX3E3KMea1dHu 4tS8pV5gTVySN7kAmW4Nx/rA6h8um+8g57JrqFXiJcHcUO/6RY3RB5XxV4DLGDUdhQIP ytLWpqG9p+TYO2YW7h0esU03RiWkJjnk+hAq4hugi6m1QCjlmMWO0nB3wML+6htX+JGi 2k+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784162647; x=1784767447; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=Wj3qOVD1qVOw8XUvfGDy0ElOIdjPdoQAPsV1zxudQMA=; b=Tfysja79ycyLvzVDLoNq3mELc6Zzwxe21IQXe7V6DFMQXr7WXXUWaoG52auM7FClv/ KpbRachvmtNKhDc1KQtpYRbFB3HhkbWP/4UqGbA1c/WpUimjkKCkwYaTAtEFvfAGIz6y V8f76/SYAAk1T0WbPO6G7qoFhzwoZBFxg1GSV0MYBw0WkM4weyuVWsGiNEPv0gSZVtrW HCfX7rJYbhcFtLVLEVAEQ2EAKg5zAy2bUAVfkWvHMTXtw3PK14RXzWpUs65ISRKz2b5d FdVFSphiCTfIAMauLsOosc4BtqLm/YxHA46UgSkMvzXoNB9KFm7KT/13oBWEvhrKod2o t/RQ== X-Forwarded-Encrypted: i=1; AHgh+Rpz+Id38xEZZactpswQLZAuHrj+kxdtFmhkIl4Jq+0Lu3wFEu0Y3HeLmSIadw6D0ZIBcOVemJm4Nz9opUY=@vger.kernel.org X-Gm-Message-State: AOJu0YzHViYtwM12UejNhkhg0MwbOYsV8exd9nz5jP4w2aqyVlPUhHHj tLFFGU6IwSUnpWBez/66bqnNdfpFT2BW0GiWirALPkS4+/+ytfK4CeH8 X-Gm-Gg: AfdE7cnJgjHgUshq4TZHbqk7YTtQ/mMJHmhALE1/w68puq4paERSJfyA2xYc9L4tyOl qhK+0BPzqUlQmw/85IlUbFxZU7jFXPcLeoeIVaWmNRjTV+eBAw6ONqjwi5F7YqsATPDtukG6qiL 4VNJJJhtYxrFXyToU6J/bliVevAKmt6POaqqmJBB6DEZECv7MOyzmjfOSJ4kH7pVXmJ6YernCxE IVMIxFmIm+uPlLKjn34BC9LQyteVj0cD/xCbwsiQSU9Ue6OKhBZJCHZQBHurgK5OixLG2VHfLN4 A1im9JWc5LkV009bzReanOYKHhgGoWdWNSLNMSSRusNuwbREqr9Yqattamx3MzPo1bCLk4YaOjv 5WEPmlDHZPfa8skcYARz1oEQtvc+V+Tqf3YJxgGGI2WhQifYr7EVie2DiXaxmI8AgtXT0Ms1Oew Le+iDNFaM6cB0UbiIJhJ8qX4bod53rxABmsbrU2tW1ytLkqOG4neV3HBqnXx+QAmJ6fQu/VP8fl spd1VroMw== X-Received: by 2002:a05:6a21:2291:b0:3bf:b40b:def2 with SMTP id adf61e73a8af0-3c36c0f6cf0mr5511456637.9.1784162646934; Wed, 15 Jul 2026 17:44:06 -0700 (PDT) Received: from l.localdomain ([76.166.220.66]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-13cd4284a92sm4107620c88.4.2026.07.15.17.44.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 17:44:06 -0700 (PDT) From: Dave Seddon To: netdev@vger.kernel.org Cc: "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Stanislav Fomichev , Tom Herbert , Willem de Bruijn , linux-kernel@vger.kernel.org, Dave Seddon Subject: [PATCH net-next v1 05/11] net: flow_dissector: add fast-path for single MPLS label + IP Date: Wed, 15 Jul 2026 17:43:51 -0700 Message-ID: <20260716004357.3652679-6-dave.seddon.ca@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> References: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Adds flow_dissect_fast_mpls() =E2=80=94 straight-line read of one MPLS label entry (ETH_P_MPLS_UC 0x8847 / ETH_P_MPLS_MC 0x8848), extracting label / TC / TTL / BoS and writing to FLOW_DISSECTOR_KEY_MPLS lse[0] when the dissector requests it. Byte-identical with the slow path's __skb_flow_dissect_mpls() called with lse_index=3D0, bos=3D1: the slow path returns FLOW_DISSECT_RET_OUT_GOOD when BoS is set without descending into the inner packet (MPLS flow hashing is on the label stack, not the inner 5-tuple), so this fast-path does the same =E2=80=94 no IP tail-call. Multi-label stacks (BoS=3D0 on the first label) defer to the slow path. A depth-2+ variant can land later, mirroring the vlan -> qinq staging used for the VLAN gates. Why this matters: MPLS is the dominant SP backbone L2.5 and the existing slow-path traverses an iteration loop per label even when the consumer just wants the label-stack flow key. The fast-path saves the iteration overhead for the common single-label case while keeping the slow path as escape for multi-label and the entropy-label case (which can only fire from label 2+ anyway). Gated by: - new DEFINE_STATIC_KEY_FALSE(flow_dissector_mpls_key) - new sysctl /proc/sys/net/flow_dissector/mpls (default 0) - dispatcher case htons(ETH_P_MPLS_UC) / htons(ETH_P_MPLS_MC) with static_branch_unlikely(&flow_dissector_mpls_key) guard When the sysctl is 0, dispatcher cost is one not-taken JMP per matching packet =E2=80=94 same per-shape pattern as the other fast-path patches. Documentation/admin-guide/sysctl/net.rst grows an `mpls` subsection under /proc/sys/net/flow_dissector/. The helper also mirrors the slow path's out_good terminal writes: for the standard flow_keys dissectors (which request no MPLS key), the slow path returns OUT_GOOD after the first LSE with nhoff advanced past it, so thoff, n_proto (the MPLS ethertype) and ip_proto (0) are still written at the exit label. Skipping those writes is observable in struct flow_keys and would break the byte-identical contract; the in-tree KUnit equivalence test added later in this series catches exactly this. Assisted-by: Claude:claude-fable-5 sparse smatch Signed-off-by: Dave Seddon --- Documentation/admin-guide/sysctl/net.rst | 19 ++++++ include/net/flow_dissector.h | 1 + net/core/flow_dissector.c | 83 ++++++++++++++++++++++++ 3 files changed, 103 insertions(+) diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin= -guide/sysctl/net.rst index 1096e6ddca39..1ee5419ffaeb 100644 --- a/Documentation/admin-guide/sysctl/net.rst +++ b/Documentation/admin-guide/sysctl/net.rst @@ -534,6 +534,25 @@ code !=3D 0). =20 Default: 0 =20 +mpls +~~~~ + +Single-label MPLS frames (EtherType ``ETH_P_MPLS_UC`` 0x8847 or +``ETH_P_MPLS_MC`` 0x8848 with bottom-of-stack bit set on the first +label). The fast-path reads the 4-byte label entry, extracts +label / TC / TTL / BoS, and writes them to FLOW_DISSECTOR_KEY_MPLS +lse[0] when requested. Byte-identical with the slow path's +__skb_flow_dissect_mpls() for lse_index=3D0, bos=3D1: the slow path +returns OUT_GOOD on BoS without descending into the inner packet +(MPLS hashing is on the label stack, not inner 5-tuple), so this +fast-path does the same. + +Multi-label stacks (BoS=3D0 on the first label) defer to slow path =E2=80= =94 +extending to depth-2 or more is a follow-up, mirroring the +vlan -> qinq staging used earlier in this series. + +Default: 0 + 3. /proc/sys/net/unix - Parameters for Unix domain sockets ---------------------------------------------------------- =20 diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h index df85162959cc..8df7821ba769 100644 --- a/include/net/flow_dissector.h +++ b/include/net/flow_dissector.h @@ -432,6 +432,7 @@ extern struct static_key_false flow_dissector_eth_ip_ke= y; extern struct static_key_false flow_dissector_vlan_key; extern struct static_key_false flow_dissector_qinq_key; extern struct static_key_false flow_dissector_pppoe_key; +extern struct static_key_false flow_dissector_mpls_key; =20 /* struct flow_keys_digest: * diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c index 21a3f5e0ea05..0ec1a6a1bb0e 100644 --- a/net/core/flow_dissector.c +++ b/net/core/flow_dissector.c @@ -48,6 +48,8 @@ DEFINE_STATIC_KEY_FALSE(flow_dissector_qinq_key); EXPORT_SYMBOL(flow_dissector_qinq_key); DEFINE_STATIC_KEY_FALSE(flow_dissector_pppoe_key); EXPORT_SYMBOL(flow_dissector_pppoe_key); +DEFINE_STATIC_KEY_FALSE(flow_dissector_mpls_key); +EXPORT_SYMBOL(flow_dissector_mpls_key); =20 /* IPv4 version/IHL byte of an option-less header: version 4, IHL 5. */ #define FLOW_DIS_IPV4_VIHL_NOOPT 0x45 @@ -1396,6 +1398,73 @@ static bool flow_dissect_fast_pppoe(const struct sk_= buff *skb, nhoff, hlen); } =20 +/* Single MPLS label (BoS=3D1): write FLOW_DISSECTOR_KEY_MPLS lse[0] + * when requested and stop -- the slow path returns OUT_GOOD on BoS + * without descending into the inner packet, so the fast path must + * not tail-call either. Multi-label stacks defer. + */ +static bool flow_dissect_fast_mpls(const struct sk_buff *skb, + struct flow_dissector *flow_dissector, + void *target_container, + const void *data, + __be16 proto, int nhoff, int hlen) +{ + struct flow_dissector_key_control *key_control; + struct flow_dissector_key_basic *key_basic; + struct flow_dissector_key_mpls *key_mpls; + struct flow_dissector_mpls_lse *lse; + const struct mpls_label *hdr; + u32 entry, label, bos; + + if (unlikely(hlen - nhoff < (int)sizeof(*hdr))) + return false; + hdr =3D (const struct mpls_label *)((const u8 *)data + nhoff); + entry =3D ntohl(hdr->entry); + bos =3D (entry & MPLS_LS_S_MASK) >> MPLS_LS_S_SHIFT; + + /* Multi-label stack: defer. */ + if (!bos) + return false; + + /* Neither MPLS key requested: the slow path returns OUT_GOOD without + * writing the key; skipping the write block matches it. + */ + label =3D (entry & MPLS_LS_LABEL_MASK) >> MPLS_LS_LABEL_SHIFT; + + if (dissector_uses_key(flow_dissector, FLOW_DISSECTOR_KEY_MPLS)) { + key_mpls =3D skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_MPLS, + target_container); + lse =3D &key_mpls->ls[0]; + lse->mpls_ttl =3D (entry & MPLS_LS_TTL_MASK) >> MPLS_LS_TTL_SHIFT; + lse->mpls_bos =3D bos; + lse->mpls_tc =3D (entry & MPLS_LS_TC_MASK) >> MPLS_LS_TC_SHIFT; + lse->mpls_label =3D label; + dissector_set_mpls_lse(key_mpls, 0); + } + + /* Mirror the slow path's out_good terminal: nhoff advances past the + * LSE, then thoff/basic are written from the final proto/ip_proto. + */ + nhoff +=3D (int)sizeof(*hdr); + if (dissector_uses_key(flow_dissector, FLOW_DISSECTOR_KEY_CONTROL)) { + key_control =3D skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_CONTROL, + target_container); + key_control->thoff =3D min_t(u16, nhoff, + skb ? skb->len : hlen); + } + if (dissector_uses_key(flow_dissector, FLOW_DISSECTOR_KEY_BASIC)) { + key_basic =3D skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_BASIC, + target_container); + key_basic->n_proto =3D proto; + key_basic->ip_proto =3D 0; + } + + return true; +} + /* Top-level dispatcher: eligibility check (only the two standard * dissectors and flag subset) + per-proto switch with per-shape * static_branch gating. Each case's branch is a forward not-taken JMP @@ -1449,6 +1518,13 @@ static bool flow_dissect_fast(const struct sk_buff *= skb, return flow_dissect_fast_pppoe(skb, flow_dissector, target_container, data, nhoff, hlen); + case htons(ETH_P_MPLS_UC): + case htons(ETH_P_MPLS_MC): + if (!static_branch_unlikely(&flow_dissector_mpls_key)) + return false; + return flow_dissect_fast_mpls(skb, flow_dissector, + target_container, data, + proto, nhoff, hlen); default: return false; } @@ -2598,6 +2674,13 @@ static struct ctl_table flow_dissector_sysctl_table[= ] =3D { .mode =3D 0644, .proc_handler =3D proc_do_static_key, }, + { + .procname =3D "mpls", + .data =3D &flow_dissector_mpls_key.key, + .maxlen =3D sizeof(flow_dissector_mpls_key), + .mode =3D 0644, + .proc_handler =3D proc_do_static_key, + }, }; =20 static int __init flow_dissector_sysctl_init(void) --=20 2.54.0 From nobody Sat Jul 25 16:44:51 2026 Received: from mail-pg1-f181.google.com (mail-pg1-f181.google.com [209.85.215.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C82A52C21F0 for ; Thu, 16 Jul 2026 00:44:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162654; cv=none; b=JYLgwbgNLhCOBNT3Va+lxNzly9zdS2YtIQsy5GRxiZJmZLCHxPYwhpWu/jrSbJFUFmVvaB96j5YfVf7cS1Ipc6/4MdQAfh2xV0rKaImYTPEoeriwm7aCENltMyI0QEPhhNTUTCK7Yo7niJrKZJkT72mJfdi0Ikme8yFkUBCWyc8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162654; c=relaxed/simple; bh=8z2dU6C3TYRJ022H9N3Ej1VRtoM6gFjZ5RUegcBnuak=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=KjXswl6AZCq9soTI0LnvV/j20LfeWwIHVaJ8VGqMZssJcN8w3/LxsRqpeDir61R3anf/7DmOB2mCaPXj4U9DNQ+aQ0UR2qni4iicaGzl06m9mBS1iqCweBsoCsjaiVl70tRlYRlP4+odpJTl/JWiOOlTSJLTjXBKdCfJxO3lhG8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Ag07XFKm; arc=none smtp.client-ip=209.85.215.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Ag07XFKm" Received: by mail-pg1-f181.google.com with SMTP id 41be03b00d2f7-ca766c1c9ccso3516686a12.0 for ; Wed, 15 Jul 2026 17:44:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784162648; x=1784767448; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=vobfv6qQPpjyzGOl008TBJ35Kdt4Q/Z2GOdMKmOELa0=; b=Ag07XFKm1byI2WJ9DgEnCYDBrgGXnCZ81pHFMN2wCPo5ZLlu6qhy6P8ioWmb7lv4PP UteJ/2MHhrWnvgnKxol8Gp+2rrcrp9b8IW214H1UvnS9ginqGXRHmYebRc0+H1VR8D9Y ZbBcbas5hs3Tfygv0OwMWn2vI9AwZWMehU3JNXWQs8CRoV+4bPdU/Uoucv3IgDXqgJv7 yzarAlqWGhR5yaIfHK43Zuz8r6VYoPHTpi0voUlD30snlr1JL604VnzOUWC1GVgTSDGF WHZmbbC5qlLmpaAWASA3ge0dM+G7X/URzd5fQtNcYpzx7DxstkwyhN3XcjweR5innD6z rIVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784162648; x=1784767448; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=vobfv6qQPpjyzGOl008TBJ35Kdt4Q/Z2GOdMKmOELa0=; b=fQSwr194YG12EEeA/WzXUBX5q0EQk7H9CPbKLPg1hglvEOyG7O8fJ/Mn2PxcRWUEmm +9SY+TaDfRCwkSXdMesN8g4eNSRmU8l2NFz4C6YjC7LdLpessB821ia9JPhoxipcRDdW PVTzGM+T+4C7UhrI1GAnusSsCzTQbDXqttbeMuf1u7aKW2RzE+fKajUt1kewaRbZUxbH uhp5XSl7Dwx5rP2kX4jPxceijLBcSdv7OLIWJrojgvf4lPOphrjetpG4q3pVPIJsw/Lz biR6uXMC4c7icH6WjszWxNkxZDJtfbT1Mxs6NtSdVZRO6ezzkGIDAW8zEajR6lmZJeTp jn6g== X-Forwarded-Encrypted: i=1; AHgh+RovBMtyWDye7GGSRQrJG4EjWNfIYTYHDoHf6PgsYvqIWxhZ4dkKDo4l0wJjERDUJY/740oCrtfjSSmD5/A=@vger.kernel.org X-Gm-Message-State: AOJu0YyTPbx4aUZdwgqtEZLtzJGolRPocB3SDTQw3CKG/aSUGUazXRYr X5Yv4W1B5Yx7DyLb0mlucFYQAvEOcFNrchYnx/E4c73+iY/oxug9x8pR X-Gm-Gg: AfdE7cljiJIaKQOqMZ+g7qUEXMB3w5oRNwohH7CxzCS1KrJKoRhuzatKsQ3rpgKwSj0 /hviUB+E7pq9cK1R0nhF7QxUdM3LlyGJ5Bqn/3MGr36NOprHQBtB1GKBcChBce74mamzzgAE4Wv YhkpBYgXK4qZP3vBcPDMo6/zsVkgtRCmXOJbiVe1z156NdIW1wgJ9gBPUQay+yNkdPjYKsFH29P d9oChIBp6jMkVk1W6y9U5F6PD/Z+ggfjrtLMNa1kZvs8k4bHb+iUDTo6pYByKTGc00c4fylu9bC ME39dzsUtdOA9wTc8LpcrF/KFxRvG69m5+bOhVjVuxPnxLQby9BAQLUNlwyioTTtSAZ4t32jFnI ndCTdvmEP1euhIDDQHoPMsYhAv+LB7Irzlo4vNfbmAiJfBY5ayeMnWPY/BhNg2EG8cbtrvghJxr 7uZmVH6Iz6F2+4u1UfcTJheg+L+aC4GN1NUW6rD7jLAGdas37tchyVdgJN4+sFARfm0co7Cra6o 7I/py5TTA== X-Received: by 2002:a05:6a20:4310:b0:3bf:e2dc:bc7d with SMTP id adf61e73a8af0-3c36c3b1de3mr5402079637.57.1784162648082; Wed, 15 Jul 2026 17:44:08 -0700 (PDT) Received: from l.localdomain ([76.166.220.66]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-13cd4284a92sm4107620c88.4.2026.07.15.17.44.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 17:44:07 -0700 (PDT) From: Dave Seddon To: netdev@vger.kernel.org Cc: "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Stanislav Fomichev , Tom Herbert , Willem de Bruijn , linux-kernel@vger.kernel.org, Dave Seddon Subject: [PATCH net-next v1 06/11] net: flow_dissector: add fast-path for IP-in-IP family (IPIP / 4in6 / 6in4) Date: Wed, 15 Jul 2026 17:43:52 -0700 Message-ID: <20260716004357.3652679-7-dave.seddon.ca@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> References: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Adds flow_dissect_fast_ipip_inner() =E2=80=94 a small recursion helper invoked from the existing flow_dissect_fast_ipv4() and flow_dissect_fast_ipv6() helpers when the outer IP's protocol (IPv4) or nexthdr (IPv6) is IPPROTO_IPIP or IPPROTO_IPV6 and the ipip gate is on. Tail-calls into the matching inner IP fast-path then stamps FLOW_DIS_ENCAPSULATION on key_control->flags. This is the series' first tunnel descent: the fast path parses through the tunnel header and dissects the inner flow, the pattern the GRE and UDP-tunnel patches later in the series reuse. Byte-identical with the slow path's IPPROTO_IPIP and IPPROTO_IPV6 switch cases in __skb_flow_dissect(): the slow path also lets the inner IP overwrite outer addrs/basic and OR's ENCAP into the control flags. The ENCAP write must happen *after* the inner returns because the existing fast-path helpers unconditionally set key_control->flags =3D 0 on entry (correct for the top-level call but clobbers the encap flag during recursion); the ipip_inner helper re-establishes ENCAP after the inner returns. Coverage: outer IPv4, protocol =3D=3D IPIP -> inner IPv4 outer IPv4, protocol =3D=3D IPV6 -> inner IPv6 (6in4) outer IPv6, nexthdr =3D=3D IPIP -> inner IPv4 (4in6) outer IPv6, nexthdr =3D=3D IPV6 -> inner IPv6 FLOW_DISSECTOR_F_STOP_BEFORE_ENCAP and STOP_AT_ENCAP don't apply because the top-level dispatcher already rejects anything beyond F_PARSE_1ST_FRAG, so the fast-path entry never sees the encap-stop flags - packets that request them go to the slow path unchanged. Gated by: - new DEFINE_STATIC_KEY_FALSE(flow_dissector_ipip_key) - new sysctl /proc/sys/net/flow_dissector/ipip (default 0) - static_branch_unlikely guard inside the v4/v6 helpers, hit only when the outer protocol/nexthdr is not TCP/UDP (i.e. the TCP/UDP fast-path was about to return false anyway) Cost when off: the not-TCP/UDP fall-through grows by one static_branch_unlikely check. The TCP/UDP hot path is unchanged =E2=80=94 the new branch is below the existing TCP/UDP fast-out. The IPv6 helper's standalone not-TCP/UDP guard becomes unreachable once the new block handles that case in full, so it is removed here. Documentation/admin-guide/sysctl/net.rst grows an `ipip` subsection under /proc/sys/net/flow_dissector/. The IPv6-outer descent mirrors the slow path's outer writes (v6addrs, addr_type) before recursing into the inner header. The slow path fills these for the outer header and the inner pass then overwrites only what it uses: an inner IPv4 overwrites just the first 8 bytes of the addrs union, leaving outer-v6 bytes in the tail. Byte-identical output means reproducing exactly that, residue included =E2=80=94 the KUnit equivalence test's 4in6 cases fail otherwise. (A non-zero outer flow label defers before any descent =E2=80=94 see the eth_ip patch's output contract =E2=80=94 so no label residue exists. The IPv4-outer descent needs no mirroring: an inner IPv4 or IPv6 write always fully covers the outer's 8 address bytes.) Assisted-by: Claude:claude-fable-5 sparse smatch Signed-off-by: Dave Seddon --- Documentation/admin-guide/sysctl/net.rst | 19 ++++ include/net/flow_dissector.h | 1 + net/core/flow_dissector.c | 117 ++++++++++++++++++++++- 3 files changed, 134 insertions(+), 3 deletions(-) diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin= -guide/sysctl/net.rst index 1ee5419ffaeb..15c8ace2c0a4 100644 --- a/Documentation/admin-guide/sysctl/net.rst +++ b/Documentation/admin-guide/sysctl/net.rst @@ -553,6 +553,25 @@ vlan -> qinq staging used earlier in this series. =20 Default: 0 =20 +ipip +~~~~ + +IPv4-in-IPv4 (``IPPROTO_IPIP`` 4), IPv6-in-IPv4 (``IPPROTO_IPV6`` 41 +inside an IPv4 outer), IPv4-in-IPv6 and IPv6-in-IPv6 tunnels. When +the outer IP fast-path notices the outer protocol/nexthdr is one of +these and this gate is on, it skips the outer header and tail-calls +into the inner IPv4 / IPv6 fast-path. The flow_keys ends up with the +inner 5-tuple (matching the slow path's behaviour, where the inner +parse overwrites the outer addrs/basic), plus +FLOW_DIS_ENCAPSULATION set on key_control->flags. + +Byte-identical with the slow path's IPPROTO_IPIP / IPPROTO_IPV6 +switch cases for the eligible shape. Defers on any miss =E2=80=94 including +fragmented outer, unusual outer IHL, or unsupported inner shapes +that the inner IP fast-path itself would defer. + +Default: 0 + 3. /proc/sys/net/unix - Parameters for Unix domain sockets ---------------------------------------------------------- =20 diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h index 8df7821ba769..d9b4b461cd05 100644 --- a/include/net/flow_dissector.h +++ b/include/net/flow_dissector.h @@ -433,6 +433,7 @@ extern struct static_key_false flow_dissector_vlan_key; extern struct static_key_false flow_dissector_qinq_key; extern struct static_key_false flow_dissector_pppoe_key; extern struct static_key_false flow_dissector_mpls_key; +extern struct static_key_false flow_dissector_ipip_key; =20 /* struct flow_keys_digest: * diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c index 0ec1a6a1bb0e..abf4fdb0100b 100644 --- a/net/core/flow_dissector.c +++ b/net/core/flow_dissector.c @@ -50,6 +50,8 @@ DEFINE_STATIC_KEY_FALSE(flow_dissector_pppoe_key); EXPORT_SYMBOL(flow_dissector_pppoe_key); DEFINE_STATIC_KEY_FALSE(flow_dissector_mpls_key); EXPORT_SYMBOL(flow_dissector_mpls_key); +DEFINE_STATIC_KEY_FALSE(flow_dissector_ipip_key); +EXPORT_SYMBOL(flow_dissector_ipip_key); =20 /* IPv4 version/IHL byte of an option-less header: version 4, IHL 5. */ #define FLOW_DIS_IPV4_VIHL_NOOPT 0x45 @@ -65,6 +67,12 @@ static bool flow_dissect_fast_ipv6(const struct sk_buff = *skb, void *target_container, const void *data, int nhoff, int hlen); +static bool flow_dissect_fast_ipip_inner(const struct sk_buff *skb, + struct flow_dissector *flow_dissector, + void *target_container, + const void *data, + __be16 inner_eth_proto, + int inner_nhoff, int hlen); =20 /* One of the two dissectors the fast-path eligibility check admits; * defined here so flow_dissect_fast() below can reference it (its keys @@ -1114,8 +1122,25 @@ static bool flow_dissect_fast_ipv4(const struct sk_b= uff *skb, return false; =20 if (unlikely(iph->protocol !=3D IPPROTO_TCP && - iph->protocol !=3D IPPROTO_UDP)) + iph->protocol !=3D IPPROTO_UDP)) { + /* IPIP / IPv6-in-IPv4 outer: defer to the inner fast-path and + * stamp ENCAP. Outer addrs are not written -- the inner pass + * overwrites them, as in the slow path. + */ + if (static_branch_unlikely(&flow_dissector_ipip_key)) { + if (iph->protocol =3D=3D IPPROTO_IPIP) + return flow_dissect_fast_ipip_inner(skb, + flow_dissector, target_container, + data, htons(ETH_P_IP), + nhoff + (int)sizeof(*iph), hlen); + if (iph->protocol =3D=3D IPPROTO_IPV6) + return flow_dissect_fast_ipip_inner(skb, + flow_dissector, target_container, + data, htons(ETH_P_IPV6), + nhoff + (int)sizeof(*iph), hlen); + } return false; + } =20 thoff =3D nhoff + (int)sizeof(*iph); =20 @@ -1203,8 +1228,49 @@ static bool flow_dissect_fast_ipv6(const struct sk_b= uff *skb, return false; =20 if (unlikely(iph->nexthdr !=3D IPPROTO_TCP && - iph->nexthdr !=3D IPPROTO_UDP)) - return false; + iph->nexthdr !=3D IPPROTO_UDP)) { + /* IPIP / IPV6-in-IPv6 tunnel outer: same pattern as the + * IPv4 helper. The nexthdr_IPIP / nexthdr_IPV6 cases + * defer to the inner IP fast-path and stamp ENCAP. + */ + bool ipip =3D static_branch_unlikely(&flow_dissector_ipip_key) && + (iph->nexthdr =3D=3D IPPROTO_IPIP || + iph->nexthdr =3D=3D IPPROTO_IPV6); + + if (!ipip) + return false; + + /* Mirror the slow path's outer-IPv6 writes before the + * descent. The slow path fills v6addrs for the outer + * header and the inner pass then overwrites only what + * it uses =E2=80=94 an inner IPv4 leaves the tail of the addrs + * union holding outer-v6 bytes. Byte-identical means + * reproducing exactly that, residue included. (The + * outer flow label is always zero here =E2=80=94 a non-zero + * label deferred above =E2=80=94 so there is no label residue.) + */ + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_IPV6_ADDRS)) { + key_addrs =3D skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_IPV6_ADDRS, + target_container); + memcpy(&key_addrs->v6addrs.src, &iph->saddr, + sizeof(key_addrs->v6addrs.src)); + memcpy(&key_addrs->v6addrs.dst, &iph->daddr, + sizeof(key_addrs->v6addrs.dst)); + key_control =3D skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_CONTROL, + target_container); + key_control->addr_type =3D FLOW_DISSECTOR_KEY_IPV6_ADDRS; + } + + return flow_dissect_fast_ipip_inner(skb, + flow_dissector, target_container, + data, + iph->nexthdr =3D=3D IPPROTO_IPIP ? + htons(ETH_P_IP) : htons(ETH_P_IPV6), + nhoff + (int)sizeof(*iph), hlen); + } =20 thoff =3D nhoff + (int)sizeof(*iph); =20 @@ -1465,6 +1531,44 @@ static bool flow_dissect_fast_mpls(const struct sk_b= uff *skb, return true; } =20 +/* IPIP / 4in6 / 6in4 inner descent: recurse into the inner IP + * fast-path, then stamp FLOW_DIS_ENCAPSULATION -- the inner helper + * zeros key_control->flags, so the ENCAP write must come after. + */ +static bool flow_dissect_fast_ipip_inner(const struct sk_buff *skb, + struct flow_dissector *flow_dissector, + void *target_container, + const void *data, + __be16 inner_eth_proto, + int inner_nhoff, int hlen) +{ + struct flow_dissector_key_control *key_control; + bool ok; + + if (inner_eth_proto =3D=3D htons(ETH_P_IP)) + ok =3D flow_dissect_fast_ipv4(skb, flow_dissector, + target_container, data, + inner_nhoff, hlen); + else if (inner_eth_proto =3D=3D htons(ETH_P_IPV6)) + ok =3D flow_dissect_fast_ipv6(skb, flow_dissector, + target_container, data, + inner_nhoff, hlen); + else + return false; + + if (!ok) + return false; + + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_CONTROL)) { + key_control =3D skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_CONTROL, + target_container); + key_control->flags |=3D FLOW_DIS_ENCAPSULATION; + } + return true; +} + /* Top-level dispatcher: eligibility check (only the two standard * dissectors and flag subset) + per-proto switch with per-shape * static_branch gating. Each case's branch is a forward not-taken JMP @@ -2681,6 +2785,13 @@ static struct ctl_table flow_dissector_sysctl_table[= ] =3D { .mode =3D 0644, .proc_handler =3D proc_do_static_key, }, + { + .procname =3D "ipip", + .data =3D &flow_dissector_ipip_key.key, + .maxlen =3D sizeof(flow_dissector_ipip_key), + .mode =3D 0644, + .proc_handler =3D proc_do_static_key, + }, }; =20 static int __init flow_dissector_sysctl_init(void) --=20 2.54.0 From nobody Sat Jul 25 16:44:51 2026 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 22AE230677E for ; Thu, 16 Jul 2026 00:44:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162658; cv=none; b=Yus9buvQHlhhcnZluOVm1vCjuk5GKPohAUEC16TzRYgbErHXiDLtskHJ9Z7Hw0YB/BvgduuP6izymLnCqyxorF6MdG3F8ZksdZtC8Cx2gAi2J9i4MzNTZRBJpgKOPu3yOg3GP4TctejxSlqSDMteVKO9OhJ3SgkODxN8O0Yio1A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162658; c=relaxed/simple; bh=dknHAWVDVr4VbWy52X7Bw83RZF3ZCZHYuBg0q4g0ZQk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=iVw43SDNTSZTk9nLISXKVFXjF0IOdbTgNLbSdFNU6/iU3gYBf/K0vPzJ6eM5ZJepPVmdc9GXJoDnYxSSnmqX0RmS4/VNDvsZ5mM9MkHOub+8m5LOb2ZVqUHBYmUYQKHEWsD+eAtFdgn3vuLVRhrW7EyF8UY3Fef5GD19kpYNBtI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=F5B2r3lv; arc=none smtp.client-ip=209.85.216.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="F5B2r3lv" Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-381216921aaso6231948a91.1 for ; Wed, 15 Jul 2026 17:44:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784162649; x=1784767449; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=3WXKkArdNhSXf7ay7wr0ijzbhyQDDS4XvU5X+d1Sp8s=; b=F5B2r3lvueKB4Rdpe/r0/yUBkLh/jUcsJZ78AUNiHHnh9ff4gxQ4fZ8EwMHnv0ye5G FBFW5fiZV9DiQ0NU6cwOv3/p3poodmWq3IXS2WdFQBlTdqVeGqfWSZNtQBdgTXFqcr8R lT/sS4eE3p7yrfVPa7w0bMAvx2I92/irqahFZPmyZFu+R+Sk+f/0F194vBxmREAbWe3w aN20fxQX5G9fMOpwB7DMXBguxSj0nOw3Y624i6KqYYy1lhJbiLmAF3lBmrQURghDNKXk +PznGhQAi+4P0yoLskBIh5vrcfAFk1CC/mLZcFXJVq/DN6gjHozsMf+F1HaqKdDxD997 jceg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784162649; x=1784767449; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=3WXKkArdNhSXf7ay7wr0ijzbhyQDDS4XvU5X+d1Sp8s=; b=soMrp/uMO9nq41uXIKesQdoQ0GyAt71k0IBV+KNz0IEvTYMiaYD3nHvWuDXBpTC7I5 JVIGGHfoJCDHzyhvMeLANWzuwXPLXHJwdFca4911Zf0UTy8CF5zOqVIBCDBzSaLqr7SW r1WqtPILa1cAQYRanYLrb5R/vheojpeGwKOVu3b8QAH/tG9ta5FewV73Fu//fS/u5UQx nnL3DINUwNG6/ZUWUm/HoUygn/uyA7ZbE3hDo6M3VxJbUBUto9d1vaFOZeFluGj94nkQ CjsqbFSWIruAXsOKYsrEVgBq7tdLSohdJoaNM00XIxehFs0oP8fw7LFVBgLKdPEXuXUN K9Rw== X-Forwarded-Encrypted: i=1; AHgh+Ro5hFasa88i8VhB3QYESDDq6H16oMWcB3jOQkhbOZT6o6o1jhEvCdk6P+jZMaaZQncIa6dYRgdaXNBRDMs=@vger.kernel.org X-Gm-Message-State: AOJu0YwGxVP1yabS2akxAW3L0YpanqxySR2OnAgwjRNS3cLqTgVKBcRx ZBfJcbwjpHPeXgTgEnXczC6sPZRkcZZwGFTRjncvGqEeb5+nAyUK1V71 X-Gm-Gg: AfdE7clXep8hRxaulEw6D1deOrMcARxhJgGiGaQQAEzfX+vhhyYlMT6dScP3FFPPAkE 4wjZwKNWC0YIjDoyY7GTvsHK9KEKDoiQ8abM/l17OQTElAi+Yj9LGqeBRSuW+9/aJrj8zaz0p2E sRuVPGZOzHrqtBiANHlIAJh9H0E9TfHqiMSVFLRQZAd7Wy8892FKHWuPCyBguhmtMulLixd2/vB 9YY1joQoG07fEz16uztBN4/XM4Cx8dax/akUoggYzJrjqYdMyO1++Vd6kwCHnF1khcNY9Y0SNi5 jejqEV4vPfc//nosoR9HINxCbAkuQaa7EYgmVf0XAcRHJQmUUuFIPbXsaXNGqaoqPzsO8zXeMgj AQnOZAkJsuemp/gpRhVp27GENF+x0k0Bc8QFWyColGW3Y38WcOaRgL7O1ImUSPerQAR/sk7SaBn FgmDfiy2F8hyau/bIROJmQLNea2+uPS3/yerCS4e6IYZ/VOivSkTAUh2pyWBm959S3T1CsoNstJ vfGizVMDQ== X-Received: by 2002:a05:6a20:9f04:b0:3c0:b3f7:e5e5 with SMTP id adf61e73a8af0-3c3573959c8mr10863618637.25.1784162649425; Wed, 15 Jul 2026 17:44:09 -0700 (PDT) Received: from l.localdomain ([76.166.220.66]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-13cd4284a92sm4107620c88.4.2026.07.15.17.44.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 17:44:08 -0700 (PDT) From: Dave Seddon To: netdev@vger.kernel.org Cc: "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Stanislav Fomichev , Tom Herbert , Willem de Bruijn , linux-kernel@vger.kernel.org, Dave Seddon Subject: [PATCH net-next v1 07/11] net: flow_dissector: add byte-identical fast-path for plain GRE inner Date: Wed, 15 Jul 2026 17:43:53 -0700 Message-ID: <20260716004357.3652679-8-dave.seddon.ca@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> References: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Adds flow_dissect_fast_gre_inner() =E2=80=94 invoked from inside the existing flow_dissect_fast_ipv4() / flow_dissect_fast_ipv6() helpers when the outer IP's protocol/nexthdr is IPPROTO_GRE and the GRE gate is on. Byte-identical with the slow path's __skb_flow_dissect_gre() for the subset where: - GRE version =3D=3D 0 (v1 =3D PPTP defers to slow path) - All GRE flags clear (no CSUM, KEY, SEQ, ROUTING) - protocol =3D=3D ETH_P_IP or ETH_P_IPV6 (no TEB, PPP, MPLS-over-GRE) In that subset, the GRE base header is exactly 4 bytes; slow path also descends to inner IP (sets *p_proto =3D hdr->protocol and returns PROTO_AGAIN) and stamps FLOW_DIS_ENCAPSULATION on key_control->flags. The fast-path produces the same output: skips the 4-byte header, tail-calls into the inner IP fast-path, re-establishes ENCAP on key_control after the inner overwrites it (same pattern as the IPIP family fast-path from the prior patch). Why this matters: plain GRE (no key, no checksum) is the dominant shape for cloud backbone tunnels (AWS GRE between regions, GCP inter-region overlays) and corporate site-to-site VPNs that don't need per-tunnel keying. The existing slow-path GRE handler does a __skb_header_pointer + flag-walk per packet; the fast-path saves that for the common case while keeping the slow path as the escape valve for GRE-with-flags, v1/PPTP, TEB, and PPP-over-GRE. Gated by: - new DEFINE_STATIC_KEY_FALSE(flow_dissector_gre_key) - new sysctl /proc/sys/net/flow_dissector/gre (default 0) - static_branch_unlikely guard inside the v4/v6 helpers, on the same not-TCP/UDP fall-through path as the IPIP gate (so the eth_ip hot path remains unchanged when this is off) Cost when off: the not-TCP/UDP fall-through grows by one static_branch_unlikely check on top of the IPIP check. The TCP/UDP hot path is untouched. GRE-with-KEY (used in some MPLS-over-GRE and cloud-overlay deployments) is a natural follow-up: same descent shape with the optional 4-byte key field read and a write to FLOW_DISSECTOR_KEY_GRE_KEYID. Documentation/admin-guide/sysctl/net.rst grows a `gre` subsection under /proc/sys/net/flow_dissector/. Classification note: the slow path already descends through plain GRE unconditionally, so this fast path is byte-identical with today's behaviour =E2=80=94 a pure CPU saving. That is unlike the UDP-tunnel descents later in the series (VXLAN/Geneve/GTP-U/FOU/GUE), where the slow path stops at the outer UDP header today and the descent itself is the new, gated behaviour. Assisted-by: Claude:claude-fable-5 sparse smatch Signed-off-by: Dave Seddon --- Documentation/admin-guide/sysctl/net.rst | 25 ++++++ include/net/flow_dissector.h | 1 + net/core/flow_dissector.c | 101 +++++++++++++++++++++-- 3 files changed, 121 insertions(+), 6 deletions(-) diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin= -guide/sysctl/net.rst index 15c8ace2c0a4..dbc819740db0 100644 --- a/Documentation/admin-guide/sysctl/net.rst +++ b/Documentation/admin-guide/sysctl/net.rst @@ -572,6 +572,31 @@ that the inner IP fast-path itself would defer. =20 Default: 0 =20 +gre +~~~ + +GRE-encapsulated inner IP (``IPPROTO_GRE`` 47). The fast-path +mirrors the slow path's __skb_flow_dissect_gre() descent for the +common subset: + +- GRE version 0 (the v1 PPTP variant defers to slow path) +- All GRE flags clear (no GRE_CSUM, GRE_KEY, GRE_SEQ, GRE_ROUTING =E2=80= =94 + i.e. plain 4-byte GRE base header) +- protocol field is ``ETH_P_IP`` 0x0800 or ``ETH_P_IPV6`` 0x86DD + (no Transparent Ethernet Bridging, no PPP-over-GRE, no MPLS- + over-GRE) + +In that subset, slow path also descends to inner IP and stamps +``key_control->flags |=3D FLOW_DIS_ENCAPSULATION``; the fast-path +produces the same output. + +GRE-with-KEY (common in MPLS-over-GRE deployments and some cloud +overlays) is a follow-up patch =E2=80=94 same descent shape with an +additional 4-byte key field read and a write to +``FLOW_DISSECTOR_KEY_GRE_KEYID``. + +Default: 0 + 3. /proc/sys/net/unix - Parameters for Unix domain sockets ---------------------------------------------------------- =20 diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h index d9b4b461cd05..ba7226a42d19 100644 --- a/include/net/flow_dissector.h +++ b/include/net/flow_dissector.h @@ -434,6 +434,7 @@ extern struct static_key_false flow_dissector_qinq_key; extern struct static_key_false flow_dissector_pppoe_key; extern struct static_key_false flow_dissector_mpls_key; extern struct static_key_false flow_dissector_ipip_key; +extern struct static_key_false flow_dissector_gre_key; =20 /* struct flow_keys_digest: * diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c index abf4fdb0100b..b79c80612dee 100644 --- a/net/core/flow_dissector.c +++ b/net/core/flow_dissector.c @@ -52,6 +52,8 @@ DEFINE_STATIC_KEY_FALSE(flow_dissector_mpls_key); EXPORT_SYMBOL(flow_dissector_mpls_key); DEFINE_STATIC_KEY_FALSE(flow_dissector_ipip_key); EXPORT_SYMBOL(flow_dissector_ipip_key); +DEFINE_STATIC_KEY_FALSE(flow_dissector_gre_key); +EXPORT_SYMBOL(flow_dissector_gre_key); =20 /* IPv4 version/IHL byte of an option-less header: version 4, IHL 5. */ #define FLOW_DIS_IPV4_VIHL_NOOPT 0x45 @@ -73,6 +75,11 @@ static bool flow_dissect_fast_ipip_inner(const struct sk= _buff *skb, const void *data, __be16 inner_eth_proto, int inner_nhoff, int hlen); +static bool flow_dissect_fast_gre_inner(const struct sk_buff *skb, + struct flow_dissector *flow_dissector, + void *target_container, + const void *data, + int nhoff, int hlen); =20 /* One of the two dissectors the fast-path eligibility check admits; * defined here so flow_dissect_fast() below can reference it (its keys @@ -1139,6 +1146,11 @@ static bool flow_dissect_fast_ipv4(const struct sk_b= uff *skb, data, htons(ETH_P_IPV6), nhoff + (int)sizeof(*iph), hlen); } + if (static_branch_unlikely(&flow_dissector_gre_key) && + iph->protocol =3D=3D IPPROTO_GRE) + return flow_dissect_fast_gre_inner(skb, flow_dissector, + target_container, data, + nhoff + (int)sizeof(*iph), hlen); return false; } =20 @@ -1236,8 +1248,10 @@ static bool flow_dissect_fast_ipv6(const struct sk_b= uff *skb, bool ipip =3D static_branch_unlikely(&flow_dissector_ipip_key) && (iph->nexthdr =3D=3D IPPROTO_IPIP || iph->nexthdr =3D=3D IPPROTO_IPV6); + bool gre =3D static_branch_unlikely(&flow_dissector_gre_key) && + iph->nexthdr =3D=3D IPPROTO_GRE; =20 - if (!ipip) + if (!ipip && !gre) return false; =20 /* Mirror the slow path's outer-IPv6 writes before the @@ -1264,11 +1278,15 @@ static bool flow_dissect_fast_ipv6(const struct sk_= buff *skb, key_control->addr_type =3D FLOW_DISSECTOR_KEY_IPV6_ADDRS; } =20 - return flow_dissect_fast_ipip_inner(skb, - flow_dissector, target_container, - data, - iph->nexthdr =3D=3D IPPROTO_IPIP ? - htons(ETH_P_IP) : htons(ETH_P_IPV6), + if (ipip) + return flow_dissect_fast_ipip_inner(skb, + flow_dissector, target_container, + data, + iph->nexthdr =3D=3D IPPROTO_IPIP ? + htons(ETH_P_IP) : htons(ETH_P_IPV6), + nhoff + (int)sizeof(*iph), hlen); + return flow_dissect_fast_gre_inner(skb, flow_dissector, + target_container, data, nhoff + (int)sizeof(*iph), hlen); } =20 @@ -1569,6 +1587,70 @@ static bool flow_dissect_fast_ipip_inner(const struc= t sk_buff *skb, return true; } =20 +/* Plain-GRE inner descent (version 0, no flags, inner IP): a 4-byte + * base header the slow path steps over with PROTO_AGAIN. Any flags, + * version 1 (PPTP) or a non-IP inner defers. ENCAP is stamped after + * the inner pass, which zeros key_control->flags. + */ +static bool flow_dissect_fast_gre_inner(const struct sk_buff *skb, + struct flow_dissector *flow_dissector, + void *target_container, + const void *data, + int nhoff, int hlen) +{ + struct flow_dissector_key_control *key_control; + const struct gre_base_hdr *hdr; + __be16 inner_proto; + int inner_nhoff; + bool ok; + + if (unlikely(hlen - nhoff < (int)sizeof(*hdr))) + return false; + hdr =3D (const struct gre_base_hdr *)((const u8 *)data + nhoff); + + /* flags field carries the GRE control flags *and* the 3-bit + * version in its low bits. A zero word means "no flags set + * AND version 0" =E2=80=94 the byte-identical fast subset. + */ + if (hdr->flags !=3D 0) + return false; + + switch (hdr->protocol) { + case htons(ETH_P_IP): + inner_proto =3D htons(ETH_P_IP); + break; + case htons(ETH_P_IPV6): + inner_proto =3D htons(ETH_P_IPV6); + break; + default: + return false; + } + + inner_nhoff =3D nhoff + (int)sizeof(*hdr); + + if (inner_proto =3D=3D htons(ETH_P_IP)) + ok =3D flow_dissect_fast_ipv4(skb, flow_dissector, + target_container, data, + inner_nhoff, hlen); + else + ok =3D flow_dissect_fast_ipv6(skb, flow_dissector, + target_container, data, + inner_nhoff, hlen); + + if (!ok) + return false; + + /* Re-establish ENCAP after the inner pass zeroed key_control->flags. */ + if (dissector_uses_key(flow_dissector, + FLOW_DISSECTOR_KEY_CONTROL)) { + key_control =3D skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_CONTROL, + target_container); + key_control->flags |=3D FLOW_DIS_ENCAPSULATION; + } + return true; +} + /* Top-level dispatcher: eligibility check (only the two standard * dissectors and flag subset) + per-proto switch with per-shape * static_branch gating. Each case's branch is a forward not-taken JMP @@ -2792,6 +2874,13 @@ static struct ctl_table flow_dissector_sysctl_table[= ] =3D { .mode =3D 0644, .proc_handler =3D proc_do_static_key, }, + { + .procname =3D "gre", + .data =3D &flow_dissector_gre_key.key, + .maxlen =3D sizeof(flow_dissector_gre_key), + .mode =3D 0644, + .proc_handler =3D proc_do_static_key, + }, }; =20 static int __init flow_dissector_sysctl_init(void) --=20 2.54.0 From nobody Sat Jul 25 16:44:51 2026 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4625D30CDB6 for ; Thu, 16 Jul 2026 00:44:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162658; cv=none; b=TZCZTMkE8aGWCJ0TOJ9sxFoPlwI8pCx0MKP8EV4xQ3+cYAUimmm+I71CR64ufQ5fsRO0kzOJBisVMVn6kS9zXLKXSmwGdAzCNLHJEy/csegMM6irbmciVV0pGTQVbLjHV1o1KlJqUw2fOEp6/xEsIYBZdG2+vZow4NL30YnIzCU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162658; c=relaxed/simple; bh=p0T+/y9J8K+qskpcjQYFTK7dBe6f1Aa0M6ApkUJiPQY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=qzV8E/br2mMLwNNWyHQkIHWFKD/VcssF18aZyeIn7ZZeyIZiJ8Wf9lXEnqeoAQNglil+q6riWm2fhZkavDi2gVnR96IGwyo8TOrLBFkow6TadFd+PI9cOSU3KS+lzgl5zFEqF4xvUL3mK9dV8KtKGk5UVUI2YScrMgtqhcWObhw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=l9qR5yxj; arc=none smtp.client-ip=209.85.214.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="l9qR5yxj" Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-2cae1a3a744so45785905ad.3 for ; Wed, 15 Jul 2026 17:44:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784162651; x=1784767451; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=zGT9oLZKq+mTL6K4GKPIHr/ZJHA0XEBzbs9yPmGkhFY=; b=l9qR5yxjxoiTVDdxjXnpVbDQ4/X5FlIE1mOoI8w3HXHwspBnjbkwxKm75zeYMvFE4C sCK/LM92efuMqBo8tloqIoKuQzkabsD3U1Y0ceDs/ov0hZepg4wvj/nBHk8yEFdZG89r ZVjOiOcZ1Rjjr2CEC7JPwUJ87yKIvQ6u2hXpehZkehpK3q3qlK/GpsmWT9YmfJFgnLgC qY3FVdyPbcZ4Ea5MF/r01UFngf2U2ggxOBZClULpk6L95SKaMsaMhf22RUpcwdxmSNk6 XQVlYfQYfqL1Wgph210BlQo+jpJ8SO67QJJb+giHNtULRyO5stPkliIcp6tr88AsEWhd U3Qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784162651; x=1784767451; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=zGT9oLZKq+mTL6K4GKPIHr/ZJHA0XEBzbs9yPmGkhFY=; b=NAZBzKk8NcKRRieCvIJCO2CnQ8SHap6PyFT0u/J6VMf6QY3m+CuatRaq1LhjlIk4jR pMQZvVvjuntYXe6P76qHmOu3NMS/nLzIPoqtdYS/mc1nkwFTl1FbT8oz49Wb7/3RSSim /DBKGPMc1/UIg2n8RYZCwdAL/PZ4VDJSDednrFlGsqMAgJys/InDNY+HLGv8oOHyEyIE zKjNP7LuNxUudFYenpn//QrJtHWTlzqDIwRI8YubaX06EjdWwLhH1fq667CGjDml/8JI eDHsqn+CS63ZWUJnbXK4nuHBdaBTaKziUJ4rCHRKrv6hx9/AWtUsYIkBdpM3no7xmClK 7ZHA== X-Forwarded-Encrypted: i=1; AHgh+Rqhoz4U0Tv4O+t4l2zDC+XGt8ztJenwQsSRO/BXQ5ab0MskaiueXLkcOoAvIezv8qKK55MFhFeP3Bv7Yys=@vger.kernel.org X-Gm-Message-State: AOJu0YzIw6eWM4G6jztfoLsTJeK7lMVZyiI9QlayzqeWRlDco+SWAK1t RRq7XRD07AVBf+e4Vfwe+RduwZegg2n5+VsG768QsN5DO0vDr9wNIIth X-Gm-Gg: AfdE7cnzrYtNUAt7GnD2SZ2fihthVbFap/DknPe9Y1Fk8wdnqUQWarBpEAQW3fxbjHE 2UXQRuyo7Eczxf+csAhWtUZBU0hAoYpV3smkYAnn6AHRGzGpXG4C10M5fF+dqgTjEcRYoyHal4F kTSeDhPgdcSVeDx1mzeSpshz12xQ7oi0iMBNOL5GDYW5FMdvAMRTcuFVch5ALXyZJ1gezaog2WB zJLtYVHYgddEr+lh8YHwo+cBzYQ0vujPRxXKiotWR25C/ke9Bl9G9VaGSQqz0QE692FV7+n8HGp Uu26DnqxbhA5syVv97AkkECzcKbqkTcwDS2st+V3IOSVf0LIQ5ZnLn4eAyksI9bw5D5n/WCIxq7 OXhkUB0Hqt77wbm3hrOwnvh2J1CuHR5emAW2d7bJQT9A+Cnj9DoGDLRZvCBmv0grjkrEP4b/YiV d+DUFelP4DMjsHmWMBQNw3hKZWaFYKu1aIWyHJ6BpJCdUqoR5qHxibeo+hGYYDdoBh/OZ91DuR/ ModSKMLcA== X-Received: by 2002:a05:6a21:320b:b0:3bf:e761:6285 with SMTP id adf61e73a8af0-3c34d5d2da4mr11806835637.29.1784162650571; Wed, 15 Jul 2026 17:44:10 -0700 (PDT) Received: from l.localdomain ([76.166.220.66]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-13cd4284a92sm4107620c88.4.2026.07.15.17.44.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 17:44:10 -0700 (PDT) From: Dave Seddon To: netdev@vger.kernel.org Cc: "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Stanislav Fomichev , Tom Herbert , Willem de Bruijn , linux-kernel@vger.kernel.org, Dave Seddon Subject: [PATCH net-next v1 08/11] net: flow_dissector: per-shape counters + /proc/net/flow_dissector_stats Date: Wed, 15 Jul 2026 17:43:54 -0700 Message-ID: <20260716004357.3652679-9-dave.seddon.ca@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> References: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable The opt-in fast-path (this series) gates each byte-identical shape behind its own net.flow_dissector.* sysctl. Operators need to see how traffic is actually composed to decide which gates to enable =E2=80=94 and that same s= ignal lets a policy layer (a userspace agent, or the auto controller proposed in a separate RFC thread) enable/disable the gates automatically instead of hand-tuning knobs. Add per-cpu counters, keyed by a small enum flow_dissector_shape covering the byte-identical shapes (eth_ip, vlan, qinq, pppoe, mpls, ipip, gre): occurrences[] a shape handled by the SLOW path. Measured while its gate is off, this approximates the eligible-fraction signal: how much traffic that shape's fast body would have handled. The slow counts sit at protocol recognition points, which are a superset of fast-path eligibility (a VLAN+ARP frame counts as vlan; an options-bearing IPv4 counts as eth_ip), so eligible% is an upper bound -- the exact hit rate is fast_hits / (occurrences + fast_hits) once a gate is on. Deliberately so: matching the fast path's full eligibility conditions in the slow path would mean re-implementing the fast path's checks there. fast_hits[] the fast body actually ran for that shape (gate on). dissects total dissects =E2=80=94 the denominator. Because the fast path returns before the slow-path classification, exactly one of occurrences/fast_hits is incremented per shaped packet depending on gate state, so (occurrences + fast_hits) / dissects is a gate-invariant eligible fraction. Placement mirrors the two paths precisely: the slow counts sit at each protocol's dissection point -- for eth_ip that is the out: exit label, taken only on a top-level, non-encapsulated, TCP/UDP terminal, so a packet counted as VLAN/IPIP/GRE is never also counted as eth_ip; the fast counts sit at each shape's success terminal, so a fall-through to the slow path is never double-counted. Expose the summed counters read-only at /proc/net/flow_dissector_stats (documented in Documentation/admin-guide/sysctl/net.rst), one line per shape with occurrences, fast_hits, computed eligible%, and the current gate state. Counters and the file are global (init_net), matching the global gates; the file is visible in the init netns only, so containers do not see it. Per-netns is a noted follow-up. Dissects fully handled by an attached netns BPF flow dissector program are deliberately not counted: the dissects increment sits after the BPF hook's early return, so the file counts only dissects that reach the in-kernel dissector. That keeps fast_hits/dissects an undiluted hit-rate for the code these gates control =E2=80=94 the signal an operator = (or the auto controller proposed in a separate RFC thread) acts on. The counters are unconditional by design, not oversight: their whole purpose is observing traffic composition while the gates are OFF, so a "stats gate" would blind exactly the signal an operator (or the auto-enable controller proposed separately) needs before enabling anything. Cost: one unconditional this_cpu_inc per dissect (the denominator) plus one this_cpu_inc at the matched shape's classification point, summed only on read. Measured on a CPU-bound pktgen soak this is +0.74% dissector time (+0.44 sigma -- within run-to-run noise); see the cover letter. The gates themselves are unaffected: an off gate stays a not-taken static branch. Assisted-by: Claude:claude-fable-5 sparse smatch Signed-off-by: Dave Seddon --- Documentation/admin-guide/sysctl/net.rst | 39 ++++ include/net/flow_dissector.h | 17 ++ net/core/flow_dissector.c | 236 ++++++++++++++++++++--- 3 files changed, 270 insertions(+), 22 deletions(-) diff --git a/Documentation/admin-guide/sysctl/net.rst b/Documentation/admin= -guide/sysctl/net.rst index dbc819740db0..3b55b2528ba2 100644 --- a/Documentation/admin-guide/sysctl/net.rst +++ b/Documentation/admin-guide/sysctl/net.rst @@ -597,6 +597,45 @@ additional 4-byte key field read and a write to =20 Default: 0 =20 +The ``/proc/net/flow_dissector_stats`` observability file +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Read-only. Reports, per byte-identical shape, the number of packets the +slow path handled (``occurrences``), the number the fast body handled +(``fast_hits``), the resulting eligible fraction, and the current gate +state, plus a total dissect count. Because the fast path returns before +the slow-path classification, exactly one of the two counters is +incremented per shaped packet depending on gate state, so +``(occurrences + fast_hits) / dissects`` is a gate-invariant measure of +how much traffic each shape's fast body would handle -- the signal an +operator uses to decide whether a gate is worth enabling. The slow-path +counts sit at protocol recognition points, a superset of fast-path +eligibility (a VLAN+ARP frame counts as vlan), so the eligible fraction +is an upper bound; the exact hit rate once a gate is on is +``fast_hits / (occurrences + fast_hits)``. The counters +are per-cpu and summed on read; the per-packet cost is one increment on +the already-hot classification path. + +An example snapshot, from a host that mostly sees plain Eth + IP traffic +with some VLAN, before any gate has been enabled:: + + # cat /proc/net/flow_dissector_stats + shape occurrences fast_hits eligible% gate + eth_ip 1866909 0 75.71 off + vlan 598847 0 24.28 off + qinq 0 0 0.00 off + pppoe 0 0 0.00 off + mpls 0 0 0.00 off + ipip 0 0 0.00 off + gre 0 0 0.00 off + dissects: 2465765 + +Read top-down: 76% of dissects are plain Eth + IP and 24% are +VLAN-tagged, so ``eth_ip`` and ``vlan`` are the shapes worth enabling on +this host; the other shapes are absent. Every gate is still off, so all +the traffic sits under ``occurrences``; enabling a gate moves that +shape's packets to ``fast_hits``. + 3. /proc/sys/net/unix - Parameters for Unix domain sockets ---------------------------------------------------------- =20 diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h index ba7226a42d19..53b16f7306c7 100644 --- a/include/net/flow_dissector.h +++ b/include/net/flow_dissector.h @@ -436,6 +436,23 @@ extern struct static_key_false flow_dissector_mpls_key; extern struct static_key_false flow_dissector_ipip_key; extern struct static_key_false flow_dissector_gre_key; =20 +/* Per-shape counters for /proc/net/flow_dissector_stats. + * occurrences[]: shape seen by the slow path (the eligible-fraction + * signal while its gate is off); fast_hits[]: fast path ran. + * Byte-identical shapes only; the behaviour-changing descents are + * never counted or auto-managed. + */ +enum flow_dissector_shape { + FLOW_DISSECTOR_SHAPE_ETH_IP, + FLOW_DISSECTOR_SHAPE_VLAN, + FLOW_DISSECTOR_SHAPE_QINQ, + FLOW_DISSECTOR_SHAPE_PPPOE, + FLOW_DISSECTOR_SHAPE_MPLS, + FLOW_DISSECTOR_SHAPE_IPIP, + FLOW_DISSECTOR_SHAPE_GRE, + FLOW_DISSECTOR_SHAPE__MAX, +}; + /* struct flow_keys_digest: * * This structure is used to hold a digest of the full flow keys. This is a diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c index b79c80612dee..2a4b9b2af834 100644 --- a/net/core/flow_dissector.c +++ b/net/core/flow_dissector.c @@ -36,6 +36,10 @@ #include #endif #include +#include +#include +#include +#include =20 /* Per-shape fast-path gates, one static_branch per shape so operators * enable only what their deployment carries. All default off. @@ -55,6 +59,35 @@ EXPORT_SYMBOL(flow_dissector_ipip_key); DEFINE_STATIC_KEY_FALSE(flow_dissector_gre_key); EXPORT_SYMBOL(flow_dissector_gre_key); =20 +/* Per-cpu, summed on read via /proc/net/flow_dissector_stats. + * occurrences[]: shape handled by the slow path (with the gate off, + * the shape's eligible-fraction signal); fast_hits[]: fast body ran; + * dissects: denominator. + */ +struct flow_dissector_stats { + u64 occurrences[FLOW_DISSECTOR_SHAPE__MAX]; + u64 fast_hits[FLOW_DISSECTOR_SHAPE__MAX]; + u64 dissects; +}; + +static DEFINE_PER_CPU(struct flow_dissector_stats, flow_dissector_pcpu_sta= ts); + +static inline void flow_dissector_count_slow(enum flow_dissector_shape sha= pe) +{ + this_cpu_inc(flow_dissector_pcpu_stats.occurrences[shape]); +} + +static inline void flow_dissector_count_fast(enum flow_dissector_shape sha= pe) +{ + this_cpu_inc(flow_dissector_pcpu_stats.fast_hits[shape]); +} + +/* Counting map: eth_ip counts in the dispatcher (only when the leaf + * did not set ENCAP), vlan/qinq/pppoe/mpls in their helpers, ipip/gre + * in the inner-descent helpers. The slow path counts where it + * recognises each shape, gate on or off. + */ + /* IPv4 version/IHL byte of an option-less header: version 4, IHL 5. */ #define FLOW_DIS_IPV4_VIHL_NOOPT 0x45 =20 @@ -1354,6 +1387,7 @@ static bool flow_dissect_fast_vlan(const struct sk_bu= ff *skb, __be16 saved_tpid; u16 tci_prio; u16 tci_id; + bool ok; =20 if (vlan_depth >=3D 1 && !static_branch_unlikely(&flow_dissector_qinq_key)) @@ -1403,22 +1437,35 @@ static bool flow_dissect_fast_vlan(const struct sk_= buff *skb, =20 switch (inner_proto) { case htons(ETH_P_IP): - return flow_dissect_fast_ipv4(skb, flow_dissector, - target_container, data, - nhoff, hlen); + ok =3D flow_dissect_fast_ipv4(skb, flow_dissector, + target_container, data, + nhoff, hlen); + break; case htons(ETH_P_IPV6): - return flow_dissect_fast_ipv6(skb, flow_dissector, - target_container, data, - nhoff, hlen); + ok =3D flow_dissect_fast_ipv6(skb, flow_dissector, + target_container, data, + nhoff, hlen); + break; case htons(ETH_P_8021Q): case htons(ETH_P_8021AD): - return flow_dissect_fast_vlan(skb, flow_dissector, - target_container, data, - inner_proto, nhoff, hlen, - vlan_depth + 1); + ok =3D flow_dissect_fast_vlan(skb, flow_dissector, + target_container, data, + inner_proto, nhoff, hlen, + vlan_depth + 1); + break; default: return false; } + + /* Count only on full success -- a miss defers and the slow path + * counts the occurrence. Depth 0 counts vlan, depth >=3D 1 qinq; a + * double-tagged hit counts both, as the slow path does. + */ + if (ok) + flow_dissector_count_fast(vlan_depth =3D=3D 0 ? + FLOW_DISSECTOR_SHAPE_VLAN : + FLOW_DISSECTOR_SHAPE_QINQ); + return ok; } =20 /* PPPoE session (RFC 2516) + 2-byte PPP protocol: write @@ -1439,6 +1486,7 @@ static bool flow_dissect_fast_pppoe(const struct sk_b= uff *skb, } *hdr; __be16 inner_eth_proto; u16 ppp_proto; + bool ok; =20 if (unlikely(hlen - nhoff < (int)sizeof(*hdr))) return false; @@ -1474,12 +1522,17 @@ static bool flow_dissect_fast_pppoe(const struct sk= _buff *skb, nhoff +=3D PPPOE_SES_HLEN; =20 if (inner_eth_proto =3D=3D htons(ETH_P_IP)) - return flow_dissect_fast_ipv4(skb, flow_dissector, - target_container, data, - nhoff, hlen); - return flow_dissect_fast_ipv6(skb, flow_dissector, - target_container, data, - nhoff, hlen); + ok =3D flow_dissect_fast_ipv4(skb, flow_dissector, + target_container, data, + nhoff, hlen); + else + ok =3D flow_dissect_fast_ipv6(skb, flow_dissector, + target_container, data, + nhoff, hlen); + + if (ok) + flow_dissector_count_fast(FLOW_DISSECTOR_SHAPE_PPPOE); + return ok; } =20 /* Single MPLS label (BoS=3D1): write FLOW_DISSECTOR_KEY_MPLS lse[0] @@ -1546,6 +1599,7 @@ static bool flow_dissect_fast_mpls(const struct sk_bu= ff *skb, key_basic->ip_proto =3D 0; } =20 + flow_dissector_count_fast(FLOW_DISSECTOR_SHAPE_MPLS); return true; } =20 @@ -1584,6 +1638,7 @@ static bool flow_dissect_fast_ipip_inner(const struct= sk_buff *skb, target_container); key_control->flags |=3D FLOW_DIS_ENCAPSULATION; } + flow_dissector_count_fast(FLOW_DISSECTOR_SHAPE_IPIP); return true; } =20 @@ -1648,9 +1703,27 @@ static bool flow_dissect_fast_gre_inner(const struct= sk_buff *skb, target_container); key_control->flags |=3D FLOW_DIS_ENCAPSULATION; } + flow_dissector_count_fast(FLOW_DISSECTOR_SHAPE_GRE); return true; } =20 +/* Did a fast-path leaf stamp FLOW_DIS_ENCAPSULATION? The dispatcher + * counts eth_ip only for plain top-level IP; descents count under + * their own shape (UDP-tunnel descents not at all). + */ +static bool flow_dissect_fast_is_encap(struct flow_dissector *flow_dissect= or, + void *target_container) +{ + struct flow_dissector_key_control *key_control; + + if (!dissector_uses_key(flow_dissector, FLOW_DISSECTOR_KEY_CONTROL)) + return false; + key_control =3D skb_flow_dissector_target(flow_dissector, + FLOW_DISSECTOR_KEY_CONTROL, + target_container); + return key_control->flags & FLOW_DIS_ENCAPSULATION; +} + /* Top-level dispatcher: eligibility check (only the two standard * dissectors and flag subset) + per-proto switch with per-shape * static_branch gating. Each case's branch is a forward not-taken JMP @@ -1689,15 +1762,24 @@ static bool flow_dissect_fast(const struct sk_buff = *skb, case htons(ETH_P_IP): if (!static_branch_unlikely(&flow_dissector_eth_ip_key)) return false; - return flow_dissect_fast_ipv4(skb, flow_dissector, - target_container, data, - nhoff, hlen); + if (!flow_dissect_fast_ipv4(skb, flow_dissector, + target_container, data, + nhoff, hlen)) + return false; + /* Plain top-level eth+IP only; descents count under their own shape. */ + if (!flow_dissect_fast_is_encap(flow_dissector, target_container)) + flow_dissector_count_fast(FLOW_DISSECTOR_SHAPE_ETH_IP); + return true; case htons(ETH_P_IPV6): if (!static_branch_unlikely(&flow_dissector_eth_ip_key)) return false; - return flow_dissect_fast_ipv6(skb, flow_dissector, - target_container, data, - nhoff, hlen); + if (!flow_dissect_fast_ipv6(skb, flow_dissector, + target_container, data, + nhoff, hlen)) + return false; + if (!flow_dissect_fast_is_encap(flow_dissector, target_container)) + flow_dissector_count_fast(FLOW_DISSECTOR_SHAPE_ETH_IP); + return true; case htons(ETH_P_PPP_SES): if (!static_branch_unlikely(&flow_dissector_pppoe_key)) return false; @@ -1751,6 +1833,8 @@ bool __skb_flow_dissect(const struct net *net, bool mpls_el =3D false; int mpls_lse =3D 0; int num_hdrs =3D 0; + int nhoff_init =3D 0; + bool eth_ip_top =3D false; u8 ip_proto =3D 0; bool ret; =20 @@ -1859,6 +1943,8 @@ bool __skb_flow_dissect(const struct net *net, /* Opt-in fast-path; flow_dissect_fast() does the eligibility check * and per-shape gating. */ + this_cpu_inc(flow_dissector_pcpu_stats.dissects); + if (flow_dissect_fast(skb, flow_dissector, target_container, data, proto, nhoff, hlen, flags)) return true; @@ -1892,6 +1978,8 @@ bool __skb_flow_dissect(const struct net *net, key_num_of_vlans->num_of_vlans =3D 0; } =20 + nhoff_init =3D nhoff; + proto_again: fdret =3D FLOW_DISSECT_RET_CONTINUE; =20 @@ -1906,6 +1994,10 @@ bool __skb_flow_dissect(const struct net *net, break; } =20 + /* Top-level eth+IPv4: eth_ip shape candidate (confirmed at out:). */ + if (nhoff =3D=3D nhoff_init) + eth_ip_top =3D true; + nhoff +=3D iph->ihl * 4; =20 ip_proto =3D iph->protocol; @@ -1954,6 +2046,9 @@ bool __skb_flow_dissect(const struct net *net, break; } =20 + if (nhoff =3D=3D nhoff_init) + eth_ip_top =3D true; + ip_proto =3D iph->nexthdr; nhoff +=3D sizeof(struct ipv6hdr); =20 @@ -2027,8 +2122,12 @@ bool __skb_flow_dissect(const struct net *net, =20 if (dissector_vlan =3D=3D FLOW_DISSECTOR_KEY_MAX) { dissector_vlan =3D FLOW_DISSECTOR_KEY_VLAN; + /* First (outer) tag: the vlan fast-path shape. */ + flow_dissector_count_slow(FLOW_DISSECTOR_SHAPE_VLAN); } else if (dissector_vlan =3D=3D FLOW_DISSECTOR_KEY_VLAN) { dissector_vlan =3D FLOW_DISSECTOR_KEY_CVLAN; + /* Second tag: the qinq (double-tag) fast-path shape. */ + flow_dissector_count_slow(FLOW_DISSECTOR_SHAPE_QINQ); } else { fdret =3D FLOW_DISSECT_RET_PROTO_AGAIN; break; @@ -2074,6 +2173,8 @@ bool __skb_flow_dissect(const struct net *net, break; } =20 + flow_dissector_count_slow(FLOW_DISSECTOR_SHAPE_PPPOE); + /* PFC (compressed 1-byte protocol) frames are not processed. * A compressed protocol field has the least significant bit of * the most significant octet set, which will fail the following @@ -2138,6 +2239,7 @@ bool __skb_flow_dissect(const struct net *net, =20 case htons(ETH_P_MPLS_UC): case htons(ETH_P_MPLS_MC): + flow_dissector_count_slow(FLOW_DISSECTOR_SHAPE_MPLS); fdret =3D __skb_flow_dissect_mpls(skb, flow_dissector, target_container, data, nhoff, hlen, mpls_lse, @@ -2235,6 +2337,7 @@ bool __skb_flow_dissect(const struct net *net, fdret =3D FLOW_DISSECT_RET_OUT_GOOD; break; } + flow_dissector_count_slow(FLOW_DISSECTOR_SHAPE_GRE); =20 fdret =3D __skb_flow_dissect_gre(skb, key_control, flow_dissector, target_container, data, @@ -2297,6 +2400,10 @@ bool __skb_flow_dissect(const struct net *net, fdret =3D FLOW_DISSECT_RET_OUT_GOOD; break; } + /* Counted below the STOP check: a stop-flagged dissect is + * one the fast path could never have handled. + */ + flow_dissector_count_slow(FLOW_DISSECTOR_SHAPE_IPIP); =20 proto =3D htons(ETH_P_IP); =20 @@ -2388,6 +2495,15 @@ bool __skb_flow_dissect(const struct net *net, key_basic->n_proto =3D proto; key_basic->ip_proto =3D ip_proto; =20 + /* eth_ip shape: top-level eth+IP, TCP/UDP, no encap -- counted here + * because the fast path returns earlier; with the gate off this is + * the shape's eligible-fraction signal. + */ + if (ret && eth_ip_top && + !(key_control->flags & FLOW_DIS_ENCAPSULATION) && + (ip_proto =3D=3D IPPROTO_TCP || ip_proto =3D=3D IPPROTO_UDP)) + flow_dissector_count_slow(FLOW_DISSECTOR_SHAPE_ETH_IP); + return ret; =20 out_bad: @@ -2883,11 +2999,87 @@ static struct ctl_table flow_dissector_sysctl_table= [] =3D { }, }; =20 +static const char * const flow_dissector_shape_names[FLOW_DISSECTOR_SHAPE_= _MAX] =3D { + [FLOW_DISSECTOR_SHAPE_ETH_IP] =3D "eth_ip", + [FLOW_DISSECTOR_SHAPE_VLAN] =3D "vlan", + [FLOW_DISSECTOR_SHAPE_QINQ] =3D "qinq", + [FLOW_DISSECTOR_SHAPE_PPPOE] =3D "pppoe", + [FLOW_DISSECTOR_SHAPE_MPLS] =3D "mpls", + [FLOW_DISSECTOR_SHAPE_IPIP] =3D "ipip", + [FLOW_DISSECTOR_SHAPE_GRE] =3D "gre", +}; + +static bool flow_dissector_shape_gate(enum flow_dissector_shape shape) +{ + switch (shape) { + case FLOW_DISSECTOR_SHAPE_ETH_IP: + return static_key_enabled(&flow_dissector_eth_ip_key); + case FLOW_DISSECTOR_SHAPE_VLAN: + return static_key_enabled(&flow_dissector_vlan_key); + case FLOW_DISSECTOR_SHAPE_QINQ: + return static_key_enabled(&flow_dissector_qinq_key); + case FLOW_DISSECTOR_SHAPE_PPPOE: + return static_key_enabled(&flow_dissector_pppoe_key); + case FLOW_DISSECTOR_SHAPE_MPLS: + return static_key_enabled(&flow_dissector_mpls_key); + case FLOW_DISSECTOR_SHAPE_IPIP: + return static_key_enabled(&flow_dissector_ipip_key); + case FLOW_DISSECTOR_SHAPE_GRE: + return static_key_enabled(&flow_dissector_gre_key); + default: + return false; + } +} + +/* /proc/net/flow_dissector_stats: per-shape occurrences (slow path), + * fast_hits, and eligible% =3D (occurrences + fast_hits) / dissects -- + * the signal a policy layer thresholds against a per-shape + * break-even. + */ +static int flow_dissector_stats_show(struct seq_file *seq, void *v) +{ + u64 occ[FLOW_DISSECTOR_SHAPE__MAX] =3D {}; + u64 fast[FLOW_DISSECTOR_SHAPE__MAX] =3D {}; + u64 dissects =3D 0; + int cpu, i; + + for_each_possible_cpu(cpu) { + const struct flow_dissector_stats *s =3D + per_cpu_ptr(&flow_dissector_pcpu_stats, cpu); + + for (i =3D 0; i < FLOW_DISSECTOR_SHAPE__MAX; i++) { + occ[i] +=3D READ_ONCE(s->occurrences[i]); + fast[i] +=3D READ_ONCE(s->fast_hits[i]); + } + dissects +=3D READ_ONCE(s->dissects); + } + + seq_printf(seq, "%-8s %16s %16s %10s %5s\n", + "shape", "occurrences", "fast_hits", "eligible%", "gate"); + for (i =3D 0; i < FLOW_DISSECTOR_SHAPE__MAX; i++) { + u64 total =3D occ[i] + fast[i]; + u64 pct_x100 =3D dissects ? div64_u64(total * 10000, dissects) : 0; + + seq_printf(seq, "%-8s %16llu %16llu %7llu.%02llu %5s\n", + flow_dissector_shape_names[i], occ[i], fast[i], + pct_x100 / 100, pct_x100 % 100, + str_on_off(flow_dissector_shape_gate(i))); + } + seq_printf(seq, "dissects: %llu\n", dissects); + return 0; +} + static int __init flow_dissector_sysctl_init(void) { if (!register_net_sysctl(&init_net, "net/flow_dissector", flow_dissector_sysctl_table)) return -ENOMEM; + + /* Global (init_net) counters =E2=80=94 matches the global gates; per-net= ns + * is a noted follow-up. A missing proc file is not fatal. + */ + proc_create_single("flow_dissector_stats", 0444, init_net.proc_net, + flow_dissector_stats_show); return 0; } late_initcall(flow_dissector_sysctl_init); --=20 2.54.0 From nobody Sat Jul 25 16:44:51 2026 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3A36E30AD05 for ; Thu, 16 Jul 2026 00:44:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162659; cv=none; b=uQwnrhuemVqM0gcxdyljbnbhuuMjdKoOWDMjXubnxSk91nM14KPUSBfzUxSnjfuXC925kVnu3DlB+6goAsF9Nri/zTY0GZqLWmEb/CfIPf/KH2xQYq4BeoDmxJ2owkPUH8xgHyN7hTgh0qDCsjiNbJBmvHmki9cneywz+2BattQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162659; c=relaxed/simple; bh=0HZwnPZ2rcelUtMycwk6J0KLMzgtNL/IZgjQYn+uei0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ugk9tFGzutZsA7IeviHQAwWnnlfjvEUIfQdIgMW1F5LqmQYbrrVv0sdATIAXnrWgWx1ccFi4DMZdZX1+0PdwmkTzEJLm4mcIOhNJzrY5bIvSohK5n9sM/oMTLE9gDopIsbLUJH3Fe/nSsal3/spR7ZkxDFiVopQty/9OpbfdlVw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=euVybSqt; arc=none smtp.client-ip=209.85.215.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="euVybSqt" Received: by mail-pg1-f169.google.com with SMTP id 41be03b00d2f7-ca12086c06eso5068590a12.0 for ; Wed, 15 Jul 2026 17:44:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784162652; x=1784767452; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=usi9ZiTk3Ofj4fjNFF5pMwohzAn15bDlv9dy6F3lFnY=; b=euVybSqtMyV8MPfOdSc06Nep0ZpP8aiDDVtVH4iAUG+4Zsu4S7F3E955x2Z67AnwcF VJ03DL3AbIDbvA+KpvEPgyCy4dRWFmhVQ4DtRId9x6GIgbxYGFMAeOz5BzKe4Fw9Hwis w1K0fYWX/PeUiwe0vPbThF4vyDLzm9VGb6bx74H7R8koHntDIatCWsq0S7ISqezMk5q8 d+DfzrvU3pxuJrtGfrb0cuoYHPtW4bVMVMeZMFU8n4jYiff8ql0szQIkGu0oiYd2f273 2nA+ZgFNH0amrbIcnJ5sB1ju0lnm8ktL1WkiiOrGLiUuoOYqP9cikV9fBIOGxHlyfW72 a4rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784162652; x=1784767452; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=usi9ZiTk3Ofj4fjNFF5pMwohzAn15bDlv9dy6F3lFnY=; b=Z6F2ep+G65LTBvrPYn7JNb1RZ+To82p6D8OAklZKHTw3rr24XWMWKeJTBdlViAoKwO J10AChYX/aBrDLQU4uTCwo3xcbDweARwwtSd7zRhsqFo5H3UsThT8PMEcrkzCNHyxJG0 mW0jgbCozDpJmhRDx+odzyo1C/3fZkdoJxJXGNhW0FGaTOalV26Fj6PKcaWTlyirvPGR rpKpKqENfcvL6/k/FxCshEs6bpWzX41mQnrUemj/VkTXMoWT91LU/0z5PUbLXNMsHkJl 6UCgkedEIy62ca4W51+2HslH5K/WILZqlkuV3LNcc6URnYxa95x0z3nqoxWYmeYnpcy9 x+tw== X-Forwarded-Encrypted: i=1; AHgh+RoUM6GE18zPoW7epA23PaytZaGUX70TX1SXHln5qgXXjnOS6TXOc5mSJpDqswyUmUSqL4DK06hI8aTT57Q=@vger.kernel.org X-Gm-Message-State: AOJu0YzNiDquYZ2ZEBnn8vSGReixqz0ux/6w+hsptyRmkQGg0VE5WV3o anOeoZAdoCxhlV+xjSQOKDtaAsDJaFSrDzkZZ9qn15/Rlufae3dfVUKU X-Gm-Gg: AfdE7cmi7k9nYhgavfLVQ/RYpFLR//Gmj6nQQDBK+ww5yX4a+mbwNFfZulLUydqqLuu Weeo8CZb8u40N0Id4pg4eCT5DrzW/w6jlCkJDlWAbZa0CtihuivaG2BjRFwEZ9jmYVy1p3WROdd VEPE/kU9ybWd36yVoohL8vLQjih145zXee0sJBbnXdMHer/6F/uyI3uBUOJab9ChOJQ7Fi7sMNu vrG1pZPaXtVRzYwSZRYyQJUGojcabB5o5mVSar2h4bLgC9Wa4p3+M8oA3i0Ikcvamu/2DqEwNHD SmuU6I76uuKDsMgQ19xKLhScSoVsW8BrgbYvms4fPy4FSWtvzWWTGL9sd/M0aUe5wlQuytdaSdT f7+aqDjfypGhIhqYBmJSVrsTWGt1+juHFpX5kYm7hB8m4jMpsTakcAj5VA4zcQogzhG+Eyj+stM bps8eZdjQ/5hqhO6k75+T0zpwFbtZe4CW8JahtART+cuayI1Py2iCB8ErJh9BwLTIl6o+jOQ1vb +URnDrTiQ== X-Received: by 2002:a05:6a20:d706:b0:3c3:4ab3:608f with SMTP id adf61e73a8af0-3c36c1eadbcmr5590169637.29.1784162651619; Wed, 15 Jul 2026 17:44:11 -0700 (PDT) Received: from l.localdomain ([76.166.220.66]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-13cd4284a92sm4107620c88.4.2026.07.15.17.44.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 17:44:11 -0700 (PDT) From: Dave Seddon To: netdev@vger.kernel.org Cc: "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Stanislav Fomichev , Tom Herbert , Willem de Bruijn , linux-kernel@vger.kernel.org, Dave Seddon Subject: [PATCH net-next v1 09/11] net: flow_dissector: bound fast-path tunnel recursion Date: Wed, 15 Jul 2026 17:43:55 -0700 Message-ID: <20260716004357.3652679-10-dave.seddon.ca@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> References: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The opt-in fast path descends into IP-in-IP and GRE tunnels by recursion: flow_dissect_fast_ipv4()/ipv6() call flow_dissect_fast_ipip_inner()/gre_inner(), which parse the inner IP header and call back into flow_dissect_fast_ipv4()/ipv6(). Unlike the slow path, which caps total header descents at MAX_FLOW_DISSECT_HDRS via skb_flow_dissect_allowed(), the fast helpers had no such bound. A crafted frame with a deeply nested tunnel chain (IP-in-IP-in-IP-in-..., or GRE) therefore drives one C stack frame per ~20 bytes of linear header. hlen is skb_headlen(), so a large, GRO-coalesced or locally crafted skb yields hundreds to thousands of levels and can exhaust the kernel stack. The inner helpers also do work after the recursive call (they stamp FLOW_DIS_ENCAPSULATION), so the frames are not tail-call eliminable. It is additionally an output divergence: past 15 headers the slow path stops and reports the 15th-level keys, while the fast path would recurse to the innermost, breaking the byte-identical contract. Thread a num_hdrs counter through the tunnel-reachable helpers and, at each tunnel descent, mirror skb_flow_dissect_allowed(): once the count exceeds MAX_FLOW_DISSECT_HDRS the helper returns false and the dissect falls back to the slow path (which caps at the same limit). For chains within the limit both paths descend fully and agree; beyond it the fast path defers to the slow path's result, so the output stays identical and the recursion is bounded. The initial count at each fast-path entry is set at or above the number of headers the slow path has already consumed (outer VLAN tags, PPPoE, the outer IP), so the fast path never descends past the point the slow path would have capped. The added KUnit equivalence test's deep-nest case exercises IPIP and GRE chains across the cap boundary; with the bound removed it fails at 16 levels, confirming it guards this fix. Assisted-by: Claude:claude-fable-5 sparse smatch Signed-off-by: Dave Seddon --- net/core/flow_dissector.c | 61 ++++++++++++++++++++++++--------------- 1 file changed, 37 insertions(+), 24 deletions(-) diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c index 2a4b9b2af834..49b256ce5c1e 100644 --- a/net/core/flow_dissector.c +++ b/net/core/flow_dissector.c @@ -91,28 +91,31 @@ static inline void flow_dissector_count_fast(enum flow_= dissector_shape shape) /* IPv4 version/IHL byte of an option-less header: version 4, IHL 5. */ #define FLOW_DIS_IPV4_VIHL_NOOPT 0x45 =20 -/* Fast-path helper forward declarations. */ +/* @num_hdrs: protocol headers consumed so far (Ethernet =3D 1, each VLAN + * tag, each outer IP header). Tunnel descents increment it and defer + * past MAX_FLOW_DISSECT_HDRS, so both paths stop at the same depth. + */ static bool flow_dissect_fast_ipv4(const struct sk_buff *skb, struct flow_dissector *flow_dissector, void *target_container, const void *data, - int nhoff, int hlen); + int nhoff, int hlen, int num_hdrs); static bool flow_dissect_fast_ipv6(const struct sk_buff *skb, struct flow_dissector *flow_dissector, void *target_container, const void *data, - int nhoff, int hlen); + int nhoff, int hlen, int num_hdrs); static bool flow_dissect_fast_ipip_inner(const struct sk_buff *skb, struct flow_dissector *flow_dissector, void *target_container, const void *data, __be16 inner_eth_proto, - int inner_nhoff, int hlen); + int inner_nhoff, int hlen, int num_hdrs); static bool flow_dissect_fast_gre_inner(const struct sk_buff *skb, struct flow_dissector *flow_dissector, void *target_container, const void *data, - int nhoff, int hlen); + int nhoff, int hlen, int num_hdrs); =20 /* One of the two dissectors the fast-path eligibility check admits; * defined here so flow_dissect_fast() below can reference it (its keys @@ -1141,7 +1144,7 @@ static bool flow_dissect_fast_ipv4(const struct sk_bu= ff *skb, struct flow_dissector *flow_dissector, void *target_container, const void *data, - int nhoff, int hlen) + int nhoff, int hlen, int num_hdrs) { struct flow_dissector_key_control *key_control; struct flow_dissector_key_addrs *key_addrs; @@ -1172,18 +1175,18 @@ static bool flow_dissect_fast_ipv4(const struct sk_= buff *skb, return flow_dissect_fast_ipip_inner(skb, flow_dissector, target_container, data, htons(ETH_P_IP), - nhoff + (int)sizeof(*iph), hlen); + nhoff + (int)sizeof(*iph), hlen, num_hdrs); if (iph->protocol =3D=3D IPPROTO_IPV6) return flow_dissect_fast_ipip_inner(skb, flow_dissector, target_container, data, htons(ETH_P_IPV6), - nhoff + (int)sizeof(*iph), hlen); + nhoff + (int)sizeof(*iph), hlen, num_hdrs); } if (static_branch_unlikely(&flow_dissector_gre_key) && iph->protocol =3D=3D IPPROTO_GRE) return flow_dissect_fast_gre_inner(skb, flow_dissector, target_container, data, - nhoff + (int)sizeof(*iph), hlen); + nhoff + (int)sizeof(*iph), hlen, num_hdrs); return false; } =20 @@ -1241,7 +1244,7 @@ static bool flow_dissect_fast_ipv6(const struct sk_bu= ff *skb, struct flow_dissector *flow_dissector, void *target_container, const void *data, - int nhoff, int hlen) + int nhoff, int hlen, int num_hdrs) { struct flow_dissector_key_control *key_control; struct flow_dissector_key_addrs *key_addrs; @@ -1317,10 +1320,10 @@ static bool flow_dissect_fast_ipv6(const struct sk_= buff *skb, data, iph->nexthdr =3D=3D IPPROTO_IPIP ? htons(ETH_P_IP) : htons(ETH_P_IPV6), - nhoff + (int)sizeof(*iph), hlen); + nhoff + (int)sizeof(*iph), hlen, num_hdrs); return flow_dissect_fast_gre_inner(skb, flow_dissector, target_container, data, - nhoff + (int)sizeof(*iph), hlen); + nhoff + (int)sizeof(*iph), hlen, num_hdrs); } =20 thoff =3D nhoff + (int)sizeof(*iph); @@ -1439,12 +1442,12 @@ static bool flow_dissect_fast_vlan(const struct sk_= buff *skb, case htons(ETH_P_IP): ok =3D flow_dissect_fast_ipv4(skb, flow_dissector, target_container, data, - nhoff, hlen); + nhoff, hlen, vlan_depth + 2); break; case htons(ETH_P_IPV6): ok =3D flow_dissect_fast_ipv6(skb, flow_dissector, target_container, data, - nhoff, hlen); + nhoff, hlen, vlan_depth + 2); break; case htons(ETH_P_8021Q): case htons(ETH_P_8021AD): @@ -1524,11 +1527,11 @@ static bool flow_dissect_fast_pppoe(const struct sk= _buff *skb, if (inner_eth_proto =3D=3D htons(ETH_P_IP)) ok =3D flow_dissect_fast_ipv4(skb, flow_dissector, target_container, data, - nhoff, hlen); + nhoff, hlen, 2); else ok =3D flow_dissect_fast_ipv6(skb, flow_dissector, target_container, data, - nhoff, hlen); + nhoff, hlen, 2); =20 if (ok) flow_dissector_count_fast(FLOW_DISSECTOR_SHAPE_PPPOE); @@ -1612,19 +1615,25 @@ static bool flow_dissect_fast_ipip_inner(const stru= ct sk_buff *skb, void *target_container, const void *data, __be16 inner_eth_proto, - int inner_nhoff, int hlen) + int inner_nhoff, int hlen, int num_hdrs) { struct flow_dissector_key_control *key_control; bool ok; =20 + /* Mirror the slow path's MAX_FLOW_DISSECT_HDRS budget; past the cap + * defer, so both paths stop at the same depth. + */ + if (++num_hdrs > MAX_FLOW_DISSECT_HDRS) + return false; + if (inner_eth_proto =3D=3D htons(ETH_P_IP)) ok =3D flow_dissect_fast_ipv4(skb, flow_dissector, target_container, data, - inner_nhoff, hlen); + inner_nhoff, hlen, num_hdrs); else if (inner_eth_proto =3D=3D htons(ETH_P_IPV6)) ok =3D flow_dissect_fast_ipv6(skb, flow_dissector, target_container, data, - inner_nhoff, hlen); + inner_nhoff, hlen, num_hdrs); else return false; =20 @@ -1651,7 +1660,7 @@ static bool flow_dissect_fast_gre_inner(const struct = sk_buff *skb, struct flow_dissector *flow_dissector, void *target_container, const void *data, - int nhoff, int hlen) + int nhoff, int hlen, int num_hdrs) { struct flow_dissector_key_control *key_control; const struct gre_base_hdr *hdr; @@ -1659,6 +1668,10 @@ static bool flow_dissect_fast_gre_inner(const struct= sk_buff *skb, int inner_nhoff; bool ok; =20 + /* Same MAX_FLOW_DISSECT_HDRS budget as flow_dissect_fast_ipip_inner(). */ + if (++num_hdrs > MAX_FLOW_DISSECT_HDRS) + return false; + if (unlikely(hlen - nhoff < (int)sizeof(*hdr))) return false; hdr =3D (const struct gre_base_hdr *)((const u8 *)data + nhoff); @@ -1686,11 +1699,11 @@ static bool flow_dissect_fast_gre_inner(const struc= t sk_buff *skb, if (inner_proto =3D=3D htons(ETH_P_IP)) ok =3D flow_dissect_fast_ipv4(skb, flow_dissector, target_container, data, - inner_nhoff, hlen); + inner_nhoff, hlen, num_hdrs); else ok =3D flow_dissect_fast_ipv6(skb, flow_dissector, target_container, data, - inner_nhoff, hlen); + inner_nhoff, hlen, num_hdrs); =20 if (!ok) return false; @@ -1764,7 +1777,7 @@ static bool flow_dissect_fast(const struct sk_buff *s= kb, return false; if (!flow_dissect_fast_ipv4(skb, flow_dissector, target_container, data, - nhoff, hlen)) + nhoff, hlen, 1)) return false; /* Plain top-level eth+IP only; descents count under their own shape. */ if (!flow_dissect_fast_is_encap(flow_dissector, target_container)) @@ -1775,7 +1788,7 @@ static bool flow_dissect_fast(const struct sk_buff *s= kb, return false; if (!flow_dissect_fast_ipv6(skb, flow_dissector, target_container, data, - nhoff, hlen)) + nhoff, hlen, 1)) return false; if (!flow_dissect_fast_is_encap(flow_dissector, target_container)) flow_dissector_count_fast(FLOW_DISSECTOR_SHAPE_ETH_IP); --=20 2.54.0 From nobody Sat Jul 25 16:44:51 2026 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BEB9D30E0CC for ; Thu, 16 Jul 2026 00:44:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162659; cv=none; b=GCDGLgJbUxILgbRgwFnTjk1d2m9Fmg2NVnBA4D/0zAn2pMNEydNmNGfZGr3KwK9cGDt5neCqe8l69E27SMyEmIC6F+EddJEHxHoQfkpOBb+uGE5M9R2QzZpjyZsUU+GqwR7PTs1rEi3/dDNPXzjThjtB7eGjlXV45b+b42vL0Mg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162659; c=relaxed/simple; bh=Lk+rcAPSYPstjCw+/+3SnzZiKF0I9iRCWmm4lkjNdJo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=t4F87yGHLxGXothF7I8z2ciOzDKOiOTHUJWBlpLWgWgZC6Vcth/Ei++4kBsAVUiOA1IeBKksj3N67vsWkgqBzXVAqQvtVQrNT+ou8ItQYbPjExs1zVNu2CDv1XScXwkftFKkutsAJffnsg6aCa5EH74jk8ExZD41BB7GXUyoJ38= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Q1tmZwQo; arc=none smtp.client-ip=209.85.216.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Q1tmZwQo" Received: by mail-pj1-f44.google.com with SMTP id 98e67ed59e1d1-38a0c7e841fso2957785a91.2 for ; Wed, 15 Jul 2026 17:44:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784162653; x=1784767453; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=B3K0iafsb2eEJ5i3nuA3zhTiG2cSIaYBe4GzOeaGblY=; b=Q1tmZwQoccUChoWNDbmq0hUpAsdciJz+02SSTiXCaWjgsGdstqFQPzKfN0OoQfdgN4 kV/PcIdIQM1A/3SYfDRXPELbXLfH9VVjy5j+HZe6nKhG2UPCM0hqX5G6i5DRNd6CMw8H 6OCTkVd+niESu/PNLtZeUVQLz55q+zKZNiIcOf768sZINujL9xnLeHOZQVYAH7t4M3ky jBV/ATra6i6puRSNaSai/aNu3nncuPj9NEcsqsteFx0daVpOeeBcgfVoiYVX+JIb53zF CU+tcrP2QgbjswScCC5VBpiOyBcgUA4AEd1Undk4Xrm1NUiMjWjycmKaugjKts1485yr DyqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784162653; x=1784767453; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=B3K0iafsb2eEJ5i3nuA3zhTiG2cSIaYBe4GzOeaGblY=; b=paBpD4DEOHyi82I+H1H2/yf/3/tEPWac6Pj/EUEO7k73KWzBuE3Pjt7lreMYqybVaD 1oWqa3PN6OLBemlgD2EuWfq5MwPlFHGjAb3Q017uAbz53Nt/DfVyo4onE8HMqaQpAYep mysgDAMgWC70J8XaiWmcLjXu6VQ/AZN4XYN19kUd0pIBKna7DamVwE8XqRW44SdKD0f8 DjGAMcs52JXLPaQ9ESNhr7hrS0CDidB3HpHpuhMR2A2BBmYM2EH/fpZTwYIUyrfizyiz 6weXenD+vDrlqIjbbLXyAwvKCYE+fbs6B/Utvd4V1T25Vx2EUJ8c9LeoK4KPctHQqa9e /W3Q== X-Forwarded-Encrypted: i=1; AHgh+RqRmVvjD4WMswt3XYIvfbqkqI7xboZ3SFE0KcCSs/TeKY/GkzJ5ardHG1+NO7T/fHzaJrStiPhm/F0Ean8=@vger.kernel.org X-Gm-Message-State: AOJu0YxuoZmso1JmImjbIw4YWiHHLeYjpkutqqOwpHGxBzH+vWkxfU4r Hc++yioymBgQIilsK74/EV+MfmZvCbWNNMZUtk4iBixEW2Hx+hfWjilN X-Gm-Gg: AfdE7cnMkr2EAQFE6o5DYsFyxm/H1oGKmEuJK5izNKYCD/PG0x6ik8B2TsnP7ijAgRz 7eUuHWownli4VWDfT817BH4RvxxX1PYmZk27dM4gwF195AqdwisNzPvGtJu1JvjMrBPLQYXtoMt TxsNMVEIY2TZPTceZd0AL6rj+i0Tm3b9JEpEthtHCu7ivSeuLYnL9XVV3gcCT1H1TPu3WuOnYcF ycnACAQRaafxYKb/kJwHpwbEa+d5eOmmEMPrNAAFKu+a0Hdh1cvR3yc0my3Z9HlbFVHfmhW1Jbc ibQm2TUq0oG2NPWumWJKWgH7tuKbydOdrWGPshFatw+y1sgGRU5U2tqzcxygVE1OnfGUVVXS+zW qNclROaDl/Vqpz2QEQAoO3x+pwrNutFUYAgi4AS8Q9h1KCMzbh11okS6zdU2sA6Pz7NKgHK6wt9 BVgBKfbTONFHzxrn4kdo0yeJO+FZQyl8tR5NjOVA9M8ZSLwdQA+q+IsD1NroPK9eOMnzRUPSC7p S05SS2cjWiFJROJ/aKT X-Received: by 2002:a05:6300:2288:b0:3c0:e6c3:224a with SMTP id adf61e73a8af0-3c110775293mr20589898637.37.1784162652913; Wed, 15 Jul 2026 17:44:12 -0700 (PDT) Received: from l.localdomain ([76.166.220.66]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-13cd4284a92sm4107620c88.4.2026.07.15.17.44.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 17:44:12 -0700 (PDT) From: Dave Seddon To: netdev@vger.kernel.org Cc: "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Stanislav Fomichev , Tom Herbert , Willem de Bruijn , linux-kernel@vger.kernel.org, Dave Seddon Subject: [PATCH net-next v1 10/11] net: flow_dissector: add KUnit fast/slow path equivalence tests Date: Wed, 15 Jul 2026 17:43:56 -0700 Message-ID: <20260716004357.3652679-11-dave.seddon.ca@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> References: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The per-shape fast paths contract to be byte-identical with the slow path: for any packet, either struct flow_keys is filled with exactly the bytes the slow path would have written, or the fast path returns false and the slow path runs. Until now that contract was only verified out of tree (a userspace A/B harness compiling the dissector both ways). Enforce it in-tree so any future divergence between the two implementations is a failing test, not a silent behaviour change. Each corpus packet is dissected twice into zeroed containers - once with every fast-path gate disabled and once with every gate enabled - and the return values and full struct flow_keys must match exactly. (A fast-path miss cannot leak partial writes to callers: flow_dissect_fast() is only invoked from __skb_flow_dissect(), which falls through to the full slow-path walk on the same container whenever the fast path returns false.) The corpus covers the eligible shapes (eth+IPv4/IPv6 x TCP/UDP, VLAN, QinQ, PPPoE session, single MPLS, IPIP/4in6/6in4, plain GRE), deliberate fast-path misses (IP options, fragments with and without FLOW_DISSECTOR_F_PARSE_1ST_FRAG, IPv6 extension headers and non-zero flow labels, triple VLAN, multi-label MPLS, non-IP PPP, flagged GRE, ICMP), a truncation sweep cutting every packet at every byte boundary, and skb-mode cases for the hardware-stripped VLAN tag entry conditions raw-data mode cannot reach. Every equivalence case runs both with and without FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL -- the flag skb_get_hash() passes on every dissect -- against both eligible dissectors, and the gates-off test's positive control asserts the fast path actually hits for that flagship flag shape (it caught the dispatcher rejecting the flag outright, which had made the fast path unreachable from RPS/RFS/fq/cake). Beyond byte equality, every check also asserts flow_hash_from_keys() equality (the hash is what consumers act on). A gates-off pass over the whole corpus proves the fast path never runs when disabled (observed via the per-shape counters), and a seeded, reproducible fuzzer mutates corpus packets and re-checks equivalence every iteration, as a guard for divergences the hand-written corpus does not name. The file-static symmetric dissector and the summed fast-hit counters are exposed to the suite through accessors compiled only under CONFIG_FLOW_DISSECTOR_KUNIT_TEST. Run with: ./tools/testing/kunit/kunit.py run --arch=3Dx86_64 \ --kconfig_add CONFIG_NET=3Dy \ --kconfig_add CONFIG_FLOW_DISSECTOR_KUNIT_TEST=3Dy \ flow_dissector_fastpath Assisted-by: Claude:claude-fable-5 sparse smatch Signed-off-by: Dave Seddon --- include/net/flow_dissector.h | 6 + net/Kconfig | 12 + net/core/Makefile | 1 + net/core/flow_dissector.c | 30 + net/core/flow_dissector_test.c | 1053 ++++++++++++++++++++++++++++++++ 5 files changed, 1102 insertions(+) create mode 100644 net/core/flow_dissector_test.c diff --git a/include/net/flow_dissector.h b/include/net/flow_dissector.h index 53b16f7306c7..b92aef8039f8 100644 --- a/include/net/flow_dissector.h +++ b/include/net/flow_dissector.h @@ -453,6 +453,12 @@ enum flow_dissector_shape { FLOW_DISSECTOR_SHAPE__MAX, }; =20 +#if IS_ENABLED(CONFIG_FLOW_DISSECTOR_KUNIT_TEST) +/* Test-only accessors, defined in flow_dissector.c. */ +struct flow_dissector *flow_keys_dissector_symmetric_kunit(void); +u64 flow_dissector_fast_hits_kunit(void); +#endif + /* struct flow_keys_digest: * * This structure is used to hold a digest of the full flow keys. This is a diff --git a/net/Kconfig b/net/Kconfig index e38477393551..d0ce8fc34429 100644 --- a/net/Kconfig +++ b/net/Kconfig @@ -540,4 +540,16 @@ config NET_TEST =20 If unsure, say N. =20 +config FLOW_DISSECTOR_KUNIT_TEST + tristate "KUnit tests for flow dissector fast/slow path equivalence" if != KUNIT_ALL_TESTS + depends on KUNIT + default KUNIT_ALL_TESTS + help + KUnit tests asserting the flow dissector's opt-in per-shape + fast paths produce byte-identical struct flow_keys output to + the slow path for every corpus packet, with the gates on and + off. + + If unsure, say N. + endif # if NET diff --git a/net/core/Makefile b/net/core/Makefile index b3fdcb4e355f..b27c69580d2c 100644 --- a/net/core/Makefile +++ b/net/core/Makefile @@ -46,6 +46,7 @@ obj-$(CONFIG_BPF_SYSCALL) +=3D sock_map.o obj-$(CONFIG_BPF_SYSCALL) +=3D bpf_sk_storage.o obj-$(CONFIG_OF) +=3D of_net.o obj-$(CONFIG_NET_TEST) +=3D net_test.o +obj-$(CONFIG_FLOW_DISSECTOR_KUNIT_TEST) +=3D flow_dissector_test.o obj-$(CONFIG_NET_DEVMEM) +=3D devmem.o obj-$(CONFIG_DEBUG_NET) +=3D lock_debug.o obj-$(CONFIG_FAIL_SKB_REALLOC) +=3D skb_fault_injection.o diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c index 49b256ce5c1e..aaed4473c211 100644 --- a/net/core/flow_dissector.c +++ b/net/core/flow_dissector.c @@ -2909,6 +2909,36 @@ EXPORT_SYMBOL(flow_keys_dissector); struct flow_dissector flow_keys_basic_dissector __read_mostly; EXPORT_SYMBOL(flow_keys_basic_dissector); =20 +#if IS_ENABLED(CONFIG_FLOW_DISSECTOR_KUNIT_TEST) +/* Test-only accessor: the symmetric dissector is file-static but is + * the second dissector the fast-path eligibility gate admits. + */ +struct flow_dissector *flow_keys_dissector_symmetric_kunit(void) +{ + return &flow_keys_dissector_symmetric; +} +EXPORT_SYMBOL_GPL(flow_keys_dissector_symmetric_kunit); + +/* Test-only accessor: summed fast-path hits, the observable for the + * suite's gates-off negative. + */ +u64 flow_dissector_fast_hits_kunit(void) +{ + u64 sum =3D 0; + int cpu, i; + + for_each_possible_cpu(cpu) { + const struct flow_dissector_stats *s =3D + per_cpu_ptr(&flow_dissector_pcpu_stats, cpu); + + for (i =3D 0; i < FLOW_DISSECTOR_SHAPE__MAX; i++) + sum +=3D READ_ONCE(s->fast_hits[i]); + } + return sum; +} +EXPORT_SYMBOL_GPL(flow_dissector_fast_hits_kunit); +#endif + static int __init init_default_flow_dissectors(void) { skb_flow_dissector_init(&flow_keys_dissector, diff --git a/net/core/flow_dissector_test.c b/net/core/flow_dissector_test.c new file mode 100644 index 000000000000..e0014affda7f --- /dev/null +++ b/net/core/flow_dissector_test.c @@ -0,0 +1,1053 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * KUnit equivalence tests for the flow dissector opt-in fast paths. + * + * Every fast-path shape contracts to be byte-identical with the slow + * path. Each corpus packet is dissected twice into zeroed containers + * -- gates off, then gates on -- and the return values, the full + * struct flow_keys and flow_hash_from_keys() must match exactly, for + * both eligible dissectors (flow_keys_dissector and the file-static + * symmetric one). (A fast-path miss cannot leak partial writes to + * callers: flow_dissect_fast() is only invoked from + * __skb_flow_dissect(), which continues into the full slow-path walk + * on the same container whenever the fast path returns false.) + */ + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define FD_TEST_BUF_LEN 512 +#define FD_FUZZ_ITERS 4000 + +/* The byte-identical fast-path gates. */ +static struct static_key_false * const fd_fast_gates[] =3D { + &flow_dissector_eth_ip_key, + &flow_dissector_vlan_key, + &flow_dissector_qinq_key, + &flow_dissector_pppoe_key, + &flow_dissector_mpls_key, + &flow_dissector_ipip_key, + &flow_dissector_gre_key, +}; + +static void fd_fast_gates_set(bool on) +{ + int i; + + for (i =3D 0; i < ARRAY_SIZE(fd_fast_gates); i++) { + if (on) + static_branch_enable(fd_fast_gates[i]); + else + static_branch_disable(fd_fast_gates[i]); + } +} + +/* --- packet builders (network-layer payloads; __skb_flow_dissect is + * called in raw-data mode with the L3 proto passed explicitly, so no + * Ethernet header is included) --- + */ + +static int put_ports(u8 *buf) +{ + __be16 *p =3D (__be16 *)buf; + + p[0] =3D htons(0x1234); + p[1] =3D htons(0x5678); + return 4; +} + +static int put_ipv4(u8 *buf, u8 ip_proto, u8 ihl, __be16 frag_off) +{ + struct iphdr *ip =3D (struct iphdr *)buf; + int hdrlen =3D ihl * 4; + + memset(ip, 0, hdrlen); + ip->version =3D 4; + ip->ihl =3D ihl; + ip->ttl =3D 64; + ip->tot_len =3D htons(hdrlen + 4); + ip->frag_off =3D frag_off; + ip->protocol =3D ip_proto; + ip->saddr =3D htonl(0xc0a80101); /* 192.168.1.1 */ + ip->daddr =3D htonl(0xc0a80202); /* 192.168.2.2 */ + return hdrlen; +} + +static int put_ipv6(u8 *buf, u8 nexthdr, u32 flow_label) +{ + struct ipv6hdr *ip6 =3D (struct ipv6hdr *)buf; + + memset(ip6, 0, sizeof(*ip6)); + ip6->version =3D 6; + ip6->flow_lbl[0] =3D (flow_label >> 16) & 0x0f; + ip6->flow_lbl[1] =3D (flow_label >> 8) & 0xff; + ip6->flow_lbl[2] =3D flow_label & 0xff; + ip6->payload_len =3D htons(4); + ip6->nexthdr =3D nexthdr; + ip6->hop_limit =3D 64; + ip6->saddr.s6_addr32[0] =3D htonl(0x20010db8); + ip6->saddr.s6_addr32[3] =3D htonl(1); + ip6->daddr.s6_addr32[0] =3D htonl(0x20010db8); + ip6->daddr.s6_addr32[3] =3D htonl(2); + return sizeof(*ip6); +} + +static int put_vlan(u8 *buf, u16 tci, __be16 inner_proto) +{ + struct vlan_hdr *vlan =3D (struct vlan_hdr *)buf; + + vlan->h_vlan_TCI =3D htons(tci); + vlan->h_vlan_encapsulated_proto =3D inner_proto; + return sizeof(*vlan); +} + +static int put_pppoe(u8 *buf, u16 ppp_proto, u16 payload_len) +{ + struct pppoe_hdr *hdr =3D (struct pppoe_hdr *)buf; + __be16 *proto =3D (__be16 *)(buf + sizeof(*hdr)); + + memset(hdr, 0, sizeof(*hdr)); + hdr->ver =3D 1; + hdr->type =3D 1; + hdr->code =3D 0; + hdr->sid =3D htons(0x42); + hdr->length =3D htons(payload_len + 2); + *proto =3D htons(ppp_proto); + return sizeof(*hdr) + 2; +} + +static int put_mpls(u8 *buf, u32 label, u32 bos) +{ + struct mpls_label *lse =3D (struct mpls_label *)buf; + + lse->entry =3D htonl((label << MPLS_LS_LABEL_SHIFT) | + (bos << MPLS_LS_S_SHIFT) | + (64 << MPLS_LS_TTL_SHIFT)); + return sizeof(*lse); +} + +static int put_gre(u8 *buf, __be16 flags, __be16 protocol) +{ + struct gre_base_hdr *gre =3D (struct gre_base_hdr *)buf; + + gre->flags =3D flags; + gre->protocol =3D protocol; + return sizeof(*gre); +} + +/* --- composite builders, one per corpus entry --- */ + +static int build_ipv4_tcp(u8 *buf) +{ + int n =3D put_ipv4(buf, IPPROTO_TCP, 5, 0); + + return n + put_ports(buf + n); +} + +static int build_ipv4_udp(u8 *buf) +{ + int n =3D put_ipv4(buf, IPPROTO_UDP, 5, 0); + + return n + put_ports(buf + n); +} + +static int build_ipv6_tcp(u8 *buf) +{ + int n =3D put_ipv6(buf, IPPROTO_TCP, 0); + + return n + put_ports(buf + n); +} + +static int build_ipv6_udp(u8 *buf) +{ + int n =3D put_ipv6(buf, IPPROTO_UDP, 0); + + return n + put_ports(buf + n); +} + +static int build_vlan_ipv4_tcp(u8 *buf) +{ + int n =3D put_vlan(buf, 100, htons(ETH_P_IP)); + + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +static int build_vlan_ipv6_udp(u8 *buf) +{ + int n =3D put_vlan(buf, 100, htons(ETH_P_IPV6)); + + n +=3D put_ipv6(buf + n, IPPROTO_UDP, 0); + return n + put_ports(buf + n); +} + +static int build_qinq_ipv4_tcp(u8 *buf) +{ + int n =3D put_vlan(buf, 100, htons(ETH_P_8021Q)); + + n +=3D put_vlan(buf + n, 200, htons(ETH_P_IP)); + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +static int build_vlan3_ipv4_tcp(u8 *buf) +{ + int n =3D put_vlan(buf, 100, htons(ETH_P_8021Q)); + + n +=3D put_vlan(buf + n, 200, htons(ETH_P_8021Q)); + n +=3D put_vlan(buf + n, 300, htons(ETH_P_IP)); + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +static int build_pppoe_ipv4_tcp(u8 *buf) +{ + int n =3D put_pppoe(buf, PPP_IP, 20 + 4); + + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +static int build_pppoe_ipv6_tcp(u8 *buf) +{ + int n =3D put_pppoe(buf, PPP_IPV6, 40 + 4); + + n +=3D put_ipv6(buf + n, IPPROTO_TCP, 0); + return n + put_ports(buf + n); +} + +static int build_pppoe_lcp(u8 *buf) +{ + int n =3D put_pppoe(buf, PPP_LCP, 4); + u8 lcp[4] =3D { 0x01, 0x01, 0x00, 0x04 }; + + memcpy(buf + n, lcp, sizeof(lcp)); + return n + sizeof(lcp); +} + +static int build_mpls_ipv4(u8 *buf) +{ + int n =3D put_mpls(buf, 1000, 1); + + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +static int build_mpls_multi(u8 *buf) +{ + int n =3D put_mpls(buf, 1000, 0); /* S=3D0: not bottom of stack */ + + n +=3D put_mpls(buf + n, 2000, 1); + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +static int build_ipip(u8 *buf) +{ + int n =3D put_ipv4(buf, IPPROTO_IPIP, 5, 0); + + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +static int build_6in4(u8 *buf) +{ + int n =3D put_ipv4(buf, IPPROTO_IPV6, 5, 0); + + n +=3D put_ipv6(buf + n, IPPROTO_TCP, 0); + return n + put_ports(buf + n); +} + +static int build_4in6(u8 *buf) +{ + int n =3D put_ipv6(buf, IPPROTO_IPIP, 0); + + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +static int build_4in6_flowlabel(u8 *buf) +{ + int n =3D put_ipv6(buf, IPPROTO_IPIP, 0xbeef); + + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +static int build_6in6(u8 *buf) +{ + int n =3D put_ipv6(buf, IPPROTO_IPV6, 0); + + n +=3D put_ipv6(buf + n, IPPROTO_TCP, 0); + return n + put_ports(buf + n); +} + +static int build_gre_ipv4(u8 *buf) +{ + int n =3D put_ipv4(buf, IPPROTO_GRE, 5, 0); + + n +=3D put_gre(buf + n, 0, htons(ETH_P_IP)); + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +static int build_gre_ipv6(u8 *buf) +{ + int n =3D put_ipv4(buf, IPPROTO_GRE, 5, 0); + + n +=3D put_gre(buf + n, 0, htons(ETH_P_IPV6)); + n +=3D put_ipv6(buf + n, IPPROTO_TCP, 0); + return n + put_ports(buf + n); +} + +static int build_gre6_ipv6(u8 *buf) +{ + int n =3D put_ipv6(buf, IPPROTO_GRE, 0); + + n +=3D put_gre(buf + n, 0, htons(ETH_P_IPV6)); + n +=3D put_ipv6(buf + n, IPPROTO_TCP, 0); + return n + put_ports(buf + n); +} + +static int build_gre6_ipv4(u8 *buf) +{ + int n =3D put_ipv6(buf, IPPROTO_GRE, 0); + + n +=3D put_gre(buf + n, 0, htons(ETH_P_IP)); + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +/* --- corner-case builders --- */ + +/* GRE variants that must defer (any non-zero flags/version word). */ +static int build_gre_seq_ipv4(u8 *buf) +{ + int n =3D put_ipv4(buf, IPPROTO_GRE, 5, 0); + + n +=3D put_gre(buf + n, GRE_SEQ, htons(ETH_P_IP)); + memset(buf + n, 0, 4); /* sequence number */ + n +=3D 4; + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +static int build_gre_key_ipv4(u8 *buf) +{ + int n =3D put_ipv4(buf, IPPROTO_GRE, 5, 0); + + n +=3D put_gre(buf + n, GRE_KEY, htons(ETH_P_IP)); + memset(buf + n, 0, 4); /* key */ + n +=3D 4; + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +static int build_gre_version1(u8 *buf) +{ + int n =3D put_ipv4(buf, IPPROTO_GRE, 5, 0); + + /* GRE version 1 (PPTP) lives in the low bits of the flags word. */ + n +=3D put_gre(buf + n, htons(0x0001), htons(ETH_P_IP)); + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +static int build_gre_teb(u8 *buf) +{ + int n =3D put_ipv4(buf, IPPROTO_GRE, 5, 0); + + /* Transparent Ethernet bridging inner: fast path defers. */ + n +=3D put_gre(buf + n, 0, htons(ETH_P_TEB)); + return n + put_ipv4(buf + n, IPPROTO_TCP, 5, 0); +} + +/* An IPv6 extension header (8 bytes, hdrlen field 0). */ +static int put_ext_hdr(u8 *buf, u8 next_proto) +{ + memset(buf, 0, 8); + buf[0] =3D next_proto; + buf[1] =3D 0; /* hdr ext len: (0 + 1) * 8 =3D 8 bytes */ + return 8; +} + +static int build_ipv6_hopopts_dest_tcp(u8 *buf) +{ + int n =3D put_ipv6(buf, IPPROTO_HOPOPTS, 0); + + n +=3D put_ext_hdr(buf + n, IPPROTO_DSTOPTS); + n +=3D put_ext_hdr(buf + n, IPPROTO_TCP); + return n + put_ports(buf + n); +} + +static int build_ipv6_frag_tcp(u8 *buf) +{ + int n =3D put_ipv6(buf, IPPROTO_FRAGMENT, 0); + + memset(buf + n, 0, 8); + buf[n] =3D IPPROTO_TCP; /* fragment header, offset 0 */ + *(__be32 *)(buf + n + 4) =3D htonl(0x12345678); + n +=3D 8; + return n + put_ports(buf + n); +} + +/* TPID ordering not covered elsewhere: 8021Q outer, 8021AD inner. */ +static int build_vlan_q_then_ad_ipv4(u8 *buf) +{ + int n =3D put_vlan(buf, 100, htons(ETH_P_8021AD)); + + n +=3D put_vlan(buf + n, 200, htons(ETH_P_IP)); + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +/* IPv4 with a larger options block (IHL 8 =3D> 12 bytes of options). */ +static int build_ipv4_options12_tcp(u8 *buf) +{ + int n =3D put_ipv4(buf, IPPROTO_TCP, 8, 0); + + return n + put_ports(buf + n); +} + +/* PPPoE with a PFC-compressed (odd MSB) PPP protocol field: defers. */ +static int build_pppoe_pfc(u8 *buf) +{ + int n =3D put_pppoe(buf, 0x2100, 20 + 4); + + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +/* Mid-stream fragment: non-zero offset AND MF still set. */ +static int build_ipv4_frag_mid_tcp(u8 *buf) +{ + int n =3D put_ipv4(buf, IPPROTO_TCP, 5, htons(IP_MF | 5)); + + return n + put_ports(buf + n); +} + +/* GRE optional-field combinations (order: csum+reserved, key, seq). */ +static int build_gre_csum_key_ipv4(u8 *buf) +{ + int n =3D put_ipv4(buf, IPPROTO_GRE, 5, 0); + + n +=3D put_gre(buf + n, GRE_CSUM | GRE_KEY, htons(ETH_P_IP)); + memset(buf + n, 0, 8); /* checksum + reserved, key */ + n +=3D 8; + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +static int build_gre_csum_key_seq_ipv4(u8 *buf) +{ + int n =3D put_ipv4(buf, IPPROTO_GRE, 5, 0); + + n +=3D put_gre(buf + n, GRE_CSUM | GRE_KEY | GRE_SEQ, htons(ETH_P_IP)); + memset(buf + n, 0, 12); /* checksum + reserved, key, seq */ + n +=3D 12; + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +/* Priority-tagged frame (802.1Q VID 0, only PCP bits in the TCI). */ +static int build_vlan_prio_ipv4_tcp(u8 *buf) +{ + int n =3D put_vlan(buf, 5 << VLAN_PRIO_SHIFT, htons(ETH_P_IP)); + + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +/* IPv6 routing extension header before TCP. */ +static int build_ipv6_routing_tcp(u8 *buf) +{ + int n =3D put_ipv6(buf, IPPROTO_ROUTING, 0); + + n +=3D put_ext_hdr(buf + n, IPPROTO_TCP); + return n + put_ports(buf + n); +} + +/* Deeply nested IPIP / GRE chains for the recursion cap. */ +static int build_ipip_nest(u8 *buf, int levels) +{ + int n =3D 0, i; + + for (i =3D 0; i < levels; i++) + n +=3D put_ipv4(buf + n, IPPROTO_IPIP, 5, 0); + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +static int build_gre_nest(u8 *buf, int levels) +{ + int n =3D 0, i; + + for (i =3D 0; i < levels; i++) { + n +=3D put_ipv4(buf + n, IPPROTO_GRE, 5, 0); + n +=3D put_gre(buf + n, 0, htons(ETH_P_IP)); + } + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +static int build_gre_csum_ipv4(u8 *buf) +{ + int n =3D put_ipv4(buf, IPPROTO_GRE, 5, 0); + + n +=3D put_gre(buf + n, GRE_CSUM, htons(ETH_P_IP)); + memset(buf + n, 0, 4); /* checksum + reserved */ + n +=3D 4; + n +=3D put_ipv4(buf + n, IPPROTO_TCP, 5, 0); + return n + put_ports(buf + n); +} + +static int build_ipv4_options_tcp(u8 *buf) +{ + int n =3D put_ipv4(buf, IPPROTO_TCP, 6, 0); /* IHL=3D6: 4B options */ + + return n + put_ports(buf + n); +} + +static int build_ipv4_frag_tcp(u8 *buf) +{ + int n =3D put_ipv4(buf, IPPROTO_TCP, 5, htons(IP_MF)); + + return n + put_ports(buf + n); +} + +static int build_ipv4_frag_offset_tcp(u8 *buf) +{ + int n =3D put_ipv4(buf, IPPROTO_TCP, 5, htons(0x0010)); + + return n + put_ports(buf + n); +} + +static int build_ipv4_icmp(u8 *buf) +{ + int n =3D put_ipv4(buf, IPPROTO_ICMP, 5, 0); + u8 icmp[8] =3D { 0x08, 0x00, 0x00, 0x00, 0x12, 0x34, 0x00, 0x01 }; + + memcpy(buf + n, icmp, sizeof(icmp)); + return n + sizeof(icmp); +} + +static int build_ipv6_hopopts_tcp(u8 *buf) +{ + int n =3D put_ipv6(buf, IPPROTO_HOPOPTS, 0); + u8 hbh[8] =3D { IPPROTO_TCP, 0, 0, 0, 0, 0, 0, 0 }; + + memcpy(buf + n, hbh, sizeof(hbh)); + n +=3D sizeof(hbh); + return n + put_ports(buf + n); +} + +static int build_ipv6_flowlabel_tcp(u8 *buf) +{ + int n =3D put_ipv6(buf, IPPROTO_TCP, 0xbeef); + + return n + put_ports(buf + n); +} + +struct fd_fast_case { + const char *name; + int (*build)(u8 *buf); + __be16 proto; + unsigned int flags; +}; + +/* htons() is not constant-foldable here without cpu_to_be16 literals; + * use cpu_to_be16 for static initializers. + */ +static const struct fd_fast_case fd_fast_cases[] =3D { + /* eligible shapes: the fast path should take these */ + { "ipv4_tcp", build_ipv4_tcp, cpu_to_be16(ETH_P_IP), 0 }, + { "ipv4_udp", build_ipv4_udp, cpu_to_be16(ETH_P_IP), 0 }, + { "ipv6_tcp", build_ipv6_tcp, cpu_to_be16(ETH_P_IPV6), 0 }, + { "ipv6_udp", build_ipv6_udp, cpu_to_be16(ETH_P_IPV6), 0 }, + { "vlan_ipv4_tcp", build_vlan_ipv4_tcp, cpu_to_be16(ETH_P_8021Q), 0 }, + { "vlan_ipv6_udp", build_vlan_ipv6_udp, cpu_to_be16(ETH_P_8021Q), 0 }, + { "qinq_ipv4_tcp", build_qinq_ipv4_tcp, cpu_to_be16(ETH_P_8021AD), 0 }, + { "pppoe_ipv4_tcp", build_pppoe_ipv4_tcp, cpu_to_be16(ETH_P_PPP_SES), 0 }, + { "pppoe_ipv6_tcp", build_pppoe_ipv6_tcp, cpu_to_be16(ETH_P_PPP_SES), 0 }, + { "mpls_ipv4", build_mpls_ipv4, cpu_to_be16(ETH_P_MPLS_UC), 0 }, + { "ipip", build_ipip, cpu_to_be16(ETH_P_IP), 0 }, + { "6in4", build_6in4, cpu_to_be16(ETH_P_IP), 0 }, + { "4in6", build_4in6, cpu_to_be16(ETH_P_IPV6), 0 }, + { "4in6_flowlabel", build_4in6_flowlabel, cpu_to_be16(ETH_P_IPV6), 0 }, + { "6in6", build_6in6, cpu_to_be16(ETH_P_IPV6), 0 }, + { "gre_ipv4", build_gre_ipv4, cpu_to_be16(ETH_P_IP), 0 }, + { "gre6_ipv4", build_gre6_ipv4, cpu_to_be16(ETH_P_IPV6), 0 }, + { "gre_ipv6", build_gre_ipv6, cpu_to_be16(ETH_P_IP), 0 }, + { "gre6_ipv6", build_gre6_ipv6, cpu_to_be16(ETH_P_IPV6), 0 }, + /* deliberate fast-path misses: must fall back with identical + * output (and, per the zeroed containers, no partial writes) + */ + { "ipv4_options_tcp", build_ipv4_options_tcp, cpu_to_be16(ETH_P_IP), 0 }, + { "ipv4_frag_mf", build_ipv4_frag_tcp, cpu_to_be16(ETH_P_IP), 0 }, + { "ipv4_frag_mf_1stfrag", build_ipv4_frag_tcp, cpu_to_be16(ETH_P_IP), + FLOW_DISSECTOR_F_PARSE_1ST_FRAG }, + { "ipv4_frag_offset", build_ipv4_frag_offset_tcp, + cpu_to_be16(ETH_P_IP), 0 }, + { "ipv4_icmp", build_ipv4_icmp, cpu_to_be16(ETH_P_IP), 0 }, + { "ipv6_hopopts_tcp", build_ipv6_hopopts_tcp, + cpu_to_be16(ETH_P_IPV6), 0 }, + { "ipv6_flowlabel_tcp", build_ipv6_flowlabel_tcp, + cpu_to_be16(ETH_P_IPV6), 0 }, + { "vlan3_ipv4_tcp", build_vlan3_ipv4_tcp, cpu_to_be16(ETH_P_8021Q), 0 }, + { "pppoe_lcp", build_pppoe_lcp, cpu_to_be16(ETH_P_PPP_SES), 0 }, + { "mpls_multi_label", build_mpls_multi, cpu_to_be16(ETH_P_MPLS_UC), 0 }, + { "gre_csum_ipv4", build_gre_csum_ipv4, cpu_to_be16(ETH_P_IP), 0 }, + /* corner cases */ + { "gre_seq_ipv4", build_gre_seq_ipv4, cpu_to_be16(ETH_P_IP), 0 }, + { "gre_key_ipv4", build_gre_key_ipv4, cpu_to_be16(ETH_P_IP), 0 }, + { "gre_version1", build_gre_version1, cpu_to_be16(ETH_P_IP), 0 }, + { "gre_teb", build_gre_teb, cpu_to_be16(ETH_P_IP), 0 }, + { "ipv6_hopopts_dest_tcp", build_ipv6_hopopts_dest_tcp, + cpu_to_be16(ETH_P_IPV6), 0 }, + { "ipv6_frag_tcp", build_ipv6_frag_tcp, cpu_to_be16(ETH_P_IPV6), 0 }, + { "vlan_q_then_ad_ipv4", build_vlan_q_then_ad_ipv4, + cpu_to_be16(ETH_P_8021Q), 0 }, + { "ipv4_options12_tcp", build_ipv4_options12_tcp, + cpu_to_be16(ETH_P_IP), 0 }, + { "pppoe_pfc", build_pppoe_pfc, cpu_to_be16(ETH_P_PPP_SES), 0 }, + { "ipv4_frag_mid", build_ipv4_frag_mid_tcp, cpu_to_be16(ETH_P_IP), 0 }, + { "ipv4_frag_mid_1stfrag", build_ipv4_frag_mid_tcp, + cpu_to_be16(ETH_P_IP), FLOW_DISSECTOR_F_PARSE_1ST_FRAG }, + { "gre_csum_key_ipv4", build_gre_csum_key_ipv4, + cpu_to_be16(ETH_P_IP), 0 }, + { "gre_csum_key_seq_ipv4", build_gre_csum_key_seq_ipv4, + cpu_to_be16(ETH_P_IP), 0 }, + { "vlan_prio_ipv4_tcp", build_vlan_prio_ipv4_tcp, + cpu_to_be16(ETH_P_8021Q), 0 }, + { "ipv6_routing_tcp", build_ipv6_routing_tcp, + cpu_to_be16(ETH_P_IPV6), 0 }, +}; + +static void fd_fast_case_to_desc(const struct fd_fast_case *c, char *desc) +{ + strscpy(desc, c->name, KUNIT_PARAM_DESC_SIZE); +} + +KUNIT_ARRAY_PARAM(fd_fast, fd_fast_cases, fd_fast_case_to_desc); + +/* Dissect with gates off then on; returns and output must match. */ +static void fd_check_one(struct kunit *test, struct flow_dissector *fd, + const u8 *data, __be16 proto, int hlen, + unsigned int flags) +{ + struct flow_keys keys_slow, keys_fast; + bool ret_slow, ret_fast; + + fd_fast_gates_set(false); + memset(&keys_slow, 0, sizeof(keys_slow)); + ret_slow =3D __skb_flow_dissect(&init_net, NULL, fd, &keys_slow, + data, proto, 0, hlen, flags); + + fd_fast_gates_set(true); + memset(&keys_fast, 0, sizeof(keys_fast)); + ret_fast =3D __skb_flow_dissect(&init_net, NULL, fd, &keys_fast, + data, proto, 0, hlen, flags); + fd_fast_gates_set(false); + + KUNIT_EXPECT_EQ(test, ret_slow, ret_fast); + KUNIT_EXPECT_MEMEQ(test, &keys_slow, &keys_fast, sizeof(keys_slow)); + /* flow_keys -> skb->hash is what consumers see; assert it too. */ + KUNIT_EXPECT_EQ(test, flow_hash_from_keys(&keys_slow), + flow_hash_from_keys(&keys_fast)); +} + +/* Check against both eligible dissectors (different used_keys). */ +static void fd_fast_check_equiv(struct kunit *test, const u8 *data, + __be16 proto, int hlen, unsigned int flags) +{ + /* skb_get_hash() passes STOP_AT_FLOW_LABEL on every dissect; each + * case must hold with and without it. + */ + unsigned int fl =3D flags | FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL; + + fd_check_one(test, &flow_keys_dissector, data, proto, hlen, flags); + fd_check_one(test, flow_keys_dissector_symmetric_kunit(), + data, proto, hlen, flags); + fd_check_one(test, &flow_keys_dissector, data, proto, hlen, fl); + fd_check_one(test, flow_keys_dissector_symmetric_kunit(), + data, proto, hlen, fl); +} + +static void fd_fast_equiv_test(struct kunit *test) +{ + const struct fd_fast_case *c =3D test->param_value; + u8 *buf; + int len; + + buf =3D kunit_kzalloc(test, FD_TEST_BUF_LEN, GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf); + + len =3D c->build(buf); + KUNIT_ASSERT_LE(test, len, FD_TEST_BUF_LEN); + + fd_fast_check_equiv(test, buf, c->proto, len, c->flags); +} + +/* Truncation sweep: the fast path must never parse (or partially + * write) anything the slow path rejected at the same length. + */ +static void fd_fast_truncation_test(struct kunit *test) +{ + int i, len, hlen; + u8 *buf; + + buf =3D kunit_kzalloc(test, FD_TEST_BUF_LEN, GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf); + + for (i =3D 0; i < ARRAY_SIZE(fd_fast_cases); i++) { + const struct fd_fast_case *c =3D &fd_fast_cases[i]; + + len =3D c->build(buf); + for (hlen =3D len - 1; hlen >=3D 0; hlen--) + fd_fast_check_equiv(test, buf, c->proto, hlen, + c->flags); + } +} + +/* skb-mode builder (for hw-accel VLAN entry conditions). */ +static struct sk_buff *fd_fast_build_skb(struct kunit *test, + int (*build)(u8 *buf), + __be16 protocol) +{ + struct sk_buff *skb; + int len; + u8 *buf; + + buf =3D kunit_kzalloc(test, FD_TEST_BUF_LEN, GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf); + len =3D build(buf); + + skb =3D alloc_skb(FD_TEST_BUF_LEN + NET_SKB_PAD, GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, skb); + skb_reserve(skb, NET_SKB_PAD); + skb_put_data(skb, buf, len); + skb->protocol =3D protocol; + skb_reset_mac_header(skb); + skb_reset_network_header(skb); + return skb; +} + +static void fd_check_one_skb(struct kunit *test, struct flow_dissector *fd, + struct sk_buff *skb) +{ + struct flow_keys keys_slow, keys_fast; + bool ret_slow, ret_fast; + + fd_fast_gates_set(false); + memset(&keys_slow, 0, sizeof(keys_slow)); + ret_slow =3D __skb_flow_dissect(&init_net, skb, fd, + &keys_slow, NULL, 0, 0, 0, 0); + + fd_fast_gates_set(true); + memset(&keys_fast, 0, sizeof(keys_fast)); + ret_fast =3D __skb_flow_dissect(&init_net, skb, fd, + &keys_fast, NULL, 0, 0, 0, 0); + fd_fast_gates_set(false); + + KUNIT_EXPECT_EQ(test, ret_slow, ret_fast); + KUNIT_EXPECT_MEMEQ(test, &keys_slow, &keys_fast, sizeof(keys_slow)); + KUNIT_EXPECT_EQ(test, flow_hash_from_keys(&keys_slow), + flow_hash_from_keys(&keys_fast)); +} + +static void fd_fast_check_equiv_skb(struct kunit *test, struct sk_buff *sk= b) +{ + fd_check_one_skb(test, &flow_keys_dissector, skb); + fd_check_one_skb(test, flow_keys_dissector_symmetric_kunit(), skb); +} + +static void fd_fast_skb_plain_test(struct kunit *test) +{ + struct sk_buff *skb; + + skb =3D fd_fast_build_skb(test, build_ipv4_tcp, htons(ETH_P_IP)); + fd_fast_check_equiv_skb(test, skb); + kfree_skb(skb); +} + +static void fd_fast_skb_hwaccel_vlan_test(struct kunit *test) +{ + struct sk_buff *skb; + + /* Plain payload; VLAN tag in skb metadata, as after hw stripping. */ + skb =3D fd_fast_build_skb(test, build_ipv4_tcp, htons(ETH_P_IP)); + __vlan_hwaccel_put_tag(skb, htons(ETH_P_8021Q), 100); + fd_fast_check_equiv_skb(test, skb); + kfree_skb(skb); +} + +static void fd_fast_skb_hwaccel_vlan_qinq_test(struct kunit *test) +{ + struct sk_buff *skb; + + /* Outer tag hw-stripped (8021AD), second tag in the payload. */ + skb =3D fd_fast_build_skb(test, build_vlan_ipv4_tcp, htons(ETH_P_8021Q)); + __vlan_hwaccel_put_tag(skb, htons(ETH_P_8021AD), 100); + fd_fast_check_equiv_skb(test, skb); + kfree_skb(skb); +} + +/* Nested tunnels: both paths must stop at MAX_FLOW_DISSECT_HDRS -- + * shallow stacks descend, past the cap the fast path defers and + * matches the slow path's capped result. + */ +static void fd_fast_deep_nest_test(struct kunit *test) +{ + int levels, len; + u8 *buf; + + buf =3D kunit_kzalloc(test, FD_TEST_BUF_LEN, GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf); + + for (levels =3D 1; levels <=3D 24; levels++) { + len =3D build_ipip_nest(buf, levels); + if (len > FD_TEST_BUF_LEN) + break; + fd_fast_check_equiv(test, buf, htons(ETH_P_IP), len, 0); + } + for (levels =3D 1; levels <=3D 16; levels++) { + len =3D build_gre_nest(buf, levels); + if (len > FD_TEST_BUF_LEN) + break; + fd_fast_check_equiv(test, buf, htons(ETH_P_IP), len, 0); + } +} + +/* Non-linear skb: the fast path reads only the linear head and must + * defer; the slow path pulls from the frag. Output must match. + */ +static void fd_fast_nonlinear_skb_test(struct kunit *test) +{ + const int linear =3D 8; + struct sk_buff *skb; + struct page *page; + u8 *buf; + int len; + + buf =3D kunit_kzalloc(test, FD_TEST_BUF_LEN, GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf); + len =3D build_ipv4_tcp(buf); + + skb =3D alloc_skb(NET_SKB_PAD + linear, GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, skb); + skb_reserve(skb, NET_SKB_PAD); + skb_put_data(skb, buf, linear); + + page =3D alloc_page(GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, page); + memcpy(page_address(page), buf + linear, len - linear); + skb_add_rx_frag(skb, 0, page, 0, len - linear, len - linear); + + skb->protocol =3D htons(ETH_P_IP); + skb_reset_mac_header(skb); + skb_reset_network_header(skb); + + fd_fast_check_equiv_skb(test, skb); + kfree_skb(skb); +} + +/* An ineligible (custom) dissector: the fast path must never run, + * so gates-on output equals gates-off for the whole corpus. + */ +static void fd_fast_ineligible_dissector_test(struct kunit *test) +{ + static const struct flow_dissector_key keys[] =3D { + { .key_id =3D FLOW_DISSECTOR_KEY_CONTROL, + .offset =3D offsetof(struct flow_keys, control) }, + { .key_id =3D FLOW_DISSECTOR_KEY_BASIC, + .offset =3D offsetof(struct flow_keys, basic) }, + { .key_id =3D FLOW_DISSECTOR_KEY_IPV4_ADDRS, + .offset =3D offsetof(struct flow_keys, addrs.v4addrs) }, + { .key_id =3D FLOW_DISSECTOR_KEY_IPV6_ADDRS, + .offset =3D offsetof(struct flow_keys, addrs.v6addrs) }, + { .key_id =3D FLOW_DISSECTOR_KEY_PORTS, + .offset =3D offsetof(struct flow_keys, ports) }, + { .key_id =3D FLOW_DISSECTOR_KEY_ICMP, + .offset =3D offsetof(struct flow_keys, icmp) }, + }; + struct flow_dissector custom; + int i, len; + u8 *buf; + + skb_flow_dissector_init(&custom, keys, ARRAY_SIZE(keys)); + + buf =3D kunit_kzalloc(test, FD_TEST_BUF_LEN, GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf); + + for (i =3D 0; i < ARRAY_SIZE(fd_fast_cases); i++) { + const struct fd_fast_case *c =3D &fd_fast_cases[i]; + + len =3D c->build(buf); + fd_check_one(test, &custom, buf, c->proto, len, c->flags); + } +} + +/* skb-mode coverage for ipv6 / pppoe / mpls / gre. */ +static void fd_fast_skb_shapes_test(struct kunit *test) +{ + static const struct { + int (*build)(u8 *buf); + __be16 proto; + } cases[] =3D { + { build_ipv6_tcp, cpu_to_be16(ETH_P_IPV6) }, + { build_pppoe_ipv4_tcp, cpu_to_be16(ETH_P_PPP_SES) }, + { build_mpls_ipv4, cpu_to_be16(ETH_P_MPLS_UC) }, + { build_gre_ipv4, cpu_to_be16(ETH_P_IP) }, + }; + int i; + + for (i =3D 0; i < ARRAY_SIZE(cases); i++) { + struct sk_buff *skb; + + skb =3D fd_fast_build_skb(test, cases[i].build, cases[i].proto); + fd_fast_check_equiv_skb(test, skb); + kfree_skb(skb); + } +} + +/* Seeded, reproducible fuzzer: mutate corpus packets and lengths, + * assert equivalence each iteration -- the guard for divergences the + * hand-written corpus does not name. + */ +static void fd_fast_fuzz_test(struct kunit *test) +{ + struct rnd_state rnd; + u8 *buf; + int i; + + buf =3D kunit_kzalloc(test, FD_TEST_BUF_LEN, GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf); + prandom_seed_state(&rnd, 0x1a2b3c4d5e6f7788ULL); + + for (i =3D 0; i < FD_FUZZ_ITERS; i++) { + const struct fd_fast_case *c; + unsigned int flags; + int len, j, nmut; + + c =3D &fd_fast_cases[prandom_u32_state(&rnd) % + ARRAY_SIZE(fd_fast_cases)]; + len =3D c->build(buf); + + /* Mutate a handful of header bytes to random values. */ + nmut =3D prandom_u32_state(&rnd) % (len + 1); + for (j =3D 0; j < nmut; j++) + buf[prandom_u32_state(&rnd) % len] =3D + prandom_u32_state(&rnd); + + /* Random truncation and random 1st-frag flag. */ + len =3D prandom_u32_state(&rnd) % (len + 1); + flags =3D (prandom_u32_state(&rnd) & 1) ? + FLOW_DISSECTOR_F_PARSE_1ST_FRAG : 0; + + fd_fast_check_equiv(test, buf, c->proto, len, flags); + } +} + +/* Gates off: the fast path must never run (counters are the + * observable). Positive controls: an eligible shape must count with + * the gates on. + */ +static void fd_fast_gates_off_test(struct kunit *test) +{ + struct flow_keys keys; + u64 before; + int i, len; + u8 *buf; + + buf =3D kunit_kzalloc(test, FD_TEST_BUF_LEN, GFP_KERNEL); + KUNIT_ASSERT_NOT_ERR_OR_NULL(test, buf); + + fd_fast_gates_set(false); + before =3D flow_dissector_fast_hits_kunit(); + for (i =3D 0; i < ARRAY_SIZE(fd_fast_cases); i++) { + len =3D fd_fast_cases[i].build(buf); + memset(&keys, 0, sizeof(keys)); + __skb_flow_dissect(&init_net, NULL, &flow_keys_dissector, + &keys, buf, fd_fast_cases[i].proto, 0, + len, fd_fast_cases[i].flags); + } + KUNIT_EXPECT_EQ(test, flow_dissector_fast_hits_kunit(), before); + + fd_fast_gates_set(true); + len =3D build_ipv4_tcp(buf); + memset(&keys, 0, sizeof(keys)); + __skb_flow_dissect(&init_net, NULL, &flow_keys_dissector, &keys, + buf, htons(ETH_P_IP), 0, len, 0); + KUNIT_EXPECT_GT(test, flow_dissector_fast_hits_kunit(), before); + + /* Flagship consumer shape: skb_get_hash() passes STOP_AT_FLOW_LABEL; + * the fast path must hit for it. + */ + before =3D flow_dissector_fast_hits_kunit(); + memset(&keys, 0, sizeof(keys)); + __skb_flow_dissect(&init_net, NULL, &flow_keys_dissector, &keys, + buf, htons(ETH_P_IP), 0, len, + FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL); + fd_fast_gates_set(false); + KUNIT_EXPECT_GT(test, flow_dissector_fast_hits_kunit(), before); +} + +static void fd_all_gates_off(void) +{ + fd_fast_gates_set(false); +} + +static int fd_fast_suite_init(struct kunit_suite *suite) +{ + fd_all_gates_off(); + return 0; +} + +static void fd_fast_suite_exit(struct kunit_suite *suite) +{ + fd_all_gates_off(); +} + +static struct kunit_case fd_fast_test_cases[] =3D { + KUNIT_CASE_PARAM(fd_fast_equiv_test, fd_fast_gen_params), + KUNIT_CASE(fd_fast_truncation_test), + KUNIT_CASE(fd_fast_deep_nest_test), + KUNIT_CASE(fd_fast_nonlinear_skb_test), + KUNIT_CASE(fd_fast_ineligible_dissector_test), + KUNIT_CASE(fd_fast_skb_plain_test), + KUNIT_CASE(fd_fast_skb_hwaccel_vlan_test), + KUNIT_CASE(fd_fast_skb_hwaccel_vlan_qinq_test), + KUNIT_CASE(fd_fast_skb_shapes_test), + KUNIT_CASE(fd_fast_fuzz_test), + KUNIT_CASE(fd_fast_gates_off_test), + {} +}; + +static struct kunit_suite fd_fast_suite =3D { + .name =3D "flow_dissector_fastpath", + .suite_init =3D fd_fast_suite_init, + .suite_exit =3D fd_fast_suite_exit, + .test_cases =3D fd_fast_test_cases, +}; + +kunit_test_suite(fd_fast_suite); + +MODULE_DESCRIPTION("KUnit fast/slow equivalence tests for the flow dissect= or"); +MODULE_LICENSE("GPL"); --=20 2.54.0 From nobody Sat Jul 25 16:44:51 2026 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 16BF23090D7 for ; Thu, 16 Jul 2026 00:44:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162657; cv=none; b=Iphs4D1kRvmv8pa+W8WfuDXxNVMYw9ppmCyo9zhQ2QL37/ahTVV5SHDHyv1/wiOjf/hMu+AL+4VnPtsD7+X4oiJ+bXRzovz7QMOocmDC4hsO3XD3RtrgXLwhj9P8W52mEMFzqhlxPrjKWZLw00iCAwJtO3d1qLcrxSPo6FJNtKM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784162657; c=relaxed/simple; bh=oATQsjjSeuM9iTtIYsSo8o1omXnNEePKGjgizoxIB3g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qTKwsJ45bDOrbt59t6dn65w8MV+BY73qcLDCt+azt31L9c3a6nlj5w5Bgi3Z+i1ge3t74ltQckkd1KLr/Xj3uJGVQ3fdoBcXxQCF6G3vpWpm9BgczmWMZid9vGP/1bjNyvrGBwkke3ufzw7pmnKHGKInTA8Y3Z69h8MkPWH3wZk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=nxTTDM1/; arc=none smtp.client-ip=209.85.214.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nxTTDM1/" Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-2cac59f8b64so70303055ad.0 for ; Wed, 15 Jul 2026 17:44:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1784162654; x=1784767454; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=+34tg30fHZs04P1TYJFjNrZ6J+jY9NedABLOWWO5q2k=; b=nxTTDM1/2LpT1iM/RhAMgb4JjOAsqrpDj7449lB02+L+TFVyuQkri83BB8WszI0VfX Wy2+LDz7xlh8VnOjB2eLWiNuht87f55dB3/svf2h72kZic/9ZcEbTryGvF0bwoKz++5L UdkZEzbvslll0MUHYeL3pKUZyf3QM5TdeRkIzgXfsJpujJloI2Def8nQBTbqZecbyL+J nt1zwPmgS69Qkh1XUr4UABpOJ6JdulDIyMoj8pUYdFj0YXsJxUc/DkVRtZK9ZZXBl1SZ 3cXbq6WpIOfYk9FgTjt9vv2/shTXQxB7KDsA4otycIP74ObEPtHO9TLkZl1DRxc3DnAq 1apg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1784162654; x=1784767454; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=+34tg30fHZs04P1TYJFjNrZ6J+jY9NedABLOWWO5q2k=; b=r99RXHzZztjAnlFuLOYM/pB5O+NR+FY8qIM6jHq5Wms/ThkLlCB7RbxsNhVxr1QbYa FtZ8CtBtG6scTVRxXC2ujUeXGMJKcymfCRFKIq4fH5VI72pKOOJPqc2brhkMFor2jg/P n8/mNcJYj7c441RMrFmzsL5aDVmxXFY4hvIQ3/id+63UZAPcjhLCnU/F945od4mUwd3j guAvCjEnzoOLxrmVPCwqgqCtmD/5eFtUdnVwAqRj9Tpa5NR153yPuKpkYEamZ+Mdhl3N RNs/ym7BNrSdkjvZ6pe1/mrGDeP4BjgqVB5c3MWnWgABlBb4eJaPbFIYPpE0Bft80SMa YA1g== X-Forwarded-Encrypted: i=1; AHgh+RpULKg8IR7CdAM4cB7qFCoUIp6O+FSTuRuiW+ZoXF4RK/F+jWY/Us0HhjyA2pkoAXMnSu5xN2XJEMFgXRc=@vger.kernel.org X-Gm-Message-State: AOJu0YxfTVQcznzlITWeRQ8YPHLFY2um5f+8iGiEODLf376v4JrfEjfo z51767uhFqc5iWHdKPlOolEBrBWIo+6lJvXapVps3yBOZDEEOscjDXWI X-Gm-Gg: AfdE7cntdPzAtWxV3gbt6pAvDQ/RjnopnxQYGU49k0fuMv7Z+CpLsyA1CPL4Yhxu8+q u5pa6tkpWAux0jX4chgfweXX7p98tAiI7RGXj6iQvkwI4bIYx7NbPrT+4Jls9b0cqyG4QuOF3rQ oSVQe0VlGONuYmvkdM3OQx/uI4kosYhn273N8Nf5UddbOG4yRLwr9m2aweslAMQAzcCxEqVfhCX XUikFI/CRBgg09Cl0YdY2/A4JUWF8dP9DgpwrYCP16bwQEVK7pJ8uzyxc9aizGElWlaEV9QmfDF lll+qKc211e4tAn7qN7qAmqejGwM8dZefINmrNAxjvhgKsAJUfN30sQB+xPl4ci/fOm7DSJZonu emULzZ3sAUhwoGv2jM/EXw/L3H4ODstqIG5w4IzKPDVm2J/xs0MEdKPmncsGG2aDPjdN0fIgRoP I/RabdJ8Eg/zRT4/ey9nQvj2HTvFTVcqDj3bXLACnRRO/gHua5jmKbPTVIc4uOLyAzlKCy1bwcC B0Wuw8/ltBpuWLzyhi0 X-Received: by 2002:a05:6a21:6d9b:b0:3b4:71a9:cd8f with SMTP id adf61e73a8af0-3c36c365548mr5456208637.41.1784162654045; Wed, 15 Jul 2026 17:44:14 -0700 (PDT) Received: from l.localdomain ([76.166.220.66]) by smtp.gmail.com with ESMTPSA id a92af1059eb24-13cd4284a92sm4107620c88.4.2026.07.15.17.44.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 15 Jul 2026 17:44:13 -0700 (PDT) From: Dave Seddon To: netdev@vger.kernel.org Cc: "David S . Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Stanislav Fomichev , Tom Herbert , Willem de Bruijn , linux-kernel@vger.kernel.org, Dave Seddon Subject: [PATCH net-next v1 11/11] Documentation: networking: add flow_dissector overview and fast-path guide Date: Wed, 15 Jul 2026 17:43:57 -0700 Message-ID: <20260716004357.3652679-12-dave.seddon.ca@gmail.com> X-Mailer: git-send-email 2.54.0 In-Reply-To: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> References: <20260716004357.3652679-1-dave.seddon.ca@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" The kernel has no general flow_dissector documentation: Documentation/bpf/ prog_flow_dissector.rst covers only the BPF override mechanism (and was itself in no toctree). Add Documentation/networking/flow_dissector.rst as a standalone overview -- what the flow dissector is, struct flow_keys and the consumers of skb->hash (RSS/RPS/RFS, ECMP/multipath, bonding/LAG, tc-flower, aRFS), the generic protocol-graph parser, and the relationship to the BPF flow dissector -- and document the opt-in fast paths added by this series: the byte-identical per-shape gates, the break-even model for when to enable a gate (with the observable /proc/net/flow_dissector_stats). The per-knob reference stays in Documentation/admin-guide/sysctl/net.rst; this doc links to it rather than duplicating it. Also index the previously-orphaned Documentation/bpf/prog_flow_dissector.rst and cross-link it with the new overview (the bpf/index.rst hunk can be split out if the BPF maintainers prefer). Assisted-by: Claude:claude-fable-5 sparse smatch Signed-off-by: Dave Seddon --- Documentation/bpf/index.rst | 1 + Documentation/bpf/prog_flow_dissector.rst | 2 + Documentation/networking/flow_dissector.rst | 133 ++++++++++++++++++++ Documentation/networking/index.rst | 1 + 4 files changed, 137 insertions(+) create mode 100644 Documentation/networking/flow_dissector.rst diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst index 0d5c6f659266..6e26731fa06f 100644 --- a/Documentation/bpf/index.rst +++ b/Documentation/bpf/index.rst @@ -23,6 +23,7 @@ that goes into great technical depth about the BPF Archit= ecture. cpumasks fs_kfuncs programs + prog_flow_dissector maps bpf_prog_run classic_vs_extended.rst diff --git a/Documentation/bpf/prog_flow_dissector.rst b/Documentation/bpf/= prog_flow_dissector.rst index f24270b8b034..f4c4648461ca 100644 --- a/Documentation/bpf/prog_flow_dissector.rst +++ b/Documentation/bpf/prog_flow_dissector.rst @@ -9,6 +9,8 @@ Overview =20 Flow dissector is a routine that parses metadata out of the packets. It's used in the various places in the networking subsystem (RFS, flow hash, et= c). +See :doc:`/networking/flow_dissector` for an overview of the flow dissector +and the built-in C implementation this program type overrides. =20 BPF flow dissector is an attempt to reimplement C-based flow dissector log= ic in BPF to gain all the benefits of BPF verifier (namely, limits on the diff --git a/Documentation/networking/flow_dissector.rst b/Documentation/ne= tworking/flow_dissector.rst new file mode 100644 index 000000000000..014f6d124b4f --- /dev/null +++ b/Documentation/networking/flow_dissector.rst @@ -0,0 +1,133 @@ +.. SPDX-License-Identifier: GPL-2.0 + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +Flow Dissector +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Overview +=3D=3D=3D=3D=3D=3D=3D=3D + +The flow dissector parses a packet's headers into a ``struct flow_keys``: = the +addresses, ports, and other fields that identify the flow the packet belon= gs +to. Its entry point is ``__skb_flow_dissect()`` in +``net/core/flow_dissector.c``. + +The dissected keys feed ``skb->hash`` (via ``flow_hash_from_keys()``), whi= ch is +the canonical flow identity consumed all over the stack, including: + +* receive-side steering: RSS, RPS, RFS and accelerated RFS (aRFS); +* transmit-side spreading: ECMP / multipath route selection and bonding / = LAG + ``xmit_hash_policy``; +* classification and offload: tc-flower. + +A single dissection therefore feeds every one of these, so the dissector s= its +on a hot path in most networking workloads. + +Two standard dissectors are built in: ``flow_keys_dissector`` (the general= one) +and ``flow_keys_dissector_symmetric`` (same source and destination hashed +symmetrically). In addition, subsystems such as tc-flower build **custom** +dissector instances at run time that request only the keys they need. A cu= stom +instance carries its own ``used_keys`` bitmask and per-key offset table, s= o its +field accesses are resolved dynamically at each dissect. + +struct flow_keys +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +``struct flow_keys`` (see ``include/net/flow_dissector.h``) collects the +dissected metadata. Its main members are: + +* ``control`` -- address type and control flags (for example + ``FLOW_DIS_ENCAPSULATION`` when the keys came from inside a tunnel, and + ``FLOW_DIS_IS_FRAGMENT``); +* ``basic`` -- L3 protocol (``n_proto``) and L4 protocol (``ip_proto``); +* ``addrs`` -- source and destination addresses (IPv4 or IPv6); +* ``ports`` -- source and destination ports; +* ``vlan`` / ``cvlan`` -- 802.1Q / 802.1AD tags; +* ``tags``, ``keyid``, ``icmp`` -- MPLS/flow-label tags, GRE/tunnel key, a= nd + ICMP identifiers. + +The dissection path +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +The built-in dissector is a generic protocol-graph parser: it walks the he= ader +chain in a loop, dispatching on the current L2/L3 ethertype and then the L4 +protocol, following encapsulation (IP-in-IP, GRE, ...) by re-entering the = loop +on the inner header. The number of headers it will descend through is boun= ded +by ``MAX_FLOW_DISSECT_HDRS`` so that a crafted deeply nested packet cannot= make +it loop without limit. + +This generic parser is referred to below as the *slow path*: it is table- = and +loop-driven and handles every protocol the dissector knows about. + +The BPF flow dissector +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +A per-network-namespace BPF program can be attached to replace the built-in +dissector for the protocols it chooses to handle. When one is attached, +``__skb_flow_dissect()`` runs it **first**; if the program returns a verdi= ct +other than ``BPF_FLOW_DISSECTOR_CONTINUE``, its result is used and the bui= lt-in +dissector (including the fast paths below) is not run for that packet. A s= ystem +running an attached BPF dissector therefore sees behaviour defined entirel= y by +its program. + +See :doc:`/bpf/prog_flow_dissector` for the BPF program type, its API, and= its +reference implementation. + +Opt-in fast paths +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +For the packet shapes that dominate real deployments, the built-in dissect= or +also provides an opt-in, straight-line *fast path* that produces the same +result as the slow-path graph walk with far fewer instructions. Restrictin= g the +fast path to the two standard dissectors turns their per-key offsets into +compile-time constants, so the parse becomes branch-light straight-line co= de; +the isolated per-dissect cost of the eligible shapes drops by roughly half, +with the largest wins on in-order cores. + +Each shape has its own gate -- a ``static_branch`` exposed as a sysctl und= er +``/proc/sys/net/flow_dissector/`` -- and **every gate is off by default**.= When +a gate is off, the only added cost is one not-taken branch per dissect, an= d the +dissector's output is exactly what it was before. The fast path runs only +*after* the BPF hook described above, so an attached BPF program always ta= kes +precedence. + +The fast paths are **byte-identical by contract**: for any packet, a fast = path +either writes exactly the ``flow_keys`` the slow path would have written, = or it +returns false and the slow path runs. There are only these two outcomes, a= nd +the equivalence is enforced in-tree by a KUnit test suite +(``CONFIG_FLOW_DISSECTOR_KUNIT_TEST``). Enabling a byte-identical shape ga= te +never changes any consumer's hash; it only makes the dissection cheaper. + +The eligible byte-identical shapes are plain Ethernet + IPv4/IPv6 + TCP/UD= P, +single and stacked VLAN (QinQ), PPPoE sessions, a single MPLS label, the +IP-in-IP family, and plain GRE. The individual knobs are documented in +:doc:`/admin-guide/sysctl/net` under ``/proc/sys/net/flow_dissector``. + +When to enable +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Turning a byte-identical shape gate on is worthwhile when enough traffic m= atches +that shape to repay the small cost a non-matching packet pays while the ga= te is +on. Concretely, if a matching packet saves ``S`` and a non-matching packet= costs +``C`` (both measured per shape and per micro-architecture), enabling the g= ate is +net-positive once the fraction of traffic matching the shape exceeds the +break-even ``p_be =3D C / (S + C)``. + +That fraction is observable: the read-only file +``/proc/net/flow_dissector_stats`` reports, per shape, how many packets th= e slow +path handled, how many the fast body handled, and the resulting eligible +fraction, so ``(occurrences + fast_hits) / dissects`` is a gate-invariant +measure of how much traffic each shape would accelerate. + +A separate RFC proposes an optional ``auto`` mode built on these counters:= the +kernel would sample them over a packet-count window and flip the +byte-identical gates itself, with hysteresis and a flip-rate cap, so the +operator does not have to tune each shape by hand. That controller is not = part +of this series. + +References +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +* :doc:`/admin-guide/sysctl/net` -- the ``/proc/sys/net/flow_dissector/`` + per-shape knob reference and the ``/proc/net/flow_dissector_stats`` file. +* :doc:`/bpf/prog_flow_dissector` -- the BPF flow dissector program type. diff --git a/Documentation/networking/index.rst b/Documentation/networking/= index.rst index 44a422ad3b05..65368050ebab 100644 --- a/Documentation/networking/index.rst +++ b/Documentation/networking/index.rst @@ -52,6 +52,7 @@ Contents: eql fib_trie filter + flow_dissector generic-hdlc generic_netlink ../netlink/specs/index --=20 2.54.0