[v2] net: erspan: set lltx to avoid sch_direct_xmit deadlock

[PATCH net v2] net: erspan: set lltx to avoid sch_direct_xmit deadlock

Posted by Yun Zhou 1 week, 5 days ago

erspan_xmit() re-enters the network stack via ip_tunnel_xmit(), causing
nested acquisition of _xmit_lock on the underlay device while already
holding the ERSPAN device's _xmit_lock. Both are ARPHRD_ETHER and share
the same lockdep class, creating an ABBA deadlock:

  sch_direct_xmit [lock erspan] -> erspan_xmit -> ip_tunnel_xmit ->
  ip_output -> __dev_queue_xmit -> sch_direct_xmit [lock underlay]

Set dev->lltx = true so HARD_TX_LOCK() skips the spinlock for ERSPAN.
This is safe as erspan_xmit() has no shared mutable state: o_seqno is
atomic, stats use atomic_long_inc, and dst_cache is per-CPU. GRETAP,
the sibling device with identical xmit structure, already sets lltx.

Closes: https://syzkaller.appspot.com/bug?extid=9bda1b9fbb7fbdf9b62b
Reported-by: syzbot+9bda1b9fbb7fbdf9b62b@syzkaller.appspotmail.com
Fixes: 84e54fe0a5ea ("gre: introduce native tunnel support for ERSPAN")
Signed-off-by: Yun Zhou <yun.zhou@windriver.com>
---
v2:
  - change subject prefix to [PATCH net]

 net/ipv4/ip_gre.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/net/ipv4/ip_gre.c b/net/ipv4/ip_gre.c
index 3efdfb4ffa21..9fbff16cda1d 100644
--- a/net/ipv4/ip_gre.c
+++ b/net/ipv4/ip_gre.c
@@ -1363,6 +1363,8 @@ static int erspan_tunnel_init(struct net_device *dev)
 	dev->features		|= GRE_FEATURES;
 	dev->hw_features	|= GRE_FEATURES;
 	dev->priv_flags		|= IFF_LIVE_ADDR_CHANGE;
+	/* Skip TX lock: xmit re-enters stack, risking ABBA with underlay */
+	dev->lltx = true;
 	netif_keep_dst(dev);
 
 	return ip_tunnel_init(dev);
-- 
2.43.0

Re: [PATCH net v2] net: erspan: set lltx to avoid sch_direct_xmit deadlock

Posted by Ido Schimmel 5 days, 12 hours ago

On Mon, Jul 13, 2026 at 11:14:35PM +0800, Yun Zhou wrote:
> erspan_xmit() re-enters the network stack via ip_tunnel_xmit(), causing
> nested acquisition of _xmit_lock on the underlay device while already
> holding the ERSPAN device's _xmit_lock. Both are ARPHRD_ETHER and share
> the same lockdep class, creating an ABBA deadlock:
> 
>   sch_direct_xmit [lock erspan] -> erspan_xmit -> ip_tunnel_xmit ->
>   ip_output -> __dev_queue_xmit -> sch_direct_xmit [lock underlay]
> 
> Set dev->lltx = true so HARD_TX_LOCK() skips the spinlock for ERSPAN.
> This is safe as erspan_xmit() has no shared mutable state: o_seqno is
> atomic, stats use atomic_long_inc, and dst_cache is per-CPU. GRETAP,
> the sibling device with identical xmit structure, already sets lltx.

erspan_xmit() (unlike gre_tap_xmit()) is performing non-atomic
__clear_bit() on shared tunnel flags and KCSAN will probably flag it.

Eric had a patch [1] that changes erspan_xmit() to use a private copy of
these flags. I think it's better to wait for Eric's patch to be merged
before setting lltx.

Eric, can you please submit v2 of your patch to net?

Also, doesn't ip6erspan suffer from the same problem? Please try to
reproduce and fix.

[1] https://lore.kernel.org/netdev/20260615140333.3161072-1-edumazet@google.com/

> 
> Closes: https://syzkaller.appspot.com/bug?extid=9bda1b9fbb7fbdf9b62b
> Reported-by: syzbot+9bda1b9fbb7fbdf9b62b@syzkaller.appspotmail.com
> Fixes: 84e54fe0a5ea ("gre: introduce native tunnel support for ERSPAN")
> Signed-off-by: Yun Zhou <yun.zhou@windriver.com>
> ---
> v2:
>   - change subject prefix to [PATCH net]
> 
>  net/ipv4/ip_gre.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/net/ipv4/ip_gre.c b/net/ipv4/ip_gre.c
> index 3efdfb4ffa21..9fbff16cda1d 100644
> --- a/net/ipv4/ip_gre.c
> +++ b/net/ipv4/ip_gre.c
> @@ -1363,6 +1363,8 @@ static int erspan_tunnel_init(struct net_device *dev)
>  	dev->features		|= GRE_FEATURES;
>  	dev->hw_features	|= GRE_FEATURES;
>  	dev->priv_flags		|= IFF_LIVE_ADDR_CHANGE;
> +	/* Skip TX lock: xmit re-enters stack, risking ABBA with underlay */
> +	dev->lltx = true;
>  	netif_keep_dst(dev);
>  
>  	return ip_tunnel_init(dev);
> -- 
> 2.43.0
>

Re: [PATCH net v2] net: erspan: set lltx to avoid sch_direct_xmit deadlock

Posted by Zhou, Yun 5 days, 10 hours ago


On 7/20/26 15:53, Ido Schimmel wrote:
> CAUTION: This email comes from a non Wind River email account!
> Do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> On Mon, Jul 13, 2026 at 11:14:35PM +0800, Yun Zhou wrote:
>> erspan_xmit() re-enters the network stack via ip_tunnel_xmit(), causing
>> nested acquisition of _xmit_lock on the underlay device while already
>> holding the ERSPAN device's _xmit_lock. Both are ARPHRD_ETHER and share
>> the same lockdep class, creating an ABBA deadlock:
>>
>>    sch_direct_xmit [lock erspan] -> erspan_xmit -> ip_tunnel_xmit ->
>>    ip_output -> __dev_queue_xmit -> sch_direct_xmit [lock underlay]
>>
>> Set dev->lltx = true so HARD_TX_LOCK() skips the spinlock for ERSPAN.
>> This is safe as erspan_xmit() has no shared mutable state: o_seqno is
>> atomic, stats use atomic_long_inc, and dst_cache is per-CPU. GRETAP,
>> the sibling device with identical xmit structure, already sets lltx.
> 
> erspan_xmit() (unlike gre_tap_xmit()) is performing non-atomic
> __clear_bit() on shared tunnel flags and KCSAN will probably flag it.
> 
> Eric had a patch [1] that changes erspan_xmit() to use a private copy of
> these flags. I think it's better to wait for Eric's patch to be merged
> before setting lltx.
> 
> Eric, can you please submit v2 of your patch to net?
> 
> Also, doesn't ip6erspan suffer from the same problem? Please try to
> reproduce and fix.
> 

The following command sequence can reproduce the problem:

ip link set lo up

ip addr add fd00:a::1/128 dev lo
ip addr add fd00:b::1/128 dev lo

ip link add ip6erspan_dev type ip6erspan \
     local fd00:a::1 remote fd00:a::2 \
     seq key 1 erspan_ver 1 erspan 1
ip link set ip6erspan_dev up

ip link add ip6gretap_dev type ip6gretap \
     local fd00:b::1 remote fd00:b::2 seq
ip link set ip6gretap_dev up

ip -6 route add fd00:a::2/128 dev ip6gretap_dev
ip -6 route add fd00:b::2/128 dev ip6erspan_dev

ip -6 neigh add fd00:a::2 lladdr 00:11:22:33:44:55 dev ip6gretap_dev
ip -6 neigh add fd00:b::2 lladdr 00:11:22:33:44:66 dev ip6erspan_dev

Re: [PATCH net v2] net: erspan: set lltx to avoid sch_direct_xmit deadlock

Posted by Eric Dumazet 5 days, 12 hours ago

On Mon, Jul 20, 2026 at 9:54 AM Ido Schimmel <idosch@nvidia.com> wrote:
>
> On Mon, Jul 13, 2026 at 11:14:35PM +0800, Yun Zhou wrote:
> > erspan_xmit() re-enters the network stack via ip_tunnel_xmit(), causing
> > nested acquisition of _xmit_lock on the underlay device while already
> > holding the ERSPAN device's _xmit_lock. Both are ARPHRD_ETHER and share
> > the same lockdep class, creating an ABBA deadlock:
> >
> >   sch_direct_xmit [lock erspan] -> erspan_xmit -> ip_tunnel_xmit ->
> >   ip_output -> __dev_queue_xmit -> sch_direct_xmit [lock underlay]
> >
> > Set dev->lltx = true so HARD_TX_LOCK() skips the spinlock for ERSPAN.
> > This is safe as erspan_xmit() has no shared mutable state: o_seqno is
> > atomic, stats use atomic_long_inc, and dst_cache is per-CPU. GRETAP,
> > the sibling device with identical xmit structure, already sets lltx.
>
> erspan_xmit() (unlike gre_tap_xmit()) is performing non-atomic
> __clear_bit() on shared tunnel flags and KCSAN will probably flag it.
>
> Eric had a patch [1] that changes erspan_xmit() to use a private copy of
> these flags. I think it's better to wait for Eric's patch to be merged
> before setting lltx.
>
> Eric, can you please submit v2 of your patch to net?

Sure, I can work on it today. Thanks!

>
> Also, doesn't ip6erspan suffer from the same problem? Please try to
> reproduce and fix.
>
> [1] https://lore.kernel.org/netdev/20260615140333.3161072-1-edumazet@google.com/
>
> >
> > Closes: https://syzkaller.appspot.com/bug?extid=9bda1b9fbb7fbdf9b62b
> > Reported-by: syzbot+9bda1b9fbb7fbdf9b62b@syzkaller.appspotmail.com
> > Fixes: 84e54fe0a5ea ("gre: introduce native tunnel support for ERSPAN")
> > Signed-off-by: Yun Zhou <yun.zhou@windriver.com>
> > ---
> > v2:
> >   - change subject prefix to [PATCH net]
> >
> >  net/ipv4/ip_gre.c | 2 ++
> >  1 file changed, 2 insertions(+)
> >
> > diff --git a/net/ipv4/ip_gre.c b/net/ipv4/ip_gre.c
> > index 3efdfb4ffa21..9fbff16cda1d 100644
> > --- a/net/ipv4/ip_gre.c
> > +++ b/net/ipv4/ip_gre.c
> > @@ -1363,6 +1363,8 @@ static int erspan_tunnel_init(struct net_device *dev)
> >       dev->features           |= GRE_FEATURES;
> >       dev->hw_features        |= GRE_FEATURES;
> >       dev->priv_flags         |= IFF_LIVE_ADDR_CHANGE;
> > +     /* Skip TX lock: xmit re-enters stack, risking ABBA with underlay */
> > +     dev->lltx = true;
> >       netif_keep_dst(dev);
> >
> >       return ip_tunnel_init(dev);
> > --
> > 2.43.0
> >

Re: [PATCH net v2] net: erspan: set lltx to avoid sch_direct_xmit deadlock

Posted by Zhou, Yun 5 days, 12 hours ago


On 7/20/26 15:53, Ido Schimmel wrote:
> CAUTION: This email comes from a non Wind River email account!
> Do not click links or open attachments unless you recognize the sender and know the content is safe.
> 
> On Mon, Jul 13, 2026 at 11:14:35PM +0800, Yun Zhou wrote:
>> erspan_xmit() re-enters the network stack via ip_tunnel_xmit(), causing
>> nested acquisition of _xmit_lock on the underlay device while already
>> holding the ERSPAN device's _xmit_lock. Both are ARPHRD_ETHER and share
>> the same lockdep class, creating an ABBA deadlock:
>>
>>    sch_direct_xmit [lock erspan] -> erspan_xmit -> ip_tunnel_xmit ->
>>    ip_output -> __dev_queue_xmit -> sch_direct_xmit [lock underlay]
>>
>> Set dev->lltx = true so HARD_TX_LOCK() skips the spinlock for ERSPAN.
>> This is safe as erspan_xmit() has no shared mutable state: o_seqno is
>> atomic, stats use atomic_long_inc, and dst_cache is per-CPU. GRETAP,
>> the sibling device with identical xmit structure, already sets lltx.
> 
> erspan_xmit() (unlike gre_tap_xmit()) is performing non-atomic
> __clear_bit() on shared tunnel flags and KCSAN will probably flag it.
> 
> Eric had a patch [1] that changes erspan_xmit() to use a private copy of
> these flags. I think it's better to wait for Eric's patch to be merged
> before setting lltx.
> 
> Eric, can you please submit v2 of your patch to net?
> 
> Also, doesn't ip6erspan suffer from the same problem? Please try to
> reproduce and fix.
> 

Yes, ip6erspan has the same problem. I will fix it in v3.

Thanks,
Yun

Re: [PATCH net v2] net: erspan: set lltx to avoid sch_direct_xmit deadlock

Posted by Zhou, Yun 6 days, 8 hours ago

Hi Ido,

Could you please help review this patch when you have a moment?

Thanks,
Yun

On 7/13/2026 11:14 PM, Yun Zhou wrote:
> erspan_xmit() re-enters the network stack via ip_tunnel_xmit(), causing
> nested acquisition of _xmit_lock on the underlay device while already
> holding the ERSPAN device's _xmit_lock. Both are ARPHRD_ETHER and share
> the same lockdep class, creating an ABBA deadlock:
> 
>    sch_direct_xmit [lock erspan] -> erspan_xmit -> ip_tunnel_xmit ->
>    ip_output -> __dev_queue_xmit -> sch_direct_xmit [lock underlay]
> 
> Set dev->lltx = true so HARD_TX_LOCK() skips the spinlock for ERSPAN.
> This is safe as erspan_xmit() has no shared mutable state: o_seqno is
> atomic, stats use atomic_long_inc, and dst_cache is per-CPU. GRETAP,
> the sibling device with identical xmit structure, already sets lltx.
> 
> Closes: https://syzkaller.appspot.com/bug?extid=9bda1b9fbb7fbdf9b62b
> Reported-by: syzbot+9bda1b9fbb7fbdf9b62b@syzkaller.appspotmail.com
> Fixes: 84e54fe0a5ea ("gre: introduce native tunnel support for ERSPAN")
> Signed-off-by: Yun Zhou <yun.zhou@windriver.com>
> ---
> v2:
>    - change subject prefix to [PATCH net]
> 
>   net/ipv4/ip_gre.c | 2 ++
>   1 file changed, 2 insertions(+)
> 
> diff --git a/net/ipv4/ip_gre.c b/net/ipv4/ip_gre.c
> index 3efdfb4ffa21..9fbff16cda1d 100644
> --- a/net/ipv4/ip_gre.c
> +++ b/net/ipv4/ip_gre.c
> @@ -1363,6 +1363,8 @@ static int erspan_tunnel_init(struct net_device *dev)
>   	dev->features		|= GRE_FEATURES;
>   	dev->hw_features	|= GRE_FEATURES;
>   	dev->priv_flags		|= IFF_LIVE_ADDR_CHANGE;
> +	/* Skip TX lock: xmit re-enters stack, risking ABBA with underlay */
> +	dev->lltx = true;
>   	netif_keep_dst(dev);
>   
>   	return ip_tunnel_init(dev);