From nobody Sat Jun 13 16:24:17 2026 Received: from unimail.uni-dortmund.de (mx1.hrz.uni-dortmund.de [129.217.128.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0DBDE49250C; Wed, 6 May 2026 14:11:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=129.217.128.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778076665; cv=none; b=NkL3VGWCLj1N5F/AQgiwgTXn+jmERhiJXLGZ2EqiLCqQOVd0Y09P/+nmynFuTIBKjrwtCZ/u2vpcFA0ytEYDC3T3JWtH0RniTGrQhk84h5m1TToTeaXojNCHRwpTsW4de0YfBzmMkxuE8CiWoWxQWMJcUnwE3fMJDpueTQQjMMs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778076665; c=relaxed/simple; bh=52b+YNhKvnHeBpft7vVFWaEBKKldagGULo7C9xvyt1w=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=P2UraNS9dOqvcEwSUpL3F+mEw1oH8e9WK/Pla5GyMlf5Tpo11DEG4InlIGvbkHSbSysforfOLw/PweWXWR/wKWRiZwmR0cdtYx27Eb0+fMrEEXj6TSLvwoBtFe3rLuUDpqtZVBKvgyLJSYmL5vA3SaFqM6Nngn86R7ougYaopMk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=tu-dortmund.de; spf=pass smtp.mailfrom=tu-dortmund.de; dkim=pass (1024-bit key) header.d=tu-dortmund.de header.i=@tu-dortmund.de header.b=L0xqu+cv; arc=none smtp.client-ip=129.217.128.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=tu-dortmund.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tu-dortmund.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=tu-dortmund.de header.i=@tu-dortmund.de header.b="L0xqu+cv" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tu-dortmund.de; s=unimail; t=1778076651; bh=52b+YNhKvnHeBpft7vVFWaEBKKldagGULo7C9xvyt1w=; h=From:To:Subject:Date:In-Reply-To:References; b=L0xqu+cv/1FHMUHTfOwLHEn0LKozvoTxqk3XR+2QsA13Hw38tq/sdYglYrPeNLkQC VYaUg7WVBv5lGjcATR0T9eqoMn/LLMXq+Xt0yG9O9jy3ayoLb0wJoHSnlvL0Ri/qi6 /kW5ZFpWvzHSihsHkbji4qtoeqrnUyL5lrgnubc4= Received: from simon-Latitude-5450.cni.e-technik.tu-dortmund.de ([129.217.186.46]) (authenticated bits=0) by unimail.uni-dortmund.de (8.19.0.1/8.19.0.1) with ESMTPSA id 646EAnWY015408 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Wed, 6 May 2026 16:10:51 +0200 (CEST) From: Simon Schippers To: willemdebruijn.kernel@gmail.com, jasowang@redhat.com, andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, mst@redhat.com, eperezma@redhat.com, leiyang@redhat.com, stephen@networkplumber.org, jon@nutanix.com, tim.gebauer@tu-dortmund.de, simon.schippers@tu-dortmund.de, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux.dev Subject: [PATCH net-next v10 1/4] tun/tap: add ptr_ring consume helper with netdev queue wakeup Date: Wed, 6 May 2026 16:10:30 +0200 Message-ID: <20260506141033.180450-2-simon.schippers@tu-dortmund.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260506141033.180450-1-simon.schippers@tu-dortmund.de> References: <20260506141033.180450-1-simon.schippers@tu-dortmund.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce tun_ring_consume() that wraps ptr_ring_consume() and calls __tun_wake_queue(). The latter wakes the stopped netdev subqueue once half of the ring capacity has been consumed, tracked via the new cons_cnt field in tun_file. cons_cnt is updated while holding the ring consumer lock, avoiding races. As a safety net, the queue is also woken when the ring becomes empty. The point is to allow the queue to be stopped when it gets full, which is required for traffic shaping - implemented by the following "avoid ptr_ring tail-drop when a qdisc is present". That patch also explains the pairing of the smp_mb() of __tun_wake_queue(). Without the corresponding queue stopping, this patch alone causes no regression for a tap setup sending to a qemu VM: 1.132 Mpps to 1.144 Mpps. Details: AMD Ryzen 5 5600X at 4.3 GHz, 3200 MHz RAM, isolated QEMU threads, pktgen sender; Avg over 50 runs @ 100,000,000 packets; SRSO and spectre v2 mitigations disabled. Co-developed-by: Tim Gebauer Signed-off-by: Tim Gebauer Signed-off-by: Simon Schippers --- drivers/net/tun.c | 54 +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 50 insertions(+), 4 deletions(-) diff --git a/drivers/net/tun.c b/drivers/net/tun.c index b183189f1853..00ecf128fe8e 100644 --- a/drivers/net/tun.c +++ b/drivers/net/tun.c @@ -145,6 +145,7 @@ struct tun_file { struct list_head next; struct tun_struct *detached; struct ptr_ring tx_ring; + int cons_cnt; struct xdp_rxq_info xdp_rxq; }; =20 @@ -557,6 +558,13 @@ void tun_ptr_free(void *ptr) } EXPORT_SYMBOL_GPL(tun_ptr_free); =20 +static void tun_reset_cons_cnt(struct tun_file *tfile) +{ + spin_lock(&tfile->tx_ring.consumer_lock); + tfile->cons_cnt =3D 0; + spin_unlock(&tfile->tx_ring.consumer_lock); +} + static void tun_queue_purge(struct tun_file *tfile) { void *ptr; @@ -564,6 +572,7 @@ static void tun_queue_purge(struct tun_file *tfile) while ((ptr =3D ptr_ring_consume(&tfile->tx_ring)) !=3D NULL) tun_ptr_free(ptr); =20 + tun_reset_cons_cnt(tfile); skb_queue_purge(&tfile->sk.sk_write_queue); skb_queue_purge(&tfile->sk.sk_error_queue); } @@ -730,6 +739,7 @@ static int tun_attach(struct tun_struct *tun, struct fi= le *file, goto out; } =20 + tun_reset_cons_cnt(tfile); tfile->queue_index =3D tun->numqueues; tfile->socket.sk->sk_shutdown &=3D ~RCV_SHUTDOWN; =20 @@ -2115,13 +2125,46 @@ static ssize_t tun_put_user(struct tun_struct *tun, return total; } =20 -static void *tun_ring_recv(struct tun_file *tfile, int noblock, int *err) +/* Callers must hold ring.consumer_lock */ +static void __tun_wake_queue(struct tun_struct *tun, + struct tun_file *tfile, int consumed) +{ + struct netdev_queue *txq =3D netdev_get_tx_queue(tun->dev, + tfile->queue_index); + + /* Paired with smp_mb__after_atomic() in tun_net_xmit() */ + smp_mb(); + if (netif_tx_queue_stopped(txq)) { + tfile->cons_cnt +=3D consumed; + if (tfile->cons_cnt >=3D tfile->tx_ring.size / 2 || + __ptr_ring_empty(&tfile->tx_ring)) { + netif_tx_wake_queue(txq); + tfile->cons_cnt =3D 0; + } + } +} + +static void *tun_ring_consume(struct tun_struct *tun, struct tun_file *tfi= le) +{ + void *ptr; + + spin_lock(&tfile->tx_ring.consumer_lock); + ptr =3D __ptr_ring_consume(&tfile->tx_ring); + if (ptr) + __tun_wake_queue(tun, tfile, 1); + + spin_unlock(&tfile->tx_ring.consumer_lock); + return ptr; +} + +static void *tun_ring_recv(struct tun_struct *tun, struct tun_file *tfile, + int noblock, int *err) { DECLARE_WAITQUEUE(wait, current); void *ptr =3D NULL; int error =3D 0; =20 - ptr =3D ptr_ring_consume(&tfile->tx_ring); + ptr =3D tun_ring_consume(tun, tfile); if (ptr) goto out; if (noblock) { @@ -2133,7 +2176,7 @@ static void *tun_ring_recv(struct tun_file *tfile, in= t noblock, int *err) =20 while (1) { set_current_state(TASK_INTERRUPTIBLE); - ptr =3D ptr_ring_consume(&tfile->tx_ring); + ptr =3D tun_ring_consume(tun, tfile); if (ptr) break; if (signal_pending(current)) { @@ -2170,7 +2213,7 @@ static ssize_t tun_do_read(struct tun_struct *tun, st= ruct tun_file *tfile, =20 if (!ptr) { /* Read frames from ring */ - ptr =3D tun_ring_recv(tfile, noblock, &err); + ptr =3D tun_ring_recv(tun, tfile, noblock, &err); if (!ptr) return err; } @@ -3406,6 +3449,8 @@ static int tun_chr_open(struct inode *inode, struct f= ile * file) return -ENOMEM; } =20 + tun_reset_cons_cnt(tfile); + mutex_init(&tfile->napi_mutex); RCU_INIT_POINTER(tfile->tun, NULL); tfile->flags =3D 0; @@ -3614,6 +3659,7 @@ static int tun_queue_resize(struct tun_struct *tun) for (i =3D 0; i < tun->numqueues; i++) { tfile =3D rtnl_dereference(tun->tfiles[i]); rings[i] =3D &tfile->tx_ring; + tun_reset_cons_cnt(tfile); } list_for_each_entry(tfile, &tun->disabled, next) rings[i++] =3D &tfile->tx_ring; --=20 2.43.0 From nobody Sat Jun 13 16:24:17 2026 Received: from unimail.uni-dortmund.de (mx1.hrz.uni-dortmund.de [129.217.128.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D9FD492508; Wed, 6 May 2026 14:11:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=129.217.128.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778076664; cv=none; b=C2H60RQKet+9Nu4OU1N4uZ8/OMahw2/Odgw/JJIuZB2slHPdS+d7m/lniOn6Ht+DjBBr8YydPnobUOwbZ3lelL9v4vVA9XyZq+EIYt0m1hEMIYhtFJIVYEfx1Zx9mOeX91vPzUSYFNeEMVr2kEckqtafCO1AE1Y2TTGt3MUpVmc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778076664; c=relaxed/simple; bh=jdku3lwlGP7dIt/wFobLtUxMN8c01eAC84JHaTYMDnc=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bgPeflzK4VAuSJlQewsppJGtLFd0oIyCKrCp3YD4ThhSDb/X8XdsSkjDTuYxn8AFhyRAJj3dCaIEOvGTVGfpVzsDtc3XXabBReqOkaUEMq2fOHH31F/QSbtPWy517P8VCN9b55rx9BN97kPAYOlKvmZQt9+MdA1Kkzi+nGCV+WA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=tu-dortmund.de; spf=pass smtp.mailfrom=tu-dortmund.de; dkim=pass (1024-bit key) header.d=tu-dortmund.de header.i=@tu-dortmund.de header.b=MhEJGJvp; arc=none smtp.client-ip=129.217.128.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=tu-dortmund.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tu-dortmund.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=tu-dortmund.de header.i=@tu-dortmund.de header.b="MhEJGJvp" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tu-dortmund.de; s=unimail; t=1778076651; bh=jdku3lwlGP7dIt/wFobLtUxMN8c01eAC84JHaTYMDnc=; h=From:To:Subject:Date:In-Reply-To:References; b=MhEJGJvpNHj0h83819p10Z5MQG3MxCjKfOYyC8ozLq9tq2QyTm0PTY8hZoi/EROMw xKCY5PSR72L1Vg7uiI/1z7IBaVEughGGwI8ujOKh/qD0x9N6xydC8mChSFh4OUoFmr GUNOq9L4g/xtQ3nW4b5ivM+QlciRHIKqKChGV6hU= Received: from simon-Latitude-5450.cni.e-technik.tu-dortmund.de ([129.217.186.46]) (authenticated bits=0) by unimail.uni-dortmund.de (8.19.0.1/8.19.0.1) with ESMTPSA id 646EAnWa015408 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Wed, 6 May 2026 16:10:51 +0200 (CEST) From: Simon Schippers To: willemdebruijn.kernel@gmail.com, jasowang@redhat.com, andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, mst@redhat.com, eperezma@redhat.com, leiyang@redhat.com, stephen@networkplumber.org, jon@nutanix.com, tim.gebauer@tu-dortmund.de, simon.schippers@tu-dortmund.de, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux.dev Subject: [PATCH net-next v10 2/4] vhost-net: wake queue of tun/tap after ptr_ring consume Date: Wed, 6 May 2026 16:10:31 +0200 Message-ID: <20260506141033.180450-3-simon.schippers@tu-dortmund.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260506141033.180450-1-simon.schippers@tu-dortmund.de> References: <20260506141033.180450-1-simon.schippers@tu-dortmund.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add tun_wake_queue() to tun.c and export it for use by vhost-net. The function validates that the file belongs to a tun/tap device, dereferences the tun_struct under RCU, and delegates to __tun_wake_queue(). vhost_net_buf_produce() now calls tun_wake_queue() after a successful batched consume of the ring to allow the netdev subqueue to be woken up. The point is to allow the queue to be stopped when it gets full, which is required for traffic shaping - implemented by the following "avoid ptr_ring tail-drop when a qdisc is present". Without the corresponding queue stopping, this patch alone causes no throughput regression for a tap+vhost-net setup sending to a qemu VM: 3.857 Mpps to 3.891 Mpps. Details: AMD Ryzen 5 5600X at 4.3 GHz, 3200 MHz RAM, isolated QEMU threads, XDP drop program active in VM, pktgen sender; Avg over 50 runs @ 100,000,000 packets. SRSO and spectre v2 mitigations disabled. Co-developed-by: Tim Gebauer Signed-off-by: Tim Gebauer Signed-off-by: Simon Schippers --- drivers/net/tun.c | 23 +++++++++++++++++++++++ drivers/vhost/net.c | 21 +++++++++++++++------ include/linux/if_tun.h | 3 +++ 3 files changed, 41 insertions(+), 6 deletions(-) diff --git a/drivers/net/tun.c b/drivers/net/tun.c index 00ecf128fe8e..fc358c4c355b 100644 --- a/drivers/net/tun.c +++ b/drivers/net/tun.c @@ -3776,6 +3776,29 @@ struct ptr_ring *tun_get_tx_ring(struct file *file) } EXPORT_SYMBOL_GPL(tun_get_tx_ring); =20 +/* Callers must hold ring.consumer_lock */ +void tun_wake_queue(struct file *file, int consumed) +{ + struct tun_file *tfile; + struct tun_struct *tun; + + if (file->f_op !=3D &tun_fops) + return; + + tfile =3D file->private_data; + if (!tfile) + return; + + rcu_read_lock(); + + tun =3D rcu_dereference(tfile->tun); + if (tun) + __tun_wake_queue(tun, tfile, consumed); + + rcu_read_unlock(); +} +EXPORT_SYMBOL_GPL(tun_wake_queue); + module_init(tun_init); module_exit(tun_cleanup); MODULE_DESCRIPTION(DRV_DESCRIPTION); diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index 80965181920c..ee583d6cc0fa 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -176,13 +176,21 @@ static void *vhost_net_buf_consume(struct vhost_net_b= uf *rxq) return ret; } =20 -static int vhost_net_buf_produce(struct vhost_net_virtqueue *nvq) +static int vhost_net_buf_produce(struct sock *sk, + struct vhost_net_virtqueue *nvq) { + struct file *file =3D sk->sk_socket->file; struct vhost_net_buf *rxq =3D &nvq->rxq; =20 rxq->head =3D 0; - rxq->tail =3D ptr_ring_consume_batched(nvq->rx_ring, rxq->queue, - VHOST_NET_BATCH); + spin_lock(&nvq->rx_ring->consumer_lock); + rxq->tail =3D __ptr_ring_consume_batched(nvq->rx_ring, rxq->queue, + VHOST_NET_BATCH); + + if (rxq->tail) + tun_wake_queue(file, rxq->tail); + + spin_unlock(&nvq->rx_ring->consumer_lock); return rxq->tail; } =20 @@ -209,14 +217,15 @@ static int vhost_net_buf_peek_len(void *ptr) return __skb_array_len_with_tag(ptr); } =20 -static int vhost_net_buf_peek(struct vhost_net_virtqueue *nvq) +static int vhost_net_buf_peek(struct sock *sk, + struct vhost_net_virtqueue *nvq) { struct vhost_net_buf *rxq =3D &nvq->rxq; =20 if (!vhost_net_buf_is_empty(rxq)) goto out; =20 - if (!vhost_net_buf_produce(nvq)) + if (!vhost_net_buf_produce(sk, nvq)) return 0; =20 out: @@ -995,7 +1004,7 @@ static int peek_head_len(struct vhost_net_virtqueue *r= vq, struct sock *sk) unsigned long flags; =20 if (rvq->rx_ring) - return vhost_net_buf_peek(rvq); + return vhost_net_buf_peek(sk, rvq); =20 spin_lock_irqsave(&sk->sk_receive_queue.lock, flags); head =3D skb_peek(&sk->sk_receive_queue); diff --git a/include/linux/if_tun.h b/include/linux/if_tun.h index 80166eb62f41..5f3e206c7a73 100644 --- a/include/linux/if_tun.h +++ b/include/linux/if_tun.h @@ -22,6 +22,7 @@ struct tun_msg_ctl { #if defined(CONFIG_TUN) || defined(CONFIG_TUN_MODULE) struct socket *tun_get_socket(struct file *); struct ptr_ring *tun_get_tx_ring(struct file *file); +void tun_wake_queue(struct file *file, int consumed); =20 static inline bool tun_is_xdp_frame(void *ptr) { @@ -55,6 +56,8 @@ static inline struct ptr_ring *tun_get_tx_ring(struct fil= e *f) return ERR_PTR(-EINVAL); } =20 +static inline void tun_wake_queue(struct file *f, int consumed) {} + static inline bool tun_is_xdp_frame(void *ptr) { return false; --=20 2.43.0 From nobody Sat Jun 13 16:24:17 2026 Received: from unimail.uni-dortmund.de (mx1.hrz.uni-dortmund.de [129.217.128.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 91AA8492523; Wed, 6 May 2026 14:11:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=129.217.128.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778076665; cv=none; b=iq3+xOq9t6dZXT8sPjHxrclIfGbmBUx1yKpC1tMz9RPbTeBgdQd1DoEuaxkrt5P5rOtGS9YueMp3UJdaAm4Irpx8REIQE/Z0VVi/gh3lyu1udtlS9H0rbMYKXPbuscoPz1ph2v9QSW94TGbW8qODkuotniVvhQfFjoUpfJUrWF0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778076665; c=relaxed/simple; bh=5Omvoy0R+ORh1eJAnCigmU7FGxPraJBDTIrpe0PSprA=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cyUGblDqNBLEYNuvsB04xckuXwZ0+vIfObUhbYZXp4ZvFqKnXCwe3R7esVUzfuXoPBrOq906GzyU0FyxFFtnp7++lzEgufbyHIEOjHXRoGk1xn86Fye/HNpKKg8DoILPQly/THr3Zs4uSQCdYjdpmP1Mh0I5TjtVBvBOHvToClo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=tu-dortmund.de; spf=pass smtp.mailfrom=tu-dortmund.de; dkim=pass (1024-bit key) header.d=tu-dortmund.de header.i=@tu-dortmund.de header.b=nAVGmD1g; arc=none smtp.client-ip=129.217.128.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=tu-dortmund.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tu-dortmund.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=tu-dortmund.de header.i=@tu-dortmund.de header.b="nAVGmD1g" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tu-dortmund.de; s=unimail; t=1778076651; bh=5Omvoy0R+ORh1eJAnCigmU7FGxPraJBDTIrpe0PSprA=; h=From:To:Subject:Date:In-Reply-To:References; b=nAVGmD1gpQJEsqWHo2sCz5IZRhGBi0WxQa3NnS7dOsFAjtCmGX0solIcnhz4sAjTL yTKMxLDltcgZbM95m0Ov3WdnTKlyF0rneZBC4JED4UeaZo6lAfYEdFyxbIbI30wgTJ CxlkrElaEZIysD+7sosBJ38gN/xV8d0oAQnlqI0E= Received: from simon-Latitude-5450.cni.e-technik.tu-dortmund.de ([129.217.186.46]) (authenticated bits=0) by unimail.uni-dortmund.de (8.19.0.1/8.19.0.1) with ESMTPSA id 646EAnWc015408 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Wed, 6 May 2026 16:10:51 +0200 (CEST) From: Simon Schippers To: willemdebruijn.kernel@gmail.com, jasowang@redhat.com, andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, mst@redhat.com, eperezma@redhat.com, leiyang@redhat.com, stephen@networkplumber.org, jon@nutanix.com, tim.gebauer@tu-dortmund.de, simon.schippers@tu-dortmund.de, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux.dev Subject: [PATCH net-next v10 3/4] ptr_ring: move free-space check into separate helper Date: Wed, 6 May 2026 16:10:32 +0200 Message-ID: <20260506141033.180450-4-simon.schippers@tu-dortmund.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260506141033.180450-1-simon.schippers@tu-dortmund.de> References: <20260506141033.180450-1-simon.schippers@tu-dortmund.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This patch moves the check for available free space for a new entry into a separate function. As a result, __ptr_ring_produce() remains logically unchanged, while the new helper allows callers to determine in advance whether subsequent __ptr_ring_produce() calls will succeed. This information can, for example, be used to temporarily stop producing until __ptr_ring_produce_peek() indicates that space is available again. Co-developed-by: Tim Gebauer Signed-off-by: Tim Gebauer Signed-off-by: Simon Schippers --- include/linux/ptr_ring.h | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/include/linux/ptr_ring.h b/include/linux/ptr_ring.h index d2c3629bbe45..0887284e5b43 100644 --- a/include/linux/ptr_ring.h +++ b/include/linux/ptr_ring.h @@ -96,6 +96,17 @@ static inline bool ptr_ring_full_bh(struct ptr_ring *r) return ret; } =20 +/* Note: callers invoking this in a loop must use a compiler barrier, + * for example cpu_relax(). Callers must hold producer_lock. + */ +static inline int __ptr_ring_produce_peek(struct ptr_ring *r) +{ + if (unlikely(!r->size) || data_race(r->queue[r->producer])) + return -ENOSPC; + + return 0; +} + /* Note: callers invoking this in a loop must use a compiler barrier, * for example cpu_relax(). Callers must hold producer_lock. * Callers are responsible for making sure pointer that is being queued @@ -103,8 +114,10 @@ static inline bool ptr_ring_full_bh(struct ptr_ring *r) */ static inline int __ptr_ring_produce(struct ptr_ring *r, void *ptr) { - if (unlikely(!r->size) || data_race(r->queue[r->producer])) - return -ENOSPC; + int p =3D __ptr_ring_produce_peek(r); + + if (p) + return p; =20 /* Make sure the pointer we are storing points to a valid data. */ /* Pairs with the dependency ordering in __ptr_ring_consume. */ --=20 2.43.0 From nobody Sat Jun 13 16:24:17 2026 Received: from unimail.uni-dortmund.de (mx1.hrz.uni-dortmund.de [129.217.128.51]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 53690492510; Wed, 6 May 2026 14:11:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=129.217.128.51 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778076666; cv=none; b=NXUltGrJ1XE2BspuMuHecVlfIGs7aZVcL5xTdMmX4QqfqFUWt3Eg4u3mud9ccp4HzqUGoTSMlG20DZtRPTsFVvCVT+4QH3b+3d73P6m6tTuDAdyO6KdCxLuhuF/JTa6G9jmX5RgA+ocgJ2R15oES24Z4YKsRvwLOjyyxjwZApbk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1778076666; c=relaxed/simple; bh=4ebU+CzqnDYV0UkRyUU+znmrPZfYugt+GMlID6BAu2c=; h=From:To:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SGq/wUKIwdOvXy5xN8rpBfVohLMII6MHf0c8UQqPRMSD62oijmPC9S+qXlItGIf94jXhzJtiGAs/IePeVZmMMnDTW6kH3hXeMXxEyzJxYJRs+4RWNv8qDXhJr5nYpVpcRRaJkZFUAwG1QzhpKxRhKCEcDK8Uz3YI/e+foX0IB80= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=tu-dortmund.de; spf=pass smtp.mailfrom=tu-dortmund.de; dkim=pass (1024-bit key) header.d=tu-dortmund.de header.i=@tu-dortmund.de header.b=BnIouqo9; arc=none smtp.client-ip=129.217.128.51 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=tu-dortmund.de Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=tu-dortmund.de Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=tu-dortmund.de header.i=@tu-dortmund.de header.b="BnIouqo9" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tu-dortmund.de; s=unimail; t=1778076651; bh=4ebU+CzqnDYV0UkRyUU+znmrPZfYugt+GMlID6BAu2c=; h=From:To:Subject:Date:In-Reply-To:References; b=BnIouqo9aw7E1ZEEos6exl6iXUN5iKqDZyZXkhLsT2Eb3KUvekuftMwDwN+dRnop2 eP6wJu07vliZ5TKH973DkXkpzQbD8ozDexKg+nJNB52A6B4sHmzM5MuJsNEGjkLCGP OP4HuNWjSJUI8tp6n4cEYckwGPwmdK1CzlxJ0No0= Received: from simon-Latitude-5450.cni.e-technik.tu-dortmund.de ([129.217.186.46]) (authenticated bits=0) by unimail.uni-dortmund.de (8.19.0.1/8.19.0.1) with ESMTPSA id 646EAnWe015408 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NOT); Wed, 6 May 2026 16:10:51 +0200 (CEST) From: Simon Schippers To: willemdebruijn.kernel@gmail.com, jasowang@redhat.com, andrew+netdev@lunn.ch, davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, mst@redhat.com, eperezma@redhat.com, leiyang@redhat.com, stephen@networkplumber.org, jon@nutanix.com, tim.gebauer@tu-dortmund.de, simon.schippers@tu-dortmund.de, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux.dev Subject: [PATCH net-next v10 4/4] tun/tap & vhost-net: avoid ptr_ring tail-drop when a qdisc is present Date: Wed, 6 May 2026 16:10:33 +0200 Message-ID: <20260506141033.180450-5-simon.schippers@tu-dortmund.de> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20260506141033.180450-1-simon.schippers@tu-dortmund.de> References: <20260506141033.180450-1-simon.schippers@tu-dortmund.de> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" This commit prevents tail-drop when a qdisc is present and the ptr_ring becomes full. Once an entry is successfully produced and the ptr_ring reaches capacity, the netdev queue is stopped instead of dropping subsequent packets. If no qdisc is present, the previous tail-drop behavior is preserved. If producing an entry fails anyways due to a race, tun_net_xmit() drops the packet. Such races are expected because LLTX is enabled and the transmit path operates without the usual locking. The __tun_wake_queue() function of the consumer races with the producer for waking/stopping the netdev queue, which could result in a stalled queue. Therefore, an smp_mb__after_atomic() is introduced that pairs with the smp_mb() of the consumer. It follows the principle of store buffering described in tools/memory-model/Documentation/recipes.txt: - The producer in tun_net_xmit() first sets __QUEUE_STATE_DRV_XOFF, followed by an smp_mb__after_atomic() (=3D smp_mb()), and then reads the ring with __ptr_ring_produce_peek(). - The consumer in __tun_wake_queue() first writes zero to the ring in __ptr_ring_consume(), followed by an smp_mb(), and then reads the queue status with netif_tx_queue_stopped(). =3D> Following the aforementioned principle, it is impossible for the producer to see a full ring (and therefore not wake the queue on the re-check) while the consumer simultaneously fails to see a stopped queue (and therefore also does not wake it). Benchmarks: The benchmarks show a slight regression in raw transmission performance when using two sending threads. Packet loss also occurs only in the two-thread sending case; no packet loss was observed with a single sending thread. Test setup: AMD Ryzen 5 5600X at 4.3 GHz, 3200 MHz RAM, isolated QEMU threads; Average over 50 runs @ 100,000,000 packets. SRSO and spectre v2 mitigations disabled. Note for tap+vhost-net: XDP drop program active in VM -> ~2.5x faster; slower for tap due to more syscalls (high utilization of entry_SYSRETQ_unsafe_stack in perf) +--------------------------+--------------+----------------+----------+ | 1 thread | Stock | Patched with | diff | | sending | | fq_codel qdisc | | +------------+-------------+--------------+----------------+----------+ | TAP | Received | 1.132 Mpps | 1.133 Mpps | +0.1% | | +-------------+--------------+----------------+----------+ | | Lost/s | 3.765 Mpps | 0 pps | | +------------+-------------+--------------+----------------+----------+ | TAP | Received | 3.857 Mpps | 3.905 Mpps | +1.2% | | +-------------+--------------+----------------+----------+ | +vhost-net | Lost/s | 0.802 Mpps | 0 pps | | +------------+-------------+--------------+----------------+----------+ +--------------------------+--------------+----------------+----------+ | 2 threads | Stock | Patched with | diff | | sending | | fq_codel qdisc | | +------------+-------------+--------------+----------------+----------+ | TAP | Received | 1.115 Mpps | 1.092 Mpps | -2.1% | | +-------------+--------------+----------------+----------+ | | Lost/s | 8.490 Mpps | 359 pps | | +------------+-------------+--------------+----------------+----------+ | TAP | Received | 3.664 Mpps | 3.549 Mpps | -3.1% | | +-------------+--------------+----------------+----------+ | +vhost-net | Lost/s | 5.330 Mpps | 832 pps | | +------------+-------------+--------------+----------------+----------+ Co-developed-by: Tim Gebauer Signed-off-by: Tim Gebauer Signed-off-by: Simon Schippers --- drivers/net/tun.c | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/drivers/net/tun.c b/drivers/net/tun.c index fc358c4c355b..d9ffbf88cfd8 100644 --- a/drivers/net/tun.c +++ b/drivers/net/tun.c @@ -1018,6 +1018,7 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb, = struct net_device *dev) struct netdev_queue *queue; struct tun_file *tfile; int len =3D skb->len; + int ret; =20 rcu_read_lock(); tfile =3D rcu_dereference(tun->tfiles[txq]); @@ -1072,13 +1073,33 @@ static netdev_tx_t tun_net_xmit(struct sk_buff *skb= , struct net_device *dev) =20 nf_reset_ct(skb); =20 - if (ptr_ring_produce(&tfile->tx_ring, skb)) { + queue =3D netdev_get_tx_queue(dev, txq); + + spin_lock(&tfile->tx_ring.producer_lock); + ret =3D __ptr_ring_produce(&tfile->tx_ring, skb); + if (!qdisc_txq_has_no_queue(queue) && + (__ptr_ring_produce_peek(&tfile->tx_ring) || ret)) { + netif_tx_stop_queue(queue); + /* Paired with smp_mb() in __tun_wake_queue() */ + smp_mb__after_atomic(); + if (!__ptr_ring_produce_peek(&tfile->tx_ring)) + netif_tx_wake_queue(queue); + } + spin_unlock(&tfile->tx_ring.producer_lock); + + if (ret) { + /* This should be a rare case if a qdisc is present, but + * can happen due to lltx. + * Since skb_tx_timestamp(), skb_orphan(), + * run_ebpf_filter() and pskb_trim() could have tinkered + * with the SKB, returning NETDEV_TX_BUSY is unsafe and + * we must drop instead. + */ drop_reason =3D SKB_DROP_REASON_FULL_RING; goto drop; } =20 /* dev->lltx requires to do our own update of trans_start */ - queue =3D netdev_get_tx_queue(dev, txq); txq_trans_cond_update(queue); =20 /* Notify and wake up reader process */ --=20 2.43.0