drivers/net/ethernet/google/gve/gve_tx.c | 12 +++++++++--- drivers/net/ethernet/google/gve/gve_tx_dqo.c | 9 ++++++++- 2 files changed, 17 insertions(+), 4 deletions(-)
From: Ankit Garg <nktgrg@google.com> This series fixes a kernel panic in the GVE driver caused by out-of-bounds array access when the network stack provides an invalid TX queue index. The issue impacts both GQI and DQO queue formats. For both cases, the driver is updated to validate the queue index and drop the packet if the index is out of range. Ankit Garg (2): gve: drop packets on invalid queue indices in GQI TX path gve: drop packets on invalid queue indices in DQO TX path drivers/net/ethernet/google/gve/gve_tx.c | 12 +++++++++--- drivers/net/ethernet/google/gve/gve_tx_dqo.c | 9 ++++++++- 2 files changed, 17 insertions(+), 4 deletions(-) -- 2.52.0.351.gbe84eed79e-goog
On Mon, 5 Jan 2026 15:25:02 -0800 Joshua Washington wrote: > This series fixes a kernel panic in the GVE driver caused by > out-of-bounds array access when the network stack provides an invalid > TX queue index. Do you know how? I seem to recall we had such issues due to bugs in the qdisc layer, most of which were fixed. Fixing this at the source, if possible, would be far preferable to sprinkling this condition to all the drivers.
On Tue, Jan 6, 2026 at 6:22 PM Jakub Kicinski <kuba@kernel.org> wrote: > > On Mon, 5 Jan 2026 15:25:02 -0800 Joshua Washington wrote: > > This series fixes a kernel panic in the GVE driver caused by > > out-of-bounds array access when the network stack provides an invalid > > TX queue index. > > Do you know how? I seem to recall we had such issues due to bugs > in the qdisc layer, most of which were fixed. > > Fixing this at the source, if possible, would be far preferable > to sprinkling this condition to all the drivers. That matches our observation—we have encountered this panic on older kernels (specifically Rocky Linux 8) but have not been able to reproduce it on recent upstream kernels. Could you point us to the specific qdisc fixes you recall? We'd like to verify if the issue we are seeing on the older kernel is indeed one of those known/fixed bugs. If it turns out this is fully resolved in the core network stack upstream, we can drop this patch for the mainline driver. However, if there is ambiguity, do you think there is value in keeping this check to prevent the driver from crashing on invalid input? Thanks, Ankit Garg
On Thu, Jan 8, 2026 at 4:36 PM Ankit Garg <nktgrg@google.com> wrote: > > On Tue, Jan 6, 2026 at 6:22 PM Jakub Kicinski <kuba@kernel.org> wrote: > > > > On Mon, 5 Jan 2026 15:25:02 -0800 Joshua Washington wrote: > > > This series fixes a kernel panic in the GVE driver caused by > > > out-of-bounds array access when the network stack provides an invalid > > > TX queue index. > > > > Do you know how? I seem to recall we had such issues due to bugs > > in the qdisc layer, most of which were fixed. > > > > Fixing this at the source, if possible, would be far preferable > > to sprinkling this condition to all the drivers. > That matches our observation—we have encountered this panic on older > kernels (specifically Rocky Linux 8) but have not been able to > reproduce it on recent upstream kernels. What is the kernel version used in Rocky Linux 8 ? Note that the test against real_num_tx_queues is done before reaching the Qdisc layer. It might help to give a stack trace of a panic. > > Could you point us to the specific qdisc fixes you recall? We'd like > to verify if the issue we are seeing on the older kernel is indeed one > of those known/fixed bugs. > > If it turns out this is fully resolved in the core network stack > upstream, we can drop this patch for the mainline driver. However, if > there is ambiguity, do you think there is value in keeping this check > to prevent the driver from crashing on invalid input? We already have many costly checks, and netdev_core_pick_tx() should already prevent such panic. > > Thanks, > Ankit Garg
On Thu, Jan 8, 2026 at 8:37 AM Eric Dumazet <edumazet@google.com> wrote: > > On Thu, Jan 8, 2026 at 4:36 PM Ankit Garg <nktgrg@google.com> wrote: > > > > On Tue, Jan 6, 2026 at 6:22 PM Jakub Kicinski <kuba@kernel.org> wrote: > > > > > > On Mon, 5 Jan 2026 15:25:02 -0800 Joshua Washington wrote: > > > > This series fixes a kernel panic in the GVE driver caused by > > > > out-of-bounds array access when the network stack provides an invalid > > > > TX queue index. > > > > > > Do you know how? I seem to recall we had such issues due to bugs > > > in the qdisc layer, most of which were fixed. > > > > > > Fixing this at the source, if possible, would be far preferable > > > to sprinkling this condition to all the drivers. > > That matches our observation—we have encountered this panic on older > > kernels (specifically Rocky Linux 8) but have not been able to > > reproduce it on recent upstream kernels. > > What is the kernel version used in Rocky Linux 8 ? > The kernel version where we observed this is 4.18.0 (full version 4.18.0-553.81.1+2.1.el8_10_ciq) > Note that the test against real_num_tx_queues is done before reaching > the Qdisc layer. > > It might help to give a stack trace of a panic. > Crash happens in the sch_direct_xmit path per the trace. I wonder if sch_direct_xmit is acting as an optimization to bypass the queueing layer, and if that is somehow bypassing the queue index checks you mentioned? I'll try to dig a bit deeper into that specific flow, but here is the trace in the meantime: Call Trace: ? __warn+0x94/0xe0 ? gve_tx+0xa9f/0xc30 [gve] ? gve_tx+0xa9f/0xc30 [gve] ? report_bug+0xb1/0xe0 ? do_error_trap+0x9e/0xd0 ? do_invalid_op+0x36/0x40 ? gve_tx+0xa9f/0xc30 [gve] ? invalid_op+0x14/0x20 ? gve_tx+0xa9f/0xc30 [gve] ? netif_skb_features+0xcf/0x2a0 dev_hard_start_xmit+0xd7/0x240 sch_direct_xmit+0x9f/0x370 __dev_queue_xmit+0xa04/0xc50 ip_finish_output2+0x26d/0x430 ? __ip_finish_output+0xdf/0x1d0 ip_output+0x70/0xf0 __ip_queue_xmit+0x165/0x400 __tcp_transmit_skb+0xa6b/0xb90 tcp_connect+0xae3/0xd40 tcp_v4_connect+0x476/0x4f0 __inet_stream_connect+0xda/0x380 > > > > Could you point us to the specific qdisc fixes you recall? We'd like > > to verify if the issue we are seeing on the older kernel is indeed one > > of those known/fixed bugs. > > > > If it turns out this is fully resolved in the core network stack > > upstream, we can drop this patch for the mainline driver. However, if > > there is ambiguity, do you think there is value in keeping this check > > to prevent the driver from crashing on invalid input? > > We already have many costly checks, and netdev_core_pick_tx() should > already prevent such panic. > > > > > Thanks, > > Ankit Garg
On Thu, Jan 8, 2026 at 9:53 PM Ankit Garg <nktgrg@google.com> wrote:
>
> On Thu, Jan 8, 2026 at 8:37 AM Eric Dumazet <edumazet@google.com> wrote:
> >
> > On Thu, Jan 8, 2026 at 4:36 PM Ankit Garg <nktgrg@google.com> wrote:
> > >
> > > On Tue, Jan 6, 2026 at 6:22 PM Jakub Kicinski <kuba@kernel.org> wrote:
> > > >
> > > > On Mon, 5 Jan 2026 15:25:02 -0800 Joshua Washington wrote:
> > > > > This series fixes a kernel panic in the GVE driver caused by
> > > > > out-of-bounds array access when the network stack provides an invalid
> > > > > TX queue index.
> > > >
> > > > Do you know how? I seem to recall we had such issues due to bugs
> > > > in the qdisc layer, most of which were fixed.
> > > >
> > > > Fixing this at the source, if possible, would be far preferable
> > > > to sprinkling this condition to all the drivers.
> > > That matches our observation—we have encountered this panic on older
> > > kernels (specifically Rocky Linux 8) but have not been able to
> > > reproduce it on recent upstream kernels.
> >
> > What is the kernel version used in Rocky Linux 8 ?
> >
> The kernel version where we observed this is 4.18.0 (full version
> 4.18.0-553.81.1+2.1.el8_10_ciq)
>
> > Note that the test against real_num_tx_queues is done before reaching
> > the Qdisc layer.
> >
> > It might help to give a stack trace of a panic.
> >
> Crash happens in the sch_direct_xmit path per the trace.
>
> I wonder if sch_direct_xmit is acting as an optimization to bypass the
> queueing layer, and if that is somehow bypassing the queue index
> checks you mentioned?
>
> I'll try to dig a bit deeper into that specific flow, but here is the
> trace in the meantime:
Jakub, the issue is that before 4.20, calling synchronize_rcu()
instead of synchronize_rcu_bh()
was probably a bug. I suspect we had more issues like that.
__dev_queue_xmit takes a rcu_read_lock_bh(), while the code (that you
added in 2018 [1])
to update the queue netif_set_real_num_tx_queues does synchronize_net()
(aka synchronize_rcu()) and in earlier times, it would mean that this
would maybe return too soon (say on preemptible kernels)
[1] commit ac5b70198adc25c73fba28de4f78adcee8f6be0b
Author: Jakub Kicinski <jakub.kicinski@netronome.com>
Date: Mon Feb 12 21:35:31 2018 -0800
net: fix race on decreasing number of TX queues
netif_set_real_num_tx_queues() can be called when netdev is up.
That usually happens when user requests change of number of
channels/rings with ethtool -L. The procedure for changing
the number of queues involves resetting the qdiscs and setting
dev->num_tx_queues to the new value. When the new value is
lower than the old one, extra care has to be taken to ensure
ordering of accesses to the number of queues vs qdisc reset.
Currently the queues are reset before new dev->num_tx_queues
is assigned, leaving a window of time where packets can be
enqueued onto the queues going down, leading to a likely
crash in the drivers, since most drivers don't check if TX
skbs are assigned to an active queue.
Fixes: e6484930d7c7 ("net: allocate tx queues in register_netdevice")
Signed-off-by: Jakub Kicinski <jakub.kicinski@netronome.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
So perhaps a fix for pre 4.20 kernel would be: (I kept the
synchronize_net() to be really cautious and because I really do not
want to test)
diff --git a/net/core/dev.c b/net/core/dev.c
index 93243479085fb1d61031ed2136f5aee22d8f313d..4dd1db70561d35fe2097afc86764dd82bfd0bf27
100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -2636,6 +2636,7 @@ int netif_set_real_num_tx_queues(struct
net_device *dev, unsigned int txq)
if (disabling) {
synchronize_net();
+ synchronize_rcu_bh();
qdisc_reset_all_tx_gt(dev, txq);
#ifdef CONFIG_XPS
netif_reset_xps_queues_gt(dev, txq);
On Fri, 30 Jan 2026 21:56:07 +0100 Eric Dumazet wrote: > Jakub, the issue is that before 4.20, calling synchronize_rcu() > instead of synchronize_rcu_bh() > was probably a bug. I suspect we had more issues like that. > > __dev_queue_xmit takes a rcu_read_lock_bh(), while the code (that you > added in 2018 [1]) > to update the queue netif_set_real_num_tx_queues does synchronize_net() > (aka synchronize_rcu()) and in earlier times, it would mean that this > would maybe return too soon (say on preemptible kernels) > > [...] > > So perhaps a fix for pre 4.20 kernel would be: (I kept the > synchronize_net() to be really cautious and because I really do not > want to test) Sounds entirely plausible, FWIW. Ankit, this would mean that you have to convince RHEL / Rocky to take Eric's patch. Oldest kernel we can patch upstream is 5.10, AFAIK.
On Thu, 8 Jan 2026 07:35:59 -0800 Ankit Garg wrote: > On Tue, Jan 6, 2026 at 6:22 PM Jakub Kicinski <kuba@kernel.org> wrote: > > On Mon, 5 Jan 2026 15:25:02 -0800 Joshua Washington wrote: > > > This series fixes a kernel panic in the GVE driver caused by > > > out-of-bounds array access when the network stack provides an invalid > > > TX queue index. > > > > Do you know how? I seem to recall we had such issues due to bugs > > in the qdisc layer, most of which were fixed. > > > > Fixing this at the source, if possible, would be far preferable > > to sprinkling this condition to all the drivers. > > That matches our observation—we have encountered this panic on older > kernels (specifically Rocky Linux 8) but have not been able to > reproduce it on recent upstream kernels. > > Could you point us to the specific qdisc fixes you recall? We'd like > to verify if the issue we are seeing on the older kernel is indeed one > of those known/fixed bugs. Very old - ac5b70198adc25 > If it turns out this is fully resolved in the core network stack > upstream, we can drop this patch for the mainline driver. However, if > there is ambiguity, do you think there is value in keeping this check > to prevent the driver from crashing on invalid input? The API contract is that the stack does not send frames for queues which don't exist (> real_num_tx_queues) down to the drivers. There's no ambiguity, IMO, if the stack sends such frames its a bug in the stack.
© 2016 - 2026 Red Hat, Inc.