Commit 94833addfaba ("net: thunderx: Unembed netdev structure") had
a go at dynamically allocating the netdev structures for the thunderx_bgx
driver.  This change results in my ThunderX box catching fire (to be fair,
it is what it does best).

The issues with this change are that:

- bgx_lmac_enable() is called *after* bgx_acpi_register_phy() and
  bgx_init_of_phy(), both expecting netdev to be a valid pointer.

- bgx_init_of_phy() populates the MAC addresses for *all* LMACs
  attached to a given BGX instance, and thus needs netdev for each of
  them to have been allocated.

There is a few things to be said about how the driver mixes LMAC and
BGX states which leads to this sorry state, but that's beside the point.

To address this, go back to a situation where all netdev structures
are allocated before the driver starts relying on them, and move the
freeing of these structures to driver removal. Someone brave enough
can always go and restructure the driver if they want.

Fixes: 94833addfaba ("net: thunderx: Unembed netdev structure")
Signed-off-by: Marc Zyngier <maz@kernel.org>
Cc: Breno Leitao <leitao@debian.org>
Cc: Sunil Goutham <sgoutham@marvell.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Jakub Kicinski <kuba@kernel.org>
Cc: Paolo Abeni <pabeni@redhat.com>
---
 .../net/ethernet/cavium/thunder/thunder_bgx.c | 30 +++++++++++++------
 1 file changed, 21 insertions(+), 9 deletions(-)

diff --git a/drivers/net/ethernet/cavium/thunder/thunder_bgx.c b/drivers/net/ethernet/cavium/thunder/thunder_bgx.c
index a40c266c37f2..608cc6af5af1 100644
--- a/drivers/net/ethernet/cavium/thunder/thunder_bgx.c
+++ b/drivers/net/ethernet/cavium/thunder/thunder_bgx.c
@@ -1054,18 +1054,12 @@ static int phy_interface_mode(u8 lmac_type)
 
 static int bgx_lmac_enable(struct bgx *bgx, u8 lmacid)
 {
-	struct lmac *lmac, **priv;
+	struct lmac *lmac;
 	u64 cfg;
 
 	lmac = &bgx->lmac[lmacid];
 	lmac->bgx = bgx;
 
-	lmac->netdev = alloc_netdev_dummy(sizeof(struct lmac *));
-	if (!lmac->netdev)
-		return -ENOMEM;
-	priv = netdev_priv(lmac->netdev);
-	*priv = lmac;
-
 	if ((lmac->lmac_type == BGX_MODE_SGMII) ||
 	    (lmac->lmac_type == BGX_MODE_QSGMII) ||
 	    (lmac->lmac_type == BGX_MODE_RGMII)) {
@@ -1191,7 +1185,6 @@ static void bgx_lmac_disable(struct bgx *bgx, u8 lmacid)
 	    (lmac->lmac_type != BGX_MODE_10G_KR) && lmac->phydev)
 		phy_disconnect(lmac->phydev);
 
-	free_netdev(lmac->netdev);
 	lmac->phydev = NULL;
 }
 
@@ -1653,6 +1646,23 @@ static int bgx_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
 
 	bgx_get_qlm_mode(bgx);
 
+	for (lmac = 0; lmac < bgx->lmac_count; lmac++) {
+		struct lmac *lmacp, **priv;
+
+		lmacp = &bgx->lmac[lmac];
+		lmacp->netdev = alloc_netdev_dummy(sizeof(struct lmac *));
+
+		if (!lmacp->netdev) {
+			for (int i = 0; i < lmac; i++)
+				free_netdev(bgx->lmac[i].netdev);
+			err = -ENOMEM;
+			goto err_enable;
+		}
+
+		priv = netdev_priv(lmacp->netdev);
+		*priv = lmacp;
+	}
+
 	err = bgx_init_phy(bgx);
 	if (err)
 		goto err_enable;
@@ -1692,8 +1702,10 @@ static void bgx_remove(struct pci_dev *pdev)
 	u8 lmac;
 
 	/* Disable all LMACs */
-	for (lmac = 0; lmac < bgx->lmac_count; lmac++)
+	for (lmac = 0; lmac < bgx->lmac_count; lmac++) {
 		bgx_lmac_disable(bgx, lmac);
+		free_netdev(bgx->lmac[lmac].netdev);
+	}
 
 	pci_free_irq(pdev, GMPX_GMI_TX_INT, bgx);
 
-- 
2.39.2

Hello Marc,

On Mon, Aug 12, 2024 at 03:13:22PM +0100, Marc Zyngier wrote:
> Commit 94833addfaba ("net: thunderx: Unembed netdev structure") had
> a go at dynamically allocating the netdev structures for the thunderx_bgx
> driver.  This change results in my ThunderX box catching fire (to be fair,
> it is what it does best).

Should I be proud of it? :-)

> The issues with this change are that:
> 
> - bgx_lmac_enable() is called *after* bgx_acpi_register_phy() and
>   bgx_init_of_phy(), both expecting netdev to be a valid pointer.
> 
> - bgx_init_of_phy() populates the MAC addresses for *all* LMACs
>   attached to a given BGX instance, and thus needs netdev for each of
>   them to have been allocated.
> 
> There is a few things to be said about how the driver mixes LMAC and
> BGX states which leads to this sorry state, but that's beside the point.
> 
> To address this, go back to a situation where all netdev structures
> are allocated before the driver starts relying on them, and move the
> freeing of these structures to driver removal. Someone brave enough
> can always go and restructure the driver if they want.
> 
> Fixes: 94833addfaba ("net: thunderx: Unembed netdev structure")
> Signed-off-by: Marc Zyngier <maz@kernel.org>

Reviewed-by: Breno Leitao <leitao@debian.org>

Thanks for taming my fiery commit.

On Tue, 13 Aug 2024 17:25:27 +0100,
Breno Leitao <leitao@debian.org> wrote:
> 
> Hello Marc,
> 
> On Mon, Aug 12, 2024 at 03:13:22PM +0100, Marc Zyngier wrote:
> > Commit 94833addfaba ("net: thunderx: Unembed netdev structure") had
> > a go at dynamically allocating the netdev structures for the thunderx_bgx
> > driver.  This change results in my ThunderX box catching fire (to be fair,
> > it is what it does best).
> 
> Should I be proud of it? :-)

It's always good practice to check that someone still cares about
terrible HW. Break it, wait for a few releases, and purge it if nobody
was looking.

Unfortunately, this is one of the few machines I have that has 16kB
page support, so I can't really turn a blind eye on the
breakage... ;-)

> 
> > The issues with this change are that:
> > 
> > - bgx_lmac_enable() is called *after* bgx_acpi_register_phy() and
> >   bgx_init_of_phy(), both expecting netdev to be a valid pointer.
> > 
> > - bgx_init_of_phy() populates the MAC addresses for *all* LMACs
> >   attached to a given BGX instance, and thus needs netdev for each of
> >   them to have been allocated.
> > 
> > There is a few things to be said about how the driver mixes LMAC and
> > BGX states which leads to this sorry state, but that's beside the point.
> > 
> > To address this, go back to a situation where all netdev structures
> > are allocated before the driver starts relying on them, and move the
> > freeing of these structures to driver removal. Someone brave enough
> > can always go and restructure the driver if they want.
> > 
> > Fixes: 94833addfaba ("net: thunderx: Unembed netdev structure")
> > Signed-off-by: Marc Zyngier <maz@kernel.org>
> 
> Reviewed-by: Breno Leitao <leitao@debian.org>
> 
> Thanks for taming my fiery commit.

No worries.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

On Mon, Aug 12, 2024 at 03:13:22PM +0100, Marc Zyngier wrote:
> Commit 94833addfaba ("net: thunderx: Unembed netdev structure") had
> a go at dynamically allocating the netdev structures for the thunderx_bgx
> driver.  This change results in my ThunderX box catching fire (to be fair,
> it is what it does best).

(I saw that :)

> The issues with this change are that:
> 
> - bgx_lmac_enable() is called *after* bgx_acpi_register_phy() and
>   bgx_init_of_phy(), both expecting netdev to be a valid pointer.
> 
> - bgx_init_of_phy() populates the MAC addresses for *all* LMACs
>   attached to a given BGX instance, and thus needs netdev for each of
>   them to have been allocated.
> 
> There is a few things to be said about how the driver mixes LMAC and
> BGX states which leads to this sorry state, but that's beside the point.
> 
> To address this, go back to a situation where all netdev structures
> are allocated before the driver starts relying on them, and move the
> freeing of these structures to driver removal. Someone brave enough
> can always go and restructure the driver if they want.
> 
> Fixes: 94833addfaba ("net: thunderx: Unembed netdev structure")
> Signed-off-by: Marc Zyngier <maz@kernel.org>

Reviewed-by: Simon Horman <horms@kernel.org>