Assume 'usxgmii' being set as initial interface mode in DTS. Now plug
a 2.5GBase-T SFP module with exposed PHY. Currently this results in
a rather bizare situation:

RTL8221B-VB-CG 2.5Gbps PHY (C45) i2c:sfp1-wan:11: rtl822x_c45_get_features: supported=00,00000000,00008000,000080ef
mtk_soc_eth 15100000.ethernet eth2: requesting link mode phy/2500base-x with support 00,00000000,00008000,0000e0ef
mtk_soc_eth 15100000.ethernet eth2: switched to phy/2500base-x link mode   <<< !!!!!!
mtk_soc_eth 15100000.ethernet eth2: major config usxgmii    <<< !!!!!!
mtk_soc_eth 15100000.ethernet eth2: phylink_mac_config: mode=phy/usxgmii/none adv=00,00000000,00000000,00000000 pause=00
mtk_soc_eth 15100000.ethernet eth2: PHY [i2c:sfp1-wan:11] driver [RTL8221B-VB-CG 2.5Gbps PHY (C45)] (irq=POLL)
mtk_soc_eth 15100000.ethernet eth2: phy: 2500base-x setting supported 00,00000000,00008000,0000e0ef advertising 00,00000000,00008000,0000e0ef

Then the link seemingly comes up (but is dead) because no subsequent
call to phylink_major_config actually configured MAC and PCS for
2500base-x mode.

This is because phylink_mac_initial_config() considers
pl->phy_state.interface if in MLO_AN_PHY mode while
phylink_sfp_set_config() only sets pl->link_config.interface.

Also set pl->phy_state.interface in phylink_sfp_set_config().

Fixes: 31eb8907aa5b ("net: phylink: allow attaching phy for SFP modules on 802.3z mode")
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
---
 drivers/net/phy/phylink.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/net/phy/phylink.c b/drivers/net/phy/phylink.c
index aa842634d5c2c..21ef77e4fe720 100644
--- a/drivers/net/phy/phylink.c
+++ b/drivers/net/phy/phylink.c
@@ -3056,6 +3056,7 @@ static void phylink_sfp_set_config(struct phylink *pl, u8 mode,
 	    pl->link_config.interface != state->interface) {
 		pl->cur_link_an_mode = mode;
 		pl->link_config.interface = state->interface;
+		pl->phy_state.interface = state->interface;
 
 		changed = true;
 
-- 
2.42.1

On Sat, Nov 25, 2023 at 04:56:20AM +0000, Daniel Golle wrote:
> Assume 'usxgmii' being set as initial interface mode in DTS. Now plug
> a 2.5GBase-T SFP module with exposed PHY. Currently this results in
> a rather bizare situation:
> 
> RTL8221B-VB-CG 2.5Gbps PHY (C45) i2c:sfp1-wan:11: rtl822x_c45_get_features: supported=00,00000000,00008000,000080ef
> mtk_soc_eth 15100000.ethernet eth2: requesting link mode phy/2500base-x with support 00,00000000,00008000,0000e0ef
> mtk_soc_eth 15100000.ethernet eth2: switched to phy/2500base-x link mode   <<< !!!!!!
> mtk_soc_eth 15100000.ethernet eth2: major config usxgmii    <<< !!!!!!
> mtk_soc_eth 15100000.ethernet eth2: phylink_mac_config: mode=phy/usxgmii/none adv=00,00000000,00000000,00000000 pause=00
> mtk_soc_eth 15100000.ethernet eth2: PHY [i2c:sfp1-wan:11] driver [RTL8221B-VB-CG 2.5Gbps PHY (C45)] (irq=POLL)
> mtk_soc_eth 15100000.ethernet eth2: phy: 2500base-x setting supported 00,00000000,00008000,0000e0ef advertising 00,00000000,00008000,0000e0ef
> 
> Then the link seemingly comes up (but is dead) because no subsequent
> call to phylink_major_config actually configured MAC and PCS for
> 2500base-x mode.
> 
> This is because phylink_mac_initial_config() considers
> pl->phy_state.interface if in MLO_AN_PHY mode while
> phylink_sfp_set_config() only sets pl->link_config.interface.
> 
> Also set pl->phy_state.interface in phylink_sfp_set_config().

Does it _actually_ matter?

When the PHY's link comes up, doesn't it get sorted out for the real
mode that will be used?

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!

On Sat, Nov 25, 2023 at 08:28:20AM +0000, Russell King (Oracle) wrote:
> On Sat, Nov 25, 2023 at 04:56:20AM +0000, Daniel Golle wrote:
> > Assume 'usxgmii' being set as initial interface mode in DTS. Now plug
> > a 2.5GBase-T SFP module with exposed PHY. Currently this results in
> > a rather bizare situation:
> > 
> > RTL8221B-VB-CG 2.5Gbps PHY (C45) i2c:sfp1-wan:11: rtl822x_c45_get_features: supported=00,00000000,00008000,000080ef
> > mtk_soc_eth 15100000.ethernet eth2: requesting link mode phy/2500base-x with support 00,00000000,00008000,0000e0ef
> > mtk_soc_eth 15100000.ethernet eth2: switched to phy/2500base-x link mode   <<< !!!!!!
> > mtk_soc_eth 15100000.ethernet eth2: major config usxgmii    <<< !!!!!!
> > mtk_soc_eth 15100000.ethernet eth2: phylink_mac_config: mode=phy/usxgmii/none adv=00,00000000,00000000,00000000 pause=00
> > mtk_soc_eth 15100000.ethernet eth2: PHY [i2c:sfp1-wan:11] driver [RTL8221B-VB-CG 2.5Gbps PHY (C45)] (irq=POLL)
> > mtk_soc_eth 15100000.ethernet eth2: phy: 2500base-x setting supported 00,00000000,00008000,0000e0ef advertising 00,00000000,00008000,0000e0ef
> > 
> > Then the link seemingly comes up (but is dead) because no subsequent
> > call to phylink_major_config actually configured MAC and PCS for
> > 2500base-x mode.
> > 
> > This is because phylink_mac_initial_config() considers
> > pl->phy_state.interface if in MLO_AN_PHY mode while
> > phylink_sfp_set_config() only sets pl->link_config.interface.
> > 
> > Also set pl->phy_state.interface in phylink_sfp_set_config().
> 
> Does it _actually_ matter?

It does matter, I'm suggesting this change because the SFP module
won't work without it.

> When the PHY's link comes up, doesn't it get sorted out for the real
> mode that will be used?

That does happen once the interface mode *changes* to anything else,
than 2500M/Full and the PHY driver sets state->interface to SGMII.
However, initially it does *not* happen, probably because phylink
ends up believing MAC and PCS are already operating in 2500base-x
mode (but that's not the case).

Previously (eg. with the BananaPi R3) this has not been a problem
because the default interface mode defined in device tree anyway has
also been 2500base-x. Now that I'm testing the same module with the
R4 which uses usxgmii by default the problem surfaced.

On Fri, Dec 01, 2023 at 06:19:20PM +0000, Daniel Golle wrote:
> On Sat, Nov 25, 2023 at 08:28:20AM +0000, Russell King (Oracle) wrote:
> > On Sat, Nov 25, 2023 at 04:56:20AM +0000, Daniel Golle wrote:
> > > Assume 'usxgmii' being set as initial interface mode in DTS. Now plug
> > > a 2.5GBase-T SFP module with exposed PHY. Currently this results in
> > > a rather bizare situation:
> > > 
> > > RTL8221B-VB-CG 2.5Gbps PHY (C45) i2c:sfp1-wan:11: rtl822x_c45_get_features: supported=00,00000000,00008000,000080ef
> > > mtk_soc_eth 15100000.ethernet eth2: requesting link mode phy/2500base-x with support 00,00000000,00008000,0000e0ef
> > > mtk_soc_eth 15100000.ethernet eth2: switched to phy/2500base-x link mode   <<< !!!!!!
> > > mtk_soc_eth 15100000.ethernet eth2: major config usxgmii    <<< !!!!!!
> > > mtk_soc_eth 15100000.ethernet eth2: phylink_mac_config: mode=phy/usxgmii/none adv=00,00000000,00000000,00000000 pause=00
> > > mtk_soc_eth 15100000.ethernet eth2: PHY [i2c:sfp1-wan:11] driver [RTL8221B-VB-CG 2.5Gbps PHY (C45)] (irq=POLL)
> > > mtk_soc_eth 15100000.ethernet eth2: phy: 2500base-x setting supported 00,00000000,00008000,0000e0ef advertising 00,00000000,00008000,0000e0ef
> > > 
> > > Then the link seemingly comes up (but is dead) because no subsequent
> > > call to phylink_major_config actually configured MAC and PCS for
> > > 2500base-x mode.
> > > 
> > > This is because phylink_mac_initial_config() considers
> > > pl->phy_state.interface if in MLO_AN_PHY mode while
> > > phylink_sfp_set_config() only sets pl->link_config.interface.
> > > 
> > > Also set pl->phy_state.interface in phylink_sfp_set_config().
> > 
> > Does it _actually_ matter?
> 
> It does matter, I'm suggesting this change because the SFP module
> won't work without it.
> 
> > When the PHY's link comes up, doesn't it get sorted out for the real
> > mode that will be used?
> 
> That does happen once the interface mode *changes* to anything else,
> than 2500M/Full and the PHY driver sets state->interface to SGMII.
> However, initially it does *not* happen, probably because phylink
> ends up believing MAC and PCS are already operating in 2500base-x
> mode (but that's not the case).
> 
> Previously (eg. with the BananaPi R3) this has not been a problem
> because the default interface mode defined in device tree anyway has
> also been 2500base-x. Now that I'm testing the same module with the
> R4 which uses usxgmii by default the problem surfaced.

Okay, I think what we have here is an ordering issue in
phylink_sfp_connect_phy(). phylink_bringup_phy() will set
phy_state.interface, but that happens after the major config has
happened.

With a built-in PHY, the sequence would be:

 - connect to PHY (which calls phylink_bringup_phy())
 - phylink_start() (which calls phylink_major_config())

whereas in this case we have the reverse. That needs to be sorted.
I can't do that over this weekend, but if you remind me I'll look
at it next week.

Thanks.

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!