[v5] PCI: vmd: Enable PCI PM's L1 substates of remapped PCIe Root Port and NVMe

[PATCH v5 0/4] PCI: vmd: Enable PCI PM's L1 substates of remapped PCIe Root Port and NVMe

Posted by Jian-Hong Pan 1 year, 9 months ago

Re-send for the version information.

Notice the VMD remapped PCIe Root Port and NVMe have PCI PM L1 substates
capability, but they are disabled originally.

Here is a failed example on ASUS B1400CEAE with enabled VMD:

10000:e0:06.0 PCI bridge: Intel Corporation 11th Gen Core Processor PCIe Controller (rev 01) (prog-if 00 [Normal decode])
    ...
    Capabilities: [200 v1] L1 PM Substates
        L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
        	  PortCommonModeRestoreTime=45us PortTPowerOnTime=50us
        L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1-
        	   T_CommonMode=45us LTR1.2_Threshold=101376ns
        L1SubCtl2: T_PwrOn=50us

10000:e1:00.0 Non-Volatile memory controller: Sandisk Corp WD Blue SN550 NVMe SSD (rev 01) (prog-if 02 [NVM Express])
    ...
    Capabilities: [900 v1] L1 PM Substates
        L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1- L1_PM_Substates+
                  PortCommonModeRestoreTime=32us PortTPowerOnTime=10us
        L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1-
                   T_CommonMode=0us LTR1.2_Threshold=0ns
        L1SubCtl2: T_PwrOn=10us

According to "PCIe r6.0, sec 5.5.4", to config the link between the PCIe
Root Port and the child device correctly:
* Ensure both devices are in D0 before enabling PCI-PM L1 PM Substates.
* Ensure L1.2 parameters: Common_Mode_Restore_Times, T_POWER_ON and
  LTR_L1.2_THRESHOLD are programmed properly on both devices before enable
  bits for L1.2.

Prepare this series to fix that.

Jian-Hong Pan (4):
  PCI: vmd: Set PCI devices to D0 before enable PCI PM's L1 substates
  PCI/ASPM: Add notes about enabling PCI-PM L1SS to pci_enable_link_state(_locked)
  PCI/ASPM: Introduce aspm_get_l1ss_cap()
  PCI/ASPM: Fix L1.2 parameters when enable link state

 drivers/pci/controller/vmd.c | 13 ++++++++----
 drivers/pci/pcie/aspm.c      | 41 ++++++++++++++++++++++++++++--------
 2 files changed, 41 insertions(+), 13 deletions(-)

-- 
2.44.0

Re: [PATCH v5 0/4] PCI: vmd: Enable PCI PM's L1 substates of remapped PCIe Root Port and NVMe

Posted by Bjorn Helgaas 1 year, 9 months ago

On Wed, Apr 24, 2024 at 06:58:15PM +0800, Jian-Hong Pan wrote:
> Re-send for the version information.
> 
> Notice the VMD remapped PCIe Root Port and NVMe have PCI PM L1 substates
> capability, but they are disabled originally.

When you post the next version, can you post the patches as responses
to the cover letter?  Patchwork and b4 don't know how to deal with
a series posted as disconnected messages.

Bjorn

Re: [PATCH v5 0/4] PCI: vmd: Enable PCI PM's L1 substates of remapped PCIe Root Port and NVMe

Posted by Nirmal Patel 1 year, 9 months ago

On Wed, 24 Apr 2024 18:58:15 +0800
Jian-Hong Pan <jhp@endlessos.org> wrote:

> Re-send for the version information.
> 
> Notice the VMD remapped PCIe Root Port and NVMe have PCI PM L1
> substates capability, but they are disabled originally.
> 
> Here is a failed example on ASUS B1400CEAE with enabled VMD:
> 
> 10000:e0:06.0 PCI bridge: Intel Corporation 11th Gen Core Processor
> PCIe Controller (rev 01) (prog-if 00 [Normal decode]) ...
>     Capabilities: [200 v1] L1 PM Substates
>         L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
> L1_PM_Substates+ PortCommonModeRestoreTime=45us PortTPowerOnTime=50us
>         L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1-
>         	   T_CommonMode=45us LTR1.2_Threshold=101376ns
>         L1SubCtl2: T_PwrOn=50us
> 
> 10000:e1:00.0 Non-Volatile memory controller: Sandisk Corp WD Blue
> SN550 NVMe SSD (rev 01) (prog-if 02 [NVM Express]) ...
>     Capabilities: [900 v1] L1 PM Substates
>         L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1-
> L1_PM_Substates+ PortCommonModeRestoreTime=32us PortTPowerOnTime=10us
>         L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2+ ASPM_L1.1-
>                    T_CommonMode=0us LTR1.2_Threshold=0ns
>         L1SubCtl2: T_PwrOn=10us
> 
> According to "PCIe r6.0, sec 5.5.4", to config the link between the
> PCIe Root Port and the child device correctly:
> * Ensure both devices are in D0 before enabling PCI-PM L1 PM
> Substates.
> * Ensure L1.2 parameters: Common_Mode_Restore_Times, T_POWER_ON and
>   LTR_L1.2_THRESHOLD are programmed properly on both devices before
> enable bits for L1.2.
> 
> Prepare this series to fix that.
> 
> Jian-Hong Pan (4):
>   PCI: vmd: Set PCI devices to D0 before enable PCI PM's L1 substates
>   PCI/ASPM: Add notes about enabling PCI-PM L1SS to
> pci_enable_link_state(_locked) PCI/ASPM: Introduce aspm_get_l1ss_cap()
>   PCI/ASPM: Fix L1.2 parameters when enable link state
> 
>  drivers/pci/controller/vmd.c | 13 ++++++++----
>  drivers/pci/pcie/aspm.c      | 41
> ++++++++++++++++++++++++++++-------- 2 files changed, 41
> insertions(+), 13 deletions(-)
> 

Hi,

We are running some tests to make sure we dont have issue with other
platforms and trying to avoid another hotplug scenario. Please wait for
our Ack before merging this patch. Thanks.

-nirmal