[PATCH v6 25/27] cxl/pci: Disable root port interrupts in RCH mode

Terry Bowman posted 27 patches 2 years, 7 months ago
There is a newer version of this series
[PATCH v6 25/27] cxl/pci: Disable root port interrupts in RCH mode
Posted by Terry Bowman 2 years, 7 months ago
The RCH root port contains root command AER registers that should not be
enabled.[1] Disable these to prevent root port interrupts.

[1] CXL 3.0 - 12.2.1.1 RCH Downstream Port-detected Errors

Signed-off-by: Terry Bowman <terry.bowman@amd.com>
Signed-off-by: Robert Richter <rrichter@amd.com>
---
 drivers/cxl/core/core.h |  6 ++++++
 drivers/cxl/core/pci.c  | 29 +++++++++++++++++++++++++++++
 drivers/cxl/core/port.c |  3 +++
 3 files changed, 38 insertions(+)

diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
index 87467c633123..880bac9db376 100644
--- a/drivers/cxl/core/core.h
+++ b/drivers/cxl/core/core.h
@@ -86,4 +86,10 @@ enum cxl_poison_trace_type {
 	CXL_POISON_TRACE_CLEAR,
 };
 
+#ifdef CONFIG_PCIEAER_CXL
+void cxl_disable_rch_root_ints(struct cxl_dport *dport);
+#else
+static inline void cxl_disable_rch_root_ints(struct cxl_dport *dport) { };
+#endif
+
 #endif /* __CXL_CORE_H__ */
diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
index 9e0eba5ccfc4..39a2f9f4f115 100644
--- a/drivers/cxl/core/pci.c
+++ b/drivers/cxl/core/pci.c
@@ -838,6 +838,35 @@ static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds)
 		cxl_handle_rdport_ras(cxlds, dport);
 }
 
+void cxl_disable_rch_root_ints(struct cxl_dport *dport)
+{
+	void __iomem *aer_base = dport->regs.dport_aer;
+	struct pci_host_bridge *bridge;
+	u32 aer_cmd_mask, aer_cmd;
+
+	if (!aer_base)
+		return;
+
+	bridge = to_pci_host_bridge(dport->dport_dev);
+
+	/*
+	 * Disable RCH root port command interrupts.
+	 * CXL 3.0 12.2.1.1 - RCH Downstream Port-detected Errors
+	 *
+	 * This sequnce may not be necessary. CXL spec states disabling
+	 * the root cmd register's interrupts is required. But, PCI spec
+	 * shows these are disabled by default on reset.
+	 */
+	if (bridge->native_cxl_error) {
+		aer_cmd_mask = (PCI_ERR_ROOT_CMD_COR_EN |
+				PCI_ERR_ROOT_CMD_NONFATAL_EN |
+				PCI_ERR_ROOT_CMD_FATAL_EN);
+		aer_cmd = readl(aer_base + PCI_ERR_ROOT_COMMAND);
+		aer_cmd &= ~aer_cmd_mask;
+		writel(aer_cmd, aer_base + PCI_ERR_ROOT_COMMAND);
+	}
+}
+
 #else
 static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds) { }
 #endif
diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
index 87a12e69aa8e..2d812bbaf05f 100644
--- a/drivers/cxl/core/port.c
+++ b/drivers/cxl/core/port.c
@@ -1035,6 +1035,9 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
 
 	cxl_dport_map_regs(dport);
 
+	if (dport->rch)
+		cxl_disable_rch_root_ints(dport);
+
 	cond_cxl_root_lock(port);
 	rc = add_dport(port, dport);
 	cond_cxl_root_unlock(port);
-- 
2.34.1
Re: [PATCH v6 25/27] cxl/pci: Disable root port interrupts in RCH mode
Posted by Jonathan Cameron 2 years, 7 months ago
On Wed, 21 Jun 2023 22:51:24 -0500
Terry Bowman <terry.bowman@amd.com> wrote:

> The RCH root port contains root command AER registers that should not be
> enabled.[1] Disable these to prevent root port interrupts.

I'm a little dubious about a 'because the spec says' so argument.
If we can describe the path by which spurious interrupts turn up then
great - if not then fair enough.

One trivial spelling thing inline. With that fixed.
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

> 
> [1] CXL 3.0 - 12.2.1.1 RCH Downstream Port-detected Errors
> 
> Signed-off-by: Terry Bowman <terry.bowman@amd.com>
> Signed-off-by: Robert Richter <rrichter@amd.com>
> ---
>  drivers/cxl/core/core.h |  6 ++++++
>  drivers/cxl/core/pci.c  | 29 +++++++++++++++++++++++++++++
>  drivers/cxl/core/port.c |  3 +++
>  3 files changed, 38 insertions(+)
> 
> diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h
> index 87467c633123..880bac9db376 100644
> --- a/drivers/cxl/core/core.h
> +++ b/drivers/cxl/core/core.h
> @@ -86,4 +86,10 @@ enum cxl_poison_trace_type {
>  	CXL_POISON_TRACE_CLEAR,
>  };
>  
> +#ifdef CONFIG_PCIEAER_CXL
> +void cxl_disable_rch_root_ints(struct cxl_dport *dport);
> +#else
> +static inline void cxl_disable_rch_root_ints(struct cxl_dport *dport) { };
> +#endif
> +
>  #endif /* __CXL_CORE_H__ */
> diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c
> index 9e0eba5ccfc4..39a2f9f4f115 100644
> --- a/drivers/cxl/core/pci.c
> +++ b/drivers/cxl/core/pci.c
> @@ -838,6 +838,35 @@ static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds)
>  		cxl_handle_rdport_ras(cxlds, dport);
>  }
>  
> +void cxl_disable_rch_root_ints(struct cxl_dport *dport)
> +{
> +	void __iomem *aer_base = dport->regs.dport_aer;
> +	struct pci_host_bridge *bridge;
> +	u32 aer_cmd_mask, aer_cmd;
> +
> +	if (!aer_base)
> +		return;
> +
> +	bridge = to_pci_host_bridge(dport->dport_dev);
> +
> +	/*
> +	 * Disable RCH root port command interrupts.
> +	 * CXL 3.0 12.2.1.1 - RCH Downstream Port-detected Errors
> +	 *
> +	 * This sequnce may not be necessary. CXL spec states disabling

Spell check. (I often forget as well :(


> +	 * the root cmd register's interrupts is required. But, PCI spec
> +	 * shows these are disabled by default on reset.
> +	 */
> +	if (bridge->native_cxl_error) {
> +		aer_cmd_mask = (PCI_ERR_ROOT_CMD_COR_EN |
> +				PCI_ERR_ROOT_CMD_NONFATAL_EN |
> +				PCI_ERR_ROOT_CMD_FATAL_EN);
> +		aer_cmd = readl(aer_base + PCI_ERR_ROOT_COMMAND);
> +		aer_cmd &= ~aer_cmd_mask;
> +		writel(aer_cmd, aer_base + PCI_ERR_ROOT_COMMAND);
> +	}
> +}
> +
>  #else
>  static void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds) { }
>  #endif
> diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
> index 87a12e69aa8e..2d812bbaf05f 100644
> --- a/drivers/cxl/core/port.c
> +++ b/drivers/cxl/core/port.c
> @@ -1035,6 +1035,9 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
>  
>  	cxl_dport_map_regs(dport);
>  
> +	if (dport->rch)
> +		cxl_disable_rch_root_ints(dport);
> +
>  	cond_cxl_root_lock(port);
>  	rc = add_dport(port, dport);
>  	cond_cxl_root_unlock(port);
Re: [PATCH v6 25/27] cxl/pci: Disable root port interrupts in RCH mode
Posted by Terry Bowman 2 years, 7 months ago
Hi Jonathan,

On 6/22/23 08:12, Jonathan Cameron wrote:
> On Wed, 21 Jun 2023 22:51:24 -0500
> Terry Bowman <terry.bowman@amd.com> wrote:
> 
>> The RCH root port contains root command AER registers that should not be
>> enabled.[1] Disable these to prevent root port interrupts.
> 
> I'm a little dubious about a 'because the spec says' so argument.
> If we can describe the path by which spurious interrupts turn up then
> great - if not then fair enough.
> 

This was added to follow the spec. The RCH downstream port contains
root port control pci express capability for enabling and disabling root
port interrupts. The interrupts are (should be) disabled by default at
powerup according to the PCI spec. We know SW does not enable because
the RCH downstream port is not enumerated or managed by a port driver. I
cant say this patch is absolutely necessary but was not comfortable with
removing it either and want to avoid undefined behavior.

Regards,
Terry
Re: [PATCH v6 25/27] cxl/pci: Disable root port interrupts in RCH mode
Posted by Jonathan Cameron 2 years, 7 months ago
On Thu, 22 Jun 2023 11:33:30 -0500
Terry Bowman <Terry.Bowman@amd.com> wrote:

> Hi Jonathan,
> 
> On 6/22/23 08:12, Jonathan Cameron wrote:
> > On Wed, 21 Jun 2023 22:51:24 -0500
> > Terry Bowman <terry.bowman@amd.com> wrote:
> >   
> >> The RCH root port contains root command AER registers that should not be
> >> enabled.[1] Disable these to prevent root port interrupts.  
> > 
> > I'm a little dubious about a 'because the spec says' so argument.
> > If we can describe the path by which spurious interrupts turn up then
> > great - if not then fair enough.
> >   
> 
> This was added to follow the spec. The RCH downstream port contains
> root port control pci express capability for enabling and disabling root
> port interrupts. The interrupts are (should be) disabled by default at
> powerup according to the PCI spec. We know SW does not enable because
> the RCH downstream port is not enumerated or managed by a port driver. I
> cant say this patch is absolutely necessary but was not comfortable with
> removing it either and want to avoid undefined behavior.
> 
Maybe we should just abuse firmware authors and blame them for potentially
having changed the default :)

Fine as is.

Jonathan

> Regards,
> Terry