Before sending Enter Mode for an Alt Mode, there is a gap between Discover
Modes and the Alt Mode driver queueing the Enter Mode VDM for the port
partner to send a message to the port.

If this message results in unregistering Alt Modes such as in a DR_SWAP,
then the following deadlock can occur with respect to the DisplayPort Alt
Mode driver:
1. The DR_SWAP state holds port->lock. Unregistering the Alt Mode driver
results in a cancel_work_sync() that waits for the current dp_altmode_work
to finish.
2. dp_altmode_work makes a call to tcpm_altmode_enter. The deadlock occurs
because tcpm_queue_vdm_unlock attempts to hold port->lock.

Before attempting to grab the lock, ensure that the port is in a state
vdm_run_state_machine can run in. Alt Mode unregistration will not occur
in these states.

Fixes: 03eafcfb60c0 ("usb: typec: tcpm: Add tcpm_queue_vdm_unlocked() helper")
Cc: stable@vger.kernel.org
Signed-off-by: RD Babiera <rdbabiera@google.com>
---
 drivers/usb/typec/tcpm/tcpm.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/usb/typec/tcpm/tcpm.c b/drivers/usb/typec/tcpm/tcpm.c
index c26fb70c3ec6..6fa1601ac259 100644
--- a/drivers/usb/typec/tcpm/tcpm.c
+++ b/drivers/usb/typec/tcpm/tcpm.c
@@ -1564,6 +1564,10 @@ static void tcpm_queue_vdm(struct tcpm_port *port, const u32 header,
 static void tcpm_queue_vdm_unlocked(struct tcpm_port *port, const u32 header,
 				    const u32 *data, int cnt, enum tcpm_transmit_type tx_sop_type)
 {
+	if (port->state != SRC_READY && port->state != SNK_READY &&
+	    port->state != SRC_VDM_IDENTITY_REQUEST)
+		return;
+
 	mutex_lock(&port->lock);
 	tcpm_queue_vdm(port, header, data, cnt, TCPC_TX_SOP);
 	mutex_unlock(&port->lock);

base-commit: 684e9f5f97eb4b7831298ffad140d5c1d426ff27
-- 
2.44.0.769.g3c40516874-goog

On Tue, Apr 23, 2024 at 08:23:57PM +0000, RD Babiera wrote:
> Before sending Enter Mode for an Alt Mode, there is a gap between Discover
> Modes and the Alt Mode driver queueing the Enter Mode VDM for the port
> partner to send a message to the port.
> 
> If this message results in unregistering Alt Modes such as in a DR_SWAP,
> then the following deadlock can occur with respect to the DisplayPort Alt
> Mode driver:
> 1. The DR_SWAP state holds port->lock. Unregistering the Alt Mode driver
> results in a cancel_work_sync() that waits for the current dp_altmode_work
> to finish.
> 2. dp_altmode_work makes a call to tcpm_altmode_enter. The deadlock occurs
> because tcpm_queue_vdm_unlock attempts to hold port->lock.
> 
> Before attempting to grab the lock, ensure that the port is in a state
> vdm_run_state_machine can run in. Alt Mode unregistration will not occur
> in these states.

I'm probable missing something, but wouldn't it be safer to check
port->state after grabbing the lock?

> Fixes: 03eafcfb60c0 ("usb: typec: tcpm: Add tcpm_queue_vdm_unlocked() helper")
> Cc: stable@vger.kernel.org
> Signed-off-by: RD Babiera <rdbabiera@google.com>
> ---
>  drivers/usb/typec/tcpm/tcpm.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/drivers/usb/typec/tcpm/tcpm.c b/drivers/usb/typec/tcpm/tcpm.c
> index c26fb70c3ec6..6fa1601ac259 100644
> --- a/drivers/usb/typec/tcpm/tcpm.c
> +++ b/drivers/usb/typec/tcpm/tcpm.c
> @@ -1564,6 +1564,10 @@ static void tcpm_queue_vdm(struct tcpm_port *port, const u32 header,
>  static void tcpm_queue_vdm_unlocked(struct tcpm_port *port, const u32 header,
>  				    const u32 *data, int cnt, enum tcpm_transmit_type tx_sop_type)
>  {
> +	if (port->state != SRC_READY && port->state != SNK_READY &&
> +	    port->state != SRC_VDM_IDENTITY_REQUEST)
> +		return;
> +
>  	mutex_lock(&port->lock);
>  	tcpm_queue_vdm(port, header, data, cnt, TCPC_TX_SOP);
>  	mutex_unlock(&port->lock);
> 
> base-commit: 684e9f5f97eb4b7831298ffad140d5c1d426ff27
> -- 
> 2.44.0.769.g3c40516874-goog

-- 
heikki

On Thu, Apr 25, 2024 at 2:49 AM Heikki Krogerus
<heikki.krogerus@linux.intel.com> wrote:
> I'm probable missing something, but wouldn't it be safer to check
> port->state after grabbing the lock?

I could have been more explicit in describing the deadlock, my bad.
But there are two
threads here:

Thread A starts in the TCPM. It is the port state machine that
transitions to DR_SWAP
and holds port->lock. When it unregisters DisplayPort Alt Mode, it goes into the
DP Alt Mode driver and hangs until any DP Alt Mode work is finished.

Thread B starts in the DP Alt Mode driver. It attempts to call tcpm_enter_mode
and the call to mutex_lock in tcpm_queue_vdm_unlock hangs because Thread A
holds the lock. Thread A will never drop the lock because it waits for Thread B
to finish.

So, the check is done before grabbing the lock because the thread needs to avoid
grabbing the lock in the first place. If port->state changes between
queueing and
sending the message, the VDM state machine will drop the message anyways
because port->state isn't in the ready state as well.

best,
rd

On Thu, Apr 25, 2024 at 10:37:21AM -0700, RD Babiera wrote:
> On Thu, Apr 25, 2024 at 2:49 AM Heikki Krogerus
> <heikki.krogerus@linux.intel.com> wrote:
> > I'm probable missing something, but wouldn't it be safer to check
> > port->state after grabbing the lock?
> 
> I could have been more explicit in describing the deadlock, my bad.
> But there are two
> threads here:
> 
> Thread A starts in the TCPM. It is the port state machine that
> transitions to DR_SWAP
> and holds port->lock. When it unregisters DisplayPort Alt Mode, it goes into the
> DP Alt Mode driver and hangs until any DP Alt Mode work is finished.
> 
> Thread B starts in the DP Alt Mode driver. It attempts to call tcpm_enter_mode
> and the call to mutex_lock in tcpm_queue_vdm_unlock hangs because Thread A
> holds the lock. Thread A will never drop the lock because it waits for Thread B
> to finish.
> 
> So, the check is done before grabbing the lock because the thread needs to avoid
> grabbing the lock in the first place. If port->state changes between
> queueing and
> sending the message, the VDM state machine will drop the message anyways
> because port->state isn't in the ready state as well.

Okay, thanks for the explanation.

Reviewed-by: Heikki Krogerus <heikki.krogerus@linux.intel.com>

-- 
heikki