[v1] tty: serial: kgdboc: fix mutex locking order for configure_kgdboc()

[PATCH] tty: serial: kgdboc: fix mutex locking order for configure_kgdboc()

Posted by John Ogness 2 years, 8 months ago

Several mutexes are taken while setting up console serial ports. In
particular, the tty_port->mutex and @console_mutex are taken:

  serial_pnp_probe
    serial8250_register_8250_port
      uart_add_one_port (locks tty_port->mutex)
        uart_configure_port
          register_console (locks @console_mutex)

In order to synchronize kgdb's tty_find_polling_driver() with
register_console(), commit 6193bc90849a ("tty: serial: kgdboc:
synchronize tty_find_polling_driver() and register_console()") takes
the @console_mutex. However, this leads to the following call chain
(with locking):

  platform_probe
    kgdboc_probe
      configure_kgdboc (locks @console_mutex)
        tty_find_polling_driver
          uart_poll_init (locks tty_port->mutex)
            uart_set_options

This is clearly deadlock potential due to the reverse lock ordering.

Since uart_set_options() requires holding @console_mutex in order to
serialize early initialization of the serial-console lock, take the
@console_mutex in uart_poll_init() instead of configure_kgdboc().

Since configure_kgdboc() was using @console_mutex for safe traversal
of the console list, change it to use the SRCU iterator instead.

Add comments to uart_set_options() kerneldoc mentioning that it
requires holding @console_mutex (aka the console_list_lock).

Fixes: 6193bc90849a ("tty: serial: kgdboc: synchronize tty_find_polling_driver() and register_console()")
Signed-off-by: John Ogness <john.ogness@linutronix.de>
---
 drivers/tty/serial/kgdboc.c      | 16 ++++------------
 drivers/tty/serial/serial_core.c |  5 +++++
 2 files changed, 9 insertions(+), 12 deletions(-)

diff --git a/drivers/tty/serial/kgdboc.c b/drivers/tty/serial/kgdboc.c
index a3ed9b34e2ab..f7df46cc1890 100644
--- a/drivers/tty/serial/kgdboc.c
+++ b/drivers/tty/serial/kgdboc.c
@@ -171,6 +171,7 @@ static int configure_kgdboc(void)
 	int err = -ENODEV;
 	char *cptr = config;
 	struct console *cons;
+	int cookie;
 
 	if (!strlen(config) || isspace(config[0])) {
 		err = 0;
@@ -189,15 +190,6 @@ static int configure_kgdboc(void)
 	if (kgdboc_register_kbd(&cptr))
 		goto do_register;
 
-	/*
-	 * tty_find_polling_driver() can call uart_set_options()
-	 * (via poll_init) to configure the uart. Take the console_list_lock
-	 * in order to synchronize against register_console(), which can also
-	 * configure the uart via uart_set_options(). This also allows safe
-	 * traversal of the console list.
-	 */
-	console_list_lock();
-
 	p = tty_find_polling_driver(cptr, &tty_line);
 	if (!p) {
 		console_list_unlock();
@@ -211,7 +203,8 @@ static int configure_kgdboc(void)
 	 */
 	console_lock();
 
-	for_each_console(cons) {
+	cookie = console_srcu_read_lock();
+	for_each_console_srcu(cons) {
 		int idx;
 		if (cons->device && cons->device(cons, &idx) == p &&
 		    idx == tty_line) {
@@ -219,11 +212,10 @@ static int configure_kgdboc(void)
 			break;
 		}
 	}
+	console_srcu_read_unlock(cookie);
 
 	console_unlock();
 
-	console_list_unlock();
-
 	kgdb_tty_driver = p;
 	kgdb_tty_line = tty_line;
 
diff --git a/drivers/tty/serial/serial_core.c b/drivers/tty/serial/serial_core.c
index b9fbbee598b8..ec874f3a567c 100644
--- a/drivers/tty/serial/serial_core.c
+++ b/drivers/tty/serial/serial_core.c
@@ -2212,6 +2212,9 @@ EXPORT_SYMBOL_GPL(uart_parse_options);
  * @parity: parity character - 'n' (none), 'o' (odd), 'e' (even)
  * @bits: number of data bits
  * @flow: flow control character - 'r' (rts)
+ *
+ * Locking: Caller must hold console_list_lock in order to serialize
+ * early initialization of the serial-console lock.
  */
 int
 uart_set_options(struct uart_port *port, struct console *co,
@@ -2619,7 +2622,9 @@ static int uart_poll_init(struct tty_driver *driver, int line, char *options)
 
 	if (!ret && options) {
 		uart_parse_options(options, &baud, &parity, &bits, &flow);
+		console_list_lock();
 		ret = uart_set_options(port, NULL, baud, parity, bits, flow);
+		console_list_unlock();
 	}
 out:
 	mutex_unlock(&tport->mutex);

base-commit: b7bfaa761d760e72a969d116517eaa12e404c262
-- 
2.30.2

Re: [PATCH] tty: serial: kgdboc: fix mutex locking order for configure_kgdboc()

Posted by Petr Mladek 2 years, 8 months ago

On Wed 2023-01-11 15:57:10, John Ogness wrote:
> Several mutexes are taken while setting up console serial ports. In
> particular, the tty_port->mutex and @console_mutex are taken:
> 
>   serial_pnp_probe
>     serial8250_register_8250_port
>       uart_add_one_port (locks tty_port->mutex)
>         uart_configure_port
>           register_console (locks @console_mutex)
> 
> In order to synchronize kgdb's tty_find_polling_driver() with
> register_console(), commit 6193bc90849a ("tty: serial: kgdboc:
> synchronize tty_find_polling_driver() and register_console()") takes
> the @console_mutex. However, this leads to the following call chain
> (with locking):
> 
>   platform_probe
>     kgdboc_probe
>       configure_kgdboc (locks @console_mutex)
>         tty_find_polling_driver
>           uart_poll_init (locks tty_port->mutex)
>             uart_set_options
> 
> This is clearly deadlock potential due to the reverse lock ordering.

Great catch!

> Since uart_set_options() requires holding @console_mutex in order to
> serialize early initialization of the serial-console lock, take the
> @console_mutex in uart_poll_init() instead of configure_kgdboc().
> 
> Since configure_kgdboc() was using @console_mutex for safe traversal
> of the console list, change it to use the SRCU iterator instead.
> 
> Add comments to uart_set_options() kerneldoc mentioning that it
> requires holding @console_mutex (aka the console_list_lock).
> 
> Fixes: 6193bc90849a ("tty: serial: kgdboc: synchronize tty_find_polling_driver() and register_console()")
> Signed-off-by: John Ogness <john.ogness@linutronix.de>

> --- a/drivers/tty/serial/kgdboc.c
> +++ b/drivers/tty/serial/kgdboc.c
> @@ -189,15 +190,6 @@ static int configure_kgdboc(void)
>  	if (kgdboc_register_kbd(&cptr))
>  		goto do_register;
>  
> -	/*
> -	 * tty_find_polling_driver() can call uart_set_options()
> -	 * (via poll_init) to configure the uart. Take the console_list_lock
> -	 * in order to synchronize against register_console(), which can also
> -	 * configure the uart via uart_set_options(). This also allows safe
> -	 * traversal of the console list.
> -	 */
> -	console_list_lock();
> -
>  	p = tty_find_polling_driver(cptr, &tty_line);
>  	if (!p) {
>  		console_list_unlock();

This console_list_unlock() should be removed here as well.

Best Regards,
Petr

PS: I still have to double check all the dependencies. I think that I need
to do it with a fresh head tomorrow.

Re: [PATCH] tty: serial: kgdboc: fix mutex locking order for configure_kgdboc()

Posted by Petr Mladek 2 years, 8 months ago

On Wed 2023-01-11 17:23:55, Petr Mladek wrote:
> On Wed 2023-01-11 15:57:10, John Ogness wrote:
> > Several mutexes are taken while setting up console serial ports. In
> > particular, the tty_port->mutex and @console_mutex are taken:
> > 
> >   serial_pnp_probe
> >     serial8250_register_8250_port
> >       uart_add_one_port (locks tty_port->mutex)
> >         uart_configure_port
> >           register_console (locks @console_mutex)
> > 
> > In order to synchronize kgdb's tty_find_polling_driver() with
> > register_console(), commit 6193bc90849a ("tty: serial: kgdboc:
> > synchronize tty_find_polling_driver() and register_console()") takes
> > the @console_mutex. However, this leads to the following call chain
> > (with locking):
> > 
> >   platform_probe
> >     kgdboc_probe
> >       configure_kgdboc (locks @console_mutex)
> >         tty_find_polling_driver
> >           uart_poll_init (locks tty_port->mutex)
> >             uart_set_options
> > 
> > This is clearly deadlock potential due to the reverse lock ordering.
> 
> Great catch!
> 
> > Since uart_set_options() requires holding @console_mutex in order to
> > serialize early initialization of the serial-console lock, take the
> > @console_mutex in uart_poll_init() instead of configure_kgdboc().
> > 
> > Since configure_kgdboc() was using @console_mutex for safe traversal
> > of the console list, change it to use the SRCU iterator instead.
> > 
> > Add comments to uart_set_options() kerneldoc mentioning that it
> > requires holding @console_mutex (aka the console_list_lock).
> > 
> > Fixes: 6193bc90849a ("tty: serial: kgdboc: synchronize tty_find_polling_driver() and register_console()")
> > Signed-off-by: John Ogness <john.ogness@linutronix.de>
> 
> > --- a/drivers/tty/serial/kgdboc.c
> > +++ b/drivers/tty/serial/kgdboc.c
> > @@ -189,15 +190,6 @@ static int configure_kgdboc(void)
> >  	if (kgdboc_register_kbd(&cptr))
> >  		goto do_register;
> >  
> > -	/*
> > -	 * tty_find_polling_driver() can call uart_set_options()
> > -	 * (via poll_init) to configure the uart. Take the console_list_lock
> > -	 * in order to synchronize against register_console(), which can also
> > -	 * configure the uart via uart_set_options(). This also allows safe
> > -	 * traversal of the console list.
> > -	 */
> > -	console_list_lock();
> > -
> >  	p = tty_find_polling_driver(cptr, &tty_line);
> >  	if (!p) {
> >  		console_list_unlock();
> 
> This console_list_unlock() should be removed here as well.

This seems to be the only problem. Otherwise, the patch looks good to
me.

After removing the superfluous console_list_unlock():

Reviewed-by: Petr Mladek <pmladek@suse.com>

John, would you mind to send v2, please?

Best Regards,
Petr

Re: [PATCH] tty: serial: kgdboc: fix mutex locking order for configure_kgdboc()

Posted by Sergey Senozhatsky 2 years, 8 months ago

On (23/01/12 16:40), Petr Mladek wrote:
> On Wed 2023-01-11 17:23:55, Petr Mladek wrote:
> > On Wed 2023-01-11 15:57:10, John Ogness wrote:
> > > Several mutexes are taken while setting up console serial ports. In
> > > particular, the tty_port->mutex and @console_mutex are taken:
> > > 
> > >   serial_pnp_probe
> > >     serial8250_register_8250_port
> > >       uart_add_one_port (locks tty_port->mutex)
> > >         uart_configure_port
> > >           register_console (locks @console_mutex)
> > > 
> > > In order to synchronize kgdb's tty_find_polling_driver() with
> > > register_console(), commit 6193bc90849a ("tty: serial: kgdboc:
> > > synchronize tty_find_polling_driver() and register_console()") takes
> > > the @console_mutex. However, this leads to the following call chain
> > > (with locking):
> > > 
> > >   platform_probe
> > >     kgdboc_probe
> > >       configure_kgdboc (locks @console_mutex)
> > >         tty_find_polling_driver
> > >           uart_poll_init (locks tty_port->mutex)
> > >             uart_set_options
> > > 
> > > This is clearly deadlock potential due to the reverse lock ordering.
[..]
> > > -	console_list_lock();
> > > -
> > >  	p = tty_find_polling_driver(cptr, &tty_line);
> > >  	if (!p) {
> > >  		console_list_unlock();
> > 
> > This console_list_unlock() should be removed here as well.
> 
> This seems to be the only problem. Otherwise, the patch looks good to
> me.

Good catch.

Reviewed-by: Sergey Senozhatsky <senozhatsky@chromium.org>