From nobody Wed Nov 27 12:30:37 2024 Received: from mail-oo1-f53.google.com (mail-oo1-f53.google.com [209.85.161.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BF25A1A0AFA for ; Wed, 9 Oct 2024 17:57:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728496678; cv=none; b=pWCHtbDT2apV6/9+j46cYq4JJ8HwvtJThL9fsswdBeR5hXDPNZTStmzWloyVuJZ4Awm8Iku69GRTvKNLDADlaD2GjYYSTNS99OGMOwyi0bibM4lGwKSAo7B+GOs2gCuWCMuUXB0UMFBE2dyU0Zi+rrTs6FRqdzqpOhJnGKffq5c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728496678; c=relaxed/simple; bh=ZRI2legnzyzaSoUkqUQGMtdIQqeHbvusRsqABYbjEiU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=N4nB81Lxeo4AiGjEzgM04xfjgho5yF9bu5E2TQeaSrtkOgPiw2xkoPXzYWunQSqceHbebxVbCqJ+qBCrYh6jVIIhlHSLcKB/wS5P2nlVnMs+LLzYpxF8agIOOzuqSm96xdhU9C3/iKmoQeOiRWOEac/0UbhjtiGmsISRYUUcZdU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=E6Km+tZZ; arc=none smtp.client-ip=209.85.161.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="E6Km+tZZ" Received: by mail-oo1-f53.google.com with SMTP id 006d021491bc7-5e7f551113dso31838eaf.0 for ; Wed, 09 Oct 2024 10:57:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1728496675; x=1729101475; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LPm/X+fl96wDokyJYUzE4ML1aoLjDUpa0pIP+DD6nQ0=; b=E6Km+tZZnx6xqg1A7wghj5JtNDTXmrcbudhDEan34pK9pFdflz8kFT37189Q/ZoMos A3VhiqYs36Cuiigx2uteYOg6XBw8KbwGN9S8Hs1A1NgI/8UrFoI1+5LnDof8x8OVQIbI Mkhax4UnQDqvp6vQi2J/kutC3dyQfFLHvLiiLep0B9Il7VBHq/Rv4egSTTT29+ngCEiw l8GplpYgEt5waPUFZoP/JN67Ka58ICLwWmvuyu0Om66Ha9ghB206jOOCEf6DP0UodwFx kI7uvW5itCOh3tOPkPxX08N+/HFZHfOx7Qvpqj0xT+ohhfSzE5TlEvj8D55L4IHl/ADp NMQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728496675; x=1729101475; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LPm/X+fl96wDokyJYUzE4ML1aoLjDUpa0pIP+DD6nQ0=; b=BmJDUZFyfGzH/l590xMe6/shPr8Ht5W5/xPeJCHzEx+mZLBG/zOdLpohu4me4zIKLe 5e8chuShLm3FqouZxbbA5xInEEgowTg76x07w3TBFTGvcc694hXGh4i8fO2uI7Ccu/O5 qxLQ4urhAgC3VP9HKiZpKlkUulgXMrZbaawmetX1OhQgzz2uSYYhFXRDEX5I5D7zschY KimP1eit4fAVceWGeNnQkcPoyTRl+sotD9rvJup9fGsmjkB/6Zle5iHAbdaIaTYqAUAP 9wVkw/iN5S5nXhm0y2qQqX+kKdAiWpICnPUan9JUic2SSgjdasyhT3fD8a/8PZfNjd2D 2f8Q== X-Gm-Message-State: AOJu0YypclQbUXDEbuE4Jj1aUeZdY6wvx3/j1J3U/Lvk7fNTXuO2SAla BgRhGrVI4vwmqxKlViUKLjcrQHCIASwxISmzZCYZ+6luMHez7vWqv5Y6p0TP X-Google-Smtp-Source: AGHT+IHq3n5qVynTl8WGUVgCMZSQGlNypP2wpgIrsgNF+ymCZyRCtVqjwo6pYhcKUfn4qky9AfEi4w== X-Received: by 2002:a05:6820:c93:b0:5e1:e748:7d2 with SMTP id 006d021491bc7-5e987a8eda4mr2540631eaf.1.1728496675654; Wed, 09 Oct 2024 10:57:55 -0700 (PDT) Received: from localhost.localdomain ([143.166.81.254]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5e7d711f219sm2442489eaf.9.2024.10.09.10.57.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Oct 2024 10:57:55 -0700 (PDT) From: Stuart Hayes To: linux-kernel@vger.kernel.org, Greg Kroah-Hartman , "Rafael J . Wysocki" , Martin Belanger , Oliver O'Halloran , Daniel Wagner , Keith Busch , Lukas Wunner , David Jeffery , Jeremy Allison , Jens Axboe , Christoph Hellwig , Sagi Grimberg , linux-nvme@lists.infradead.org, Nathan Chancellor , Jan Kiszka , Bert Karwatzki Cc: Stuart Hayes Subject: [PATCH v9 1/4] driver core: don't always lock parent in shutdown Date: Wed, 9 Oct 2024 12:57:43 -0500 Message-Id: <20241009175746.46758-2-stuart.w.hayes@gmail.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20241009175746.46758-1-stuart.w.hayes@gmail.com> References: <20241009175746.46758-1-stuart.w.hayes@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Don't lock a parent device unless it is needed in device_shutdown. This is in preparation for making device shutdown asynchronous, when it will be needed to allow children of a common parent to shut down simultaneously. Signed-off-by: Stuart Hayes Signed-off-by: David Jeffery --- drivers/base/core.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/base/core.c b/drivers/base/core.c index a4c853411a6b..2bf9730db056 100644 --- a/drivers/base/core.c +++ b/drivers/base/core.c @@ -4816,7 +4816,7 @@ void device_shutdown(void) spin_unlock(&devices_kset->list_lock); =20 /* hold lock to avoid race with probe/release */ - if (parent) + if (parent && dev->bus && dev->bus->need_parent_lock) device_lock(parent); device_lock(dev); =20 @@ -4840,7 +4840,7 @@ void device_shutdown(void) } =20 device_unlock(dev); - if (parent) + if (parent && dev->bus && dev->bus->need_parent_lock) device_unlock(parent); =20 put_device(dev); --=20 2.39.3 From nobody Wed Nov 27 12:30:37 2024 Received: from mail-oo1-f41.google.com (mail-oo1-f41.google.com [209.85.161.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 30E261D356C for ; Wed, 9 Oct 2024 17:57:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728496679; cv=none; b=Rck0pPNEB235O/oxHCkvUWW2rze+IHPp/NwwJ0D2ZhFjetOOnRTbJIdlP3JUgH2+CPM+loNjw9oswxElGD81+TVhfXRXbm5u3oNE6DgJ1m/6Sa6eQ20gy1JTDvG9246mWiWOiPvikgCUEuQRNqha13Jb+NUcE+PpatKnbVRpWik= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728496679; c=relaxed/simple; bh=0rTi+zR6nxurhg2CC++4MHJnM0nVd582QJCaKUM/Mtg=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=k8ovhcDMqdcA3zhs7R9waJj20SseGF05TcK1dZpULOk2rJgKhLZi6/ipMBY7anRUrscLrtMJBsxg2jb/mFMwbqduzlP2VXnFW4L+5f1ql/+9YvztZSvQPufPPCmbdpwAaZObXtV+hnLLIMFEgG0yiX0SNbIONSRf0IVOnbX9BIo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=M605sFaC; arc=none smtp.client-ip=209.85.161.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="M605sFaC" Received: by mail-oo1-f41.google.com with SMTP id 006d021491bc7-5d5eec95a74so29123eaf.1 for ; Wed, 09 Oct 2024 10:57:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1728496677; x=1729101477; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yuL/8aIw/afMIlL0zgRWmNHbdROCz4p8+8UOkY7QcKo=; b=M605sFaCeAnqFUREuGGjcrOl0ECktPNIuhB0zJZ3X4Ovx4gDyTqPcI7I+QQJxyROU4 QkH8gd6FFTi1nc7k3Un4SoJUR46eVdN7kaBmVeFxoV0e5n9eL3i/6riJgeMe5OIF8aBP UbuP6va8ju6vMJLiXlaKiqHdvJz2wAbhLjSRggwSKu6hi2PgMpt5WoH03DH4L50154RR rxJYUXkVfi+2Des5lqcA50EPuoN0mMyXLLQJfQ5XhQ8eHDHCsOcIKyv/aM0W14Dr7LeW BmDqZIjGATB99sh/6mo+XcnorMbJJfYKEqDxjq4++EWzfq5LWdGkqL1mkEAagonZIrif 6F+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728496677; x=1729101477; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yuL/8aIw/afMIlL0zgRWmNHbdROCz4p8+8UOkY7QcKo=; b=eV3Kry033sb/oCzBXedzKBT+vpMQAWUQSnsnZqMmYVzPGHe3z66flQWnRTLTyeScos KdntBv6hYpws80CP8OzeLz6q7gko4rNrAVTW7bztTPtFItHkPn3MjsypFWmp7ks1he7+ myzfRBCx7KJEI2jdYoYB1Yv2PBa7ppKGxZMHQNPkucpXIy8t+fd9spWU4uGJ9MwOoVVu HsUt/lrS6J22Jq7zkgsAKEMcUw/dJUmuvJOA3phm/kNNvkQGNg1QD+xLRxfyedihr7jN 26Slz/gnxRy72lFo/WpKqinV9KPwF57KNWjmzjW0BzVlk4ilHS2Lvp+AkzOHM88jTAt+ tvog== X-Gm-Message-State: AOJu0Yx1Zp3EhwU2LBR+21kuEOoQaW+YTTwMIKS1VxhzZbGHUUy4J8x9 NVh30SluVbEIaKh+Hmsbl1713L7yZ5w31Y33/sxqsC6tgrXwBxBiC60uwLKp X-Google-Smtp-Source: AGHT+IE/e+vGPUrJ26IlI7cLrBAxrlVYHJ5MhFchi6chEvdYi6UN+vPbB7ACvmCE2pKN5Yd2RD2hpQ== X-Received: by 2002:a05:6820:1992:b0:5e5:c49a:b53 with SMTP id 006d021491bc7-5e987c3014amr2857685eaf.7.1728496677079; Wed, 09 Oct 2024 10:57:57 -0700 (PDT) Received: from localhost.localdomain ([143.166.81.254]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5e7d711f219sm2442489eaf.9.2024.10.09.10.57.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Oct 2024 10:57:56 -0700 (PDT) From: Stuart Hayes To: linux-kernel@vger.kernel.org, Greg Kroah-Hartman , "Rafael J . Wysocki" , Martin Belanger , Oliver O'Halloran , Daniel Wagner , Keith Busch , Lukas Wunner , David Jeffery , Jeremy Allison , Jens Axboe , Christoph Hellwig , Sagi Grimberg , linux-nvme@lists.infradead.org, Nathan Chancellor , Jan Kiszka , Bert Karwatzki Cc: Stuart Hayes Subject: [PATCH v9 2/4] driver core: separate function to shutdown one device Date: Wed, 9 Oct 2024 12:57:44 -0500 Message-Id: <20241009175746.46758-3-stuart.w.hayes@gmail.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20241009175746.46758-1-stuart.w.hayes@gmail.com> References: <20241009175746.46758-1-stuart.w.hayes@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make a separate function for the part of device_shutdown() that does the shutown for a single device. This is in preparation for making device shutdown asynchronous. Signed-off-by: Stuart Hayes Signed-off-by: David Jeffery --- drivers/base/core.c | 66 ++++++++++++++++++++++++--------------------- 1 file changed, 36 insertions(+), 30 deletions(-) diff --git a/drivers/base/core.c b/drivers/base/core.c index 2bf9730db056..4482382fb947 100644 --- a/drivers/base/core.c +++ b/drivers/base/core.c @@ -4779,6 +4779,41 @@ int device_change_owner(struct device *dev, kuid_t k= uid, kgid_t kgid) } EXPORT_SYMBOL_GPL(device_change_owner); =20 +static void shutdown_one_device(struct device *dev) +{ + /* hold lock to avoid race with probe/release */ + if (dev->parent && dev->bus && dev->bus->need_parent_lock) + device_lock(dev->parent); + device_lock(dev); + + /* Don't allow any more runtime suspends */ + pm_runtime_get_noresume(dev); + pm_runtime_barrier(dev); + + if (dev->class && dev->class->shutdown_pre) { + if (initcall_debug) + dev_info(dev, "shutdown_pre\n"); + dev->class->shutdown_pre(dev); + } + if (dev->bus && dev->bus->shutdown) { + if (initcall_debug) + dev_info(dev, "shutdown\n"); + dev->bus->shutdown(dev); + } else if (dev->driver && dev->driver->shutdown) { + if (initcall_debug) + dev_info(dev, "shutdown\n"); + dev->driver->shutdown(dev); + } + + device_unlock(dev); + if (dev->parent && dev->bus && dev->bus->need_parent_lock) + device_unlock(dev->parent); + + put_device(dev); + if (dev->parent) + put_device(dev->parent); +} + /** * device_shutdown - call ->shutdown() on each device to shutdown. */ @@ -4815,36 +4850,7 @@ void device_shutdown(void) list_del_init(&dev->kobj.entry); spin_unlock(&devices_kset->list_lock); =20 - /* hold lock to avoid race with probe/release */ - if (parent && dev->bus && dev->bus->need_parent_lock) - device_lock(parent); - device_lock(dev); - - /* Don't allow any more runtime suspends */ - pm_runtime_get_noresume(dev); - pm_runtime_barrier(dev); - - if (dev->class && dev->class->shutdown_pre) { - if (initcall_debug) - dev_info(dev, "shutdown_pre\n"); - dev->class->shutdown_pre(dev); - } - if (dev->bus && dev->bus->shutdown) { - if (initcall_debug) - dev_info(dev, "shutdown\n"); - dev->bus->shutdown(dev); - } else if (dev->driver && dev->driver->shutdown) { - if (initcall_debug) - dev_info(dev, "shutdown\n"); - dev->driver->shutdown(dev); - } - - device_unlock(dev); - if (parent && dev->bus && dev->bus->need_parent_lock) - device_unlock(parent); - - put_device(dev); - put_device(parent); + shutdown_one_device(dev); =20 spin_lock(&devices_kset->list_lock); } --=20 2.39.3 From nobody Wed Nov 27 12:30:37 2024 Received: from mail-ot1-f41.google.com (mail-ot1-f41.google.com [209.85.210.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7DE571E0DA8 for ; Wed, 9 Oct 2024 17:57:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728496681; cv=none; b=LafQBWV1Kbgb6ScXF1MIBzJwWrbCPdvVUkUzey5zH06WEH0tEuVCwrAINUfqpbA/rAZnCb/HxJKtqBICuhv3ml/j+k228Ak4ceJiOjDFcywLOkf4YQtddJopFt1MDyh0fSFFd8yGhS8KZQavbb4c3aCJrHA6sPcy9FQ6N1fRCAw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728496681; c=relaxed/simple; bh=pWDG5wKcqqnqRQbHr/2elXDqtzsVTOSauHHyjuqzGQ0=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=oiEa11/1pvtRtr82i9Fc5NSQH0vxSZLR25PW/m0quSG2ZvGhO+U3AA7+qyqb7OOTCtPICCG/MjTvf9BuupCX82qYxePJXwyFvoDBnkqAHD+D6bhKijrmvR8D8i7IcOk9q1hrLAzttnAiqu7NJLZ0OznlTEPXTaHhvYTEjZ16CNw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=MJa9n2J3; arc=none smtp.client-ip=209.85.210.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MJa9n2J3" Received: by mail-ot1-f41.google.com with SMTP id 46e09a7af769-716a7d5c9fbso60301a34.0 for ; Wed, 09 Oct 2024 10:57:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1728496678; x=1729101478; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=CbKbrq2eBZoafCjGOvWmirCqOvc1rlNEfL7PzJ9PmSg=; b=MJa9n2J3pc5w687pEtOt6eChERqkYg6ZXsvOlNWONTKXV4N9Hbl4vrjQXgRqCRW5SD jH73bObZHHvkjRVHTEi9TQ61//9sRLPSbc6OUsqRIDYTcbVxdqjy+YVsC2R/wMIRaWVA 8WTKn/K4jRbpZcg5y4257suBZSLlrpqwO92qYmVzaHNkOFqFJNDb0oZ9aaDZ0abPUxhV 59rbdA8Sh0CwAe0P7gXHdOuUscQlLx1E4tQKwvkCGVQMBTxdMQqP0R6Bar5rDoJ5WOhe F2cE+9QKgqI1rrGgKVa40PGyuNK0BJ4V07VJSCJeJCB1oJ0+Fgf/VQr1/2i5mquv/uzt yOmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728496678; x=1729101478; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CbKbrq2eBZoafCjGOvWmirCqOvc1rlNEfL7PzJ9PmSg=; b=bDTJhEEzsZhQ81n/QfoavTv2gz9/u2okkhFgK5XWi7Dz6luo7bt+IPXyIPiafTfxsy bCfiI1riAU824xFG5jEwKYePti8PDFuDLEjcUDULwiFc905by0U0GgGaRZ9PSyU8Ep9I WQRBuPHN1U6KHAdjNP2lEmNzcgYRXn2QWJu9gE7dKYdwWnryNwdVH4rSkmEzJhVVgcNE /MwElCoJ8YB/gGEIOEsa2lQImbrHpeb1rwF9VUP81a0f/4plbfwOq8+89sIgDfXWbfh+ n8/1VyjFIHh/QxJj0p01iNfXgNj5ILGidXRB6yeak7sQyPefm4NTXYS2ZTy43B4GsDbN PNQA== X-Gm-Message-State: AOJu0Yxuu8qHjgaOLUPD9xKAL5Ve6rJUnGylbCvJ/X4LznvUfau4+Wcw i3TRkpbThOeWzXQTYB3qX31wcSY1YAAQnXsLXFBerS4gP1DT7qv2RMgtXkG+ X-Google-Smtp-Source: AGHT+IHsDJkfTNGPYZjTJw4QewG2zd6CFF2l5EXBDuN2TTJHxDt9CybyyYW3NqGWR12c/J4eH2GThw== X-Received: by 2002:a05:6830:380c:b0:715:4e38:a1ad with SMTP id 46e09a7af769-716a9ffae17mr1682332a34.21.1728496678513; Wed, 09 Oct 2024 10:57:58 -0700 (PDT) Received: from localhost.localdomain ([143.166.81.254]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5e7d711f219sm2442489eaf.9.2024.10.09.10.57.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Oct 2024 10:57:58 -0700 (PDT) From: Stuart Hayes To: linux-kernel@vger.kernel.org, Greg Kroah-Hartman , "Rafael J . Wysocki" , Martin Belanger , Oliver O'Halloran , Daniel Wagner , Keith Busch , Lukas Wunner , David Jeffery , Jeremy Allison , Jens Axboe , Christoph Hellwig , Sagi Grimberg , linux-nvme@lists.infradead.org, Nathan Chancellor , Jan Kiszka , Bert Karwatzki Cc: Stuart Hayes Subject: [PATCH v9 3/4] driver core: shut down devices asynchronously Date: Wed, 9 Oct 2024 12:57:45 -0500 Message-Id: <20241009175746.46758-4-stuart.w.hayes@gmail.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20241009175746.46758-1-stuart.w.hayes@gmail.com> References: <20241009175746.46758-1-stuart.w.hayes@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add code to allow asynchronous shutdown of devices, ensuring that each device is shut down before its parents & suppliers. Any devices that are ordered in the devices_kset list such that a parent or supplier would shut down before its child or consumer are shut down synchronously (as they are without this patch). Only devices with drivers that have async_shutdown_enable enabled will be shut down asynchronously. This can dramatically reduce system shutdown/reboot time on systems that have multiple devices that take many seconds to shut down (like certain NVMe drives). On one system tested, the shutdown time went from 11 minutes without this patch to 55 seconds with the patch. Signed-off-by: Stuart Hayes Signed-off-by: David Jeffery --- drivers/base/base.h | 4 ++ drivers/base/core.c | 81 ++++++++++++++++++++++++++++++++++- include/linux/device/driver.h | 2 + 3 files changed, 86 insertions(+), 1 deletion(-) diff --git a/drivers/base/base.h b/drivers/base/base.h index 8cf04a557bdb..ea18aa70f151 100644 --- a/drivers/base/base.h +++ b/drivers/base/base.h @@ -10,6 +10,7 @@ * shared outside of the drivers/base/ directory. * */ +#include #include =20 /** @@ -97,6 +98,8 @@ struct driver_private { * the device; typically because it depends on another driver getting * probed first. * @async_driver - pointer to device driver awaiting probe via async_probe + * @shutdown_after - used during device shutdown to ensure correct shutdown + * ordering. * @device - pointer back to the struct device that this structure is * associated with. * @dead - This device is currently either in the process of or has been @@ -114,6 +117,7 @@ struct device_private { struct list_head deferred_probe; const struct device_driver *async_driver; char *deferred_probe_reason; + async_cookie_t shutdown_after; struct device *device; u8 dead:1; }; diff --git a/drivers/base/core.c b/drivers/base/core.c index 4482382fb947..bde3fdeafbdb 100644 --- a/drivers/base/core.c +++ b/drivers/base/core.c @@ -9,6 +9,7 @@ */ =20 #include +#include #include #include #include @@ -3524,6 +3525,7 @@ static int device_private_init(struct device *dev) klist_init(&dev->p->klist_children, klist_children_get, klist_children_put); INIT_LIST_HEAD(&dev->p->deferred_probe); + dev->p->shutdown_after =3D 0; return 0; } =20 @@ -4779,6 +4781,8 @@ int device_change_owner(struct device *dev, kuid_t ku= id, kgid_t kgid) } EXPORT_SYMBOL_GPL(device_change_owner); =20 +static ASYNC_DOMAIN(sd_domain); + static void shutdown_one_device(struct device *dev) { /* hold lock to avoid race with probe/release */ @@ -4814,12 +4818,61 @@ static void shutdown_one_device(struct device *dev) put_device(dev->parent); } =20 +static bool device_wants_async_shutdown(struct device *dev) +{ + if (dev->driver && dev->driver->async_shutdown_enable) + return true; + + return false; +} + +/** + * shutdown_one_device_async + * @data: the pointer to the struct device to be shutdown + * @cookie: not used + * + * Shuts down one device, after waiting for previous device to shut down (= for + * synchronous shutdown) or waiting for device's last child or consumer to + * be shutdown (for async shutdown). + * + * shutdown_after is set to the shutdown cookie of the last child or consu= mer + * of this device (if any). + */ +static void shutdown_one_device_async(void *data, async_cookie_t cookie) +{ + struct device *dev =3D data; + async_cookie_t wait =3D cookie; + + if (device_wants_async_shutdown(dev)) { + wait =3D dev->p->shutdown_after + 1; + /* + * To prevent system hang, revert to sync shutdown in the event + * that shutdown_after would make this shutdown wait for a + * shutdown that hasn't been scheduled yet. + * + * This can happen if a parent or supplier is not ordered in the + * devices_kset list before a child or consumer, which is not + * expected. + */ + if (wait > cookie) { + wait =3D cookie; + dev_warn(dev, "Unsafe shutdown ordering, forcing sync order\n"); + } + } + + async_synchronize_cookie_domain(wait, &sd_domain); + shutdown_one_device(dev); +} + /** * device_shutdown - call ->shutdown() on each device to shutdown. */ void device_shutdown(void) { struct device *dev, *parent; + async_cookie_t cookie; + struct device_link *link; + int idx; =20 wait_for_device_probe(); device_block_probing(); @@ -4850,11 +4903,37 @@ void device_shutdown(void) list_del_init(&dev->kobj.entry); spin_unlock(&devices_kset->list_lock); =20 - shutdown_one_device(dev); + get_device(dev); + get_device(parent); + + cookie =3D async_schedule_domain(shutdown_one_device_async, + dev, &sd_domain); + /* + * Ensure any parent & suppliers will wait for this device to + * shut down + */ + if (parent) { + parent->p->shutdown_after =3D cookie; + put_device(parent); + } + + idx =3D device_links_read_lock(); + list_for_each_entry_rcu(link, &dev->links.suppliers, c_node, + device_links_read_lock_held()) { + /* + * sync_state_only devlink consumers aren't dependent on + * suppliers + */ + if (!device_link_flag_is_sync_state_only(link->flags)) + link->supplier->p->shutdown_after =3D cookie; + } + device_links_read_unlock(idx); + put_device(dev); =20 spin_lock(&devices_kset->list_lock); } spin_unlock(&devices_kset->list_lock); + async_synchronize_full_domain(&sd_domain); } =20 /* diff --git a/include/linux/device/driver.h b/include/linux/device/driver.h index 5c04b8e3833b..14c9211b82d6 100644 --- a/include/linux/device/driver.h +++ b/include/linux/device/driver.h @@ -56,6 +56,7 @@ enum probe_type { * @mod_name: Used for built-in modules. * @suppress_bind_attrs: Disables bind/unbind via sysfs. * @probe_type: Type of the probe (synchronous or asynchronous) to use. + * @async_shutdown_enable: Enables devices to be shutdown asynchronously. * @of_match_table: The open firmware table. * @acpi_match_table: The ACPI match table. * @probe: Called to query the existence of a specific device, @@ -102,6 +103,7 @@ struct device_driver { =20 bool suppress_bind_attrs; /* disables bind/unbind via sysfs */ enum probe_type probe_type; + bool async_shutdown_enable; =20 const struct of_device_id *of_match_table; const struct acpi_device_id *acpi_match_table; --=20 2.39.3 From nobody Wed Nov 27 12:30:37 2024 Received: from mail-oi1-f175.google.com (mail-oi1-f175.google.com [209.85.167.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C2D491E103E for ; Wed, 9 Oct 2024 17:58:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728496682; cv=none; b=uSFZTXcIHgojiWZ3EglJz2GsvBFM4CDTp/cRVY7A6PnTphWQVo4YBNA2m+3oYEqC77tWEW0bt6ThNBbKyPYhdSgWkc0MQT68K9IC3tq7r5TvkZNz9g99ZDH8A0wgBzfWfNPpXx+D57SM+jV7EPzxzo7xqvwrhBZVq4Riw2nxIpc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1728496682; c=relaxed/simple; bh=1S9Bz1OBNCsBfE4aviNr4ZXeNfaumRaYVRvRaRjyzMI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=d1Ff6GPrcaujKhLOu+MBsUGz+miazrdSuMYc4TuuBtpRoR6cJgxRFv66Mn6iFgLxhqIG18vHN8GN7t+do4x2bX22rTzD4BDKSZ8gjQ9Zf+Y4xSkFBlPtAqTdnzdq8tkFsFxLLBzy/I/lCsSvA1ara/VOItYeB+GvlL7ZVh202Kw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=FN6Ju+4A; arc=none smtp.client-ip=209.85.167.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FN6Ju+4A" Received: by mail-oi1-f175.google.com with SMTP id 5614622812f47-3e3e7dcf7b3so57752b6e.0 for ; Wed, 09 Oct 2024 10:58:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1728496680; x=1729101480; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RjdY2kbsntsJxkIY+YoEwppZ+eHyqiNWOeFPHP9zjwU=; b=FN6Ju+4AoaHj9O9VLT25L+bPwoER2tOX9P9YcoSB5Z+f7uO1Sk8vhEQwm9360ZRcys 4kLG9dbNKpAdiW7pQBRaSsnJrKO9sCtqcVAMl+FxCTXQkZSZsarvJQOOkTO11msD3hOO DBSKZsm8EDbsTmIyvvh4I/ASD6X2AlKBzLjmTY9mMHEuePjdoNWeyi72ftocBqj8hj0D eGqyB1Fpg3k4sTJ/kU8hGZgpV2qU45ZEJnQvYVkaI39Zg1KFGMQ4WpD0BZM0Z6gLVhEw w8ek3e1TEUrffT9l6PFjMh4Tua12x4sKx05DnOaciQ0lu2K2sJ1DjCshTHUFQFVJeeJY BdwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728496680; x=1729101480; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RjdY2kbsntsJxkIY+YoEwppZ+eHyqiNWOeFPHP9zjwU=; b=ZYrm+uQBM4Mtt6eO4ID8IRbZIrKnuMftb9GqiSzQjoMoOxUjj9D5llFLOZfur4qsQ5 oo2soPWYVcHGmdhJXqgS1mq4w+e/KTk6Kfym0ao9VRpnT4kouDAIdUsfhtUoxsClOqYN y6nx6mmWZNLXxYuFB07UFWTV1CtA37JN+hcWIA5jKGB5G5Wwfr0zaMjZJUO5uYw9esH+ 7VrzJnoY3Lu4Uyyk1g684lFWdk4BDyFO4ZKPe6xX0Gjtd31BHair2n2CvaQDYBZ7DyuF rpC/LisLcp90R2KfY38U/30nmxcfnVHBrBBUdP+zJE2dAdoPQX5f8G5jjwkiCZoTCRpT vFGQ== X-Gm-Message-State: AOJu0YwzBKHvOOZ24tl6JpYHnTZDfQrbUl7qce6o+I32jMPiFixcI7Hx LVZmfIE6OUGoqnIhDwve/VLrkpD2v4EGpB0wOJUSvxX+RCoh8J7NEc25d+S8 X-Google-Smtp-Source: AGHT+IE2oGK/MSVHYCoJinOlidq5VY3r6qCb0/MqDXQTbVF1E62v1vJwCGEoMCNIzqsr8vGUNFmhFg== X-Received: by 2002:a05:6808:219e:b0:3e0:3b50:6fcd with SMTP id 5614622812f47-3e4d7526e1dmr367763b6e.5.1728496679791; Wed, 09 Oct 2024 10:57:59 -0700 (PDT) Received: from localhost.localdomain ([143.166.81.254]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5e7d711f219sm2442489eaf.9.2024.10.09.10.57.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Oct 2024 10:57:59 -0700 (PDT) From: Stuart Hayes To: linux-kernel@vger.kernel.org, Greg Kroah-Hartman , "Rafael J . Wysocki" , Martin Belanger , Oliver O'Halloran , Daniel Wagner , Keith Busch , Lukas Wunner , David Jeffery , Jeremy Allison , Jens Axboe , Christoph Hellwig , Sagi Grimberg , linux-nvme@lists.infradead.org, Nathan Chancellor , Jan Kiszka , Bert Karwatzki Cc: Stuart Hayes Subject: [PATCH v9 4/4] nvme-pci: Make driver prefer asynchronous shutdown Date: Wed, 9 Oct 2024 12:57:46 -0500 Message-Id: <20241009175746.46758-5-stuart.w.hayes@gmail.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20241009175746.46758-1-stuart.w.hayes@gmail.com> References: <20241009175746.46758-1-stuart.w.hayes@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Set the driver default to enable asynchronous shutdown. Signed-off-by: Stuart Hayes Signed-off-by: David Jeffery --- drivers/nvme/host/pci.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index 7990c3f22ecf..1cbff7537788 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -3597,6 +3597,7 @@ static struct pci_driver nvme_driver =3D { .shutdown =3D nvme_shutdown, .driver =3D { .probe_type =3D PROBE_PREFER_ASYNCHRONOUS, + .async_shutdown_enable =3D true, #ifdef CONFIG_PM_SLEEP .pm =3D &nvme_dev_pm_ops, #endif --=20 2.39.3