From nobody Mon Dec 15 09:41:38 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.libvirt.org designates 8.43.85.245 as permitted sender) client-ip=8.43.85.245; envelope-from=devel-bounces@lists.libvirt.org; helo=lists.libvirt.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of lists.libvirt.org designates 8.43.85.245 as permitted sender) smtp.mailfrom=devel-bounces@lists.libvirt.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.libvirt.org (lists.libvirt.org [8.43.85.245]) by mx.zohomail.com with SMTPS id 174179993131724.955588558102477; Wed, 12 Mar 2025 10:18:51 -0700 (PDT) Received: by lists.libvirt.org (Postfix, from userid 996) id 3A68F1E4A; Wed, 12 Mar 2025 13:18:50 -0400 (EDT) Received: from lists.libvirt.org (localhost [IPv6:::1]) by lists.libvirt.org (Postfix) with ESMTP id 35BAF1A8D; Wed, 12 Mar 2025 13:18:16 -0400 (EDT) Received: by lists.libvirt.org (Postfix, from userid 996) id 652411A1B; Wed, 12 Mar 2025 13:18:12 -0400 (EDT) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.libvirt.org (Postfix) with ESMTPS id BBEEB1659 for ; Wed, 12 Mar 2025 13:18:11 -0400 (EDT) Received: from mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-36-Ie29eHfhN7-lcw3-lo1LMg-1; Wed, 12 Mar 2025 13:18:10 -0400 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-04.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 5BF9E1955D61 for ; Wed, 12 Mar 2025 17:18:09 +0000 (UTC) Received: from toolbx.redhat.com (unknown [10.42.28.57]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 66AAE18001DE; Wed, 12 Mar 2025 17:18:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on lists.libvirt.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,RCVD_IN_VALIDITY_RPBL_BLOCKED, RCVD_IN_VALIDITY_SAFE_BLOCKED,SPF_HELO_NONE autolearn=unavailable autolearn_force=no version=3.4.4 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1741799891; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cFOBi+/HxScqc1+RV6aHaOLKagCteIXU0Cz/YLEcW6Y=; b=GpvDZr0vRezTwa+tUdxIDbiZ4izslGkMC76xefBmJZ9qiUdxT8hboDtkEyGzNjOGIrqvPG si1iy7XesKmYVesQZvoHNIoXCDDUPdBYSHGJVWWimrCo5EGNj55Jgb+9wd81Pr9q63SmnQ MmzPO7FF03J83uIQCTxiUw8E6spY6c4= X-MC-Unique: Ie29eHfhN7-lcw3-lo1LMg-1 X-Mimecast-MFC-AGG-ID: Ie29eHfhN7-lcw3-lo1LMg_1741799889 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: devel@lists.libvirt.org Cc: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Subject: [PATCH v2 03/22] hypervisor: expand available shutdown actions Date: Wed, 12 Mar 2025 17:17:43 +0000 Message-ID: <20250312171802.1854985-4-berrange@redhat.com> In-Reply-To: <20250312171802.1854985-1-berrange@redhat.com> References: <20250312171802.1854985-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: jV3oOOO6_GEF6hX_XTy9bHKPULt5VAf9UUktxECIUJI_1741799889 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Message-ID-Hash: 7XFNSLS3L5USZSISSDZ6GE47INKZSUEK X-Message-ID-Hash: 7XFNSLS3L5USZSISSDZ6GE47INKZSUEK X-MailFrom: berrange@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-config-1; header-match-config-2; header-match-config-3; header-match-devel.lists.libvirt.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header X-Mailman-Version: 3.2.2 Precedence: list List-Id: Development discussions about the libvirt library & tools Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1741799933150019100 Content-Type: text/plain; charset="utf-8" The auto shutdown code can currently only perform managed save, which may fail in some cases, for example when PCI devices are assigned. On failure, shutdown inhibitors remain in place which may be undesirable. This expands the logic to try a sequence of operations * Managed save * Graceful shutdown * Forced poweroff Each of these operations can be enabled or disabled, but they are always applied in this order. With the shutdown option, a configurable time is allowed for shutdown to complete, defaulting to 30 seconds, before moving onto the forced poweroff phase. Signed-off-by: Daniel P. Berrang=C3=A9 Reviewed-by: Peter Krempa --- src/hypervisor/domain_driver.c | 130 ++++++++++++++++++++++++++++----- src/hypervisor/domain_driver.h | 7 ++ src/qemu/qemu_driver.c | 3 + 3 files changed, 121 insertions(+), 19 deletions(-) diff --git a/src/hypervisor/domain_driver.c b/src/hypervisor/domain_driver.c index 0ca482275f..e625726c07 100644 --- a/src/hypervisor/domain_driver.c +++ b/src/hypervisor/domain_driver.c @@ -720,9 +720,25 @@ virDomainDriverAutoShutdown(virDomainDriverAutoShutdow= nConfig *cfg) g_autoptr(virConnect) conn =3D NULL; int numDomains =3D 0; size_t i; - int state; virDomainPtr *domains =3D NULL; - g_autofree unsigned int *flags =3D NULL; + + VIR_DEBUG("Run autoshutdown uri=3D%s trySave=3D%d tryShutdown=3D%d pow= eroff=3D%d waitShutdownSecs=3D%u", + cfg->uri, cfg->trySave, cfg->tryShutdown, cfg->poweroff, + cfg->waitShutdownSecs); + + /* + * Ideally guests will shutdown in a few seconds, but it would + * not be unsual for it to take a while longer, especially under + * load, or if the guest OS has inhibitors slowing down shutdown. + * + * If we wait too long, then guests which ignore the shutdown + * request will significantly delay host shutdown. + * + * Pick 30 seconds as a moderately safe default, assuming that + * most guests are well behaved. + */ + if (cfg->waitShutdownSecs <=3D 0) + cfg->waitShutdownSecs =3D 30; =20 if (!(conn =3D virConnectOpen(cfg->uri))) goto cleanup; @@ -732,31 +748,107 @@ virDomainDriverAutoShutdown(virDomainDriverAutoShutd= ownConfig *cfg) VIR_CONNECT_LIST_DOMAINS_AC= TIVE)) < 0) goto cleanup; =20 - flags =3D g_new0(unsigned int, numDomains); + VIR_DEBUG("Auto shutdown with %d running domains", numDomains); + if (cfg->trySave) { + g_autofree unsigned int *flags =3D g_new0(unsigned int, numDomains= ); + for (i =3D 0; i < numDomains; i++) { + int state; + /* + * Pause all VMs to make them stop dirtying pages, + * so save is quicker. We remember if any VMs were + * paused so we can restore that on resume. + */ + flags[i] =3D VIR_DOMAIN_SAVE_RUNNING; + if (virDomainGetState(domains[i], &state, NULL, 0) =3D=3D 0) { + if (state =3D=3D VIR_DOMAIN_PAUSED) + flags[i] =3D VIR_DOMAIN_SAVE_PAUSED; + } + if (flags[i] & VIR_DOMAIN_SAVE_RUNNING) + virDomainSuspend(domains[i]); + } + + for (i =3D 0; i < numDomains; i++) { + if (virDomainManagedSave(domains[i], flags[i]) < 0) { + VIR_WARN("Unable to perform managed save of '%s': %s", + virDomainGetName(domains[i]), + virGetLastErrorMessage()); + if (flags[i] & VIR_DOMAIN_SAVE_RUNNING) + virDomainResume(domains[i]); + continue; + } + virObjectUnref(domains[i]); + domains[i] =3D NULL; + } + } + + if (cfg->tryShutdown) { + GTimer *timer =3D NULL; + for (i =3D 0; i < numDomains; i++) { + if (domains[i] =3D=3D NULL) + continue; + if (virDomainShutdown(domains[i]) < 0) { + VIR_WARN("Unable to request graceful shutdown of '%s': %s", + virDomainGetName(domains[i]), + virGetLastErrorMessage()); + break; + } + } + + timer =3D g_timer_new(); + while (1) { + bool anyRunning =3D false; + for (i =3D 0; i < numDomains; i++) { + if (!domains[i]) + continue; =20 - /* First we pause all VMs to make them stop dirtying - pages, etc. We remember if any VMs were paused so - we can restore that on resume. */ - for (i =3D 0; i < numDomains; i++) { - flags[i] =3D VIR_DOMAIN_SAVE_RUNNING; - if (virDomainGetState(domains[i], &state, NULL, 0) =3D=3D 0) { - if (state =3D=3D VIR_DOMAIN_PAUSED) - flags[i] =3D VIR_DOMAIN_SAVE_PAUSED; + if (virDomainIsActive(domains[i]) =3D=3D 1) { + anyRunning =3D true; + } else { + virObjectUnref(domains[i]); + domains[i] =3D NULL; + } + } + + if (!anyRunning) + break; + if (g_timer_elapsed(timer, NULL) > cfg->waitShutdownSecs) + break; + g_usleep(1000*500); } - virDomainSuspend(domains[i]); + g_timer_destroy(timer); } =20 - /* Then we save the VMs to disk */ - for (i =3D 0; i < numDomains; i++) - if (virDomainManagedSave(domains[i], flags[i]) < 0) - VIR_WARN("Unable to perform managed save of '%s': %s", - virDomainGetName(domains[i]), - virGetLastErrorMessage()); + if (cfg->poweroff) { + for (i =3D 0; i < numDomains; i++) { + if (domains[i] =3D=3D NULL) + continue; + /* + * NB might fail if we gave up on waiting for + * virDomainShutdown, but it then completed anyway, + * hence we're not checking for failure + */ + virDomainDestroy(domains[i]); + + virObjectUnref(domains[i]); + domains[i] =3D NULL; + } + } =20 cleanup: if (domains) { - for (i =3D 0; i < numDomains; i++) + /* Anything non-NULL in this list indicates none of + * the configured ations were successful in processing + * the domain. There's not much we can do about that + * as the host is powering off, logging at least lets + * admins know + */ + for (i =3D 0; i < numDomains; i++) { + if (domains[i] =3D=3D NULL) + continue; + VIR_WARN("Domain '%s' not successfully shut off by any action", + virDomainGetName(domains[i])); virObjectUnref(domains[i]); + } VIR_FREE(domains); } } diff --git a/src/hypervisor/domain_driver.h b/src/hypervisor/domain_driver.h index f36db5c6f0..ff68517edd 100644 --- a/src/hypervisor/domain_driver.h +++ b/src/hypervisor/domain_driver.h @@ -93,6 +93,13 @@ void virDomainDriverAutoStart(virDomainObjList *domains, =20 typedef struct _virDomainDriverAutoShutdownConfig { const char *uri; + bool trySave; + bool tryShutdown; + bool poweroff; + unsigned int waitShutdownSecs; /* Seconds to wait for VM to shutdown + * before moving onto next action. + * If 0 a default is used (currently 30= secs) + */ } virDomainDriverAutoShutdownConfig; =20 void virDomainDriverAutoShutdown(virDomainDriverAutoShutdownConfig *cfg); diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 7e6ce5f69b..946f1c7e96 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -947,6 +947,9 @@ qemuStateStop(void) g_autoptr(virQEMUDriverConfig) cfg =3D virQEMUDriverGetConfig(qemu_dri= ver); virDomainDriverAutoShutdownConfig ascfg =3D { .uri =3D cfg->uri, + .trySave =3D true, + .tryShutdown =3D false, + .poweroff =3D false, }; =20 if (!qemu_driver->privileged) --=20 2.48.1