From nobody Tue Jan 21 07:23:31 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.libvirt.org designates 8.43.85.245 as permitted sender) client-ip=8.43.85.245; envelope-from=devel-bounces@lists.libvirt.org; helo=lists.libvirt.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of lists.libvirt.org designates 8.43.85.245 as permitted sender) smtp.mailfrom=devel-bounces@lists.libvirt.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.libvirt.org (lists.libvirt.org [8.43.85.245]) by mx.zohomail.com with SMTPS id 173636570058227.63068622460787; Wed, 8 Jan 2025 11:48:20 -0800 (PST) Received: by lists.libvirt.org (Postfix, from userid 996) id A37F1144B; Wed, 8 Jan 2025 14:48:19 -0500 (EST) Received: from lists.libvirt.org (localhost [IPv6:::1]) by lists.libvirt.org (Postfix) with ESMTP id 60972136E; Wed, 8 Jan 2025 14:43:45 -0500 (EST) Received: by lists.libvirt.org (Postfix, from userid 996) id 07F241498; Wed, 8 Jan 2025 14:43:41 -0500 (EST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.libvirt.org (Postfix) with ESMTPS id 1DBBD1450 for ; Wed, 8 Jan 2025 14:43:15 -0500 (EST) Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-348-VMhD1aCuMpu08ALjvvOBbw-1; Wed, 08 Jan 2025 14:43:13 -0500 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id B6F6B19772C6 for ; Wed, 8 Jan 2025 19:43:12 +0000 (UTC) Received: from toolbx.redhat.com (unknown [10.42.28.103]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id DD633300019E; Wed, 8 Jan 2025 19:43:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on lists.libvirt.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=5.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H2,RCVD_IN_VALIDITY_RPBL_BLOCKED, RCVD_IN_VALIDITY_SAFE_BLOCKED,SPF_HELO_NONE autolearn=unavailable autolearn_force=no version=3.4.4 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1736365394; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KWbjacy0MtMTnb+TGOHwusCLs9eR2abQH4k7jHbo4Jk=; b=P0F/3fs91daDOpDHgCd2NxEedI5XiWChaV6o93Tt9IoMrneWKWfc01GLbhmQ8GaH/sUhDZ BASmozvGPYBNIB6XBcJeQqebdyp0rS4Skp1LClKXRZ/HuIOjgFv5m7Nyi7A+Yup+bJ6YHL 1T2XIX5ocoS8o2W6XvKONCks6S5ztdg= X-MC-Unique: VMhD1aCuMpu08ALjvvOBbw-1 X-Mimecast-MFC-AGG-ID: VMhD1aCuMpu08ALjvvOBbw From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: devel@lists.libvirt.org Cc: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Subject: [PATCH 09/26] hypervisor: expand available shutdown actions Date: Wed, 8 Jan 2025 19:42:42 +0000 Message-ID: <20250108194259.1171990-10-berrange@redhat.com> In-Reply-To: <20250108194259.1171990-1-berrange@redhat.com> References: <20250108194259.1171990-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: Nm4o_VWJQ_KoHTSEDPDDKbLv49ikou5sLY_Q79rMMvg_1736365392 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Message-ID-Hash: CUCW4ACTPQQX553YBZMYX6FZFP6MVVCZ X-Message-ID-Hash: CUCW4ACTPQQX553YBZMYX6FZFP6MVVCZ X-MailFrom: berrange@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-config-1; header-match-config-2; header-match-config-3; header-match-devel.lists.libvirt.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header X-Mailman-Version: 3.2.2 Precedence: list List-Id: Development discussions about the libvirt library & tools Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1736365728834116600 Content-Type: text/plain; charset="utf-8" The auto shutdown code can currently only perform managed save, which may fail in some cases, for example when PCI devices are assigned. On failure, shutdown inhibitors remain in place which may be undesirable. This expands the logic to try a sequence of operations * Managed save * Graceful shutdown * Forced poweroff Each of these operations can be enabled or disabled, but they are always applied in this order. With the shutdown option, a configurable time is allowed for shutdown to complete, defaulting to 30 seconds, before moving onto the forced poweroff phase. Signed-off-by: Daniel P. Berrang=C3=A9 --- src/hypervisor/domain_driver.c | 113 +++++++++++++++++++++++++++------ src/hypervisor/domain_driver.h | 4 ++ src/qemu/qemu_driver.c | 3 + 3 files changed, 100 insertions(+), 20 deletions(-) diff --git a/src/hypervisor/domain_driver.c b/src/hypervisor/domain_driver.c index 949e3d01f1..ea3f1cbfcd 100644 --- a/src/hypervisor/domain_driver.c +++ b/src/hypervisor/domain_driver.c @@ -714,9 +714,26 @@ virDomainDriverAutoShutdown(virDomainDriverAutoShutdow= nConfig *cfg) g_autoptr(virConnect) conn =3D NULL; int numDomains =3D 0; size_t i; - int state; virDomainPtr *domains =3D NULL; - g_autofree unsigned int *flags =3D NULL; + + VIR_DEBUG("Run autoshutdown uri=3D%s trySave=3D%d tryShutdown=3D%d pow= eroff=3D%d" + "waitShutdownSecs=3D%d", + cfg->uri, cfg->trySave, cfg->tryShutdown, cfg->poweroff, + cfg->waitShutdownSecs); + + /* + * Ideally guests will shutdown in a few seconds, but it would + * not be unsual for it to take a while longer, especially under + * load, or if the guest OS has inhibitors slowing down shutdown. + * + * If we wait too long, then guests which ignore the shutdown + * request will significantly delay host shutdown. + * + * Pick 30 seconds as a moderately safe default, assuming that + * most guests are well behaved. + */ + if (cfg->waitShutdownSecs <=3D 0) + cfg->waitShutdownSecs =3D 30; =20 if (!(conn =3D virConnectOpen(cfg->uri))) goto cleanup; @@ -726,31 +743,87 @@ virDomainDriverAutoShutdown(virDomainDriverAutoShutdo= wnConfig *cfg) VIR_CONNECT_LIST_DOMAINS_AC= TIVE)) < 0) goto cleanup; =20 - flags =3D g_new0(unsigned int, numDomains); + VIR_DEBUG("Auto shutdown with %d running domains", numDomains); + if (cfg->trySave) { + g_autofree unsigned int *flags =3D g_new0(unsigned int, numDomains= ); + for (i =3D 0; i < numDomains; i++) { + int state; + /* + * Pause all VMs to make them stop dirtying pages, + * so save is quicker. We remember if any VMs were + * paused so we can restore that on resume. + */ + flags[i] =3D VIR_DOMAIN_SAVE_RUNNING; + if (virDomainGetState(domains[i], &state, NULL, 0) =3D=3D 0) { + if (state =3D=3D VIR_DOMAIN_PAUSED) + flags[i] =3D VIR_DOMAIN_SAVE_PAUSED; + } + virDomainSuspend(domains[i]); + } + + for (i =3D 0; i < numDomains; i++) { + if (virDomainManagedSave(domains[i], flags[i]) < 0) { + VIR_WARN("Unable to perform managed save of '%s': %s", + virDomainGetName(domains[i]), + virGetLastErrorMessage()); + continue; + } + virObjectUnref(domains[i]); + domains[i] =3D NULL; + } + } + + if (cfg->tryShutdown) { + GTimer *timer =3D NULL; + for (i =3D 0; i < numDomains; i++) { + if (domains[i] =3D=3D NULL) + continue; + if (virDomainShutdown(domains[i]) < 0) { + VIR_WARN("Unable to perform graceful shutdown of '%s': %s", + virDomainGetName(domains[i]), + virGetLastErrorMessage()); + break; + } + } + + timer =3D g_timer_new(); + while (1) { + bool anyRunning =3D false; + for (i =3D 0; i < numDomains; i++) { + if (!domains[i]) + continue; =20 - /* First we pause all VMs to make them stop dirtying - pages, etc. We remember if any VMs were paused so - we can restore that on resume. */ - for (i =3D 0; i < numDomains; i++) { - flags[i] =3D VIR_DOMAIN_SAVE_RUNNING; - if (virDomainGetState(domains[i], &state, NULL, 0) =3D=3D 0) { - if (state =3D=3D VIR_DOMAIN_PAUSED) - flags[i] =3D VIR_DOMAIN_SAVE_PAUSED; + if (virDomainIsActive(domains[i]) =3D=3D 1) { + anyRunning =3D true; + } else { + virObjectUnref(domains[i]); + domains[i] =3D NULL; + } + } + + if (!anyRunning) + break; + if (g_timer_elapsed(timer, NULL) > cfg->waitShutdownSecs) + break; + g_usleep(1000*500); } - virDomainSuspend(domains[i]); + g_timer_destroy(timer); } =20 - /* Then we save the VMs to disk */ - for (i =3D 0; i < numDomains; i++) - if (virDomainManagedSave(domains[i], flags[i]) < 0) - VIR_WARN("Unable to perform managed save of '%s': %s", - virDomainGetName(domains[i]), - virGetLastErrorMessage()); + if (cfg->poweroff) { + for (i =3D 0; i < numDomains; i++) { + if (domains[i] =3D=3D NULL) + continue; + virDomainDestroy(domains[i]); + } + } =20 cleanup: if (domains) { - for (i =3D 0; i < numDomains; i++) - virObjectUnref(domains[i]); + for (i =3D 0; i < numDomains; i++) { + if (domains[i]) + virObjectUnref(domains[i]); + } VIR_FREE(domains); } } diff --git a/src/hypervisor/domain_driver.h b/src/hypervisor/domain_driver.h index d1588ea712..25c4b0c664 100644 --- a/src/hypervisor/domain_driver.h +++ b/src/hypervisor/domain_driver.h @@ -92,6 +92,10 @@ void virDomainDriverAutoStart(virDomainObjList *domains, =20 typedef struct _virDomainDriverAutoShutdownConfig { const char *uri; + bool trySave; + bool tryShutdown; + bool poweroff; + int waitShutdownSecs; } virDomainDriverAutoShutdownConfig; =20 void virDomainDriverAutoShutdown(virDomainDriverAutoShutdownConfig *cfg); diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 103369ac93..2c97a6fb98 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -947,6 +947,9 @@ qemuStateStop(void) g_autoptr(virQEMUDriverConfig) cfg =3D virQEMUDriverGetConfig(qemu_dri= ver); virDomainDriverAutoShutdownConfig ascfg =3D { .uri =3D cfg->uri, + .trySave =3D true, + .tryShutdown =3D false, + .poweroff =3D false, }; =20 if (!qemu_driver->privileged) --=20 2.47.1