From nobody Mon Feb 9 12:43:14 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 207.211.31.120 as permitted sender) client-ip=207.211.31.120; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-1.mimecast.com; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of redhat.com designates 207.211.31.120 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=fail(p=none dis=none) header.from=virtuozzo.com ARC-Seal: i=1; a=rsa-sha256; t=1594719227; cv=none; d=zohomail.com; s=zohoarc; b=Wzae6YT/GrkCgYlLkBD3wh1q1VpiyUlz3JFoj6YIZygE1MBSDUyFbS4iXeocxYL8jY+ST6ntB1Jyl950tfBDc/zIBYsa0YAOAGTVnxDH1C3wbwP52zqUM+gNh3UiFfPI/A1P1tE6mQ9zhDmSEA+r3TWpPif41QKw82lpTxMwIPI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1594719227; h=Content-Type:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:Message-ID:References:Sender:Subject:To; bh=2kEr15b1ARxW1NpKWybO0IAMGdkK/vXW1qI/1ZaiBT0=; b=JAMZCOnaYbDV1OD4UccGDnMT5VQiRDZb6hnif4O/sPL4TPgILuJbuyPQpkNBQJd0kEWl561qnJVprEexCFWOfpZJbFTBi9rhAq2LwhGCG9dAEgqv6KEOIhjvP766G6zNzFkuwUlVKtNHkPyOoxTBL29b3PFMVa0l2C2uySdfz08= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of redhat.com designates 207.211.31.120 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) by mx.zohomail.com with SMTPS id 159471922710664.33756265204943; Tue, 14 Jul 2020 02:33:47 -0700 (PDT) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-389-o27t1hAeOJCdmMmznktWXg-1; Tue, 14 Jul 2020 05:33:42 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 740B8802B40; Tue, 14 Jul 2020 09:33:36 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 511ED10013D0; Tue, 14 Jul 2020 09:33:36 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 1E627180597C; Tue, 14 Jul 2020 09:33:36 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 06E9XNgh027936 for ; Tue, 14 Jul 2020 05:33:23 -0400 Received: by smtp.corp.redhat.com (Postfix) id 8F31A10CD61C; Tue, 14 Jul 2020 09:33:23 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast06.extmail.prod.ext.rdu2.redhat.com [10.11.55.22]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4B63110CD6DC for ; Tue, 14 Jul 2020 09:33:21 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1F0651832D25 for ; Tue, 14 Jul 2020 09:33:21 +0000 (UTC) Received: from relay3.sw.ru (relay.sw.ru [185.231.240.75]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-507-eRi9sWOGOO-1B91sOGyHsg-1; Tue, 14 Jul 2020 05:33:18 -0400 Received: from [172.16.50.220] (helo=vz7.localdomain) by relay3.sw.ru with esmtp (Exim 4.93) (envelope-from ) id 1jvHJB-00044B-Nr for libvir-list@redhat.com; Tue, 14 Jul 2020 12:33:05 +0300 X-MC-Unique: o27t1hAeOJCdmMmznktWXg-1 X-MC-Unique: eRi9sWOGOO-1B91sOGyHsg-1 From: Nikolay Shirokovskiy To: libvir-list@redhat.com Subject: [PATCH 05/10] rpc: finish all threads before exiting main loop Date: Tue, 14 Jul 2020 12:32:56 +0300 Message-Id: <1594719181-546271-6-git-send-email-nshirokovskiy@virtuozzo.com> In-Reply-To: <1594719181-546271-1-git-send-email-nshirokovskiy@virtuozzo.com> References: <1594719181-546271-1-git-send-email-nshirokovskiy@virtuozzo.com> X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-loop: libvir-list@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Mimecast-Spam-Score: 2 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Currently we have issues like [1] on libvirtd shutdown as we cleanup while = RPC and other threads are still running. Let's finish all threads other then ma= in before cleanup. The approach to finish threads is suggested in [2]. In order to finish RPC threads serving API calls we let the event loop run but stop accepting new = API calls and block processing any pending API calls. We also inform all driver= s of shutdown so they can prepare for shutdown too. Then we wait for all RPC thr= eads and driver's background thread to finish. If finishing takes more then 15s = we just exit as we can't safely cleanup in time. [1] https://bugzilla.redhat.com/show_bug.cgi?id=3D1828207 [2] https://www.redhat.com/archives/libvir-list/2020-April/msg01328.html Signed-off-by: Nikolay Shirokovskiy --- src/remote/remote_daemon.c | 3 -- src/rpc/virnetdaemon.c | 82 ++++++++++++++++++++++++++++++++++++++++++= +++- src/rpc/virnetserver.c | 8 +++++ src/rpc/virnetserver.h | 1 + 4 files changed, 90 insertions(+), 4 deletions(-) diff --git a/src/remote/remote_daemon.c b/src/remote/remote_daemon.c index 1aa9bfc..222bb5f 100644 --- a/src/remote/remote_daemon.c +++ b/src/remote/remote_daemon.c @@ -1201,9 +1201,6 @@ int main(int argc, char **argv) { 0, "shutdown", NULL, NULL); =20 cleanup: - /* Keep cleanup order in inverse order of startup */ - virNetDaemonClose(dmn); - virNetlinkEventServiceStopAll(); =20 if (driversInitialized) { diff --git a/src/rpc/virnetdaemon.c b/src/rpc/virnetdaemon.c index bb81a43..c4b31c6 100644 --- a/src/rpc/virnetdaemon.c +++ b/src/rpc/virnetdaemon.c @@ -23,6 +23,7 @@ #include #include =20 +#include "libvirt_internal.h" #include "virnetdaemon.h" #include "virlog.h" #include "viralloc.h" @@ -69,7 +70,10 @@ struct _virNetDaemon { virHashTablePtr servers; virJSONValuePtr srvObject; =20 + int finishTimer; bool quit; + bool finished; + bool graceful; =20 unsigned int autoShutdownTimeout; size_t autoShutdownInhibitions; @@ -80,6 +84,11 @@ struct _virNetDaemon { =20 static virClassPtr virNetDaemonClass; =20 +static int +daemonServerClose(void *payload, + const void *key G_GNUC_UNUSED, + void *opaque G_GNUC_UNUSED); + static void virNetDaemonDispose(void *obj) { @@ -796,11 +805,53 @@ daemonServerProcessClients(void *payload, return 0; } =20 +static int +daemonServerShutdownWait(void *payload, + const void *key G_GNUC_UNUSED, + void *opaque G_GNUC_UNUSED) +{ + virNetServerPtr srv =3D payload; + + virNetServerShutdownWait(srv); + return 0; +} + +static void +daemonShutdownWait(void *opaque) +{ + virNetDaemonPtr dmn =3D opaque; + bool graceful =3D false; + + virHashForEach(dmn->servers, daemonServerShutdownWait, NULL); + if (virStateShutdownWait() < 0) + goto finish; + + graceful =3D true; + + finish: + virObjectLock(dmn); + dmn->graceful =3D graceful; + virEventUpdateTimeout(dmn->finishTimer, 0); + virObjectUnlock(dmn); +} + +static void +virNetDaemonFinishTimer(int timerid G_GNUC_UNUSED, + void *opaque) +{ + virNetDaemonPtr dmn =3D opaque; + + virObjectLock(dmn); + dmn->finished =3D true; + virObjectUnlock(dmn); +} + void virNetDaemonRun(virNetDaemonPtr dmn) { int timerid =3D -1; bool timerActive =3D false; + virThread shutdownThread; =20 virObjectLock(dmn); =20 @@ -811,6 +862,9 @@ virNetDaemonRun(virNetDaemonPtr dmn) } =20 dmn->quit =3D false; + dmn->finishTimer =3D -1; + dmn->finished =3D false; + dmn->graceful =3D false; =20 if (dmn->autoShutdownTimeout && (timerid =3D virEventAddTimeout(-1, @@ -826,7 +880,7 @@ virNetDaemonRun(virNetDaemonPtr dmn) virSystemdNotifyStartup(); =20 VIR_DEBUG("dmn=3D%p quit=3D%d", dmn, dmn->quit); - while (!dmn->quit) { + while (!dmn->finished) { /* A shutdown timeout is specified, so check * if any drivers have active state, if not * shutdown after timeout seconds @@ -857,6 +911,32 @@ virNetDaemonRun(virNetDaemonPtr dmn) virObjectLock(dmn); =20 virHashForEach(dmn->servers, daemonServerProcessClients, NULL); + + if (dmn->quit && dmn->finishTimer =3D=3D -1) { + virHashForEach(dmn->servers, daemonServerClose, NULL); + if (virStateShutdown() < 0) + break; + + if ((dmn->finishTimer =3D virEventAddTimeout(15 * 1000, + virNetDaemonFinishT= imer, + dmn, NULL)) < 0) { + VIR_WARN("Failed to register finish timer."); + break; + } + + if (virThreadCreateFull(&shutdownThread, true, daemonShutdownW= ait, + "daemon-shutdown", false, dmn) < 0) { + VIR_WARN("Failed to register join thread."); + break; + } + } + } + + if (dmn->graceful) { + virThreadJoin(&shutdownThread); + } else { + VIR_WARN("Make forcefull daemon shutdown"); + exit(EXIT_FAILURE); } =20 cleanup: diff --git a/src/rpc/virnetserver.c b/src/rpc/virnetserver.c index e0a2386..79ea9f6 100644 --- a/src/rpc/virnetserver.c +++ b/src/rpc/virnetserver.c @@ -942,9 +942,17 @@ void virNetServerClose(virNetServerPtr srv) for (i =3D 0; i < srv->nclients; i++) virNetServerClientClose(srv->clients[i]); =20 + virThreadPoolStop(srv->workers); + virObjectUnlock(srv); } =20 +void +virNetServerShutdownWait(virNetServerPtr srv) +{ + virThreadPoolDrain(srv->workers); +} + static inline size_t virNetServerTrackPendingAuthLocked(virNetServerPtr srv) { diff --git a/src/rpc/virnetserver.h b/src/rpc/virnetserver.h index 1c6a2ef..112a51d 100644 --- a/src/rpc/virnetserver.h +++ b/src/rpc/virnetserver.h @@ -56,6 +56,7 @@ virNetServerPtr virNetServerNewPostExecRestart(virJSONVal= uePtr object, ATTRIBUTE_NONNULL(4) ATTRIBUTE_NONNULL(5) ATTRIBUTE_NONNULL(6); =20 void virNetServerClose(virNetServerPtr srv); +void virNetServerShutdownWait(virNetServerPtr srv); =20 virJSONValuePtr virNetServerPreExecRestart(virNetServerPtr srv); =20 --=20 1.8.3.1