From nobody Sun Nov 22 16:13:42 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.libvirt.org designates 38.145.34.151 as permitted sender) client-ip=38.145.34.151; envelope-from=devel-bounces@lists.libvirt.org; helo=lists.libvirt.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of lists.libvirt.org designates 38.145.34.151 as permitted sender) smtp.mailfrom=devel-bounces@lists.libvirt.org; dmarc=pass(p=reject dis=none) header.from=lists.libvirt.org ARC-Seal: i=1; a=rsa-sha256; t=1790004759; cv=none; d=zohomail.com; s=zohoarc; b=ePwZW31tX0CywYuPgAgpBj1RX6FjH+xmVwlMn02N1sWlnIINei3kWQfheDIXSTgfArskC/mrRMRHmdjSQ/aH1xF/laiYG4jYtdsHY4a1s1nHDTkTZekvHuE73h4UIoXesJ2d3HBygRoBAiSUHD/9lDyRzy7rWqrzjlkYQ2J4FJc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1790004759; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Owner:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:Subject:Subject:To:To:Message-Id; bh=vWv+uI05aF5vFNVT95jH1hloSv/gkTvPLZpevSTBVww=; b=jX2MSv0jXG6qKyWfCvVxQSNT/G+AsODyPtCthSIz/qBKmzY3+/a3XrzxWzf56XKM2ZWamysoZbunnYfsx1J4H8fWjdbV5bXqkfgCUJw7tO1+RPvUMTeylQ18oze6pVlI5RIadz/ejuRQddaX75D/MAg4HrmAEMX/AkZgTu0/rtA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of lists.libvirt.org designates 38.145.34.151 as permitted sender) smtp.mailfrom=devel-bounces@lists.libvirt.org; dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.libvirt.org (lists.libvirt.org [38.145.34.151]) by mx.zohomail.com with SMTPS id 1790004759465293.684067048541; Mon, 21 Sep 2026 08:32:39 -0700 (PDT) Received: by lists.libvirt.org (Postfix, from userid 993) id E21663FCAD; Mon, 21 Sep 2026 11:32:37 -0400 (EDT) Received: from [172.19.199.16] (unknown [10.16.107.18]) by lists.libvirt.org (Postfix) with ESMTP id 190A03FB97; Mon, 21 Sep 2026 11:29:56 -0400 (EDT) Received: by lists.libvirt.org (Postfix, from userid 993) id 3F67A3F316; Mon, 21 Sep 2026 11:25:12 -0400 (EDT) Received: from mail-wm2-x10.google.com (mail-wm2-x10.google.com [IPv6:2a00:1450:4864:31::10]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (3072 bits) server-digest SHA256) (No client certificate requested) by lists.libvirt.org (Postfix) with ESMTPS id 0F12C3F340 for ; Mon, 21 Sep 2026 11:25:10 -0400 (EDT) Received: by mail-wm2-x10.google.com with SMTP id 5b1f17b1804b1-49b912e2406so16249075e9.1 for ; Mon, 21 Sep 2026 08:25:10 -0700 (PDT) Received: from athena.sw.ru ([2a06:5b06:b600:300:5745:5560:519b:4337]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49fcd0f7e05sm248149245e9.4.2026.09.21.08.25.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 08:25:09 -0700 (PDT) X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-26) on lists.libvirt.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE,SPF_HELO_NONE autolearn=unavailable autolearn_force=no version=4.0.1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=openvz.org; s=google; t=1790004310; x=1790609110; darn=lists.libvirt.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=vWv+uI05aF5vFNVT95jH1hloSv/gkTvPLZpevSTBVww=; b=nv695CJqATn7x+8I9WKOiqXwgO8pe169kDQ0a0HMXeeD35YbNs9+quYfn39dHJXLiT jvgkBzzAt7VGfTA6hxRWC+FMr4c7a0sDIFSqli9Jx8k+10fhh9L2U5E478EmYea6u+Al 9msvNdxhkdAobm/ey0vhe5Xl3nKtn8JuQ9GjvrtypN7M5VI8fWhPii0jAh+m7N88DVKs 9CNdt2bw5E5jpnHLHKRGUUiSsq9UUfF/bD22oA9kwK0Nkd9srIg8OGU7JV85WCISMOrd OTRNIPn8Zz2190V3NQSIry4S2WfDBSBZjFSNQkIQdAgE9Hf+sj8eZH2QYdrag/3ZMH+N gnHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790004310; x=1790609110; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=vWv+uI05aF5vFNVT95jH1hloSv/gkTvPLZpevSTBVww=; b=oyxiJzVvwKEHoOPe3aAlcgWu25JlsU8+Nf0bf+WhxjAJtSEdjb9m9jW5cT4hVgY8t9 GVPPBhHTRDrzPucq1HwMsoecZ4QfNwSibh+S5LGl02GQT0092QpwM2A4gHQTsUqmsLUm Ecj3OvAjF7K+Piu7vtIAaOAtiFgZZFH7q3Tf76EfeCWuwHvp/GLWck9a9ZlqHJYF0p7K hr0y177M+U2BqhvhEuBg23cEuYWGIVfE7b6FFO5XnIpXnVHCi69dZE/hycvdzjFjHWt1 Po1anGQfdyw2NsncVRnj6Eh11jgHJdtCtbcYXyAWXymQ2veSGQw4sO0+CK90vOCOadEP x3KA== X-Gm-Message-State: AFuF++n1TaC4JlcHaFpXCS2AgkAP6han2ZDxas1LgxXFcEYRDIh7jBh9 6qUPH0YtHcgBgJoQupbW3nFyQ0n1BWVKCtusbh8/QFDP74T2p+ShQGw17vz7ztktmTABRFD1Fss 1fa9e X-Gm-Gg: AYBFou1TVtJ9zmE6Yb3WGx+VQRnUAph3Ha905uCrzo1IjFCu5doqnGW87W6JhpPiT1A XSabjwnfItwRGhv0TPgAEsSQu0qyPd2FA5v59htmhBxWd8qhzcf/ZJx7gJ+ihO58yjIE1J/YW73 eKyPmczy0WNRUcQOerngog1UtOBVN1i/4u1mizbglwN7dSEvyi/n1DBN8jeC5O93ofzeE+iTJAU i88W1gUUJVPXEj50YjO0ki1IMkmkrWjA2yIGeXNOXnwu/OF+XVoeG+Hx+/KmmHRRxBnG2E84YJM MxjWt1/f2IcPgCAmyiAO6zU2Zrekis94Bz0o/HGYHPRgTitQ3fQGJX56xLczFRnnVxbwMzUIUPm geH79lQf6xes84WhsHZ+Ksstzg5nitf9uSFsHB0rNz7ZX84sAq1r+o+teYnnjs1k7SpRa///WmT y1Gqw2E/pknjtjNHytkoCEJPGiuLU35ZDE0dmr15UzfgGizjdCU+MMX4rf07stifW2jAIM X-Received: by 2002:a05:600c:4592:b0:49e:6836:5386 with SMTP id 5b1f17b1804b1-49fd8517e22mr229395e9.0.1790004309676; Mon, 21 Sep 2026 08:25:09 -0700 (PDT) To: devel@lists.libvirt.org Subject: [PATCH 1/2] remote: fix daemon deadlock when a stream event closes its client Date: Mon, 21 Sep 2026 17:25:04 +0200 Message-ID: <20260921152505.1604682-2-den@openvz.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260921152505.1604682-1-den@openvz.org> References: <20260921152505.1604682-1-den@openvz.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-MailFrom: den@openvz.org X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; header-match-devel.lists.libvirt.org-0; emergency; member-moderation Message-ID-Hash: UWOZWYV24Y6H56QD234QAH6HXOK2MY6S X-Message-ID-Hash: UWOZWYV24Y6H56QD234QAH6HXOK2MY6S X-Mailman-Approved-At: Mon, 21 Sep 2026 15:29:42 +0000 CC: den@openvz.org X-Mailman-Version: 3.3.10 Precedence: list List-Id: Development discussions about the libvirt library & tools Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: "Denis V. Lunev via Devel" Reply-To: "Denis V. Lunev" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1790004763071158500 Content-Type: text/plain; charset="utf-8" From: Denis V. Lunev daemonStreamEvent() holds priv->lock for its whole body, and every error path in it calls virNetServerClientClose(). That runs the client close hook synchronously in the same thread, and remoteClientCloseFunc() -> remoteClientFreePrivateCallbacks() takes priv->lock again. The lock is a plain non-recursive mutex, so the thread blocks on itself and never returns. The thread is the daemon main loop, so the host stops answering RPC entirely while its guests keep running. Only a restart of the daemon recovers. Every route from daemonStreamEvent() to the close goes through a virNetServerProgramSend* failure, and those fail once the client is marked wantClose. It therefore needs a client marked for close while one of its streams is still registered and still delivering events, which is why the deadlock has gone unnoticed for so long. It was hit in the field by a stream whose client had just died, but a console or a volume transfer reaches the same line. Move the body into a helper that reports whether the client has to go, and close it in the caller once the lock guard is out of scope. daemonRemoveClientStream() keeps running under the lock, and virNetServerClientClose() only ever needed the client object lock. Fixes: 6386dd897df5 ("remote: add mutex when freeing private callbacks") Signed-off-by: Denis V. Lunev Reviewed-by: Michal Privoznik --- src/remote/remote_daemon_stream.c | 58 +++++++++++++++++++------------ 1 file changed, 36 insertions(+), 22 deletions(-) diff --git a/src/remote/remote_daemon_stream.c b/src/remote/remote_daemon_s= tream.c index 4faaf99a90..0841bb0c78 100644 --- a/src/remote/remote_daemon_stream.c +++ b/src/remote/remote_daemon_stream.c @@ -110,15 +110,16 @@ daemonStreamMessageFinished(virNetMessage *msg, =20 =20 /* - * Callback that gets invoked when a stream becomes writable/readable + * Returns true if the client has to be closed, which the caller does + * after dropping priv->lock. */ -static void -daemonStreamEvent(virStreamPtr st, int events, void *opaque) +static bool +daemonStreamEventLocked(virNetServerClient *client, + virStreamPtr st, + int events) { - virNetServerClient *client =3D opaque; - daemonClientStream *stream; daemonClientPrivate *priv =3D virNetServerClientGetPrivateData(client); - VIR_LOCK_GUARD lock =3D virLockGuardLock(&priv->lock); + daemonClientStream *stream; =20 stream =3D priv->streams; while (stream) { @@ -130,7 +131,7 @@ daemonStreamEvent(virStreamPtr st, int events, void *op= aque) if (!stream) { VIR_WARN("event for client=3D%p stream st=3D%p, but missing stream= state", client, st); virStreamEventRemoveCallback(st); - return; + return false; } =20 VIR_DEBUG("st=3D%p events=3D%d EOF=3D%d closed=3D%d", st, events, stre= am->recvEOF, stream->closed); @@ -139,8 +140,7 @@ daemonStreamEvent(virStreamPtr st, int events, void *op= aque) (events & VIR_STREAM_EVENT_WRITABLE)) { if (daemonStreamHandleWrite(client, stream) < 0) { daemonRemoveClientStream(client, stream); - virNetServerClientClose(client); - return; + return true; } } =20 @@ -149,8 +149,7 @@ daemonStreamEvent(virStreamPtr st, int events, void *op= aque) events =3D events & ~(VIR_STREAM_EVENT_READABLE); if (daemonStreamHandleRead(client, stream) < 0) { daemonRemoveClientStream(client, stream); - virNetServerClientClose(client); - return; + return true; } /* If we detected EOF during read processing, * then clear hangup/error conditions, since @@ -174,8 +173,7 @@ daemonStreamEvent(virStreamPtr st, int events, void *op= aque) if (daemonStreamHandleFinish(client, stream, msg) < 0) { virNetMessageFree(msg); daemonRemoveClientStream(client, stream); - virNetServerClientClose(client); - return; + return true; } break; case VIR_NET_ERROR: @@ -184,8 +182,7 @@ daemonStreamEvent(virStreamPtr st, int events, void *op= aque) if (daemonStreamHandleAbort(client, stream, msg) < 0) { virNetMessageFree(msg); daemonRemoveClientStream(client, stream); - virNetServerClientClose(client); - return; + return true; } break; } @@ -203,8 +200,7 @@ daemonStreamEvent(virStreamPtr st, int events, void *op= aque) stream->recvEOF =3D true; if (!(msg =3D virNetMessageNew(false))) { daemonRemoveClientStream(client, stream); - virNetServerClientClose(client); - return; + return true; } msg->cb =3D daemonStreamMessageFinished; msg->opaque =3D stream; @@ -217,8 +213,7 @@ daemonStreamEvent(virStreamPtr st, int events, void *op= aque) "", 0) < 0) { virNetMessageFree(msg); daemonRemoveClientStream(client, stream); - virNetServerClientClose(client); - return; + return true; } } =20 @@ -258,9 +253,7 @@ daemonStreamEvent(virStreamPtr st, int events, void *op= aque) stream->serial); } daemonRemoveClientStream(client, stream); - if (ret < 0) - virNetServerClientClose(client); - return; + return ret < 0; } =20 if (stream->closed) { @@ -268,6 +261,27 @@ daemonStreamEvent(virStreamPtr st, int events, void *o= paque) } else { daemonStreamUpdateEvents(stream); } + + return false; +} + + +/* + * Callback that gets invoked when a stream becomes writable/readable + */ +static void +daemonStreamEvent(virStreamPtr st, int events, void *opaque) +{ + virNetServerClient *client =3D opaque; + daemonClientPrivate *priv =3D virNetServerClientGetPrivateData(client); + bool needClose =3D false; + + VIR_WITH_MUTEX_LOCK_GUARD(&priv->lock) { + needClose =3D daemonStreamEventLocked(client, st, events); + } + + if (needClose) + virNetServerClientClose(client); } =20 =20 --=20 2.53.0 From nobody Sun Nov 22 16:13:42 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.libvirt.org designates 38.145.34.151 as permitted sender) client-ip=38.145.34.151; envelope-from=devel-bounces@lists.libvirt.org; helo=lists.libvirt.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of lists.libvirt.org designates 38.145.34.151 as permitted sender) smtp.mailfrom=devel-bounces@lists.libvirt.org; dmarc=pass(p=reject dis=none) header.from=lists.libvirt.org ARC-Seal: i=1; a=rsa-sha256; t=1790004888; cv=none; d=zohomail.com; s=zohoarc; b=Y6r+ZLsTugmqlcb2FBAfnAxqwrjHKT6cc2yEkaC99NX1423ZlgBl2FHiSXSPmGM9MxwHsJeKPK4jXPnfRh9cUV3SgD9mClLPlyjcsG4A7junfF2q9KVKDFoYk4mNood8iLfgb660nB1o27e4UiVg2cWUKQTJi1BlqLPMkjGsYUM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1790004888; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Owner:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:Subject:Subject:To:To:Message-Id; bh=j8khnHN5KAGTBNse6DOG2z7NVlrX9ROeHndLUl0detg=; b=KVyIaXHwXjQpCJ05vR3KfHnqJKr3KujNbFieme/h7fS/NE1UIE33a7+KmvaDkOwjpZZZ1GEYD2r6CxqPeYGHbK1F7WaRtomQGWP2zwMfg6TKUqDFOS7ous+teAKtPhhXD50wAymHzBnVdqbMF7QGsiC3VhMxdHzxwsg3kiqGrJs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of lists.libvirt.org designates 38.145.34.151 as permitted sender) smtp.mailfrom=devel-bounces@lists.libvirt.org; dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.libvirt.org (lists.libvirt.org [38.145.34.151]) by mx.zohomail.com with SMTPS id 1790004888586798.2258600194247; Mon, 21 Sep 2026 08:34:48 -0700 (PDT) Received: by lists.libvirt.org (Postfix, from userid 993) id 157CA3FC9D; Mon, 21 Sep 2026 11:34:47 -0400 (EDT) Received: from [172.19.199.16] (unknown [10.16.107.18]) by lists.libvirt.org (Postfix) with ESMTP id 49D503FC7D; Mon, 21 Sep 2026 11:30:01 -0400 (EDT) Received: by lists.libvirt.org (Postfix, from userid 993) id 7EAC33F316; Mon, 21 Sep 2026 11:25:13 -0400 (EDT) Received: from mail-wm2-x11.google.com (mail-wm2-x11.google.com [IPv6:2a00:1450:4864:31::11]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (3072 bits) server-digest SHA256) (No client certificate requested) by lists.libvirt.org (Postfix) with ESMTPS id 42C3A3F337 for ; Mon, 21 Sep 2026 11:25:12 -0400 (EDT) Received: by mail-wm2-x11.google.com with SMTP id 5b1f17b1804b1-49d1ca5b0d6so23380395e9.0 for ; Mon, 21 Sep 2026 08:25:12 -0700 (PDT) Received: from athena.sw.ru ([2a06:5b06:b600:300:5745:5560:519b:4337]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-49fcd0f7e05sm248149245e9.4.2026.09.21.08.25.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2026 08:25:10 -0700 (PDT) X-Spam-Checker-Version: SpamAssassin 4.0.1 (2024-03-26) on lists.libvirt.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,SPF_HELO_NONE autolearn=unavailable autolearn_force=no version=4.0.1 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=openvz.org; s=google; t=1790004311; x=1790609111; darn=lists.libvirt.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=j8khnHN5KAGTBNse6DOG2z7NVlrX9ROeHndLUl0detg=; b=kNp1T/nZgRxq7vy9TDIYcmN2v3ig+mGVb+rCH/DWhRR8peaVOiboL+LU61aSSpSB3B X2fDAcdW7p2AAhC7dw3IQ/HqbbfokFFXUI7crv0LZw7B80AkvoGJVLOQcz8sm4qsOTyb 1YSgNconJXZ/t4zS9zzZIMLIqMvnnBkbKezaaJ27GnovKi0X5NJxws6EdTmcs2IZUZPx eJRqTDplFAczh3HIo/qZUTg5o/oluM+QqHPnzwtdwX6glZxLnqsQiOKbKpb/eudMGI6G 0Os8d0cCS3gze54bwlRC3E+yI3ugm7Wh4046yvvSdlUbUypQTODz8isVwhwASH1sUBY0 CPFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1790004311; x=1790609111; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to:content-type; bh=j8khnHN5KAGTBNse6DOG2z7NVlrX9ROeHndLUl0detg=; b=xrCjQGYgTv0y/HlCLJ4pUj+cjbCWBzOgV08BYJFy5pSj+wugeSqOsNndjPJdLQ3HNB QIUNuiPWkfDUuU6RmKzVnJq6lAJPw8QtThmY282dnkrktcYxJeOh0FsVJnsfPlxP2IVu 3ysOHC5lpIEtOfYllh1FIEDX9ThdLVfEWd30lUqzbqk8b9XYVemrrt+0TOke29r8y0kO xUL91DdnUl3W9AUwr8jin+ZrTG8ZtuEIc0DWeXo1iqAoalXqxZZn8ZVqFpYV6iP3RSQW eNip9pvDKk1rk/qLkycuYWWUyREC84E/ArYFu/s19EmoxRL73wl4gYWs+4phJMAeg5RF PN2Q== X-Gm-Message-State: AFuF++maXIT8LHUX5rkz8zBkc1qKCVtw9zaLkf7UzGXGOnBZA3RkvuBd 3tbNRxR5/OeqHpA/vKmLHwyGPcYE0Lz8MvuViMWMIgbn7ThMZ0ZhfeOQuOIcvbwUkx3ik7pw4/C lf3MZ X-Gm-Gg: AYBFou3gHgxXwgH+dPlayOXicMuOc0xrq9YdwYgIRXNY2oem1rzqx7+1ICm0wrlJGfD W8sFnzrRxN1lPqdH3mFE4Fz7OrsVzvz7k20sto0VEDWj4ghhhhtWOtGE3c0zeWKg+UjB8GBRcdI NKifTP0xsJmd80Pf3beSub0nPI/PnGFB5Y2mFCg3vvFtKNzoMDziASkgKiJR4K366Su5tku3gJ0 hp1wRYDpRFSHUDP/5GKYqZqxdG33MzcupnpvIxcKcZ01eGrtz0SSvfeS/ClmZZ4UHyoyTDY4As1 yd9p5Tt5Xw94SLqm+fBrNSGjf5/kxNwuOBqo/0nmm4RFmnZ+2NzwcFPuEjgPEMs58Mx9mEsJQQR TtvAXopldbRGQPfA5xM+VpVz38KdE7BdiXStvvfwEdzowREndCHvrqrM8XuWg0nZG0KKVRYnQiz 9BnevdXq9pHoxArbgOz8rBbzFXP1j3ZyoL4PeXE2IFf8N1AbBRLgLILcpB2ylxJmHybmG5 X-Received: by 2002:a05:600c:1548:b0:49e:6ac4:b76e with SMTP id 5b1f17b1804b1-49fc574eed4mr161944895e9.30.1790004311032; Mon, 21 Sep 2026 08:25:11 -0700 (PDT) To: devel@lists.libvirt.org Subject: [PATCH 2/2] tests: add daemonstreamtest covering the stream close deadlock Date: Mon, 21 Sep 2026 17:25:05 +0200 Message-ID: <20260921152505.1604682-3-den@openvz.org> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260921152505.1604682-1-den@openvz.org> References: <20260921152505.1604682-1-den@openvz.org> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-MailFrom: den@openvz.org X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; loop; banned-address; header-match-devel.lists.libvirt.org-0; emergency; member-moderation Message-ID-Hash: WX2CKK5JKEOLB53XQWCBLOIP4PXWDQNI X-Message-ID-Hash: WX2CKK5JKEOLB53XQWCBLOIP4PXWDQNI X-Mailman-Approved-At: Mon, 21 Sep 2026 15:29:42 +0000 CC: den@openvz.org X-Mailman-Version: 3.3.10 Precedence: list List-Id: Development discussions about the libvirt library & tools Archived-At: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: From: "Denis V. Lunev via Devel" Reply-To: "Denis V. Lunev" X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1790004891276158500 Content-Type: text/plain; charset="utf-8" From: Denis V. Lunev daemonStreamEvent() must not close the client while holding priv->lock: the close hook takes that lock too and runs in the same thread. A regression there hangs the daemon rather than failing anything. It tests deterministically. virNetServerClientImmediateClose() sets wantClose without closing anything, which is the state every route to the close needs, and an fd stream over a pipe fires as soon as the write end goes away. So the test gives a client a close hook that takes priv->lock, as remoteClientFreePrivateCallbacks() does, registers a stream, marks the client for close and hangs up the pipe. A close under the lock blocks the event loop and the alarm reports it. Building remote_daemon_stream.c into a test pulls in the generated protocol headers through remote_daemon.h, hence the extra sources and the new variable that keeps the daemon listing the file once. Signed-off-by: Denis V. Lunev Reviewed-by: Michal Privoznik --- src/remote/meson.build | 7 +- tests/daemonstreamtest.c | 212 +++++++++++++++++++++++++++++++++++++++ tests/meson.build | 9 ++ 3 files changed, 226 insertions(+), 2 deletions(-) create mode 100644 tests/daemonstreamtest.c diff --git a/src/remote/meson.build b/src/remote/meson.build index 3525bc34b2..a1d858779b 100644 --- a/src/remote/meson.build +++ b/src/remote/meson.build @@ -52,12 +52,15 @@ foreach name : [ 'remote', 'qemu', 'lxc' ] rpc_probe_files +=3D files(protocol_x) endforeach =20 +remote_daemon_stream_sources =3D files( + 'remote_daemon_stream.c', +) + remote_daemon_sources =3D files( 'remote_daemon.c', 'remote_daemon_config.c', 'remote_daemon_dispatch.c', - 'remote_daemon_stream.c', -) +) + remote_daemon_stream_sources =20 remote_daemon_generated =3D [] =20 diff --git a/tests/daemonstreamtest.c b/tests/daemonstreamtest.c new file mode 100644 index 0000000000..586bdcc1e9 --- /dev/null +++ b/tests/daemonstreamtest.c @@ -0,0 +1,212 @@ +/* + * daemonstreamtest.c: test the daemon side of client streams + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library. If not, see + * . + */ + +#include + +#include +#include +#include + +#include "testutils.h" +#include "virerror.h" +#include "virevent.h" +#include "virfdstream.h" +#include "virfile.h" +#include "virthread.h" +#include "virutil.h" +#include "remote/remote_daemon_stream.h" +#include "rpc/virnetserverclient.h" +#include "rpc/virnetserverprogram.h" + +#define VIR_FROM_THIS VIR_FROM_RPC + +#ifndef WIN32 + +#define TEST_PROGRAM 0x11111111 +#define TEST_TIMEOUT 5 + +static bool closeHookDone; + +static void * +testClientPrivNew(virNetServerClient *client G_GNUC_UNUSED, + void *opaque G_GNUC_UNUSED) +{ + daemonClientPrivate *priv =3D g_new0(daemonClientPrivate, 1); + + if (virMutexInit(&priv->lock) < 0) { + g_free(priv); + return NULL; + } + + return priv; +} + + +static void +testClientPrivFree(void *opaque) +{ + daemonClientPrivate *priv =3D opaque; + + virMutexDestroy(&priv->lock); + g_free(priv); +} + + +/* Same locking as remoteClientFreePrivateCallbacks(), which is what the + * daemon installs here. */ +static void +testClientClose(virNetServerClient *client) +{ + daemonClientPrivate *priv =3D virNetServerClientGetPrivateData(client); + VIR_LOCK_GUARD lock =3D virLockGuardLock(&priv->lock); + + closeHookDone =3D true; +} + + +static void +testAlarm(int sig G_GNUC_UNUSED) +{ + static const char msg[] =3D "daemonStreamEvent() deadlocked closing th= e client\n"; + + ignore_value(safewrite(STDERR_FILENO, msg, sizeof(msg) - 1)); + _exit(EXIT_FAILURE); +} + + +/* A stream event delivered for a client that is already marked for close + * must not deadlock the thread running the event loop. The error reply + * cannot be queued to such a client, so the event handler closes it, and + * the close hook takes the same lock the handler holds. */ +static int +testStreamEventClose(const void *opaque G_GNUC_UNUSED) +{ + struct virNetMessageHeader header =3D { .proc =3D 1, .serial =3D 1 }; + virConnectPtr conn =3D NULL; + virNetServerClient *client =3D NULL; + virNetServerProgram *prog =3D NULL; + daemonClientStream *stream =3D NULL; + virNetSocket *sock =3D NULL; + virStreamPtr st =3D NULL; + int pipeFD[2] =3D { -1, -1 }; + int sv[2] =3D { -1, -1 }; + int ret =3D -1; + size_t i; + + closeHookDone =3D false; + + if (!(conn =3D virConnectOpen("test:///default"))) + return -1; + + if (socketpair(PF_UNIX, SOCK_STREAM, 0, sv) < 0) { + virReportSystemError(errno, "%s", "Cannot create socket pair"); + return -1; + } + + if (virNetSocketNewConnectSockFD(sv[0], &sock) < 0) + goto cleanup; + sv[0] =3D -1; + + if (!(client =3D virNetServerClientNew(1, sock, 0, false, 1, NULL, + testClientPrivNew, NULL, + testClientPrivFree, NULL))) + goto cleanup; + + virNetServerClientSetCloseHook(client, testClientClose); + + if (!(prog =3D virNetServerProgramNew(TEST_PROGRAM, 1, NULL, 0))) + goto cleanup; + + if (virPipe(pipeFD) < 0) + goto cleanup; + + if (!(st =3D virStreamNew(conn, VIR_STREAM_NONBLOCK))) + goto cleanup; + + if (virFDStreamOpen(st, pipeFD[0]) < 0) + goto cleanup; + pipeFD[0] =3D -1; + + if (!(stream =3D daemonCreateClientStream(client, st, prog, &header, f= alse))) + goto cleanup; + + if (daemonAddClientStream(client, stream, true) < 0) + goto cleanup; + st =3D NULL; + + virNetServerClientImmediateClose(client); + + /* Hangs up the read end, so the stream reports EOF */ + VIR_FORCE_CLOSE(pipeFD[1]); + + signal(SIGALRM, testAlarm); + alarm(TEST_TIMEOUT); + + for (i =3D 0; i < 100 && !closeHookDone; i++) { + if (virEventRunDefaultImpl() < 0) + break; + } + + alarm(0); + + if (!closeHookDone) { + fprintf(stderr, "Client was not closed by the stream event\n"); + goto cleanup; + } + + ret =3D 0; + + cleanup: + virObjectUnref(st); + if (conn) + virConnectClose(conn); + virObjectUnref(prog); + virObjectUnref(sock); + if (client) + virNetServerClientClose(client); + virObjectUnref(client); + VIR_FORCE_CLOSE(pipeFD[0]); + VIR_FORCE_CLOSE(pipeFD[1]); + VIR_FORCE_CLOSE(sv[0]); + VIR_FORCE_CLOSE(sv[1]); + return ret; +} + + +static int +mymain(void) +{ + int ret =3D 0; + + virEventRegisterDefaultImpl(); + + if (virTestRun("Stream event on a client marked for close", + testStreamEventClose, NULL) < 0) + ret =3D -1; + + return ret =3D=3D 0 ? EXIT_SUCCESS : EXIT_FAILURE; +} +VIR_TEST_MAIN(mymain); +#else +static int +mymain(void) +{ + return EXIT_AM_SKIP; +} +VIR_TEST_MAIN(mymain); +#endif diff --git a/tests/meson.build b/tests/meson.build index 9d300de41d..2a97a18a52 100644 --- a/tests/meson.build +++ b/tests/meson.build @@ -492,6 +492,15 @@ endif =20 if conf.has('WITH_REMOTE') tests +=3D [ + { + 'name': 'daemonstreamtest', + 'sources': [ + 'daemonstreamtest.c', + remote_daemon_stream_sources, + remote_protocol_generated, + ], + 'include': [ remote_inc_dir ], + }, { 'name': 'virnetdaemontest' }, { 'name': 'virnetmessagetest' }, { 'name': 'virnetserverclienttest' }, --=20 2.53.0