From nobody Fri May 17 12:14:34 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) client-ip=170.10.133.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1661356018; cv=none; d=zohomail.com; s=zohoarc; b=BcVUJNMfeeVzQFE/Y/bPQl/n8YJpg7/qlYHeuZn2TOVKGjTIdB56ZhVWZ5w7nzht2Gl3isLEYeWKY+RV4Que2yC6QTTHZHvdprn6wi50Kt/nfcbbFgioZ5Yv++7Gw78bby+OluWPRL4EQTGA22mTqgBZKXOAc22EsnEj+QvON/M= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1661356018; h=Content-Type:Content-Transfer-Encoding:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=Akuef5LqFj9jFKeK+NJdfQlqsD4z1/3/Pe/GwhcXo24=; b=dW5RH7+gBR+sUchtHUjibMHlf5yPOPDLOTPmL9YMCXfdLaUyf1cGyTyqGEy4mYwsbs4auiBsHEzFvWXl7dJ0DsuAmHn3l0aIP+qaQwEtgh8EfwTngVXNxEyb5j06TMmSwxu2aSLP4zWQGdSu2z83MMymaBBOhFVlckp57CSFbYY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mx.zohomail.com with SMTPS id 1661356018435927.2244640429609; Wed, 24 Aug 2022 08:46:58 -0700 (PDT) Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-142-jQITFBj7MrOQ_nD04YyUsw-1; Wed, 24 Aug 2022 11:46:48 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 478A9811E83; Wed, 24 Aug 2022 15:46:46 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (unknown [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id AB27C1415137; Wed, 24 Aug 2022 15:46:45 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 8FD6C1946A5B; Wed, 24 Aug 2022 15:46:45 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id E3C1A1946A42 for ; Wed, 24 Aug 2022 15:21:04 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id D5C47492CA5; Wed, 24 Aug 2022 15:21:04 +0000 (UTC) Received: from localhost.localdomain.com (unknown [10.33.36.34]) by smtp.corp.redhat.com (Postfix) with ESMTP id 53A5A492C3B; Wed, 24 Aug 2022 15:21:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1661356017; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=Akuef5LqFj9jFKeK+NJdfQlqsD4z1/3/Pe/GwhcXo24=; b=hu7+b/8aC64Zn5wse8IuxDl5CPnDvZMXvZCCKWWZwvXSV+sBQO4JbqeanHSW6pxqRKcW/M 4HVT1+cEQblbMMoVZ8/zFJn18xH0bPZHIG/Fl0JQ8Ur2RN3kPNwVSG4b63a9uzozcyQoWT CJLsO5rEw5cjieI2wKndiSZGzp+SeeM= X-MC-Unique: jQITFBj7MrOQ_nD04YyUsw-1 X-Original-To: libvir-list@listman.corp.redhat.com From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: libvir-list@redhat.com Subject: [libvirt PATCH] src: warn if client hits the max requests limit Date: Wed, 24 Aug 2022 16:21:02 +0100 Message-Id: <20220824152102.700487-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libvir-list-bounces@redhat.com Sender: "libvir-list" X-Scanned-By: MIMEDefang 2.85 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1661356020604100001 Since they are simply normal RPC messages, the keep alive packets are subject to the "max_client_requests" limit just like any API calls. Thus, if a client hits the 'max_client_requests' limit and all the pending API calls take a long time to complete, it may result in keep-alives firing and dropping the client connection. This has been seen by a number of users with the default value of max_client_requests=3D5, by issuing 5 concurrent live migration operations. By printing a warning message when this happens, admins will be alerted to the fact that their active clients are exceeding the default client requests limit. Signed-off-by: Daniel P. Berrang=C3=A9 --- I'm a little wary of this change. If we use anything less than VIR_WARN it is not going to be useful, as we need it visible by default. At the same time though I'm concerned that this might expose very many deployments using an unreasonably low max_client_requests value for their workload. For example OpenStack deployment tools have often left this on the default setting and have been known to exceed it with live migration running concurrently. One possible optimization would be to only issue this warning once per connected client, so we don't spam repeatedly ? src/rpc/virnetserverclient.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/rpc/virnetserverclient.c b/src/rpc/virnetserverclient.c index d57ca07167..0d82726194 100644 --- a/src/rpc/virnetserverclient.c +++ b/src/rpc/virnetserverclient.c @@ -1259,6 +1259,10 @@ static virNetMessage *virNetServerClientDispatchRead= (virNetServerClient *client) client->rx->buffer =3D g_new0(char, client->rx->bufferLeng= th); client->nrequests++; } + } else { + VIR_WARN("Client hit max requests limit %zd. This may result " + "in keep-alive timeouts. Consider tuning the " + "max_client_requests server parameter", client->nrequ= ests); } virNetServerClientUpdateEvent(client); =20 --=20 2.37.2