From nobody Fri May 17 02:41:49 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) client-ip=170.10.133.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1665147400; cv=none; d=zohomail.com; s=zohoarc; b=g2k22oqJQCLwwEUAKQZC1FVXDfXg4/ScvJvYzRLxtga1sVF2BU5tY5YJj6p0qCGCCEEgHOvcQfhxfKfdyxNonUVWMjWopaNF+6jubCg04+6+nsMqSYYfGW27DhvPsB2ZSsFn6j7O/734i90/ZCkE6/1HAYUgBFbIkb8OJTsDP8Q= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1665147400; h=Content-Type:Content-Transfer-Encoding:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=QDoJoVA+j/gaATzJVN2fIllgV0uslv862Y8wmtjCp/g=; b=nCDH2ei6yI4WKzVyS+qlQK8wy9pUFlTvjYX/gEWDE4yTcDkBwwfWtjGlGFxdAv4/AuAL5vQRxFWz/3X5GJ0nZXjX1bnWAigKP2YhsB45PEh9DwGBjEkiBqfhHSvqBLel5n3SuVIglwCvWQTtyhFL1XuWSv6jG6Z0RmNlCgMrgdk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mx.zohomail.com with SMTPS id 1665147400425330.4383104287256; Fri, 7 Oct 2022 05:56:40 -0700 (PDT) Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-610-xlB5POKrNHSko0viiRGEKQ-1; Fri, 07 Oct 2022 08:56:35 -0400 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4EF54185A7A4; Fri, 7 Oct 2022 12:56:33 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (unknown [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8218A14583FB; Fri, 7 Oct 2022 12:56:31 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 289401946A52; Fri, 7 Oct 2022 12:56:31 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id B03801946A51 for ; Fri, 7 Oct 2022 12:56:29 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id A14E2145BA50; Fri, 7 Oct 2022 12:56:29 +0000 (UTC) Received: from localhost.localdomain.com (unknown [10.33.36.42]) by smtp.corp.redhat.com (Postfix) with ESMTP id 184E6145BA4F; Fri, 7 Oct 2022 12:56:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1665147399; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=QDoJoVA+j/gaATzJVN2fIllgV0uslv862Y8wmtjCp/g=; b=Ppz0XCIha336Q20HHpoZ5HtZLgJmBYTE1q7L+HowGVW2ThcoBiX2BNTA6HQpRlWm79oLbA jTBI76PB5tPT2WrZ/V4UhmBW8M9Ka7MuVEhzxgzSWzc9/iqO7HYElt0ocbWvyPPuAJKyVU rQYWrQHloHsO3g2StcXIfb3Hz/XnhIg= X-MC-Unique: xlB5POKrNHSko0viiRGEKQ-1 X-Original-To: libvir-list@listman.corp.redhat.com From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: libvir-list@redhat.com Subject: [libvirt PATCH v2] src: warn if client hits the max requests limit Date: Fri, 7 Oct 2022 13:56:27 +0100 Message-Id: <20221007125627.1473332-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libvir-list-bounces@redhat.com Sender: "libvir-list" X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1665147402580100001 Since they are simply normal RPC messages, the keep alive packets are subject to the "max_client_requests" limit just like any API calls. Thus, if a client hits the 'max_client_requests' limit and all the pending API calls take a long time to complete, it may result in keep-alives firing and dropping the client connection. This has been seen by a number of users with the default value of max_client_requests=3D5, by issuing 5 concurrent live migration operations. By printing a warning message when this happens, admins will be alerted to the fact that their active clients are exceeding the default client requests limit. Signed-off-by: Daniel P. Berrang=C3=A9 Reviewed-by: Peter Krempa --- In v2: - now only warn once per client to avoid log spam src/rpc/virnetserverclient.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/rpc/virnetserverclient.c b/src/rpc/virnetserverclient.c index 30f6af7be5..c9a4eb521e 100644 --- a/src/rpc/virnetserverclient.c +++ b/src/rpc/virnetserverclient.c @@ -93,6 +93,9 @@ struct _virNetServerClient * throttling calculations */ size_t nrequests; size_t nrequests_max; + /* True if we've warned about nrequests hittin + * the server limit already */ + bool nrequests_warning; /* Zero or one messages being received. Zero if * nrequests >=3D max_clients and throttling */ virNetMessage *rx; @@ -1261,6 +1264,11 @@ static virNetMessage *virNetServerClientDispatchRead= (virNetServerClient *client) client->rx->buffer =3D g_new0(char, client->rx->bufferLeng= th); client->nrequests++; } + } else if (!client->nrequests_warning) { + client->nrequests_warning =3D true; + VIR_WARN("Client hit max requests limit %zd. This may result " + "in keep-alive timeouts. Consider tuning the " + "max_client_requests server parameter", client->nrequ= ests); } virNetServerClientUpdateEvent(client); =20 --=20 2.37.3