From nobody Fri May 3 23:32:33 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.24 as permitted sender) client-ip=209.132.183.24; envelope-from=libvir-list-bounces@redhat.com; helo=mx3-phx2.redhat.com; Authentication-Results: mx.zoho.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.24 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; Return-Path: Received: from mx3-phx2.redhat.com (mx3-phx2.redhat.com [209.132.183.24]) by mx.zohomail.com with SMTPS id 1489597111218671.538316769792; Wed, 15 Mar 2017 09:58:31 -0700 (PDT) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by mx3-phx2.redhat.com (8.13.8/8.13.8) with ESMTP id v2FGtA6s006788; Wed, 15 Mar 2017 12:55:10 -0400 Received: from int-mx13.intmail.prod.int.phx2.redhat.com (int-mx13.intmail.prod.int.phx2.redhat.com [10.5.11.26]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id v2FGt8Eh027649 for ; Wed, 15 Mar 2017 12:55:08 -0400 Received: from t460.redhat.com (ovpn-117-127.ams2.redhat.com [10.36.117.127]) by int-mx13.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id v2FGt6cD012364; Wed, 15 Mar 2017 12:55:07 -0400 From: "Daniel P. Berrange" To: libvir-list@redhat.com Date: Wed, 15 Mar 2017 16:55:04 +0000 Message-Id: <20170315165504.1566-1-berrange@redhat.com> X-Scanned-By: MIMEDefang 2.68 on 10.5.11.26 X-loop: libvir-list@redhat.com Subject: [libvirt] [PATCH] Increase default file handle limits for daemons X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-ZohoMail: RSF_0 Z_629925259 SPT_0 Content-Type: text/plain; charset="utf-8" Linux still defaults to a 1024 open file handle limit. This causes scalability problems for libvirtd / virtlockd / virtlogd on large hosts which might want > 1024 guest to be running. In fact if each guest needs > 1 FD, we can't even get to 500 guests. This is not good enough when we see machines with 100's of physical cores and TBs of RAM. In comparison to other memory requirements of libvirtd & related daemons, the resource usage associated with open file handles is essentially line noise. It is thus reasonable to increase the limits unconditionally for all installs. Signed-off-by: Daniel P. Berrange --- daemon/libvirtd.service.in | 7 +++++-- src/locking/virtlockd.service.in | 4 ++++ src/logging/virtlogd.service.in | 5 +++++ 3 files changed, 14 insertions(+), 2 deletions(-) diff --git a/daemon/libvirtd.service.in b/daemon/libvirtd.service.in index c72dde5..22fc156 100644 --- a/daemon/libvirtd.service.in +++ b/daemon/libvirtd.service.in @@ -24,8 +24,11 @@ ExecStart=3D@sbindir@/libvirtd $LIBVIRTD_ARGS ExecReload=3D/bin/kill -HUP $MAINPID KillMode=3Dprocess Restart=3Don-failure -# Override the maximum number of opened files -#LimitNOFILE=3D2048 +# At least 1 FD per guest, often 2 (eg qemu monitor + qemu agent). +# If we want to support 2048 guests, we'll typically need 4096 FDs +# If changing this, also consider virtlogd.service & virtlockd.service +# limits which are also related to number of guests +LimitNOFILE=3D8192 =20 [Install] WantedBy=3Dmulti-user.target diff --git a/src/locking/virtlockd.service.in b/src/locking/virtlockd.servi= ce.in index 69b568f..c369591 100644 --- a/src/locking/virtlockd.service.in +++ b/src/locking/virtlockd.service.in @@ -13,6 +13,10 @@ ExecReload=3D/bin/kill -USR1 $MAINPID # cause the machine to be fenced (rebooted), so make # sure we discourage OOM killer OOMScoreAdjust=3D-900 +# Needs to allow for max guests * average disks per guest +# libvirtd.service written to expect 4096 guests, so if we +# allow for 4 disks per guest, we get: +LimitNOFILE=3D16384 =20 [Install] Also=3Dvirtlockd.socket diff --git a/src/logging/virtlogd.service.in b/src/logging/virtlogd.service= .in index 09e0740..be039c6 100644 --- a/src/logging/virtlogd.service.in +++ b/src/logging/virtlogd.service.in @@ -13,6 +13,11 @@ ExecReload=3D/bin/kill -USR1 $MAINPID # cause the machine to be fenced (rebooted), so make # sure we discourage OOM killer OOMScoreAdjust=3D-900 +# Need to have at least one file open per guest (eg QEMU +# stdio log), but might be more (eg serial console logs) +# libvirtd.service written to expect 2048 guests, so if we +# guess at 2 log files per guest here (stdio + 1 serial): +LimitNOFILE=3D8192 =20 [Install] Also=3Dvirtlogd.socket --=20 2.9.3 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list