From nobody Thu Mar 28 14:27:37 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1634046133; cv=none; d=zohomail.com; s=zohoarc; b=FqzNj3KMyjkAWVxRgoIZkp310VJYtNzXUhsdnDdK2B4ZKI8l94yRaw2zEtfIIug5GLrRUgRGS+air/Cfd8M+aEoRf1N6dIcZyUOy9Qr3KaZjmJFd7EqnmtyaOadFxhVhM8sNC/ib5uTKesGiFuVlOJbbUUOFwiusSuabDNCdiaU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1634046133; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=Q5L7RANFuY0lTjcILncAay9zDfSZ+zR/cGF7re2yY04=; b=MW8X6rvIYc2sRR6sTE0ulNUA7nLZGQzwbeBhh7SpoEZFR+PWvxvx2rE+afLBAx6rzdZLxM4dYd9A8FFTqNEHUe4YXFPWfzC7NSLBajII4IEYrPSF1QPCUfaQShhiFqZNJEVa7PAmh13x7WDJwwLWqYXDP+5Pa3dmOjb3gQK5v3s= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 163404613378794.73821491109845; Tue, 12 Oct 2021 06:42:13 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.207219.362977 (Exim 4.92) (envelope-from ) id 1maI2U-0001iQ-A1; Tue, 12 Oct 2021 13:41:54 +0000 Received: by outflank-mailman (output) from mailman id 207219.362977; Tue, 12 Oct 2021 13:41:54 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1maI2U-0001iJ-5q; Tue, 12 Oct 2021 13:41:54 +0000 Received: by outflank-mailman (input) for mailman id 207219; Tue, 12 Oct 2021 13:41:52 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1maI2S-0001i8-Tf for xen-devel@lists.xenproject.org; Tue, 12 Oct 2021 13:41:52 +0000 Received: from smtp-out1.suse.de (unknown [195.135.220.28]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 6e277c9a-b0a5-479e-9360-bdd38a5ac81b; Tue, 12 Oct 2021 13:41:52 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 2C83E22185; Tue, 12 Oct 2021 13:41:51 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id EC2B813E43; Tue, 12 Oct 2021 13:41:50 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id cOxCOJ6QZWGabgAAMHmgww (envelope-from ); Tue, 12 Oct 2021 13:41:50 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 6e277c9a-b0a5-479e-9360-bdd38a5ac81b DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1634046111; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Q5L7RANFuY0lTjcILncAay9zDfSZ+zR/cGF7re2yY04=; b=k3qCzLkwOK9uYRQWGTlPf0TWQ5zGIPjtZ04AryLo8EoEVfngOLI3Nw1SrV2nR+gNavOSnw AsVQXpzUIGDAkXvkBBL/Ov0TeF3JQOmuFsFxIU5wEz5Thb1mjedgvbcbIpEC6wyPc9kYjl GmFocuIifMbKqo8SPNtKUZoVyovSsQ8= From: Juergen Gross To: xen-devel@lists.xenproject.org Cc: Juergen Gross , Ian Jackson , Wei Liu Subject: [PATCH v6 1/2] tools/xenstore: set oom score for xenstore daemon on Linux Date: Tue, 12 Oct 2021 15:41:47 +0200 Message-Id: <20211012134148.6280-2-jgross@suse.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20211012134148.6280-1-jgross@suse.com> References: <20211012134148.6280-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) X-ZM-MESSAGEID: 1634047037698100001 Content-Type: text/plain; charset="utf-8" Xenstored is absolutely mandatory for a Xen host and it can't be restarted, so being killed by OOM-killer in case of memory shortage is to be avoided. Set /proc/$pid/oom_score_adj (if available) per default to -500 (this translates to 50% of dom0 memory size) in order to allow xenstored to use large amounts of memory without being killed. The percentage of dom0 memory above which the oom killer is allowed to kill xenstored can be set via XENSTORED_OOM_MEM_THRESHOLD in xencommons. Make sure the pid file isn't a left-over from a previous run delete it before starting xenstored. Signed-off-by: Juergen Gross Reviewed-by: Ian Jackson --- V2: - set oom score from launch script (Julien Grall) - split off open file descriptor limit setting (Julien Grall) V3: - make oom killer threshold configurable (Julien Grall) V4: - extend comment (Ian Jackson) --- tools/hotplug/Linux/init.d/sysconfig.xencommons.in | 9 +++++++++ tools/hotplug/Linux/launch-xenstore.in | 6 ++++++ 2 files changed, 15 insertions(+) diff --git a/tools/hotplug/Linux/init.d/sysconfig.xencommons.in b/tools/hot= plug/Linux/init.d/sysconfig.xencommons.in index 00cf7f91d4..b83101ab7e 100644 --- a/tools/hotplug/Linux/init.d/sysconfig.xencommons.in +++ b/tools/hotplug/Linux/init.d/sysconfig.xencommons.in @@ -48,6 +48,15 @@ XENSTORED_ARGS=3D # Only evaluated if XENSTORETYPE is "daemon". #XENSTORED_TRACE=3D[yes|on|1] =20 +## Type: integer +## Default: 50 +# +# Percentage of dom0 memory size the xenstore daemon can use before the +# OOM killer is allowed to kill it. +# The specified value is multiplied by -10 and echoed to +# /proc/PID/oom_score_adj. +#XENSTORED_OOM_MEM_THRESHOLD=3D50 + ## Type: string ## Default: @LIBEXEC@/boot/xenstore-stubdom.gz # diff --git a/tools/hotplug/Linux/launch-xenstore.in b/tools/hotplug/Linux/l= aunch-xenstore.in index 019f9d6f4d..1747c96065 100644 --- a/tools/hotplug/Linux/launch-xenstore.in +++ b/tools/hotplug/Linux/launch-xenstore.in @@ -59,11 +59,17 @@ test -f @CONFIG_DIR@/@CONFIG_LEAF_DIR@/xencommons && . = @CONFIG_DIR@/@CONFIG_LEAF echo "No xenstored found" exit 1 } + [ -z "$XENSTORED_OOM_MEM_THRESHOLD" ] || XENSTORED_OOM_MEM_THRESHOLD=3D50 + XS_OOM_SCORE=3D-$(($XENSTORED_OOM_MEM_THRESHOLD * 10)) + + rm -f @XEN_RUN_DIR@/xenstored.pid =20 echo -n Starting $XENSTORED... $XENSTORED --pid-file @XEN_RUN_DIR@/xenstored.pid $XENSTORED_ARGS =20 systemd-notify --booted 2>/dev/null || timeout_xenstore $XENSTORED || exi= t 1 + XS_PID=3D`cat @XEN_RUN_DIR@/xenstored.pid` + echo $XS_OOM_SCORE >/proc/$XS_PID/oom_score_adj =20 exit 0 } --=20 2.26.2 From nobody Thu Mar 28 14:27:37 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass(p=quarantine dis=none) header.from=suse.com ARC-Seal: i=1; a=rsa-sha256; t=1634046134; cv=none; d=zohomail.com; s=zohoarc; b=mAJa+iLhIN5l6hNVw7Pug86ZZyE4dq5TJ8jf/S6Dj3ORFYykGrGYiSH5qMtU/Qmlju2E+AkaJJNaOB/8PrRJg/9frKeNQPHmWevi+nTFv6imz6g8HTWgVbdW0yoqSjcssXC26YBN0qeEYGIckimeLWZ5/0vxEg2FSfp0gGJJ/gg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1634046134; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=SEIm+LDplzdzplbZtDj1JKs78X3CX933Us/EMZK8kUQ=; b=aL0p/n7MGUqgw/ZfsZu2fVMvz2m+fBhHOTx+jb/4y/AYBv33wd9qcUL92ryM4JCJcd9Ncnzuq2d97iBqJvuq7pteGVS12zamUNtdhwtU0R/JLykT+cbNosxZkQgzt/4vz0ybBX+zxLR4HDNyRCXrx7+TAlB9791hpwEM8oCG+dU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1634046134298770.8081456888102; Tue, 12 Oct 2021 06:42:14 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.207221.362999 (Exim 4.92) (envelope-from ) id 1maI2Y-0002HP-Vl; Tue, 12 Oct 2021 13:41:58 +0000 Received: by outflank-mailman (output) from mailman id 207221.362999; Tue, 12 Oct 2021 13:41:58 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1maI2Y-0002HG-SP; Tue, 12 Oct 2021 13:41:58 +0000 Received: by outflank-mailman (input) for mailman id 207221; Tue, 12 Oct 2021 13:41:57 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1maI2X-0001i7-N1 for xen-devel@lists.xenproject.org; Tue, 12 Oct 2021 13:41:57 +0000 Received: from smtp-out2.suse.de (unknown [195.135.220.29]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 27ae2b7f-2b62-11ec-8129-12813bfff9fa; Tue, 12 Oct 2021 13:41:52 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 51D2F200AD; Tue, 12 Oct 2021 13:41:51 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 2774F13BC9; Tue, 12 Oct 2021 13:41:51 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id aLQ6CJ+QZWGabgAAMHmgww (envelope-from ); Tue, 12 Oct 2021 13:41:51 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 27ae2b7f-2b62-11ec-8129-12813bfff9fa DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1634046111; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SEIm+LDplzdzplbZtDj1JKs78X3CX933Us/EMZK8kUQ=; b=DeH1SPQanWNTalBCfT28ybemBt/DXolwcq5aIAsmOeLi69vKGr6llMAujDPpxgEj7UaG9I WrgWUnDVeIJfO6pbffBCNn8fwYAE8OeO2Y9nhLhVzUQ8PWtUDKcCWq5uRcepdOi9FOyhbw YifqJlTD/O76cBP2JIhEzv4Zw8SllxI= From: Juergen Gross To: xen-devel@lists.xenproject.org Cc: Juergen Gross , Ian Jackson , Wei Liu Subject: [PATCH v6 2/2] tools/xenstore: set open file descriptor limit for xenstored Date: Tue, 12 Oct 2021 15:41:48 +0200 Message-Id: <20211012134148.6280-3-jgross@suse.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20211012134148.6280-1-jgross@suse.com> References: <20211012134148.6280-1-jgross@suse.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @suse.com) X-ZM-MESSAGEID: 1634046135771100005 Content-Type: text/plain; charset="utf-8" Add a configuration item for the maximum number of open file descriptors xenstored should be allowed to have. The default should be "unlimited" in order not to restrict xenstored in the number of domains it can support, but unfortunately the kernel is normally limiting the maximum value via /proc/sys/fs/nr_open [1], [2]. So check that file to exist and if it does, limit the maximum value to the one specified by /proc/sys/fs/nr_open. As an aid for the admin configuring the value add a comment specifying the common needs of xenstored for the different domain types. [1]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/com= mit/?id=3D60fd760fb9ff7034360bab7137c917c0330628c2 [2]: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/com= mit/?id=3D0c2d64fb6cae9aae480f6a46cfe79f8d7d48b59f Signed-off-by: Juergen Gross Reviewed-by: Ian Jackson --- V2: - set ulimit form launch script (Julien Grall) - split off from original patch (Julien Grall) V4: - switch to directly configuring the limit of file descriptors instead of domains (Ian Jackson) V5: - use /proc/sys/fs/nr_open (Ian Jackson) V6: - start daemon directly via prlimit --- .../Linux/init.d/sysconfig.xencommons.in | 13 ++++++++++++ tools/hotplug/Linux/launch-xenstore.in | 21 ++++++++++++++++++- 2 files changed, 33 insertions(+), 1 deletion(-) diff --git a/tools/hotplug/Linux/init.d/sysconfig.xencommons.in b/tools/hot= plug/Linux/init.d/sysconfig.xencommons.in index b83101ab7e..433e4849af 100644 --- a/tools/hotplug/Linux/init.d/sysconfig.xencommons.in +++ b/tools/hotplug/Linux/init.d/sysconfig.xencommons.in @@ -32,6 +32,19 @@ # Changing this requires a reboot to take effect. #XENSTORED=3D@XENSTORED@ =20 +## Type: string +## Default: unlimited +# +# Select maximum number of file descriptors xenstored is allowed to have +# opened at one time. +# For each HVM domain xenstored might need up to 5 open file descriptors, +# PVH and PV domains will require up to 3 open file descriptors. Additiona= lly +# 20-30 file descriptors will be opened for internal uses. +# The specified value (including "unlimited") will be capped by the conten= ts +# of /proc/sys/fs/nr_open if existing. +# Only evaluated if XENSTORETYPE is "daemon". +#XENSTORED_MAX_OPEN_FDS=3Dunlimited + ## Type: string ## Default: "" # diff --git a/tools/hotplug/Linux/launch-xenstore.in b/tools/hotplug/Linux/l= aunch-xenstore.in index 1747c96065..8438af9977 100644 --- a/tools/hotplug/Linux/launch-xenstore.in +++ b/tools/hotplug/Linux/launch-xenstore.in @@ -54,6 +54,7 @@ test -f @CONFIG_DIR@/@CONFIG_LEAF_DIR@/xencommons && . @C= ONFIG_DIR@/@CONFIG_LEAF =20 [ "$XENSTORETYPE" =3D "daemon" ] && { [ -z "$XENSTORED_TRACE" ] || XENSTORED_ARGS=3D"$XENSTORED_ARGS -T @XEN_LO= G_DIR@/xenstored-trace.log" + [ -z "$XENSTORED_MAX_OPEN_FDS" ] && XENSTORED_MAX_OPEN_FDS=3Dunlimited [ -z "$XENSTORED" ] && XENSTORED=3D@XENSTORED@ [ -x "$XENSTORED" ] || { echo "No xenstored found" @@ -62,10 +63,28 @@ test -f @CONFIG_DIR@/@CONFIG_LEAF_DIR@/xencommons && . = @CONFIG_DIR@/@CONFIG_LEAF [ -z "$XENSTORED_OOM_MEM_THRESHOLD" ] || XENSTORED_OOM_MEM_THRESHOLD=3D50 XS_OOM_SCORE=3D-$(($XENSTORED_OOM_MEM_THRESHOLD * 10)) =20 + [ "$XENSTORED_MAX_OPEN_FDS" =3D "unlimited" ] || { + [ -z "${XENSTORED_MAX_OPEN_FDS//[0-9]}" ] && + [ -n "$XENSTORED_MAX_OPEN_FDS" ] || { + echo "XENSTORED_MAX_OPEN_FDS=3D$XENSTORED_MAX_OPEN_FDS invalid" + echo "Setting to default \"unlimited\"." + XENSTORED_MAX_OPEN_FDS=3Dunlimited + } + } + [ -r /proc/sys/fs/nr_open ] && { + MAX_FDS=3D`cat /proc/sys/fs/nr_open` + [ "$XENSTORED_MAX_OPEN_FDS" =3D "unlimited" ] && XENSTORED_MAX_OPEN_FDS= =3D$MAX_FDS + [ $XENSTORED_MAX_OPEN_FDS -gt $MAX_FDS ] && { + echo "XENSTORED_MAX_OPEN_FDS exceeds system limit." + echo "Setting to \"$MAX_FDS\"." + XENSTORED_MAX_OPEN_FDS=3D$MAX_FDS + } + } + rm -f @XEN_RUN_DIR@/xenstored.pid =20 echo -n Starting $XENSTORED... - $XENSTORED --pid-file @XEN_RUN_DIR@/xenstored.pid $XENSTORED_ARGS + prlimit --nofile=3D$XENSTORED_MAX_OPEN_FDS $XENSTORED --pid-file @XEN_RUN= _DIR@/xenstored.pid $XENSTORED_ARGS =20 systemd-notify --booted 2>/dev/null || timeout_xenstore $XENSTORED || exi= t 1 XS_PID=3D`cat @XEN_RUN_DIR@/xenstored.pid` --=20 2.26.2