[v3] tools/xenstore: set resource limits of xenstored

[PATCH v3 1/2] tools/xenstore: set oom score for xenstore daemon on Linux

Posted by Juergen Gross 4 years, 6 months ago

Xenstored is absolutely mandatory for a Xen host and it can't be
restarted, so being killed by OOM-killer in case of memory shortage is
to be avoided.

Set /proc/$pid/oom_score_adj (if available) per default to -500 (this
translates to 50% of dom0 memory size) in order to allow xenstored to
use large amounts of memory without being killed.

The percentage of dom0 memory above which the oom killer is allowed to
kill xenstored can be set via XENSTORED_OOM_MEM_THRESHOLD in
xencommons.

Make sure the pid file isn't a left-over from a previous run delete it
before starting xenstored.

Signed-off-by: Juergen Gross <jgross@suse.com>
---
V2:
- set oom score from launch script (Julien Grall)
- split off open file descriptor limit setting (Julien Grall)
V3:
- make oom killer threshold configurable (Julien Grall)
---
 tools/hotplug/Linux/init.d/sysconfig.xencommons.in | 7 +++++++
 tools/hotplug/Linux/launch-xenstore.in             | 6 ++++++
 2 files changed, 13 insertions(+)

diff --git a/tools/hotplug/Linux/init.d/sysconfig.xencommons.in b/tools/hotplug/Linux/init.d/sysconfig.xencommons.in
index 00cf7f91d4..5ad4fe0818 100644
--- a/tools/hotplug/Linux/init.d/sysconfig.xencommons.in
+++ b/tools/hotplug/Linux/init.d/sysconfig.xencommons.in
@@ -48,6 +48,13 @@ XENSTORED_ARGS=
 # Only evaluated if XENSTORETYPE is "daemon".
 #XENSTORED_TRACE=[yes|on|1]
 
+## Type: integer
+## Default: 50
+#
+# Percentage of dom0 memory size the xenstore daemon can use before the
+# OOM killer is allowed to kill it.
+#XENSTORED_OOM_MEM_THRESHOLD=50
+
 ## Type: string
 ## Default: @LIBEXEC@/boot/xenstore-stubdom.gz
 #
diff --git a/tools/hotplug/Linux/launch-xenstore.in b/tools/hotplug/Linux/launch-xenstore.in
index 019f9d6f4d..1747c96065 100644
--- a/tools/hotplug/Linux/launch-xenstore.in
+++ b/tools/hotplug/Linux/launch-xenstore.in
@@ -59,11 +59,17 @@ test -f @CONFIG_DIR@/@CONFIG_LEAF_DIR@/xencommons && . @CONFIG_DIR@/@CONFIG_LEAF
 		echo "No xenstored found"
 		exit 1
 	}
+	[ -z "$XENSTORED_OOM_MEM_THRESHOLD" ] || XENSTORED_OOM_MEM_THRESHOLD=50
+	XS_OOM_SCORE=-$(($XENSTORED_OOM_MEM_THRESHOLD * 10))
+
+	rm -f @XEN_RUN_DIR@/xenstored.pid
 
 	echo -n Starting $XENSTORED...
 	$XENSTORED --pid-file @XEN_RUN_DIR@/xenstored.pid $XENSTORED_ARGS
 
 	systemd-notify --booted 2>/dev/null || timeout_xenstore $XENSTORED || exit 1
+	XS_PID=`cat @XEN_RUN_DIR@/xenstored.pid`
+	echo $XS_OOM_SCORE >/proc/$XS_PID/oom_score_adj
 
 	exit 0
 }
-- 
2.26.2

[PATCH v3 1/2] tools/xenstore: set oom score for xenstore daemon on Linux

Posted by Ian Jackson 4 years, 6 months ago

Juergen Gross writes ("[PATCH v3 1/2] tools/xenstore: set oom score for xenstore daemon on Linux"):
> Xenstored is absolutely mandatory for a Xen host and it can't be
> restarted, so being killed by OOM-killer in case of memory shortage is
> to be avoided.
> 
> Set /proc/$pid/oom_score_adj (if available) per default to -500 (this
> translates to 50% of dom0 memory size) in order to allow xenstored to
> use large amounts of memory without being killed.
...
> +## Type: integer
> +## Default: 50
> +#
> +# Percentage of dom0 memory size the xenstore daemon can use before the
> +# OOM killer is allowed to kill it.
> +#XENSTORED_OOM_MEM_THRESHOLD=50
> +
>  ## Type: string
>  ## Default: @LIBEXEC@/boot/xenstore-stubdom.gz

Thanks for working on this.  I approve of the principle.

I have one question about detail:

>  	}
> +	[ -z "$XENSTORED_OOM_MEM_THRESHOLD" ] || XENSTORED_OOM_MEM_THRESHOLD=50
> +	XS_OOM_SCORE=-$(($XENSTORED_OOM_MEM_THRESHOLD * 10))
> +
> +	rm -f @XEN_RUN_DIR@/xenstored.pid
...
> +	XS_PID=`cat @XEN_RUN_DIR@/xenstored.pid`
> +	echo $XS_OOM_SCORE >/proc/$XS_PID/oom_score_adj

The effect of all this is that the value specified in
XENSTORED_OOM_MEM_THRESHOLD is transformed before being echoed into
/proc, by being multiplied by -10.

Of course an alternative would be to ask the user to specify the
tuneable directly but given its rather more obscure semantics I think
it is reasonable to have this done by the script.

But maybe we could add something to the doc comment ?

Eg
  # (The specified value is multiplied by -10 and echoed into
  # /proc/PID/oom_score_adj.)

?

Thanks,
Ian.

Re: [PATCH v3 1/2] tools/xenstore: set oom score for xenstore daemon on Linux

Posted by Juergen Gross 4 years, 5 months ago

On 30.07.21 15:26, Ian Jackson wrote:
> Juergen Gross writes ("[PATCH v3 1/2] tools/xenstore: set oom score for xenstore daemon on Linux"):
>> Xenstored is absolutely mandatory for a Xen host and it can't be
>> restarted, so being killed by OOM-killer in case of memory shortage is
>> to be avoided.
>>
>> Set /proc/$pid/oom_score_adj (if available) per default to -500 (this
>> translates to 50% of dom0 memory size) in order to allow xenstored to
>> use large amounts of memory without being killed.
> ...
>> +## Type: integer
>> +## Default: 50
>> +#
>> +# Percentage of dom0 memory size the xenstore daemon can use before the
>> +# OOM killer is allowed to kill it.
>> +#XENSTORED_OOM_MEM_THRESHOLD=50
>> +
>>   ## Type: string
>>   ## Default: @LIBEXEC@/boot/xenstore-stubdom.gz
> 
> Thanks for working on this.  I approve of the principle.
> 
> I have one question about detail:
> 
>>   	}
>> +	[ -z "$XENSTORED_OOM_MEM_THRESHOLD" ] || XENSTORED_OOM_MEM_THRESHOLD=50
>> +	XS_OOM_SCORE=-$(($XENSTORED_OOM_MEM_THRESHOLD * 10))
>> +
>> +	rm -f @XEN_RUN_DIR@/xenstored.pid
> ...
>> +	XS_PID=`cat @XEN_RUN_DIR@/xenstored.pid`
>> +	echo $XS_OOM_SCORE >/proc/$XS_PID/oom_score_adj
> 
> The effect of all this is that the value specified in
> XENSTORED_OOM_MEM_THRESHOLD is transformed before being echoed into
> /proc, by being multiplied by -10.

Yes.

> Of course an alternative would be to ask the user to specify the
> tuneable directly but given its rather more obscure semantics I think
> it is reasonable to have this done by the script.

Correct. Otherwise the user would need to know about the oom_score_adj
ABI.

> But maybe we could add something to the doc comment ?
> 
> Eg
>    # (The specified value is multiplied by -10 and echoed into
>    # /proc/PID/oom_score_adj.)
> 
> ?

Why? This is an internal implementation detail. I don't see why the
user needs to know how this is accomplished. What is unclear with the
XENSTORED_OOM_MEM_THRESHOLD semantics as described?

There is no other parameter with an explanation how it's semantics are
being accomplished.


Juergen

Re: [PATCH v3 1/2] tools/xenstore: set oom score for xenstore daemon on Linux

Posted by Ian Jackson 4 years, 4 months ago

Juergen Gross writes ("Re: [PATCH v3 1/2] tools/xenstore: set oom score for xenstore daemon on Linux"):
> Correct. Otherwise the user would need to know about the oom_score_adj
> ABI.

I think they might know about it.  In particular, sysadmins might well
be used to configuring this directly (for other daemons).

> On 30.07.21 15:26, Ian Jackson wrote:
> > But maybe we could add something to the doc comment ?
> > 
> > Eg
> >    # (The specified value is multiplied by -10 and echoed into
> >    # /proc/PID/oom_score_adj.)
> > 
> > ?
> 
> Why? This is an internal implementation detail. I don't see why the
> user needs to know how this is accomplished. What is unclear with the
> XENSTORED_OOM_MEM_THRESHOLD semantics as described?

The underlying interface is both also-publicly-exposed, and has a
nonobvious mapping.  Speaking as a sometime sysadmin, I would often
appreciate something like this.

> There is no other parameter with an explanation how it's semantics are
> being accomplished.

I don't think the other parameters are as strange as this.

Ian.