[libvirt] [PATCH 8/8] docs: Improve documentation related to memory locking

Andrea Bolognani posted 8 patches 8 years, 10 months ago
There is a newer version of this series
[libvirt] [PATCH 8/8] docs: Improve documentation related to memory locking
Posted by Andrea Bolognani 8 years, 10 months ago
---
 docs/formatdomain.html.in | 36 +++++++++++++++++++++++-------------
 1 file changed, 23 insertions(+), 13 deletions(-)

diff --git a/docs/formatdomain.html.in b/docs/formatdomain.html.in
index 4a3123e..5906de6 100644
--- a/docs/formatdomain.html.in
+++ b/docs/formatdomain.html.in
@@ -937,14 +937,18 @@
       <dt><code>locked</code></dt>
       <dd>When set and supported by the hypervisor, memory pages belonging
         to the domain will be locked in host's memory and the host will not
-        be allowed to swap them out. For QEMU/KVM this requires
-        <code>hard_limit</code> <a href="#elementsMemoryTuning">memory tuning</a>
-        element to be used and set to the maximum memory configured for the
-        domain plus any memory consumed by the QEMU process itself. Beware of
-        setting the memory limit too high (and thus allowing the domain to lock
-        most of the host's memory). Doing so may be dangerous to both the
-        domain and the host itself since the host's kernel may run out of
-        memory. <span class="since">Since 1.0.6</span></dd>
+        be allowed to swap them out, which might be required for some
+        workloads such as RT. For QEMU/KVM guests, the memory used by the QEMU
+        process itself will be locked too: unlike guest memory, this is an
+        amount libvirt has no way of figuring out in advance, so it has to
+        remove the limit on locked memory altogether. This can be very
+        dangerous as the host might run out of memory and be unable to reclaim
+        it from the guest, so using this option is discouraged unless your
+        workload demands it; even then, it's highly recommended to set an
+        <code>hard_limit</code> (see
+        <a href="#elementsMemoryTuning">memory tuning</a>) on memory allocation
+        suitable for the specific environment at the same time to mitigate
+        the risks described above. <span class="since">Since 1.0.6</span></dd>
        <dt><code>source</code></dt>
        <dd>In this attribute you can switch to file memorybacking or keep default anonymous.</dd>
        <dt><code>access</code></dt>
@@ -989,12 +993,18 @@
       <dt><code>hard_limit</code></dt>
       <dd> The optional <code>hard_limit</code> element is the maximum memory
         the guest can use. The units for this value are kibibytes (i.e. blocks
-        of 1024 bytes). <strong>However, users of QEMU and KVM are strongly
-        advised not to set this limit as domain may get killed by the kernel
-        if the guess is too low. To determine the memory needed for a process
-        to run is an
+        of 1024 bytes). Users of QEMU and KVM are strongly advised not to set
+        this limit as domain may get killed by the kernel if the guess is too
+        low, and determining the memory needed for a process to run is an
         <a href="http://en.wikipedia.org/wiki/Undecidable_problem">
-        undecidable problem</a>.</strong></dd>
+        undecidable problem</a>; that said, if you already set
+        <code>locked</code> in
+        <a href="#elementsMemoryBacking">memory backing</a> because your
+        workload demands it, you'll have to take into account the specifics of
+        your deployment and figure out a value for <code>hard_limit</code> that
+        balances the risk of your guest being killed because the limit was set
+        too low and the risk of your host crashing because it cannot reclaim
+        the memory used by the guest due to <code>locked</code>. Good luck!</dd>
       <dt><code>soft_limit</code></dt>
       <dd> The optional <code>soft_limit</code> element is the memory limit to
         enforce during memory contention. The units for this value are
-- 
2.7.4

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 8/8] docs: Improve documentation related to memory locking
Posted by Luiz Capitulino 8 years, 10 months ago
On Thu, 23 Mar 2017 19:16:47 +0100
Andrea Bolognani <abologna@redhat.com> wrote:

> ---
>  docs/formatdomain.html.in | 36 +++++++++++++++++++++++-------------
>  1 file changed, 23 insertions(+), 13 deletions(-)
> 
> diff --git a/docs/formatdomain.html.in b/docs/formatdomain.html.in
> index 4a3123e..5906de6 100644
> --- a/docs/formatdomain.html.in
> +++ b/docs/formatdomain.html.in
> @@ -937,14 +937,18 @@
>        <dt><code>locked</code></dt>
>        <dd>When set and supported by the hypervisor, memory pages belonging
>          to the domain will be locked in host's memory and the host will not
> -        be allowed to swap them out. For QEMU/KVM this requires
> -        <code>hard_limit</code> <a href="#elementsMemoryTuning">memory tuning</a>
> -        element to be used and set to the maximum memory configured for the
> -        domain plus any memory consumed by the QEMU process itself. Beware of
> -        setting the memory limit too high (and thus allowing the domain to lock
> -        most of the host's memory). Doing so may be dangerous to both the
> -        domain and the host itself since the host's kernel may run out of
> -        memory. <span class="since">Since 1.0.6</span></dd>
> +        be allowed to swap them out, which might be required for some
> +        workloads such as RT. For QEMU/KVM guests, the memory used by the QEMU

Minor, but I'd do s/RT/real-time. As this doc is for the general population,
RT may not be a know term for everyone.

> +        process itself will be locked too: unlike guest memory, this is an
> +        amount libvirt has no way of figuring out in advance, so it has to
> +        remove the limit on locked memory altogether. This can be very
> +        dangerous as the host might run out of memory and be unable to reclaim
> +        it from the guest,

I'd rewrite this to:

"""
This option has a drawback and a possible security risk for the host. If
the host is running out of memory, it will be unable to reclaim the
memory locked by this guest which could cause the host to run out of
memory. In particular, a malicious guest could be able to lock as much
memory it wants, causing a DDoS attack in the host. For setups where
this may have a significant impact, it is highly recommended to use
<hard_limit> to prevent this attack.
"""

> so using this option is discouraged unless your
> +        workload demands it; even then, it's highly recommended to set an
> +        <code>hard_limit</code> (see
> +        <a href="#elementsMemoryTuning">memory tuning</a>) on memory allocation
> +        suitable for the specific environment at the same time to mitigate
> +        the risks described above. <span class="since">Since 1.0.6</span></dd>
>         <dt><code>source</code></dt>
>         <dd>In this attribute you can switch to file memorybacking or keep default anonymous.</dd>
>         <dt><code>access</code></dt>
> @@ -989,12 +993,18 @@
>        <dt><code>hard_limit</code></dt>
>        <dd> The optional <code>hard_limit</code> element is the maximum memory
>          the guest can use. The units for this value are kibibytes (i.e. blocks
> -        of 1024 bytes). <strong>However, users of QEMU and KVM are strongly
> -        advised not to set this limit as domain may get killed by the kernel
> -        if the guess is too low. To determine the memory needed for a process
> -        to run is an
> +        of 1024 bytes). Users of QEMU and KVM are strongly advised not to set
> +        this limit as domain may get killed by the kernel if the guess is too
> +        low, and determining the memory needed for a process to run is an
>          <a href="http://en.wikipedia.org/wiki/Undecidable_problem">
> -        undecidable problem</a>.</strong></dd>
> +        undecidable problem</a>; that said, if you already set
> +        <code>locked</code> in
> +        <a href="#elementsMemoryBacking">memory backing</a> because your
> +        workload demands it, you'll have to take into account the specifics of
> +        your deployment and figure out a value for <code>hard_limit</code> that
> +        balances the risk of your guest being killed because the limit was set
> +        too low and the risk of your host crashing because it cannot reclaim
> +        the memory used by the guest due to <code>locked</code>. Good luck!</dd>
>        <dt><code>soft_limit</code></dt>
>        <dd> The optional <code>soft_limit</code> element is the memory limit to
>          enforce during memory contention. The units for this value are

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 8/8] docs: Improve documentation related to memory locking
Posted by Andrea Bolognani 8 years, 10 months ago
On Fri, 2017-03-24 at 13:58 -0400, Luiz Capitulino wrote:
[...]
> > +        be allowed to swap them out, which might be required for some
> > +        workloads such as RT. For QEMU/KVM guests, the memory used by the QEMU
> 
> Minor, but I'd do s/RT/real-time. As this doc is for the general population,
> RT may not be a know term for everyone.

Sure.

> > +        process itself will be locked too: unlike guest memory, this is an
> > +        amount libvirt has no way of figuring out in advance, so it has to
> > +        remove the limit on locked memory altogether. This can be very
> > +        dangerous as the host might run out of memory and be unable to reclaim
> > +        it from the guest,
> 
> I'd rewrite this to:
> 
> """
> This option has a drawback and a possible security risk for the host. If
> the host is running out of memory, it will be unable to reclaim the
> memory locked by this guest which could cause the host to run out of
> memory. In particular, a malicious guest could be able to lock as much
> memory it wants, causing a DDoS attack in the host. For setups where
> this may have a significant impact, it is highly recommended to use
> <hard_limit> to prevent this attack.
> """

Another stab at it (which plugs into my original version):

  [...] remove the limit on locked memory altogether. Thus,
  enabling this option opens up to a potential security risk:
  the host will be unable to reclaim the locked memory back
  from the guest when it's running out of memory, which means
  a malicious guest allocating large amounts of locked memory
  could cause a denial-of-service attach on the host. Because
  of this, using the option is discouraged unless your [...]

Does it look reasonable?

-- 
Andrea Bolognani / Red Hat / Virtualization

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 8/8] docs: Improve documentation related to memory locking
Posted by Luiz Capitulino 8 years, 10 months ago
On Mon, 27 Mar 2017 12:33:48 +0200
Andrea Bolognani <abologna@redhat.com> wrote:

> On Fri, 2017-03-24 at 13:58 -0400, Luiz Capitulino wrote:
> [...]
> > > +        be allowed to swap them out, which might be required for some
> > > +        workloads such as RT. For QEMU/KVM guests, the memory used by the QEMU  
> > 
> > Minor, but I'd do s/RT/real-time. As this doc is for the general population,
> > RT may not be a know term for everyone.  
> 
> Sure.
> 
> > > +        process itself will be locked too: unlike guest memory, this is an
> > > +        amount libvirt has no way of figuring out in advance, so it has to
> > > +        remove the limit on locked memory altogether. This can be very
> > > +        dangerous as the host might run out of memory and be unable to reclaim
> > > +        it from the guest,  
> > 
> > I'd rewrite this to:
> > 
> > """
> > This option has a drawback and a possible security risk for the host. If
> > the host is running out of memory, it will be unable to reclaim the
> > memory locked by this guest which could cause the host to run out of
> > memory. In particular, a malicious guest could be able to lock as much
> > memory it wants, causing a DDoS attack in the host. For setups where
> > this may have a significant impact, it is highly recommended to use
> > <hard_limit> to prevent this attack.
> > """  
> 
> Another stab at it (which plugs into my original version):
> 
>   [...] remove the limit on locked memory altogether. Thus,
>   enabling this option opens up to a potential security risk:
>   the host will be unable to reclaim the locked memory back
>   from the guest when it's running out of memory, which means
>   a malicious guest allocating large amounts of locked memory
>   could cause a denial-of-service attach on the host. Because
>   of this, using the option is discouraged unless your [...]
> 
> Does it look reasonable?

That looks fine, although I'd drop "discouraged" because that's
not helpful to those who must use the feature. I think it's better
to objectively explain what the problems are and how to prevent or
mitigate them. That's what I tried to do in my paragraph.

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 8/8] docs: Improve documentation related to memory locking
Posted by Andrea Bolognani 8 years, 10 months ago
On Tue, 2017-03-28 at 09:43 -0400, Luiz Capitulino wrote:
> > Another stab at it (which plugs into my original version):
> > 
> >   [...] remove the limit on locked memory altogether. Thus,
> >   enabling this option opens up to a potential security risk:
> >   the host will be unable to reclaim the locked memory back
> >   from the guest when it's running out of memory, which means
> >   a malicious guest allocating large amounts of locked memory
> >   could cause a denial-of-service attach on the host. Because
> >   of this, using the option is discouraged unless your [...]
> > 
> > Does it look reasonable?
> 
> That looks fine, although I'd drop "discouraged" because that's
> not helpful to those who must use the feature. I think it's better
> to objectively explain what the problems are and how to prevent or
> mitigate them. That's what I tried to do in my paragraph.

The strong wording is intentional: we really, really don't
want people to enable this unless their setup can't work
without it.

-- 
Andrea Bolognani / Red Hat / Virtualization

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 8/8] docs: Improve documentation related to memory locking
Posted by Luiz Capitulino 8 years, 10 months ago
On Tue, 28 Mar 2017 18:26:12 +0200
Andrea Bolognani <abologna@redhat.com> wrote:

> On Tue, 2017-03-28 at 09:43 -0400, Luiz Capitulino wrote:
> > > Another stab at it (which plugs into my original version):
> > > 
> > >   [...] remove the limit on locked memory altogether. Thus,
> > >   enabling this option opens up to a potential security risk:
> > >   the host will be unable to reclaim the locked memory back
> > >   from the guest when it's running out of memory, which means
> > >   a malicious guest allocating large amounts of locked memory
> > >   could cause a denial-of-service attach on the host. Because
> > >   of this, using the option is discouraged unless your [...]
> > > 
> > > Does it look reasonable?  
> > 
> > That looks fine, although I'd drop "discouraged" because that's
> > not helpful to those who must use the feature. I think it's better
> > to objectively explain what the problems are and how to prevent or
> > mitigate them. That's what I tried to do in my paragraph.  
> 
> The strong wording is intentional: we really, really don't
> want people to enable this unless their setup can't work
> without it.

I don't see how using strong wording is going to be helpful
to users who need to use the feature.

The documentation should be helpful and technically clear. It
should instruct, not cause fear.

Your last paragraph is an improvement, but I really think this
mindset against <locked/> is very wrong.

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 8/8] docs: Improve documentation related to memory locking
Posted by Martin Kletzander 8 years, 10 months ago
On Tue, Mar 28, 2017 at 12:40:04PM -0400, Luiz Capitulino wrote:
>On Tue, 28 Mar 2017 18:26:12 +0200
>Andrea Bolognani <abologna@redhat.com> wrote:
>
>> On Tue, 2017-03-28 at 09:43 -0400, Luiz Capitulino wrote:
>> > > Another stab at it (which plugs into my original version):
>> > > 
>> > >   [...] remove the limit on locked memory altogether. Thus,
>> > >   enabling this option opens up to a potential security risk:
>> > >   the host will be unable to reclaim the locked memory back
>> > >   from the guest when it's running out of memory, which means
>> > >   a malicious guest allocating large amounts of locked memory
>> > >   could cause a denial-of-service attach on the host. Because
>> > >   of this, using the option is discouraged unless your [...]
>> > > 
>> > > Does it look reasonable?
>> > 
>> > That looks fine, although I'd drop "discouraged" because that's
>> > not helpful to those who must use the feature. I think it's better
>> > to objectively explain what the problems are and how to prevent or
>> > mitigate them. That's what I tried to do in my paragraph.
>>
>> The strong wording is intentional: we really, really don't
>> want people to enable this unless their setup can't work
>> without it.
>
>I don't see how using strong wording is going to be helpful
>to users who need to use the feature.
>
>The documentation should be helpful and technically clear. It
>should instruct, not cause fear.
>
>Your last paragraph is an improvement, but I really think this
>mindset against <locked/> is very wrong.
>

I must say I'm kinda on Luiz's side here.  It is clear what everything
does and how does it work and what are the risks.  The discouragement is
subjective *and* redundant as all the risks and technicalities are
explained.  If, for some admin/developer, "your machine may crash" is
not enough, but "your machine may crash, you should not do this" will
make everything rainbows and sunshine, then such person should rather be
user of its own system and not anything else.

>--
>libvir-list mailing list
>libvir-list@redhat.com
>https://www.redhat.com/mailman/listinfo/libvir-list
--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH 8/8] docs: Improve documentation related to memory locking
Posted by Andrea Bolognani 8 years, 10 months ago
On Tue, 2017-03-28 at 12:40 -0400, Luiz Capitulino wrote:
> > The strong wording is intentional: we really, really don't
> > want people to enable this unless their setup can't work
> > without it.
> 
> I don't see how using strong wording is going to be helpful
> to users who need to use the feature.
> 
> The documentation should be helpful and technically clear. It
> should instruct, not cause fear.
> 
> Your last paragraph is an improvement, but I really think this
> mindset against <locked/> is very wrong.

I believe the current wording strikes a good balance
between informing users who really need the feature about
the potential risks involved and mitigation options that
are available to them, and dissuading people who don't
actually need memory locking from messing with it.

Based on that, I'm not going to be spending any more time
on it, but you're of course welcome to improve it further
and post the resulting patches on the list :)

-- 
Andrea Bolognani / Red Hat / Virtualization

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list