From nobody Tue May 7 07:31:08 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=libvir-list-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1561649398; cv=none; d=zoho.com; s=zohoarc; b=HbSwpmPFaTbRCeq7M0Td+eAM8nQUT6jp109+gCIYevFUtOEBRGWCjUZV5NUuHEWk2Qu1kvqJrhlkpmEB/4p1wEYS38vPfzPgSxB2PM1IcLHCDJI+Ib5p5nGALdeMbWKy4M6SOxzvErbKEdUBRZF/fm5z7FBn2S29ZmU4q1171dw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1561649398; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To:ARC-Authentication-Results; bh=pkRCm83NIq+9xMsHaTtabfWVd5wXqdLIcWqjEuqcO+c=; b=XlH3QNt3BElY8Ga7OEG3nacSd4bVwNx3P8XtPreMgzJ+kEUYCVz1bkZbw3uzFcmLS6eAHi6ykQ1PfwBiWOK0tep308+BLtWhalRW6ZWci1xrCFFAMyvhXddS6BFjBNsidM611ot3eOvqyf6siYpXAbJyTtX+ogAkEv+TFOr0ZF0= ARC-Authentication-Results: i=1; mx.zoho.com; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1561649398470457.00511003775796; Thu, 27 Jun 2019 08:29:58 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id E9F5659473; Thu, 27 Jun 2019 15:28:59 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 731111001B13; Thu, 27 Jun 2019 15:28:43 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 4ABCD1806B19; Thu, 27 Jun 2019 15:28:17 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id x5RFSBcZ021867 for ; Thu, 27 Jun 2019 11:28:11 -0400 Received: by smtp.corp.redhat.com (Postfix) id 77AF05C1B4; Thu, 27 Jun 2019 15:28:11 +0000 (UTC) Received: from beluga.usersys.redhat.com (unknown [10.43.2.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id AA3D35C298; Thu, 27 Jun 2019 15:28:08 +0000 (UTC) From: Erik Skultety To: libvir-list@redhat.com Date: Thu, 27 Jun 2019 17:27:56 +0200 Message-Id: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-loop: libvir-list@redhat.com Cc: Erik Skultety Subject: [libvirt] [PATCH] docs: Provide documentation for SEV launch security (DO NOT PUSH) X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Thu, 27 Jun 2019 15:29:33 +0000 (UTC) --- I sent this as a patch to get a review on the contents itself, but I believ= e it should live as an article on our wiki page afterwards. docs/launch_security_sev.html.in | 536 +++++++++++++++++++++++++++++++ 1 file changed, 536 insertions(+) create mode 100644 docs/launch_security_sev.html.in diff --git a/docs/launch_security_sev.html.in b/docs/launch_security_sev.ht= ml.in new file mode 100644 index 0000000000..42db10b33a --- /dev/null +++ b/docs/launch_security_sev.html.in @@ -0,0 +1,536 @@ + + + + + + + + + + + + + libvirt: Launch security with AMD SEV + + + + +
+ +

+ Storage encryption in modern public cloud computing is a common = practice. + However, from the point of view of a user of these cloud workloa= ds, a + significant amount of trust needs to be put in the cloud platfor= m security as + well as integrity (was the hypervisor tampered?). For this reaso= n there's ever + rising demand for securing data in use, i.e. memory encryption. + One of the solutions addressing this matter is AMD SEV. +

+

AMD SEV

+

+ SEV (Secure Encrypted Virtualization) is a feature extension of = AMD's SME (Secure + Memory Encryption) intended for KVM virtual machines which is su= pported + primarily on AMD's EPYC CPU line. In contrast to SME, SEV uses a= unique memory encryption + key for each VM. The whole encryption of memory pages is complet= ely transparent + to the hypervisor and happens in the AMD firmware. + For more details about the technology itself, you can visit + AMD's developer portal<= /a>. +

+

+ Enabling SEV on the host + =C2=B6 +

+

+ Before VMs can make use of the SEV feature you need to make sure= your + AMD CPU does support SEV. You can check whether SEV is among the= CPU + flags with: +

+ + 
+$ cat /proc/cpuinfo | grep sev
+...
+sme ssbd sev ibpb
+ +

+ Next step is to enable SEV in the kernel, because it is disabled= by default. + This is done by putting the following onto the kernel command li= ne: +

+ 
+mem_encrypt=3Don kvm_amd.sev=3D1
+
+ +

+ To make the changes persistent, append the above to the variable= holding + parameters of the kernel command line in + /etc/default/grub to preserve SEV settings across r= eboots +

+ + 
+$ cat /etc/default/grub
+...
+GRUB_CMDLINE_LINUX=3D"... mem_encrypt=3Don kvm_amd.sev=3D1"
+$ grub2-mkconfig -o /boot/efi/EFI/<distro>/grub.cfg
+ +

+ mem_encrypt=3Don turns on the SME memory encryption f= eature on + the host which is required for SEV to work. The kvm_amd.sev<= /code> + parameter actually enables SEV in the kvm module. It can be set on= the + command line alongside mem_encrypt like shown above, = or it + can be put into a module config under /etc/modprobe.d/ +

+ 
+$ cat /etc/modprobe.d/sev.conf
+options kvm_amd sev=3D1
+
+

+ After rebooting the host, you should see SEV being enabled in th= e kernel: +

+ 
+$ cat /sys/module/kvm_amd/parameters/sev
+1
+
+ +

+ Checking SEV support in the virt stack + =C2=B6 +

+

+ Note: All of the commands bellow need to be run with root privi= leges. +

+

+ First make sure you have the following packages in the specified= versions: +

+
    +
  • + libvirt >=3D 4.5.0 (>5.1.0 recommended due to additional SEV b= ugfixes) +
    • +
  • + QEMU >=3D 2.12.0 +
    • +
+

+ To confirm that the virtualization stack supports SEV, run the f= ollowing: +

+ 
+# virsh domcapabilities
+<domainCapabilities>
+...
+  <features>
+    ...
+    <sev supported=3D'yes'>
+      <cbitpos>47</cbitpos>
+      <reducedPhysBits>1</reducedPhysBits>
+    </sev>
+    ...
+  </features>
+</domainCapabilities>
+

+ Note that if libvirt was already installed and libvirtd running = before enabling SEV in the kernel followed by the host reboot you need to f= orce libvirtd + to re-probe both the host and QEMU capabilities. First stop libv= irtd: +

+ 
+# systemctl stop libvirtd.service
+
+

+ Now you need to clean the capabilities cache: +

+ 
+# rm -f /var/cache/libvirt/qemu/capabilities/*
+
+

+ If you now restart libvirtd, it will re-probe the capabilities a= nd if + you now run: +

+ 
+# virsh domcapabilities
+
+

+ SEV should be listed as supported. If you still see: +

+ 
+<sev supported=3D'no'/>
+
+

+ it means one of two things: +

    +
  1. + libvirt does support SEV, but either QEMU or the host does n= ot +
    2. +
  2. + you have libvirt <=3D5.1.0 which suffered from getting a + 'Permission denied' on /dev/sev beca= use + of the default permissions on the character device which preve= nted + QEMU from opening it during capabilities probing - you can eit= her + manually tweak the permissions so that QEMU has access to it or + preferably install libvirt 5.1.0 or higher +
    3. +
+

+

+ VM Configuration + =C2=B6 +

+

+ SEV is enabled in the XML by specifying the + <= ;launchSecurity> element. However, specifying launchSecurity<= /code> isn't + enough to boot an SEV VM. Further configuration requirements are d= iscussed + below. +

+ +

Machine type

+

+ Even though both Q35 and legacy PC machine types (for PC see also + "virtio") can be used with SEV, usage of the legacy PC machine t= ype is + strongly discouraged, since depending on how your OVMF package w= as + built (e.g. including features like SecureBoot or SMM) Q35 may e= ven be + required. +

+ +
Q35
+
+...
+<os>
+  <type arch=3D'x86_64' machine=3D'pc-q35-3.0'>hvm</type>
+  ...
+</os>
+...
+ +
i440fx (discouraged)
+ 
+...
+<os>
+  <type arch=3D'x86_64' machine=3D'pc-i440fx-3.0'>hvm</type>
+  ...
+</os>
+...
+
+ +

Boot loader

+

+ SEV is only going to work with OVMF (UEFI), so you'll need to po= int libvirt to + the correct OVMF binary. +

+ 
+...
+<os>
+  <type arch=3D'x86_64' machine=3D'pc-q35-3.0'>hvm</type>
+  <loader readonly=3D'yes' type=3D'pflash'>/usr/share/edk2/ovmf/OVMF=
_CODE.fd</loader>
+</os>
+...
+ +

Memory

+

+ Internally, SEV expects that the encrypted memory pages won't be= swapped out or move + around so the VM memory needs to be pinned in physical RAM which= will be + handled by QEMU. Apart from that, certain memory regions allocat= ed by QEMU + itself (UEFI pflash, device ROMs, video RAM, etc.) have to be en= crypted as + well. This causes a conflict in how libvirt tries to protect the= host. + By default, libvirt enforces a memory hard limit on each VM's cg= roup in order + to protect the host from malicious QEMU to allocate and lock all= the available + memory. This limit corresponds to the total memory allocation fo= r the VM given + by <currentMemory> element. However, trying t= o account for the additional + memory regions QEMU allocates when calculating the limit in an a= utomated manner + is non-deterministic. One way to resolve this is to set the hard= limit manually. + +

+ Note: Figuring out the right number so that your guest boots and= isn't killed is + challenging, but 256MiB extra memory over the total guest RAM sh= ould suffice for + most workloads and may serve as a good starting point. + + For example, a domain with 4GB memory with a 256MiB extra hard l= imit would look + like this: +

+

+ 
+# virsh edit <domain>
+<domain>
+  ...
+  <currentMemory unit=3D'KiB'>4194304</currentMemory>
+  <memtune>
+    <hard_limit unit=3D'KiB'>4456448</hard_limit>
+  </memtune>
+  ...
+</domain>
+

+ There's another, preferred method of taking care of the limits by + using the<memoryBacking> element along with t= he + <locked/> subelement: +

+ 
+<domain>
+  ...
+  <memoryBacking>
+    <locked/>
+  </memoryBacking>
+  ...
+</domain>
+

+ What that does is that it tells libvirt not to force any hard li= mit (well, + unlimited) upon the VM cgroup. The obvious advantage is that one= doesn't need + to determine the hard limit for every single SEV-enabled VM. How= ever, there is + a significant security-related drawback to this approach. Since = no hard limit + is applied, a malicious QEMU could perform a DoS attack by locki= ng all of the + host's available memory. The way to avoid this issue and to prot= ect the host is + to enforce a bigger hard limit on the master cgroup containing a= ll of the VMs + - on systemd this is machine.slice. +

+ 
+# systemctl set-property machine.slice MemoryHigh=3D<value>
+

+ To put even stricter measures in place which would involve the O= OM killer, use + 

+# systemctl set-property machine.slice MemoryMax=3D<value>
+ instead. Alternatively, you can create a systemd config (don't f= orget + to reload systemd configuration in this case): + 
+# cat << EOF > /etc/systemd/system.control/machine.slice.d/90-Mem=
oryMax.conf
+MemoryMax=3D<value>
+EOF
+ The trade-off to keep in mind with the second approach is that t= he VMs + can still perform DoS on each other. +

+

Virtio

+

+ In order to make virtio devices work, we need to enable emulated= IOMMU + on the devices so that virtual DMA can work. +

+ 
+# virsh edit <domain>
+<domain>
+  ...
+  <controller type=3D'virtio-serial' index=3D'0'>
+    <driver iommu=3D'on'/>
+  </controller>
+  <controller type=3D'scsi' index=3D'0' model=3D'virtio-scsi'>
+    <driver iommu=3D'on'/>
+  </controller>
+  ...
+  <memballoon model=3D'virtio'>
+    <driver iommu=3D'on'/>
+  </memballoon>
+  <rng model=3D'virtio'>
+    <backend model=3D'random'>/dev/urandom</backend>
+    <driver iommu=3D'on'/>
+  </rng>
+  ...
+<domain>
+

+ If you for some reason want to use the legacy PC machine type, fur= ther changes + to the virtio + configuration is required, because SEV will not work with Virtio &= lt;1.0. In + libvirt, this is handled by using the virtio-non-transitional devi= ce model + (libvirt >=3D 5.2.0 required). + +

+ Note: some devices like video devices don't + support non-transitional model, which means that virtio GPU cann= ot be used. +

+

+ 
+<domain>
+  ...
+  <devices>
+    ...
+    <memballoon model=3D'virtio-non-transitional'>
+      <driver iommu=3D'on'/>
+    </memballoon>
+  </devices>
+  ...
+</domain>
+ + +

+ Limitations + =C2=B6 +

+

+ Currently, the boot disk cannot be of type virtio-blk, instead, vi= rtio-scsi + needs to be used if virtio is desired. This limitation is expected= to be lifted + with future releases of kernel (the kernel used at the time of wri= ting the + article is 5.0.14). + If you still cannot start an SEV VM, it could be because of wrong = SELinux label on the /dev/sev device with selinux-policy <3= .14.2.40 which prevents QEMU from touching the device. This can be resolved= by upgrading the package, tuning the selinux policy rules manually to allo= w svirt_t to access the device (see audit2allow on how to do t= hat) or putting SELinux into permissive mode (discouraged). +

+

+ Full domain XML examples + =C2=B6 +

+
Q35 machine
+ 
+<domain type=3D'kvm'>
+  <name>sev-dummy</name>
+  <memory unit=3D'KiB'>4194304</memory>
+  <currentMemory unit=3D'KiB'>4194304</currentMemory>
+  <memoryBacking>
+    <locked/>
+  </memoryBacking>
+  <vcpu placement=3D'static'>4</vcpu>
+  <os>
+    <type arch=3D'x86_64' machine=3D'pc-q35-3.0'>hvm</type>
+    <loader readonly=3D'yes' type=3D'pflash'>/usr/share/edk2/ovmf/OV=
MF_CODE.fd</loader>
+    <nvram>/var/lib/libvirt/qemu/nvram/sev-dummy_VARS.fd</nvram&g=
t;
+  </os>
+  <features>
+    <acpi/>
+    <apic/>
+    <vmport state=3D'off'/>
+  </features>
+  <cpu mode=3D'host-model' check=3D'partial'>
+    <model fallback=3D'allow'/>
+  </cpu>
+  <clock offset=3D'utc'>
+    <timer name=3D'rtc' tickpolicy=3D'catchup'/>
+    <timer name=3D'pit' tickpolicy=3D'delay'/>
+    <timer name=3D'hpet' present=3D'no'/>
+  </clock>
+  <on_poweroff>destroy</on_poweroff>
+  <on_reboot>restart</on_reboot>
+  <on_crash>destroy</on_crash>
+  <pm>
+    <suspend-to-mem enabled=3D'no'/>
+    <suspend-to-disk enabled=3D'no'/>
+  </pm>
+  <devices>
+    <emulator>/usr/bin/qemu-kvm</emulator>
+    <disk type=3D'file' device=3D'disk'>
+      <driver name=3D'qemu' type=3D'qcow2'/>
+      <source file=3D'/var/lib/libvirt/images/sev-dummy.qcow2'/>
+      <target dev=3D'sda' bus=3D'scsi'/>
+      <boot order=3D'1'/>
+    </disk>
+    <controller type=3D'virtio-serial' index=3D'0'>
+      <driver iommu=3D'on'/>
+    </controller>
+    <controller type=3D'scsi' index=3D'0' model=3D'virtio-scsi'>
+      <driver iommu=3D'on'/>
+    </controller>
+    <interface type=3D'network'>
+      <mac address=3D'52:54:00:cc:56:90'/>
+      <source network=3D'default'/>
+      <model type=3D'virtio'/>
+      <driver iommu=3D'on'/>
+    </interface>
+    <graphics type=3D'spice' autoport=3D'yes'>
+      <listen type=3D'address'/>
+      <gl enable=3D'no'/>
+    </graphics>
+    <video>
+      <model type=3D'qxl'/>
+    </video>
+    <memballoon model=3D'virtio'>
+      <driver iommu=3D'on'/>
+    </memballoon>
+    <rng model=3D'virtio'>
+      <driver iommu=3D'on'/>
+    </rng>
+  </devices>
+  <launchSecurity type=3D'sev'>
+    <cbitpos>47</cbitpos>
+    <reducedPhysBits>1</reducedPhysBits>
+    <policy>0x0003</policy>
+  </launchSecurity>
+</domain>
+ +
PC-i440fx machine:
+ 
+<domain type=3D'kvm'>
+  <name>sev-dummy-legacy</name>
+  <memory unit=3D'KiB'>4194304</memory>
+  <currentMemory unit=3D'KiB'>4194304</currentMemory>
+  <memtune>
+    <hard_limit unit=3D'KiB'>5242880</hard_limit>
+  </memtune>
+  <vcpu placement=3D'static'>4</vcpu>
+  <os>
+    <type arch=3D'x86_64' machine=3D'pc-i440fx-3.0'>hvm</type>
+    <loader readonly=3D'yes' type=3D'pflash'>/usr/share/edk2/ovmf/OV=
MF_CODE.fd</loader>
+    <nvram>/var/lib/libvirt/qemu/nvram/sev-dummy_VARS.fd</nvram&g=
t;
+    <boot dev=3D'hd'/>
+  </os>
+  <features>
+  <acpi/>
+  <apic/>
+  <vmport state=3D'off'/>
+  </features>
+  <cpu mode=3D'host-model' check=3D'partial'>
+    <model fallback=3D'allow'/>
+  </cpu>
+  <clock offset=3D'utc'>
+    <timer name=3D'rtc' tickpolicy=3D'catchup'/>
+    <timer name=3D'pit' tickpolicy=3D'delay'/>
+    <timer name=3D'hpet' present=3D'no'/>
+  </clock>
+  <on_poweroff>destroy</on_poweroff>
+  <on_reboot>restart</on_reboot>
+  <on_crash>destroy</on_crash>
+  <pm>
+    <suspend-to-mem enabled=3D'no'/>
+    <suspend-to-disk enabled=3D'no'/>
+  </pm>
+  <devices>
+    <emulator>/usr/bin/qemu-kvm</emulator>
+    <disk type=3D'file' device=3D'disk'>
+      <driver name=3D'qemu' type=3D'qcow2'/>
+      <source file=3D'/var/lib/libvirt/images/sev-dummy-seabios.qcow2'/=
>
+      <target dev=3D'sda' bus=3D'sata'/>
+    </disk>
+    <interface type=3D'network'>
+      <mac address=3D'52:54:00:d8:96:c8'/>
+      <source network=3D'default'/>
+      <model type=3D'virtio-non-transitional'/>
+    </interface>
+    <serial type=3D'pty'>
+      <target type=3D'isa-serial' port=3D'0'>
+        <model name=3D'isa-serial'/>
+      </target>
+    </serial>
+    <console type=3D'pty'>
+      <target type=3D'serial' port=3D'0'/>
+    </console>
+    <input type=3D'tablet' bus=3D'usb'>
+      <address type=3D'usb' bus=3D'0' port=3D'1'/>
+    </input>
+    <input type=3D'mouse' bus=3D'ps2'/>
+    <input type=3D'keyboard' bus=3D'ps2'/>
+    <graphics type=3D'spice' autoport=3D'yes'>
+      <listen type=3D'address'/>
+      <gl enable=3D'no'/>
+    </graphics>
+    <video>
+      <model type=3D'qxl' ram=3D'65536' vram=3D'65536' vgamem=3D'16384'=
 heads=3D'1' primary=3D'yes'/>
+    </video>
+    <memballoon model=3D'virtio-non-transitional'>
+      <driver iommu=3D'on'/>
+    </memballoon>
+      <rng model=3D'virtio-non-transitional'>
+    <driver iommu=3D'on'/>
+    </rng>
+  </devices>
+  <launchSecurity type=3D'sev'>
+    <cbitpos>47</cbitpos>
+    <reducedPhysBits>1</reducedPhysBits>
+    <policy>0x0003</policy>
+  </launchSecurity>
+</domain>
+
+ + -- 2.21.0 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list