Let's add some details about VM templating, focusing on the VM memory
configuration only.
There is much more to VM templating (VM state? block devices?), but I leave
that as future work.
Signed-off-by: David Hildenbrand <david@redhat.com>
---
docs/vm-templating.txt | 109 +++++++++++++++++++++++++++++++++++++++++
1 file changed, 109 insertions(+)
create mode 100644 docs/vm-templating.txt
diff --git a/docs/vm-templating.txt b/docs/vm-templating.txt
new file mode 100644
index 0000000000..419362c1ea
--- /dev/null
+++ b/docs/vm-templating.txt
@@ -0,0 +1,109 @@
+QEMU VM templating
+==================
+
+This document explains how to use VM templating in QEMU.
+
+For now, the focus is on VM memory aspects, and not about how to save and
+restore other VM state (i.e., migrate-to-file with 'x-ignore-shared').
+
+Overview
+--------
+
+With VM templating, a single template VM serves as the starting point for
+new VMs. This allows for fast and efficient replication of VMs, resulting
+in fast startup times and reduced memory consumption.
+
+Conceptually, the VM state is frozen, to then be used as a basis for new
+VMs. The Copy-On-Write mechanism in the operating systems makes
+sure that new VMs are able to read template VM memory; however, any
+modifications stay private and don't modify the original template VM or any
+other created VM.
+
+Memory configuration
+--------------------
+
+In order to create the template VM, we have to make sure that VM memory
+ends up in a file, from where it can be reused for the new VMs:
+
+Supply VM RAM via memory-backend-file, with 'share=on' (modifications go
+to the file) and 'readonly=off' (open the file writable). Note that
+'readonly=off' is implicit.
+
+In the following command-line example, a 2GB VM is created, whereby VM RAM
+is to be stored in the 'template' file.
+
+ qemu [...] -m 2g \
+ -object memory-backend-file,id=pc.ram,mem-path=template,size=2g,share=on,... \
+ -machine q35,memory-backend=pc.ram',
+
+If multiple memory backends are used (vNUMA, DIMMs), configure all
+memory backends accordingly.
+
+Once the VM is in the desired state, stop the VM and save other VM state,
+leaving the current state of VM RAM reside in the file.
+
+In order to have a new VM be based on a template VM, we have to
+configure VM RAM to be based on a template VM RAM file; however, the VM
+should not be able to modify file content.
+
+Supply VM RAM via memory-backend-file, with 'share=off' (modifications stay
+private), 'readonly=on' (open the file readonly) and 'rom=off' (don't make
+the memory readonly for the VM). Note that 'share=off' is implicit and
+that other VM state has to be restored separately.
+
+In the following command-line example, a 2GB VM is created based on the
+existing 2GB file 'template'.
+
+ qemu [...] -m 2g \
+ -object memory-backend-file,id=pc.ram,mem-path=template,size=2g,readonly=on,rom=off,... \
+ -machine q35,memory-backend=pc.ram',
+
+If multiple memory backends are used (vNUMA, DIMMs), configure all
+memory backends accordingly.
+
+Note that '-mem-path' cannot be used for VM templating when creating the
+template VM or when starting new VMs based on a template VM.
+
+Incompatible features
+---------------------
+
+Some features are incompatible with VM templating, as the underlying file
+cannot be modified to discard VM RAM, or to actually share memory with
+another process.
+
+vhost-user and multi-process QEMU
+'''''''''''''''''''''''''''''''''
+
+vhost-user and multi-process QEMU are incompatible with VM templating.
+These technologies rely on shared memory, however, the template VMs
+don't actually share memory ('share=off'), even though they are file-based.
+
+virtio-balloon
+''''''''''''''
+
+virtio-balloon inflation and "free page reporting" cannot discard VM RAM
+and will repeatedly report errors. While virtio-balloon can be used
+for template VMs (e.g., report VM RAM stats), "free page reporting"
+should be disabled and the balloon should not be inflated.
+
+virtio-mem
+''''''''''
+
+virtio-mem cannot discard VM RAM that is managed by the virtio-mem
+device. virtio-mem will fail early when realizing the device. To use
+VM templating with virtio-mem, either hotplug virtio-mem devices to the new
+VM, or don't supply any memory to the template VM using virtio-mem
+(requested-size=0), not using a template VM file as memory backend for the
+virtio-mem device.
+
+VM migration
+''''''''''''
+
+For VM migration, "x-release-ram" similarly relies on discarding of VM
+RAM on the migration source to free up migrated RAM, and will
+repeatedly report errors.
+
+Postcopy live migration fails discarding VM RAM on the migration
+destination early and refuses to activate postcopy live migration. Note
+that postcopy live migration usually only works on selected filesystems
+(shmem/tmpfs, hugetlbfs) either way.
--
2.41.0On Tue, 22 Aug 2023 at 12:49, David Hildenbrand <david@redhat.com> wrote: > > Let's add some details about VM templating, focusing on the VM memory > configuration only. > > There is much more to VM templating (VM state? block devices?), but I leave > that as future work. > > Signed-off-by: David Hildenbrand <david@redhat.com> > --- > docs/vm-templating.txt | 109 +++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 109 insertions(+) > create mode 100644 docs/vm-templating.txt No new .txt files in docs/, please. Use rst, and incorporate the information into the correct parts of the manual structure. thanks -- PMM
On 22.08.23 16:23, Peter Maydell wrote: > On Tue, 22 Aug 2023 at 12:49, David Hildenbrand <david@redhat.com> wrote: >> >> Let's add some details about VM templating, focusing on the VM memory >> configuration only. >> >> There is much more to VM templating (VM state? block devices?), but I leave >> that as future work. >> >> Signed-off-by: David Hildenbrand <david@redhat.com> >> --- >> docs/vm-templating.txt | 109 +++++++++++++++++++++++++++++++++++++++++ >> 1 file changed, 109 insertions(+) >> create mode 100644 docs/vm-templating.txt > > No new .txt files in docs/, please. Use rst, and incorporate > the information into the correct parts of the manual structure. Thanks, already raised by Daniel. Will be an RST and moved under docs/system. -- Cheers, David / dhildenb
On Tue, Aug 22, 2023 at 01:44:57PM +0200, David Hildenbrand wrote: > Let's add some details about VM templating, focusing on the VM memory > configuration only. > > There is much more to VM templating (VM state? block devices?), but I leave > that as future work. Then there's the supposedly "unique" hardware identifiers, most notably VM UUID & NIC MAC addr that don't change if you create many VMs from a "template". Or from the guest OS there are "unique" things like /etc/machine-id, SSH host keys, web server certificates, etc. The vmgenid device at least provides a way for guest OS to get notified to update its unique resources/identifiers, but doesn't solve the overall VM UUID. NIC MAC addr could be solved by hotunplug+plug either side of creating the template & instantiating the template. > > Signed-off-by: David Hildenbrand <david@redhat.com> > --- > docs/vm-templating.txt | 109 +++++++++++++++++++++++++++++++++++++++++ Can you make this doument RST from the start and link to it from somewhere appropriate in our documentation. Perhaps it should live under the docs/system/ directory ? > 1 file changed, 109 insertions(+) > create mode 100644 docs/vm-templating.txt > > diff --git a/docs/vm-templating.txt b/docs/vm-templating.txt > new file mode 100644 > index 0000000000..419362c1ea > --- /dev/null > +++ b/docs/vm-templating.txt > @@ -0,0 +1,109 @@ > +QEMU VM templating > +================== > + > +This document explains how to use VM templating in QEMU. > + > +For now, the focus is on VM memory aspects, and not about how to save and > +restore other VM state (i.e., migrate-to-file with 'x-ignore-shared'). > + > +Overview > +-------- > + > +With VM templating, a single template VM serves as the starting point for > +new VMs. This allows for fast and efficient replication of VMs, resulting > +in fast startup times and reduced memory consumption. > + > +Conceptually, the VM state is frozen, to then be used as a basis for new > +VMs. The Copy-On-Write mechanism in the operating systems makes > +sure that new VMs are able to read template VM memory; however, any > +modifications stay private and don't modify the original template VM or any > +other created VM. I feel like we should have a paragraph at the top here explicitly calling out the dangers of templating, wrt to unique data in the hardware and guest OS. Don't have to provide solutions, just more of a scarcy "here be dragons" warning to users who might be tempted to try this. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 22.08.23 15:47, Daniel P. Berrangé wrote: > On Tue, Aug 22, 2023 at 01:44:57PM +0200, David Hildenbrand wrote: >> Let's add some details about VM templating, focusing on the VM memory >> configuration only. >> >> There is much more to VM templating (VM state? block devices?), but I leave >> that as future work. > > Then there's the supposedly "unique" hardware identifiers, most notably > VM UUID & NIC MAC addr that don't change if you create many VMs from > a "template". Or from the guest OS there are "unique" things like > /etc/machine-id, SSH host keys, web server certificates, etc. > > The vmgenid device at least provides a way for guest OS to get notified > to update its unique resources/identifiers, but doesn't solve the overall > VM UUID. NIC MAC addr could be solved by hotunplug+plug either side of > creating the template & instantiating the template. > >> >> Signed-off-by: David Hildenbrand <david@redhat.com> >> --- >> docs/vm-templating.txt | 109 +++++++++++++++++++++++++++++++++++++++++ > > Can you make this doument RST from the start and link to it from > somewhere appropriate in our documentation. Perhaps it should live > under the docs/system/ directory ? I blindly did what memory-hotplug.txt and nvdimm.txt do. I can make it a RST and move under docs/system [+ link it in the index] > >> 1 file changed, 109 insertions(+) >> create mode 100644 docs/vm-templating.txt >> >> diff --git a/docs/vm-templating.txt b/docs/vm-templating.txt >> new file mode 100644 >> index 0000000000..419362c1ea >> --- /dev/null >> +++ b/docs/vm-templating.txt >> @@ -0,0 +1,109 @@ >> +QEMU VM templating >> +================== >> + >> +This document explains how to use VM templating in QEMU. >> + >> +For now, the focus is on VM memory aspects, and not about how to save and >> +restore other VM state (i.e., migrate-to-file with 'x-ignore-shared'). >> + >> +Overview >> +-------- >> + >> +With VM templating, a single template VM serves as the starting point for >> +new VMs. This allows for fast and efficient replication of VMs, resulting >> +in fast startup times and reduced memory consumption. >> + >> +Conceptually, the VM state is frozen, to then be used as a basis for new >> +VMs. The Copy-On-Write mechanism in the operating systems makes >> +sure that new VMs are able to read template VM memory; however, any >> +modifications stay private and don't modify the original template VM or any >> +other created VM. > > I feel like we should have a paragraph at the top here explicitly calling > out the dangers of templating, wrt to unique data in the hardware and guest > OS. Don't have to provide solutions, just more of a scarcy "here be dragons" > warning to users who might be tempted to try this. Agreed, I'll use some of your information above, thanks! -- Cheers, David / dhildenb
© 2016 - 2026 Red Hat, Inc.