[Qemu-devel] [PATCH v4 0/3] pc: mmap kernel (ELF image) and initrd

Stefano Garzarella posted 3 patches 36 weeks ago
Test docker-clang@ubuntu passed
Test s390x passed
Test asan passed
Test docker-mingw@fedora passed
Test FreeBSD passed
Test checkpatch passed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20190724143105.307042-1-sgarzare@redhat.com
Maintainers: "Michael S. Tsirkin" <mst@redhat.com>, Richard Henderson <rth@twiddle.net>, Paolo Bonzini <pbonzini@redhat.com>, Eduardo Habkost <ehabkost@redhat.com>, Marcel Apfelbaum <marcel.apfelbaum@gmail.com>
hw/core/loader.c     | 38 +++++++++++++++++++-----
hw/i386/pc.c         | 17 ++++++++---
include/hw/elf_ops.h | 71 ++++++++++++++++++++++++++------------------
include/hw/i386/pc.h |  1 +
include/hw/loader.h  |  5 ++--
5 files changed, 89 insertions(+), 43 deletions(-)

[Qemu-devel] [PATCH v4 0/3] pc: mmap kernel (ELF image) and initrd

Posted by Stefano Garzarella 36 weeks ago
In order to reduce the memory footprint when PVH kernel and initrd
are used, we map them into memory instead of reading them.
In this way we can share them between multiple instances of QEMU.

v4:
  - Patch 1: fix the rom_add_elf_program() comment [Paolo]
  - Patch 2:
    ~ fix the missing of g_mapped_file_unref() in the success case [Paolo]
    ~ fix the rom_add_elf_program() comment [Paolo]

v3: https://patchew.org/QEMU/20190724112531.232260-1-sgarzare@redhat.com/
v2: https://patchew.org/QEMU/20190723140445.12748-1-sgarzare@redhat.com/

These are the results using a PVH kernel and initrd (cpio):
- memory footprint (using smem) [MB]
        QEMU              before                   now
    # instances        USS      PSS            USS      PSS
         1           102.0M   105.8M         102.3M   106.2M
         2            94.6M   101.2M          72.3M    90.1M
         4            94.1M    98.0M          72.0M    81.5M
         8            94.0M    96.2M          71.8M    76.9M
        16            93.9M    95.1M          71.6M    74.3M

    Initrd size: 3.0M
    Kernel
        image size: 28M
        sections size [size -A -d vmlinux]:  18.9M

- boot time [ms]
                          before                   now
 qemu_init_end:           63.85                   55.91
 linux_start_kernel:      82.11 (+18.26)          74.51 (+18.60)
 linux_start_user:       169.94 (+87.83)         159.06 (+84.56)

QEMU command used:
./qemu-system-x86_64 -bios /path/to/seabios/out/bios.bin -no-hpet \
    -machine q35,accel=kvm,kernel_irqchip,nvdimm,sata=off,smbus=off,vmport=off \
    -cpu host -m 1G -smp 1 -vga none -display none -no-user-config -nodefaults \
    -kernel /path/to/vmlinux -initrd /path/to/rootfs.cpio \
    -append 'root=/dev/mem0 ro console=hvc0 pci=lastbus=0 nosmap'

Stefano Garzarella (3):
  loader: Handle memory-mapped ELFs
  elf-ops.h: Map into memory the ELF to load
  hw/i386/pc: Map into memory the initrd

 hw/core/loader.c     | 38 +++++++++++++++++++-----
 hw/i386/pc.c         | 17 ++++++++---
 include/hw/elf_ops.h | 71 ++++++++++++++++++++++++++------------------
 include/hw/i386/pc.h |  1 +
 include/hw/loader.h  |  5 ++--
 5 files changed, 89 insertions(+), 43 deletions(-)

-- 
2.20.1


Re: [Qemu-devel] [PATCH v4 0/3] pc: mmap kernel (ELF image) and initrd

Posted by Montes, Julio 36 weeks ago
Hi Stefano

Here the results

https://pasteboard.co/Ipu3DO4.png
https://pasteboard.co/Ipu3L69.png

boot time with initrd is a little bit better

Thanks

-
Julio


On Wed, 2019-07-24 at 16:31 +0200, Stefano Garzarella wrote:
> In order to reduce the memory footprint when PVH kernel and initrd
> are used, we map them into memory instead of reading them.
> In this way we can share them between multiple instances of QEMU.
> 
> v4:
>   - Patch 1: fix the rom_add_elf_program() comment [Paolo]
>   - Patch 2:
>     ~ fix the missing of g_mapped_file_unref() in the success case
> [Paolo]
>     ~ fix the rom_add_elf_program() comment [Paolo]
> 
> v3: 
> https://patchew.org/QEMU/20190724112531.232260-1-sgarzare@redhat.com/
> v2: 
> https://patchew.org/QEMU/20190723140445.12748-1-sgarzare@redhat.com/
> 
> These are the results using a PVH kernel and initrd (cpio):
> - memory footprint (using smem) [MB]
>         QEMU              before                   now
>     # instances        USS      PSS            USS      PSS
>          1           102.0M   105.8M         102.3M   106.2M
>          2            94.6M   101.2M          72.3M    90.1M
>          4            94.1M    98.0M          72.0M    81.5M
>          8            94.0M    96.2M          71.8M    76.9M
>         16            93.9M    95.1M          71.6M    74.3M
> 
>     Initrd size: 3.0M
>     Kernel
>         image size: 28M
>         sections size [size -A -d vmlinux]:  18.9M
> 
> - boot time [ms]
>                           before                   now
>  qemu_init_end:           63.85                   55.91
>  linux_start_kernel:      82.11 (+18.26)          74.51 (+18.60)
>  linux_start_user:       169.94 (+87.83)         159.06 (+84.56)
> 
> QEMU command used:
> ./qemu-system-x86_64 -bios /path/to/seabios/out/bios.bin -no-hpet \
>     -machine
> q35,accel=kvm,kernel_irqchip,nvdimm,sata=off,smbus=off,vmport=off \
>     -cpu host -m 1G -smp 1 -vga none -display none -no-user-config
> -nodefaults \
>     -kernel /path/to/vmlinux -initrd /path/to/rootfs.cpio \
>     -append 'root=/dev/mem0 ro console=hvc0 pci=lastbus=0 nosmap'
> 
> Stefano Garzarella (3):
>   loader: Handle memory-mapped ELFs
>   elf-ops.h: Map into memory the ELF to load
>   hw/i386/pc: Map into memory the initrd
> 
>  hw/core/loader.c     | 38 +++++++++++++++++++-----
>  hw/i386/pc.c         | 17 ++++++++---
>  include/hw/elf_ops.h | 71 ++++++++++++++++++++++++++--------------
> ----
>  include/hw/i386/pc.h |  1 +
>  include/hw/loader.h  |  5 ++--
>  5 files changed, 89 insertions(+), 43 deletions(-)
> 

Re: [Qemu-devel] [PATCH v4 0/3] pc: mmap kernel (ELF image) and initrd

Posted by Stefano Garzarella 36 weeks ago
On Wed, Jul 24, 2019 at 04:35:23PM +0000, Montes, Julio wrote:
> Hi Stefano
> 
> Here the results
> 
> https://pasteboard.co/Ipu3DO4.png
> https://pasteboard.co/Ipu3L69.png
> 
> boot time with initrd is a little bit better
> 

Thanks for sharing!

It is interesting that 'inkernel' time with initrd is almost twice the time of
nvdimm.


Stefano

Re: [Qemu-devel] [PATCH v4 0/3] pc: mmap kernel (ELF image) and initrd

Posted by Paolo Bonzini 36 weeks ago
On 24/07/19 16:31, Stefano Garzarella wrote:
> In order to reduce the memory footprint when PVH kernel and initrd
> are used, we map them into memory instead of reading them.
> In this way we can share them between multiple instances of QEMU.
> 
> v4:
>   - Patch 1: fix the rom_add_elf_program() comment [Paolo]
>   - Patch 2:
>     ~ fix the missing of g_mapped_file_unref() in the success case [Paolo]
>     ~ fix the rom_add_elf_program() comment [Paolo]
> 
> v3: https://patchew.org/QEMU/20190724112531.232260-1-sgarzare@redhat.com/
> v2: https://patchew.org/QEMU/20190723140445.12748-1-sgarzare@redhat.com/
> 
> These are the results using a PVH kernel and initrd (cpio):
> - memory footprint (using smem) [MB]
>         QEMU              before                   now
>     # instances        USS      PSS            USS      PSS
>          1           102.0M   105.8M         102.3M   106.2M
>          2            94.6M   101.2M          72.3M    90.1M
>          4            94.1M    98.0M          72.0M    81.5M
>          8            94.0M    96.2M          71.8M    76.9M
>         16            93.9M    95.1M          71.6M    74.3M
> 
>     Initrd size: 3.0M
>     Kernel
>         image size: 28M
>         sections size [size -A -d vmlinux]:  18.9M
> 
> - boot time [ms]
>                           before                   now
>  qemu_init_end:           63.85                   55.91
>  linux_start_kernel:      82.11 (+18.26)          74.51 (+18.60)
>  linux_start_user:       169.94 (+87.83)         159.06 (+84.56)
> 
> QEMU command used:
> ./qemu-system-x86_64 -bios /path/to/seabios/out/bios.bin -no-hpet \
>     -machine q35,accel=kvm,kernel_irqchip,nvdimm,sata=off,smbus=off,vmport=off \
>     -cpu host -m 1G -smp 1 -vga none -display none -no-user-config -nodefaults \
>     -kernel /path/to/vmlinux -initrd /path/to/rootfs.cpio \
>     -append 'root=/dev/mem0 ro console=hvc0 pci=lastbus=0 nosmap'
> 
> Stefano Garzarella (3):
>   loader: Handle memory-mapped ELFs
>   elf-ops.h: Map into memory the ELF to load
>   hw/i386/pc: Map into memory the initrd
> 
>  hw/core/loader.c     | 38 +++++++++++++++++++-----
>  hw/i386/pc.c         | 17 ++++++++---
>  include/hw/elf_ops.h | 71 ++++++++++++++++++++++++++------------------
>  include/hw/i386/pc.h |  1 +
>  include/hw/loader.h  |  5 ++--
>  5 files changed, 89 insertions(+), 43 deletions(-)
> 

Queued, thanks.

Paolo

Re: [Qemu-devel] [PATCH v4 0/3] pc: mmap kernel (ELF image) and initrd

Posted by Dr. David Alan Gilbert 36 weeks ago
* Stefano Garzarella (sgarzare@redhat.com) wrote:
> In order to reduce the memory footprint when PVH kernel and initrd
> are used, we map them into memory instead of reading them.
> In this way we can share them between multiple instances of QEMU.
> 
> v4:
>   - Patch 1: fix the rom_add_elf_program() comment [Paolo]
>   - Patch 2:
>     ~ fix the missing of g_mapped_file_unref() in the success case [Paolo]
>     ~ fix the rom_add_elf_program() comment [Paolo]
> 
> v3: https://patchew.org/QEMU/20190724112531.232260-1-sgarzare@redhat.com/
> v2: https://patchew.org/QEMU/20190723140445.12748-1-sgarzare@redhat.com/

Two high level questions:
   a) What happens if someone tries to migrate the VM - I don't think
it's too unusual for people to run with -kernel/-initrd in situations
where they migrate.

   b) Are there situations where you can't mmap but you can validly
read it?  For example, running with an ELF built for 4k page alignment
on a host with 64k host pages?

Dave

> 
> These are the results using a PVH kernel and initrd (cpio):
> - memory footprint (using smem) [MB]
>         QEMU              before                   now
>     # instances        USS      PSS            USS      PSS
>          1           102.0M   105.8M         102.3M   106.2M
>          2            94.6M   101.2M          72.3M    90.1M
>          4            94.1M    98.0M          72.0M    81.5M
>          8            94.0M    96.2M          71.8M    76.9M
>         16            93.9M    95.1M          71.6M    74.3M
> 
>     Initrd size: 3.0M
>     Kernel
>         image size: 28M
>         sections size [size -A -d vmlinux]:  18.9M
> 
> - boot time [ms]
>                           before                   now
>  qemu_init_end:           63.85                   55.91
>  linux_start_kernel:      82.11 (+18.26)          74.51 (+18.60)
>  linux_start_user:       169.94 (+87.83)         159.06 (+84.56)
> 
> QEMU command used:
> ./qemu-system-x86_64 -bios /path/to/seabios/out/bios.bin -no-hpet \
>     -machine q35,accel=kvm,kernel_irqchip,nvdimm,sata=off,smbus=off,vmport=off \
>     -cpu host -m 1G -smp 1 -vga none -display none -no-user-config -nodefaults \
>     -kernel /path/to/vmlinux -initrd /path/to/rootfs.cpio \
>     -append 'root=/dev/mem0 ro console=hvc0 pci=lastbus=0 nosmap'
> 
> Stefano Garzarella (3):
>   loader: Handle memory-mapped ELFs
>   elf-ops.h: Map into memory the ELF to load
>   hw/i386/pc: Map into memory the initrd
> 
>  hw/core/loader.c     | 38 +++++++++++++++++++-----
>  hw/i386/pc.c         | 17 ++++++++---
>  include/hw/elf_ops.h | 71 ++++++++++++++++++++++++++------------------
>  include/hw/i386/pc.h |  1 +
>  include/hw/loader.h  |  5 ++--
>  5 files changed, 89 insertions(+), 43 deletions(-)
> 
> -- 
> 2.20.1
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK