[Xen-devel] [PATCH v4 00/16] Add support for qemu-xen runnning in a Linux-based stubdomain.

Marek Marczykowski-Górecki posted 16 patches 4 years, 2 months ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/xen tags/patchew/cover.f819645cd9f5cf7a6f692f9661cfb4e670a2cd08.1579055705.git-series.marmarek@invisiblethingslab.com
.gitignore                          |   1 +-
configure                           |  14 +-
docs/configure                      |  14 +-
docs/man/xl.cfg.5.pod.in            |  23 +-
docs/misc/stubdom.txt               | 103 ++++++-
stubdom/configure                   |  14 +-
tools/Rules.mk                      |   2 +-
tools/config.h.in                   |   3 +-
tools/configure                     |  46 +--
tools/configure.ac                  |   9 +-
tools/libvchan/Makefile             |   7 +-
tools/libvchan/init.c               |   3 +-
tools/libvchan/init.c.rej           |  60 ++++-
tools/libvchan/vchan-socket-proxy.c | 469 +++++++++++++++++++++++++++++-
tools/libxl/libxl_create.c          |  37 +-
tools/libxl/libxl_dm.c              | 437 ++++++++++++++++++++++-----
tools/libxl/libxl_internal.h        |  19 +-
tools/libxl/libxl_mem.c             |   6 +-
tools/libxl/libxl_qmp.c             |  25 +-
tools/libxl/libxl_types.idl         |   3 +-
tools/xl/xl_parse.c                 |   7 +-
21 files changed, 1151 insertions(+), 151 deletions(-)
create mode 100644 tools/libvchan/init.c.rej
create mode 100644 tools/libvchan/vchan-socket-proxy.c
[Xen-devel] [PATCH v4 00/16] Add support for qemu-xen runnning in a Linux-based stubdomain.
Posted by Marek Marczykowski-Górecki 4 years, 2 months ago
General idea is to allow freely set device_model_version and
device_model_stubdomain_override and choose the right options based on this choice.
Also, allow to specific path to stubdomain kernel/ramdisk, for greater flexibility.

First two patches add documentation about expected toolstack-stubdomain-qemu
interface, both for MiniOS stubdomain and Linux stubdomain.

Initial version has no QMP support - in initial patches it is completely
disabled, which means no suspend/restore and no PCI passthrough.

Later patches add QMP over libvchan connection support. The actual connection
is made in a separate process. As discussed on Xen Summit 2019, this allows to
apply some basic checks and/or filtering (not part of this series), to limit
libxl exposure for potentially malicious stubdomain.

The actual stubdomain implementation is here:

    https://github.com/marmarek/qubes-vmm-xen-stubdom-linux
    (branch for-upstream, tag for-upstream-v3)

See readme there for build instructions.
Beware: building on Debian is dangerous, as it require installing "dracut",
which will remove initramfs-tools. You may end up with broken initrd on
your host.

Few comments/questions about the stubdomain code:

1. There are extra patches for qemu that are necessary to run it in stubdomain.
While it is desirable to upstream them, I think it can be done after merging
libxl part. Stubdomain's qemu build will in most cases be separate anyway, to
limit qemu's dependencies (so the stubdomain size).

2. By default Linux hvc-xen console frontend is unreliable for data transfer
(qemu state save/restore) - it drops data sent faster than client is reading
it. To fix it, console device needs to be switched into raw mode
(`stty raw /dev/hvc1`). Especially for restoring qemu state it is tricky, as it
would need to be done before opening the device, but stty (obviously) needs to
open the device first. To solve this problem, for now the repository contains
kernel patch which changes the default for all hvc consoles. Again, this isn't
practical problem, as the kernel for stubdomain is built separately. But it
would be nice to have something working with vanilla kernel. I see those options:
  - convert it to kernel cmdline parameter (hvc_console_raw=1 ?)
  - use channels instead of consoles (and on the kernel side change the default
    to "raw" only for channels); while in theory better design, libxl part will
    be more complex, as channels can be connected to sockets but not files, so
    libxl would need to read/write to it exactly when qemu write/read the data,
    not before/after as it is done now

Remaining parts for eliminating dom0's instance of qemu:
 - do not force QDISK backend for CDROM
 - multiple consoles support in xenconsoled

Changes in v2:
 - apply review comments by Jason Andryuk
Changes in v3:
 - rework qemu arguments handling (separate xenstore keys, instead of \x1b separator)
 - add QMP over libvchan, instead of console
 - add protocol documentation
 - a lot of minor changes, see individual patches for full changes list
 - split xenconsoled patches into separate series
Changes in v4:
 - extract vchan connection into a separate process
 - rebase on master
 - various fixes

Cc: Simon Gaiser <simon@invisiblethingslab.com>
Cc: Eric Shelton <eshelton@pobox.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Wei Liu <wei.liu2@citrix.com>

Eric Shelton (1):
  libxl: Handle Linux stubdomain specific QEMU options.

Marek Marczykowski-Górecki (15):
  Document ioemu MiniOS stubdomain protocol
  Document ioemu Linux stubdomain protocol
  libxl: fix qemu-trad cmdline for no sdl/vnc case
  libxl: Allow running qemu-xen in stubdomain
  libxl: write qemu arguments into separate xenstore keys
  xl: add stubdomain related options to xl config parser
  tools/libvchan: notify server when client is connected
  libxl: add save/restore support for qemu-xen in stubdomain
  tools: add missing libxenvchan cflags
  tools: add simple vchan-socket-proxy
  libxl: use vchan for QMP access with Linux stubdomain
  Regenerate autotools files
  libxl: require qemu in dom0 even if stubdomain is in use
  libxl: ignore emulated IDE disks beyond the first 4
  libxl: consider also qemu in stubdomain in libxl__dm_active check

 .gitignore                          |   1 +-
 configure                           |  14 +-
 docs/configure                      |  14 +-
 docs/man/xl.cfg.5.pod.in            |  23 +-
 docs/misc/stubdom.txt               | 103 ++++++-
 stubdom/configure                   |  14 +-
 tools/Rules.mk                      |   2 +-
 tools/config.h.in                   |   3 +-
 tools/configure                     |  46 +--
 tools/configure.ac                  |   9 +-
 tools/libvchan/Makefile             |   7 +-
 tools/libvchan/init.c               |   3 +-
 tools/libvchan/init.c.rej           |  60 ++++-
 tools/libvchan/vchan-socket-proxy.c | 469 +++++++++++++++++++++++++++++-
 tools/libxl/libxl_create.c          |  37 +-
 tools/libxl/libxl_dm.c              | 437 ++++++++++++++++++++++-----
 tools/libxl/libxl_internal.h        |  19 +-
 tools/libxl/libxl_mem.c             |   6 +-
 tools/libxl/libxl_qmp.c             |  25 +-
 tools/libxl/libxl_types.idl         |   3 +-
 tools/xl/xl_parse.c                 |   7 +-
 21 files changed, 1151 insertions(+), 151 deletions(-)
 create mode 100644 tools/libvchan/init.c.rej
 create mode 100644 tools/libvchan/vchan-socket-proxy.c

base-commit: fae249d23413b2bf7d98a97d8f649cf7d102c1ae
-- 
git-series 0.9.1

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: [Xen-devel] [PATCH v4 00/16] Add support for qemu-xen runnning in a Linux-based stubdomain.
Posted by Jason Andryuk 4 years, 2 months ago
On Tue, Jan 14, 2020 at 9:42 PM Marek Marczykowski-Górecki
<marmarek@invisiblethingslab.com> wrote:

<snip>

> Later patches add QMP over libvchan connection support. The actual connection
> is made in a separate process. As discussed on Xen Summit 2019, this allows to
> apply some basic checks and/or filtering (not part of this series), to limit
> libxl exposure for potentially malicious stubdomain.

Thanks for working on this!  I think the separate process is nicer.

> The actual stubdomain implementation is here:
>
>     https://github.com/marmarek/qubes-vmm-xen-stubdom-linux
>     (branch for-upstream, tag for-upstream-v3)
>
> See readme there for build instructions.
> Beware: building on Debian is dangerous, as it require installing "dracut",
> which will remove initramfs-tools. You may end up with broken initrd on
> your host.

Just as an FYI, Marek's use of dracut is mainly for dracut-install to
copy a binary & dependent libraries when generating the initramfs
(https://github.com/marmarek/qubes-vmm-xen-stubdom-linux/blob/master/rootfs/gen).
The initramfs isn't running dracut scripts.  Using initramfs-tools
hook-functions:copy_exec() for similar functionality is a possibility.

> 1. There are extra patches for qemu that are necessary to run it in stubdomain.
> While it is desirable to upstream them, I think it can be done after merging
> libxl part. Stubdomain's qemu build will in most cases be separate anyway, to
> limit qemu's dependencies (so the stubdomain size).

A mostly unpatched QEMU works for networking & disk.  The exception is
PCI passthrough, which needs some patches.  I tested this by removing
patches from Marek's repo, except for the seccomp ones and
disable-nic-option-rom.patch.  Without disable-nic-option-rom.patch,
QEMU fails to start with 'failed to find romfile "efi-rtl8139.rom"'

One issue I've noticed is QEMU ~4.1 calls getrandom() during startup.
In a stubdom there is insufficient entropy, so QEMU blocks and stubdom
startup times out.  You can avoid getrandom() blocking with
CONFIG_RANDOM_TRUST_CPU or
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=50ee7529ec4500c88f8664560770a7a1b65db72b
or some other way of adding entropy.

Regards,
Jason

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel