[PATCH v7 00/19] Add support for qemu-xen runnning in a Linux-based stubdomain

Jason Andryuk posted 19 patches 22 weeks ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/xen tags/patchew/20200519015503.115236-1-jandryuk@gmail.com
Maintainers: Wei Liu <wl@xen.org>, Andrew Cooper <andrew.cooper3@citrix.com>, Anthony PERARD <anthony.perard@citrix.com>, Jan Beulich <jbeulich@suse.com>, Ian Jackson <ian.jackson@eu.citrix.com>, George Dunlap <george.dunlap@citrix.com>, Julien Grall <julien@xen.org>, Stefano Stabellini <sstabellini@kernel.org>
.gitignore                          |   1 +
configure                           |  14 +-
docs/configure                      |  14 +-
docs/man/xl.cfg.5.pod.in            |  27 +-
docs/misc/stubdom.txt               | 105 ++++++
docs/misc/xenstore-paths.pandoc     |   5 +
stubdom/configure                   |  14 +-
tools/Rules.mk                      |   2 +-
tools/config.h.in                   |   3 +
tools/configure                     |  46 ++-
tools/configure.ac                  |   9 +
tools/libvchan/Makefile             |   8 +-
tools/libvchan/init.c               |   3 +
tools/libvchan/vchan-socket-proxy.c | 478 ++++++++++++++++++++++++
tools/libxl/libxl_aoutils.c         |  32 ++
tools/libxl/libxl_create.c          |  46 ++-
tools/libxl/libxl_dm.c              | 558 +++++++++++++++++++++-------
tools/libxl/libxl_domain.c          |  10 +
tools/libxl/libxl_internal.h        |  22 ++
tools/libxl/libxl_mem.c             |   6 +-
tools/libxl/libxl_qmp.c             |  27 +-
tools/libxl/libxl_types.idl         |   3 +
tools/xl/xl_parse.c                 |   7 +
23 files changed, 1230 insertions(+), 210 deletions(-)
create mode 100644 tools/libvchan/vchan-socket-proxy.c

[PATCH v7 00/19] Add support for qemu-xen runnning in a Linux-based stubdomain

Posted by Jason Andryuk 22 weeks ago
Hi,

Below is mostly the v6 cover leter with a few additions.  All patches
are Acked except for
  libxl: Use libxl__xs_* in libxl__write_stub_dmargs
  libxl: write qemu arguments into separate xenstore keys

These are refactored into Ian's preferred form (I think).  A single loop
for the xenstore transaction with behaviour differing for MiniOS and
Linux stubdoms.

In coordination with Marek, I'm making a submission of his patches for Linux
stubdomain device-model support.  I made a few of my own additions, but Marek
did the heavy lifting.  Thank you, Marek.

General idea is to allow freely set device_model_version and
device_model_stubdomain_override and choose the right options based on this
choice.  Also, allow to specific path to stubdomain kernel/ramdisk, for greater
flexibility.

First two patches add documentation about expected toolstack-stubdomain-qemu
interface, both for MiniOS stubdomain and Linux stubdomain.

Initial version has no QMP support - in initial patches it is completely
disabled, which means no suspend/restore and no PCI passthrough.

Later patches add QMP over libvchan connection support. The actual connection
is made in a separate process. As discussed on Xen Summit 2019, this allows to
apply some basic checks and/or filtering (not part of this series), to limit
libxl exposure for potentially malicious stubdomain.

Jason's additions ensure the qmp-proxy (vchan-socket-proxy) processes and
sockets are cleaned up and add some documentation.

The actual stubdomain implementation is here:

   https://github.com/jandryuk/qubes-vmm-xen-stubdom-linux
   (branch initramfs-tools, tag for-upstream-v6)

See readme there for build instructions.  Marek's version requires dracut.  I
have hacked up a version usable with initramfs-tools.

The v6 version is needed to be compatible with these changes in the v6 posting:

 - Mini-OS stubdoms use dmargs xenstore key as a string.  Linux stubdoms
   use dm-argv as a directory for numbered entries.

 - The hardcoded "fd:3" for the restore image is replaced with the
   placehodler string $STUBDOM_RESTORE_INCOMING_ARG.  The stubdom
   initscript needs to replace that with a an "fd:$FD" string to the
   hooked up console 2.

Few comments/questions about the stubdomain code:

1. There are extra patches for qemu that are necessary to run it in stubdomain.
While it is desirable to upstream them, I think it can be done after merging
libxl part. Stubdomain's qemu build will in most cases be separate anyway, to
limit qemu's dependencies (so the stubdomain size).

2. By default Linux hvc-xen console frontend is unreliable for data transfer
(qemu state save/restore) - it drops data sent faster than client is reading
it. To fix it, console device needs to be switched into raw mode (`stty raw
/dev/hvc1`). Especially for restoring qemu state it is tricky, as it would need
to be done before opening the device, but stty (obviously) needs to open the
device first. To solve this problem, for now the repository contains kernel
patch which changes the default for all hvc consoles. Again, this isn't
practical problem, as the kernel for stubdomain is built separately. But it
would be nice to have something working with vanilla kernel. I see those
options:
  - convert it to kernel cmdline parameter (hvc_console_raw=1 ?)
  - use channels instead of consoles (and on the kernel side change the default
    to "raw" only for channels); while in theory better design, libxl part will
    be more complex, as channels can be connected to sockets but not files, so
    libxl would need to read/write to it exactly when qemu write/read the data,
    not before/after as it is done now

Remaining parts for eliminating dom0's instance of qemu:
 - do not force QDISK backend for CDROM
 - multiple consoles support in xenconsoled

Changes in v2:
 - apply review comments by Jason Andryuk
Changes in v3:
 - rework qemu arguments handling (separate xenstore keys, instead of \x1b separator)
 - add QMP over libvchan, instead of console
 - add protocol documentation
 - a lot of minor changes, see individual patches for full changes list
 - split xenconsoled patches into separate series
Changes in v4:
 - extract vchan connection into a separate process
 - rebase on master
 - various fixes
Changes in v5:
 - Marek: apply review comments from Jason Andryuk
 - Jason: Clean up qmp-proxy processes and sockets
Changes in v6:
 - Squash vchan-proxy kill and socket cleanup into "libxl: use vchan for
   QMP access with Linux stubdomain".
 - Use dm-argv as the xenstore directory for the QEMU arguments.
 - Use $STUBDOM_RESTORE_INCOMING_ARG as a placeholder instead of
   hardcoding "fd:3".
 - Comment to re-run autotools.
 - Add Acked-by from Ian Jackson where approriate.
Changes in v7:
 - Have libxl__write_stub_dmargs handle both minios and linux qemu
   arguments.
 - Add Acked-by from Ian Jackson where approriate.

Eric Shelton (1):
  libxl: Handle Linux stubdomain specific QEMU options.

Jason Andryuk (4):
  libxl: Use libxl__xs_* in libxl__write_stub_dmargs
  libxl: Refactor kill_device_model to libxl__kill_xs_path
  docs: Add device-model-domid to xenstore-paths
  libxl: Check stubdomain kernel & ramdisk presence

Marek Marczykowski-Górecki (14):
  Document ioemu MiniOS stubdomain protocol
  Document ioemu Linux stubdomain protocol
  libxl: fix qemu-trad cmdline for no sdl/vnc case
  libxl: Allow running qemu-xen in stubdomain
  libxl: write qemu arguments into separate xenstore keys
  xl: add stubdomain related options to xl config parser
  tools/libvchan: notify server when client is connected
  libxl: add save/restore support for qemu-xen in stubdomain
  tools: add missing libxenvchan cflags
  tools: add simple vchan-socket-proxy
  libxl: use vchan for QMP access with Linux stubdomain
  libxl: require qemu in dom0 for multiple stubdomain consoles
  libxl: ignore emulated IDE disks beyond the first 4
  libxl: consider also qemu in stubdomain in libxl__dm_active check

 .gitignore                          |   1 +
 configure                           |  14 +-
 docs/configure                      |  14 +-
 docs/man/xl.cfg.5.pod.in            |  27 +-
 docs/misc/stubdom.txt               | 105 ++++++
 docs/misc/xenstore-paths.pandoc     |   5 +
 stubdom/configure                   |  14 +-
 tools/Rules.mk                      |   2 +-
 tools/config.h.in                   |   3 +
 tools/configure                     |  46 ++-
 tools/configure.ac                  |   9 +
 tools/libvchan/Makefile             |   8 +-
 tools/libvchan/init.c               |   3 +
 tools/libvchan/vchan-socket-proxy.c | 478 ++++++++++++++++++++++++
 tools/libxl/libxl_aoutils.c         |  32 ++
 tools/libxl/libxl_create.c          |  46 ++-
 tools/libxl/libxl_dm.c              | 558 +++++++++++++++++++++-------
 tools/libxl/libxl_domain.c          |  10 +
 tools/libxl/libxl_internal.h        |  22 ++
 tools/libxl/libxl_mem.c             |   6 +-
 tools/libxl/libxl_qmp.c             |  27 +-
 tools/libxl/libxl_types.idl         |   3 +
 tools/xl/xl_parse.c                 |   7 +
 23 files changed, 1230 insertions(+), 210 deletions(-)
 create mode 100644 tools/libvchan/vchan-socket-proxy.c

-- 
2.25.1


Re: [PATCH v7 00/19] Add support for qemu-xen runnning in a Linux-based stubdomain

Posted by George Dunlap 21 weeks ago
> On May 19, 2020, at 2:54 AM, Jason Andryuk <jandryuk@gmail.com> wrote:
> 
> General idea is to allow freely set device_model_version and
> device_model_stubdomain_override and choose the right options based on this
> choice.  Also, allow to specific path to stubdomain kernel/ramdisk, for greater
> flexibility.

Excited to see this patch series get in.  But I didn’t really notice any documents explaining how to actually use it — is there a blog post anywhere describing how to get the kernel / initrd image and so on?

Also, would it be possible to add a follow-up series which modifies SUPPORT.md and CHANGELOG.md?

Thanks,
 -George

RE: [PATCH v7 00/19] Add support for qemu-xen runnning in a Linux-based stubdomain

Posted by Paul Durrant 21 weeks ago
> -----Original Message-----
> From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of George Dunlap
> Sent: 22 May 2020 10:11
> To: Jason Andryuk <jandryuk@gmail.com>
> Cc: Stefano Stabellini <sstabellini@kernel.org>; Julien Grall <julien@xen.org>; Samuel Thibault
> <samuel.thibault@ens-lyon.org>; Wei Liu <wl@xen.org>; Andrew Cooper <Andrew.Cooper3@citrix.com>; Jan
> Beulich <jbeulich@suse.com>; Ian Jackson <Ian.Jackson@citrix.com>; Anthony Perard
> <anthony.perard@citrix.com>; xen-devel <xen-devel@lists.xenproject.org>; Daniel De Graaf
> <dgdegra@tycho.nsa.gov>
> Subject: Re: [PATCH v7 00/19] Add support for qemu-xen runnning in a Linux-based stubdomain
> 
> 
> > On May 19, 2020, at 2:54 AM, Jason Andryuk <jandryuk@gmail.com> wrote:
> >
> > General idea is to allow freely set device_model_version and
> > device_model_stubdomain_override and choose the right options based on this
> > choice.  Also, allow to specific path to stubdomain kernel/ramdisk, for greater
> > flexibility.
> 
> Excited to see this patch series get in.  But I didn’t really notice any documents explaining how to
> actually use it — is there a blog post anywhere describing how to get the kernel / initrd image and so
> on?
> 
> Also, would it be possible to add a follow-up series which modifies SUPPORT.md and CHANGELOG.md?

Yes please. In future I think we should encourage the patch to CHANGELOG.md to be the last patch of a series such as this.

  Paul

> 
> Thanks,
>  -George


Re: [PATCH v7 00/19] Add support for qemu-xen runnning in a Linux-based stubdomain

Posted by Jason Andryuk 21 weeks ago
On Fri, May 22, 2020 at 5:54 AM Paul Durrant <xadimgnik@gmail.com> wrote:
>
> > -----Original Message-----
> > From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of George Dunlap
> > Sent: 22 May 2020 10:11
> > To: Jason Andryuk <jandryuk@gmail.com>
> > Cc: Stefano Stabellini <sstabellini@kernel.org>; Julien Grall <julien@xen.org>; Samuel Thibault
> > <samuel.thibault@ens-lyon.org>; Wei Liu <wl@xen.org>; Andrew Cooper <Andrew.Cooper3@citrix.com>; Jan
> > Beulich <jbeulich@suse.com>; Ian Jackson <Ian.Jackson@citrix.com>; Anthony Perard
> > <anthony.perard@citrix.com>; xen-devel <xen-devel@lists.xenproject.org>; Daniel De Graaf
> > <dgdegra@tycho.nsa.gov>
> > Subject: Re: [PATCH v7 00/19] Add support for qemu-xen runnning in a Linux-based stubdomain
> >
> >
> > > On May 19, 2020, at 2:54 AM, Jason Andryuk <jandryuk@gmail.com> wrote:
> > >
> > > General idea is to allow freely set device_model_version and
> > > device_model_stubdomain_override and choose the right options based on this
> > > choice.  Also, allow to specific path to stubdomain kernel/ramdisk, for greater
> > > flexibility.
> >
> > Excited to see this patch series get in.  But I didn’t really notice any documents explaining how to
> > actually use it — is there a blog post anywhere describing how to get the kernel / initrd image and so
> > on?

Yeah, it's not really collected anywhere, but below are the quick
start instructions.

The cover letter mentioned this repo (forked from Marek's):
https://github.com/jandryuk/qubes-vmm-xen-stubdom-linux
   (branch initramfs-tools, tag for-upstream-v6)

clone it and then run:
$ make get-sources
$ make -f Makefile.stubdom

output:
kernel: build/linux/arch/x86/boot/bzImage
ramdisk: build/rootfs/stubdom-linux-rootfs

To make them available system wide, copy to
/usr/lib/xen/boot/qemu-stubdom-linux-kernel and
/usr/lib/xen/boot/qemu-stubdom-linux-rootfs respectively. Obviously
this should match your installation's "$lib/xen/boot/" location.

A second option is to set paths to those files manually in a VM's
xl.cfg with stubdomain_kernel="/path" and stubdomain_ramdisk="/path"

Update your xl configuration with:
device_model_stubdomain_override = 1
device_model_version = "qemu-xen"

Start the domain and that should be it.   Maybe additionally use
serial = "pty" to access the VM with `xl console -t serial $NAME`.

Some limitations are here:
https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=docs/misc/stubdom.txt;h=c717a95d17d2e562639a5574e89df3c4db8712fa;hb=HEAD#l124
Limitations:
 - PCI passthrough require permissive mode
 - only one nic is supported
 - at most 26 emulated disks are supported (more are still available
as PV disks)
 - graphics output (VNC/SDL/Spice) not supported

> > Also, would it be possible to add a follow-up series which modifies SUPPORT.md and CHANGELOG.md?
>
> Yes please. In future I think we should encourage the patch to CHANGELOG.md to be the last patch of a series such as this.

I can do this.  What is the SUPPORT status for this?

Regards,
Jason

Re: [PATCH v7 00/19] Add support for qemu-xen runnning in a Linux-based stubdomain

Posted by Ian Jackson 21 weeks ago
Jason Andryuk writes ("Re: [PATCH v7 00/19] Add support for qemu-xen runnning in a Linux-based stubdomain"):
> I can do this.  What is the SUPPORT status for this?

I think that given we aren't testing it upstream, the answer
probably has to be "Tech Preview".

In general, I would love to see this (including your stub builder)
being tested by osstest.

Ian.

Re: [PATCH v7 00/19] Add support for qemu-xen runnning in a Linux-based stubdomain

Posted by Wei Liu 22 weeks ago
I have queued up the first 5 patches for committing today.

Wei.