Since QEMU 2.12 there's a new vfio-pci device property 'display' with values
on/off/auto. This special kind of display allows using a mediated device which
is a VGA compatible device for a display output.
There are 2 different implementations of how the device output is handled,
referred to as dmabuf and vfio region mapping (currently NVIDIA uses the latter,
while Intel relies on the former). From libvirt's perspective the important
difference is that dmabuf requires OpenGL support whereas vfio regions don't
(it will of course work even with OpenGL enabled). There's a catch though -
because of several constraints in the vendor drivers (as discussed here [1]),
there currently isn't a reasonable way for libvirt (other than spawning a dummy
 QEMU instance) to probe such mediated devices for the display mode they use.
This the nr.1 reason why libvirt is not going to support the value 'auto' with
QEMU and will default to 'off' instead in all cases to stay safe the least and
therefore is going to rely on users being able to configure this properly
otherwise they'll get an error.

Once there's a way for libvirt to probe the nature of the display-capable
mediated devices, we can consider adding support for 'auto' value meaning that
libvirt is going to take care of adding an appropriate Spice/VNC graphics
device depending on the system if these are missing in the config, otherwise
the user's choice is always favoured.

TL;DR:
- we have a new attribute value for vfio-pci mediated devices called 'display'
    -> devices can now format this new 'display=on/off' property to the cmdline

- if user enables the vfio display (display=on) but doesn't enable OpenGL for
    Spice, we automatically assume the usage of '-display egl-headless'
    (uses local drm nodes) which works both for Spice and VNC
    -> if OpenGL is enabled, then '-display egl-headless' is not necessary

Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1475770
[1] https://www.redhat.com/archives/libvir-list/2018-May/msg00243.html

Erik Skultety (7):
  conf: Remove a redundant model/address-type check in mdev post parse
  qemu: command: Move graphics iteration to its own function
  conf: Introduce virDomainDefHasSpiceGL helper
  conf: Introduce new <hostdev> attribute 'display'
  qemu: caps: Add vfio-pci.display capability
  qemu: domain: Set default vfio-pci display value depending on
    capability
  qemu: command: Enable formatting vfio-pci.display option onto cmdline

 docs/formatdomain.html.in                          | 16 ++++-
 docs/schemas/domaincommon.rng                      |  5 ++
 src/conf/domain_conf.c                             | 56 +++++++++++----
 src/conf/domain_conf.h                             |  4 ++
 src/libvirt_private.syms                           |  1 +
 src/qemu/qemu_capabilities.c                       |  2 +
 src/qemu/qemu_capabilities.h                       |  1 +
 src/qemu/qemu_command.c                            | 84 +++++++++++++++-------
 src/qemu/qemu_domain.c                             | 75 +++++++++++++++++++
 tests/qemucapabilitiesdata/caps_2.12.0.aarch64.xml |  1 +
 tests/qemucapabilitiesdata/caps_2.12.0.ppc64.xml   |  1 +
 tests/qemucapabilitiesdata/caps_2.12.0.s390x.xml   |  1 +
 tests/qemucapabilitiesdata/caps_2.12.0.x86_64.xml  |  1 +
 .../hostdev-mdev-display-missing-graphics.xml      | 35 +++++++++
 .../hostdev-mdev-display-spice-no-opengl.args      | 32 +++++++++
 .../hostdev-mdev-display-spice-no-opengl.xml       | 41 +++++++++++
 .../hostdev-mdev-display-spice-opengl.args         | 31 ++++++++
 .../hostdev-mdev-display-spice-opengl.xml          | 41 +++++++++++
 .../qemuxml2argvdata/hostdev-mdev-display-vnc.args | 32 +++++++++
 .../qemuxml2argvdata/hostdev-mdev-display-vnc.xml  | 39 ++++++++++
 tests/qemuxml2argvtest.c                           | 23 ++++++
 .../hostdev-mdev-display-spice-no-opengl.xml       | 47 ++++++++++++
 .../hostdev-mdev-display-spice-opengl.xml          | 48 +++++++++++++
 .../hostdev-mdev-display-vnc.xml                   | 47 ++++++++++++
 tests/qemuxml2xmltest.c                            |  3 +
 25 files changed, 626 insertions(+), 41 deletions(-)
 create mode 100644 tests/qemuxml2argvdata/hostdev-mdev-display-missing-graphics.xml
 create mode 100644 tests/qemuxml2argvdata/hostdev-mdev-display-spice-no-opengl.args
 create mode 100644 tests/qemuxml2argvdata/hostdev-mdev-display-spice-no-opengl.xml
 create mode 100644 tests/qemuxml2argvdata/hostdev-mdev-display-spice-opengl.args
 create mode 100644 tests/qemuxml2argvdata/hostdev-mdev-display-spice-opengl.xml
 create mode 100644 tests/qemuxml2argvdata/hostdev-mdev-display-vnc.args
 create mode 100644 tests/qemuxml2argvdata/hostdev-mdev-display-vnc.xml
 create mode 100644 tests/qemuxml2xmloutdata/hostdev-mdev-display-spice-no-opengl.xml
 create mode 100644 tests/qemuxml2xmloutdata/hostdev-mdev-display-spice-opengl.xml
 create mode 100644 tests/qemuxml2xmloutdata/hostdev-mdev-display-vnc.xml

--
2.14.3

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

CC'ing Gerd to comment since he implemented the feature in QEMU.

Erik

On Wed, May 30, 2018 at 03:42:54PM +0200, Erik Skultety wrote:
> Since QEMU 2.12 there's a new vfio-pci device property 'display' with values
> on/off/auto. This special kind of display allows using a mediated device which
> is a VGA compatible device for a display output.
> There are 2 different implementations of how the device output is handled,
> referred to as dmabuf and vfio region mapping (currently NVIDIA uses the latter,
> while Intel relies on the former). From libvirt's perspective the important
> difference is that dmabuf requires OpenGL support whereas vfio regions don't
> (it will of course work even with OpenGL enabled). There's a catch though -
> because of several constraints in the vendor drivers (as discussed here [1]),
> there currently isn't a reasonable way for libvirt (other than spawning a dummy
>  QEMU instance) to probe such mediated devices for the display mode they use.
> This the nr.1 reason why libvirt is not going to support the value 'auto' with
> QEMU and will default to 'off' instead in all cases to stay safe the least and
> therefore is going to rely on users being able to configure this properly
> otherwise they'll get an error.
>
> Once there's a way for libvirt to probe the nature of the display-capable
> mediated devices, we can consider adding support for 'auto' value meaning that
> libvirt is going to take care of adding an appropriate Spice/VNC graphics
> device depending on the system if these are missing in the config, otherwise
> the user's choice is always favoured.
>
> TL;DR:
> - we have a new attribute value for vfio-pci mediated devices called 'display'
>     -> devices can now format this new 'display=on/off' property to the cmdline
>
> - if user enables the vfio display (display=on) but doesn't enable OpenGL for
>     Spice, we automatically assume the usage of '-display egl-headless'
>     (uses local drm nodes) which works both for Spice and VNC
>     -> if OpenGL is enabled, then '-display egl-headless' is not necessary
>
> Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1475770
> [1] https://www.redhat.com/archives/libvir-list/2018-May/msg00243.html
>
> Erik Skultety (7):
>   conf: Remove a redundant model/address-type check in mdev post parse
>   qemu: command: Move graphics iteration to its own function
>   conf: Introduce virDomainDefHasSpiceGL helper
>   conf: Introduce new <hostdev> attribute 'display'
>   qemu: caps: Add vfio-pci.display capability
>   qemu: domain: Set default vfio-pci display value depending on
>     capability
>   qemu: command: Enable formatting vfio-pci.display option onto cmdline
>
>  docs/formatdomain.html.in                          | 16 ++++-
>  docs/schemas/domaincommon.rng                      |  5 ++
>  src/conf/domain_conf.c                             | 56 +++++++++++----
>  src/conf/domain_conf.h                             |  4 ++
>  src/libvirt_private.syms                           |  1 +
>  src/qemu/qemu_capabilities.c                       |  2 +
>  src/qemu/qemu_capabilities.h                       |  1 +
>  src/qemu/qemu_command.c                            | 84 +++++++++++++++-------
>  src/qemu/qemu_domain.c                             | 75 +++++++++++++++++++
>  tests/qemucapabilitiesdata/caps_2.12.0.aarch64.xml |  1 +
>  tests/qemucapabilitiesdata/caps_2.12.0.ppc64.xml   |  1 +
>  tests/qemucapabilitiesdata/caps_2.12.0.s390x.xml   |  1 +
>  tests/qemucapabilitiesdata/caps_2.12.0.x86_64.xml  |  1 +
>  .../hostdev-mdev-display-missing-graphics.xml      | 35 +++++++++
>  .../hostdev-mdev-display-spice-no-opengl.args      | 32 +++++++++
>  .../hostdev-mdev-display-spice-no-opengl.xml       | 41 +++++++++++
>  .../hostdev-mdev-display-spice-opengl.args         | 31 ++++++++
>  .../hostdev-mdev-display-spice-opengl.xml          | 41 +++++++++++
>  .../qemuxml2argvdata/hostdev-mdev-display-vnc.args | 32 +++++++++
>  .../qemuxml2argvdata/hostdev-mdev-display-vnc.xml  | 39 ++++++++++
>  tests/qemuxml2argvtest.c                           | 23 ++++++
>  .../hostdev-mdev-display-spice-no-opengl.xml       | 47 ++++++++++++
>  .../hostdev-mdev-display-spice-opengl.xml          | 48 +++++++++++++
>  .../hostdev-mdev-display-vnc.xml                   | 47 ++++++++++++
>  tests/qemuxml2xmltest.c                            |  3 +
>  25 files changed, 626 insertions(+), 41 deletions(-)
>  create mode 100644 tests/qemuxml2argvdata/hostdev-mdev-display-missing-graphics.xml
>  create mode 100644 tests/qemuxml2argvdata/hostdev-mdev-display-spice-no-opengl.args
>  create mode 100644 tests/qemuxml2argvdata/hostdev-mdev-display-spice-no-opengl.xml
>  create mode 100644 tests/qemuxml2argvdata/hostdev-mdev-display-spice-opengl.args
>  create mode 100644 tests/qemuxml2argvdata/hostdev-mdev-display-spice-opengl.xml
>  create mode 100644 tests/qemuxml2argvdata/hostdev-mdev-display-vnc.args
>  create mode 100644 tests/qemuxml2argvdata/hostdev-mdev-display-vnc.xml
>  create mode 100644 tests/qemuxml2xmloutdata/hostdev-mdev-display-spice-no-opengl.xml
>  create mode 100644 tests/qemuxml2xmloutdata/hostdev-mdev-display-spice-opengl.xml
>  create mode 100644 tests/qemuxml2xmloutdata/hostdev-mdev-display-vnc.xml
>
> --
> 2.14.3
>
> --
> libvir-list mailing list
> libvir-list@redhat.com
> https://www.redhat.com/mailman/listinfo/libvir-list

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

  Hi,

> > TL;DR:
> > - we have a new attribute value for vfio-pci mediated devices called 'display'
> >     -> devices can now format this new 'display=on/off' property to the cmdline

Good.

> > - if user enables the vfio display (display=on) but doesn't enable OpenGL for
> >     Spice, we automatically assume the usage of '-display egl-headless'

Hmm, I think it would be better to have egl-headless explicitly
configured in the domain xml instead of doing it automagically depending
on configuration.

First, I'd prefer to allow non-opengl configuration for vfio displays.
nvidia has its own non-standard ways of doing things, which causes
problems now and then, so I'd like to have an easy way out in case
using egl-headless on nvidia fails to work properly.

It's a bit ugly that we have to depend on qemu throwing errors in case
the vfio display uses dma-bufs (and therefore requires opengl) then
because libvirt can't easily probe the vfio display type.

Second, you can use egl-headless for other use cases too, for example to
use virgl with vnc.

cheers,
  Gerd

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

On Tue, Jun 05, 2018 at 11:52:03AM +0200, Gerd Hoffmann wrote:
>   Hi,
>
> > > TL;DR:
> > > - we have a new attribute value for vfio-pci mediated devices called 'display'
> > >     -> devices can now format this new 'display=on/off' property to the cmdline
>
> Good.
>
> > > - if user enables the vfio display (display=on) but doesn't enable OpenGL for
> > >     Spice, we automatically assume the usage of '-display egl-headless'
>
> Hmm, I think it would be better to have egl-headless explicitly
> configured in the domain xml instead of doing it automagically depending
> on configuration.

I was thinking about that earlier, then we had a private conversation where I
asked why there's no documentation on egl-headless in QEMU to which you replied
that the future of egl-headless being uncertain, especially once there's remote
opengl renderer support (as opposed to local nowadays). So I took a bit of a
safe path here and only made the changes to the command line, thus leaving us
with a way of easily ditching that if QEMU decides to deprecate egl-headless
completely. So, if QEMU can guarantee supporting egl-headless (of course you
can discourage usage of it...) then we surely can expose it through the XML,
but I'd really wouldn't want to end up with an XML setting which we won't be
able to satisfy at some point in the future. More importantly though,
egl-headless was introduced in 2.10 IIRC, but I don't see any way for libvirt
to use any kind of introspection here, there's no capability for egl-headless,
I see it defined as a value for DisplayType in the qapi scheme (ui.json), but
I don't see any query command. The way I handled it here is easy, I'm querying
vfio display support which was introduced in 2.12 which is newer, but I'm not
sure about the virgl + vnc case below, because that one doesn't need vfio
display=on/off support.

>
> First, I'd prefer to allow non-opengl configuration for vfio displays.
> nvidia has its own non-standard ways of doing things, which causes
> problems now and then, so I'd like to have an easy way out in case
> using egl-headless on nvidia fails to work properly.
>
> It's a bit ugly that we have to depend on qemu throwing errors in case
> the vfio display uses dma-bufs (and therefore requires opengl) then
> because libvirt can't easily probe the vfio display type.
>
> Second, you can use egl-headless for other use cases too, for example to
> use virgl with vnc.

Yep, that makes sense for us to support.

Erik

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

On Wed, Jun 06, 2018 at 02:49:24PM +0200, Erik Skultety wrote:
> On Tue, Jun 05, 2018 at 11:52:03AM +0200, Gerd Hoffmann wrote:
> >   Hi,
> >
> > > > TL;DR:
> > > > - we have a new attribute value for vfio-pci mediated devices called 'display'
> > > >     -> devices can now format this new 'display=on/off' property to the cmdline
> >
> > Good.
> >
> > > > - if user enables the vfio display (display=on) but doesn't enable OpenGL for
> > > >     Spice, we automatically assume the usage of '-display egl-headless'
> >
> > Hmm, I think it would be better to have egl-headless explicitly
> > configured in the domain xml instead of doing it automagically depending
> > on configuration.
> 
> I was thinking about that earlier, then we had a private conversation where I
> asked why there's no documentation on egl-headless in QEMU to which you replied
> that the future of egl-headless being uncertain, especially once there's remote
> opengl renderer support (as opposed to local nowadays). So I took a bit of a
> safe path here and only made the changes to the command line, thus leaving us
> with a way of easily ditching that if QEMU decides to deprecate egl-headless
> completely. So, if QEMU can guarantee supporting egl-headless (of course you
> can discourage usage of it...) then we surely can expose it through the XML,

Well, I guess if libvirt uses it it pretty much has to stay anyway ...

It isn't much of a burden code-wise, most of the underlying egl/opengl
code is needed for -spice gl=on too.  It is just that once spice got
remote support for gl=on mode it is alot less useful because native
spice support clearly is the better route performance-wise.

> egl-headless was introduced in 2.10 IIRC, but I don't see any way for libvirt
> to use any kind of introspection here, there's no capability for egl-headless,

Yea, right, isn't there.

> I see it defined as a value for DisplayType in the qapi scheme (ui.json), but
> I don't see any query command.

Also the enum is alwayws present, no matter whenever qemu was compiled
with opengl support or not.  So even in case you could query it it
wouldn't buy you anything.

How is spice/vnc support probed for btw?

cheers,
  Gerd

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list

On Thu, Jun 07, 2018 at 08:01:23AM +0200, Gerd Hoffmann wrote:
> On Wed, Jun 06, 2018 at 02:49:24PM +0200, Erik Skultety wrote:
> > On Tue, Jun 05, 2018 at 11:52:03AM +0200, Gerd Hoffmann wrote:
> > >   Hi,
> > >
> > > > > TL;DR:
> > > > > - we have a new attribute value for vfio-pci mediated devices called 'display'
> > > > >     -> devices can now format this new 'display=on/off' property to the cmdline
> > >
> > > Good.
> > >
> > > > > - if user enables the vfio display (display=on) but doesn't enable OpenGL for
> > > > >     Spice, we automatically assume the usage of '-display egl-headless'
> > >
> > > Hmm, I think it would be better to have egl-headless explicitly
> > > configured in the domain xml instead of doing it automagically depending
> > > on configuration.
> >
> > I was thinking about that earlier, then we had a private conversation where I
> > asked why there's no documentation on egl-headless in QEMU to which you replied
> > that the future of egl-headless being uncertain, especially once there's remote
> > opengl renderer support (as opposed to local nowadays). So I took a bit of a
> > safe path here and only made the changes to the command line, thus leaving us
> > with a way of easily ditching that if QEMU decides to deprecate egl-headless
> > completely. So, if QEMU can guarantee supporting egl-headless (of course you
> > can discourage usage of it...) then we surely can expose it through the XML,
>
> Well, I guess if libvirt uses it it pretty much has to stay anyway ...

It would be beneficial to add it to the man page and/or --help output then.

>
> It isn't much of a burden code-wise, most of the underlying egl/opengl
> code is needed for -spice gl=on too.  It is just that once spice got
> remote support for gl=on mode it is alot less useful because native
> spice support clearly is the better route performance-wise.
>
> > egl-headless was introduced in 2.10 IIRC, but I don't see any way for libvirt
> > to use any kind of introspection here, there's no capability for egl-headless,
>
> Yea, right, isn't there.
>
> > I see it defined as a value for DisplayType in the qapi scheme (ui.json), but
> > I don't see any query command.
>
> Also the enum is alwayws present, no matter whenever qemu was compiled
> with opengl support or not.  So even in case you could query it it
> wouldn't buy you anything.

In which case the only thing we can do is let qemu fail with an internal error
rather than failing in libvirt because of a missing capability.

>
> How is spice/vnc support probed for btw?

we execute 'query-commands' and then looking for in "query-spice" and
"query-vnc" literals in the list of returned commands.

Erik

--
libvir-list mailing list
libvir-list@redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list