[RFC PATCH v1 00/15] Xen PV backend support for KVM/Xen guests

David Woodhouse posted 15 patches 1 year, 3 months ago
Failed in applying to current master (apply log)
accel/xen/xen-all.c                           |   69 +-
hw/9pfs/meson.build                           |    2 +-
hw/9pfs/xen-9p-backend.c                      |   32 +-
hw/9pfs/xen-9pfs.h                            |    4 +-
hw/block/dataplane/meson.build                |    2 +-
hw/block/dataplane/xen-block.c                |   12 +-
hw/block/meson.build                          |    2 +-
hw/block/xen-block.c                          |    1 -
hw/block/xen_blkif.h                          |    5 +-
hw/char/meson.build                           |    2 +-
hw/char/xen_console.c                         |   59 +-
hw/display/meson.build                        |    2 +-
hw/display/xenfb.c                            |   38 +-
hw/i386/kvm/xen_evtchn.c                      |   15 +
hw/i386/kvm/xen_gnttab.c                      |  320 +++++++-
hw/i386/kvm/xen_gnttab.h                      |    1 +
hw/i386/kvm/xen_xenstore.c                    |   13 +-
hw/i386/pc_piix.c                             |   17 +
hw/i386/xen/xen-hvm.c                         |   45 +-
hw/i386/xen/xen-mapcache.c                    |    2 +-
hw/i386/xen/xen_platform.c                    |    7 +-
hw/net/xen_nic.c                              |   27 +-
hw/usb/meson.build                            |    2 +-
hw/usb/xen-usb.c                              |   31 +-
hw/xen/meson.build                            |    6 +-
hw/xen/trace-events                           |    2 +-
hw/xen/xen-bus-helper.c                       |   61 +-
hw/xen/xen-bus.c                              |  394 ++--------
hw/xen/xen-legacy-backend.c                   |  256 ++-----
hw/xen/xen-operations.c                       |  487 ++++++++++++
hw/xen/xen_devconfig.c                        |    4 +-
hw/xen/xen_pt.c                               |    2 +-
hw/xen/xen_pt.h                               |    2 +-
hw/xen/xen_pt_config_init.c                   |    2 +-
hw/xen/xen_pt_graphics.c                      |    1 -
hw/xen/xen_pt_msi.c                           |    4 +-
hw/xen/xen_pvdev.c                            |   63 +-
include/hw/xen/interface/grant_table.h        |   36 -
include/hw/xen/interface/io/blkif.h           |  712 -----------------
include/hw/xen/interface/io/console.h         |   46 --
include/hw/xen/interface/io/fbif.h            |  156 ----
include/hw/xen/interface/io/kbdif.h           |  566 --------------
include/hw/xen/interface/io/netif.h           | 1010 -------------------------
include/hw/xen/interface/io/protocols.h       |   42 -
include/hw/xen/interface/io/ring.h            |  474 ------------
include/hw/xen/interface/io/usbif.h           |  254 -------
include/hw/xen/interface/io/xenbus.h          |   70 --
include/hw/xen/xen-bus-helper.h               |   25 +-
include/hw/xen/xen-bus.h                      |   21 +-
include/hw/xen/xen-legacy-backend.h           |   24 +-
include/hw/xen/xen.h                          |   24 +-
include/hw/xen/xen_backend_ops.h              |  393 ++++++++++
include/hw/xen/{xen_common.h => xen_native.h} |   75 +-
include/hw/xen/xen_pvdev.h                    |    6 +-
softmmu/globals.c                             |    4 +
target/i386/kvm/xen-emu.c                     |    5 +
56 files changed, 1651 insertions(+), 4286 deletions(-)
[RFC PATCH v1 00/15] Xen PV backend support for KVM/Xen guests
Posted by David Woodhouse 1 year, 3 months ago
This is a very early preview of phase 2, in which we make the PV backend 
drivers build. Mostly by providing operations tables that can be backed 
either by the true Xen toolstack libraries, or by the internal emulated 
versions. But also a bunch of untangling of headers to keep things sane.

Some parts of it are relatively straightforward; others less so. In
particular, it looks really hard to provide a contiguous virtual mapping
of multiple potentially discontiguous pages granted by the guest. To
fix that we might actually need the guest memory blocks to be backed
by real files (perhaps deleted or shmem) in order that they can be mapped
again in at a different virtual address. So for now we'll limit the
back ends to mapping a single grant ref at a time.

https://git.infradead.org/users/dwmw2/qemu.git/shortlog/refs/heads/xenfv-kvm-backends-1

David Woodhouse (14):
      hw/xen: Add evtchn operations to allow redirection to internal emulation
      hw/xen: Add emulated evtchn ops
      hw/xen: Add gnttab operations to allow redirection to internal emulation
      hw/xen: Pass grant ref to gnttab unmap
      hw/xen: Add foreignmem operations to allow redirection to internal emulation
      hw/xen: Move xenstore_store_pv_console_info to xen_console.c
      hw/xen: Use XEN_PAGE_SIZE in PV backend drivers
      hw/xen: Rename xen_common.h to xen_native.h
      hw/xen: Build PV backend drivers for XENFV_MACHINE
      hw/xen: Map guest XENSTORE_PFN grant in emulated Xenstore
      hw/xen: Add backend implementation of grant table operations
      hw/xen: Implement soft reset for emulated gnttab
      hw/xen: Remove old version of Xen headers
      i386/xen: Initialize XenBus and legacy backends from pc_init1()

Paul Durrant (1):
      hw/xen: Add xenstore operations to allow redirection to internal emulation

 accel/xen/xen-all.c                           |   69 +-
 hw/9pfs/meson.build                           |    2 +-
 hw/9pfs/xen-9p-backend.c                      |   32 +-
 hw/9pfs/xen-9pfs.h                            |    4 +-
 hw/block/dataplane/meson.build                |    2 +-
 hw/block/dataplane/xen-block.c                |   12 +-
 hw/block/meson.build                          |    2 +-
 hw/block/xen-block.c                          |    1 -
 hw/block/xen_blkif.h                          |    5 +-
 hw/char/meson.build                           |    2 +-
 hw/char/xen_console.c                         |   59 +-
 hw/display/meson.build                        |    2 +-
 hw/display/xenfb.c                            |   38 +-
 hw/i386/kvm/xen_evtchn.c                      |   15 +
 hw/i386/kvm/xen_gnttab.c                      |  320 +++++++-
 hw/i386/kvm/xen_gnttab.h                      |    1 +
 hw/i386/kvm/xen_xenstore.c                    |   13 +-
 hw/i386/pc_piix.c                             |   17 +
 hw/i386/xen/xen-hvm.c                         |   45 +-
 hw/i386/xen/xen-mapcache.c                    |    2 +-
 hw/i386/xen/xen_platform.c                    |    7 +-
 hw/net/xen_nic.c                              |   27 +-
 hw/usb/meson.build                            |    2 +-
 hw/usb/xen-usb.c                              |   31 +-
 hw/xen/meson.build                            |    6 +-
 hw/xen/trace-events                           |    2 +-
 hw/xen/xen-bus-helper.c                       |   61 +-
 hw/xen/xen-bus.c                              |  394 ++--------
 hw/xen/xen-legacy-backend.c                   |  256 ++-----
 hw/xen/xen-operations.c                       |  487 ++++++++++++
 hw/xen/xen_devconfig.c                        |    4 +-
 hw/xen/xen_pt.c                               |    2 +-
 hw/xen/xen_pt.h                               |    2 +-
 hw/xen/xen_pt_config_init.c                   |    2 +-
 hw/xen/xen_pt_graphics.c                      |    1 -
 hw/xen/xen_pt_msi.c                           |    4 +-
 hw/xen/xen_pvdev.c                            |   63 +-
 include/hw/xen/interface/grant_table.h        |   36 -
 include/hw/xen/interface/io/blkif.h           |  712 -----------------
 include/hw/xen/interface/io/console.h         |   46 --
 include/hw/xen/interface/io/fbif.h            |  156 ----
 include/hw/xen/interface/io/kbdif.h           |  566 --------------
 include/hw/xen/interface/io/netif.h           | 1010 -------------------------
 include/hw/xen/interface/io/protocols.h       |   42 -
 include/hw/xen/interface/io/ring.h            |  474 ------------
 include/hw/xen/interface/io/usbif.h           |  254 -------
 include/hw/xen/interface/io/xenbus.h          |   70 --
 include/hw/xen/xen-bus-helper.h               |   25 +-
 include/hw/xen/xen-bus.h                      |   21 +-
 include/hw/xen/xen-legacy-backend.h           |   24 +-
 include/hw/xen/xen.h                          |   24 +-
 include/hw/xen/xen_backend_ops.h              |  393 ++++++++++
 include/hw/xen/{xen_common.h => xen_native.h} |   75 +-
 include/hw/xen/xen_pvdev.h                    |    6 +-
 softmmu/globals.c                             |    4 +
 target/i386/kvm/xen-emu.c                     |    5 +
 56 files changed, 1651 insertions(+), 4286 deletions(-)
Re: [RFC PATCH v1 00/15] Xen PV backend support for KVM/Xen guests
Posted by Joao Martins 1 year, 3 months ago
On 10/01/2023 12:37, David Woodhouse wrote:
> Some parts of it are relatively straightforward; others less so. In
> particular, it looks really hard to provide a contiguous virtual mapping
> of multiple potentially discontiguous pages granted by the guest. To
> fix that we might actually need the guest memory blocks to be backed
> by real files (perhaps deleted or shmem) in order that they can be mapped
> again in at a different virtual address. 

I wonder if we really need to go to that extent.

As far as Qemu emulated-Xen is concerned, gref is mostly a different lookup
mechanism a GPA i.e. an index on a table (that the VMM knows about) that
references a GPA. Perhaps if it's simpler to teach the backends to deal with
discontiguous grefs (if there's such a case today even).

The only user of multi-gref mapping is the block xen driver ... and only for
mapping the shared ring if I understood correctly. But even there you could
probably twist it... considering that the multi-gref ring is contiguous is guest
address space, thus the gref -> HVA conversion ius contiguous too (?). So that
way you could still return a single HVA (provided that map-grant implementation
validates the backing frame contiguosity).

> So for now we'll limit the
> back ends to mapping a single grant ref at a time.
> 
Which is not a practical limitation right now. One grant ref is actually fine
for the everything else that is not the block shared-ring.  Xen devices in Qemu
seem to be using the legacy backend interface, and thus (un)mapping one grant at
a time, or otherwise copying grefs.

> https://git.infradead.org/users/dwmw2/qemu.git/shortlog/refs/heads/xenfv-kvm-backends-1
> 
Cool stuff! A lot better than the RFC redirection layer

> David Woodhouse (14):
>       hw/xen: Remove old version of Xen headers

This patch looks more appropriate to your earlier v6 (?)
Re: [RFC PATCH v1 00/15] Xen PV backend support for KVM/Xen guests
Posted by David Woodhouse 1 year, 3 months ago
On Tue, 2023-01-10 at 15:43 +0000, Joao Martins wrote:
> On 10/01/2023 12:37, David Woodhouse wrote:
> > Some parts of it are relatively straightforward; others less so. In
> > particular, it looks really hard to provide a contiguous virtual mapping
> > of multiple potentially discontiguous pages granted by the guest. To
> > fix that we might actually need the guest memory blocks to be backed
> > by real files (perhaps deleted or shmem) in order that they can be mapped
> > again in at a different virtual address. 
> 
> I wonder if we really need to go to that extent.
> 
> As far as Qemu emulated-Xen is concerned, gref is mostly a different lookup
> mechanism a GPA i.e. an index on a table (that the VMM knows about) that
> references a GPA. Perhaps if it's simpler to teach the backends to deal with
> discontiguous grefs (if there's such a case today even).

We thought about making the back ends deal with discontiguous grefs by
returning an array[nr_refs] of pointers to each one.

The problem is that in some cases the ring entries don't even land on
page boundaries, leaving one entry crossing from one page to the next.
For example the 32-bit blkif response is 12 bytes, which means it lines
up at the end of the *first* page (64-byte ring header + 336 @12 byte
entries == 4096) but it won't line up at the end of the *second* page.

Which is still not insurmountable, but it's kind of awful.

> The only user of multi-gref mapping is the block xen driver ... and only for
> mapping the shared ring if I understood correctly. But even there you could
> probably twist it... considering that the multi-gref ring is contiguous is guest
> address space, thus the gref -> HVA conversion ius contiguous too (?). So that
> way you could still return a single HVA (provided that map-grant implementation
> validates the backing frame contiguosity).

There's no reason a guest couldn't vmalloc its multi-page ring and have
a virtually contiguous view for itself, while the actual GPAs are
discontiguous. Sure, that's probably not the common case, but I didn't
really want to have a multi-page mapping succeed *sometimes*.

> > So for now we'll limit the
> > back ends to mapping a single grant ref at a time.
>
> Which is not a practical limitation right now.

That sentence can be read both ways. I first read it as "that
limitation is impractical". But prefer to read it as "that doesn't
limit us in practice", and I *think* the latter is what you meant from
the rest of the paragraph...

>  One grant ref is actually fine
> for the everything else that is not the block shared-ring.  Xen devices in Qemu
> seem to be using the legacy backend interface, and thus (un)mapping one grant at
> a time, or otherwise copying grefs.

Right. We can absolutely live with that limitation for now; we'll teach
xen-block not to advertise a higher max-ring-page-order if ring_order
if the MAP_MULTIPLE feature isn't set.

And I absolutely don't give a crap about 9p or usb just yet. :)

> > https://git.infradead.org/users/dwmw2/qemu.git/shortlog/refs/heads/xenfv-kvm-backends-1
> > 
> Cool stuff! A lot better than the RFC redirection layer
> 
> > David Woodhouse (14):
> >       hw/xen: Remove old version of Xen headers
> 
> This patch looks more appropriate to your earlier v6 (?)

I would have liked that but it depends on a bit on the header
untangling that happens in between. As we introduce the backend ops
structures, rename xen_common.h → xen_native.h and remove that from the
C files that don't need it, we subtly fix things such that this final
patch removing the old headers makes it *look* easy. 



Re: [RFC PATCH v1 00/15] Xen PV backend support for KVM/Xen guests
Posted by Joao Martins 1 year, 3 months ago
On 10/01/2023 16:52, David Woodhouse wrote:
> On Tue, 2023-01-10 at 15:43 +0000, Joao Martins wrote:
>> On 10/01/2023 12:37, David Woodhouse wrote:
>> The only user of multi-gref mapping is the block xen driver ... and only for
>> mapping the shared ring if I understood correctly. But even there you could
>> probably twist it... considering that the multi-gref ring is contiguous is guest
>> address space, thus the gref -> HVA conversion ius contiguous too (?). So that
>> way you could still return a single HVA (provided that map-grant implementation
>> validates the backing frame contiguosity).
> 
> There's no reason a guest couldn't vmalloc its multi-page ring and have
> a virtually contiguous view for itself, while the actual GPAs are
> discontiguous. Sure, that's probably not the common case, but I didn't
> really want to have a multi-page mapping succeed *sometimes*.
> 
Right. I don't remember that ever being the case on frontends, but I could be
very wrong

This was just a suggesting not trying too hard to fix the problem while
maintaining single/contiguous gref limitation (if it didn't meant much more work
to you). Provided this only happens when configuring device. An alternative is
to /try finding GPA contiguosity/ and should it not be, then you switch device
xenbus status to error if you fail to map the guest grefs (This is a valid
scenario still in Xen kernel backends IIRC). Sadly backends don't re-read
features when re-connecting so probably not exposing ring-order at all as safest
if it can only be supported "half way"

>>> So for now we'll limit the
>>> back ends to mapping a single grant ref at a time.
>>
>> Which is not a practical limitation right now.
> 
> That sentence can be read both ways. I first read it as "that
> limitation is impractical". But prefer to read it as "that doesn't
> limit us in practice", and I *think* the latter is what you meant from
> the rest of the paragraph...
> 
Correct. The latter sounds like proper english on what I wanted to say

>>  One grant ref is actually fine
>> for the everything else that is not the block shared-ring.  Xen devices in Qemu
>> seem to be using the legacy backend interface, and thus (un)mapping one grant at
>> a time, or otherwise copying grefs.
> 
> Right. We can absolutely live with that limitation for now; we'll teach
> xen-block not to advertise a higher max-ring-page-order if ring_order
> if the MAP_MULTIPLE feature isn't set.
> 
> And I absolutely don't give a crap about 9p or usb just yet. :)
> 
:)

9p is actually for rings as well (same as block). USB is the only one where it
uses it in the datapath, which is likely to see issues more often.

Re: [RFC PATCH v1 00/15] Xen PV backend support for KVM/Xen guests
Posted by Joao Martins 1 year, 3 months ago
On 10/01/2023 15:43, Joao Martins wrote:
> On 10/01/2023 12:37, David Woodhouse wrote:
>> So for now we'll limit the
>> back ends to mapping a single grant ref at a time.
>>
> Which is not a practical limitation right now. 

Gah, no. xen-usb and 9p do use possibly-discontiguous grant maps :(

> One grant ref is actually fine
> for the everything else that is not the block shared-ring.  Xen devices in Qemu
> seem to be using the legacy backend interface, and thus (un)mapping one grant at
> a time, or otherwise copying grefs.
>