Q35 support using Qemu's device emulation.  I based the patches from 2017
found on the mailing list here:
https://lists.xenproject.org/archives/html/xen-devel/2018-03/msg01176.html

I have been using a version of these patches on Xen 4.16 with Qemu
version 4.1 for over 6 months.  The guest VMs are very stable, and PCIe
PT is working as was designed (all of the PCIe devices are on the root
PCIe device).  I have successfully passed through GPUs, NICs, etc. I was
asked by those in the community to attempt to once again upstream the
patches.  I have them working with Seabios and OVMF (patches are needed
to OVMF which I will be sending to the mailing list).  I will be sending
my Qemu patches to their mailing list in hopes of getting everything
upstreamed. The Qemu patches allow for the xenvbd to properly unplug the
AHCI SATA device, and all xen pv windows drivers work as intended.

I used the original author of the patches to get a majority of this to work:
Alexey Gerasimenko.  I fixed the patches to be in line with the upstream
Qemu and Xen versions.  Any original issues may still exist; however, I
am sure in time they can be improved. If the code doesn't exist then they 
can't be actively looked at by the community.

I am not an expert on the Q35 chipset or PCIe technology.  This is my
first patch to this mailing list.

Recap of changes to Qemu, only here for reference:
  pc/xen: Xen Q35 support: provide IRQ handling for PCI devices
  pc/q35: Apply PCI bus BSEL property for Xen PCI device hotplug
  acpi/ich9: Multiple fixes for S3 support
  acpi/pcihp: Allow for machines to hotplug when using Xen
  isa/lpc_ich9: Write lpc configuration for xen IRQs
  q35/acpi/xen: Provide ACPI PCI hotplug interface for Xen on Q35
  q35/xen: Add Xen platform device support for Q35
  q35: Fix incorrect values for PCIEXBAR masks
  xen/pt: XenHostPCIDevice: provide functions for PCI Capabilities and
    PCIe Extended Capabilities enumeration
  xen/pt: avoid reading PCIe device type and cap version multiple times
  xen/pt: determine the legacy/PCIe mode for a passed through device
  xen/pt: Xen PCIe passthrough support for Q35: bypass PCIe topology
    check
  xen/pt: add support for PCIe Extended Capabilities and larger config
    space
  xen/pt: handle PCIe Extended Capabilities Next register
  xen/pt: allow to hide PCIe Extended Capabilities
  xen/pt: add Vendor-specific PCIe Extended Capability descriptor and
    sizing
  xen/pt: add fixed-size PCIe Extended Capabilities descriptors
  xen/pt: add AER PCIe Extended Capability descriptor and sizing
  xen/pt: add descriptors and size calculation for
    RCLD/ACS/PMUX/DPA/MCAST/TPH/DPC PCIe Extended Capabilities
  xen/pt: add Resizable BAR PCIe Extended Capability descriptor and
    sizing
  xen/pt: add VC/VC9/MFVC PCIe Extended Capabilities descriptors and
    sizing

Changes that would be going to upstream Qemu:
 hw/acpi/ich9.c                |   22 +-
 hw/acpi/pcihp.c               |    6 +-
 hw/core/machine.c             |   19 +
 hw/i386/pc_piix.c             |    3 +-
 hw/i386/pc_q35.c              |   39 +-
 hw/i386/xen/xen-hvm.c         |    7 +-
 hw/i386/xen/xen_platform.c    |   19 +-
 hw/ide/ahci.c                 |    2 +-
 hw/isa/lpc_ich9.c             |   53 +-
 hw/isa/piix3.c                |    2 +-
 hw/pci-host/q35.c             |   28 +-
 hw/pci/pci.c                  |   17 +
 hw/xen/xen-host-pci-device.c  |  106 +++-
 hw/xen/xen-host-pci-device.h  |    6 +-
 hw/xen/xen_pt.c               |   49 +-
 hw/xen/xen_pt.h               |   19 +-
 hw/xen/xen_pt_config_init.c   | 1103 ++++++++++++++++++++++++++++++---
 include/hw/acpi/ich9.h        |    1 +
 include/hw/acpi/pcihp.h       |    2 +
 include/hw/boards.h           |    1 +
 include/hw/i386/pc.h          |    3 +
 include/hw/pci-host/q35.h     |    4 +-
 include/hw/pci/pci.h          |    3 +
 include/hw/southbridge/ich9.h |    1 +
 include/hw/xen/xen.h          |    4 +-
 qemu-options.hx               |    1 +
 softmmu/datadir.c             |    1 -
 softmmu/qdev-monitor.c        |    3 +-
 stubs/xen-hw-stub.c           |    4 +-
 29 files changed, 1396 insertions(+), 132 deletions(-)
*** These patches are not found in this series, if they are supposed to
be I will send them here as well. ***

Recap of changes to Xen:
  libacpi: new DSDT ACPI table for Q35
  Makefile: build and use new DSDT table for Q35
  hvmloader: add function to query an emulated machine type (i440/Q35)
  hvmloader: add ACPI enabling for Q35
  hvmloader: add Q35 DSDT table loading
  hvmloader: add basic Q35 support
  hvmloader: allocate MMCONFIG area in the MMIO hole + minor code
    refactoring
  hvmloader: seabios dsdt set to load correct config
  hvmloader: ovmf dsdt set to load correct config
  libxl: Q35 support (new option device_model_machine)
  libxl: Xen Platform device support for Q35
  libacpi: build ACPI MCFG table if requested
  hvmloader: use libacpi to build MCFG table
  docs: provide description for device_model_machine option

Joel Upham (1):
  Q35 support

 docs/man/xl.cfg.5.pod.in             |  27 ++
 tools/firmware/hvmloader/Makefile    |   2 +-
 tools/firmware/hvmloader/config.h    |   6 +
 tools/firmware/hvmloader/hvmloader.c |  13 +-
 tools/firmware/hvmloader/ovmf.c      |   7 +-
 tools/firmware/hvmloader/pci.c       | 285 +++++++++----
 tools/firmware/hvmloader/pci_regs.h  |   6 +
 tools/firmware/hvmloader/seabios.c   |   5 +
 tools/firmware/hvmloader/util.c      | 130 +++++-
 tools/firmware/hvmloader/util.h      |  10 +
 tools/libacpi/Makefile               |  10 +-
 tools/libacpi/acpi2_0.h              |  21 +
 tools/libacpi/build.c                |  43 ++
 tools/libacpi/dsdt_q35.asl           | 578 +++++++++++++++++++++++++++
 tools/libacpi/libacpi.h              |   4 +
 tools/libs/light/libxl_dm.c          |  60 ++-
 tools/libs/light/libxl_types.idl     |   8 +
 tools/xl/xl_parse.c                  |  14 +
 18 files changed, 1120 insertions(+), 109 deletions(-)
 create mode 100644 tools/libacpi/dsdt_q35.asl

-- 
2.34.1

On Sun, Jun 18, 2023 at 06:22:01PM -0400, Joel Upham wrote:
> Q35 support using Qemu's device emulation.  I based the patches from 2017
> found on the mailing list here:
> https://lists.xenproject.org/archives/html/xen-devel/2018-03/msg01176.html
> 
> I have been using a version of these patches on Xen 4.16 with Qemu
> version 4.1 for over 6 months.  The guest VMs are very stable, and PCIe
> PT is working as was designed (all of the PCIe devices are on the root
> PCIe device).  I have successfully passed through GPUs, NICs, etc. I was
> asked by those in the community to attempt to once again upstream the
> patches.  I have them working with Seabios and OVMF (patches are needed
> to OVMF which I will be sending to the mailing list).  I will be sending
> my Qemu patches to their mailing list in hopes of getting everything
> upstreamed. The Qemu patches allow for the xenvbd to properly unplug the
> AHCI SATA device, and all xen pv windows drivers work as intended.
> 
> I used the original author of the patches to get a majority of this to work:
> Alexey Gerasimenko.  I fixed the patches to be in line with the upstream
> Qemu and Xen versions.  Any original issues may still exist; however, I
> am sure in time they can be improved. If the code doesn't exist then they 
> can't be actively looked at by the community.
> 
> I am not an expert on the Q35 chipset or PCIe technology.  This is my
> first patch to this mailing list.
> 
> Recap of changes to Qemu, only here for reference:
>   pc/xen: Xen Q35 support: provide IRQ handling for PCI devices
>   pc/q35: Apply PCI bus BSEL property for Xen PCI device hotplug
>   acpi/ich9: Multiple fixes for S3 support
>   acpi/pcihp: Allow for machines to hotplug when using Xen
>   isa/lpc_ich9: Write lpc configuration for xen IRQs
>   q35/acpi/xen: Provide ACPI PCI hotplug interface for Xen on Q35
>   q35/xen: Add Xen platform device support for Q35
>   q35: Fix incorrect values for PCIEXBAR masks
>   xen/pt: XenHostPCIDevice: provide functions for PCI Capabilities and
>     PCIe Extended Capabilities enumeration
>   xen/pt: avoid reading PCIe device type and cap version multiple times
>   xen/pt: determine the legacy/PCIe mode for a passed through device
>   xen/pt: Xen PCIe passthrough support for Q35: bypass PCIe topology
>     check
>   xen/pt: add support for PCIe Extended Capabilities and larger config
>     space
>   xen/pt: handle PCIe Extended Capabilities Next register
>   xen/pt: allow to hide PCIe Extended Capabilities
>   xen/pt: add Vendor-specific PCIe Extended Capability descriptor and
>     sizing
>   xen/pt: add fixed-size PCIe Extended Capabilities descriptors
>   xen/pt: add AER PCIe Extended Capability descriptor and sizing
>   xen/pt: add descriptors and size calculation for
>     RCLD/ACS/PMUX/DPA/MCAST/TPH/DPC PCIe Extended Capabilities
>   xen/pt: add Resizable BAR PCIe Extended Capability descriptor and
>     sizing
>   xen/pt: add VC/VC9/MFVC PCIe Extended Capabilities descriptors and
>     sizing
> 
> Changes that would be going to upstream Qemu:
>  hw/acpi/ich9.c                |   22 +-
>  hw/acpi/pcihp.c               |    6 +-
>  hw/core/machine.c             |   19 +
>  hw/i386/pc_piix.c             |    3 +-
>  hw/i386/pc_q35.c              |   39 +-
>  hw/i386/xen/xen-hvm.c         |    7 +-
>  hw/i386/xen/xen_platform.c    |   19 +-
>  hw/ide/ahci.c                 |    2 +-
>  hw/isa/lpc_ich9.c             |   53 +-
>  hw/isa/piix3.c                |    2 +-
>  hw/pci-host/q35.c             |   28 +-
>  hw/pci/pci.c                  |   17 +
>  hw/xen/xen-host-pci-device.c  |  106 +++-
>  hw/xen/xen-host-pci-device.h  |    6 +-
>  hw/xen/xen_pt.c               |   49 +-
>  hw/xen/xen_pt.h               |   19 +-
>  hw/xen/xen_pt_config_init.c   | 1103 ++++++++++++++++++++++++++++++---
>  include/hw/acpi/ich9.h        |    1 +
>  include/hw/acpi/pcihp.h       |    2 +
>  include/hw/boards.h           |    1 +
>  include/hw/i386/pc.h          |    3 +
>  include/hw/pci-host/q35.h     |    4 +-
>  include/hw/pci/pci.h          |    3 +
>  include/hw/southbridge/ich9.h |    1 +
>  include/hw/xen/xen.h          |    4 +-
>  qemu-options.hx               |    1 +
>  softmmu/datadir.c             |    1 -
>  softmmu/qdev-monitor.c        |    3 +-
>  stubs/xen-hw-stub.c           |    4 +-
>  29 files changed, 1396 insertions(+), 132 deletions(-)
> *** These patches are not found in this series, if they are supposed to
> be I will send them here as well. ***

It's unclear, do the QEMU patches depend on the Xen side?  I have a
suspicion you likely want to post them mostly at the same time, but
using two different patch series (one for Xen and one for QEMU).

> Recap of changes to Xen:
>   libacpi: new DSDT ACPI table for Q35
>   Makefile: build and use new DSDT table for Q35
>   hvmloader: add function to query an emulated machine type (i440/Q35)
>   hvmloader: add ACPI enabling for Q35
>   hvmloader: add Q35 DSDT table loading
>   hvmloader: add basic Q35 support
>   hvmloader: allocate MMCONFIG area in the MMIO hole + minor code
>     refactoring
>   hvmloader: seabios dsdt set to load correct config
>   hvmloader: ovmf dsdt set to load correct config
>   libxl: Q35 support (new option device_model_machine)
>   libxl: Xen Platform device support for Q35
>   libacpi: build ACPI MCFG table if requested
>   hvmloader: use libacpi to build MCFG table
>   docs: provide description for device_model_machine option
> 
> Joel Upham (1):
>   Q35 support

I'm afraid this needs to be split into a patch series, with proper
commit messages in order to be considered for review.  The current
huge patch is impossible to review at all.

Also you need to keep the Signed-off-by of the original author in the
patches.

Thanks, Roger.

I actually realized that after submitting yesterday and have been splitting
them like the original had done for both Qemu and Xen. Thanks again for
getting back to me.

-Joel

On Mon, Jun 19, 2023 at 7:22 AM Roger Pau Monné <roger.pau@citrix.com>
wrote:

> On Sun, Jun 18, 2023 at 06:22:01PM -0400, Joel Upham wrote:
> > Q35 support using Qemu's device emulation.  I based the patches from 2017
> > found on the mailing list here:
> >
> https://lists.xenproject.org/archives/html/xen-devel/2018-03/msg01176.html
> >
> > I have been using a version of these patches on Xen 4.16 with Qemu
> > version 4.1 for over 6 months.  The guest VMs are very stable, and PCIe
> > PT is working as was designed (all of the PCIe devices are on the root
> > PCIe device).  I have successfully passed through GPUs, NICs, etc. I was
> > asked by those in the community to attempt to once again upstream the
> > patches.  I have them working with Seabios and OVMF (patches are needed
> > to OVMF which I will be sending to the mailing list).  I will be sending
> > my Qemu patches to their mailing list in hopes of getting everything
> > upstreamed. The Qemu patches allow for the xenvbd to properly unplug the
> > AHCI SATA device, and all xen pv windows drivers work as intended.
> >
> > I used the original author of the patches to get a majority of this to
> work:
> > Alexey Gerasimenko.  I fixed the patches to be in line with the upstream
> > Qemu and Xen versions.  Any original issues may still exist; however, I
> > am sure in time they can be improved. If the code doesn't exist then
> they
> > can't be actively looked at by the community.
> >
> > I am not an expert on the Q35 chipset or PCIe technology.  This is my
> > first patch to this mailing list.
> >
> > Recap of changes to Qemu, only here for reference:
> >   pc/xen: Xen Q35 support: provide IRQ handling for PCI devices
> >   pc/q35: Apply PCI bus BSEL property for Xen PCI device hotplug
> >   acpi/ich9: Multiple fixes for S3 support
> >   acpi/pcihp: Allow for machines to hotplug when using Xen
> >   isa/lpc_ich9: Write lpc configuration for xen IRQs
> >   q35/acpi/xen: Provide ACPI PCI hotplug interface for Xen on Q35
> >   q35/xen: Add Xen platform device support for Q35
> >   q35: Fix incorrect values for PCIEXBAR masks
> >   xen/pt: XenHostPCIDevice: provide functions for PCI Capabilities and
> >     PCIe Extended Capabilities enumeration
> >   xen/pt: avoid reading PCIe device type and cap version multiple times
> >   xen/pt: determine the legacy/PCIe mode for a passed through device
> >   xen/pt: Xen PCIe passthrough support for Q35: bypass PCIe topology
> >     check
> >   xen/pt: add support for PCIe Extended Capabilities and larger config
> >     space
> >   xen/pt: handle PCIe Extended Capabilities Next register
> >   xen/pt: allow to hide PCIe Extended Capabilities
> >   xen/pt: add Vendor-specific PCIe Extended Capability descriptor and
> >     sizing
> >   xen/pt: add fixed-size PCIe Extended Capabilities descriptors
> >   xen/pt: add AER PCIe Extended Capability descriptor and sizing
> >   xen/pt: add descriptors and size calculation for
> >     RCLD/ACS/PMUX/DPA/MCAST/TPH/DPC PCIe Extended Capabilities
> >   xen/pt: add Resizable BAR PCIe Extended Capability descriptor and
> >     sizing
> >   xen/pt: add VC/VC9/MFVC PCIe Extended Capabilities descriptors and
> >     sizing
> >
> > Changes that would be going to upstream Qemu:
> >  hw/acpi/ich9.c                |   22 +-
> >  hw/acpi/pcihp.c               |    6 +-
> >  hw/core/machine.c             |   19 +
> >  hw/i386/pc_piix.c             |    3 +-
> >  hw/i386/pc_q35.c              |   39 +-
> >  hw/i386/xen/xen-hvm.c         |    7 +-
> >  hw/i386/xen/xen_platform.c    |   19 +-
> >  hw/ide/ahci.c                 |    2 +-
> >  hw/isa/lpc_ich9.c             |   53 +-
> >  hw/isa/piix3.c                |    2 +-
> >  hw/pci-host/q35.c             |   28 +-
> >  hw/pci/pci.c                  |   17 +
> >  hw/xen/xen-host-pci-device.c  |  106 +++-
> >  hw/xen/xen-host-pci-device.h  |    6 +-
> >  hw/xen/xen_pt.c               |   49 +-
> >  hw/xen/xen_pt.h               |   19 +-
> >  hw/xen/xen_pt_config_init.c   | 1103 ++++++++++++++++++++++++++++++---
> >  include/hw/acpi/ich9.h        |    1 +
> >  include/hw/acpi/pcihp.h       |    2 +
> >  include/hw/boards.h           |    1 +
> >  include/hw/i386/pc.h          |    3 +
> >  include/hw/pci-host/q35.h     |    4 +-
> >  include/hw/pci/pci.h          |    3 +
> >  include/hw/southbridge/ich9.h |    1 +
> >  include/hw/xen/xen.h          |    4 +-
> >  qemu-options.hx               |    1 +
> >  softmmu/datadir.c             |    1 -
> >  softmmu/qdev-monitor.c        |    3 +-
> >  stubs/xen-hw-stub.c           |    4 +-
> >  29 files changed, 1396 insertions(+), 132 deletions(-)
> > *** These patches are not found in this series, if they are supposed to
> > be I will send them here as well. ***
>
> It's unclear, do the QEMU patches depend on the Xen side?  I have a
> suspicion you likely want to post them mostly at the same time, but
> using two different patch series (one for Xen and one for QEMU).
>
> > Recap of changes to Xen:
> >   libacpi: new DSDT ACPI table for Q35
> >   Makefile: build and use new DSDT table for Q35
> >   hvmloader: add function to query an emulated machine type (i440/Q35)
> >   hvmloader: add ACPI enabling for Q35
> >   hvmloader: add Q35 DSDT table loading
> >   hvmloader: add basic Q35 support
> >   hvmloader: allocate MMCONFIG area in the MMIO hole + minor code
> >     refactoring
> >   hvmloader: seabios dsdt set to load correct config
> >   hvmloader: ovmf dsdt set to load correct config
> >   libxl: Q35 support (new option device_model_machine)
> >   libxl: Xen Platform device support for Q35
> >   libacpi: build ACPI MCFG table if requested
> >   hvmloader: use libacpi to build MCFG table
> >   docs: provide description for device_model_machine option
> >
> > Joel Upham (1):
> >   Q35 support
>
> I'm afraid this needs to be split into a patch series, with proper
> commit messages in order to be considered for review.  The current
> huge patch is impossible to review at all.
>
> Also you need to keep the Signed-off-by of the original author in the
> patches.
>
> Thanks, Roger.
>