Hello all,

This is the fourth revision of a patch set implementing an accelerator
for the MSHV kernel driver, exposing HyperV to Linux "Dom0" hosts in
various scenarios. Thanks for the feedback to the previous revision, I
tried to incorporate those. The changes in the currenct patchset beyond
the suggested fixes are mostly related to the replacement of retired
ioctl calls that will not part of the upstreamed MSHV kernel driver.

Best regards,

magnus

Changelog:

v3 => v4

- Addressed code review comments
- Extended mention of MSHV accelerator in documentation
- Reworked the QMP `query-mshv` command to work like other accelerator's
  implementations (availability of the accelerator, instead of issueing
  stat() on /dev/mshv)
- Added a HMP `info mshv` command
- Replaced ioctls `MSHV_GET_VP_REGISTERS`, `MSHV_SET_VP_REGISTERS`,
  and `MSHV_TRANSLATE_GVA` with calls to `MSHV_ROOT_HVCALL`
- Allocate static pages in cpu->accel for the HVCALL input/output params
  (to avoid allocations in the MMIO exit path)

v2 => v3

- Addressed code review comments (style)
- Reserve GSI 01-23 for IO-APIC pins (this resolved a problem in which MSI
  routes would be overwritten with interrupts from legacy devices, breaking
  irqfd notification for virtio-blk queues)
- Guard memory slot table with mutex and RCU mechanism (multiple threads
  might access the memory slot table, and in the event of an UNMAPPED_GPA
  exit we need to query the table for an unmapped region covering that GPA)
- Include memory slot manager in MshvState
- Produce mshv.h kernel header with ./scripts/update-linux-headers.sh from
  linux 6.16 (not all UAPI definitions are defined in the upstream kernel,
  hence we ship hw/hyper/hvgdk*.h and hw/hyperv/hvhdk*.h headers)
- Added a QMP command query-mshv (a requirement for integration into
  higher-level tooling)
- Removed handling of HALT vm exit, since this is not a supported HV
  message any more.
- Added 2 maintainers from Microsoft for the respective file hierarchy
- Added mshv as accelerator option in the documentation

RFC (v1) => v2

- Addressed code review comments (style, consilidation).
- Rewrote the logic that handles overlap-in-userspace mappings to use
  a static list of slots, inspired by the HVF accelerator code.
- Fixed a bug that wrote corrupt payload in a MSHV_SET_MSI_ROUTING
  call, preventing vhost=on to work on tap network devices.
- Removed an erronous truncation of guest addresses to 32bit when
  registering ioeventfd's using MSHV_IOEVENTFD. This resulted in
  shadowing of low memory when ioevents were registered with
  addresses beyond the 4gb barrier and thus unexpected "unmapped gpa"
  vm exits in lower mem regions (impacting io performance).
- Fixed problem in which the MSI routing table was committed for KVM
  KVM instead of MSHV in virtio-pci bus initialization.
- Added handler for HLT vm exits.
- The above fixes removed a few limitation present in the previous
  revision:
  - Guest with machine type "pc" are booting (testing is still mostly
    performed with q35)
  - Tap network devices can be used with vhost=on option.
  - Seabios can be used with >2.75G memory and multiple virtio-pci
    devices
  - I/O performance improvement as extranous MMIO vm exits are avoided
    by registering ioevents with a correct address.

Notes:

- A discrete kernel ioctl "set_immediate_exit" (to avoid a race condition
  when handling terminiation signals like ctrl-a x) has been tested and
  proven to mitigate the problem. Since other consumers of /dev/mshv have
  simular requirements as QEMU, we opted to iterate a bit more on the
  respective kernel interface.

Magnus Kulke (26):
  accel: Add Meson and config support for MSHV accelerator
  target/i386/emulate: Allow instruction decoding from stream
  target/i386/mshv: Add x86 decoder/emu implementation
  hw/intc: Generalize APIC helper names from kvm_* to accel_*
  include/hw/hyperv: Add MSHV ABI header definitions
  linux-headers/linux: Add mshv.h headers
  accel/mshv: Add accelerator skeleton
  accel/mshv: Register memory region listeners
  accel/mshv: Initialize VM partition
  accel/mshv: Add vCPU creation and execution loop
  accel/mshv: Add vCPU signal handling
  target/i386/mshv: Add CPU create and remove logic
  target/i386/mshv: Implement mshv_store_regs()
  target/i386/mshv: Implement mshv_get_standard_regs()
  target/i386/mshv: Implement mshv_get_special_regs()
  target/i386/mshv: Implement mshv_arch_put_registers()
  target/i386/mshv: Set local interrupt controller state
  target/i386/mshv: Register CPUID entries with MSHV
  target/i386/mshv: Register MSRs with MSHV
  target/i386/mshv: Integrate x86 instruction decoder/emulator
  target/i386/mshv: Write MSRs to the hypervisor
  target/i386/mshv: Implement mshv_vcpu_run()
  accel/mshv: Handle overlapping mem mappings
  target/i386/mshv: Use preallocated page for hvcall
  docs: Add mshv to documentation
  MAINTAINERS: Add maintainers for mshv accelerator

Praveen K Paladugu (1):
  qapi/accel: Allow to query mshv capabilities

 MAINTAINERS                      |   15 +
 accel/Kconfig                    |    3 +
 accel/accel-irq.c                |  106 ++
 accel/meson.build                |    3 +-
 accel/mshv/irq.c                 |  397 +++++++
 accel/mshv/mem.c                 |  562 ++++++++++
 accel/mshv/meson.build           |    9 +
 accel/mshv/mshv-all.c            |  728 ++++++++++++
 accel/mshv/msr.c                 |  374 +++++++
 accel/mshv/trace-events          |   33 +
 accel/mshv/trace.h               |   14 +
 docs/about/build-platforms.rst   |    2 +-
 docs/devel/codebase.rst          |    2 +-
 docs/glossary.rst                |    6 +-
 docs/system/introduction.rst     |    3 +
 hmp-commands-info.hx             |   13 +
 hw/core/machine-hmp-cmds.c       |   15 +
 hw/core/machine-qmp-cmds.c       |   14 +
 hw/intc/apic.c                   |    8 +
 hw/intc/ioapic.c                 |   20 +-
 hw/virtio/virtio-pci.c           |   21 +-
 include/hw/hyperv/hvgdk.h        |   20 +
 include/hw/hyperv/hvgdk_mini.h   |  817 ++++++++++++++
 include/hw/hyperv/hvhdk.h        |  249 +++++
 include/hw/hyperv/hvhdk_mini.h   |  102 ++
 include/monitor/hmp.h            |    1 +
 include/system/accel-irq.h       |   37 +
 include/system/hw_accel.h        |    1 +
 include/system/mshv.h            |  196 ++++
 linux-headers/linux/mshv.h       |  291 +++++
 meson.build                      |   11 +
 meson_options.txt                |    2 +
 qapi/accelerator.json            |   29 +
 qemu-options.hx                  |   16 +-
 scripts/meson-buildoptions.sh    |    3 +
 scripts/update-linux-headers.sh  |    2 +-
 target/i386/cpu.h                |    4 +-
 target/i386/emulate/meson.build  |    7 +-
 target/i386/emulate/x86_decode.c |   27 +-
 target/i386/emulate/x86_decode.h |    9 +
 target/i386/emulate/x86_emu.c    |    3 +-
 target/i386/emulate/x86_emu.h    |    2 +
 target/i386/meson.build          |    2 +
 target/i386/mshv/meson.build     |    8 +
 target/i386/mshv/mshv-cpu.c      | 1764 ++++++++++++++++++++++++++++++
 target/i386/mshv/x86.c           |  297 +++++
 46 files changed, 6208 insertions(+), 40 deletions(-)
 create mode 100644 accel/accel-irq.c
 create mode 100644 accel/mshv/irq.c
 create mode 100644 accel/mshv/mem.c
 create mode 100644 accel/mshv/meson.build
 create mode 100644 accel/mshv/mshv-all.c
 create mode 100644 accel/mshv/msr.c
 create mode 100644 accel/mshv/trace-events
 create mode 100644 accel/mshv/trace.h
 create mode 100644 include/hw/hyperv/hvgdk.h
 create mode 100644 include/hw/hyperv/hvgdk_mini.h
 create mode 100644 include/hw/hyperv/hvhdk.h
 create mode 100644 include/hw/hyperv/hvhdk_mini.h
 create mode 100644 include/system/accel-irq.h
 create mode 100644 include/system/mshv.h
 create mode 100644 linux-headers/linux/mshv.h
 create mode 100644 target/i386/mshv/meson.build
 create mode 100644 target/i386/mshv/mshv-cpu.c
 create mode 100644 target/i386/mshv/x86.c

-- 
2.34.1

Hi Magnus,

On 16/9/25 18:48, Magnus Kulke wrote:
> Hello all,
> 
> This is the fourth revision of a patch set implementing an accelerator
> for the MSHV kernel driver, exposing HyperV to Linux "Dom0" hosts in
> various scenarios. Thanks for the feedback to the previous revision, I
> tried to incorporate those. The changes in the currenct patchset beyond
> the suggested fixes are mostly related to the replacement of retired
> ioctl calls that will not part of the upstreamed MSHV kernel driver.


> Magnus Kulke (26):
>    accel: Add Meson and config support for MSHV accelerator

>    accel/mshv: Add accelerator skeleton
>    accel/mshv: Register memory region listeners
>    accel/mshv: Initialize VM partition
>    accel/mshv: Add vCPU creation and execution loop
>    accel/mshv: Add vCPU signal handling

>    accel/mshv: Handle overlapping mem mappings
Due to my generic work on accelerators, I'll have to refactor these
patches. Obviously I don't want to break your implementation! Can
you add some (functional?) tests? Ideally we should be running
tests on our CI to ensure code doesn't bitrot.

Regards,

Phil.

On Thu, Oct 02, 2025 at 10:30:56AM +0200, Philippe Mathieu-Daudé wrote:
> Due to my generic work on accelerators, I'll have to refactor these
> patches. Obviously I don't want to break your implementation! Can
> you add some (functional?) tests? Ideally we should be running
> tests on our CI to ensure code doesn't bitrot.
> 
> Regards,
> 
> Phil.

Hey Phil,

yes, that's a good point. I assume for functional tests we
will face the challenge those will require external infra, because
eventually there needs to be a HyperV Hypervisor running somewhere.

Is there any precedent/prior art in QEMU (e.g. for HVF or WHPX) that we
could follow?

FWIW, in smoke tests for the patch series I've been using a nested
HyperV hypervisor that is shipped as dll and made available as UEFI
protocol to the MSHV driver (I _think_ that is how it works, I'm a
bit out of my depth here since I don't know how those things are
glued together in detail).

For reference: https://github.com/mkulke/qemu/actions/runs/18187719634

There will be other virt topologies supported by the MSHV driver and
the nested option is not upstreamed to mainline yet. However, from
QEMU's perspective those topologies do not matter, they share the same
ABI of the kernel driver.

So we could do something similar, provide/maintain a VM on Azure with a
similar nested HyperV hypervisor configuration that we can used for
testing. Would this make sense?

Best,

magnus

> On 2. Oct 2025, at 11:10, Magnus Kulke <magnuskulke@linux.microsoft.com> wrote:
> 
> On Thu, Oct 02, 2025 at 10:30:56AM +0200, Philippe Mathieu-Daudé wrote:
>> Due to my generic work on accelerators, I'll have to refactor these
>> patches. Obviously I don't want to break your implementation! Can
>> you add some (functional?) tests? Ideally we should be running
>> tests on our CI to ensure code doesn't bitrot.
>> 
>> Regards,
>> 
>> Phil.
> 
> Hey Phil,
> 
> yes, that's a good point. I assume for functional tests we
> will face the challenge those will require external infra, because
> eventually there needs to be a HyperV Hypervisor running somewhere.
Worth noting that x86_64 MSHV on Linux is available to the public as part of Azure Linux,
and runs inside of a Qemu VM just fine. (with the catch that MBEC isn’t currently emulated by KVM)

Perhaps having an Azure Linux instance in CI could be the right thing to do?

Thank you,

> Is there any precedent/prior art in QEMU (e.g. for HVF or WHPX) that we
> could follow?
> 
> FWIW, in smoke tests for the patch series I've been using a nested
> HyperV hypervisor that is shipped as dll and made available as UEFI
> protocol to the MSHV driver (I _think_ that is how it works, I'm a
> bit out of my depth here since I don't know how those things are
> glued together in detail).
> 
> For reference: https://github.com/mkulke/qemu/actions/runs/18187719634
> 
> There will be other virt topologies supported by the MSHV driver and
> the nested option is not upstreamed to mainline yet. However, from
> QEMU's perspective those topologies do not matter, they share the same
> ABI of the kernel driver.
> 
> So we could do something similar, provide/maintain a VM on Azure with a
> similar nested HyperV hypervisor configuration that we can used for
> testing. Would this make sense?
> 
> Best,
> 
> magnus
> 

On Thu, Oct 02, 2025 at 04:07:39PM +0200, Mohamed Mediouni wrote:
> Worth noting that x86_64 MSHV on Linux is available to the public as part of Azure Linux,
> and runs inside of a Qemu VM just fine. (with the catch that MBEC isn’t currently emulated by KVM)
> 
> Perhaps having an Azure Linux instance in CI could be the right thing to do?
> 

Yes, that's right, MSHV is shipped as part of Azure Linux images, albeit
the mshv kernel driver in the AL kernel tree has more features that
are being upstreamed to mainline incrementally over time.

If I understand correctly, your suggestion is to run HyperV as a nested
hypervisor inside a QEMU VM (with KVM accel) on the CI machine that runs
on on baremetal/L0? (mshv-on-kvm)

We haven't tested this scenario yet, AFAIK the images that ship the
HyperV bits are meant to run as part of proprietary Azure products, so
the CI setup wouldn't really be representative of a real-world user
workloads.

We'll investigate if this is a feasible option, but I suspect that it'll
be more straight forward to provide CI runners as L1 compute on Azure
cloud (or run tests downstream somewhere if this is not trivial to
integrate).

best,

magnus

> On 2. Oct 2025, at 17:36, Magnus Kulke <magnuskulke@linux.microsoft.com> wrote:
> 
> If I understand correctly, your suggestion is to run HyperV as a nested
> hypervisor inside a QEMU VM (with KVM accel) on the CI machine that runs
> on on baremetal/L0? (mshv-on-kvm)

Yes. I’ve been playing around with that configuration using cloud-hypervisor
on an AMD Zen 4 host and it works fine.

Thanks,

On Thu, Oct 02, 2025 at 10:30:56AM +0200, Philippe Mathieu-Daudé wrote:
> Hi Magnus,
> 
> On 16/9/25 18:48, Magnus Kulke wrote:
> > Hello all,
> > 
> > This is the fourth revision of a patch set implementing an accelerator
> > for the MSHV kernel driver, exposing HyperV to Linux "Dom0" hosts in
> > various scenarios. Thanks for the feedback to the previous revision, I
> > tried to incorporate those. The changes in the currenct patchset beyond
> > the suggested fixes are mostly related to the replacement of retired
> > ioctl calls that will not part of the upstreamed MSHV kernel driver.
> 
> 
> > Magnus Kulke (26):
> >    accel: Add Meson and config support for MSHV accelerator
> 
> >    accel/mshv: Add accelerator skeleton
> >    accel/mshv: Register memory region listeners
> >    accel/mshv: Initialize VM partition
> >    accel/mshv: Add vCPU creation and execution loop
> >    accel/mshv: Add vCPU signal handling
> >    accel/mshv: Handle overlapping mem mappings
>
> Due to my generic work on accelerators, I'll have to refactor these
> patches. Obviously I don't want to break your implementation! Can
> you add some (functional?) tests? Ideally we should be running
> tests on our CI to ensure code doesn't bitrot.

NB our CI systems don't have HyperV available, so it is unrealistic
for us to test it in our CI upstream.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On 2/10/25 10:41, Daniel P. Berrangé wrote:
> On Thu, Oct 02, 2025 at 10:30:56AM +0200, Philippe Mathieu-Daudé wrote:
>> Hi Magnus,
>>
>> On 16/9/25 18:48, Magnus Kulke wrote:
>>> Hello all,
>>>
>>> This is the fourth revision of a patch set implementing an accelerator
>>> for the MSHV kernel driver, exposing HyperV to Linux "Dom0" hosts in
>>> various scenarios. Thanks for the feedback to the previous revision, I
>>> tried to incorporate those. The changes in the currenct patchset beyond
>>> the suggested fixes are mostly related to the replacement of retired
>>> ioctl calls that will not part of the upstreamed MSHV kernel driver.
>>
>>
>>> Magnus Kulke (26):
>>>     accel: Add Meson and config support for MSHV accelerator
>>
>>>     accel/mshv: Add accelerator skeleton
>>>     accel/mshv: Register memory region listeners
>>>     accel/mshv: Initialize VM partition
>>>     accel/mshv: Add vCPU creation and execution loop
>>>     accel/mshv: Add vCPU signal handling
>>>     accel/mshv: Handle overlapping mem mappings
>>
>> Due to my generic work on accelerators, I'll have to refactor these
>> patches. Obviously I don't want to break your implementation! Can
>> you add some (functional?) tests? Ideally we should be running
>> tests on our CI to ensure code doesn't bitrot.
> 
> NB our CI systems don't have HyperV available, so it is unrealistic
> for us to test it in our CI upstream.

OK. Hopefully meson will autodetect it on x86_64 Linux hosts so at least
we'll compile & link.
That said, IIUC our Xen-only job should now use --disable-mshv:

.gitlab-ci.d/crossbuilds.yml:181:cross-amd64-xen-only:
.gitlab-ci.d/crossbuilds.yml-182-  extends: .cross_accel_build_job
.gitlab-ci.d/crossbuilds.yml-183-  needs:
.gitlab-ci.d/crossbuilds.yml-184-    - job: amd64-debian-cross-container
.gitlab-ci.d/crossbuilds.yml-185-  variables:
.gitlab-ci.d/crossbuilds.yml-186-    IMAGE: debian-amd64-cross
.gitlab-ci.d/crossbuilds.yml-187-    ACCEL: xen
.gitlab-ci.d/crossbuilds.yml-188-    EXTRA_CONFIGURE_OPTS: --disable-tcg 
--disable-kvm
.gitlab-ci.d/crossbuilds.yml-189-


Tests are optionally run on CI, having a test we can run locally is
still valuable.

On Tue, Sep 16, 2025 at 06:48:20PM +0200, Magnus Kulke wrote:
> Hello all,
> 
> This is the fourth revision of a patch set implementing an accelerator
> for the MSHV kernel driver, exposing HyperV to Linux "Dom0" hosts in
> various scenarios. Thanks for the feedback to the previous revision, I
> tried to incorporate those. The changes in the currenct patchset beyond
> the suggested fixes are mostly related to the replacement of retired
> ioctl calls that will not part of the upstreamed MSHV kernel driver.

Externally we follow Microsoft's convention of calling Linux the "root
partition" in this setup. "Dom0" is an internal term. We drew parallels
to Xen at the beginning of the project.

> Magnus Kulke (26):
>   accel: Add Meson and config support for MSHV accelerator
>   target/i386/emulate: Allow instruction decoding from stream
>   target/i386/mshv: Add x86 decoder/emu implementation
>   hw/intc: Generalize APIC helper names from kvm_* to accel_*
>   include/hw/hyperv: Add MSHV ABI header definitions
>   linux-headers/linux: Add mshv.h headers
>   accel/mshv: Add accelerator skeleton
>   accel/mshv: Register memory region listeners
>   accel/mshv: Initialize VM partition
>   accel/mshv: Add vCPU creation and execution loop
>   accel/mshv: Add vCPU signal handling
>   target/i386/mshv: Add CPU create and remove logic
>   target/i386/mshv: Implement mshv_store_regs()
>   target/i386/mshv: Implement mshv_get_standard_regs()
>   target/i386/mshv: Implement mshv_get_special_regs()
>   target/i386/mshv: Implement mshv_arch_put_registers()
>   target/i386/mshv: Set local interrupt controller state
>   target/i386/mshv: Register CPUID entries with MSHV
>   target/i386/mshv: Register MSRs with MSHV
>   target/i386/mshv: Integrate x86 instruction decoder/emulator
>   target/i386/mshv: Write MSRs to the hypervisor
>   target/i386/mshv: Implement mshv_vcpu_run()
>   accel/mshv: Handle overlapping mem mappings
>   target/i386/mshv: Use preallocated page for hvcall
>   docs: Add mshv to documentation
>   MAINTAINERS: Add maintainers for mshv accelerator
> 
> Praveen K Paladugu (1):
>   qapi/accel: Allow to query mshv capabilities
> 

I've gone through this patch series and it generally looks good to me.

   Acked-by: Wei Liu <wei.liu@kenrel.org>

It would be great if we can merge this soon so we can do more testing
and implement other missing features.

Wei