From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

Hello all.

The purpose of this patch series is to add support for restricting memory access under Xen using specific
grant table [1] based DMA-mapping layer. Patch series is based on Juergen Gross’ initial work [2] which implies
using grant references instead of raw guest physical addresses (GPA) for the virtio communications (some
kind of the software IOMMU).

You can find RFC-V3 patch series (and previous discussions) at [3].

!!! Please note, the only diff between V3 and V4 is in commit #5, also I have collected the acks (commits ##4-7).

The high level idea is to create new Xen’s grant table based DMA-mapping layer for the guest Linux whose main
purpose is to provide a special 64-bit DMA address which is formed by using the grant reference (for a page
to be shared with the backend) with offset and setting the highest address bit (this is for the backend to
be able to distinguish grant ref based DMA address from normal GPA). For this to work we need the ability
to allocate contiguous (consecutive) grant references for multi-page allocations. And the backend then needs
to offer VIRTIO_F_ACCESS_PLATFORM and VIRTIO_F_VERSION_1 feature bits (it must support virtio-mmio modern
transport for 64-bit addresses in the virtqueue).

Xen's grant mapping mechanism is the secure and safe solution to share pages between domains which proven
to work and works for years (in the context of traditional Xen PV drivers for example). So far, the foreign
mapping is used for the virtio backend to map and access guest memory. With the foreign mapping, the backend
is able to map arbitrary pages from the guest memory (or even from Dom0 memory). And as the result, the malicious
backend which runs in a non-trusted domain can take advantage of this. Instead, with the grant mapping
the backend is only allowed to map pages which were explicitly granted by the guest before and nothing else.
According to the discussions in various mainline threads this solution would likely be welcome because it
perfectly fits in the security model Xen provides.

What is more, the grant table based solution requires zero changes to the Xen hypervisor itself at least
with virtio-mmio and DT (in comparison, for example, with "foreign mapping + virtio-iommu" solution which would
require the whole new complex emulator in hypervisor in addition to new functionality/hypercall to pass IOVA
from the virtio backend running elsewhere to the hypervisor and translate it to the GPA before mapping into
P2M or denying the foreign mapping request if no corresponding IOVA-GPA mapping present in the IOMMU page table
for that particular device). We only need to update toolstack to insert "xen,grant-dma" IOMMU node (to be referred
by the virtio-mmio device using "iommus" property) when creating a guest device-tree (this is an indicator for
the guest to use Xen grant mappings scheme for that device with the endpoint ID being used as ID of Xen domain
where the corresponding backend is running, the backend domid is used as an argument to the grant mapping APIs).
It worth mentioning that toolstack patch is based on non upstreamed yet “Virtio support for toolstack on Arm”
series which is on review now [4].

Please note the following:
- Patch series only covers Arm and virtio-mmio (device-tree) for now. To enable the restricted memory access
  feature on Arm the following option should be set:
  CONFIG_XEN_VIRTIO=y
- Patch series is based on "kernel: add new infrastructure for platform_has() support" patch series which
  is on review now [5]
- Xen should be built with the following options:
  CONFIG_IOREQ_SERVER=y
  CONFIG_EXPERT=y

Patch series is rebased on "for-linus-5.19" branch [1] with "platform_has()" series applied and tested on Renesas
Salvator-X board + H3 ES3.0 SoC (Arm64) with standalone userspace (non-Qemu) virtio-mmio based virtio-disk backend
running in Driver domain and Linux guest running on existing virtio-blk driver (frontend). No issues were observed.
Guest domain 'reboot/destroy' use-cases work properly.
I have also tested other use-cases such as assigning several virtio block devices or a mix of virtio and Xen PV block
devices to the guest. Patch series was build-tested on Arm32 and x86.

1. Xen changes located at (last patch):
https://github.com/otyshchenko1/xen/commits/libxl_virtio_next2_1
2. Linux changes located at (last 8 patches):
https://github.com/otyshchenko1/linux/commits/virtio_grant9
3. virtio-disk changes located at:
https://github.com/otyshchenko1/virtio-disk/commits/virtio_grant

Any feedback/help would be highly appreciated.

[1] https://xenbits.xenproject.org/docs/4.16-testing/misc/grant-tables.txt
[2] https://www.youtube.com/watch?v=IrlEdaIUDPk
[3] https://lore.kernel.org/xen-devel/1649963973-22879-1-git-send-email-olekstysh@gmail.com/
    https://lore.kernel.org/xen-devel/1650646263-22047-1-git-send-email-olekstysh@gmail.com/
    https://lore.kernel.org/xen-devel/1651947548-4055-1-git-send-email-olekstysh@gmail.com/
    https://lore.kernel.org/xen-devel/1653944417-17168-1-git-send-email-olekstysh@gmail.com/
[4] https://lore.kernel.org/xen-devel/1654106261-28044-1-git-send-email-olekstysh@gmail.com/
    https://lore.kernel.org/xen-devel/1653944813-17970-1-git-send-email-olekstysh@gmail.com/
[5] https://lore.kernel.org/xen-devel/20220504155703.13336-1-jgross@suse.com/
[6] https://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git/log/?h=for-linus-5.19

Juergen Gross (3):
  xen/grants: support allocating consecutive grants
  xen/grant-dma-ops: Add option to restrict memory access under Xen
  xen/virtio: Enable restricted memory access using Xen grant mappings

Oleksandr Tyshchenko (5):
  arm/xen: Introduce xen_setup_dma_ops()
  dt-bindings: Add xen,grant-dma IOMMU description for xen-grant DMA ops
  xen/grant-dma-iommu: Introduce stub IOMMU driver
  xen/grant-dma-ops: Retrieve the ID of backend's domain for DT devices
  arm/xen: Assign xen-grant DMA ops for xen-grant DMA devices

 .../devicetree/bindings/iommu/xen,grant-dma.yaml   |  39 +++
 arch/arm/include/asm/xen/xen-ops.h                 |   2 +
 arch/arm/mm/dma-mapping.c                          |   7 +-
 arch/arm/xen/enlighten.c                           |   2 +
 arch/arm64/include/asm/xen/xen-ops.h               |   2 +
 arch/arm64/mm/dma-mapping.c                        |   7 +-
 arch/x86/xen/enlighten_hvm.c                       |   2 +
 arch/x86/xen/enlighten_pv.c                        |   2 +
 drivers/xen/Kconfig                                |  20 ++
 drivers/xen/Makefile                               |   2 +
 drivers/xen/grant-dma-iommu.c                      |  78 +++++
 drivers/xen/grant-dma-ops.c                        | 345 +++++++++++++++++++++
 drivers/xen/grant-table.c                          | 251 ++++++++++++---
 include/xen/arm/xen-ops.h                          |  18 ++
 include/xen/grant_table.h                          |   4 +
 include/xen/xen-ops.h                              |  13 +
 include/xen/xen.h                                  |   8 +
 17 files changed, 756 insertions(+), 46 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/iommu/xen,grant-dma.yaml
 create mode 100644 arch/arm/include/asm/xen/xen-ops.h
 create mode 100644 arch/arm64/include/asm/xen/xen-ops.h
 create mode 100644 drivers/xen/grant-dma-iommu.c
 create mode 100644 drivers/xen/grant-dma-ops.c
 create mode 100644 include/xen/arm/xen-ops.h

-- 
2.7.4

Hi Oleksandr,

On Mon, Jun 6, 2022 at 10:16 AM Oleksandr Tyshchenko
<olekstysh@gmail.com> wrote:
> The high level idea is to create new Xen’s grant table based DMA-mapping layer for the guest Linux whose main
> purpose is to provide a special 64-bit DMA address which is formed by using the grant reference (for a page
> to be shared with the backend) with offset and setting the highest address bit (this is for the backend to
> be able to distinguish grant ref based DMA address from normal GPA). For this to work we need the ability
> to allocate contiguous (consecutive) grant references for multi-page allocations. And the backend then needs
> to offer VIRTIO_F_ACCESS_PLATFORM and VIRTIO_F_VERSION_1 feature bits (it must support virtio-mmio modern
> transport for 64-bit addresses in the virtqueue).

I was trying your series, from Linus's tree now and started seeing
boot failures,
failed to mount rootfs. And the reason probably is these messages:

[ 1.222498] virtio_scsi virtio1: device must provide VIRTIO_F_ACCESS_PLATFORM
[ 1.316334] virtio_net virtio0: device must provide VIRTIO_F_ACCESS_PLATFORM

I understand from your email that the backends need to offer
VIRTIO_F_ACCESS_PLATFORM flag now, but should this requirement be a
bit soft ? I mean shouldn't we allow both types of backends to run with the same
kernel, ones that offer this feature and others that don't ? The ones that don't
offer the feature, should continue to work like they used to, i.e.
without the restricted
memory access feature.

I am testing Xen currently with help of Qemu  over my x86 desktop and
these backends
(scsi and net) are part of QEMU itself I think, and I don't really
want to go and make the
change there.

Thanks.

--
Viresh

On 15.06.22 09:23, Viresh Kumar wrote:
> Hi Oleksandr,


Hello Viresh


>
> On Mon, Jun 6, 2022 at 10:16 AM Oleksandr Tyshchenko
> <olekstysh@gmail.com> wrote:
>> The high level idea is to create new Xen’s grant table based DMA-mapping layer for the guest Linux whose main
>> purpose is to provide a special 64-bit DMA address which is formed by using the grant reference (for a page
>> to be shared with the backend) with offset and setting the highest address bit (this is for the backend to
>> be able to distinguish grant ref based DMA address from normal GPA). For this to work we need the ability
>> to allocate contiguous (consecutive) grant references for multi-page allocations. And the backend then needs
>> to offer VIRTIO_F_ACCESS_PLATFORM and VIRTIO_F_VERSION_1 feature bits (it must support virtio-mmio modern
>> transport for 64-bit addresses in the virtqueue).
> I was trying your series, from Linus's tree now and started seeing
> boot failures,
> failed to mount rootfs. And the reason probably is these messages:
>
> [ 1.222498] virtio_scsi virtio1: device must provide VIRTIO_F_ACCESS_PLATFORM
> [ 1.316334] virtio_net virtio0: device must provide VIRTIO_F_ACCESS_PLATFORM
>
> I understand from your email that the backends need to offer
> VIRTIO_F_ACCESS_PLATFORM flag now, but should this requirement be a
> bit soft ? I mean shouldn't we allow both types of backends to run with the same
> kernel, ones that offer this feature and others that don't ? The ones that don't
> offer the feature, should continue to work like they used to, i.e.
> without the restricted
> memory access feature.
> I am testing Xen currently with help of Qemu  over my x86 desktop and
> these backends
> (scsi and net) are part of QEMU itself I think, and I don't really
> want to go and make the
> change there.


Thank you for testing on x86.


I assume your guest type in HVM. Within current series the 
PLATFORM_VIRTIO_RESTRICTED_MEM_ACCESS is set for *all* type of Xen 
guests if CONFIG_XEN_VIRTIO is enabled.

I have to admit that from the very beginning it could be possible to 
configure for PV and HVM guests separately [1] because the usage of 
grant mappings for virtio is mandatory for paravirtualized guest, but 
not strictly necessary for the fully virtualized guests (if the backends 
are in Dom0). But it was decided to drop these extra options (including 
XEN_HVM_VIRTIO_GRANT) and leave only single one CONFIG_XEN_VIRTIO.

I see that Juergen has already pushed a fix.

Sorry for the inconvenience.



[1] 
https://lore.kernel.org/xen-devel/1649963973-22879-3-git-send-email-olekstysh@gmail.com/


>
> Thanks.
>
> --
> Viresh

-- 
Regards,

Oleksandr Tyshchenko

On 02.06.22 21:23, Oleksandr Tyshchenko wrote:
> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
> 
> Hello all.
> 
> The purpose of this patch series is to add support for restricting memory access under Xen using specific
> grant table [1] based DMA-mapping layer. Patch series is based on Juergen Gross’ initial work [2] which implies
> using grant references instead of raw guest physical addresses (GPA) for the virtio communications (some
> kind of the software IOMMU).
> 
> You can find RFC-V3 patch series (and previous discussions) at [3].
> 
> !!! Please note, the only diff between V3 and V4 is in commit #5, also I have collected the acks (commits ##4-7).
> 
> The high level idea is to create new Xen’s grant table based DMA-mapping layer for the guest Linux whose main
> purpose is to provide a special 64-bit DMA address which is formed by using the grant reference (for a page
> to be shared with the backend) with offset and setting the highest address bit (this is for the backend to
> be able to distinguish grant ref based DMA address from normal GPA). For this to work we need the ability
> to allocate contiguous (consecutive) grant references for multi-page allocations. And the backend then needs
> to offer VIRTIO_F_ACCESS_PLATFORM and VIRTIO_F_VERSION_1 feature bits (it must support virtio-mmio modern
> transport for 64-bit addresses in the virtqueue).
> 
> Xen's grant mapping mechanism is the secure and safe solution to share pages between domains which proven
> to work and works for years (in the context of traditional Xen PV drivers for example). So far, the foreign
> mapping is used for the virtio backend to map and access guest memory. With the foreign mapping, the backend
> is able to map arbitrary pages from the guest memory (or even from Dom0 memory). And as the result, the malicious
> backend which runs in a non-trusted domain can take advantage of this. Instead, with the grant mapping
> the backend is only allowed to map pages which were explicitly granted by the guest before and nothing else.
> According to the discussions in various mainline threads this solution would likely be welcome because it
> perfectly fits in the security model Xen provides.
> 
> What is more, the grant table based solution requires zero changes to the Xen hypervisor itself at least
> with virtio-mmio and DT (in comparison, for example, with "foreign mapping + virtio-iommu" solution which would
> require the whole new complex emulator in hypervisor in addition to new functionality/hypercall to pass IOVA
> from the virtio backend running elsewhere to the hypervisor and translate it to the GPA before mapping into
> P2M or denying the foreign mapping request if no corresponding IOVA-GPA mapping present in the IOMMU page table
> for that particular device). We only need to update toolstack to insert "xen,grant-dma" IOMMU node (to be referred
> by the virtio-mmio device using "iommus" property) when creating a guest device-tree (this is an indicator for
> the guest to use Xen grant mappings scheme for that device with the endpoint ID being used as ID of Xen domain
> where the corresponding backend is running, the backend domid is used as an argument to the grant mapping APIs).
> It worth mentioning that toolstack patch is based on non upstreamed yet “Virtio support for toolstack on Arm”
> series which is on review now [4].
> 
> Please note the following:
> - Patch series only covers Arm and virtio-mmio (device-tree) for now. To enable the restricted memory access
>    feature on Arm the following option should be set:
>    CONFIG_XEN_VIRTIO=y
> - Patch series is based on "kernel: add new infrastructure for platform_has() support" patch series which
>    is on review now [5]
> - Xen should be built with the following options:
>    CONFIG_IOREQ_SERVER=y
>    CONFIG_EXPERT=y
> 
> Patch series is rebased on "for-linus-5.19" branch [1] with "platform_has()" series applied and tested on Renesas
> Salvator-X board + H3 ES3.0 SoC (Arm64) with standalone userspace (non-Qemu) virtio-mmio based virtio-disk backend
> running in Driver domain and Linux guest running on existing virtio-blk driver (frontend). No issues were observed.
> Guest domain 'reboot/destroy' use-cases work properly.
> I have also tested other use-cases such as assigning several virtio block devices or a mix of virtio and Xen PV block
> devices to the guest. Patch series was build-tested on Arm32 and x86.
> 
> 1. Xen changes located at (last patch):
> https://github.com/otyshchenko1/xen/commits/libxl_virtio_next2_1
> 2. Linux changes located at (last 8 patches):
> https://github.com/otyshchenko1/linux/commits/virtio_grant9
> 3. virtio-disk changes located at:
> https://github.com/otyshchenko1/virtio-disk/commits/virtio_grant
> 
> Any feedback/help would be highly appreciated.
> 
> [1] https://xenbits.xenproject.org/docs/4.16-testing/misc/grant-tables.txt
> [2] https://www.youtube.com/watch?v=IrlEdaIUDPk
> [3] https://lore.kernel.org/xen-devel/1649963973-22879-1-git-send-email-olekstysh@gmail.com/
>      https://lore.kernel.org/xen-devel/1650646263-22047-1-git-send-email-olekstysh@gmail.com/
>      https://lore.kernel.org/xen-devel/1651947548-4055-1-git-send-email-olekstysh@gmail.com/
>      https://lore.kernel.org/xen-devel/1653944417-17168-1-git-send-email-olekstysh@gmail.com/
> [4] https://lore.kernel.org/xen-devel/1654106261-28044-1-git-send-email-olekstysh@gmail.com/
>      https://lore.kernel.org/xen-devel/1653944813-17970-1-git-send-email-olekstysh@gmail.com/
> [5] https://lore.kernel.org/xen-devel/20220504155703.13336-1-jgross@suse.com/
> [6] https://git.kernel.org/pub/scm/linux/kernel/git/xen/tip.git/log/?h=for-linus-5.19
> 
> Juergen Gross (3):
>    xen/grants: support allocating consecutive grants
>    xen/grant-dma-ops: Add option to restrict memory access under Xen
>    xen/virtio: Enable restricted memory access using Xen grant mappings
> 
> Oleksandr Tyshchenko (5):
>    arm/xen: Introduce xen_setup_dma_ops()
>    dt-bindings: Add xen,grant-dma IOMMU description for xen-grant DMA ops
>    xen/grant-dma-iommu: Introduce stub IOMMU driver
>    xen/grant-dma-ops: Retrieve the ID of backend's domain for DT devices
>    arm/xen: Assign xen-grant DMA ops for xen-grant DMA devices
> 
>   .../devicetree/bindings/iommu/xen,grant-dma.yaml   |  39 +++
>   arch/arm/include/asm/xen/xen-ops.h                 |   2 +
>   arch/arm/mm/dma-mapping.c                          |   7 +-
>   arch/arm/xen/enlighten.c                           |   2 +
>   arch/arm64/include/asm/xen/xen-ops.h               |   2 +
>   arch/arm64/mm/dma-mapping.c                        |   7 +-
>   arch/x86/xen/enlighten_hvm.c                       |   2 +
>   arch/x86/xen/enlighten_pv.c                        |   2 +
>   drivers/xen/Kconfig                                |  20 ++
>   drivers/xen/Makefile                               |   2 +
>   drivers/xen/grant-dma-iommu.c                      |  78 +++++
>   drivers/xen/grant-dma-ops.c                        | 345 +++++++++++++++++++++
>   drivers/xen/grant-table.c                          | 251 ++++++++++++---
>   include/xen/arm/xen-ops.h                          |  18 ++
>   include/xen/grant_table.h                          |   4 +
>   include/xen/xen-ops.h                              |  13 +
>   include/xen/xen.h                                  |   8 +
>   17 files changed, 756 insertions(+), 46 deletions(-)
>   create mode 100644 Documentation/devicetree/bindings/iommu/xen,grant-dma.yaml
>   create mode 100644 arch/arm/include/asm/xen/xen-ops.h
>   create mode 100644 arch/arm64/include/asm/xen/xen-ops.h
>   create mode 100644 drivers/xen/grant-dma-iommu.c
>   create mode 100644 drivers/xen/grant-dma-ops.c
>   create mode 100644 include/xen/arm/xen-ops.h
> 

Series pushed to xen/tip.git for-linus-5.19a


Juergen