From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>

Hello all.

The purpose of this RFC patch series is to add support for restricting memory access under Xen using specific
grant table based DMA ops layer. Patch series is based on Juergen Gross’ initial work [1] which implies using
grant references instead of raw guest physical addresses (GPA) for the virtio communications (some kind of
the software IOMMU).

The high level idea is to create new Xen’s grant table based DMA ops layer for the guest Linux whose main
purpose is to provide a special 64-bit DMA address which is formed by using the grant reference (for a page
to be shared with the backend) with offset and setting the highest address bit (this is for the backend to
be able to distinguish grant ref based DMA address from normal GPA). For this to work we need the ability
to allocate contiguous (consecutive) grant references for multi-page allocations. And the backend then needs
to offer VIRTIO_F_ACCESS_PLATFORM and VIRTIO_F_VERSION_1 feature bits (it must support virtio-mmio modern
transport for 64-bit addresses in the virtqueue).

Xen's grant mapping mechanism is the secure and safe solution to share pages between domains which proven
to work and works for years (in the context of traditional Xen PV drivers for example). So far, the foreign
mapping is used for the virtio backend to map and access guest memory. With the foreign mapping, the backend
is able to map arbitrary pages from the guest memory (or even from Dom0 memory). And as the result, the malicious
backend which runs in a non-trusted domain can take advantage of this. Instead, with the grant mapping
the backend is only allowed to map pages which were explicitly granted by the guest before and nothing else. 
According to the discussions in various mainline threads this solution would likely be welcome because it
perfectly fits in the security model Xen provides. 

What is more, the grant table based solution requires zero changes to the Xen hypervisor itself at least
with virtio-mmio and DT (in comparison, for example, with "foreign mapping + virtio-iommu" solution which would
require the whole new complex emulator in hypervisor in addition to new functionality/hypercall to pass IOVA
from the virtio backend running elsewhere to the hypervisor and translate it to the GPA before mapping into
P2M or denying the foreign mapping request if no corresponding IOVA-GPA mapping present in the IOMMU page table
for that particular device). We only need to update toolstack to insert a new "xen,dev-domid" property to
the virtio-mmio device node when creating a guest device-tree (this is an indicator for the guest to use grants
and the ID of Xen domain where the corresponding backend resides, it is used as an argument to the grant mapping
APIs). It worth mentioning that toolstack patch is based on non  upstreamed yet “Virtio support for toolstack
on Arm” series which is on review now [2].

Please note the following:
- Patch series only covers Arm and virtio-mmio (device-tree) for now. To enable the restricted memory access
  feature on Arm the following options should be set:
  CONFIG_XEN_VIRTIO = y
  CONFIG_XEN_HVM_VIRTIO_GRANT = y
- Some callbacks in xen-virtio DMA ops layer (map_sg/unmap_sg, etc) are not implemented yet as they are not
  needed/used in the first prototype

Patch series is rebased on Linux 5.18-rc2 tag and tested on Renesas Salvator-X board + H3 ES3.0 SoC (Arm64)
with standalone userspace (non-Qemu) virtio-mmio based virtio-disk backend running in Driver domain and Linux
guest running on existing virtio-blk driver (frontend). No issues were observed. Guest domain 'reboot/destroy'
use-cases work properly. I have also tested other use-cases such as assigning several virtio block devices
or a mix of virtio and Xen PV block devices to the guest. 

1. Xen changes located at (last patch):
https://github.com/otyshchenko1/xen/commits/libxl_virtio_next
2. Linux changes located at:
https://github.com/otyshchenko1/linux/commits/virtio_grant5
3. virtio-disk changes located at:
https://github.com/otyshchenko1/virtio-disk/commits/virtio_grant

Any feedback/help would be highly appreciated.

[1] https://www.youtube.com/watch?v=IrlEdaIUDPk
[2] https://lore.kernel.org/xen-devel/1649442065-8332-1-git-send-email-olekstysh@gmail.com/

Juergen Gross (2):
  xen/grants: support allocating consecutive grants
  virtio: add option to restrict memory access under Xen

Oleksandr Tyshchenko (4):
  dt-bindings: xen: Add xen,dev-domid property description for
    xen-virtio layer
  virtio: Various updates to xen-virtio DMA ops layer
  arm/xen: Introduce xen_setup_dma_ops()
  arm/xen: Assign xen-virtio DMA ops for virtio devices in Xen guests

 .../devicetree/bindings/virtio/xen,dev-domid.yaml  |  39 +++
 arch/arm/include/asm/xen/xen-ops.h                 |   1 +
 arch/arm/mm/dma-mapping.c                          |   5 +-
 arch/arm/xen/enlighten.c                           |  11 +
 arch/arm64/include/asm/xen/xen-ops.h               |   1 +
 arch/arm64/mm/dma-mapping.c                        |   5 +-
 arch/x86/mm/init.c                                 |  15 +
 arch/x86/mm/mem_encrypt.c                          |   5 -
 arch/x86/xen/Kconfig                               |   9 +
 drivers/xen/Kconfig                                |  20 ++
 drivers/xen/Makefile                               |   1 +
 drivers/xen/grant-table.c                          | 238 +++++++++++++--
 drivers/xen/xen-virtio.c                           | 335 +++++++++++++++++++++
 include/xen/arm/xen-ops.h                          |  20 ++
 include/xen/grant_table.h                          |   4 +
 include/xen/xen-ops.h                              |  13 +
 16 files changed, 679 insertions(+), 43 deletions(-)
 create mode 100644 Documentation/devicetree/bindings/virtio/xen,dev-domid.yaml
 create mode 100644 arch/arm/include/asm/xen/xen-ops.h
 create mode 100644 arch/arm64/include/asm/xen/xen-ops.h
 create mode 100644 drivers/xen/xen-virtio.c
 create mode 100644 include/xen/arm/xen-ops.h

-- 
2.7.4

I can only see three out of 6 patches on the linux-arm-kernel list,
which makes reviewing this impossible.  Also please Cc me directly
on any series doing crazy things with dma ops.  Thanks!

On 15.04.22 10:41, Christoph Hellwig wrote:

Hello Christoph

> I can only see three out of 6 patches on the linux-arm-kernel list,
> which makes reviewing this impossible.


Oops, I will add linux-arm-kernel. I blindly followed what 
get_maintainer.pl suggested for each patch plus added manually some Xen 
folks,
but, indeed, the first three patches add the base of this enabling work.


> Also please Cc me directly
> on any series doing crazy things with dma ops.  Thanks!

yes, sure.


-- 
Regards,

Oleksandr Tyshchenko

On Thu, Apr 14, 2022 at 10:19:27PM +0300, Oleksandr Tyshchenko wrote:
> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
> 
> Hello all.
> 
> The purpose of this RFC patch series is to add support for restricting memory access under Xen using specific
> grant table based DMA ops layer. Patch series is based on Juergen Gross’ initial work [1] which implies using
> grant references instead of raw guest physical addresses (GPA) for the virtio communications (some kind of
> the software IOMMU).
> 
> The high level idea is to create new Xen’s grant table based DMA ops layer for the guest Linux whose main
> purpose is to provide a special 64-bit DMA address which is formed by using the grant reference (for a page
> to be shared with the backend) with offset and setting the highest address bit (this is for the backend to
> be able to distinguish grant ref based DMA address from normal GPA). For this to work we need the ability
> to allocate contiguous (consecutive) grant references for multi-page allocations. And the backend then needs
> to offer VIRTIO_F_ACCESS_PLATFORM and VIRTIO_F_VERSION_1 feature bits (it must support virtio-mmio modern
> transport for 64-bit addresses in the virtqueue).

I'm not enough of a xen expert to review this, and I didn't get
all patches, but I'm very happy to see that approach being
taken. VIRTIO_F_ACCESS_PLATFORM and VIRTIO_F_VERSION_1 are
exactly the way to declare not all of memory is accessible.
Thanks!

> Xen's grant mapping mechanism is the secure and safe solution to share pages between domains which proven
> to work and works for years (in the context of traditional Xen PV drivers for example). So far, the foreign
> mapping is used for the virtio backend to map and access guest memory. With the foreign mapping, the backend
> is able to map arbitrary pages from the guest memory (or even from Dom0 memory). And as the result, the malicious
> backend which runs in a non-trusted domain can take advantage of this. Instead, with the grant mapping
> the backend is only allowed to map pages which were explicitly granted by the guest before and nothing else. 
> According to the discussions in various mainline threads this solution would likely be welcome because it
> perfectly fits in the security model Xen provides. 
> 
> What is more, the grant table based solution requires zero changes to the Xen hypervisor itself at least
> with virtio-mmio and DT (in comparison, for example, with "foreign mapping + virtio-iommu" solution which would
> require the whole new complex emulator in hypervisor in addition to new functionality/hypercall to pass IOVA
> from the virtio backend running elsewhere to the hypervisor and translate it to the GPA before mapping into
> P2M or denying the foreign mapping request if no corresponding IOVA-GPA mapping present in the IOMMU page table
> for that particular device). We only need to update toolstack to insert a new "xen,dev-domid" property to
> the virtio-mmio device node when creating a guest device-tree (this is an indicator for the guest to use grants
> and the ID of Xen domain where the corresponding backend resides, it is used as an argument to the grant mapping
> APIs). It worth mentioning that toolstack patch is based on non  upstreamed yet “Virtio support for toolstack
> on Arm” series which is on review now [2].
> 
> Please note the following:
> - Patch series only covers Arm and virtio-mmio (device-tree) for now. To enable the restricted memory access
>   feature on Arm the following options should be set:
>   CONFIG_XEN_VIRTIO = y
>   CONFIG_XEN_HVM_VIRTIO_GRANT = y
> - Some callbacks in xen-virtio DMA ops layer (map_sg/unmap_sg, etc) are not implemented yet as they are not
>   needed/used in the first prototype
> 
> Patch series is rebased on Linux 5.18-rc2 tag and tested on Renesas Salvator-X board + H3 ES3.0 SoC (Arm64)
> with standalone userspace (non-Qemu) virtio-mmio based virtio-disk backend running in Driver domain and Linux
> guest running on existing virtio-blk driver (frontend). No issues were observed. Guest domain 'reboot/destroy'
> use-cases work properly. I have also tested other use-cases such as assigning several virtio block devices
> or a mix of virtio and Xen PV block devices to the guest. 
> 
> 1. Xen changes located at (last patch):
> https://github.com/otyshchenko1/xen/commits/libxl_virtio_next
> 2. Linux changes located at:
> https://github.com/otyshchenko1/linux/commits/virtio_grant5
> 3. virtio-disk changes located at:
> https://github.com/otyshchenko1/virtio-disk/commits/virtio_grant
> 
> Any feedback/help would be highly appreciated.
> 
> [1] https://www.youtube.com/watch?v=IrlEdaIUDPk
> [2] https://lore.kernel.org/xen-devel/1649442065-8332-1-git-send-email-olekstysh@gmail.com/
> 
> Juergen Gross (2):
>   xen/grants: support allocating consecutive grants
>   virtio: add option to restrict memory access under Xen
> 
> Oleksandr Tyshchenko (4):
>   dt-bindings: xen: Add xen,dev-domid property description for
>     xen-virtio layer
>   virtio: Various updates to xen-virtio DMA ops layer
>   arm/xen: Introduce xen_setup_dma_ops()
>   arm/xen: Assign xen-virtio DMA ops for virtio devices in Xen guests
> 
>  .../devicetree/bindings/virtio/xen,dev-domid.yaml  |  39 +++
>  arch/arm/include/asm/xen/xen-ops.h                 |   1 +
>  arch/arm/mm/dma-mapping.c                          |   5 +-
>  arch/arm/xen/enlighten.c                           |  11 +
>  arch/arm64/include/asm/xen/xen-ops.h               |   1 +
>  arch/arm64/mm/dma-mapping.c                        |   5 +-
>  arch/x86/mm/init.c                                 |  15 +
>  arch/x86/mm/mem_encrypt.c                          |   5 -
>  arch/x86/xen/Kconfig                               |   9 +
>  drivers/xen/Kconfig                                |  20 ++
>  drivers/xen/Makefile                               |   1 +
>  drivers/xen/grant-table.c                          | 238 +++++++++++++--
>  drivers/xen/xen-virtio.c                           | 335 +++++++++++++++++++++
>  include/xen/arm/xen-ops.h                          |  20 ++
>  include/xen/grant_table.h                          |   4 +
>  include/xen/xen-ops.h                              |  13 +
>  16 files changed, 679 insertions(+), 43 deletions(-)
>  create mode 100644 Documentation/devicetree/bindings/virtio/xen,dev-domid.yaml
>  create mode 100644 arch/arm/include/asm/xen/xen-ops.h
>  create mode 100644 arch/arm64/include/asm/xen/xen-ops.h
>  create mode 100644 drivers/xen/xen-virtio.c
>  create mode 100644 include/xen/arm/xen-ops.h
> 
> -- 
> 2.7.4

On 15.04.22 11:44, Michael S. Tsirkin wrote:


Hello Michael



> On Thu, Apr 14, 2022 at 10:19:27PM +0300, Oleksandr Tyshchenko wrote:
>> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
>>
>> Hello all.
>>
>> The purpose of this RFC patch series is to add support for restricting memory access under Xen using specific
>> grant table based DMA ops layer. Patch series is based on Juergen Gross’ initial work [1] which implies using
>> grant references instead of raw guest physical addresses (GPA) for the virtio communications (some kind of
>> the software IOMMU).
>>
>> The high level idea is to create new Xen’s grant table based DMA ops layer for the guest Linux whose main
>> purpose is to provide a special 64-bit DMA address which is formed by using the grant reference (for a page
>> to be shared with the backend) with offset and setting the highest address bit (this is for the backend to
>> be able to distinguish grant ref based DMA address from normal GPA). For this to work we need the ability
>> to allocate contiguous (consecutive) grant references for multi-page allocations. And the backend then needs
>> to offer VIRTIO_F_ACCESS_PLATFORM and VIRTIO_F_VERSION_1 feature bits (it must support virtio-mmio modern
>> transport for 64-bit addresses in the virtqueue).
> I'm not enough of a xen expert to review this, and I didn't get
> all patches, but I'm very happy to see that approach being
> taken. VIRTIO_F_ACCESS_PLATFORM and VIRTIO_F_VERSION_1 are
> exactly the way to declare not all of memory is accessible.
> Thanks!

I am happy to hear that! Thank you.


Regarding the "all patches" I have already redirect missing ones, I hope 
you and Christoph will get them.

Sorry for the inconvenience.


>
>> Xen's grant mapping mechanism is the secure and safe solution to share pages between domains which proven
>> to work and works for years (in the context of traditional Xen PV drivers for example). So far, the foreign
>> mapping is used for the virtio backend to map and access guest memory. With the foreign mapping, the backend
>> is able to map arbitrary pages from the guest memory (or even from Dom0 memory). And as the result, the malicious
>> backend which runs in a non-trusted domain can take advantage of this. Instead, with the grant mapping
>> the backend is only allowed to map pages which were explicitly granted by the guest before and nothing else.
>> According to the discussions in various mainline threads this solution would likely be welcome because it
>> perfectly fits in the security model Xen provides.
>>
>> What is more, the grant table based solution requires zero changes to the Xen hypervisor itself at least
>> with virtio-mmio and DT (in comparison, for example, with "foreign mapping + virtio-iommu" solution which would
>> require the whole new complex emulator in hypervisor in addition to new functionality/hypercall to pass IOVA
>> from the virtio backend running elsewhere to the hypervisor and translate it to the GPA before mapping into
>> P2M or denying the foreign mapping request if no corresponding IOVA-GPA mapping present in the IOMMU page table
>> for that particular device). We only need to update toolstack to insert a new "xen,dev-domid" property to
>> the virtio-mmio device node when creating a guest device-tree (this is an indicator for the guest to use grants
>> and the ID of Xen domain where the corresponding backend resides, it is used as an argument to the grant mapping
>> APIs). It worth mentioning that toolstack patch is based on non  upstreamed yet “Virtio support for toolstack
>> on Arm” series which is on review now [2].
>>
>> Please note the following:
>> - Patch series only covers Arm and virtio-mmio (device-tree) for now. To enable the restricted memory access
>>    feature on Arm the following options should be set:
>>    CONFIG_XEN_VIRTIO = y
>>    CONFIG_XEN_HVM_VIRTIO_GRANT = y
>> - Some callbacks in xen-virtio DMA ops layer (map_sg/unmap_sg, etc) are not implemented yet as they are not
>>    needed/used in the first prototype
>>
>> Patch series is rebased on Linux 5.18-rc2 tag and tested on Renesas Salvator-X board + H3 ES3.0 SoC (Arm64)
>> with standalone userspace (non-Qemu) virtio-mmio based virtio-disk backend running in Driver domain and Linux
>> guest running on existing virtio-blk driver (frontend). No issues were observed. Guest domain 'reboot/destroy'
>> use-cases work properly. I have also tested other use-cases such as assigning several virtio block devices
>> or a mix of virtio and Xen PV block devices to the guest.
>>
>> 1. Xen changes located at (last patch):
>> https://github.com/otyshchenko1/xen/commits/libxl_virtio_next
>> 2. Linux changes located at:
>> https://github.com/otyshchenko1/linux/commits/virtio_grant5
>> 3. virtio-disk changes located at:
>> https://github.com/otyshchenko1/virtio-disk/commits/virtio_grant
>>
>> Any feedback/help would be highly appreciated.
>>
>> [1] https://www.youtube.com/watch?v=IrlEdaIUDPk
>> [2] https://lore.kernel.org/xen-devel/1649442065-8332-1-git-send-email-olekstysh@gmail.com/
>>
>> Juergen Gross (2):
>>    xen/grants: support allocating consecutive grants
>>    virtio: add option to restrict memory access under Xen
>>
>> Oleksandr Tyshchenko (4):
>>    dt-bindings: xen: Add xen,dev-domid property description for
>>      xen-virtio layer
>>    virtio: Various updates to xen-virtio DMA ops layer
>>    arm/xen: Introduce xen_setup_dma_ops()
>>    arm/xen: Assign xen-virtio DMA ops for virtio devices in Xen guests
>>
>>   .../devicetree/bindings/virtio/xen,dev-domid.yaml  |  39 +++
>>   arch/arm/include/asm/xen/xen-ops.h                 |   1 +
>>   arch/arm/mm/dma-mapping.c                          |   5 +-
>>   arch/arm/xen/enlighten.c                           |  11 +
>>   arch/arm64/include/asm/xen/xen-ops.h               |   1 +
>>   arch/arm64/mm/dma-mapping.c                        |   5 +-
>>   arch/x86/mm/init.c                                 |  15 +
>>   arch/x86/mm/mem_encrypt.c                          |   5 -
>>   arch/x86/xen/Kconfig                               |   9 +
>>   drivers/xen/Kconfig                                |  20 ++
>>   drivers/xen/Makefile                               |   1 +
>>   drivers/xen/grant-table.c                          | 238 +++++++++++++--
>>   drivers/xen/xen-virtio.c                           | 335 +++++++++++++++++++++
>>   include/xen/arm/xen-ops.h                          |  20 ++
>>   include/xen/grant_table.h                          |   4 +
>>   include/xen/xen-ops.h                              |  13 +
>>   16 files changed, 679 insertions(+), 43 deletions(-)
>>   create mode 100644 Documentation/devicetree/bindings/virtio/xen,dev-domid.yaml
>>   create mode 100644 arch/arm/include/asm/xen/xen-ops.h
>>   create mode 100644 arch/arm64/include/asm/xen/xen-ops.h
>>   create mode 100644 drivers/xen/xen-virtio.c
>>   create mode 100644 include/xen/arm/xen-ops.h
>>
>> -- 
>> 2.7.4

-- 
Regards,

Oleksandr Tyshchenko