hw/arm/smmu-common.c | 74 ++- hw/arm/smmuv3-internal.h | 128 +++++- hw/arm/smmuv3.c | 844 ++++++++++++++++++++++++++++++----- hw/arm/trace-events | 7 +- hw/arm/virt.c | 5 + include/hw/arm/smmu-common.h | 23 +- include/hw/arm/smmuv3.h | 27 ++ 7 files changed, 968 insertions(+), 140 deletions(-)
Hi all, This patch series introduces initial support for emulating the Arm SMMUv3 Secure State. As Pierrick pointed out in a previous discussion [1], full Secure SMMUv3 emulation is a notable missing piece in QEMU. While the FVP model has some support, its limited PCIe capabilities make it challenging for complex use cases. The ability to properly manage device DMA from a secure context is a critical prerequisite for enabling device assignment (passthrough) for confidential computing solutions like Arm CCA and related research such as virtCCA [2]. This series aims to build that foundational support in QEMU. This work is being proposed as an RFC. It introduces a significant amount of new logic, including the core concept of modeling parallel secure and non-secure contexts within a single SMMUv3 device. I am seeking feedback on the overall approach, the core refactoring, and the implementation details before proceeding further. The series begins by implementing the components of the secure programming interface, then progressively refactors the core SMMU logic to handle secure and non-secure contexts in parallel. Secure Interface Implementation: The initial patches add the secure-side registers, implement their read/write logic, and enable the secure command and event queues. This includes the S_INIT mechanism and the new secure TLB invalidation commands. Core Logic Refactoring: The next set of patches makes the core SMMU functions security-state aware. This involves plumbing an is_secure context flag through the main code paths and adding logic to route SMMU-originated memory accesses to the correct (Secure or Non-secure) address space. Cache Isolation: With the core logic now aware of security states, the following patches refactor the configuration and translation lookup caches. The cache keys are modified to include the security context, ensuring that secure and non-secure entries for the same device or address are properly isolated and preventing aliasing. Framework Integration: The final patch connects the SMMU's internal security context to the generic QEMU IOMMU framework by using the iommu_index to represent the architectural SEC_SID. To validate this work, I performed the following tests: Non-Secure Regression: To ensure that existing functionality remains intact, I ran a nested virtualization test. A TCG guest was created on the host, with iommu=smmuv3 and with an emulated PCIe NVMe device assigned. Command line of TCG VM is below: qemu-system-aarch64 \ -machine virt,virtualization=on,gic-version=3,iommu=smmuv3 \ -cpu max -smp 1 -m 4080M \ -accel tcg,thread=single,tb-size=512 \ -kernel Image \ -append 'nokaslr root=/dev/vda rw rootfstype=ext4 iommu.passthrough=on' \ -device pcie-root-port,bus=pcie.0,id=rp0,addr=0x4.0,chassis=1,port=0x10 \ -device pcie-root-port,bus=pcie.0,id=rp1,addr=0x5.0,chassis=2,port=0x11 \ -drive if=none,file=u2204fs.img.qcow2,format=qcow2,id=hd0 \ -device virtio-blk-device,drive=hd0 \ -qmp unix:/tmp/qmp-sock12,server=on,wait=off \ -netdev user,id=eth0,hostfwd=tcp::10022-:22,hostfwd=tcp::59922-:5922 \ -device virtio-net-device,netdev=eth0 \ -drive if=none,file=nvme.img,format=raw,id=nvme0 \ -device nvme,drive=nvme0,serial=deadbeef \ -d unimp,guest_errors -trace events=smmu-events.txt -D qemu.log -nographic Inside this TCG VM, a KVM guest was launched, and the same NVMe device was re-assigned to it via VFIO. Command line of KVM VM inside TCG VM is below: sudo qemu-system-aarch64 \ -enable-kvm -m 1024 -cpu host -M virt \ -machine virt,gic-version=3 \ -cpu max -append "nokaslr" -smp 1 \ -monitor stdio \ -kernel 5.15.Image \ -initrd rootfs.cpio.gz \ -display vnc=:22,id=primary \ -device vfio-pci,host=00:01.0 The KVM guest was able to perform I/O on the device correctly, confirming that the non-secure path is not broken. Secure Register/Command Interface: I set up an OP-TEE + Hafnium environment. Hafnium's smmuv3_driver_init function was used to test the secure register I/O and command queue functionality (excluding translation). As Hafnium assumes larger queue and StreamID sizes than are practical without TTST support, I temporarily patched Hafnium to use smaller values, allowing its driver to initialize the emulated secure SMMU successfully. Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not support QEMU, and no secure device assignment feature exists yet, I created a custom platform device to test the secure translation flow. To trigger the translation logic, I initiated MMIO writes to this device from within Hafnium. The device's MMIO callback handler then performed DMA accesses via its IOMMU region, exercising the secure translation path. While SMMUv3 is typically used for PCIe on physical SoCs, the architecture allows its use with platform devices via a stream-id binding in the device tree. The test harness required some non-standard modifications to decouple the SMMU from its tight integration with PCIe. The code for this test device is available for review at [3]. README.md with detailed instructions is also provided. I've attempted to follow all of the guidance in the "Submitting a Patch" guide, but as this is my first series of this scale, I apologize if I missed anything and welcome all feedback. Thanks, Tang [1] https://lists.nongnu.org/archive/html/qemu-devel/2025-06/msg02940.html [2] https://arxiv.org/abs/2306.11011 [3] https://github.com/hnusdr/qemu Tao Tang (11): hw/arm/smmuv3: Introduce secure registers and commands hw/arm/smmuv3: Implement read/write logic for secure registers hw/arm/smmuv3: Implement S_INIT for secure initialization hw/arm/smmuv3: Enable command processing for the Secure state hw/arm/smmuv3: Support secure event queue and error handling hw/arm/smmuv3: Plumb security state through core functions hw/arm/smmuv3: Add separate address space for secure SMMU accesses hw/arm/smmuv3: Enable secure-side stage 2 TLB invalidations hw/arm/smmuv3: Make the configuration cache security-state aware hw/arm/smmuv3: Differentiate secure TLB entries via keying hw/arm/smmuv3: Use iommu_index to represent SEC_SID hw/arm/smmu-common.c | 74 ++- hw/arm/smmuv3-internal.h | 128 +++++- hw/arm/smmuv3.c | 844 ++++++++++++++++++++++++++++++----- hw/arm/trace-events | 7 +- hw/arm/virt.c | 5 + include/hw/arm/smmu-common.h | 23 +- include/hw/arm/smmuv3.h | 27 ++ 7 files changed, 968 insertions(+), 140 deletions(-) -- 2.34.1
On 8/6/25 8:11 AM, Tao Tang wrote: > Hi all, > > This patch series introduces initial support for emulating the Arm SMMUv3 > Secure State. > > As Pierrick pointed out in a previous discussion [1], full Secure SMMUv3 > emulation is a notable missing piece in QEMU. While the FVP model has > some support, its limited PCIe capabilities make it challenging for > complex use cases. The ability to properly manage device DMA from a > secure context is a critical prerequisite for enabling device assignment > (passthrough) for confidential computing solutions like Arm CCA and > related research such as virtCCA [2]. This series aims to build that > foundational support in QEMU. > Thanks for posting this series, it's definitely an important piece missing for emulating newer SMMU versions. > This work is being proposed as an RFC. It introduces a significant amount > of new logic, including the core concept of modeling parallel secure and > non-secure contexts within a single SMMUv3 device. I am seeking feedback > on the overall approach, the core refactoring, and the implementation > details before proceeding further. > > The series begins by implementing the components of the secure programming > interface, then progressively refactors the core SMMU logic to handle > secure and non-secure contexts in parallel. > > Secure Interface Implementation: The initial patches add the > secure-side registers, implement their read/write logic, and enable > the secure command and event queues. This includes the S_INIT > mechanism and the new secure TLB invalidation commands. > > Core Logic Refactoring: The next set of patches makes the core SMMU > functions security-state aware. This involves plumbing an is_secure > context flag through the main code paths and adding logic to route > SMMU-originated memory accesses to the correct (Secure or Non-secure) > address space. > > Cache Isolation: With the core logic now aware of security states, > the following patches refactor the configuration and translation > lookup caches. The cache keys are modified to include the security > context, ensuring that secure and non-secure entries for the same > device or address are properly isolated and preventing aliasing. > > Framework Integration: The final patch connects the SMMU's internal > security context to the generic QEMU IOMMU framework by using the > iommu_index to represent the architectural SEC_SID. > > To validate this work, I performed the following tests: > > Non-Secure Regression: To ensure that existing functionality remains > intact, I ran a nested virtualization test. A TCG guest was created on > the host, with iommu=smmuv3 and with an emulated PCIe NVMe device assigned. > Command line of TCG VM is below: > > qemu-system-aarch64 \ > -machine virt,virtualization=on,gic-version=3,iommu=smmuv3 \ > -cpu max -smp 1 -m 4080M \ > -accel tcg,thread=single,tb-size=512 \ > -kernel Image \ > -append 'nokaslr root=/dev/vda rw rootfstype=ext4 iommu.passthrough=on' \ > -device pcie-root-port,bus=pcie.0,id=rp0,addr=0x4.0,chassis=1,port=0x10 \ > -device pcie-root-port,bus=pcie.0,id=rp1,addr=0x5.0,chassis=2,port=0x11 \ > -drive if=none,file=u2204fs.img.qcow2,format=qcow2,id=hd0 \ > -device virtio-blk-device,drive=hd0 \ > -qmp unix:/tmp/qmp-sock12,server=on,wait=off \ > -netdev user,id=eth0,hostfwd=tcp::10022-:22,hostfwd=tcp::59922-:5922 \ > -device virtio-net-device,netdev=eth0 \ > -drive if=none,file=nvme.img,format=raw,id=nvme0 \ > -device nvme,drive=nvme0,serial=deadbeef \ > -d unimp,guest_errors -trace events=smmu-events.txt -D qemu.log -nographic > > Inside this TCG VM, a KVM guest was launched, and the same NVMe device was > re-assigned to it via VFIO. > Command line of KVM VM inside TCG VM is below: > > sudo qemu-system-aarch64 \ > -enable-kvm -m 1024 -cpu host -M virt \ > -machine virt,gic-version=3 \ > -cpu max -append "nokaslr" -smp 1 \ > -monitor stdio \ > -kernel 5.15.Image \ > -initrd rootfs.cpio.gz \ > -display vnc=:22,id=primary \ > -device vfio-pci,host=00:01.0 > > The KVM guest was able to perform I/O on the device > correctly, confirming that the non-secure path is not broken. > > Secure Register/Command Interface: I set up an OP-TEE + Hafnium > environment. Hafnium's smmuv3_driver_init function was used to test > the secure register I/O and command queue functionality (excluding > translation). As Hafnium assumes larger queue and StreamID sizes than > are practical without TTST support, I temporarily patched Hafnium to > use smaller values, allowing its driver to initialize the emulated > secure SMMU successfully. > Would that be possible to share your changes, and build instructions for this? While working on SMMU emulation, we finally left this on the side due to lack of a software stack being able to use secure SMMU, as we were not aware that Hafnium + op-tee could make use of it. > Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not > support QEMU, and no secure device assignment feature exists yet, I > created a custom platform device to test the secure translation flow. > To trigger the translation logic, I initiated MMIO writes to this > device from within Hafnium. The device's MMIO callback handler then > performed DMA accesses via its IOMMU region, exercising the secure > translation path. While SMMUv3 is typically used for PCIe on > physical SoCs, the architecture allows its use with platform devices > via a stream-id binding in the device tree. The test harness > required some non-standard modifications to decouple the SMMU from > its tight integration with PCIe. The code for this test device is > available for review at [3]. README.md with detailed instructions is > also provided. > I am not sure about the current policy in QEMU for test oriented devices, but it would be really useful to have something similar upstream (Note: it's out of the scope of this series). One challenge working with SMMU emulation is that reproducing setups and triggering specific code paths is hard to achieve, due to the indirect use of SMMU feature (through DMA) and the complex software stack usually involved. Having something upstream available to work on SMMU emulation, at least on device side, would be a great addition. Eric, Peter, is this something that would be acceptable to merge? > I've attempted to follow all of the guidance in the "Submitting a Patch" > guide, but as this is my first series of this scale, I apologize if I > missed anything and welcome all feedback. > > Thanks, > Tang > > [1] https://lists.nongnu.org/archive/html/qemu-devel/2025-06/msg02940.html > [2] https://arxiv.org/abs/2306.11011 I was not aware of it, thanks for sharing this excellent paper. > [3] https://github.com/hnusdr/qemu > > Tao Tang (11): > hw/arm/smmuv3: Introduce secure registers and commands > hw/arm/smmuv3: Implement read/write logic for secure registers > hw/arm/smmuv3: Implement S_INIT for secure initialization > hw/arm/smmuv3: Enable command processing for the Secure state > hw/arm/smmuv3: Support secure event queue and error handling > hw/arm/smmuv3: Plumb security state through core functions > hw/arm/smmuv3: Add separate address space for secure SMMU accesses > hw/arm/smmuv3: Enable secure-side stage 2 TLB invalidations > hw/arm/smmuv3: Make the configuration cache security-state aware > hw/arm/smmuv3: Differentiate secure TLB entries via keying > hw/arm/smmuv3: Use iommu_index to represent SEC_SID > > hw/arm/smmu-common.c | 74 ++- > hw/arm/smmuv3-internal.h | 128 +++++- > hw/arm/smmuv3.c | 844 ++++++++++++++++++++++++++++++----- > hw/arm/trace-events | 7 +- > hw/arm/virt.c | 5 + > include/hw/arm/smmu-common.h | 23 +- > include/hw/arm/smmuv3.h | 27 ++ > 7 files changed, 968 insertions(+), 140 deletions(-) > > -- > 2.34.1 > > Regards, Pierrick
On 2025/8/7 05:28, Pierrick Bouvier wrote: > On 8/6/25 8:11 AM, Tao Tang wrote: >> Hi all, >> >> This patch series introduces initial support for emulating the Arm >> SMMUv3 >> Secure State. >> >> As Pierrick pointed out in a previous discussion [1], full Secure SMMUv3 >> emulation is a notable missing piece in QEMU. While the FVP model has >> some support, its limited PCIe capabilities make it challenging for >> complex use cases. The ability to properly manage device DMA from a >> secure context is a critical prerequisite for enabling device assignment >> (passthrough) for confidential computing solutions like Arm CCA and >> related research such as virtCCA [2]. This series aims to build that >> foundational support in QEMU. >> > > Thanks for posting this series, it's definitely an important piece > missing for emulating newer SMMU versions. > >> This work is being proposed as an RFC. It introduces a significant >> amount >> of new logic, including the core concept of modeling parallel secure and >> non-secure contexts within a single SMMUv3 device. I am seeking feedback >> on the overall approach, the core refactoring, and the implementation >> details before proceeding further. >> >> The series begins by implementing the components of the secure >> programming >> interface, then progressively refactors the core SMMU logic to handle >> secure and non-secure contexts in parallel. >> >> Secure Interface Implementation: The initial patches add the >> secure-side registers, implement their read/write logic, and enable >> the secure command and event queues. This includes the S_INIT >> mechanism and the new secure TLB invalidation commands. >> >> Core Logic Refactoring: The next set of patches makes the core SMMU >> functions security-state aware. This involves plumbing an is_secure >> context flag through the main code paths and adding logic to route >> SMMU-originated memory accesses to the correct (Secure or Non-secure) >> address space. >> >> Cache Isolation: With the core logic now aware of security states, >> the following patches refactor the configuration and translation >> lookup caches. The cache keys are modified to include the security >> context, ensuring that secure and non-secure entries for the same >> device or address are properly isolated and preventing aliasing. >> >> Framework Integration: The final patch connects the SMMU's internal >> security context to the generic QEMU IOMMU framework by using the >> iommu_index to represent the architectural SEC_SID. >> >> To validate this work, I performed the following tests: >> >> Non-Secure Regression: To ensure that existing functionality remains >> intact, I ran a nested virtualization test. A TCG guest was created on >> the host, with iommu=smmuv3 and with an emulated PCIe NVMe device >> assigned. >> Command line of TCG VM is below: >> >> qemu-system-aarch64 \ >> -machine virt,virtualization=on,gic-version=3,iommu=smmuv3 \ >> -cpu max -smp 1 -m 4080M \ >> -accel tcg,thread=single,tb-size=512 \ >> -kernel Image \ >> -append 'nokaslr root=/dev/vda rw rootfstype=ext4 >> iommu.passthrough=on' \ >> -device >> pcie-root-port,bus=pcie.0,id=rp0,addr=0x4.0,chassis=1,port=0x10 \ >> -device >> pcie-root-port,bus=pcie.0,id=rp1,addr=0x5.0,chassis=2,port=0x11 \ >> -drive if=none,file=u2204fs.img.qcow2,format=qcow2,id=hd0 \ >> -device virtio-blk-device,drive=hd0 \ >> -qmp unix:/tmp/qmp-sock12,server=on,wait=off \ >> -netdev user,id=eth0,hostfwd=tcp::10022-:22,hostfwd=tcp::59922-:5922 \ >> -device virtio-net-device,netdev=eth0 \ >> -drive if=none,file=nvme.img,format=raw,id=nvme0 \ >> -device nvme,drive=nvme0,serial=deadbeef \ >> -d unimp,guest_errors -trace events=smmu-events.txt -D qemu.log >> -nographic >> >> Inside this TCG VM, a KVM guest was launched, and the same NVMe >> device was >> re-assigned to it via VFIO. >> Command line of KVM VM inside TCG VM is below: >> >> sudo qemu-system-aarch64 \ >> -enable-kvm -m 1024 -cpu host -M virt \ >> -machine virt,gic-version=3 \ >> -cpu max -append "nokaslr" -smp 1 \ >> -monitor stdio \ >> -kernel 5.15.Image \ >> -initrd rootfs.cpio.gz \ >> -display vnc=:22,id=primary \ >> -device vfio-pci,host=00:01.0 >> >> The KVM guest was able to perform I/O on the device >> correctly, confirming that the non-secure path is not broken. >> >> Secure Register/Command Interface: I set up an OP-TEE + Hafnium >> environment. Hafnium's smmuv3_driver_init function was used to test >> the secure register I/O and command queue functionality (excluding >> translation). As Hafnium assumes larger queue and StreamID sizes than >> are practical without TTST support, I temporarily patched Hafnium to >> use smaller values, allowing its driver to initialize the emulated >> secure SMMU successfully. >> > > Would that be possible to share your changes, and build instructions > for this? While working on SMMU emulation, we finally left this on the > side due to lack of a software stack being able to use secure SMMU, as > we were not aware that Hafnium + op-tee could make use of it. > Hi Pierrick, Thanks for your interest! I'm very happy to share my work on this. I've documented the setup process, including our code modifications and the step-by-step build instructions in this link: https://hnusdr.github.io/2025/08/09/Test-Secure-SMMU-with-Hafnium-ENG The core point of these changes is to enable the SMMUv3 feature in Hafnium. This leads to numerous read/write operations on SMMUv3 secure registers and various queue manipulations within the smmuv3_driver_init function in Hafnium. However, it's important to note that this initialization process itself does not initiate any DMA memory access that would trigger the smmuv3_translate flow. Even so, we've devised a method to test the subsequent Secure Translation Path by leveraging the smmuv3-test platform device. This approach allows us to verify the entire SMMUv3 flow, from initialization to translation. >> Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not >> support QEMU, and no secure device assignment feature exists yet, I >> created a custom platform device to test the secure translation flow. >> To trigger the translation logic, I initiated MMIO writes to this >> device from within Hafnium. The device's MMIO callback handler then >> performed DMA accesses via its IOMMU region, exercising the secure >> translation path. While SMMUv3 is typically used for PCIe on >> physical SoCs, the architecture allows its use with platform devices >> via a stream-id binding in the device tree. The test harness >> required some non-standard modifications to decouple the SMMU from >> its tight integration with PCIe. The code for this test device is >> available for review at [3]. README.md with detailed instructions is >> also provided. >> > > I am not sure about the current policy in QEMU for test oriented > devices, but it would be really useful to have something similar > upstream (Note: it's out of the scope of this series). > One challenge working with SMMU emulation is that reproducing setups > and triggering specific code paths is hard to achieve, due to the > indirect use of SMMU feature (through DMA) and the complex software > stack usually involved. > Having something upstream available to work on SMMU emulation, at > least on device side, would be a great addition. > > Eric, Peter, is this something that would be acceptable to merge? > Looking ahead, my plan is to refactor the smmuv3-test platform device. The goal is to make it self-contained within QEMU, removing the current dependency on Hafnium to trigger its operations. I plan to submit this as a separate RFC patch series in the next few days. However, the current version at [3] is already sufficient to test the code logic introduced in this RFC. >> I've attempted to follow all of the guidance in the "Submitting a Patch" >> guide, but as this is my first series of this scale, I apologize if I >> missed anything and welcome all feedback. >> >> Thanks, >> Tang >> >> [1] >> https://lists.nongnu.org/archive/html/qemu-devel/2025-06/msg02940.html >> [2] https://arxiv.org/abs/2306.11011 > > I was not aware of it, thanks for sharing this excellent paper. > >> [3] https://github.com/hnusdr/qemu >> >> Tao Tang (11): >> hw/arm/smmuv3: Introduce secure registers and commands >> hw/arm/smmuv3: Implement read/write logic for secure registers >> hw/arm/smmuv3: Implement S_INIT for secure initialization >> hw/arm/smmuv3: Enable command processing for the Secure state >> hw/arm/smmuv3: Support secure event queue and error handling >> hw/arm/smmuv3: Plumb security state through core functions >> hw/arm/smmuv3: Add separate address space for secure SMMU accesses >> hw/arm/smmuv3: Enable secure-side stage 2 TLB invalidations >> hw/arm/smmuv3: Make the configuration cache security-state aware >> hw/arm/smmuv3: Differentiate secure TLB entries via keying >> hw/arm/smmuv3: Use iommu_index to represent SEC_SID >> >> hw/arm/smmu-common.c | 74 ++- >> hw/arm/smmuv3-internal.h | 128 +++++- >> hw/arm/smmuv3.c | 844 ++++++++++++++++++++++++++++++----- >> hw/arm/trace-events | 7 +- >> hw/arm/virt.c | 5 + >> include/hw/arm/smmu-common.h | 23 +- >> include/hw/arm/smmuv3.h | 27 ++ >> 7 files changed, 968 insertions(+), 140 deletions(-) >> >> -- >> 2.34.1 >> >> > > Regards, > Pierrick -- Best, Tang
On 8/10/25 9:11 AM, Tao Tang wrote: > > On 2025/8/7 05:28, Pierrick Bouvier wrote: >> On 8/6/25 8:11 AM, Tao Tang wrote: >>> Hi all, >>> >>> This patch series introduces initial support for emulating the Arm >>> SMMUv3 >>> Secure State. >>> >>> As Pierrick pointed out in a previous discussion [1], full Secure SMMUv3 >>> emulation is a notable missing piece in QEMU. While the FVP model has >>> some support, its limited PCIe capabilities make it challenging for >>> complex use cases. The ability to properly manage device DMA from a >>> secure context is a critical prerequisite for enabling device assignment >>> (passthrough) for confidential computing solutions like Arm CCA and >>> related research such as virtCCA [2]. This series aims to build that >>> foundational support in QEMU. >>> >> >> Thanks for posting this series, it's definitely an important piece >> missing for emulating newer SMMU versions. >> >>> This work is being proposed as an RFC. It introduces a significant >>> amount >>> of new logic, including the core concept of modeling parallel secure and >>> non-secure contexts within a single SMMUv3 device. I am seeking feedback >>> on the overall approach, the core refactoring, and the implementation >>> details before proceeding further. >>> >>> The series begins by implementing the components of the secure >>> programming >>> interface, then progressively refactors the core SMMU logic to handle >>> secure and non-secure contexts in parallel. >>> >>> Secure Interface Implementation: The initial patches add the >>> secure-side registers, implement their read/write logic, and enable >>> the secure command and event queues. This includes the S_INIT >>> mechanism and the new secure TLB invalidation commands. >>> >>> Core Logic Refactoring: The next set of patches makes the core SMMU >>> functions security-state aware. This involves plumbing an is_secure >>> context flag through the main code paths and adding logic to route >>> SMMU-originated memory accesses to the correct (Secure or Non-secure) >>> address space. >>> >>> Cache Isolation: With the core logic now aware of security states, >>> the following patches refactor the configuration and translation >>> lookup caches. The cache keys are modified to include the security >>> context, ensuring that secure and non-secure entries for the same >>> device or address are properly isolated and preventing aliasing. >>> >>> Framework Integration: The final patch connects the SMMU's internal >>> security context to the generic QEMU IOMMU framework by using the >>> iommu_index to represent the architectural SEC_SID. >>> >>> To validate this work, I performed the following tests: >>> >>> Non-Secure Regression: To ensure that existing functionality remains >>> intact, I ran a nested virtualization test. A TCG guest was created on >>> the host, with iommu=smmuv3 and with an emulated PCIe NVMe device >>> assigned. >>> Command line of TCG VM is below: >>> >>> qemu-system-aarch64 \ >>> -machine virt,virtualization=on,gic-version=3,iommu=smmuv3 \ >>> -cpu max -smp 1 -m 4080M \ >>> -accel tcg,thread=single,tb-size=512 \ >>> -kernel Image \ >>> -append 'nokaslr root=/dev/vda rw rootfstype=ext4 >>> iommu.passthrough=on' \ >>> -device >>> pcie-root-port,bus=pcie.0,id=rp0,addr=0x4.0,chassis=1,port=0x10 \ >>> -device >>> pcie-root-port,bus=pcie.0,id=rp1,addr=0x5.0,chassis=2,port=0x11 \ >>> -drive if=none,file=u2204fs.img.qcow2,format=qcow2,id=hd0 \ >>> -device virtio-blk-device,drive=hd0 \ >>> -qmp unix:/tmp/qmp-sock12,server=on,wait=off \ >>> -netdev user,id=eth0,hostfwd=tcp::10022-:22,hostfwd=tcp::59922-:5922 \ >>> -device virtio-net-device,netdev=eth0 \ >>> -drive if=none,file=nvme.img,format=raw,id=nvme0 \ >>> -device nvme,drive=nvme0,serial=deadbeef \ >>> -d unimp,guest_errors -trace events=smmu-events.txt -D qemu.log >>> -nographic >>> >>> Inside this TCG VM, a KVM guest was launched, and the same NVMe >>> device was >>> re-assigned to it via VFIO. >>> Command line of KVM VM inside TCG VM is below: >>> >>> sudo qemu-system-aarch64 \ >>> -enable-kvm -m 1024 -cpu host -M virt \ >>> -machine virt,gic-version=3 \ >>> -cpu max -append "nokaslr" -smp 1 \ >>> -monitor stdio \ >>> -kernel 5.15.Image \ >>> -initrd rootfs.cpio.gz \ >>> -display vnc=:22,id=primary \ >>> -device vfio-pci,host=00:01.0 >>> >>> The KVM guest was able to perform I/O on the device >>> correctly, confirming that the non-secure path is not broken. >>> >>> Secure Register/Command Interface: I set up an OP-TEE + Hafnium >>> environment. Hafnium's smmuv3_driver_init function was used to test >>> the secure register I/O and command queue functionality (excluding >>> translation). As Hafnium assumes larger queue and StreamID sizes than >>> are practical without TTST support, I temporarily patched Hafnium to >>> use smaller values, allowing its driver to initialize the emulated >>> secure SMMU successfully. >>> >> >> Would that be possible to share your changes, and build instructions >> for this? While working on SMMU emulation, we finally left this on the >> side due to lack of a software stack being able to use secure SMMU, as >> we were not aware that Hafnium + op-tee could make use of it. >> > Hi Pierrick, > > Thanks for your interest! I'm very happy to share my work on this. I've > documented the setup process, including our code modifications and the > step-by-step build instructions in this link: > > https://hnusdr.github.io/2025/08/09/Test-Secure-SMMU-with-Hafnium-ENG > Thanks for taking the time to assemble all this in a comprehensible post, I'll give it a try when I have some spare time. > > The core point of these changes is to enable the SMMUv3 feature in > Hafnium. This leads to numerous read/write operations on SMMUv3 secure > registers and various queue manipulations within the smmuv3_driver_init > function in Hafnium. > > However, it's important to note that this initialization process itself > does not initiate any DMA memory access that would trigger the > smmuv3_translate flow. > I understand the difference. It can be tricky to generate specific translation scenarios, which is where a custom test device can really help. > Even so, we've devised a method to test the subsequent Secure > Translation Path by leveraging the smmuv3-test platform device. This > approach allows us to verify the entire SMMUv3 flow, from initialization > to translation. > Does it rely on a custom driver integration into an existing firmware or the kernel? > >>> Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not >>> support QEMU, and no secure device assignment feature exists yet, I >>> created a custom platform device to test the secure translation flow. >>> To trigger the translation logic, I initiated MMIO writes to this >>> device from within Hafnium. The device's MMIO callback handler then >>> performed DMA accesses via its IOMMU region, exercising the secure >>> translation path. While SMMUv3 is typically used for PCIe on >>> physical SoCs, the architecture allows its use with platform devices >>> via a stream-id binding in the device tree. The test harness >>> required some non-standard modifications to decouple the SMMU from >>> its tight integration with PCIe. The code for this test device is >>> available for review at [3]. README.md with detailed instructions is >>> also provided. >>> >> >> I am not sure about the current policy in QEMU for test oriented >> devices, but it would be really useful to have something similar >> upstream (Note: it's out of the scope of this series). >> One challenge working with SMMU emulation is that reproducing setups >> and triggering specific code paths is hard to achieve, due to the >> indirect use of SMMU feature (through DMA) and the complex software >> stack usually involved. >> Having something upstream available to work on SMMU emulation, at >> least on device side, would be a great addition. >> >> Eric, Peter, is this something that would be acceptable to merge? >> > > Looking ahead, my plan is to refactor the smmuv3-test platform device. > The goal is to make it self-contained within QEMU, removing the current > dependency on Hafnium to trigger its operations. I plan to submit this > as a separate RFC patch series in the next few days. > This is very welcome. Once this is in place, it would be great to add a new test to make sure things don't regress, and from where we can iterate. By self-contained within QEMU, do you mean a QTest based test? Regards, Pierrick
On 2025/8/13 02:04, Pierrick Bouvier wrote: > On 8/10/25 9:11 AM, Tao Tang wrote: >> >> On 2025/8/7 05:28, Pierrick Bouvier wrote: >>> On 8/6/25 8:11 AM, Tao Tang wrote: >>>> Hi all, >>>> >>>> This patch series introduces initial support for emulating the Arm >>>> SMMUv3 >>>> Secure State. >>>> ------------------------------<snip>------------------------------ >>>> >>>> >>>> ------------------------------<snip>------------------------------ >>>> Secure Register/Command Interface: I set up an OP-TEE + Hafnium >>>> environment. Hafnium's smmuv3_driver_init function was used to test >>>> the secure register I/O and command queue functionality (excluding >>>> translation). As Hafnium assumes larger queue and StreamID sizes than >>>> are practical without TTST support, I temporarily patched Hafnium to >>>> use smaller values, allowing its driver to initialize the emulated >>>> secure SMMU successfully. >>>> >>> >>> Would that be possible to share your changes, and build instructions >>> for this? While working on SMMU emulation, we finally left this on the >>> side due to lack of a software stack being able to use secure SMMU, as >>> we were not aware that Hafnium + op-tee could make use of it. >>> >> Hi Pierrick, >> >> Thanks for your interest! I'm very happy to share my work on this. I've >> documented the setup process, including our code modifications and the >> step-by-step build instructions in this link: >> >> https://hnusdr.github.io/2025/08/09/Test-Secure-SMMU-with-Hafnium-ENG >> > > Thanks for taking the time to assemble all this in a comprehensible > post, I'll give it a try when I have some spare time. Hi Pierrick, You're welcome, and please feel free to let me know if you run into any issues. > >> >> The core point of these changes is to enable the SMMUv3 feature in >> Hafnium. This leads to numerous read/write operations on SMMUv3 secure >> registers and various queue manipulations within the smmuv3_driver_init >> function in Hafnium. >> >> However, it's important to note that this initialization process itself >> does not initiate any DMA memory access that would trigger the >> smmuv3_translate flow. >> > > I understand the difference. It can be tricky to generate specific > translation scenarios, which is where a custom test device can really > help. > >> Even so, we've devised a method to test the subsequent Secure >> Translation Path by leveraging the smmuv3-test platform device. This >> approach allows us to verify the entire SMMUv3 flow, from initialization >> to translation. >> > > Does it rely on a custom driver integration into an existing firmware > or the kernel? > >> >>>> Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not >>>> support QEMU, and no secure device assignment feature exists yet, I >>>> created a custom platform device to test the secure translation flow. >>>> To trigger the translation logic, I initiated MMIO writes to this >>>> device from within Hafnium. The device's MMIO callback handler then >>>> performed DMA accesses via its IOMMU region, exercising the secure >>>> translation path. While SMMUv3 is typically used for PCIe on >>>> physical SoCs, the architecture allows its use with platform devices >>>> via a stream-id binding in the device tree. The test harness >>>> required some non-standard modifications to decouple the SMMU from >>>> its tight integration with PCIe. The code for this test device is >>>> available for review at [3]. README.md with detailed instructions is >>>> also provided. >>>> >>> >>> I am not sure about the current policy in QEMU for test oriented >>> devices, but it would be really useful to have something similar >>> upstream (Note: it's out of the scope of this series). >>> One challenge working with SMMU emulation is that reproducing setups >>> and triggering specific code paths is hard to achieve, due to the >>> indirect use of SMMU feature (through DMA) and the complex software >>> stack usually involved. >>> Having something upstream available to work on SMMU emulation, at >>> least on device side, would be a great addition. >>> >>> Eric, Peter, is this something that would be acceptable to merge? >>> >> >> Looking ahead, my plan is to refactor the smmuv3-test platform device. >> The goal is to make it self-contained within QEMU, removing the current >> dependency on Hafnium to trigger its operations. I plan to submit this >> as a separate RFC patch series in the next few days. >> > > This is very welcome. Once this is in place, it would be great to add > a new test to make sure things don't regress, and from where we can > iterate. > By self-contained within QEMU, do you mean a QTest based test? > > Regards, > Pierrick Thanks for the follow-up and the great questions. To answer your question about the custom test driver: yes, the current implementation of the smmuv3-test device relies on integration with Hafnium to be triggered. The test flow is initiated when Hafnium performs an mmio_write32 to the smmuv3-test device's MMIO space. This triggers the device's read/write callback in QEMU. Inside this callback, I use address_space_write/read to first populate the necessary SMMU structures (STEs, CDs, PTEs) in guest secure memory, and then perform another address_space_write/read to an IOMMU-protected region of smmuv3-test. It is this final access that exercises the full secure translation path. So for now, it is indeed dependent on the Hafnium firmware for debugging. That brings me to your next point about making the test "self-contained". My goal is exactly to remove this dependency. I'm currently exploring ways to achieve this within QEMU—for instance, by using QMP commands or another monitor interface to replace the mmio_write32 action that Hafnium currently provides. This is what I meant by "self-contained": allowing us to test the entire SMMUv3 translation flow relying only on the QEMU emulator itself, without any specific guest firmware or kernel. This leads perfectly to your question about QTest. I'm not very familiar with it, so your suggestion is very helpful. Is QTest the standard or required framework for implementing this kind of self-contained test device? I would appreciate any guidance or pointers you could provide on this topic. Thanks again for the valuable discussion. Best regards, Tao
Hi Pierrick, On 2025/8/15 13:49, Tao Tang wrote: > > On 2025/8/13 02:04, Pierrick Bouvier wrote: >> On 8/10/25 9:11 AM, Tao Tang wrote: >>> >>> On 2025/8/7 05:28, Pierrick Bouvier wrote: >>>> On 8/6/25 8:11 AM, Tao Tang wrote: >>>>> Hi all, >>>>> >>>>> This patch series introduces initial support for emulating the Arm >>>>> SMMUv3 >>>>> Secure State. >>>>> ------------------------------<snip>------------------------------ >>>>> >>>>> >>>>> ------------------------------<snip>------------------------------ >>>>> Secure Register/Command Interface: I set up an OP-TEE + Hafnium >>>>> environment. Hafnium's smmuv3_driver_init function was used to test >>>>> the secure register I/O and command queue functionality (excluding >>>>> translation). As Hafnium assumes larger queue and StreamID sizes than >>>>> are practical without TTST support, I temporarily patched Hafnium to >>>>> use smaller values, allowing its driver to initialize the emulated >>>>> secure SMMU successfully. >>>>> >>>> >>>> Would that be possible to share your changes, and build instructions >>>> for this? While working on SMMU emulation, we finally left this on the >>>> side due to lack of a software stack being able to use secure SMMU, as >>>> we were not aware that Hafnium + op-tee could make use of it. >>>> >>> Hi Pierrick, >>> >>> Thanks for your interest! I'm very happy to share my work on this. I've >>> documented the setup process, including our code modifications and the >>> step-by-step build instructions in this link: >>> >>> https://hnusdr.github.io/2025/08/09/Test-Secure-SMMU-with-Hafnium-ENG >>> >> >> Thanks for taking the time to assemble all this in a comprehensible >> post, I'll give it a try when I have some spare time. > > Hi Pierrick, > > You're welcome, and please feel free to let me know if you run into > any issues. > > >> >>> >>> The core point of these changes is to enable the SMMUv3 feature in >>> Hafnium. This leads to numerous read/write operations on SMMUv3 secure >>> registers and various queue manipulations within the smmuv3_driver_init >>> function in Hafnium. >>> >>> However, it's important to note that this initialization process itself >>> does not initiate any DMA memory access that would trigger the >>> smmuv3_translate flow. >>> >> >> I understand the difference. It can be tricky to generate specific >> translation scenarios, which is where a custom test device can really >> help. >> >>> Even so, we've devised a method to test the subsequent Secure >>> Translation Path by leveraging the smmuv3-test platform device. This >>> approach allows us to verify the entire SMMUv3 flow, from >>> initialization >>> to translation. >>> >> >> Does it rely on a custom driver integration into an existing firmware >> or the kernel? >> >>> >>>>> Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not >>>>> support QEMU, and no secure device assignment feature exists yet, I >>>>> created a custom platform device to test the secure translation flow. >>>>> To trigger the translation logic, I initiated MMIO writes to this >>>>> device from within Hafnium. The device's MMIO callback handler then >>>>> performed DMA accesses via its IOMMU region, exercising the secure >>>>> translation path. While SMMUv3 is typically used for PCIe on >>>>> physical SoCs, the architecture allows its use with platform devices >>>>> via a stream-id binding in the device tree. The test harness >>>>> required some non-standard modifications to decouple the SMMU from >>>>> its tight integration with PCIe. The code for this test device is >>>>> available for review at [3]. README.md with detailed instructions is >>>>> also provided. >>>>> >>>> >>>> I am not sure about the current policy in QEMU for test oriented >>>> devices, but it would be really useful to have something similar >>>> upstream (Note: it's out of the scope of this series). >>>> One challenge working with SMMU emulation is that reproducing setups >>>> and triggering specific code paths is hard to achieve, due to the >>>> indirect use of SMMU feature (through DMA) and the complex software >>>> stack usually involved. >>>> Having something upstream available to work on SMMU emulation, at >>>> least on device side, would be a great addition. >>>> >>>> Eric, Peter, is this something that would be acceptable to merge? >>>> >>> >>> Looking ahead, my plan is to refactor the smmuv3-test platform device. >>> The goal is to make it self-contained within QEMU, removing the current >>> dependency on Hafnium to trigger its operations. I plan to submit this >>> as a separate RFC patch series in the next few days. >>> >> >> This is very welcome. Once this is in place, it would be great to add >> a new test to make sure things don't regress, and from where we can >> iterate. >> By self-contained within QEMU, do you mean a QTest based test? >> >> Regards, >> Pierrick > > > Thanks for the follow-up and the great questions. > > To answer your question about the custom test driver: yes, the current > implementation of the smmuv3-test device relies on integration with > Hafnium to be triggered. > > The test flow is initiated when Hafnium performs an mmio_write32 to > the smmuv3-test device's MMIO space. This triggers the device's > read/write callback in QEMU. Inside this callback, I use > address_space_write/read to first populate the necessary SMMU > structures (STEs, CDs, PTEs) in guest secure memory, and then perform > another address_space_write/read to an IOMMU-protected region of > smmuv3-test. It is this final access that exercises the full secure > translation path. So for now, it is indeed dependent on the Hafnium > firmware for debugging. > > That brings me to your next point about making the test > "self-contained". My goal is exactly to remove this dependency. I'm > currently exploring ways to achieve this within QEMU—for instance, by > using QMP commands or another monitor interface to replace the > mmio_write32 action that Hafnium currently provides. This is what I > meant by "self-contained": allowing us to test the entire SMMUv3 > translation flow relying only on the QEMU emulator itself, without any > specific guest firmware or kernel. > > This leads perfectly to your question about QTest. I'm not very > familiar with it, so your suggestion is very helpful. Is QTest the > standard or required framework for implementing this kind of > self-contained test device? I would appreciate any guidance or > pointers you could provide on this topic. Hi Pierrick, I've just submitted a new SMMU test device that enables qtest-based testing without needing to boot any kernel or firmware. You can find the patch here: https://lists.nongnu.org/archive/html/qemu-devel/2025-09/msg05366.html I wanted to send a special thank you for your suggestion to use qtest. It was a pivotal piece of advice that unblocked this entire effort. To give you some context, my initial approach was much more complex. Based on the method we discussed earlier ([https://github.com/hnusdr/qemu/tree/smmuv3-test-device]), I had added a new memory map in virt.c and heavily modified the SMMU code to create a non-standard platform device. This setup was coupled with the Hafnium firmware for testing, which was clearly not a sustainable long-term solution. My next idea was to create a self-contained device using QMP to make test easier. It was at that point you suggested I look into qtest, and that's when I realized qtest already works on the exact QMP mechanism I was considering. That was the missing piece for me, and it directly led to this new test device patch. If you or others have a moment, I would greatly appreciate any review or feedback on whether this test device is a suitable approach for our testing needs. Thanks again for your invaluable guidance. Best, Tao
On 10/8/25 18:11, Tao Tang wrote: > > On 2025/8/7 05:28, Pierrick Bouvier wrote: >> On 8/6/25 8:11 AM, Tao Tang wrote: >>> Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not >>> support QEMU, and no secure device assignment feature exists yet, I >>> created a custom platform device to test the secure translation flow. >>> To trigger the translation logic, I initiated MMIO writes to this >>> device from within Hafnium. The device's MMIO callback handler then >>> performed DMA accesses via its IOMMU region, exercising the secure >>> translation path. While SMMUv3 is typically used for PCIe on >>> physical SoCs, the architecture allows its use with platform devices >>> via a stream-id binding in the device tree. The test harness >>> required some non-standard modifications to decouple the SMMU from >>> its tight integration with PCIe. The code for this test device is >>> available for review at [3]. README.md with detailed instructions is >>> also provided. >>> >> >> I am not sure about the current policy in QEMU for test oriented >> devices, but it would be really useful to have something similar >> upstream (Note: it's out of the scope of this series). >> One challenge working with SMMU emulation is that reproducing setups >> and triggering specific code paths is hard to achieve, due to the >> indirect use of SMMU feature (through DMA) and the complex software >> stack usually involved. >> Having something upstream available to work on SMMU emulation, at >> least on device side, would be a great addition. >> >> Eric, Peter, is this something that would be acceptable to merge? This shouldn't be an issue, we already have some: $ git ls-files|fgrep testdev chardev/testdev.c docs/specs/pci-testdev.rst hw/hyperv/hyperv_testdev.c hw/misc/pc-testdev.c hw/misc/pci-testdev.c hw/tricore/tricore_testdevice.c > Looking ahead, my plan is to refactor the smmuv3-test platform device. > The goal is to make it self-contained within QEMU, removing the current > dependency on Hafnium to trigger its operations. I plan to submit this > as a separate RFC patch series in the next few days.
On 8/11/25 3:26 AM, Philippe Mathieu-Daudé wrote: > On 10/8/25 18:11, Tao Tang wrote: >> >> On 2025/8/7 05:28, Pierrick Bouvier wrote: >>> On 8/6/25 8:11 AM, Tao Tang wrote: > > >>>> Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not >>>> support QEMU, and no secure device assignment feature exists yet, I >>>> created a custom platform device to test the secure translation flow. >>>> To trigger the translation logic, I initiated MMIO writes to this >>>> device from within Hafnium. The device's MMIO callback handler then >>>> performed DMA accesses via its IOMMU region, exercising the secure >>>> translation path. While SMMUv3 is typically used for PCIe on >>>> physical SoCs, the architecture allows its use with platform devices >>>> via a stream-id binding in the device tree. The test harness >>>> required some non-standard modifications to decouple the SMMU from >>>> its tight integration with PCIe. The code for this test device is >>>> available for review at [3]. README.md with detailed instructions is >>>> also provided. >>>> >>> >>> I am not sure about the current policy in QEMU for test oriented >>> devices, but it would be really useful to have something similar >>> upstream (Note: it's out of the scope of this series). >>> One challenge working with SMMU emulation is that reproducing setups >>> and triggering specific code paths is hard to achieve, due to the >>> indirect use of SMMU feature (through DMA) and the complex software >>> stack usually involved. >>> Having something upstream available to work on SMMU emulation, at >>> least on device side, would be a great addition. >>> >>> Eric, Peter, is this something that would be acceptable to merge? > > > This shouldn't be an issue, we already have some: > > $ git ls-files|fgrep testdev > chardev/testdev.c > docs/specs/pci-testdev.rst > hw/hyperv/hyperv_testdev.c > hw/misc/pc-testdev.c > hw/misc/pci-testdev.c > hw/tricore/tricore_testdevice.c > Looks good indeed, and we have the TEST_DEVICES category for that. > >> Looking ahead, my plan is to refactor the smmuv3-test platform device. >> The goal is to make it self-contained within QEMU, removing the current >> dependency on Hafnium to trigger its operations. I plan to submit this >> as a separate RFC patch series in the next few days. >
Hi Tao, sharing this with Jean-Philippe and Mostafa, who might be interested to review. On 8/6/25 8:11 AM, Tao Tang wrote: > Hi all, > > This patch series introduces initial support for emulating the Arm SMMUv3 > Secure State. > > As Pierrick pointed out in a previous discussion [1], full Secure SMMUv3 > emulation is a notable missing piece in QEMU. While the FVP model has > some support, its limited PCIe capabilities make it challenging for > complex use cases. The ability to properly manage device DMA from a > secure context is a critical prerequisite for enabling device assignment > (passthrough) for confidential computing solutions like Arm CCA and > related research such as virtCCA [2]. This series aims to build that > foundational support in QEMU. > > This work is being proposed as an RFC. It introduces a significant amount > of new logic, including the core concept of modeling parallel secure and > non-secure contexts within a single SMMUv3 device. I am seeking feedback > on the overall approach, the core refactoring, and the implementation > details before proceeding further. > > The series begins by implementing the components of the secure programming > interface, then progressively refactors the core SMMU logic to handle > secure and non-secure contexts in parallel. > > Secure Interface Implementation: The initial patches add the > secure-side registers, implement their read/write logic, and enable > the secure command and event queues. This includes the S_INIT > mechanism and the new secure TLB invalidation commands. > > Core Logic Refactoring: The next set of patches makes the core SMMU > functions security-state aware. This involves plumbing an is_secure > context flag through the main code paths and adding logic to route > SMMU-originated memory accesses to the correct (Secure or Non-secure) > address space. > > Cache Isolation: With the core logic now aware of security states, > the following patches refactor the configuration and translation > lookup caches. The cache keys are modified to include the security > context, ensuring that secure and non-secure entries for the same > device or address are properly isolated and preventing aliasing. > > Framework Integration: The final patch connects the SMMU's internal > security context to the generic QEMU IOMMU framework by using the > iommu_index to represent the architectural SEC_SID. > > To validate this work, I performed the following tests: > > Non-Secure Regression: To ensure that existing functionality remains > intact, I ran a nested virtualization test. A TCG guest was created on > the host, with iommu=smmuv3 and with an emulated PCIe NVMe device assigned. > Command line of TCG VM is below: > > qemu-system-aarch64 \ > -machine virt,virtualization=on,gic-version=3,iommu=smmuv3 \ > -cpu max -smp 1 -m 4080M \ > -accel tcg,thread=single,tb-size=512 \ > -kernel Image \ > -append 'nokaslr root=/dev/vda rw rootfstype=ext4 iommu.passthrough=on' \ > -device pcie-root-port,bus=pcie.0,id=rp0,addr=0x4.0,chassis=1,port=0x10 \ > -device pcie-root-port,bus=pcie.0,id=rp1,addr=0x5.0,chassis=2,port=0x11 \ > -drive if=none,file=u2204fs.img.qcow2,format=qcow2,id=hd0 \ > -device virtio-blk-device,drive=hd0 \ > -qmp unix:/tmp/qmp-sock12,server=on,wait=off \ > -netdev user,id=eth0,hostfwd=tcp::10022-:22,hostfwd=tcp::59922-:5922 \ > -device virtio-net-device,netdev=eth0 \ > -drive if=none,file=nvme.img,format=raw,id=nvme0 \ > -device nvme,drive=nvme0,serial=deadbeef \ > -d unimp,guest_errors -trace events=smmu-events.txt -D qemu.log -nographic > > Inside this TCG VM, a KVM guest was launched, and the same NVMe device was > re-assigned to it via VFIO. > Command line of KVM VM inside TCG VM is below: > > sudo qemu-system-aarch64 \ > -enable-kvm -m 1024 -cpu host -M virt \ > -machine virt,gic-version=3 \ > -cpu max -append "nokaslr" -smp 1 \ > -monitor stdio \ > -kernel 5.15.Image \ > -initrd rootfs.cpio.gz \ > -display vnc=:22,id=primary \ > -device vfio-pci,host=00:01.0 > > The KVM guest was able to perform I/O on the device > correctly, confirming that the non-secure path is not broken. > > Secure Register/Command Interface: I set up an OP-TEE + Hafnium > environment. Hafnium's smmuv3_driver_init function was used to test > the secure register I/O and command queue functionality (excluding > translation). As Hafnium assumes larger queue and StreamID sizes than > are practical without TTST support, I temporarily patched Hafnium to > use smaller values, allowing its driver to initialize the emulated > secure SMMU successfully. > > Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not > support QEMU, and no secure device assignment feature exists yet, I > created a custom platform device to test the secure translation flow. > To trigger the translation logic, I initiated MMIO writes to this > device from within Hafnium. The device's MMIO callback handler then > performed DMA accesses via its IOMMU region, exercising the secure > translation path. While SMMUv3 is typically used for PCIe on > physical SoCs, the architecture allows its use with platform devices > via a stream-id binding in the device tree. The test harness > required some non-standard modifications to decouple the SMMU from > its tight integration with PCIe. The code for this test device is > available for review at [3]. README.md with detailed instructions is > also provided. > > I've attempted to follow all of the guidance in the "Submitting a Patch" > guide, but as this is my first series of this scale, I apologize if I > missed anything and welcome all feedback. > > Thanks, > Tang > > [1] https://lists.nongnu.org/archive/html/qemu-devel/2025-06/msg02940.html > [2] https://arxiv.org/abs/2306.11011 > [3] https://github.com/hnusdr/qemu > > Tao Tang (11): > hw/arm/smmuv3: Introduce secure registers and commands > hw/arm/smmuv3: Implement read/write logic for secure registers > hw/arm/smmuv3: Implement S_INIT for secure initialization > hw/arm/smmuv3: Enable command processing for the Secure state > hw/arm/smmuv3: Support secure event queue and error handling > hw/arm/smmuv3: Plumb security state through core functions > hw/arm/smmuv3: Add separate address space for secure SMMU accesses > hw/arm/smmuv3: Enable secure-side stage 2 TLB invalidations > hw/arm/smmuv3: Make the configuration cache security-state aware > hw/arm/smmuv3: Differentiate secure TLB entries via keying > hw/arm/smmuv3: Use iommu_index to represent SEC_SID > > hw/arm/smmu-common.c | 74 ++- > hw/arm/smmuv3-internal.h | 128 +++++- > hw/arm/smmuv3.c | 844 ++++++++++++++++++++++++++++++----- > hw/arm/trace-events | 7 +- > hw/arm/virt.c | 5 + > include/hw/arm/smmu-common.h | 23 +- > include/hw/arm/smmuv3.h | 27 ++ > 7 files changed, 968 insertions(+), 140 deletions(-) > > -- > 2.34.1 > >
On Wed, Aug 06, 2025 at 11:11:23PM +0800, Tao Tang wrote: > Hi all, > > This patch series introduces initial support for emulating the Arm SMMUv3 > Secure State. > > As Pierrick pointed out in a previous discussion [1], full Secure SMMUv3 > emulation is a notable missing piece in QEMU. While the FVP model has > some support, its limited PCIe capabilities make it challenging for > complex use cases. The ability to properly manage device DMA from a > secure context is a critical prerequisite for enabling device assignment > (passthrough) for confidential computing solutions like Arm CCA and > related research such as virtCCA [2]. This series aims to build that > foundational support in QEMU. > > This work is being proposed as an RFC. It introduces a significant amount > of new logic, including the core concept of modeling parallel secure and > non-secure contexts within a single SMMUv3 device. I am seeking feedback > on the overall approach, the core refactoring, and the implementation > details before proceeding further. > > The series begins by implementing the components of the secure programming > interface, then progressively refactors the core SMMU logic to handle > secure and non-secure contexts in parallel. > > Secure Interface Implementation: The initial patches add the > secure-side registers, implement their read/write logic, and enable > the secure command and event queues. This includes the S_INIT > mechanism and the new secure TLB invalidation commands. > > Core Logic Refactoring: The next set of patches makes the core SMMU > functions security-state aware. This involves plumbing an is_secure > context flag through the main code paths and adding logic to route > SMMU-originated memory accesses to the correct (Secure or Non-secure) > address space. > > Cache Isolation: With the core logic now aware of security states, > the following patches refactor the configuration and translation > lookup caches. The cache keys are modified to include the security > context, ensuring that secure and non-secure entries for the same > device or address are properly isolated and preventing aliasing. > > Framework Integration: The final patch connects the SMMU's internal > security context to the generic QEMU IOMMU framework by using the > iommu_index to represent the architectural SEC_SID. > > To validate this work, I performed the following tests: > > Non-Secure Regression: To ensure that existing functionality remains > intact, I ran a nested virtualization test. A TCG guest was created on > the host, with iommu=smmuv3 and with an emulated PCIe NVMe device assigned. > Command line of TCG VM is below: > > qemu-system-aarch64 \ > -machine virt,virtualization=on,gic-version=3,iommu=smmuv3 \ > -cpu max -smp 1 -m 4080M \ > -accel tcg,thread=single,tb-size=512 \ > -kernel Image \ > -append 'nokaslr root=/dev/vda rw rootfstype=ext4 iommu.passthrough=on' \ > -device pcie-root-port,bus=pcie.0,id=rp0,addr=0x4.0,chassis=1,port=0x10 \ > -device pcie-root-port,bus=pcie.0,id=rp1,addr=0x5.0,chassis=2,port=0x11 \ > -drive if=none,file=u2204fs.img.qcow2,format=qcow2,id=hd0 \ > -device virtio-blk-device,drive=hd0 \ > -qmp unix:/tmp/qmp-sock12,server=on,wait=off \ > -netdev user,id=eth0,hostfwd=tcp::10022-:22,hostfwd=tcp::59922-:5922 \ > -device virtio-net-device,netdev=eth0 \ > -drive if=none,file=nvme.img,format=raw,id=nvme0 \ > -device nvme,drive=nvme0,serial=deadbeef \ > -d unimp,guest_errors -trace events=smmu-events.txt -D qemu.log -nographic > > Inside this TCG VM, a KVM guest was launched, and the same NVMe device was > re-assigned to it via VFIO. > Command line of KVM VM inside TCG VM is below: > > sudo qemu-system-aarch64 \ > -enable-kvm -m 1024 -cpu host -M virt \ > -machine virt,gic-version=3 \ > -cpu max -append "nokaslr" -smp 1 \ > -monitor stdio \ > -kernel 5.15.Image \ > -initrd rootfs.cpio.gz \ > -display vnc=:22,id=primary \ > -device vfio-pci,host=00:01.0 > > The KVM guest was able to perform I/O on the device > correctly, confirming that the non-secure path is not broken. I gave the patches a quick test and they seem to have broken my nested setup, I will look more into it and let you know what I find. Thanks, Mostafa > > Secure Register/Command Interface: I set up an OP-TEE + Hafnium > environment. Hafnium's smmuv3_driver_init function was used to test > the secure register I/O and command queue functionality (excluding > translation). As Hafnium assumes larger queue and StreamID sizes than > are practical without TTST support, I temporarily patched Hafnium to > use smaller values, allowing its driver to initialize the emulated > secure SMMU successfully. > > Secure Translation Path: Since the TF-A SMMUv3 Test Engine does not > support QEMU, and no secure device assignment feature exists yet, I > created a custom platform device to test the secure translation flow. > To trigger the translation logic, I initiated MMIO writes to this > device from within Hafnium. The device's MMIO callback handler then > performed DMA accesses via its IOMMU region, exercising the secure > translation path. While SMMUv3 is typically used for PCIe on > physical SoCs, the architecture allows its use with platform devices > via a stream-id binding in the device tree. The test harness > required some non-standard modifications to decouple the SMMU from > its tight integration with PCIe. The code for this test device is > available for review at [3]. README.md with detailed instructions is > also provided. > > I've attempted to follow all of the guidance in the "Submitting a Patch" > guide, but as this is my first series of this scale, I apologize if I > missed anything and welcome all feedback. > > Thanks, > Tang > > [1] https://lists.nongnu.org/archive/html/qemu-devel/2025-06/msg02940.html > [2] https://arxiv.org/abs/2306.11011 > [3] https://github.com/hnusdr/qemu > > Tao Tang (11): > hw/arm/smmuv3: Introduce secure registers and commands > hw/arm/smmuv3: Implement read/write logic for secure registers > hw/arm/smmuv3: Implement S_INIT for secure initialization > hw/arm/smmuv3: Enable command processing for the Secure state > hw/arm/smmuv3: Support secure event queue and error handling > hw/arm/smmuv3: Plumb security state through core functions > hw/arm/smmuv3: Add separate address space for secure SMMU accesses > hw/arm/smmuv3: Enable secure-side stage 2 TLB invalidations > hw/arm/smmuv3: Make the configuration cache security-state aware > hw/arm/smmuv3: Differentiate secure TLB entries via keying > hw/arm/smmuv3: Use iommu_index to represent SEC_SID > > hw/arm/smmu-common.c | 74 ++- > hw/arm/smmuv3-internal.h | 128 +++++- > hw/arm/smmuv3.c | 844 ++++++++++++++++++++++++++++++----- > hw/arm/trace-events | 7 +- > hw/arm/virt.c | 5 + > include/hw/arm/smmu-common.h | 23 +- > include/hw/arm/smmuv3.h | 27 ++ > 7 files changed, 968 insertions(+), 140 deletions(-) > > -- > 2.34.1 > >
© 2016 - 2025 Red Hat, Inc.