target/arm/cpu-features.h | 15 ++ target/arm/cpu.h | 27 ++++ target/arm/internals.h | 23 +++ target/arm/cpu.c | 9 ++ target/arm/helper.c | 259 +++++++++++++++++++++++++++++++++- target/arm/tcg/cpu64.c | 7 +- docs/system/arm/emulation.rst | 5 + 7 files changed, 339 insertions(+), 6 deletions(-)
Changes for v8: - Re-order SCTLR2 and TCR2 so that they are independent of MEC. - Enable the SCTLR2 and TCR2 enable bits. - Squash 3 smaller MEC patches together. This still fails the RME tests, because we still need TF-A rebuilt with ENABLE_FEAT_SCTLR2 and ENABLE_FEAT_TCR2. Pierrick, since you have just done such a build, could you re-test with this series? r~ Gustavo Romero (4): target/arm: Implement FEAT_SCTLR2 and enable with -cpu max target/arm: Implement FEAT_TCR2 and enable with -cpu max target/arm: Implement FEAT_MEC registers target/arm: Enable FEAT_MEC in -cpu max target/arm/cpu-features.h | 15 ++ target/arm/cpu.h | 27 ++++ target/arm/internals.h | 23 +++ target/arm/cpu.c | 9 ++ target/arm/helper.c | 259 +++++++++++++++++++++++++++++++++- target/arm/tcg/cpu64.c | 7 +- docs/system/arm/emulation.rst | 5 + 7 files changed, 339 insertions(+), 6 deletions(-) -- 2.43.0
On 7/14/25 8:58 AM, Richard Henderson wrote: > Changes for v8: > - Re-order SCTLR2 and TCR2 so that they are independent of MEC. > - Enable the SCTLR2 and TCR2 enable bits. > - Squash 3 smaller MEC patches together. > > This still fails the RME tests, because we still need TF-A rebuilt > with ENABLE_FEAT_SCTLR2 and ENABLE_FEAT_TCR2. Pierrick, since you > have just done such a build, could you re-test with this series? > I tested that on my local Realm enabled setup and I can confirm this solved the issue and current series works. Both flags are needed in TF-A. ENABLE_FEAT_TCR2 is needed to boot host, and ENABLE_FEAT_SCTLR2 is needed to boot nested guest. As I'm off today, I'll update that properly tomorrow when I have time, and not rush things. I'll update RME images for sbsa and virt tests + device passthrough test, and post associated patches. Thanks, Pierrick
On 7/14/25 10:09 AM, Pierrick Bouvier wrote: > On 7/14/25 8:58 AM, Richard Henderson wrote: >> Changes for v8: >> - Re-order SCTLR2 and TCR2 so that they are independent of MEC. >> - Enable the SCTLR2 and TCR2 enable bits. >> - Squash 3 smaller MEC patches together. >> >> This still fails the RME tests, because we still need TF-A rebuilt >> with ENABLE_FEAT_SCTLR2 and ENABLE_FEAT_TCR2. Pierrick, since you >> have just done such a build, could you re-test with this series? >> > > I tested that on my local Realm enabled setup and I can confirm this > solved the issue and current series works. > Both flags are needed in TF-A. ENABLE_FEAT_TCR2 is needed to boot host, > and ENABLE_FEAT_SCTLR2 is needed to boot nested guest. > > As I'm off today, I'll update that properly tomorrow when I have time, > and not rush things. I'll update RME images for sbsa and virt tests + > device passthrough test, and post associated patches. > Richard, patches updating tests images have been sent [1]. I tested them with and without current series. You're welcome to merge that in current series (preferably before the current series, to not break bisection for concerned tests). [1] https://lore.kernel.org/qemu-devel/20250715212335.2215509-1-pierrick.bouvier@linaro.org/T/#t Regards, Pierrick
Hi folks, Richard, thanks for v8. Pierrick, thanks for testing it. :) On 7/14/25 14:09, Pierrick Bouvier wrote: > On 7/14/25 8:58 AM, Richard Henderson wrote: >> Changes for v8: >> - Re-order SCTLR2 and TCR2 so that they are independent of MEC. >> - Enable the SCTLR2 and TCR2 enable bits. >> - Squash 3 smaller MEC patches together. >> >> This still fails the RME tests, because we still need TF-A rebuilt >> with ENABLE_FEAT_SCTLR2 and ENABLE_FEAT_TCR2. Pierrick, since you >> have just done such a build, could you re-test with this series? >> > > I tested that on my local Realm enabled setup and I can confirm this solved the issue and current series works. > Both flags are needed in TF-A. ENABLE_FEAT_TCR2 is needed to boot host, and ENABLE_FEAT_SCTLR2 is needed to boot nested guest. I'm a bit confused because the QEMU RME tests, afaics, uses OP-TEE, not TF-A. I've built TF-A using the scripts in [0], enabling ENABLE_FEAT_TCR2 and ENABLE_FEAT_SCTLR2, but no way to get it booting. I understand we can embed a OP-TEE into the TF_A via BL32=<optee_image> when building TF-A. Is that what you're using? Thanks. Cheers, Gustavo [0] https://github.com/pbo-linaro/qemu-linux-stack.git > As I'm off today, I'll update that properly tomorrow when I have time, and not rush things. I'll update RME images for sbsa and virt tests + device passthrough test, and post associated patches. > > Thanks, > Pierrick
On 7/14/25 4:31 PM, Gustavo Romero wrote: > Hi folks, > > Richard, thanks for v8. Pierrick, thanks for testing it. :) > > On 7/14/25 14:09, Pierrick Bouvier wrote: >> On 7/14/25 8:58 AM, Richard Henderson wrote: >>> Changes for v8: >>> - Re-order SCTLR2 and TCR2 so that they are independent of MEC. >>> - Enable the SCTLR2 and TCR2 enable bits. >>> - Squash 3 smaller MEC patches together. >>> >>> This still fails the RME tests, because we still need TF-A rebuilt >>> with ENABLE_FEAT_SCTLR2 and ENABLE_FEAT_TCR2. Pierrick, since you >>> have just done such a build, could you re-test with this series? >>> >> >> I tested that on my local Realm enabled setup and I can confirm this solved the issue and current series works. >> Both flags are needed in TF-A. ENABLE_FEAT_TCR2 is needed to boot host, and ENABLE_FEAT_SCTLR2 is needed to boot nested guest. > > I'm a bit confused because the QEMU RME tests, afaics, uses OP-TEE, not TF-A. I've built TF-A > using the scripts in [0], enabling ENABLE_FEAT_TCR2 and ENABLE_FEAT_SCTLR2, but no way to get > it booting. I understand we can embed a OP-TEE into the TF_A via BL32=<optee_image> when > building TF-A. Is that what you're using? > I agree it's confusing. In short, no, OP-TEE is not used anywhere for Realms, only TF-A and RMM are used in our images. It seems that OP-TEE is a term used generically to represent any firmware running in secure mode, but it's a *totally* different software than TF-A + RMM. Naming OP-TEE like this is like if Linux would have been named "OP-kernel". The RME tests we have are based on this excellent tutorial [0], and build is automated with 'qemu-rme-stack' [1], that simply follows those instructions. [0] https://linaro.atlassian.net/wiki/spaces/QEMU/pages/30128767027/Device+Assignment+Enabled+RME+Stack+on+QEMU [1] https://github.com/pbo-linaro/qemu-rme-stack To add to the confusion (*get ready*), [0] uses OP-TEE build, which is simply a build system for op-tee + other things, but in the variant it generates, OP-TEE itself is not included. Yes, that's utterly confusing and took me quite some time to discover it, after talking with Mathieu himself. They removed it explicitely to make the stack more simple. In short, for Realms, forget about OP-TEE. If you want to see it by yourself: https://git.codelinaro.org/linaro/dcap/op-tee-4.2.0/build/-/blob/cca/v8/qemu_v8_cca.mk?ref_type=heads#L172 ... TF_A_FLAGS ?= \ BL33=$(BL33_BIN) \ PLAT=qemu \ QEMU_USE_GIC_DRIVER=$(TFA_GIC_DRIVER) \ DEBUG=$(TF_A_DEBUG) \ LOG_LEVEL=$(TF_A_LOGLVL) \ ENABLE_RME=1 \ RMM=$(RMM_BIN) ... TF_A_FLAGS_BL32_OPTEE = BL32=$(OPTEE_OS_HEADER_V2_BIN) TF_A_FLAGS_BL32_OPTEE += BL32_EXTRA1=$(OPTEE_OS_PAGER_V2_BIN) TF_A_FLAGS_BL32_OPTEE += BL32_EXTRA2=$(OPTEE_OS_PAGEABLE_V2_BIN) TF_A_FLAGS_SPMC_AT_EL_n = $(TF_A_FLAGS_BL32_OPTEE) SPD=opteed ... #TF_A_FLAGS += $(TF_A_FLAGS_SPMC_AT_EL_$(SPMC_AT_EL)) The last line shows that OP_TEE flags are **NOT** added to TF_A_FLAGS build flags. The qemu_v8_cca.mk build file was copied from qemu_v8.mk, which itself has the OP-TEE inclusion. --- Recently, I had to generate a custom rootfs, and I experimented generating it directly from docker images, to avoid rebuilding the world using Buildroot. Once it worked, I realized it was a good opportunity to rebuild the rest of the stack too. The result is 'qemu-linux-stack' [2]. Master branch has only tf-a + uboot (no Realm support), while rme branch [3], which supports Realm, uses tf-a + rmm + edk2 instead. I removed u-boot as I couldn't get it to boot, and I knew that edk2 worked. One branch is only one configuration (and it will stay this way). [2] https://github.com/pbo-linaro/qemu-linux-stack [3] https://github.com/pbo-linaro/qemu-linux-stack/tree/rme Beyond the personal knowledge I got through that, I hope it can be used for others for who it's confusing about what runs before start_kernel, and I guess I'm not the only one who didn't know about that. In the end, things are not too complicated, but as most of the build systems out there (OP-TEE build, shrinkwrap, ...) try to be "generic and versatile", it ends up being complicated. I prefer basic and straightforward script shells to lenghty documentation and wiki pages, but it's a personal choice. --- Now, coming to the change introduced by this series, and supporting FEAT_SCTRL2 and FEAT_TCR2, all those images need to be updated [0], [1], [2], [3], because essentially, TF-A itself must be patched to support this. I was about to mention that, and mention that I can send a PR directly to it once we have this merged on QEMU side. The change in arm-trusted-firmware is quite simple: +diff --git a/plat/qemu/common/common.mk b/plat/qemu/common/common.mk +index 751511cf8..6bc108492 100644 +--- a/plat/qemu/common/common.mk ++++ b/plat/qemu/common/common.mk +@@ -122,6 +122,10 @@ ENABLE_FEAT_FGT := 2 + # 8.7 + ENABLE_FEAT_HCX := 2 + ++# 8.8 ++ENABLE_FEAT_TCR2 := 2 ++ENABLE_FEAT_SCTLR2 := 2 ++ I'll push all that tomorrow on master, and rebase rmm and device_passthrough on top of it. I'll add this to original RME images, it's less direct though, as it's needed to update the tf-a fork, and then the optee build system. Sorry for this very long email. At least, all is there. > Thanks. > > > Cheers, > Gustavo > > [0] https://github.com/pbo-linaro/qemu-linux-stack.git > >> As I'm off today, I'll update that properly tomorrow when I have time, and not rush things. I'll update RME images for sbsa and virt tests + device passthrough test, and post associated patches. >> >> Thanks, >> Pierrick >
Hi Pierrick,
On 7/14/25 22:26, Pierrick Bouvier wrote:
> On 7/14/25 4:31 PM, Gustavo Romero wrote:
>> Hi folks,
>>
>> Richard, thanks for v8. Pierrick, thanks for testing it. :)
>>
>> On 7/14/25 14:09, Pierrick Bouvier wrote:
>>> On 7/14/25 8:58 AM, Richard Henderson wrote:
>>>> Changes for v8:
>>>> - Re-order SCTLR2 and TCR2 so that they are independent of MEC.
>>>> - Enable the SCTLR2 and TCR2 enable bits.
>>>> - Squash 3 smaller MEC patches together.
>>>>
>>>> This still fails the RME tests, because we still need TF-A rebuilt
>>>> with ENABLE_FEAT_SCTLR2 and ENABLE_FEAT_TCR2. Pierrick, since you
>>>> have just done such a build, could you re-test with this series?
>>>>
>>>
>>> I tested that on my local Realm enabled setup and I can confirm this solved the issue and current series works.
>>> Both flags are needed in TF-A. ENABLE_FEAT_TCR2 is needed to boot host, and ENABLE_FEAT_SCTLR2 is needed to boot nested guest.
>>
>> I'm a bit confused because the QEMU RME tests, afaics, uses OP-TEE, not TF-A. I've built TF-A
>> using the scripts in [0], enabling ENABLE_FEAT_TCR2 and ENABLE_FEAT_SCTLR2, but no way to get
>> it booting. I understand we can embed a OP-TEE into the TF_A via BL32=<optee_image> when
>> building TF-A. Is that what you're using?
>>
>
> I agree it's confusing. In short, no, OP-TEE is not used anywhere for Realms, only TF-A and RMM are used in our images. It seems that OP-TEE is a term used generically to represent any firmware running in secure mode, but it's a *totally* different software than TF-A + RMM. Naming OP-TEE like this is like if Linux would have been named "OP-kernel".
Got it, so we use TF-A + RMM in the test images. Thanks for the clarifications.
Isn't the generic term (or concept) actually TEE (not OP-TEE) and OP-TEE is a real software stack that implements the TEE spec, i.e., OP-TEE is code in https://github.com/OP-TEE)?
> The RME tests we have are based on this excellent tutorial [0], and build is automated with 'qemu-rme-stack' [1], that simply follows those instructions.
>
> [0] https://linaro.atlassian.net/wiki/spaces/QEMU/pages/30128767027/Device+Assignment+Enabled+RME+Stack+on+QEMU
Should [0] be:
https://linaro.atlassian.net/wiki/spaces/QEMU/pages/29051027459/Building+an+RME+stack+for+QEMU ?
> [1] https://github.com/pbo-linaro/qemu-rme-stack
>
> To add to the confusion (*get ready*), [0] uses OP-TEE build, which is simply a build system for op-tee + other things, but in the variant it generates, OP-TEE itself is not included. Yes, that's utterly confusing and took me quite some time to discover it, after talking with Mathieu himself. They removed it explicitely to make the stack more simple. In short, for Realms, forget about OP-TEE.
ah, that's what confused me! I see now.
> If you want to see it by yourself:
>
> https://git.codelinaro.org/linaro/dcap/op-tee-4.2.0/build/-/blob/cca/v8/qemu_v8_cca.mk?ref_type=heads#L172
>
> ...
> TF_A_FLAGS ?= \
> BL33=$(BL33_BIN) \
> PLAT=qemu \
> QEMU_USE_GIC_DRIVER=$(TFA_GIC_DRIVER) \
> DEBUG=$(TF_A_DEBUG) \
> LOG_LEVEL=$(TF_A_LOGLVL) \
> ENABLE_RME=1 \
> RMM=$(RMM_BIN)
> ...
> TF_A_FLAGS_BL32_OPTEE = BL32=$(OPTEE_OS_HEADER_V2_BIN)
> TF_A_FLAGS_BL32_OPTEE += BL32_EXTRA1=$(OPTEE_OS_PAGER_V2_BIN)
> TF_A_FLAGS_BL32_OPTEE += BL32_EXTRA2=$(OPTEE_OS_PAGEABLE_V2_BIN)
> TF_A_FLAGS_SPMC_AT_EL_n = $(TF_A_FLAGS_BL32_OPTEE) SPD=opteed
> ...
> #TF_A_FLAGS += $(TF_A_FLAGS_SPMC_AT_EL_$(SPMC_AT_EL))
>
> The last line shows that OP_TEE flags are **NOT** added to TF_A_FLAGS build flags.
> The qemu_v8_cca.mk build file was copied from qemu_v8.mk, which itself has the OP-TEE inclusion.
>
> ---
>
> Recently, I had to generate a custom rootfs, and I experimented generating it directly from docker images, to avoid rebuilding the world using Buildroot. Once it worked, I realized it was a good opportunity to rebuild the rest of the stack too. The result is 'qemu-linux-stack' [2].
>
> Master branch has only tf-a + uboot (no Realm support), while rme branch [3], which supports Realm, uses tf-a + rmm + edk2 instead. I removed u-boot as I couldn't get it to boot, and I knew that edk2 worked.
>
> One branch is only one configuration (and it will stay this way).
>
> [2] https://github.com/pbo-linaro/qemu-linux-stack
> [3] https://github.com/pbo-linaro/qemu-linux-stack/tree/rme
>
> Beyond the personal knowledge I got through that, I hope it can be used for others for who it's confusing about what runs before start_kernel, and I guess I'm not the only one who didn't know about that. In the end, things are not too complicated, but as most of the build systems out there (OP-TEE build, shrinkwrap, ...) try to be "generic and versatile", it ends up being complicated. I prefer basic and straightforward script shells to lenghty documentation and wiki pages, but it's a personal choice.
>
> ---
>
> Now, coming to the change introduced by this series, and supporting FEAT_SCTRL2 and FEAT_TCR2, all those images need to be updated [0], [1], [2], [3], because essentially, TF-A itself must be patched to support this. I was about to mention that, and mention that I can send a PR directly to it once we have this merged on QEMU side.
>
> The change in arm-trusted-firmware is quite simple:
>
> +diff --git a/plat/qemu/common/common.mk b/plat/qemu/common/common.mk
> +index 751511cf8..6bc108492 100644
> +--- a/plat/qemu/common/common.mk
> ++++ b/plat/qemu/common/common.mk
> +@@ -122,6 +122,10 @@ ENABLE_FEAT_FGT := 2
> + # 8.7
> + ENABLE_FEAT_HCX := 2
> +
> ++# 8.8
> ++ENABLE_FEAT_TCR2 := 2
> ++ENABLE_FEAT_SCTLR2 := 2
> ++
>
> I'll push all that tomorrow on master, and rebase rmm and device_passthrough on top of it.
> I'll add this to original RME images, it's less direct though, as it's needed to update the tf-a fork, and then the optee build system.
So, I still fail to understand how you are generating the out/bin/flash.bin used in our images.
First I tried to use the https://github.com/pbo-linaro/qemu-rme-stack, since it's cited in the tests. I tried:
$ ./container.sh ./build_virt.sh
and get an error which seems a commit mismatch in some of branches used by repo:
repo: reusing existing repo client checkout in /mnt/git/qemu-rme-stack
Traceback (most recent call last):
File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 869, in <module>
_Main(sys.argv[1:])
File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 845, in _Main
result = repo._Run(name, gopts, argv) or 0
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 293, in _Run
result = run()
^^^^^
File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 274, in <lambda>
lambda: self._RunLong(name, gopts, argv, git_trace2_event_log) or 0
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 437, in _RunLong
execute_command()
File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 403, in execute_command
execute_command_helper()
File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 369, in execute_command_helper
result = cmd.Execute(copts, cargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/git/qemu-rme-stack/.repo/repo/subcmds/init.py", line 400, in Execute
self._SyncManifest(opt)
File "/mnt/git/qemu-rme-stack/.repo/repo/subcmds/init.py", line 146, in _SyncManifest
if not self.manifest.manifestProject.Sync(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/git/qemu-rme-stack/.repo/repo/project.py", line 4704, in Sync
self.MetaBranchSwitch(submodules=submodules, verbose=verbose)
File "/mnt/git/qemu-rme-stack/.repo/repo/project.py", line 4173, in MetaBranchSwitch
self.Sync_LocalHalf(syncbuf, submodules=submodules, verbose=verbose)
File "/mnt/git/qemu-rme-stack/.repo/repo/project.py", line 1636, in Sync_LocalHalf
lost = self._revlist(not_rev(revid), HEAD)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/git/qemu-rme-stack/.repo/repo/project.py", line 3676, in _revlist
return self.work_git.rev_list(*a, **kw)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/mnt/git/qemu-rme-stack/.repo/repo/project.py", line 3908, in rev_list
p.Wait()
File "/mnt/git/qemu-rme-stack/.repo/repo/git_command.py", line 556, in Wait
self.VerifyCommand()
File "/mnt/git/qemu-rme-stack/.repo/repo/git_command.py", line 546, in VerifyCommand
raise GitCommandError(
git_command.GitCommandError: GitCommandError: 'rev-list ^6fd1cc667671a12cfc8789a390c990446e621f8f HEAD --' on manifests failed
stderr: fatal: bad revision 'HEAD'
Then I tried to build it from https://github.com/pbo-linaro/qemu-linux-stack/tree/rme and out/flash.bin
simply doesn't boot the kernel...
> Sorry for this very long email. At least, all is there.
Nope, that's quite helpful. This firmware stack is a mess :)
Cheers,
Gustavo
On 7/15/25 8:13 PM, Gustavo Romero wrote: > Hi Pierrick, > > On 7/14/25 22:26, Pierrick Bouvier wrote: >> On 7/14/25 4:31 PM, Gustavo Romero wrote: >>> Hi folks, >>> >>> Richard, thanks for v8. Pierrick, thanks for testing it. :) >>> >>> On 7/14/25 14:09, Pierrick Bouvier wrote: >>>> On 7/14/25 8:58 AM, Richard Henderson wrote: >>>>> Changes for v8: >>>>> - Re-order SCTLR2 and TCR2 so that they are independent of MEC. >>>>> - Enable the SCTLR2 and TCR2 enable bits. >>>>> - Squash 3 smaller MEC patches together. >>>>> >>>>> This still fails the RME tests, because we still need TF-A rebuilt >>>>> with ENABLE_FEAT_SCTLR2 and ENABLE_FEAT_TCR2. Pierrick, since you >>>>> have just done such a build, could you re-test with this series? >>>>> >>>> >>>> I tested that on my local Realm enabled setup and I can confirm this solved the issue and current series works. >>>> Both flags are needed in TF-A. ENABLE_FEAT_TCR2 is needed to boot host, and ENABLE_FEAT_SCTLR2 is needed to boot nested guest. >>> >>> I'm a bit confused because the QEMU RME tests, afaics, uses OP-TEE, not TF-A. I've built TF-A >>> using the scripts in [0], enabling ENABLE_FEAT_TCR2 and ENABLE_FEAT_SCTLR2, but no way to get >>> it booting. I understand we can embed a OP-TEE into the TF_A via BL32=<optee_image> when >>> building TF-A. Is that what you're using? >>> >> >> I agree it's confusing. In short, no, OP-TEE is not used anywhere for Realms, only TF-A and RMM are used in our images. It seems that OP-TEE is a term used generically to represent any firmware running in secure mode, but it's a *totally* different software than TF-A + RMM. Naming OP-TEE like this is like if Linux would have been named "OP-kernel". > > Got it, so we use TF-A + RMM in the test images. Thanks for the clarifications. > > Isn't the generic term (or concept) actually TEE (not OP-TEE) and OP-TEE is a real software stack that implements the TEE spec, i.e., OP-TEE is code in https://github.com/OP-TEE)? > Yes, correct. The term is TEE, OP-TEE is the project you mentioned. For some reason, everybody seems to think that OP-TEE is *the* unique TEE implementation is Arm world. Thus my analogy with Linux that would be named OP-Kernel. > >> The RME tests we have are based on this excellent tutorial [0], and build is automated with 'qemu-rme-stack' [1], that simply follows those instructions. >> >> [0] https://linaro.atlassian.net/wiki/spaces/QEMU/pages/30128767027/Device+Assignment+Enabled+RME+Stack+on+QEMU > > Should [0] be: > > https://linaro.atlassian.net/wiki/spaces/QEMU/pages/29051027459/Building+an+RME+stack+for+QEMU ? > Correct, bad copy paste sorry. > >> [1] https://github.com/pbo-linaro/qemu-rme-stack >> >> To add to the confusion (*get ready*), [0] uses OP-TEE build, which is simply a build system for op-tee + other things, but in the variant it generates, OP-TEE itself is not included. Yes, that's utterly confusing and took me quite some time to discover it, after talking with Mathieu himself. They removed it explicitely to make the stack more simple. In short, for Realms, forget about OP-TEE. > > ah, that's what confused me! I see now. > > >> If you want to see it by yourself: >> >> https://git.codelinaro.org/linaro/dcap/op-tee-4.2.0/build/-/blob/cca/v8/qemu_v8_cca.mk?ref_type=heads#L172 >> >> ... >> TF_A_FLAGS ?= \ >> BL33=$(BL33_BIN) \ >> PLAT=qemu \ >> QEMU_USE_GIC_DRIVER=$(TFA_GIC_DRIVER) \ >> DEBUG=$(TF_A_DEBUG) \ >> LOG_LEVEL=$(TF_A_LOGLVL) \ >> ENABLE_RME=1 \ >> RMM=$(RMM_BIN) >> ... >> TF_A_FLAGS_BL32_OPTEE = BL32=$(OPTEE_OS_HEADER_V2_BIN) >> TF_A_FLAGS_BL32_OPTEE += BL32_EXTRA1=$(OPTEE_OS_PAGER_V2_BIN) >> TF_A_FLAGS_BL32_OPTEE += BL32_EXTRA2=$(OPTEE_OS_PAGEABLE_V2_BIN) >> TF_A_FLAGS_SPMC_AT_EL_n = $(TF_A_FLAGS_BL32_OPTEE) SPD=opteed >> ... >> #TF_A_FLAGS += $(TF_A_FLAGS_SPMC_AT_EL_$(SPMC_AT_EL)) >> >> The last line shows that OP_TEE flags are **NOT** added to TF_A_FLAGS build flags. >> The qemu_v8_cca.mk build file was copied from qemu_v8.mk, which itself has the OP-TEE inclusion. >> >> --- >> >> Recently, I had to generate a custom rootfs, and I experimented generating it directly from docker images, to avoid rebuilding the world using Buildroot. Once it worked, I realized it was a good opportunity to rebuild the rest of the stack too. The result is 'qemu-linux-stack' [2]. >> >> Master branch has only tf-a + uboot (no Realm support), while rme branch [3], which supports Realm, uses tf-a + rmm + edk2 instead. I removed u-boot as I couldn't get it to boot, and I knew that edk2 worked. >> >> One branch is only one configuration (and it will stay this way). >> >> [2] https://github.com/pbo-linaro/qemu-linux-stack >> [3] https://github.com/pbo-linaro/qemu-linux-stack/tree/rme >> >> Beyond the personal knowledge I got through that, I hope it can be used for others for who it's confusing about what runs before start_kernel, and I guess I'm not the only one who didn't know about that. In the end, things are not too complicated, but as most of the build systems out there (OP-TEE build, shrinkwrap, ...) try to be "generic and versatile", it ends up being complicated. I prefer basic and straightforward script shells to lenghty documentation and wiki pages, but it's a personal choice. >> >> --- >> >> Now, coming to the change introduced by this series, and supporting FEAT_SCTRL2 and FEAT_TCR2, all those images need to be updated [0], [1], [2], [3], because essentially, TF-A itself must be patched to support this. I was about to mention that, and mention that I can send a PR directly to it once we have this merged on QEMU side. >> >> The change in arm-trusted-firmware is quite simple: >> >> +diff --git a/plat/qemu/common/common.mk b/plat/qemu/common/common.mk >> +index 751511cf8..6bc108492 100644 >> +--- a/plat/qemu/common/common.mk >> ++++ b/plat/qemu/common/common.mk >> +@@ -122,6 +122,10 @@ ENABLE_FEAT_FGT := 2 >> + # 8.7 >> + ENABLE_FEAT_HCX := 2 >> + >> ++# 8.8 >> ++ENABLE_FEAT_TCR2 := 2 >> ++ENABLE_FEAT_SCTLR2 := 2 >> ++ >> >> I'll push all that tomorrow on master, and rebase rmm and device_passthrough on top of it. >> I'll add this to original RME images, it's less direct though, as it's needed to update the tf-a fork, and then the optee build system. > > So, I still fail to understand how you are generating the out/bin/flash.bin used in our images. > > First I tried to use the https://github.com/pbo-linaro/qemu-rme-stack, since it's cited in the tests. I tried: > > $ ./container.sh ./build_virt.sh > > and get an error which seems a commit mismatch in some of branches used by repo: > > repo: reusing existing repo client checkout in /mnt/git/qemu-rme-stack > Traceback (most recent call last): > File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 869, in <module> > _Main(sys.argv[1:]) > File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 845, in _Main > result = repo._Run(name, gopts, argv) or 0 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 293, in _Run > result = run() > ^^^^^ > File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 274, in <lambda> > lambda: self._RunLong(name, gopts, argv, git_trace2_event_log) or 0 > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 437, in _RunLong > execute_command() > File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 403, in execute_command > execute_command_helper() > File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 369, in execute_command_helper > result = cmd.Execute(copts, cargs) > ^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/mnt/git/qemu-rme-stack/.repo/repo/subcmds/init.py", line 400, in Execute > self._SyncManifest(opt) > File "/mnt/git/qemu-rme-stack/.repo/repo/subcmds/init.py", line 146, in _SyncManifest > if not self.manifest.manifestProject.Sync( > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/mnt/git/qemu-rme-stack/.repo/repo/project.py", line 4704, in Sync > self.MetaBranchSwitch(submodules=submodules, verbose=verbose) > File "/mnt/git/qemu-rme-stack/.repo/repo/project.py", line 4173, in MetaBranchSwitch > self.Sync_LocalHalf(syncbuf, submodules=submodules, verbose=verbose) > File "/mnt/git/qemu-rme-stack/.repo/repo/project.py", line 1636, in Sync_LocalHalf > lost = self._revlist(not_rev(revid), HEAD) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/mnt/git/qemu-rme-stack/.repo/repo/project.py", line 3676, in _revlist > return self.work_git.rev_list(*a, **kw) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/mnt/git/qemu-rme-stack/.repo/repo/project.py", line 3908, in rev_list > p.Wait() > File "/mnt/git/qemu-rme-stack/.repo/repo/git_command.py", line 556, in Wait > self.VerifyCommand() > File "/mnt/git/qemu-rme-stack/.repo/repo/git_command.py", line 546, in VerifyCommand > raise GitCommandError( > git_command.GitCommandError: GitCommandError: 'rev-list ^6fd1cc667671a12cfc8789a390c990446e621f8f HEAD --' on manifests failed > stderr: fatal: bad revision 'HEAD' > I'm not sure what is happening. I gave it a try now (from a clean repo), and it works as expected. I had weird errors when stopping sync with ctrl-c in the past too. Anyway, clean and retry. > > Then I tried to build it from https://github.com/pbo-linaro/qemu-linux-stack/tree/rme and out/flash.bin > simply doesn't boot the kernel... > Maybe you built previously master branch, then switched to rme and rebuilt. In this case, your kernel and tf-a are not updated. I added a note in README, there is no dependency check when switching branches, you're expected to run git clean by yourself. Both shrinkwrap and OP-TEE build don't really handle that properly to be honest... It's not ideal, but I don't want to go down the rabbit hole of reimplementing any kind of half baked Make based dependency graph. Maybe we could use git submodules or repo in the future, but both have their own drawbacks too. > >> Sorry for this very long email. At least, all is there. > > Nope, that's quite helpful. This firmware stack is a mess :) > > > Cheers, > Gustavo
Hi Pierrick! On 7/16/25 02:56, Pierrick Bouvier wrote: > On 7/15/25 8:13 PM, Gustavo Romero wrote: >> Hi Pierrick, >> >> On 7/14/25 22:26, Pierrick Bouvier wrote: >>> On 7/14/25 4:31 PM, Gustavo Romero wrote: >>>> Hi folks, >>>> >>>> Richard, thanks for v8. Pierrick, thanks for testing it. :) >>>> >>>> On 7/14/25 14:09, Pierrick Bouvier wrote: >>>>> On 7/14/25 8:58 AM, Richard Henderson wrote: >>>>>> Changes for v8: >>>>>> - Re-order SCTLR2 and TCR2 so that they are independent of MEC. >>>>>> - Enable the SCTLR2 and TCR2 enable bits. >>>>>> - Squash 3 smaller MEC patches together. >>>>>> >>>>>> This still fails the RME tests, because we still need TF-A rebuilt >>>>>> with ENABLE_FEAT_SCTLR2 and ENABLE_FEAT_TCR2. Pierrick, since you >>>>>> have just done such a build, could you re-test with this series? >>>>>> >>>>> >>>>> I tested that on my local Realm enabled setup and I can confirm this solved the issue and current series works. >>>>> Both flags are needed in TF-A. ENABLE_FEAT_TCR2 is needed to boot host, and ENABLE_FEAT_SCTLR2 is needed to boot nested guest. >>>> >>>> I'm a bit confused because the QEMU RME tests, afaics, uses OP-TEE, not TF-A. I've built TF-A >>>> using the scripts in [0], enabling ENABLE_FEAT_TCR2 and ENABLE_FEAT_SCTLR2, but no way to get >>>> it booting. I understand we can embed a OP-TEE into the TF_A via BL32=<optee_image> when >>>> building TF-A. Is that what you're using? >>>> >>> >>> I agree it's confusing. In short, no, OP-TEE is not used anywhere for Realms, only TF-A and RMM are used in our images. It seems that OP-TEE is a term used generically to represent any firmware running in secure mode, but it's a *totally* different software than TF-A + RMM. Naming OP-TEE like this is like if Linux would have been named "OP-kernel". >> >> Got it, so we use TF-A + RMM in the test images. Thanks for the clarifications. >> >> Isn't the generic term (or concept) actually TEE (not OP-TEE) and OP-TEE is a real software stack that implements the TEE spec, i.e., OP-TEE is code in https://github.com/OP-TEE)? >> > > Yes, correct. The term is TEE, OP-TEE is the project you mentioned. For some reason, everybody seems to think that OP-TEE is *the* unique TEE implementation is Arm world. > Thus my analogy with Linux that would be named OP-Kernel. > >> >>> The RME tests we have are based on this excellent tutorial [0], and build is automated with 'qemu-rme-stack' [1], that simply follows those instructions. >>> >>> [0] https://linaro.atlassian.net/wiki/spaces/QEMU/pages/30128767027/Device+Assignment+Enabled+RME+Stack+on+QEMU >> >> Should [0] be: >> >> https://linaro.atlassian.net/wiki/spaces/QEMU/pages/29051027459/Building+an+RME+stack+for+QEMU ? >> > > Correct, bad copy paste sorry. > >> >>> [1] https://github.com/pbo-linaro/qemu-rme-stack >>> >>> To add to the confusion (*get ready*), [0] uses OP-TEE build, which is simply a build system for op-tee + other things, but in the variant it generates, OP-TEE itself is not included. Yes, that's utterly confusing and took me quite some time to discover it, after talking with Mathieu himself. They removed it explicitely to make the stack more simple. In short, for Realms, forget about OP-TEE. >> >> ah, that's what confused me! I see now. >> >> >>> If you want to see it by yourself: >>> >>> https://git.codelinaro.org/linaro/dcap/op-tee-4.2.0/build/-/blob/cca/v8/qemu_v8_cca.mk?ref_type=heads#L172 >>> >>> ... >>> TF_A_FLAGS ?= \ >>> BL33=$(BL33_BIN) \ >>> PLAT=qemu \ >>> QEMU_USE_GIC_DRIVER=$(TFA_GIC_DRIVER) \ >>> DEBUG=$(TF_A_DEBUG) \ >>> LOG_LEVEL=$(TF_A_LOGLVL) \ >>> ENABLE_RME=1 \ >>> RMM=$(RMM_BIN) >>> ... >>> TF_A_FLAGS_BL32_OPTEE = BL32=$(OPTEE_OS_HEADER_V2_BIN) >>> TF_A_FLAGS_BL32_OPTEE += BL32_EXTRA1=$(OPTEE_OS_PAGER_V2_BIN) >>> TF_A_FLAGS_BL32_OPTEE += BL32_EXTRA2=$(OPTEE_OS_PAGEABLE_V2_BIN) >>> TF_A_FLAGS_SPMC_AT_EL_n = $(TF_A_FLAGS_BL32_OPTEE) SPD=opteed >>> ... >>> #TF_A_FLAGS += $(TF_A_FLAGS_SPMC_AT_EL_$(SPMC_AT_EL)) >>> >>> The last line shows that OP_TEE flags are **NOT** added to TF_A_FLAGS build flags. >>> The qemu_v8_cca.mk build file was copied from qemu_v8.mk, which itself has the OP-TEE inclusion. >>> >>> --- >>> >>> Recently, I had to generate a custom rootfs, and I experimented generating it directly from docker images, to avoid rebuilding the world using Buildroot. Once it worked, I realized it was a good opportunity to rebuild the rest of the stack too. The result is 'qemu-linux-stack' [2]. >>> >>> Master branch has only tf-a + uboot (no Realm support), while rme branch [3], which supports Realm, uses tf-a + rmm + edk2 instead. I removed u-boot as I couldn't get it to boot, and I knew that edk2 worked. >>> >>> One branch is only one configuration (and it will stay this way). >>> >>> [2] https://github.com/pbo-linaro/qemu-linux-stack >>> [3] https://github.com/pbo-linaro/qemu-linux-stack/tree/rme >>> >>> Beyond the personal knowledge I got through that, I hope it can be used for others for who it's confusing about what runs before start_kernel, and I guess I'm not the only one who didn't know about that. In the end, things are not too complicated, but as most of the build systems out there (OP-TEE build, shrinkwrap, ...) try to be "generic and versatile", it ends up being complicated. I prefer basic and straightforward script shells to lenghty documentation and wiki pages, but it's a personal choice. >>> >>> --- >>> >>> Now, coming to the change introduced by this series, and supporting FEAT_SCTRL2 and FEAT_TCR2, all those images need to be updated [0], [1], [2], [3], because essentially, TF-A itself must be patched to support this. I was about to mention that, and mention that I can send a PR directly to it once we have this merged on QEMU side. >>> >>> The change in arm-trusted-firmware is quite simple: >>> >>> +diff --git a/plat/qemu/common/common.mk b/plat/qemu/common/common.mk >>> +index 751511cf8..6bc108492 100644 >>> +--- a/plat/qemu/common/common.mk >>> ++++ b/plat/qemu/common/common.mk >>> +@@ -122,6 +122,10 @@ ENABLE_FEAT_FGT := 2 >>> + # 8.7 >>> + ENABLE_FEAT_HCX := 2 >>> + >>> ++# 8.8 >>> ++ENABLE_FEAT_TCR2 := 2 >>> ++ENABLE_FEAT_SCTLR2 := 2 >>> ++ >>> >>> I'll push all that tomorrow on master, and rebase rmm and device_passthrough on top of it. >>> I'll add this to original RME images, it's less direct though, as it's needed to update the tf-a fork, and then the optee build system. >> >> So, I still fail to understand how you are generating the out/bin/flash.bin used in our images. >> >> First I tried to use the https://github.com/pbo-linaro/qemu-rme-stack, since it's cited in the tests. I tried: >> >> $ ./container.sh ./build_virt.sh >> >> and get an error which seems a commit mismatch in some of branches used by repo: >> >> repo: reusing existing repo client checkout in /mnt/git/qemu-rme-stack >> Traceback (most recent call last): >> File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 869, in <module> >> _Main(sys.argv[1:]) >> File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 845, in _Main >> result = repo._Run(name, gopts, argv) or 0 >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 293, in _Run >> result = run() >> ^^^^^ >> File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 274, in <lambda> >> lambda: self._RunLong(name, gopts, argv, git_trace2_event_log) or 0 >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 437, in _RunLong >> execute_command() >> File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 403, in execute_command >> execute_command_helper() >> File "/mnt/git/qemu-rme-stack/.repo/repo/main.py", line 369, in execute_command_helper >> result = cmd.Execute(copts, cargs) >> ^^^^^^^^^^^^^^^^^^^^^^^^^ >> File "/mnt/git/qemu-rme-stack/.repo/repo/subcmds/init.py", line 400, in Execute >> self._SyncManifest(opt) >> File "/mnt/git/qemu-rme-stack/.repo/repo/subcmds/init.py", line 146, in _SyncManifest >> if not self.manifest.manifestProject.Sync( >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> File "/mnt/git/qemu-rme-stack/.repo/repo/project.py", line 4704, in Sync >> self.MetaBranchSwitch(submodules=submodules, verbose=verbose) >> File "/mnt/git/qemu-rme-stack/.repo/repo/project.py", line 4173, in MetaBranchSwitch >> self.Sync_LocalHalf(syncbuf, submodules=submodules, verbose=verbose) >> File "/mnt/git/qemu-rme-stack/.repo/repo/project.py", line 1636, in Sync_LocalHalf >> lost = self._revlist(not_rev(revid), HEAD) >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> File "/mnt/git/qemu-rme-stack/.repo/repo/project.py", line 3676, in _revlist >> return self.work_git.rev_list(*a, **kw) >> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ >> File "/mnt/git/qemu-rme-stack/.repo/repo/project.py", line 3908, in rev_list >> p.Wait() >> File "/mnt/git/qemu-rme-stack/.repo/repo/git_command.py", line 556, in Wait >> self.VerifyCommand() >> File "/mnt/git/qemu-rme-stack/.repo/repo/git_command.py", line 546, in VerifyCommand >> raise GitCommandError( >> git_command.GitCommandError: GitCommandError: 'rev-list ^6fd1cc667671a12cfc8789a390c990446e621f8f HEAD --' on manifests failed >> stderr: fatal: bad revision 'HEAD' >> > > I'm not sure what is happening. I gave it a try now (from a clean repo), and it works as expected. I had weird errors when stopping sync with ctrl-c in the past too. Anyway, clean and retry. OK >> >> Then I tried to build it from https://github.com/pbo-linaro/qemu-linux-stack/tree/rme and out/flash.bin >> simply doesn't boot the kernel... >> > > Maybe you built previously master branch, then switched to rme and rebuilt. In this case, your kernel and tf-a are not updated. I added a note in README, there is no dependency check when switching branches, you're expected to run git clean by yourself. > Both shrinkwrap and OP-TEE build don't really handle that properly to be honest... > > It's not ideal, but I don't want to go down the rabbit hole of reimplementing any kind of half baked Make based dependency graph. > Maybe we could use git submodules or repo in the future, but both have their own drawbacks too. Won't dare to blame you for it, we need to keep ourselves sane XD You did a great work already putting it in a single place and using bash scripts as much as possible :) Cheers, Gustavo >> >>> Sorry for this very long email. At least, all is there. >> >> Nope, that's quite helpful. This firmware stack is a mess :) >> >> >> Cheers, >> Gustavo >
© 2016 - 2025 Red Hat, Inc.