The Armv-R architecture profile was designed to support use cases
that have a high sensitivity to deterministic execution. (e.g.
Fuel Injection, Brake control, Drive trains, Motor control etc)

Arm announced Armv8-R in 2013, it is the latest generation Arm
architecture targeted at the Real-time profile. It introduces
virtualization at the highest security level while retaining the
Protected Memory System Architecture (PMSA) based on a Memory
Protection Unit (MPU). In 2020, Arm announced Cortex-R82,
which is the first Arm 64-bit Cortex-R processor based on Armv8-R64.
The latest Armv8-R64 document can be found [1]. And the features of
Armv8-R64 architecture:
  - An exception model that is compatible with the Armv8-A model
  - Virtualization with support for guest operating systems
  - PMSA virtualization using MPUs In EL2.
  - Adds support for the 64-bit A64 instruction set.
  - Supports up to 48-bit physical addressing.
  - Supports three Exception Levels (ELs)
        - Secure EL2 - The Highest Privilege
        - Secure EL1 - RichOS (MMU) or RTOS (MPU)
        - Secure EL0 - Application Workloads
 - Supports only a single Security state - Secure.
 - MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.

These patch series are implementing the Armv8-R64 MPU support
for Xen, which are based on the discussion of
"Proposal for Porting Xen to Armv8-R64 - DraftC" [1].

We will implement the Armv8-R64 and MPU support in three stages:
1. Boot Xen itself to idle thread, do not create any guests on it.
2. Support to boot MPU and MMU domains on Armv8-R64 Xen.
3. SMP and other advanced features of Xen support on Armv8-R64.

We will split these patches to several parts, this series is the
part#1, the full PoC can be found in [3]. More software for
Armv8-R64 can be found in [4];

[1] https://developer.arm.com/documentation/ddi0600/latest
[2] https://lists.xenproject.org/archives/html/xen-devel/2022-05/msg00643.html
[3] https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc/-/tree/poc/r82-mpu-v2
[4] https://armv8r64-refstack.docs.arm.com/en/v4.0/

Penny Zheng (3):
  xen/arm64: create boot-time MPU protection regions
  xen/arm64: introduce helpers for MPU enable/disable
  xen/arm64: add setup_fixmap and remove_identity_mapping for MPU

Wei Chen (8):
  xen/arm: remove xen_phys_start and xenheap_phys_end from config.h
  xen/arm: add iounmap after initrd has been loaded in domain_build
  xen/arm: disable EFI boot services for MPU systems
  xen/arm: adjust Xen TLB helpers for Armv8-R64 PMSA
  xen/arm: define Xen start address for FVP BaseR platform
  xen/arm: split MMU and MPU config files from config.h
  xen/arm: implement FIXMAP_ADDR for MPU systems
  xen/arm64: move MMU related code from head.S to head_mmu.S

 xen/arch/arm/Kconfig                          |  15 +-
 xen/arch/arm/arm64/Makefile                   |   5 +
 xen/arch/arm/arm64/head.S                     | 429 ++----------------
 xen/arch/arm/arm64/head_mmu.S                 | 364 +++++++++++++++
 xen/arch/arm/arm64/head_mpu.S                 | 154 +++++++
 xen/arch/arm/domain_build.c                   |   2 +
 xen/arch/arm/include/asm/arm64/flushtlb.h     |  25 +
 xen/arch/arm/include/asm/arm64/macros.h       |  52 ++-
 xen/arch/arm/include/asm/arm64/mpu.h          |  13 +
 xen/arch/arm/include/asm/arm64/sysregs.h      |  89 ++++
 xen/arch/arm/include/asm/config.h             |  99 +---
 xen/arch/arm/include/asm/config_mmu.h         | 119 +++++
 xen/arch/arm/include/asm/config_mpu.h         |  29 ++
 xen/arch/arm/include/asm/fixmap.h             |  25 +
 xen/arch/arm/include/asm/flushtlb.h           |  22 +
 .../arm/include/asm/platforms/fvp_baser.h     |  18 +
 xen/arch/arm/platforms/Kconfig                |  16 +-
 17 files changed, 976 insertions(+), 500 deletions(-)
 create mode 100644 xen/arch/arm/arm64/head_mmu.S
 create mode 100644 xen/arch/arm/arm64/head_mpu.S
 create mode 100644 xen/arch/arm/include/asm/arm64/mpu.h
 create mode 100644 xen/arch/arm/include/asm/config_mmu.h
 create mode 100644 xen/arch/arm/include/asm/config_mpu.h
 create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h

-- 
2.25.1

Sorry for the wrong patch version. This is the first version of this series.

Cheers,
Wei Chen

> -----Original Message-----
> From: Wei Chen <wei.chen@arm.com>
> Sent: 2022年11月4日 18:08
> To: xen-devel@lists.xenproject.org
> Cc: nd <nd@arm.com>; Wei Chen <Wei.Chen@arm.com>; Stefano Stabellini
> <sstabellini@kernel.org>; Julien Grall <julien@xen.org>; Bertrand Marquis
> <Bertrand.Marquis@arm.com>; Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>
> Subject: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen -
> Part#1
> 
> The Armv-R architecture profile was designed to support use cases
> that have a high sensitivity to deterministic execution. (e.g.
> Fuel Injection, Brake control, Drive trains, Motor control etc)
> 
> Arm announced Armv8-R in 2013, it is the latest generation Arm
> architecture targeted at the Real-time profile. It introduces
> virtualization at the highest security level while retaining the
> Protected Memory System Architecture (PMSA) based on a Memory
> Protection Unit (MPU). In 2020, Arm announced Cortex-R82,
> which is the first Arm 64-bit Cortex-R processor based on Armv8-R64.
> The latest Armv8-R64 document can be found [1]. And the features of
> Armv8-R64 architecture:
>   - An exception model that is compatible with the Armv8-A model
>   - Virtualization with support for guest operating systems
>   - PMSA virtualization using MPUs In EL2.
>   - Adds support for the 64-bit A64 instruction set.
>   - Supports up to 48-bit physical addressing.
>   - Supports three Exception Levels (ELs)
>         - Secure EL2 - The Highest Privilege
>         - Secure EL1 - RichOS (MMU) or RTOS (MPU)
>         - Secure EL0 - Application Workloads
>  - Supports only a single Security state - Secure.
>  - MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.
> 
> These patch series are implementing the Armv8-R64 MPU support
> for Xen, which are based on the discussion of
> "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
> 
> We will implement the Armv8-R64 and MPU support in three stages:
> 1. Boot Xen itself to idle thread, do not create any guests on it.
> 2. Support to boot MPU and MMU domains on Armv8-R64 Xen.
> 3. SMP and other advanced features of Xen support on Armv8-R64.
> 
> We will split these patches to several parts, this series is the
> part#1, the full PoC can be found in [3]. More software for
> Armv8-R64 can be found in [4];
> 
> [1] https://developer.arm.com/documentation/ddi0600/latest
> [2] https://lists.xenproject.org/archives/html/xen-devel/2022-
> 05/msg00643.html
> [3] https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc/-
> /tree/poc/r82-mpu-v2
> [4] https://armv8r64-refstack.docs.arm.com/en/v4.0/
> 
> Penny Zheng (3):
>   xen/arm64: create boot-time MPU protection regions
>   xen/arm64: introduce helpers for MPU enable/disable
>   xen/arm64: add setup_fixmap and remove_identity_mapping for MPU
> 
> Wei Chen (8):
>   xen/arm: remove xen_phys_start and xenheap_phys_end from config.h
>   xen/arm: add iounmap after initrd has been loaded in domain_build
>   xen/arm: disable EFI boot services for MPU systems
>   xen/arm: adjust Xen TLB helpers for Armv8-R64 PMSA
>   xen/arm: define Xen start address for FVP BaseR platform
>   xen/arm: split MMU and MPU config files from config.h
>   xen/arm: implement FIXMAP_ADDR for MPU systems
>   xen/arm64: move MMU related code from head.S to head_mmu.S
> 
>  xen/arch/arm/Kconfig                          |  15 +-
>  xen/arch/arm/arm64/Makefile                   |   5 +
>  xen/arch/arm/arm64/head.S                     | 429 ++----------------
>  xen/arch/arm/arm64/head_mmu.S                 | 364 +++++++++++++++
>  xen/arch/arm/arm64/head_mpu.S                 | 154 +++++++
>  xen/arch/arm/domain_build.c                   |   2 +
>  xen/arch/arm/include/asm/arm64/flushtlb.h     |  25 +
>  xen/arch/arm/include/asm/arm64/macros.h       |  52 ++-
>  xen/arch/arm/include/asm/arm64/mpu.h          |  13 +
>  xen/arch/arm/include/asm/arm64/sysregs.h      |  89 ++++
>  xen/arch/arm/include/asm/config.h             |  99 +---
>  xen/arch/arm/include/asm/config_mmu.h         | 119 +++++
>  xen/arch/arm/include/asm/config_mpu.h         |  29 ++
>  xen/arch/arm/include/asm/fixmap.h             |  25 +
>  xen/arch/arm/include/asm/flushtlb.h           |  22 +
>  .../arm/include/asm/platforms/fvp_baser.h     |  18 +
>  xen/arch/arm/platforms/Kconfig                |  16 +-
>  17 files changed, 976 insertions(+), 500 deletions(-)
>  create mode 100644 xen/arch/arm/arm64/head_mmu.S
>  create mode 100644 xen/arch/arm/arm64/head_mpu.S
>  create mode 100644 xen/arch/arm/include/asm/arm64/mpu.h
>  create mode 100644 xen/arch/arm/include/asm/config_mmu.h
>  create mode 100644 xen/arch/arm/include/asm/config_mpu.h
>  create mode 100644 xen/arch/arm/include/asm/platforms/fvp_baser.h
> 
> --
> 2.25.1

Hi Wei,

On 04/11/2022 10:07, Wei Chen wrote:
> The Armv-R architecture profile was designed to support use cases
> that have a high sensitivity to deterministic execution. (e.g.
> Fuel Injection, Brake control, Drive trains, Motor control etc)
> 
> Arm announced Armv8-R in 2013, it is the latest generation Arm
> architecture targeted at the Real-time profile. It introduces
> virtualization at the highest security level while retaining the
> Protected Memory System Architecture (PMSA) based on a Memory
> Protection Unit (MPU). In 2020, Arm announced Cortex-R82,
> which is the first Arm 64-bit Cortex-R processor based on Armv8-R64.
> The latest Armv8-R64 document can be found [1]. And the features of
> Armv8-R64 architecture:
>    - An exception model that is compatible with the Armv8-A model
>    - Virtualization with support for guest operating systems
>    - PMSA virtualization using MPUs In EL2.
>    - Adds support for the 64-bit A64 instruction set.
>    - Supports up to 48-bit physical addressing.
>    - Supports three Exception Levels (ELs)
>          - Secure EL2 - The Highest Privilege
>          - Secure EL1 - RichOS (MMU) or RTOS (MPU)
>          - Secure EL0 - Application Workloads
>   - Supports only a single Security state - Secure.
>   - MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.
> 
> These patch series are implementing the Armv8-R64 MPU support
> for Xen, which are based on the discussion of
> "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
> 
> We will implement the Armv8-R64 and MPU support in three stages:
> 1. Boot Xen itself to idle thread, do not create any guests on it.

I read this as I can build Xen and see it boots (not creating domain). 
However... HAS_MPU is not defined and I was expecting mm.c to be 
modified to cater the MPU support. So I am a bit ensure what the series 
is actually doing.

Cheers,

-- 
Julien Grall

Hi Julien,

> >   - Supports only a single Security state - Secure.
> >   - MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.
> >
> > These patch series are implementing the Armv8-R64 MPU support
> > for Xen, which are based on the discussion of
> > "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
> >
> > We will implement the Armv8-R64 and MPU support in three stages:
> > 1. Boot Xen itself to idle thread, do not create any guests on it.
> 
> I read this as I can build Xen and see it boots (not creating domain).
> However... HAS_MPU is not defined and I was expecting mm.c to be
> modified to cater the MPU support. So I am a bit ensure what the series
> is actually doing.
> 

These 11 patches are part#1 of stage#1, the full stage#1 has about 30
patches. We have some concerns if we send so many patches at once, the
review pressure of maintainers may be very high, so we only choose about
10 to send as part of it. But this also means that we can't do a relatively
complete thing in this part#1 series.

We want to hear some suggestions from you to make so many patches can be
Reviewed efficiently. Can we send the patches by stages, even the stage#1
will have about 30 patches?

Cheers,
Wei Chen

> Cheers,
> 
> --
> Julien Grall

On 07/11/2022 09:52, Wei Chen wrote:
> Hi Julien,

Hi,

> 
>>>    - Supports only a single Security state - Secure.
>>>    - MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.
>>>
>>> These patch series are implementing the Armv8-R64 MPU support
>>> for Xen, which are based on the discussion of
>>> "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
>>>
>>> We will implement the Armv8-R64 and MPU support in three stages:
>>> 1. Boot Xen itself to idle thread, do not create any guests on it.
>>
>> I read this as I can build Xen and see it boots (not creating domain).
>> However... HAS_MPU is not defined and I was expecting mm.c to be
>> modified to cater the MPU support. So I am a bit ensure what the series
>> is actually doing.
>>
> 
> These 11 patches are part#1 of stage#1, the full stage#1 has about 30
> patches. We have some concerns if we send so many patches at once, the
> review pressure of maintainers may be very high, so we only choose about
> 10 to send as part of it. But this also means that we can't do a relatively
> complete thing in this part#1 series.
> 
> We want to hear some suggestions from you to make so many patches can be
> Reviewed efficiently. Can we send the patches by stages, even the stage#1
> will have about 30 patches?

30 patches in a go is no too bad. I would personally prefer that because 
at least I have better idea of the shape of the code after stage#1 and 
also possibly test it (I need to check if I have access for the ARMv8-R 
model).

Cheers,

-- 
Julien Grall

Hi Julien,

> -----Original Message-----
> From: Julien Grall <julien@xen.org>
> Sent: 2022年11月7日 18:16
> To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> <Volodymyr_Babchuk@epam.com>
> Subject: Re: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen -
> Part#1
> 
> 
> 
> On 07/11/2022 09:52, Wei Chen wrote:
> > Hi Julien,
> 
> Hi,
> 
> >
> >>>    - Supports only a single Security state - Secure.
> >>>    - MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.
> >>>
> >>> These patch series are implementing the Armv8-R64 MPU support
> >>> for Xen, which are based on the discussion of
> >>> "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
> >>>
> >>> We will implement the Armv8-R64 and MPU support in three stages:
> >>> 1. Boot Xen itself to idle thread, do not create any guests on it.
> >>
> >> I read this as I can build Xen and see it boots (not creating domain).
> >> However... HAS_MPU is not defined and I was expecting mm.c to be
> >> modified to cater the MPU support. So I am a bit ensure what the series
> >> is actually doing.
> >>
> >
> > These 11 patches are part#1 of stage#1, the full stage#1 has about 30
> > patches. We have some concerns if we send so many patches at once, the
> > review pressure of maintainers may be very high, so we only choose about
> > 10 to send as part of it. But this also means that we can't do a
> relatively
> > complete thing in this part#1 series.
> >
> > We want to hear some suggestions from you to make so many patches can be
> > Reviewed efficiently. Can we send the patches by stages, even the
> stage#1
> > will have about 30 patches?
> 
> 30 patches in a go is no too bad. I would personally prefer that because
> at least I have better idea of the shape of the code after stage#1 and
> also possibly test it (I need to check if I have access for the ARMv8-R
> model).
> 

I also prefer to this way. After we have addressed the comments in
this series, we will send the full stage#1 patches together in v2.

For Armv8-R model, you can download Armv8-R AEM FVP model from link[1].
It's license free.

[1] https://developer.arm.com/downloads/-/arm-ecosystem-models

Cheers,
Wei Chen

> Cheers,
> 
> --
> Julien Grall

On Mon, 7 Nov 2022, Wei Chen wrote:
> Hi Julien,
> 
> > -----Original Message-----
> > From: Julien Grall <julien@xen.org>
> > Sent: 2022年11月7日 18:16
> > To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> > Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
> > Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> > <Volodymyr_Babchuk@epam.com>
> > Subject: Re: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen -
> > Part#1
> > 
> > 
> > 
> > On 07/11/2022 09:52, Wei Chen wrote:
> > > Hi Julien,
> > 
> > Hi,
> > 
> > >
> > >>>    - Supports only a single Security state - Secure.
> > >>>    - MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.
> > >>>
> > >>> These patch series are implementing the Armv8-R64 MPU support
> > >>> for Xen, which are based on the discussion of
> > >>> "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
> > >>>
> > >>> We will implement the Armv8-R64 and MPU support in three stages:
> > >>> 1. Boot Xen itself to idle thread, do not create any guests on it.
> > >>
> > >> I read this as I can build Xen and see it boots (not creating domain).
> > >> However... HAS_MPU is not defined and I was expecting mm.c to be
> > >> modified to cater the MPU support. So I am a bit ensure what the series
> > >> is actually doing.
> > >>
> > >
> > > These 11 patches are part#1 of stage#1, the full stage#1 has about 30
> > > patches. We have some concerns if we send so many patches at once, the
> > > review pressure of maintainers may be very high, so we only choose about
> > > 10 to send as part of it. But this also means that we can't do a
> > relatively
> > > complete thing in this part#1 series.
> > >
> > > We want to hear some suggestions from you to make so many patches can be
> > > Reviewed efficiently. Can we send the patches by stages, even the
> > stage#1
> > > will have about 30 patches?
> > 
> > 30 patches in a go is no too bad. I would personally prefer that because
> > at least I have better idea of the shape of the code after stage#1 and
> > also possibly test it (I need to check if I have access for the ARMv8-R
> > model).
> > 
> 
> I also prefer to this way. After we have addressed the comments in
> this series, we will send the full stage#1 patches together in v2.


One suggestion to make things easier to review and to commit is to
organize the series in a way so that the first 10 patches can still be
committed first independently, even if all 30 patches are sent together.

Or alternatively only send 10 patches but also add a link to a
github/gitlab tree with all the 30+ patches so that maintainers can have
a look how the whole work fit together.

I think we are all on the same page -- I just wanted to highlight that
we don't have to finish the review of all 30 patches before we can start
committing some of the initial patches in the series.

Cheers,

Stefano

Hi Stefano, Julien,

> -----Original Message-----
> From: Stefano Stabellini <sstabellini@kernel.org>
> Sent: 2022年11月11日 6:26
> To: Wei Chen <Wei.Chen@arm.com>
> Cc: Julien Grall <julien@xen.org>; xen-devel@lists.xenproject.org; nd
> <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>; Bertrand
> Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> <Volodymyr_Babchuk@epam.com>
> Subject: RE: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to Xen -
> Part#1
> 
> On Mon, 7 Nov 2022, Wei Chen wrote:
> > Hi Julien,
> >
> > > -----Original Message-----
> > > From: Julien Grall <julien@xen.org>
> > > Sent: 2022年11月7日 18:16
> > > To: Wei Chen <Wei.Chen@arm.com>; xen-devel@lists.xenproject.org
> > > Cc: nd <nd@arm.com>; Stefano Stabellini <sstabellini@kernel.org>;
> Bertrand
> > > Marquis <Bertrand.Marquis@arm.com>; Volodymyr Babchuk
> > > <Volodymyr_Babchuk@epam.com>
> > > Subject: Re: [PATCH v6 00/11] xen/arm: Add Armv8-R64 MPU support to
> Xen -
> > > Part#1
> > >
> > >
> > >
> > > On 07/11/2022 09:52, Wei Chen wrote:
> > > > Hi Julien,
> > >
> > > Hi,
> > >
> > > >
> > > >>>    - Supports only a single Security state - Secure.
> > > >>>    - MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.
> > > >>>
> > > >>> These patch series are implementing the Armv8-R64 MPU support
> > > >>> for Xen, which are based on the discussion of
> > > >>> "Proposal for Porting Xen to Armv8-R64 - DraftC" [1].
> > > >>>
> > > >>> We will implement the Armv8-R64 and MPU support in three stages:
> > > >>> 1. Boot Xen itself to idle thread, do not create any guests on it.
> > > >>
> > > >> I read this as I can build Xen and see it boots (not creating
> domain).
> > > >> However... HAS_MPU is not defined and I was expecting mm.c to be
> > > >> modified to cater the MPU support. So I am a bit ensure what the
> series
> > > >> is actually doing.
> > > >>
> > > >
> > > > These 11 patches are part#1 of stage#1, the full stage#1 has about
> 30
> > > > patches. We have some concerns if we send so many patches at once,
> the
> > > > review pressure of maintainers may be very high, so we only choose
> about
> > > > 10 to send as part of it. But this also means that we can't do a
> > > relatively
> > > > complete thing in this part#1 series.
> > > >
> > > > We want to hear some suggestions from you to make so many patches
> can be
> > > > Reviewed efficiently. Can we send the patches by stages, even the
> > > stage#1
> > > > will have about 30 patches?
> > >
> > > 30 patches in a go is no too bad. I would personally prefer that
> because
> > > at least I have better idea of the shape of the code after stage#1 and
> > > also possibly test it (I need to check if I have access for the ARMv8-
> R
> > > model).
> > >
> >
> > I also prefer to this way. After we have addressed the comments in
> > this series, we will send the full stage#1 patches together in v2.
> 
> 
> One suggestion to make things easier to review and to commit is to
> organize the series in a way so that the first 10 patches can still be
> committed first independently, even if all 30 patches are sent together.
> 

I think this is foreseeable, and we have done in this way internally.
Every patch can be built and will not broken other architectures.

> Or alternatively only send 10 patches but also add a link to a
> github/gitlab tree with all the 30+ patches so that maintainers can have
> a look how the whole work fit together.
> 

In this series we have linked the gitlab branch with the full patches.

> I think we are all on the same page -- I just wanted to highlight that
> we don't have to finish the review of all 30 patches before we can start
> committing some of the initial patches in the series.
> 

Yes, I agree with it. And above solutions are ok for us. There will not
be much difference in efforts for these two ways for us, so if you guys
think which method is the most efficient, I will follow it.

Cheers,
Wei Chen

> Cheers,
> 
> Stefano