[PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

Julien Grall posted 3 patches 1 week ago
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/xen tags/patchew/20200518113008.15422-1-julien@xen.org
Maintainers: Julien Grall <julien@xen.org>, Stefano Stabellini <sstabellini@kernel.org>, Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>
xen/arch/arm/domain_build.c                | 32 +++++++++++++---------
xen/arch/arm/platform.c                    |  5 ++++
xen/arch/arm/platforms/brcm-raspberry-pi.c |  1 +
xen/include/asm-arm/mm.h                   |  2 ++
xen/include/asm-arm/numa.h                 |  5 ----
xen/include/asm-arm/platform.h             |  2 ++
6 files changed, 29 insertions(+), 18 deletions(-)

[PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

Posted by Julien Grall 1 week ago
From: Julien Grall <jgrall@amazon.com>

Hi all,

At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
only use the first GB of memory.

This is because several devices cannot DMA above 1GB but Xen doesn't
necessarily allocate memory for Dom0 below 1GB.

This small series is trying to address the problem by allowing a
platform to restrict where Dom0 banks are allocated.

This is also a candidate for Xen 4.14. Without it, a user will not be
able to use all the RAM on the Raspberry Pi 4.

This series has only be slighlty tested. I would appreciate more test on
the Rasbperry Pi 4 to confirm this removing the restriction.

Cheers,

Cc: paul@xen.org

Julien Grall (3):
  xen/arm: Allow a platform to override the DMA width
  xen/arm: Take into account the DMA width when allocating Dom0 memory
    banks
  xen/arm: plat: Allocate as much as possible memory below 1GB for dom0
    for RPI

 xen/arch/arm/domain_build.c                | 32 +++++++++++++---------
 xen/arch/arm/platform.c                    |  5 ++++
 xen/arch/arm/platforms/brcm-raspberry-pi.c |  1 +
 xen/include/asm-arm/mm.h                   |  2 ++
 xen/include/asm-arm/numa.h                 |  5 ----
 xen/include/asm-arm/platform.h             |  2 ++
 6 files changed, 29 insertions(+), 18 deletions(-)

-- 
2.17.1


Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

Posted by Stefano Stabellini 1 week ago
On Mon, 18 May 2020, Julien Grall wrote:
> From: Julien Grall <jgrall@amazon.com>
> 
> Hi all,
> 
> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
> only use the first GB of memory.
> 
> This is because several devices cannot DMA above 1GB but Xen doesn't
> necessarily allocate memory for Dom0 below 1GB.
> 
> This small series is trying to address the problem by allowing a
> platform to restrict where Dom0 banks are allocated.
> 
> This is also a candidate for Xen 4.14. Without it, a user will not be
> able to use all the RAM on the Raspberry Pi 4.

The series looks good to me aside from the couple of minor issues being
discussed


> This series has only be slighlty tested. I would appreciate more test on
> the Rasbperry Pi 4 to confirm this removing the restriction.
> 
> Cheers,
> 
> Cc: paul@xen.org
> 
> Julien Grall (3):
>   xen/arm: Allow a platform to override the DMA width
>   xen/arm: Take into account the DMA width when allocating Dom0 memory
>     banks
>   xen/arm: plat: Allocate as much as possible memory below 1GB for dom0
>     for RPI
> 
>  xen/arch/arm/domain_build.c                | 32 +++++++++++++---------
>  xen/arch/arm/platform.c                    |  5 ++++
>  xen/arch/arm/platforms/brcm-raspberry-pi.c |  1 +
>  xen/include/asm-arm/mm.h                   |  2 ++
>  xen/include/asm-arm/numa.h                 |  5 ----
>  xen/include/asm-arm/platform.h             |  2 ++
>  6 files changed, 29 insertions(+), 18 deletions(-)
> 
> -- 
> 2.17.1
> 

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

Posted by Julien Grall 1 week ago
Hi Stefano,

On 20/05/2020 23:13, Stefano Stabellini wrote:
> On Mon, 18 May 2020, Julien Grall wrote:
>> From: Julien Grall <jgrall@amazon.com>
>>
>> Hi all,
>>
>> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
>> only use the first GB of memory.
>>
>> This is because several devices cannot DMA above 1GB but Xen doesn't
>> necessarily allocate memory for Dom0 below 1GB.
>>
>> This small series is trying to address the problem by allowing a
>> platform to restrict where Dom0 banks are allocated.
>>
>> This is also a candidate for Xen 4.14. Without it, a user will not be
>> able to use all the RAM on the Raspberry Pi 4.
> 
> The series looks good to me aside from the couple of minor issues being
> discussed

Thanks, I have sent the v2 yesterday but forgot to call 
add_maintainers.pl ([1]). Do you want me to resend it with you CCed?

Cheers,

[1] <20200519172028.31169-1-julien@xen.org>

-- 
Julien Grall

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

Posted by Stefano Stabellini 1 week ago
On Thu, 21 May 2020, Julien Grall wrote:
> Hi Stefano,
> 
> On 20/05/2020 23:13, Stefano Stabellini wrote:
> > On Mon, 18 May 2020, Julien Grall wrote:
> > > From: Julien Grall <jgrall@amazon.com>
> > > 
> > > Hi all,
> > > 
> > > At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
> > > only use the first GB of memory.
> > > 
> > > This is because several devices cannot DMA above 1GB but Xen doesn't
> > > necessarily allocate memory for Dom0 below 1GB.
> > > 
> > > This small series is trying to address the problem by allowing a
> > > platform to restrict where Dom0 banks are allocated.
> > > 
> > > This is also a candidate for Xen 4.14. Without it, a user will not be
> > > able to use all the RAM on the Raspberry Pi 4.
> > 
> > The series looks good to me aside from the couple of minor issues being
> > discussed
> 
> Thanks, I have sent the v2 yesterday but forgot to call add_maintainers.pl
> ([1]). Do you want me to resend it with you CCed?
> 
> Cheers,
> 
> [1] <20200519172028.31169-1-julien@xen.org>

No worries I found them now

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

Posted by Tamas K Lengyel 1 week ago
On Mon, May 18, 2020 at 5:32 AM Julien Grall <julien@xen.org> wrote:
>
> From: Julien Grall <jgrall@amazon.com>
>
> Hi all,
>
> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
> only use the first GB of memory.
>
> This is because several devices cannot DMA above 1GB but Xen doesn't
> necessarily allocate memory for Dom0 below 1GB.
>
> This small series is trying to address the problem by allowing a
> platform to restrict where Dom0 banks are allocated.
>
> This is also a candidate for Xen 4.14. Without it, a user will not be
> able to use all the RAM on the Raspberry Pi 4.
>
> This series has only be slighlty tested. I would appreciate more test on
> the Rasbperry Pi 4 to confirm this removing the restriction.

Hi Julien,
could you post a git branch somewhere? I can try this on my rpi4 that
already runs 4.13.

Thanks,
Tamas

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

Posted by Julien Grall 1 week ago

On 19/05/2020 04:08, Tamas K Lengyel wrote:
> On Mon, May 18, 2020 at 5:32 AM Julien Grall <julien@xen.org> wrote:
>>
>> From: Julien Grall <jgrall@amazon.com>
>>
>> Hi all,
>>
>> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
>> only use the first GB of memory.
>>
>> This is because several devices cannot DMA above 1GB but Xen doesn't
>> necessarily allocate memory for Dom0 below 1GB.
>>
>> This small series is trying to address the problem by allowing a
>> platform to restrict where Dom0 banks are allocated.
>>
>> This is also a candidate for Xen 4.14. Without it, a user will not be
>> able to use all the RAM on the Raspberry Pi 4.
>>
>> This series has only be slighlty tested. I would appreciate more test on
>> the Rasbperry Pi 4 to confirm this removing the restriction.
> 
> Hi Julien,

Hi,

> could you post a git branch somewhere? I can try this on my rpi4 that
> already runs 4.13.

I have pushed a branch based on unstable and the v2 of the series:

git://xenbits.xen.org/people/julieng/xen-unstable.git

branch arm-dma/v2

Thank you in advance for the testing!

Cheers,

-- 
Julien Grall

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

Posted by Tamas K Lengyel 1 week ago
On Tue, May 19, 2020 at 11:23 AM Julien Grall <julien@xen.org> wrote:
>
>
>
> On 19/05/2020 04:08, Tamas K Lengyel wrote:
> > On Mon, May 18, 2020 at 5:32 AM Julien Grall <julien@xen.org> wrote:
> >>
> >> From: Julien Grall <jgrall@amazon.com>
> >>
> >> Hi all,
> >>
> >> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
> >> only use the first GB of memory.
> >>
> >> This is because several devices cannot DMA above 1GB but Xen doesn't
> >> necessarily allocate memory for Dom0 below 1GB.
> >>
> >> This small series is trying to address the problem by allowing a
> >> platform to restrict where Dom0 banks are allocated.
> >>
> >> This is also a candidate for Xen 4.14. Without it, a user will not be
> >> able to use all the RAM on the Raspberry Pi 4.
> >>
> >> This series has only be slighlty tested. I would appreciate more test on
> >> the Rasbperry Pi 4 to confirm this removing the restriction.
> >
> > Hi Julien,
>
> Hi,
>
> > could you post a git branch somewhere? I can try this on my rpi4 that
> > already runs 4.13.
>
> I have pushed a branch based on unstable and the v2 of the series:
>
> git://xenbits.xen.org/people/julieng/xen-unstable.git
>
> branch arm-dma/v2
>

I've updated my image I built with
https://github.com/tklengyel/xen-rpi4-builder a while ago and I've
defined 2048m as total_mem and Xen seems to be booting fine and passes
execution to dom0. With 512m being set as the Xen cmdline for dom0_mem
it was working. When I increased the mem for dom0 the boot is now
stuck at:

[    1.427788] of_cfs_init
[    1.429667] of_cfs_init: OK
[    1.432561] clk: Not disabling unused clocks
[    1.437239] Waiting for root device /dev/mmcblk0p2...
[    1.451599] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
[    1.458156] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[    1.464729] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[    1.472804] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
[    1.479370] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[    1.546902] random: fast init done
[    1.564590] mmc1: new high speed SDIO card at address 0001

Could this be because the DTB I compiled from a fresh checkout of
https://github.com/raspberrypi/linux.git branch rpi-4.19.y whereas the
kernel itself is from a checkout ~5 months ago? I guess that must be
the cause because even if I decrease the dom0_mem to 512m it still
gets stuck at the same spot whereas it was booting fine before.

Tamas

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

Posted by Roman Shaposhnik 1 week ago
On Tue, May 19, 2020 at 4:44 PM Tamas K Lengyel
<tamas.k.lengyel@gmail.com> wrote:
>
> On Tue, May 19, 2020 at 11:23 AM Julien Grall <julien@xen.org> wrote:
> >
> >
> >
> > On 19/05/2020 04:08, Tamas K Lengyel wrote:
> > > On Mon, May 18, 2020 at 5:32 AM Julien Grall <julien@xen.org> wrote:
> > >>
> > >> From: Julien Grall <jgrall@amazon.com>
> > >>
> > >> Hi all,
> > >>
> > >> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
> > >> only use the first GB of memory.
> > >>
> > >> This is because several devices cannot DMA above 1GB but Xen doesn't
> > >> necessarily allocate memory for Dom0 below 1GB.
> > >>
> > >> This small series is trying to address the problem by allowing a
> > >> platform to restrict where Dom0 banks are allocated.
> > >>
> > >> This is also a candidate for Xen 4.14. Without it, a user will not be
> > >> able to use all the RAM on the Raspberry Pi 4.
> > >>
> > >> This series has only be slighlty tested. I would appreciate more test on
> > >> the Rasbperry Pi 4 to confirm this removing the restriction.
> > >
> > > Hi Julien,
> >
> > Hi,
> >
> > > could you post a git branch somewhere? I can try this on my rpi4 that
> > > already runs 4.13.
> >
> > I have pushed a branch based on unstable and the v2 of the series:
> >
> > git://xenbits.xen.org/people/julieng/xen-unstable.git
> >
> > branch arm-dma/v2
> >
>
> I've updated my image I built with
> https://github.com/tklengyel/xen-rpi4-builder a while ago and I've
> defined 2048m as total_mem and Xen seems to be booting fine and passes
> execution to dom0. With 512m being set as the Xen cmdline for dom0_mem
> it was working. When I increased the mem for dom0 the boot is now
> stuck at:
>
> [    1.427788] of_cfs_init
> [    1.429667] of_cfs_init: OK
> [    1.432561] clk: Not disabling unused clocks
> [    1.437239] Waiting for root device /dev/mmcblk0p2...
> [    1.451599] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
> [    1.458156] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> [    1.464729] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> [    1.472804] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
> [    1.479370] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> [    1.546902] random: fast init done
> [    1.564590] mmc1: new high speed SDIO card at address 0001
>
> Could this be because the DTB I compiled from a fresh checkout of
> https://github.com/raspberrypi/linux.git branch rpi-4.19.y whereas the
> kernel itself is from a checkout ~5 months ago? I guess that must be
> the cause because even if I decrease the dom0_mem to 512m it still
> gets stuck at the same spot whereas it was booting fine before.

Stefano and I are testing the fix right now -- for now just set your
Dom0 mem to less than 512m.

Thanks,
Roman.

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

Posted by Tamas K Lengyel 1 week ago
On Tue, May 19, 2020 at 5:50 PM Roman Shaposhnik <roman@zededa.com> wrote:
>
> On Tue, May 19, 2020 at 4:44 PM Tamas K Lengyel
> <tamas.k.lengyel@gmail.com> wrote:
> >
> > On Tue, May 19, 2020 at 11:23 AM Julien Grall <julien@xen.org> wrote:
> > >
> > >
> > >
> > > On 19/05/2020 04:08, Tamas K Lengyel wrote:
> > > > On Mon, May 18, 2020 at 5:32 AM Julien Grall <julien@xen.org> wrote:
> > > >>
> > > >> From: Julien Grall <jgrall@amazon.com>
> > > >>
> > > >> Hi all,
> > > >>
> > > >> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
> > > >> only use the first GB of memory.
> > > >>
> > > >> This is because several devices cannot DMA above 1GB but Xen doesn't
> > > >> necessarily allocate memory for Dom0 below 1GB.
> > > >>
> > > >> This small series is trying to address the problem by allowing a
> > > >> platform to restrict where Dom0 banks are allocated.
> > > >>
> > > >> This is also a candidate for Xen 4.14. Without it, a user will not be
> > > >> able to use all the RAM on the Raspberry Pi 4.
> > > >>
> > > >> This series has only be slighlty tested. I would appreciate more test on
> > > >> the Rasbperry Pi 4 to confirm this removing the restriction.
> > > >
> > > > Hi Julien,
> > >
> > > Hi,
> > >
> > > > could you post a git branch somewhere? I can try this on my rpi4 that
> > > > already runs 4.13.
> > >
> > > I have pushed a branch based on unstable and the v2 of the series:
> > >
> > > git://xenbits.xen.org/people/julieng/xen-unstable.git
> > >
> > > branch arm-dma/v2
> > >
> >
> > I've updated my image I built with
> > https://github.com/tklengyel/xen-rpi4-builder a while ago and I've
> > defined 2048m as total_mem and Xen seems to be booting fine and passes
> > execution to dom0. With 512m being set as the Xen cmdline for dom0_mem
> > it was working. When I increased the mem for dom0 the boot is now
> > stuck at:
> >
> > [    1.427788] of_cfs_init
> > [    1.429667] of_cfs_init: OK
> > [    1.432561] clk: Not disabling unused clocks
> > [    1.437239] Waiting for root device /dev/mmcblk0p2...
> > [    1.451599] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
> > [    1.458156] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> > [    1.464729] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> > [    1.472804] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
> > [    1.479370] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> > [    1.546902] random: fast init done
> > [    1.564590] mmc1: new high speed SDIO card at address 0001
> >
> > Could this be because the DTB I compiled from a fresh checkout of
> > https://github.com/raspberrypi/linux.git branch rpi-4.19.y whereas the
> > kernel itself is from a checkout ~5 months ago? I guess that must be
> > the cause because even if I decrease the dom0_mem to 512m it still
> > gets stuck at the same spot whereas it was booting fine before.
>
> Stefano and I are testing the fix right now -- for now just set your
> Dom0 mem to less than 512m.

Actually seems to work after I recompiled the kernel and reinstalled
all kernel modules. Xen boots with 4gb RAM and dom0 boots with 2g:

xl info:
...
total_memory           : 3956
free_memory            : 1842

cat /proc/meminfo
MemTotal:        1963844 kB

I get an emergency shell during boot on the console complaining about
xenbr0 not coming up but if I just hit continue it boots fine and the
network is up. So AFAICT things are good.

Cheers,
Tamas

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

Posted by Roman Shaposhnik 1 week ago
On Tue, May 19, 2020, 7:15 PM Tamas K Lengyel <tamas.k.lengyel@gmail.com>
wrote:

> On Tue, May 19, 2020 at 5:50 PM Roman Shaposhnik <roman@zededa.com> wrote:
> >
> > On Tue, May 19, 2020 at 4:44 PM Tamas K Lengyel
> > <tamas.k.lengyel@gmail.com> wrote:
> > >
> > > On Tue, May 19, 2020 at 11:23 AM Julien Grall <julien@xen.org> wrote:
> > > >
> > > >
> > > >
> > > > On 19/05/2020 04:08, Tamas K Lengyel wrote:
> > > > > On Mon, May 18, 2020 at 5:32 AM Julien Grall <julien@xen.org>
> wrote:
> > > > >>
> > > > >> From: Julien Grall <jgrall@amazon.com>
> > > > >>
> > > > >> Hi all,
> > > > >>
> > > > >> At the moment, a user who wants to boot Xen on the Raspberry Pi 4
> can
> > > > >> only use the first GB of memory.
> > > > >>
> > > > >> This is because several devices cannot DMA above 1GB but Xen
> doesn't
> > > > >> necessarily allocate memory for Dom0 below 1GB.
> > > > >>
> > > > >> This small series is trying to address the problem by allowing a
> > > > >> platform to restrict where Dom0 banks are allocated.
> > > > >>
> > > > >> This is also a candidate for Xen 4.14. Without it, a user will
> not be
> > > > >> able to use all the RAM on the Raspberry Pi 4.
> > > > >>
> > > > >> This series has only be slighlty tested. I would appreciate more
> test on
> > > > >> the Rasbperry Pi 4 to confirm this removing the restriction.
> > > > >
> > > > > Hi Julien,
> > > >
> > > > Hi,
> > > >
> > > > > could you post a git branch somewhere? I can try this on my rpi4
> that
> > > > > already runs 4.13.
> > > >
> > > > I have pushed a branch based on unstable and the v2 of the series:
> > > >
> > > > git://xenbits.xen.org/people/julieng/xen-unstable.git
> > > >
> > > > branch arm-dma/v2
> > > >
> > >
> > > I've updated my image I built with
> > > https://github.com/tklengyel/xen-rpi4-builder a while ago and I've
> > > defined 2048m as total_mem and Xen seems to be booting fine and passes
> > > execution to dom0. With 512m being set as the Xen cmdline for dom0_mem
> > > it was working. When I increased the mem for dom0 the boot is now
> > > stuck at:
> > >
> > > [    1.427788] of_cfs_init
> > > [    1.429667] of_cfs_init: OK
> > > [    1.432561] clk: Not disabling unused clocks
> > > [    1.437239] Waiting for root device /dev/mmcblk0p2...
> > > [    1.451599] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
> > > [    1.458156] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> > > [    1.464729] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> > > [    1.472804] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
> > > [    1.479370] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> > > [    1.546902] random: fast init done
> > > [    1.564590] mmc1: new high speed SDIO card at address 0001
> > >
> > > Could this be because the DTB I compiled from a fresh checkout of
> > > https://github.com/raspberrypi/linux.git branch rpi-4.19.y whereas the
> > > kernel itself is from a checkout ~5 months ago? I guess that must be
> > > the cause because even if I decrease the dom0_mem to 512m it still
> > > gets stuck at the same spot whereas it was booting fine before.
> >
> > Stefano and I are testing the fix right now -- for now just set your
> > Dom0 mem to less than 512m.
>
> Actually seems to work after I recompiled the kernel and reinstalled
> all kernel modules. Xen boots with 4gb RAM and dom0 boots with 2g:
>
> xl info:
> ...
> total_memory           : 3956
> free_memory            : 1842
>
> cat /proc/meminfo
> MemTotal:        1963844 kB
>
> I get an emergency shell during boot on the console complaining about
> xenbr0 not coming up but if I just hit continue it boots fine and the
> network is up. So AFAICT things are good.
>

What exact version of the kernel are you using and what did you build it
from?

FWIW: 5.6.x clearly has an issue with DMA.

Thanks,
Roman.

>

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

Posted by Tamas K Lengyel 1 week ago
On Tue, May 19, 2020 at 8:28 PM Roman Shaposhnik <roman@zededa.com> wrote:
>
> On Tue, May 19, 2020, 7:15 PM Tamas K Lengyel <tamas.k.lengyel@gmail.com> wrote:
>>
>> On Tue, May 19, 2020 at 5:50 PM Roman Shaposhnik <roman@zededa.com> wrote:
>> >
>> > On Tue, May 19, 2020 at 4:44 PM Tamas K Lengyel
>> > <tamas.k.lengyel@gmail.com> wrote:
>> > >
>> > > On Tue, May 19, 2020 at 11:23 AM Julien Grall <julien@xen.org> wrote:
>> > > >
>> > > >
>> > > >
>> > > > On 19/05/2020 04:08, Tamas K Lengyel wrote:
>> > > > > On Mon, May 18, 2020 at 5:32 AM Julien Grall <julien@xen.org> wrote:
>> > > > >>
>> > > > >> From: Julien Grall <jgrall@amazon.com>
>> > > > >>
>> > > > >> Hi all,
>> > > > >>
>> > > > >> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
>> > > > >> only use the first GB of memory.
>> > > > >>
>> > > > >> This is because several devices cannot DMA above 1GB but Xen doesn't
>> > > > >> necessarily allocate memory for Dom0 below 1GB.
>> > > > >>
>> > > > >> This small series is trying to address the problem by allowing a
>> > > > >> platform to restrict where Dom0 banks are allocated.
>> > > > >>
>> > > > >> This is also a candidate for Xen 4.14. Without it, a user will not be
>> > > > >> able to use all the RAM on the Raspberry Pi 4.
>> > > > >>
>> > > > >> This series has only be slighlty tested. I would appreciate more test on
>> > > > >> the Rasbperry Pi 4 to confirm this removing the restriction.
>> > > > >
>> > > > > Hi Julien,
>> > > >
>> > > > Hi,
>> > > >
>> > > > > could you post a git branch somewhere? I can try this on my rpi4 that
>> > > > > already runs 4.13.
>> > > >
>> > > > I have pushed a branch based on unstable and the v2 of the series:
>> > > >
>> > > > git://xenbits.xen.org/people/julieng/xen-unstable.git
>> > > >
>> > > > branch arm-dma/v2
>> > > >
>> > >
>> > > I've updated my image I built with
>> > > https://github.com/tklengyel/xen-rpi4-builder a while ago and I've
>> > > defined 2048m as total_mem and Xen seems to be booting fine and passes
>> > > execution to dom0. With 512m being set as the Xen cmdline for dom0_mem
>> > > it was working. When I increased the mem for dom0 the boot is now
>> > > stuck at:
>> > >
>> > > [    1.427788] of_cfs_init
>> > > [    1.429667] of_cfs_init: OK
>> > > [    1.432561] clk: Not disabling unused clocks
>> > > [    1.437239] Waiting for root device /dev/mmcblk0p2...
>> > > [    1.451599] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
>> > > [    1.458156] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
>> > > [    1.464729] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
>> > > [    1.472804] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
>> > > [    1.479370] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
>> > > [    1.546902] random: fast init done
>> > > [    1.564590] mmc1: new high speed SDIO card at address 0001
>> > >
>> > > Could this be because the DTB I compiled from a fresh checkout of
>> > > https://github.com/raspberrypi/linux.git branch rpi-4.19.y whereas the
>> > > kernel itself is from a checkout ~5 months ago? I guess that must be
>> > > the cause because even if I decrease the dom0_mem to 512m it still
>> > > gets stuck at the same spot whereas it was booting fine before.
>> >
>> > Stefano and I are testing the fix right now -- for now just set your
>> > Dom0 mem to less than 512m.
>>
>> Actually seems to work after I recompiled the kernel and reinstalled
>> all kernel modules. Xen boots with 4gb RAM and dom0 boots with 2g:
>>
>> xl info:
>> ...
>> total_memory           : 3956
>> free_memory            : 1842
>>
>> cat /proc/meminfo
>> MemTotal:        1963844 kB
>>
>> I get an emergency shell during boot on the console complaining about
>> xenbr0 not coming up but if I just hit continue it boots fine and the
>> network is up. So AFAICT things are good.
>
>
> What exact version of the kernel are you using and what did you build it from?
>
> FWIW: 5.6.x clearly has an issue with DMA.

As I said above: https://github.com/raspberrypi/linux.git branch
rpi-4.19.y, I applied the Linux patches from the xen-rpi4-builder
repo, just changing the dom0_mem option in patch 1. I reverted the
xen-rpi4-builder a couple revisions as to not build using the DTB
overlay.

Tamas

Re: [PATCH for-4.14 0/3] Remove the 1GB limitation on Rasberry Pi 4

Posted by Stefano Stabellini 1 week ago
On Tue, 19 May 2020, Tamas K Lengyel wrote:
> On Tue, May 19, 2020 at 11:23 AM Julien Grall <julien@xen.org> wrote:
> >
> >
> >
> > On 19/05/2020 04:08, Tamas K Lengyel wrote:
> > > On Mon, May 18, 2020 at 5:32 AM Julien Grall <julien@xen.org> wrote:
> > >>
> > >> From: Julien Grall <jgrall@amazon.com>
> > >>
> > >> Hi all,
> > >>
> > >> At the moment, a user who wants to boot Xen on the Raspberry Pi 4 can
> > >> only use the first GB of memory.
> > >>
> > >> This is because several devices cannot DMA above 1GB but Xen doesn't
> > >> necessarily allocate memory for Dom0 below 1GB.
> > >>
> > >> This small series is trying to address the problem by allowing a
> > >> platform to restrict where Dom0 banks are allocated.
> > >>
> > >> This is also a candidate for Xen 4.14. Without it, a user will not be
> > >> able to use all the RAM on the Raspberry Pi 4.
> > >>
> > >> This series has only be slighlty tested. I would appreciate more test on
> > >> the Rasbperry Pi 4 to confirm this removing the restriction.
> > >
> > > Hi Julien,
> >
> > Hi,
> >
> > > could you post a git branch somewhere? I can try this on my rpi4 that
> > > already runs 4.13.
> >
> > I have pushed a branch based on unstable and the v2 of the series:
> >
> > git://xenbits.xen.org/people/julieng/xen-unstable.git
> >
> > branch arm-dma/v2
> >
> 
> I've updated my image I built with
> https://github.com/tklengyel/xen-rpi4-builder a while ago and I've
> defined 2048m as total_mem and Xen seems to be booting fine and passes
> execution to dom0. With 512m being set as the Xen cmdline for dom0_mem
> it was working. When I increased the mem for dom0 the boot is now
> stuck at:
> 
> [    1.427788] of_cfs_init
> [    1.429667] of_cfs_init: OK
> [    1.432561] clk: Not disabling unused clocks
> [    1.437239] Waiting for root device /dev/mmcblk0p2...
> [    1.451599] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
> [    1.458156] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> [    1.464729] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> [    1.472804] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
> [    1.479370] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
> [    1.546902] random: fast init done
> [    1.564590] mmc1: new high speed SDIO card at address 0001
> 
> Could this be because the DTB I compiled from a fresh checkout of
> https://github.com/raspberrypi/linux.git branch rpi-4.19.y whereas the
> kernel itself is from a checkout ~5 months ago? I guess that must be
> the cause because even if I decrease the dom0_mem to 512m it still
> gets stuck at the same spot whereas it was booting fine before.

Just so that you are aware, there is a known issue with setting dom0_mem
greater than 512M. I have a WIP patch to fix it in Linux that I plan to
send to the list soon.