arch/x86/kernel/sys_x86_64.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Commit 03475167fda5 ("x86: Increase brk randomness entropy for 64-bit
systems") increased the brk randomness from 32 MiB to 1 GiB. MAP_32BIT
looks between 1 GiB and 2 GiB for an unmapped area. Depending on the
randomization, a heap starting high enough and being big enough can use
up all the area that MAP_32BIT looks at, leading to allocation failures.
For example, if the heap starts at 800 MiB and is 1.2 GiB large,
allocations with MAP_32BIT will always fail despite unused addresses
below 800 MiB.
Lower the begin of the address space which is available to MAP32_BIT
from 0x40000000 to 0x10000000 to give mmap more room if the randomly
allocated brk address turns out to be unfavourable high. This allows
mmap to allocate up to 75% more space.
Signed-off-by: Simon Liebold <simonlie@amazon.de>
---
Notes:
Background: LuaJIT v2.0 uses MAP32_BIT for allocating memory. Because of
the restriction of MAP32_BIT to limit all allocation of mmap to the
address space from 1 GiB to 2 GiB, LuaJIT v2.0 can fail to work,
depending on the random location of brk.
I tested this change using the following reproducer:
int main() {
uintptr_t mmap_end = 0x80000000;
uintptr_t heap_start = (uintptr_t)sbrk(0);
printf("heap start: %p\n", heap_start);
uintptr_t alloc_size = mmap_end - heap_start;
uintptr_t heap_end = (uintptr_t)sbrk(alloc_size);
printf("heap allocated until: %p\n", heap_end);
void* addr = mmap(NULL,
8,
PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_32BIT,
-1,
0);
if(addr == MAP_FAILED)
printf("mmap allocation failed\n");
else
printf("mmap allocation at %p\n", addr);
return 0;
}
Before the change, the allocation failed:
[root@localhost ~]# ./repro
heap start: 0x24bce000
heap allocated until: 0x24bef000
mmap allocation failed
After the change, it succeeded:
[root@localhost ~]# ./repro
heap start: 0x38f24000
heap allocated until: 0x38f45000
mmap allocation at 0x11962000
Note that this does not guarantee to succeed. If the randomized heap
start is below 0x10000000, it still fails.
arch/x86/kernel/sys_x86_64.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/x86/kernel/sys_x86_64.c b/arch/x86/kernel/sys_x86_64.c
index 776ae6fa7f2d..29c6277aa31e 100644
--- a/arch/x86/kernel/sys_x86_64.c
+++ b/arch/x86/kernel/sys_x86_64.c
@@ -100,7 +100,7 @@ static void find_start_end(unsigned long addr, unsigned long flags,
conflicts with the heap, but we assume that glibc
malloc knows how to fall back to mmap. Give it 1GB
of playground for now. -AK */
- *begin = 0x40000000;
+ *begin = 0x10000000;
*end = 0x80000000;
if (current->flags & PF_RANDOMIZE) {
*begin = randomize_page(*begin, 0x02000000);
base-commit: 260f6f4fda93c8485c8037865c941b42b9cba5d2
--
2.47.3
Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597On 8/25/25 03:48, Simon Liebold wrote:
> Commit 03475167fda5 ("x86: Increase brk randomness entropy for 64-bit
> systems") increased the brk randomness from 32 MiB to 1 GiB. MAP_32BIT
> looks between 1 GiB and 2 GiB for an unmapped area. Depending on the
> randomization, a heap starting high enough and being big enough can use
> up all the area that MAP_32BIT looks at, leading to allocation failures.
Isn't that still a really unreasonably gigantic heap?
Would you mind posting some actual /proc/$pid/maps output from one of
the failure cases?
Dave Hansen <dave.hansen@intel.com> writes:
> On 8/25/25 03:48, Simon Liebold wrote:
>> Commit 03475167fda5 ("x86: Increase brk randomness entropy for 64-bit
>> systems") increased the brk randomness from 32 MiB to 1 GiB. MAP_32BIT
>> looks between 1 GiB and 2 GiB for an unmapped area. Depending on the
>> randomization, a heap starting high enough and being big enough can use
>> up all the area that MAP_32BIT looks at, leading to allocation failures.
>
> Isn't that still a really unreasonably gigantic heap?
>
> Would you mind posting some actual /proc/$pid/maps output from one of
> the failure cases?
Hello Dave,
we are seeing heaps this large on real workloads. This is a Lua
application using LuaJit v2.0.
This is an excerpt from the output of one of the failure cases:
00400000-00566000 r-xp 00000000 103:01 4476567 [...]/bin/httpd.orig
00765000-0076b000 r--p 00165000 103:01 4476567 [...]/bin/httpd.orig
0076b000-00772000 rw-p 0016b000 103:01 4476567 [...]/bin/httpd.orig
00772000-00778000 rw-p 00000000 00:00 0
34a21000-35021000 rw-p 00000000 00:00 0 [heap]
35021000-82ea7000 rw-p 00000000 00:00 0 [heap]
7fee7c0ba000-7fee7c11f000 r-xp 00000000 103:01 3836609 [...]/lib/libluajit-5.1.so
7fee7c11f000-7fee7c31f000 ---p 00065000 103:01 3836609 [...]/lib/libluajit-5.1.so
7fee7c31f000-7fee7c321000 r--p 00065000 103:01 3836609 [...]/lib/libluajit-5.1.so
7fee7c321000-7fee7c322000 rw-p 00067000 103:01 3836609 [...]/lib/libluajit-5.1.so
[Other maps at high addresses...]
Regards,
Simon Liebold
Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597
Simon Liebold <simonlie@amazon.de> writes:
> Dave Hansen <dave.hansen@intel.com> writes:
>
>> On 8/25/25 03:48, Simon Liebold wrote:
>>> Commit 03475167fda5 ("x86: Increase brk randomness entropy for 64-bit
>>> systems") increased the brk randomness from 32 MiB to 1 GiB. MAP_32BIT
>>> looks between 1 GiB and 2 GiB for an unmapped area. Depending on the
>>> randomization, a heap starting high enough and being big enough can use
>>> up all the area that MAP_32BIT looks at, leading to allocation failures.
>>
>> Isn't that still a really unreasonably gigantic heap?
>>
>> Would you mind posting some actual /proc/$pid/maps output from one of
>> the failure cases?
>
> Hello Dave,
>
> we are seeing heaps this large on real workloads. This is a Lua
> application using LuaJit v2.0.
>
> This is an excerpt from the output of one of the failure cases:
>
> 00400000-00566000 r-xp 00000000 103:01 4476567 [...]/bin/httpd.orig
> 00765000-0076b000 r--p 00165000 103:01 4476567 [...]/bin/httpd.orig
> 0076b000-00772000 rw-p 0016b000 103:01 4476567 [...]/bin/httpd.orig
> 00772000-00778000 rw-p 00000000 00:00 0
> 34a21000-35021000 rw-p 00000000 00:00 0 [heap]
> 35021000-82ea7000 rw-p 00000000 00:00 0 [heap]
> 7fee7c0ba000-7fee7c11f000 r-xp 00000000 103:01 3836609 [...]/lib/libluajit-5.1.so
> 7fee7c11f000-7fee7c31f000 ---p 00065000 103:01 3836609 [...]/lib/libluajit-5.1.so
> 7fee7c31f000-7fee7c321000 r--p 00065000 103:01 3836609 [...]/lib/libluajit-5.1.so
> 7fee7c321000-7fee7c322000 rw-p 00067000 103:01 3836609 [...]/lib/libluajit-5.1.so
> [Other maps at high addresses...]
>
> Regards,
>
> Simon Liebold
Hi Dave,
I wanted to follow up on our previous discussion about the brk
randomization and MAP_32BIT allocation failures.
The heap overflow we're seeing is standard LuaJIT v2.0 behavior under
certain workloads, not just custom programs with unusual memory
patterns. Given LuaJIT's widespread production use, these MAP_32BIT
allocation failures could impact many users.
While I understand this might be considered an edge case, it does make
MAP_32BIT allocations less reliable in practice.
Do you have any thoughts on this?
Regards,
Simon Liebold
Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597
© 2016 - 2025 Red Hat, Inc.