Embedding the symbol table can shift sections and flip relaxation
decisions, changing code size and thus the set of emitted symbols
(e.g. gap end markers). Re-link with a regenerated table when its
size differs from the previous pass.
This mirrors commit 35de7285d508 ("Arm: do a 4th linking pass if
necessary"), which riscv didn't receive when the underlying
pass-2-vs-pass-3 check was extended to it in commit aa786d6e3467
("non-x86/symbols: check table sizes don't change between linking
passes 2 and 3").
Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
---
I faced this issue again in downstream:
https://gitlab.com/xen-project/people/olkur/xen/-/jobs/15171254706
It was found that the difference between .xen-syms.0 and .xen-syms.1 is in
alignment(?) gap between turn_on_mmu() and the end of simple_strtoull.
Specifically 0xffffffffc00c05b0 is present in pass 1 but not present in
pass 2 (where end of simple_strtoull() is just equal to ffffffffc00c05c0)
because code was shifted:
```
.xen-syms.0: file format elf64-littleriscv
...
ffffffffc00c0534: f2e794e3 bne a5,a4,ffffffffc00c045c <simple_strtoull+0x18>
ffffffffc00c0538: 00280813 addi a6,a6,2
ffffffffc00c053c: f21ff06f j ffffffffc00c045c <simple_strtoull+0x18>
ffffffffc00c0540: 00068813 mv a6,a3
ffffffffc00c0544: 00800613 li a2,8
ffffffffc00c0548: f15ff06f j ffffffffc00c045c <simple_strtoull+0x18>
ffffffffc00c054c: 00277713 andi a4,a4,2
ffffffffc00c0550: 04070463 beqz a4,ffffffffc00c0598 <simple_strtoull+0x154>
ffffffffc00c0554: fe06869b addiw a3,a3,-32
ffffffffc00c0558: 0ff6f693 zext.b a3,a3
ffffffffc00c055c: fc96879b addiw a5,a3,-55
ffffffffc00c0560: 04c7f263 bgeu a5,a2,ffffffffc00c05a4 <simple_strtoull+0x160>
ffffffffc00c0564: 02a60533 mul a0,a2,a0
ffffffffc00c0568: 00f50533 add a0,a0,a5
ffffffffc00c056c: 00180813 addi a6,a6,1
ffffffffc00c0570: 00084683 lbu a3,0(a6)
ffffffffc00c0574: 0006879b sext.w a5,a3
ffffffffc00c0578: 00d30733 add a4,t1,a3
ffffffffc00c057c: 00074703 lbu a4,0(a4)
ffffffffc00c0580: 04477893 andi a7,a4,68
ffffffffc00c0584: 02088063 beqz a7,ffffffffc00c05a4 <simple_strtoull+0x160>
ffffffffc00c0588: 00477893 andi a7,a4,4
ffffffffc00c058c: fc0880e3 beqz a7,ffffffffc00c054c <simple_strtoull+0x108>
ffffffffc00c0590: fd07879b addiw a5,a5,-48
ffffffffc00c0594: fcdff06f j ffffffffc00c0560 <simple_strtoull+0x11c>
ffffffffc00c0598: fc97879b addiw a5,a5,-55
ffffffffc00c059c: fc5ff06f j ffffffffc00c0560 <simple_strtoull+0x11c>
ffffffffc00c05a0: 00000513 li a0,0
ffffffffc00c05a4: 00058463 beqz a1,ffffffffc00c05ac <simple_strtoull+0x168>
ffffffffc00c05a8: 0105b023 sd a6,0(a1)
ffffffffc00c05ac: 00008067 ret
...
ffffffffc00c05c0 <turn_on_mmu>:
ffffffffc00c05c0: 12000073 sfence.vma
ffffffffc00c05c4: 00800293 li t0,8
ffffffffc00c05c8: 03c29293 slli t0,t0,0x3c
ffffffffc00c05cc: 000fc317 auipc t1,0xfc
ffffffffc00c05d0: a3430313 addi t1,t1,-1484 # ffffffffc01bc000 <stage1_pgtbl_root>
ffffffffc00c05d4: 00c35313 srli t1,t1,0xc
ffffffffc00c05d8: 00536333 or t1,t1,t0
ffffffffc00c05dc: 18031073 csrw satp,t1
ffffffffc00c05e0: 00050067 jr a0
ffffffffc00c05e4 <_ident_end>:
ffffffffc00c05e4: 0000 .insn 2, 0x0000
...
```
So the tool (symbols.c) emits an entry at start+size only when a gap
follows; the gap closed in pass 2. (look at the code of want_symbol_end()
in symbols.c) what leads to difference in one entry in symbols table between
passes.
---
xen/arch/riscv/Makefile | 16 ++++++++++++++--
1 file changed, 14 insertions(+), 2 deletions(-)
diff --git a/xen/arch/riscv/Makefile b/xen/arch/riscv/Makefile
index fc6b34661111..046f73f4d87c 100644
--- a/xen/arch/riscv/Makefile
+++ b/xen/arch/riscv/Makefile
@@ -50,9 +50,21 @@ $(TARGET)-syms: $(objtree)/prelink.o $(obj)/xen.lds
| $(objtree)/tools/symbols $(all_symbols) --sysv --sort \
> $(dot-target).2.S
$(MAKE) $(build)=$(@D) $(dot-target).2.o
- $(call compare-symbol-tables, $(dot-target).1.o, $(dot-target).2.o)
+ if ! { $(call compare-symbol-tables, $(dot-target).1.o, $(dot-target).2.o) >/dev/null; }; \
+ then \
+ set -e; \
+ $(LD) $(XEN_LDFLAGS) -T $(obj)/xen.lds $< $(build_id_linker) \
+ $(dot-target).2.o -o $(dot-target).2; \
+ $(NM) -pa --format=sysv $(dot-target).2 \
+ | $(objtree)/tools/symbols $(all_symbols) --sysv --sort \
+ > $(dot-target).3.S; \
+ $(MAKE) $(build)=$(@D) $(dot-target).3.o; \
+ $(call compare-symbol-tables, $(dot-target).2.o, $(dot-target).3.o); \
+ else \
+ ln -sf $(dot-target).2.o $(dot-target).3.o; \
+ fi
$(LD) $(XEN_LDFLAGS) -T $(obj)/xen.lds $< $(build_id_linker) \
- $(dot-target).2.o -o $@
+ $(dot-target).3.o -o $@
$(NM) -pa --format=sysv $@ \
| $(objtree)/tools/symbols --all-symbols --xensyms --sysv --sort \
> $@.map
--
2.54.0On 06.07.2026 17:58, Oleksii Kurochko wrote:
> Embedding the symbol table can shift sections and flip relaxation
> decisions, changing code size and thus the set of emitted symbols
> (e.g. gap end markers).
What difference in symbols is there? (And: Do you mean the ELF symbol
table or what tools/symbols emits?) On ...
> Re-link with a regenerated table when its
> size differs from the previous pass.
>
> This mirrors commit 35de7285d508 ("Arm: do a 4th linking pass if
> necessary"), which riscv didn't receive when the underlying
> pass-2-vs-pass-3 check was extended to it in commit aa786d6e3467
> ("non-x86/symbols: check table sizes don't change between linking
> passes 2 and 3").
... Arm the problem was with a symbol (dis)appearing from/in the ELF
symbol table. The situation looks to be different here. Sadly ...
> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
> ---
> I faced this issue again in downstream:
> https://gitlab.com/xen-project/people/olkur/xen/-/jobs/15171254706
>
> It was found that the difference between .xen-syms.0 and .xen-syms.1 is in
> alignment(?) gap between turn_on_mmu() and the end of simple_strtoull.
> Specifically 0xffffffffc00c05b0 is present in pass 1 but not present in
> pass 2 (where end of simple_strtoull() is just equal to ffffffffc00c05c0)
> because code was shifted:
>
> ```
> .xen-syms.0: file format elf64-littleriscv
>
> ...
>
> ffffffffc00c0534: f2e794e3 bne a5,a4,ffffffffc00c045c <simple_strtoull+0x18>
> ffffffffc00c0538: 00280813 addi a6,a6,2
> ffffffffc00c053c: f21ff06f j ffffffffc00c045c <simple_strtoull+0x18>
> ffffffffc00c0540: 00068813 mv a6,a3
> ffffffffc00c0544: 00800613 li a2,8
> ffffffffc00c0548: f15ff06f j ffffffffc00c045c <simple_strtoull+0x18>
> ffffffffc00c054c: 00277713 andi a4,a4,2
> ffffffffc00c0550: 04070463 beqz a4,ffffffffc00c0598 <simple_strtoull+0x154>
> ffffffffc00c0554: fe06869b addiw a3,a3,-32
> ffffffffc00c0558: 0ff6f693 zext.b a3,a3
> ffffffffc00c055c: fc96879b addiw a5,a3,-55
> ffffffffc00c0560: 04c7f263 bgeu a5,a2,ffffffffc00c05a4 <simple_strtoull+0x160>
> ffffffffc00c0564: 02a60533 mul a0,a2,a0
> ffffffffc00c0568: 00f50533 add a0,a0,a5
> ffffffffc00c056c: 00180813 addi a6,a6,1
> ffffffffc00c0570: 00084683 lbu a3,0(a6)
> ffffffffc00c0574: 0006879b sext.w a5,a3
> ffffffffc00c0578: 00d30733 add a4,t1,a3
> ffffffffc00c057c: 00074703 lbu a4,0(a4)
> ffffffffc00c0580: 04477893 andi a7,a4,68
> ffffffffc00c0584: 02088063 beqz a7,ffffffffc00c05a4 <simple_strtoull+0x160>
> ffffffffc00c0588: 00477893 andi a7,a4,4
> ffffffffc00c058c: fc0880e3 beqz a7,ffffffffc00c054c <simple_strtoull+0x108>
> ffffffffc00c0590: fd07879b addiw a5,a5,-48
> ffffffffc00c0594: fcdff06f j ffffffffc00c0560 <simple_strtoull+0x11c>
> ffffffffc00c0598: fc97879b addiw a5,a5,-55
> ffffffffc00c059c: fc5ff06f j ffffffffc00c0560 <simple_strtoull+0x11c>
> ffffffffc00c05a0: 00000513 li a0,0
> ffffffffc00c05a4: 00058463 beqz a1,ffffffffc00c05ac <simple_strtoull+0x168>
> ffffffffc00c05a8: 0105b023 sd a6,0(a1)
> ffffffffc00c05ac: 00008067 ret
> ...
>
> ffffffffc00c05c0 <turn_on_mmu>:
> ffffffffc00c05c0: 12000073 sfence.vma
> ffffffffc00c05c4: 00800293 li t0,8
> ffffffffc00c05c8: 03c29293 slli t0,t0,0x3c
> ffffffffc00c05cc: 000fc317 auipc t1,0xfc
> ffffffffc00c05d0: a3430313 addi t1,t1,-1484 # ffffffffc01bc000 <stage1_pgtbl_root>
> ffffffffc00c05d4: 00c35313 srli t1,t1,0xc
> ffffffffc00c05d8: 00536333 or t1,t1,t0
> ffffffffc00c05dc: 18031073 csrw satp,t1
> ffffffffc00c05e0: 00050067 jr a0
>
> ffffffffc00c05e4 <_ident_end>:
> ffffffffc00c05e4: 0000 .insn 2, 0x0000
> ...
> ```
... with all of this disassembly provided it still doesn't become clear what
changed from .xen-syms.0 to .xen-syms.1.
Jan
On 7/6/26 6:13 PM, Jan Beulich wrote:
> On 06.07.2026 17:58, Oleksii Kurochko wrote:
>> Embedding the symbol table can shift sections and flip relaxation
>> decisions, changing code size and thus the set of emitted symbols
>> (e.g. gap end markers).
>
> What difference in symbols is there? (And: Do you mean the ELF symbol
> table or what tools/symbols emits?) On ...
The ELF symbol tables of .xen-syms.0 and .xen-syms.1 are identical
(same names, types, and count; only addresses past a certain point
differ). The difference is in what tools/symbols emits: the
address-only "end of symbol" entry produced when want_symbol_end()
is true, i.e. when a sized text symbol's end doesn't reach the next
symbol's address:
static bool want_symbol_end(unsigned int idx)
{
return table[idx].size &&
(idx + 1 == table_cnt ||
table[idx].addr + table[idx].size < table[idx + 1].addr);
}
In pass 1 (generated from .xen-syms.0): simple_strtoull is at
0xffffffffc00c0444 with size 0x16c, ending at 0xffffffffc00c05b0,
while turn_on_mmu (alignment-pinned) starts at 0xffffffffc00c05c0.
16-byte gap -> the end marker at ...c05b0 is emitted.
In pass 2 (generated from .xen-syms.1): simple_strtoull moved to
0xffffffffc00c0454 (same size), ending at exactly 0xffffffffc00c05c0
== turn_on_mmu. No gap -> no end marker. Hence symbols_addresses
shrinks by 8 and symbols_names by 1 (the marker's empty name).
>
>> Re-link with a regenerated table when its
>> size differs from the previous pass.
>>
>> This mirrors commit 35de7285d508 ("Arm: do a 4th linking pass if
>> necessary"), which riscv didn't receive when the underlying
>> pass-2-vs-pass-3 check was extended to it in commit aa786d6e3467
>> ("non-x86/symbols: check table sizes don't change between linking
>> passes 2 and 3").
>
> ... Arm the problem was with a symbol (dis)appearing from/in the ELF
> symbol table. The situation looks to be different here. Sadly ...
I will rephrase then this paragraph to:
This takes the same remedy as commit 35de7285d508 ("Arm: do a 4th
linking pass if necessary"), though the underlying cause differs:
here no symbol (dis)appears from the ELF symbol table; instead the
set of end markers emitted by tools/symbols legitimately changes,
because want_symbol_end() depends on code layout, and layout differs
between passes due to linker relaxation.
Or it could be just dropped.
>
>> Signed-off-by: Oleksii Kurochko <oleksii.kurochko@gmail.com>
>> ---
>> I faced this issue again in downstream:
>> https://gitlab.com/xen-project/people/olkur/xen/-/jobs/15171254706
>>
>> It was found that the difference between .xen-syms.0 and .xen-syms.1 is in
>> alignment(?) gap between turn_on_mmu() and the end of simple_strtoull.
>> Specifically 0xffffffffc00c05b0 is present in pass 1 but not present in
>> pass 2 (where end of simple_strtoull() is just equal to ffffffffc00c05c0)
>> because code was shifted:
>>
>> ```
>> .xen-syms.0: file format elf64-littleriscv
>>
>> ...
>>
>> ffffffffc00c0534: f2e794e3 bne a5,a4,ffffffffc00c045c <simple_strtoull+0x18>
>> ffffffffc00c0538: 00280813 addi a6,a6,2
>> ffffffffc00c053c: f21ff06f j ffffffffc00c045c <simple_strtoull+0x18>
>> ffffffffc00c0540: 00068813 mv a6,a3
>> ffffffffc00c0544: 00800613 li a2,8
>> ffffffffc00c0548: f15ff06f j ffffffffc00c045c <simple_strtoull+0x18>
>> ffffffffc00c054c: 00277713 andi a4,a4,2
>> ffffffffc00c0550: 04070463 beqz a4,ffffffffc00c0598 <simple_strtoull+0x154>
>> ffffffffc00c0554: fe06869b addiw a3,a3,-32
>> ffffffffc00c0558: 0ff6f693 zext.b a3,a3
>> ffffffffc00c055c: fc96879b addiw a5,a3,-55
>> ffffffffc00c0560: 04c7f263 bgeu a5,a2,ffffffffc00c05a4 <simple_strtoull+0x160>
>> ffffffffc00c0564: 02a60533 mul a0,a2,a0
>> ffffffffc00c0568: 00f50533 add a0,a0,a5
>> ffffffffc00c056c: 00180813 addi a6,a6,1
>> ffffffffc00c0570: 00084683 lbu a3,0(a6)
>> ffffffffc00c0574: 0006879b sext.w a5,a3
>> ffffffffc00c0578: 00d30733 add a4,t1,a3
>> ffffffffc00c057c: 00074703 lbu a4,0(a4)
>> ffffffffc00c0580: 04477893 andi a7,a4,68
>> ffffffffc00c0584: 02088063 beqz a7,ffffffffc00c05a4 <simple_strtoull+0x160>
>> ffffffffc00c0588: 00477893 andi a7,a4,4
>> ffffffffc00c058c: fc0880e3 beqz a7,ffffffffc00c054c <simple_strtoull+0x108>
>> ffffffffc00c0590: fd07879b addiw a5,a5,-48
>> ffffffffc00c0594: fcdff06f j ffffffffc00c0560 <simple_strtoull+0x11c>
>> ffffffffc00c0598: fc97879b addiw a5,a5,-55
>> ffffffffc00c059c: fc5ff06f j ffffffffc00c0560 <simple_strtoull+0x11c>
>> ffffffffc00c05a0: 00000513 li a0,0
>> ffffffffc00c05a4: 00058463 beqz a1,ffffffffc00c05ac <simple_strtoull+0x168>
>> ffffffffc00c05a8: 0105b023 sd a6,0(a1)
>> ffffffffc00c05ac: 00008067 ret
>> ...
>>
>> ffffffffc00c05c0 <turn_on_mmu>:
>> ffffffffc00c05c0: 12000073 sfence.vma
>> ffffffffc00c05c4: 00800293 li t0,8
>> ffffffffc00c05c8: 03c29293 slli t0,t0,0x3c
>> ffffffffc00c05cc: 000fc317 auipc t1,0xfc
>> ffffffffc00c05d0: a3430313 addi t1,t1,-1484 # ffffffffc01bc000 <stage1_pgtbl_root>
>> ffffffffc00c05d4: 00c35313 srli t1,t1,0xc
>> ffffffffc00c05d8: 00536333 or t1,t1,t0
>> ffffffffc00c05dc: 18031073 csrw satp,t1
>> ffffffffc00c05e0: 00050067 jr a0
>>
>> ffffffffc00c05e4 <_ident_end>:
>> ffffffffc00c05e4: 0000 .insn 2, 0x0000
>> ...
>> ```
>
> ... with all of this disassembly provided it still doesn't become clear what
> changed from .xen-syms.0 to .xen-syms.1.
The disassembly was meant to show the 16-byte padding gap between the
end of simple_strtoull() and (aligned) turn_on_mmu() in .xen-syms.0.
In .xen-syms.1 that gap is gone, which is what removes the end marker
and shrinks symbols_addresses by 8 and symbols_names by 1, as
explained above.
The actual code change is earlier in .text. Inserting the symbol
table (~24k of .rodata) pushes .init.text from 0xffffffffc0148000
to 0xffffffffc014e000. Several calls from .text into .init.text
(e.g. to alloc_boot_pages()/init_boot_pages()) thereby cross JAL's
±1MB reach, so the linker can no longer relax them:
.xen-syms.0:
<caller>: jal ffffffffc0152000 <alloc_boot_pages>
.xen-syms.1:
<caller>: auipc ra,0xfc
jalr 1044(ra) # ffffffffc015800c <alloc_boot_pages>
Four such call sites grow .text by 16 bytes in total, which is what
shifts simple_strtoull by +0x10 and closes the padding gap before
the (aligned) turn_on_mmu.
~ Oleksii
On 07.07.2026 10:19, Oleksii Kurochko wrote:
> On 7/6/26 6:13 PM, Jan Beulich wrote:
>> On 06.07.2026 17:58, Oleksii Kurochko wrote:
>>> Embedding the symbol table can shift sections and flip relaxation
>>> decisions, changing code size and thus the set of emitted symbols
>>> (e.g. gap end markers).
>>
>> What difference in symbols is there? (And: Do you mean the ELF symbol
>> table or what tools/symbols emits?) On ...
>
> The ELF symbol tables of .xen-syms.0 and .xen-syms.1 are identical
> (same names, types, and count; only addresses past a certain point
> differ). The difference is in what tools/symbols emits: the
> address-only "end of symbol" entry produced when want_symbol_end()
> is true, i.e. when a sized text symbol's end doesn't reach the next
> symbol's address:
>
> static bool want_symbol_end(unsigned int idx)
> {
> return table[idx].size &&
> (idx + 1 == table_cnt ||
> table[idx].addr + table[idx].size < table[idx + 1].addr);
> }
>
> In pass 1 (generated from .xen-syms.0): simple_strtoull is at
> 0xffffffffc00c0444 with size 0x16c, ending at 0xffffffffc00c05b0,
> while turn_on_mmu (alignment-pinned) starts at 0xffffffffc00c05c0.
> 16-byte gap -> the end marker at ...c05b0 is emitted.
>
> In pass 2 (generated from .xen-syms.1): simple_strtoull moved to
> 0xffffffffc00c0454 (same size), ending at exactly 0xffffffffc00c05c0
> == turn_on_mmu. No gap -> no end marker. Hence symbols_addresses
> shrinks by 8 and symbols_names by 1 (the marker's empty name).
>
>>
>>> Re-link with a regenerated table when its
>>> size differs from the previous pass.
>>>
>>> This mirrors commit 35de7285d508 ("Arm: do a 4th linking pass if
>>> necessary"), which riscv didn't receive when the underlying
>>> pass-2-vs-pass-3 check was extended to it in commit aa786d6e3467
>>> ("non-x86/symbols: check table sizes don't change between linking
>>> passes 2 and 3").
>>
>> ... Arm the problem was with a symbol (dis)appearing from/in the ELF
>> symbol table. The situation looks to be different here. Sadly ...
>
> I will rephrase then this paragraph to:
>
> This takes the same remedy as commit 35de7285d508 ("Arm: do a 4th
> linking pass if necessary"), though the underlying cause differs:
> here no symbol (dis)appears from the ELF symbol table; instead the
> set of end markers emitted by tools/symbols legitimately changes,
> because want_symbol_end() depends on code layout, and layout differs
> between passes due to linker relaxation.
>
> Or it could be just dropped.
Please don't drop it.
Jan
© 2016 - 2026 Red Hat, Inc.