Makefile | 4 +++- init/Kconfig | 15 +++++++++++++++ lib/Kconfig.debug | 15 +++++++++++++++ mm/vmalloc.c | 1 + rust/Makefile | 26 ++++++++++++++++++++++---- rust/exports.c | 5 ++++- rust/helpers/atomic.c | 5 ----- rust/helpers/helpers.c | 31 +++++++++++++++++++++++++++++++ scripts/Makefile.build | 5 ++++- scripts/atomic/gen-rust-atomic-helpers.sh | 5 ----- 10 files changed, 95 insertions(+), 17 deletions(-)
Currently the only way for Rust code to call a static inline function is
to go through a helper in rust/helpers/. This introduces performance
costs due to additional function calls and also clutters backtraces and
flame graphs with helper symbols.
To get rid of these helper symbols, provide functionality to inline
helpers into Rust using llvm-link. This option complements full LTO, by
being much cheaper and avoiding incompatibility with BTF.
I ran a microbenchmark showing the benefit of this. All the benchmark
does is call refcount_inc() in a loop. This was chosen since refcounting
is quite hot in Binder. The results are that Rust spends 6.35 ns per
call vs 5.73 ns per call in C. When enabling this option, the two
languages become equally fast, and disassembly confirms the exact same
machine code is used (in particular there is no call to
rust_helper_refcount_inc). Benchmarking Binder also results in an
improvement from this change.
This patch is complementary to:
https://lore.kernel.org/all/20251202-define-rust-helper-v1-0-a2e13cbc17a6@google.com/
Signed-off-by: Alice Ryhl <aliceryhl@google.com>
---
Alice Ryhl (1):
vmalloc: export vrealloc_node_align_noprof
Gary Guo (3):
rust: helpers: #define __rust_helper
kbuild: rust: add `CONFIG_RUSTC_CLANG_LLVM_COMPATIBLE`
build: rust: provide an option to inline C helpers into Rust
Makefile | 4 +++-
init/Kconfig | 15 +++++++++++++++
lib/Kconfig.debug | 15 +++++++++++++++
mm/vmalloc.c | 1 +
rust/Makefile | 26 ++++++++++++++++++++++----
rust/exports.c | 5 ++++-
rust/helpers/atomic.c | 5 -----
rust/helpers/helpers.c | 31 +++++++++++++++++++++++++++++++
scripts/Makefile.build | 5 ++++-
scripts/atomic/gen-rust-atomic-helpers.sh | 5 -----
10 files changed, 95 insertions(+), 17 deletions(-)
---
base-commit: 54e3eae855629702c566bd2e130d9f40e7f35bde
change-id: 20251202-inline-helpers-996f4db65e18
Best regards,
--
Alice Ryhl <aliceryhl@google.com>
Hi Alice, Alice Ryhl <aliceryhl@google.com> writes: > Currently the only way for Rust code to call a static inline function is > to go through a helper in rust/helpers/. This introduces performance > costs due to additional function calls and also clutters backtraces and > flame graphs with helper symbols. > > To get rid of these helper symbols, provide functionality to inline > helpers into Rust using llvm-link. This option complements full LTO, by > being much cheaper and avoiding incompatibility with BTF. > > I ran a microbenchmark showing the benefit of this. All the benchmark > does is call refcount_inc() in a loop. This was chosen since refcounting > is quite hot in Binder. The results are that Rust spends 6.35 ns per > call vs 5.73 ns per call in C. When enabling this option, the two > languages become equally fast, and disassembly confirms the exact same > machine code is used (in particular there is no call to > rust_helper_refcount_inc). Benchmarking Binder also results in an > improvement from this change. > > This patch is complementary to: > https://lore.kernel.org/all/20251202-define-rust-helper-v1-0-a2e13cbc17a6@google.com/ > > Signed-off-by: Alice Ryhl <aliceryhl@google.com> I get the following modpost errors when building with this applied on top of v6.19-rc4: ERROR: modpost: "__SCK__WARN_trap" [drivers/gpu/drm/nova/nova.ko] undefined! ERROR: modpost: "__SCK__WARN_trap" [drivers/gpu/nova-core/nova_core.ko] undefined! ERROR: modpost: "__SCK__WARN_trap" [drivers/block/rnull/rnull_mod.ko] undefined! ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_minimal.ko] undefined! ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_misc_device.ko] undefined! ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_print.ko] undefined! ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_dma.ko] undefined! ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_driver_pci.ko] undefined! ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_driver_platform.ko] undefined! ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_driver_faux.ko] undefined! I also applied the series adding the inline macro [1]. Best regards, Andreas Hindborg [1] https://lore.kernel.org/rust-for-linux/20260105-define-rust-helper-v2-0-51da5f454a67@google.com/
On Wed, Jan 07, 2026 at 01:23:38PM +0100, Andreas Hindborg wrote: > Hi Alice, > > Alice Ryhl <aliceryhl@google.com> writes: > > > Currently the only way for Rust code to call a static inline function is > > to go through a helper in rust/helpers/. This introduces performance > > costs due to additional function calls and also clutters backtraces and > > flame graphs with helper symbols. > > > > To get rid of these helper symbols, provide functionality to inline > > helpers into Rust using llvm-link. This option complements full LTO, by > > being much cheaper and avoiding incompatibility with BTF. > > > > I ran a microbenchmark showing the benefit of this. All the benchmark > > does is call refcount_inc() in a loop. This was chosen since refcounting > > is quite hot in Binder. The results are that Rust spends 6.35 ns per > > call vs 5.73 ns per call in C. When enabling this option, the two > > languages become equally fast, and disassembly confirms the exact same > > machine code is used (in particular there is no call to > > rust_helper_refcount_inc). Benchmarking Binder also results in an > > improvement from this change. > > > > This patch is complementary to: > > https://lore.kernel.org/all/20251202-define-rust-helper-v1-0-a2e13cbc17a6@google.com/ > > > > Signed-off-by: Alice Ryhl <aliceryhl@google.com> > > I get the following modpost errors when building with this applied on top > of v6.19-rc4: > > ERROR: modpost: "__SCK__WARN_trap" [drivers/gpu/drm/nova/nova.ko] undefined! > ERROR: modpost: "__SCK__WARN_trap" [drivers/gpu/nova-core/nova_core.ko] undefined! > ERROR: modpost: "__SCK__WARN_trap" [drivers/block/rnull/rnull_mod.ko] undefined! > ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_minimal.ko] undefined! > ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_misc_device.ko] undefined! > ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_print.ko] undefined! > ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_dma.ko] undefined! > ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_driver_pci.ko] undefined! > ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_driver_platform.ko] undefined! > ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_driver_faux.ko] undefined! Looks like it used the wrong static_call_mod() version, was MODULE defined?
"Peter Zijlstra" <peterz@infradead.org> writes: > On Wed, Jan 07, 2026 at 01:23:38PM +0100, Andreas Hindborg wrote: >> Hi Alice, >> >> Alice Ryhl <aliceryhl@google.com> writes: >> >> > Currently the only way for Rust code to call a static inline function is >> > to go through a helper in rust/helpers/. This introduces performance >> > costs due to additional function calls and also clutters backtraces and >> > flame graphs with helper symbols. >> > >> > To get rid of these helper symbols, provide functionality to inline >> > helpers into Rust using llvm-link. This option complements full LTO, by >> > being much cheaper and avoiding incompatibility with BTF. >> > >> > I ran a microbenchmark showing the benefit of this. All the benchmark >> > does is call refcount_inc() in a loop. This was chosen since refcounting >> > is quite hot in Binder. The results are that Rust spends 6.35 ns per >> > call vs 5.73 ns per call in C. When enabling this option, the two >> > languages become equally fast, and disassembly confirms the exact same >> > machine code is used (in particular there is no call to >> > rust_helper_refcount_inc). Benchmarking Binder also results in an >> > improvement from this change. >> > >> > This patch is complementary to: >> > https://lore.kernel.org/all/20251202-define-rust-helper-v1-0-a2e13cbc17a6@google.com/ >> > >> > Signed-off-by: Alice Ryhl <aliceryhl@google.com> >> >> I get the following modpost errors when building with this applied on top >> of v6.19-rc4: >> >> ERROR: modpost: "__SCK__WARN_trap" [drivers/gpu/drm/nova/nova.ko] undefined! >> ERROR: modpost: "__SCK__WARN_trap" [drivers/gpu/nova-core/nova_core.ko] undefined! >> ERROR: modpost: "__SCK__WARN_trap" [drivers/block/rnull/rnull_mod.ko] undefined! >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_minimal.ko] undefined! >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_misc_device.ko] undefined! >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_print.ko] undefined! >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_dma.ko] undefined! >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_driver_pci.ko] undefined! >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_driver_platform.ko] undefined! >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_driver_faux.ko] undefined! > > Looks like it used the wrong static_call_mod() version, was MODULE > defined? CONFIG_MODULES=y, yes I built without the patch first, then applied the series, enabled the option via menuconfig and ran the build. I thought maybe some dependency check is messed up so I retried the build from a clean state. Same result. Best regards, Andreas Hindborg
On Wed, Jan 07, 2026 at 02:12:10PM +0100, Andreas Hindborg wrote: > "Peter Zijlstra" <peterz@infradead.org> writes: > > > On Wed, Jan 07, 2026 at 01:23:38PM +0100, Andreas Hindborg wrote: > >> Hi Alice, > >> > >> Alice Ryhl <aliceryhl@google.com> writes: > >> > >> > Currently the only way for Rust code to call a static inline function is > >> > to go through a helper in rust/helpers/. This introduces performance > >> > costs due to additional function calls and also clutters backtraces and > >> > flame graphs with helper symbols. > >> > > >> > To get rid of these helper symbols, provide functionality to inline > >> > helpers into Rust using llvm-link. This option complements full LTO, by > >> > being much cheaper and avoiding incompatibility with BTF. > >> > > >> > I ran a microbenchmark showing the benefit of this. All the benchmark > >> > does is call refcount_inc() in a loop. This was chosen since refcounting > >> > is quite hot in Binder. The results are that Rust spends 6.35 ns per > >> > call vs 5.73 ns per call in C. When enabling this option, the two > >> > languages become equally fast, and disassembly confirms the exact same > >> > machine code is used (in particular there is no call to > >> > rust_helper_refcount_inc). Benchmarking Binder also results in an > >> > improvement from this change. > >> > > >> > This patch is complementary to: > >> > https://lore.kernel.org/all/20251202-define-rust-helper-v1-0-a2e13cbc17a6@google.com/ > >> > > >> > Signed-off-by: Alice Ryhl <aliceryhl@google.com> > >> > >> I get the following modpost errors when building with this applied on top > >> of v6.19-rc4: > >> > >> ERROR: modpost: "__SCK__WARN_trap" [drivers/gpu/drm/nova/nova.ko] undefined! > >> ERROR: modpost: "__SCK__WARN_trap" [drivers/gpu/nova-core/nova_core.ko] undefined! > >> ERROR: modpost: "__SCK__WARN_trap" [drivers/block/rnull/rnull_mod.ko] undefined! > >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_minimal.ko] undefined! > >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_misc_device.ko] undefined! > >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_print.ko] undefined! > >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_dma.ko] undefined! > >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_driver_pci.ko] undefined! > >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_driver_platform.ko] undefined! > >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_driver_faux.ko] undefined! > > > > Looks like it used the wrong static_call_mod() version, was MODULE > > defined? > > CONFIG_MODULES=y, yes > > I built without the patch first, then applied the series, enabled the > option via menuconfig and ran the build. I thought maybe some dependency > check is messed up so I retried the build from a clean state. Same > result. No, I mean -DMODULE. Note how the quiet_cmd_bindgen target has -DMODULE, but the new quiet_cmd_rust_helper target does not.
"Peter Zijlstra" <peterz@infradead.org> writes:
> On Wed, Jan 07, 2026 at 02:12:10PM +0100, Andreas Hindborg wrote:
>> "Peter Zijlstra" <peterz@infradead.org> writes:
>>
>> > On Wed, Jan 07, 2026 at 01:23:38PM +0100, Andreas Hindborg wrote:
>> >> Hi Alice,
>> >>
>> >> Alice Ryhl <aliceryhl@google.com> writes:
>> >>
>> >> > Currently the only way for Rust code to call a static inline function is
>> >> > to go through a helper in rust/helpers/. This introduces performance
>> >> > costs due to additional function calls and also clutters backtraces and
>> >> > flame graphs with helper symbols.
>> >> >
>> >> > To get rid of these helper symbols, provide functionality to inline
>> >> > helpers into Rust using llvm-link. This option complements full LTO, by
>> >> > being much cheaper and avoiding incompatibility with BTF.
>> >> >
>> >> > I ran a microbenchmark showing the benefit of this. All the benchmark
>> >> > does is call refcount_inc() in a loop. This was chosen since refcounting
>> >> > is quite hot in Binder. The results are that Rust spends 6.35 ns per
>> >> > call vs 5.73 ns per call in C. When enabling this option, the two
>> >> > languages become equally fast, and disassembly confirms the exact same
>> >> > machine code is used (in particular there is no call to
>> >> > rust_helper_refcount_inc). Benchmarking Binder also results in an
>> >> > improvement from this change.
>> >> >
>> >> > This patch is complementary to:
>> >> > https://lore.kernel.org/all/20251202-define-rust-helper-v1-0-a2e13cbc17a6@google.com/
>> >> >
>> >> > Signed-off-by: Alice Ryhl <aliceryhl@google.com>
>> >>
>> >> I get the following modpost errors when building with this applied on top
>> >> of v6.19-rc4:
>> >>
>> >> ERROR: modpost: "__SCK__WARN_trap" [drivers/gpu/drm/nova/nova.ko] undefined!
>> >> ERROR: modpost: "__SCK__WARN_trap" [drivers/gpu/nova-core/nova_core.ko] undefined!
>> >> ERROR: modpost: "__SCK__WARN_trap" [drivers/block/rnull/rnull_mod.ko] undefined!
>> >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_minimal.ko] undefined!
>> >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_misc_device.ko] undefined!
>> >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_print.ko] undefined!
>> >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_dma.ko] undefined!
>> >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_driver_pci.ko] undefined!
>> >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_driver_platform.ko] undefined!
>> >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_driver_faux.ko] undefined!
>> >
>> > Looks like it used the wrong static_call_mod() version, was MODULE
>> > defined?
>>
>> CONFIG_MODULES=y, yes
>>
>> I built without the patch first, then applied the series, enabled the
>> option via menuconfig and ran the build. I thought maybe some dependency
>> check is messed up so I retried the build from a clean state. Same
>> result.
>
> No, I mean -DMODULE. Note how the quiet_cmd_bindgen target has -DMODULE,
> but the new quiet_cmd_rust_helper target does not.
Ah, that did the trick, thanks!
diff --git a/rust/Makefile b/rust/Makefile
index 5365d53b6cf96..08d3dc1038cf5 100644
--- a/rust/Makefile
+++ b/rust/Makefile
@@ -483,7 +483,7 @@ $(obj)/bindings/bindings_helpers_generated.rs: $(src)/helpers/helpers.c FORCE
quiet_cmd_rust_helper = HELPER $@
cmd_rust_helper = \
- $(CC) $(filter-out $(CFLAGS_REMOVE_helpers/helpers.o), $(c_flags)) -c -g0 $< -emit-llvm -o $@
+ $(CC) $(filter-out $(CFLAGS_REMOVE_helpers/helpers.o), $(c_flags)) -DMODULE -c -g0 $< -emit-llvm -o $@
$(obj)/helpers/helpers.bc: $(obj)/helpers/helpers.c FORCE
+$(call if_changed_dep,rust_helper)
Is -DMOUDLE always appropriate to pass to the helpers? The helpers are
also inlined into non module code.
Best regards,
Andreas Hindborg
On Wed, Jan 7, 2026 at 2:40 PM Andreas Hindborg <a.hindborg@kernel.org> wrote: > > "Peter Zijlstra" <peterz@infradead.org> writes: > > > On Wed, Jan 07, 2026 at 02:12:10PM +0100, Andreas Hindborg wrote: > >> "Peter Zijlstra" <peterz@infradead.org> writes: > >> > >> > On Wed, Jan 07, 2026 at 01:23:38PM +0100, Andreas Hindborg wrote: > >> >> Hi Alice, > >> >> > >> >> Alice Ryhl <aliceryhl@google.com> writes: > >> >> > >> >> > Currently the only way for Rust code to call a static inline function is > >> >> > to go through a helper in rust/helpers/. This introduces performance > >> >> > costs due to additional function calls and also clutters backtraces and > >> >> > flame graphs with helper symbols. > >> >> > > >> >> > To get rid of these helper symbols, provide functionality to inline > >> >> > helpers into Rust using llvm-link. This option complements full LTO, by > >> >> > being much cheaper and avoiding incompatibility with BTF. > >> >> > > >> >> > I ran a microbenchmark showing the benefit of this. All the benchmark > >> >> > does is call refcount_inc() in a loop. This was chosen since refcounting > >> >> > is quite hot in Binder. The results are that Rust spends 6.35 ns per > >> >> > call vs 5.73 ns per call in C. When enabling this option, the two > >> >> > languages become equally fast, and disassembly confirms the exact same > >> >> > machine code is used (in particular there is no call to > >> >> > rust_helper_refcount_inc). Benchmarking Binder also results in an > >> >> > improvement from this change. > >> >> > > >> >> > This patch is complementary to: > >> >> > https://lore.kernel.org/all/20251202-define-rust-helper-v1-0-a2e13cbc17a6@google.com/ > >> >> > > >> >> > Signed-off-by: Alice Ryhl <aliceryhl@google.com> > >> >> > >> >> I get the following modpost errors when building with this applied on top > >> >> of v6.19-rc4: > >> >> > >> >> ERROR: modpost: "__SCK__WARN_trap" [drivers/gpu/drm/nova/nova.ko] undefined! > >> >> ERROR: modpost: "__SCK__WARN_trap" [drivers/gpu/nova-core/nova_core.ko] undefined! > >> >> ERROR: modpost: "__SCK__WARN_trap" [drivers/block/rnull/rnull_mod.ko] undefined! > >> >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_minimal.ko] undefined! > >> >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_misc_device.ko] undefined! > >> >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_print.ko] undefined! > >> >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_dma.ko] undefined! > >> >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_driver_pci.ko] undefined! > >> >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_driver_platform.ko] undefined! > >> >> ERROR: modpost: "__SCK__WARN_trap" [samples/rust/rust_driver_faux.ko] undefined! > >> > > >> > Looks like it used the wrong static_call_mod() version, was MODULE > >> > defined? > >> > >> CONFIG_MODULES=y, yes > >> > >> I built without the patch first, then applied the series, enabled the > >> option via menuconfig and ran the build. I thought maybe some dependency > >> check is messed up so I retried the build from a clean state. Same > >> result. > > > > No, I mean -DMODULE. Note how the quiet_cmd_bindgen target has -DMODULE, > > but the new quiet_cmd_rust_helper target does not. > > Ah, that did the trick, thanks! > > diff --git a/rust/Makefile b/rust/Makefile > index 5365d53b6cf96..08d3dc1038cf5 100644 > --- a/rust/Makefile > +++ b/rust/Makefile > @@ -483,7 +483,7 @@ $(obj)/bindings/bindings_helpers_generated.rs: $(src)/helpers/helpers.c FORCE > > quiet_cmd_rust_helper = HELPER $@ > cmd_rust_helper = \ > - $(CC) $(filter-out $(CFLAGS_REMOVE_helpers/helpers.o), $(c_flags)) -c -g0 $< -emit-llvm -o $@ > + $(CC) $(filter-out $(CFLAGS_REMOVE_helpers/helpers.o), $(c_flags)) -DMODULE -c -g0 $< -emit-llvm -o $@ > > $(obj)/helpers/helpers.bc: $(obj)/helpers/helpers.c FORCE > +$(call if_changed_dep,rust_helper) > > Is -DMOUDLE always appropriate to pass to the helpers? The helpers are > also inlined into non module code. With this option, helpers.bc gets linked into every Rust object file separately (and symbols are internalized). We might want a separate .bc file depending on whether the Rust object files is part of the core kernel or a .ko file. Alice
On Wed, Jan 07, 2026 at 02:42:25PM +0100, Alice Ryhl wrote: > With this option, helpers.bc gets linked into every Rust object file > separately (and symbols are internalized). We might want a separate > .bc file depending on whether the Rust object files is part of the > core kernel or a .ko file. Right, building it twice, once with and once with -DMODULE owuldn't be the end of the world I suppose :-) Also, I ran across Eurydice during the x-mas break, would it not be possible to compile all the .rs muck to C using that and then building the whole lot like 'normal'?
© 2016 - 2026 Red Hat, Inc.