[PATCH v4 0/2] kbuild: distributed build support for Clang ThinLTO

xur@google.com posted 2 patches 3 months, 1 week ago
There is a newer version of this series
.gitignore                 |  2 +
Makefile                   | 25 +++++-------
arch/Kconfig               | 19 +++++++++
scripts/Makefile.lib       |  7 ++++
scripts/Makefile.thinlto   | 40 ++++++++++++++++++
scripts/Makefile.vmlinux_a | 83 ++++++++++++++++++++++++++++++++++++++
scripts/mod/modpost.c      | 15 +++++--
7 files changed, 174 insertions(+), 17 deletions(-)
create mode 100644 scripts/Makefile.thinlto
create mode 100644 scripts/Makefile.vmlinux_a
[PATCH v4 0/2] kbuild: distributed build support for Clang ThinLTO
Posted by xur@google.com 3 months, 1 week ago
From: Rong Xu <xur@google.com>

This V4 is based on the reimplementation provided by Masahiro Yamada
on May 26. The difference is that this version now saves the compiler
flags (_c_flags) from the Front-End (FE) compilation and re-uses them
for the subsequent Back-End (BE) compilation.

Rong Xu (2):
  kbuild: move vmlinux.a build rule to scripts/Makefile.vmlinux_a
  kbuild: distributed build support for Clang ThinLTO

 .gitignore                 |  2 +
 Makefile                   | 25 +++++-------
 arch/Kconfig               | 19 +++++++++
 scripts/Makefile.lib       |  7 ++++
 scripts/Makefile.thinlto   | 40 ++++++++++++++++++
 scripts/Makefile.vmlinux_a | 83 ++++++++++++++++++++++++++++++++++++++
 scripts/mod/modpost.c      | 15 +++++--
 7 files changed, 174 insertions(+), 17 deletions(-)
 create mode 100644 scripts/Makefile.thinlto
 create mode 100644 scripts/Makefile.vmlinux_a


base-commit: dcb6fa37fd7bc9c3d2b066329b0d27dedf8becaa
-- 
2.51.1.838.g19442a804e-goog
Re: [PATCH v4 0/2] kbuild: distributed build support for Clang ThinLTO
Posted by Piotr Gorski 3 months, 1 week ago
I tried using this on kernel 6.17.5, but unfortunately I get this error. 
I previously tested the version in CachyOS: 
https://github.com/CachyOS/linux/commit/68984063b5e27854272241549e47cb2930ef9710 
and https://github.com/CachyOS/linux/ 
commit/00a3ff73a566b7e27b4ced46af4ce27177d9fc37 and everything works fine.

/bin/sh: -c: line 1: syntax error near unexpected token `('
/bin/sh: -c: line 1: `set -e;    trap 'rm -f 
drivers/net/wireguard/main.o; trap - HUP; kill -s HUP $$' HUP; trap 'rm 
-f drivers/net/wireguard/main.o; trap - INT; kill -s INT $$' INT;  trap 
'rm -f drivers/net/wireguard/main.o
; trap - QUIT; kill -s QUIT $$' QUIT;  trap 'rm -f 
drivers/net/wireguard/main.o; trap - TERM; kill -s TERM $$' TERM; trap 
'rm -f drivers/net/wireguard/main.o; trap - PIPE; kill -s PIPE $$' PIPE; 
printf '\n%s\n' 'saved_c_flag
s_drivers/net/wireguard/main.o := -D__KERNEL__ --target=x86_64-linux-gnu 
-fintegrated-as -Werror=ignored-optimization-argument 
-Werror=option-ignored -std=gnu11 -fshort-wchar -funsigned-char 
-fno-common -fno-PIE -fno-strict-a
liasing -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx 
-fcf-protection=branch -fno-jump-tables -m64 -falign-loops=1 -mno-80387 
-mno-fp-ret-in-387 -mstack-alignment=8 -mskip-rax-setup -march=znver4 
-mno-red-zone -mcmodel=kern
el -mstack-protector-guard-reg=gs 
-mstack-protector-guard-symbol=__ref_stack_chk_guard -Wno-sign-compare 
-fno-asynchronous-unwind-tables -mretpoline-external-thunk 
-mindirect-branch-cs-prefix -mfunction-return=thunk-extern -m
harden-sls=all -fpatchable-function-entry=16,16 
-fno-delete-null-pointer-checks -O3 -fstack-protector-strong 
-ftrivial-auto-var-init=zero -fno-stack-clash-protection -pg -mfentry 
-DCC_USING_NOP_MCOUNT -DCC_USING_FENTRY -fno-l
to -flto=thin -fsplit-lto-unit -fvisibility=hidden -falign-functions=16 
-fstrict-flex-arrays=3 -fno-strict-overflow -fno-stack-check 
-fno-builtin-wcslen -Wall -Wextra -Wundef 
-Werror=implicit-function-declaration -Werror=impl
icit-int -Werror=return-type -Wno-format-security -Wno-trigraphs 
-Wno-frame-address -Wno-address-of-packed-member -Wmissing-declarations 
-Wmissing-prototypes -Wframe-larger-than=2048 -Wno-gnu 
-Wno-format-overflow-non-kprintf
-Wno-format-truncation-non-kprintf -Wno-default-const-init-unsafe 
-Wno-pointer-sign -Wcast-function-type 
-Wno-unterminated-string-initialization -Wimplicit-fallthrough 
-Werror=date-time -Wenum-conversion -Wunused -Wno-unused-
but-set-variable -Wno-unused-const-variable -Wno-format-overflow 
-Wno-override-init -Wno-pointer-to-enum-cast 
-Wno-tautological-constant-out-of-range-compare -Wno-unaligned-access 
-Wno-enum-compare-conditional -Wno-missing-fi
eld-initializers -Wno-type-limits -Wno-shift-negative-value 
-Wno-enum-enum-conversion -Wno-sign-compare -Wno-unused-parameter -g 
-gdwarf-5 -D'pr_fmt(fmt)=KBUILD_MODNAME ": " fmt'' >> 
drivers/net/wireguard/.main.o.cmd'
make[5]: *** [scripts/Makefile.build:287: drivers/net/wireguard/main.o] 
Error 2
make[5]: *** Deleting file 'drivers/net/wireguard/main.o'
make[4]: *** [scripts/Makefile.build:556: drivers/net/wireguard] Error 2

LLVM/CLANG: 21.1.4

If necessary, I can send more logs.

Best regards,

Piotr

W dniu 27.10.2025 o 23:09, xur@google.com pisze:
> From: Rong Xu <xur@google.com>
>
> This V4 is based on the reimplementation provided by Masahiro Yamada
> on May 26. The difference is that this version now saves the compiler
> flags (_c_flags) from the Front-End (FE) compilation and re-uses them
> for the subsequent Back-End (BE) compilation.
>
> Rong Xu (2):
>    kbuild: move vmlinux.a build rule to scripts/Makefile.vmlinux_a
>    kbuild: distributed build support for Clang ThinLTO
>
>   .gitignore                 |  2 +
>   Makefile                   | 25 +++++-------
>   arch/Kconfig               | 19 +++++++++
>   scripts/Makefile.lib       |  7 ++++
>   scripts/Makefile.thinlto   | 40 ++++++++++++++++++
>   scripts/Makefile.vmlinux_a | 83 ++++++++++++++++++++++++++++++++++++++
>   scripts/mod/modpost.c      | 15 +++++--
>   7 files changed, 174 insertions(+), 17 deletions(-)
>   create mode 100644 scripts/Makefile.thinlto
>   create mode 100644 scripts/Makefile.vmlinux_a
>
>
> base-commit: dcb6fa37fd7bc9c3d2b066329b0d27dedf8becaa
Re: [PATCH v4 0/2] kbuild: distributed build support for Clang ThinLTO
Posted by Rong Xu 3 months, 1 week ago
Thanks for testing this!

The culprit was the string -D'pr_fmt(fmt)=KBUILD_MODNAME ": " fmt'.

The previous patch parsed c_flags and removed -D% and a bunch of other
FE flags. Masahiro suggested using _c_flags instead.
I erroneously assumed there were no -D% in _c_flags. But this is
wrong. It seems that we do need to filter out -D% even using _c_flags.

I'll send an updated patch shortly.

Best regards,

-Rong


On Tue, Oct 28, 2025 at 3:43 AM Piotr Gorski <piotrgorski@cachyos.org> wrote:
>
> I tried using this on kernel 6.17.5, but unfortunately I get this error.
> I previously tested the version in CachyOS:
> https://github.com/CachyOS/linux/commit/68984063b5e27854272241549e47cb2930ef9710
> and https://github.com/CachyOS/linux/
> commit/00a3ff73a566b7e27b4ced46af4ce27177d9fc37 and everything works fine.
>
> /bin/sh: -c: line 1: syntax error near unexpected token `('
> /bin/sh: -c: line 1: `set -e;    trap 'rm -f
> drivers/net/wireguard/main.o; trap - HUP; kill -s HUP $$' HUP; trap 'rm
> -f drivers/net/wireguard/main.o; trap - INT; kill -s INT $$' INT;  trap
> 'rm -f drivers/net/wireguard/main.o
> ; trap - QUIT; kill -s QUIT $$' QUIT;  trap 'rm -f
> drivers/net/wireguard/main.o; trap - TERM; kill -s TERM $$' TERM; trap
> 'rm -f drivers/net/wireguard/main.o; trap - PIPE; kill -s PIPE $$' PIPE;
> printf '\n%s\n' 'saved_c_flag
> s_drivers/net/wireguard/main.o := -D__KERNEL__ --target=x86_64-linux-gnu
> -fintegrated-as -Werror=ignored-optimization-argument
> -Werror=option-ignored -std=gnu11 -fshort-wchar -funsigned-char
> -fno-common -fno-PIE -fno-strict-a
> liasing -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx
> -fcf-protection=branch -fno-jump-tables -m64 -falign-loops=1 -mno-80387
> -mno-fp-ret-in-387 -mstack-alignment=8 -mskip-rax-setup -march=znver4
> -mno-red-zone -mcmodel=kern
> el -mstack-protector-guard-reg=gs
> -mstack-protector-guard-symbol=__ref_stack_chk_guard -Wno-sign-compare
> -fno-asynchronous-unwind-tables -mretpoline-external-thunk
> -mindirect-branch-cs-prefix -mfunction-return=thunk-extern -m
> harden-sls=all -fpatchable-function-entry=16,16
> -fno-delete-null-pointer-checks -O3 -fstack-protector-strong
> -ftrivial-auto-var-init=zero -fno-stack-clash-protection -pg -mfentry
> -DCC_USING_NOP_MCOUNT -DCC_USING_FENTRY -fno-l
> to -flto=thin -fsplit-lto-unit -fvisibility=hidden -falign-functions=16
> -fstrict-flex-arrays=3 -fno-strict-overflow -fno-stack-check
> -fno-builtin-wcslen -Wall -Wextra -Wundef
> -Werror=implicit-function-declaration -Werror=impl
> icit-int -Werror=return-type -Wno-format-security -Wno-trigraphs
> -Wno-frame-address -Wno-address-of-packed-member -Wmissing-declarations
> -Wmissing-prototypes -Wframe-larger-than=2048 -Wno-gnu
> -Wno-format-overflow-non-kprintf
> -Wno-format-truncation-non-kprintf -Wno-default-const-init-unsafe
> -Wno-pointer-sign -Wcast-function-type
> -Wno-unterminated-string-initialization -Wimplicit-fallthrough
> -Werror=date-time -Wenum-conversion -Wunused -Wno-unused-
> but-set-variable -Wno-unused-const-variable -Wno-format-overflow
> -Wno-override-init -Wno-pointer-to-enum-cast
> -Wno-tautological-constant-out-of-range-compare -Wno-unaligned-access
> -Wno-enum-compare-conditional -Wno-missing-fi
> eld-initializers -Wno-type-limits -Wno-shift-negative-value
> -Wno-enum-enum-conversion -Wno-sign-compare -Wno-unused-parameter -g
> -gdwarf-5 -D'pr_fmt(fmt)=KBUILD_MODNAME ": " fmt'' >>
> drivers/net/wireguard/.main.o.cmd'
> make[5]: *** [scripts/Makefile.build:287: drivers/net/wireguard/main.o]
> Error 2
> make[5]: *** Deleting file 'drivers/net/wireguard/main.o'
> make[4]: *** [scripts/Makefile.build:556: drivers/net/wireguard] Error 2
>
> LLVM/CLANG: 21.1.4
>
> If necessary, I can send more logs.
>
> Best regards,
>
> Piotr
>
> W dniu 27.10.2025 o 23:09, xur@google.com pisze:
> > From: Rong Xu <xur@google.com>
> >
> > This V4 is based on the reimplementation provided by Masahiro Yamada
> > on May 26. The difference is that this version now saves the compiler
> > flags (_c_flags) from the Front-End (FE) compilation and re-uses them
> > for the subsequent Back-End (BE) compilation.
> >
> > Rong Xu (2):
> >    kbuild: move vmlinux.a build rule to scripts/Makefile.vmlinux_a
> >    kbuild: distributed build support for Clang ThinLTO
> >
> >   .gitignore                 |  2 +
> >   Makefile                   | 25 +++++-------
> >   arch/Kconfig               | 19 +++++++++
> >   scripts/Makefile.lib       |  7 ++++
> >   scripts/Makefile.thinlto   | 40 ++++++++++++++++++
> >   scripts/Makefile.vmlinux_a | 83 ++++++++++++++++++++++++++++++++++++++
> >   scripts/mod/modpost.c      | 15 +++++--
> >   7 files changed, 174 insertions(+), 17 deletions(-)
> >   create mode 100644 scripts/Makefile.thinlto
> >   create mode 100644 scripts/Makefile.vmlinux_a
> >
> >
> > base-commit: dcb6fa37fd7bc9c3d2b066329b0d27dedf8becaa