Kbuild provides scripts/Makefile.host to build host programs used for
building the kernel. Unfortunately, there are two exceptions that opt
out of Kbuild. The build system under tools/ is a cheesy replica, and
cause issues. I was recently poked about a problem in the tools build
system, which I do not maintain (and nobody maintains). [1]

Without a comment, people might believe this is the right location
because that is where objtool lives, even if a more robust Kbuild
syntax satisfies their needs. [2]

[1]: https://lore.kernel.org/linux-kbuild/ZnIYWBgrJ-IJtqK8@google.com/T/#m8ece130dd0e23c6f2395ed89070161948dee8457
[2]: https://lore.kernel.org/all/20240618200501.GA1611012@google.com/

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Acked-by: Nicolas Schier <nicolas@fjasle.eu>
Reviewed-by: Brian Norris <briannorris@chromium.org>
---

Changes in v2:
  - rephase the comment for clarification

 Makefile | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/Makefile b/Makefile
index 768d3dc107f8..943899656977 100644
--- a/Makefile
+++ b/Makefile
@@ -1328,6 +1328,12 @@ prepare: tools/bpf/resolve_btfids
 endif
 endif
 
+# The tools build system is not a part of Kbuild and tends to introduce
+# its own unique issues. If you need to integrate a new tool into Kbuild,
+# please consider locating that tool outside the tools/ tree and using the
+# standard Kbuild "hostprogs" syntax instead of adding a new tools/* entry
+# here. See Documentation/kbuild/makefiles.rst for details.
+
 PHONY += resolve_btfids_clean
 
 resolve_btfids_O = $(abspath $(objtree))/tools/bpf/resolve_btfids
-- 
2.43.0

On Thu, Jul 18, 2024 at 03:28:19AM +0900, Masahiro Yamada wrote:
> Kbuild provides scripts/Makefile.host to build host programs used for
> building the kernel. Unfortunately, there are two exceptions that opt
> out of Kbuild. The build system under tools/ is a cheesy replica, and
> cause issues. I was recently poked about a problem in the tools build
> system, which I do not maintain (and nobody maintains). [1]
> 
> Without a comment, people might believe this is the right location
> because that is where objtool lives, even if a more robust Kbuild
> syntax satisfies their needs. [2]

I think the original idea (from Ingo?) was to make objtool portable so
it could be easily copied and built separately without getting too
intertwined with the kernel source.

I think that's still a useful goal.  To my knowledge it's been used in
at least one other code base and could be used elsewhere going forward
as much of its functionality might be transferable to other code bases.

Also being in tools helps it shares library code (libsubcmd) and
synced headers with perf (and others).

If there's some other way to make it portable and allow it to share code
with other tools that then I wouldn't object to moving it to scripts.

Or, if the main problem is that there are two custom build systems (one
of them being a cheap knockoff), could kbuild be made portable enough to
be used in tools?

-- 
Josh

On Sat, Jul 20, 2024 at 7:55 AM Josh Poimboeuf <jpoimboe@kernel.org> wrote:
>
> On Thu, Jul 18, 2024 at 03:28:19AM +0900, Masahiro Yamada wrote:
> > Kbuild provides scripts/Makefile.host to build host programs used for
> > building the kernel. Unfortunately, there are two exceptions that opt
> > out of Kbuild. The build system under tools/ is a cheesy replica, and
> > cause issues. I was recently poked about a problem in the tools build
> > system, which I do not maintain (and nobody maintains). [1]
> >
> > Without a comment, people might believe this is the right location
> > because that is where objtool lives, even if a more robust Kbuild
> > syntax satisfies their needs. [2]
>
> I think the original idea (from Ingo?) was to make objtool portable so
> it could be easily copied and built separately without getting too
> intertwined with the kernel source.
>
> I think that's still a useful goal.  To my knowledge it's been used in
> at least one other code base and could be used elsewhere going forward
> as much of its functionality might be transferable to other code bases.
>
> Also being in tools helps it shares library code (libsubcmd) and
> synced headers with perf (and others).
>
> If there's some other way to make it portable and allow it to share code
> with other tools that then I wouldn't object to moving it to scripts.
>
> Or, if the main problem is that there are two custom build systems (one
> of them being a cheap knockoff), could kbuild be made portable enough to
> be used in tools?


I checked Debian and Fedora, but I did not find such a package that provides
objtool as a standalone tool.

In reality, objtool is tightly bound to the kernel source.
If people had thought it useful outside the kernel tree,
such a disto package would have been widely available.

I do not think there is a good reason to complicate Kbuild
based on the hypothetical statement.


-- 
Best Regards
Masahiro Yamada

Hi Masahiro,

On Wed, Jul 17, 2024 at 6:28 PM Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> Kbuild provides scripts/Makefile.host to build host programs used for
> building the kernel. Unfortunately, there are two exceptions that opt
> out of Kbuild. The build system under tools/ is a cheesy replica, and
> cause issues. I was recently poked about a problem in the tools build
> system, which I do not maintain (and nobody maintains). [1]
>
> Without a comment, people might believe this is the right location
> because that is where objtool lives, even if a more robust Kbuild
> syntax satisfies their needs. [2]
>
> [1]: https://lore.kernel.org/linux-kbuild/ZnIYWBgrJ-IJtqK8@google.com/T/#m8ece130dd0e23c6f2395ed89070161948dee8457
> [2]: https://lore.kernel.org/all/20240618200501.GA1611012@google.com/
>
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> Acked-by: Nicolas Schier <nicolas@fjasle.eu>
> Reviewed-by: Brian Norris <briannorris@chromium.org>
> ---
>
> Changes in v2:
>   - rephase the comment for clarification
>
>  Makefile | 6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/Makefile b/Makefile
> index 768d3dc107f8..943899656977 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -1328,6 +1328,12 @@ prepare: tools/bpf/resolve_btfids
>  endif
>  endif
>
> +# The tools build system is not a part of Kbuild and tends to introduce
> +# its own unique issues. If you need to integrate a new tool into Kbuild,
> +# please consider locating that tool outside the tools/ tree and using the
> +# standard Kbuild "hostprogs" syntax instead of adding a new tools/* entry
> +# here. See Documentation/kbuild/makefiles.rst for details.
> +
>  PHONY += resolve_btfids_clean
>
>  resolve_btfids_O = $(abspath $(objtree))/tools/bpf/resolve_btfids

Looks good to me, thanks for adding the clarification!

Reviewed-by: Sami Tolvanen <samitolvanen@google.com>

Sami