Allow the unwind_user symbol to be used by GPL modules, for instance
LTTng.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Steven Rostedt (Google) <rostedt@goodmis.org>
Cc: Josh Poimboeuf <jpoimboe@kernel.org>
Cc: Masami Hiramatsu <mhiramat@kernel.org>
Cc: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Andrii Nakryiko <andrii@kernel.org>
Cc: Indu Bhagat <indu.bhagat@oracle.com>
Cc: "Jose E. Marchesi" <jemarch@gnu.org>
Cc: Beau Belgrave <beaub@linux.microsoft.com>
Cc: Jens Remus <jremus@linux.ibm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
---
 kernel/unwind/user.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kernel/unwind/user.c b/kernel/unwind/user.c
index 349bdd72390b..9aefbfc8427a 100644
--- a/kernel/unwind/user.c
+++ b/kernel/unwind/user.c
@@ -187,3 +187,4 @@ int unwind_user(struct unwind_stacktrace *trace, unsigned int max_entries)
 
 	return 0;
 }
+EXPORT_SYMBOL_GPL(unwind_user);
-- 
2.43.0

On Wed, Jul 09, 2025 at 05:25:49PM -0400, Mathieu Desnoyers wrote:
> Allow the unwind_user symbol to be used by GPL modules, for instance
> LTTng.

I don't see a LTTng submission or any other user in this series.
So the usual prohibition against adding unused exports applies here
as usual.

[ Adding the TAB to this as well ]

On Fri, 11 Jul 2025 00:36:28 -0700
Christoph Hellwig <hch@infradead.org> wrote:

> On Wed, Jul 09, 2025 at 05:25:49PM -0400, Mathieu Desnoyers wrote:
> > Allow the unwind_user symbol to be used by GPL modules, for instance
> > LTTng.  
> 
> I don't see a LTTng submission or any other user in this series.
> So the usual prohibition against adding unused exports applies here
> as usual.

I want to bring up this discussion. I understand there's a policy not to
add EXPORT_SYMBOL_GPL() unless there's a current user of it in the kernel
proper. My question is, does this policy have to be draconian?

The LTTng project and specifically its maintainer Mathieu has a long
history of working along side of the Linux kernel and actively adding
enhancements to the code. This is not an Nvidia or VMware situation where
the modules add special secret sauce that they do not want to share. This
is a long standing kernel contributor and maintainer that has proved
himself time and time again to be an asset to the Linux kernel community.

It's not even that LTTng wants to be out of tree. There's been several
attempts to get LTTng merged. But they all failed due to disagreements in
approaches and other aspects of the design. With perf and ftrace already in
the kernel which focus toward helping kernel developers debug their issue,
several maintainers do not see the point of LTTng. But LTTng focuses more
on non kernel developers. Mathieu and I have come to a conclusion that
LTTng and ftrace have two different audiences. And we actively work with
each other to improve both ftrace and LTTng.

LTTng was redesigned to work as a very non-intrusive external kernel module
that could be ported to newer kernels without much pain. This approach has
worked for both Mathieu and the kernel maintainers for several years.

It is not because Mathieu doesn't want LTTng in the kernel that is keeping
it out of the tree. He would very much like it in-tree! LTTng is not in the
kernel because the kernel maintainers themselves have been blocking it. I
do not feel that we should be also add the extra burden to prevent any call
it could use from being EXPORT_SYMBOL_GPL().

Mathieu has been very helpful in the work to develop this deferred unwind
code that will help ftrace, perf and BPF. I would also like it to help
LTTng. The only way to do so is to have an EXPORT_SYMBOL_GPL() for it.

Now you can argue that we need to get LTTng into mainline before it can use
this new work that the maintainer of LTTng has been very involved in to
develop. But that would probably take a long time and from past
experiences, may fail completely. So for now, I am saying that we give
LTTng an exception to the rule that functions can not be
EXPORT_SYMBOL_GPL() unless there's a in-tree kernel module that requires it.

-- Steve

On Fri, Jul 11, 2025 at 06:57:42AM -0400, Steven Rostedt wrote:
> I want to bring up this discussion. I understand there's a policy not to
> add EXPORT_SYMBOL_GPL() unless there's a current user of it in the kernel
> proper. My question is, does this policy have to be draconian?

Yes.  Making exception for friends and family is like Austrian law
enforcement and thus errodes the whole principle.  If you think lttng
is useful help ustreaming it.

On Sun, 13 Jul 2025 23:39:29 -0700
Christoph Hellwig <hch@infradead.org> wrote:

> On Fri, Jul 11, 2025 at 06:57:42AM -0400, Steven Rostedt wrote:
> > I want to bring up this discussion. I understand there's a policy not to
> > add EXPORT_SYMBOL_GPL() unless there's a current user of it in the kernel
> > proper. My question is, does this policy have to be draconian?  
> 
> Yes.  Making exception for friends and family is like Austrian law

This has nothing to do with Mathieu being a friend. He's a long time Linux
kernel contributor and has played a key role in developing a new feature
that will help both perf and ftrace, but without the EXPORT_SYMBOL_GPL(),
LTTng can't use it. It's basically saying "thank you Mathieu for helping us
with this new feature, now go F*** off!"

> enforcement and thus errodes the whole principle.  If you think lttng
> is useful help ustreaming it.

That's the entire point. I have no stake in LTTng getting upstream. I just
sympathize with Mathieu as he has been very helpful in improving ftrace. To
me, open source software should work with each other.

Now, I would not block LTTng from getting upstream. I had a conversation
with Mathieu to discuss why it failed last time, and that's because we went
with the approach of trying to merge the features of ftrace and LTTng where
it made sense, and that became way more work than either of us had, and we
found no real benefit from it (besides getting LTTng upstream). It would
require at least 3 or 4 full months of manpower which I don't have time for.

From what Mathieu told me, LTTng's kernel module is 75K SLOC, with 20 years
of development. It already has its own ecosystem. The only practical way
for it to get upstream is if it were to become its own subsystem (like
kernel/lttng).

Who's going to review 75K of SLOC? I don't have the time to. The only
reason I would want LTTng in the kernel is to officially welcome Mathieu
into our community. Other than that, I have no need for LTTng.

What would you recommend Mathieu to do? He's been helping us for several
years and the work he does helps perf and ftrace and other parts of the
kernel (he's helped improve RCU). But to get LTTng upstream, it would take
a massive amount of work. And Mathieu has tried before without success. Why
should he spend all this time to break up the code into reviewable portions
without anyone willing to review it?

As I mentioned before, Mathieu is at a bit of a disadvantage. His customers
are non-kernel developers. But he needs the interest of kernel developers
to review his code. But he can't find anyone willing to do that. He doesn't
want to waste his time trying spending months of work to get LTTng upstream
if there's no guarantee that it will be accepted.

Would be willing to review his work?

-- Steve

On Mon, Jul 14, 2025 at 06:27:24AM -0400, Steven Rostedt wrote:
> This has nothing to do with Mathieu being a friend. He's a long time Linux
> kernel contributor and has played a key role in developing a new feature
> that will help both perf and ftrace, but without the EXPORT_SYMBOL_GPL(),
> LTTng can't use it. It's basically saying "thank you Mathieu for helping us
> with this new feature, now go F*** off!"

You don't have to be as explicit, but otherwise that's exactly how
it works.  No one gets a free ride just because they are nice and/or
contributed something.

The rest of your mail looks just as confused.

On Mon, 14 Jul 2025 04:38:33 -0700
Christoph Hellwig <hch@infradead.org> wrote:

> On Mon, Jul 14, 2025 at 06:27:24AM -0400, Steven Rostedt wrote:
> > This has nothing to do with Mathieu being a friend. He's a long time Linux
> > kernel contributor and has played a key role in developing a new feature
> > that will help both perf and ftrace, but without the EXPORT_SYMBOL_GPL(),
> > LTTng can't use it. It's basically saying "thank you Mathieu for helping us
> > with this new feature, now go F*** off!"  
> 
> You don't have to be as explicit, but otherwise that's exactly how
> it works.  No one gets a free ride just because they are nice and/or
> contributed something.

Why is that?

And yes, I still consider it draconian.

> 
> The rest of your mail looks just as confused.

Let me rephrase it then.

How would you recommend getting LTTng into the kernel? It's a relatively
large project that has 75K of lines of code with development that lasted
around 20 years.

Should it be one big code drop?

Should Mathieu copy the history of his git tree into the kernel/lttng
directory and suggest a git pull request to Linus?

To break it up now, into reviewable patches would be a huge undertaking.
And who is going to review it? I don't have the time, do you?

Basically, Mathieu has been a good Linux kernel community member, and even
wants his code upstream. But when he's the only one with a stake in getting
it upstream, and it's a long time large project, there's no easy path for
him to do it.

Now he's helped out with a simple feature that lets perf and ftrace get
user space stack traces without the need for frame pointers, and he can't
use it unless his code gets upstream.

Have any suggestions for him, or do you just not care? But you are one of
the gate keepers of EXPORT_SYMBOL_GPL() which affects him.

-- Steve

On Mon, Jul 14, 2025 at 07:54:26AM -0400, Steven Rostedt wrote:
> On Mon, 14 Jul 2025 04:38:33 -0700
> Christoph Hellwig <hch@infradead.org> wrote:
> 
> > On Mon, Jul 14, 2025 at 06:27:24AM -0400, Steven Rostedt wrote:
> > > This has nothing to do with Mathieu being a friend. He's a long time Linux
> > > kernel contributor and has played a key role in developing a new feature
> > > that will help both perf and ftrace, but without the EXPORT_SYMBOL_GPL(),
> > > LTTng can't use it. It's basically saying "thank you Mathieu for helping us
> > > with this new feature, now go F*** off!"  
> > 
> > You don't have to be as explicit, but otherwise that's exactly how
> > it works.  No one gets a free ride just because they are nice and/or
> > contributed something.
> 
> Why is that?

Why would it be any different?  We have a clear reason both for technical
reasons, and to get code upstream.  Making exceptions for vaguely defined
friends and family defeats the entire purpose.

If you want to help Mathieu or others do that by putting your effort
behind the cause instead of making up exceptions.

> How would you recommend getting LTTng into the kernel? It's a relatively
> large project that has 75K of lines of code with development that lasted
> around 20 years.

I honestly don't care.  Not my business.  And you're probably also
asking the wrong question, as those giant old out of tree projects
tend to be a mess because of that.  The right question is really what
functionality does LTTng have that we want in the kernel and work on
that.

On Tue, 15 Jul 2025 00:19:16 -0700
Christoph Hellwig <hch@infradead.org> wrote:

> I honestly don't care.  Not my business.  And you're probably also
> asking the wrong question, as those giant old out of tree projects
> tend to be a mess because of that.  The right question is really what
> functionality does LTTng have that we want in the kernel and work on
> that.

See my reply to Linus in the other thread you were Cc'd on.

https://lore.kernel.org/all/20250715052459.0000e119@gandalf.local.home/

-- Steve

On Mon, 14 Jul 2025 07:54:26 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> And yes, I still consider it draconian.

It's not that Mathieu doesn't want to work with upstream. It's more like
upstream doesn't want to work with Mathieu.

And I'll admit that I haven't helped in this regard. In the past I've
considered LTTng as more of an annoyance. But over the years of working
with Mathieu, we have found that ftrace and LTTng have two different
audiences with a little overlap. And because Mathieu has helped out where
he could, I started helping him where I could. That's called, collaboration.

-- Steve

On Mon, Jul 14, 2025 at 07:54:26AM -0400, Steven Rostedt wrote:
> On Mon, 14 Jul 2025 04:38:33 -0700
> Christoph Hellwig <hch@infradead.org> wrote:
> 
> > On Mon, Jul 14, 2025 at 06:27:24AM -0400, Steven Rostedt wrote:
> > > This has nothing to do with Mathieu being a friend. He's a long time Linux
> > > kernel contributor and has played a key role in developing a new feature
> > > that will help both perf and ftrace, but without the EXPORT_SYMBOL_GPL(),
> > > LTTng can't use it. It's basically saying "thank you Mathieu for helping us
> > > with this new feature, now go F*** off!"  
> > 
> > You don't have to be as explicit, but otherwise that's exactly how
> > it works.  No one gets a free ride just because they are nice and/or
> > contributed something.
> 
> Why is that?
> 
> And yes, I still consider it draconian.
> 
> > 
> > The rest of your mail looks just as confused.
> 
> Let me rephrase it then.
> 
> How would you recommend getting LTTng into the kernel? It's a relatively
> large project that has 75K of lines of code with development that lasted
> around 20 years.

The same exact way any such out-of-tree project gets merged into the
kernel tree.  This isn't unusual, why is anyone confused here?

> To break it up now, into reviewable patches would be a huge undertaking.
> And who is going to review it? I don't have the time, do you?

The maintainers of the subsystems in which the new code interacts with
are the reviewers, as always.  Again, nothing unusual here.

> Have any suggestions for him, or do you just not care? But you are one of
> the gate keepers of EXPORT_SYMBOL_GPL() which affects him.

This is not the issue at all, sorry Just submit it like normal.

I think there's a whole in-kernel document that describes the process :)

thanks,

greg k-h

On Mon, 14 Jul 2025 13:59:56 +0200
Greg KH <gregkh@linuxfoundation.org> wrote:

> The maintainers of the subsystems in which the new code interacts with
> are the reviewers, as always.  Again, nothing unusual here.

What subsystems does it interact with? It is an observer, it doesn't affect
any subsystem. It's basically stand alone. It only needs to use the
tracepoint infrastructure (which BTW, Matiheu is the maintainer of).

Just like perf and ftrace. They are stand alone subsystems where other
subsystems may interact with them, but they don't really interact with
anything else (besides the normal utilities like allocation of memory and
such).

> 
> > Have any suggestions for him, or do you just not care? But you are one of
> > the gate keepers of EXPORT_SYMBOL_GPL() which affects him.  
> 
> This is not the issue at all, sorry Just submit it like normal.

So he could just send a thousand patch series that doesn't touch any other
subsystem? It would have a diffstat that touches nothing but kernel/lttng,
kernel/Makefile and kernel/Kconfig. Who's going to review it? Who's going
to accept it?


-- Steve

On Mon, Jul 14, 2025 at 08:20:33AM -0400, Steven Rostedt wrote:
> On Mon, 14 Jul 2025 13:59:56 +0200
> Greg KH <gregkh@linuxfoundation.org> wrote:
> 
> > The maintainers of the subsystems in which the new code interacts with
> > are the reviewers, as always.  Again, nothing unusual here.
> 
> What subsystems does it interact with? It is an observer, it doesn't affect
> any subsystem. It's basically stand alone. It only needs to use the
> tracepoint infrastructure (which BTW, Matiheu is the maintainer of).
> 
> Just like perf and ftrace. They are stand alone subsystems where other
> subsystems may interact with them, but they don't really interact with
> anything else (besides the normal utilities like allocation of memory and
> such).
> 
> > 
> > > Have any suggestions for him, or do you just not care? But you are one of
> > > the gate keepers of EXPORT_SYMBOL_GPL() which affects him.  
> > 
> > This is not the issue at all, sorry Just submit it like normal.
> 
> So he could just send a thousand patch series that doesn't touch any other
> subsystem? It would have a diffstat that touches nothing but kernel/lttng,
> kernel/Makefile and kernel/Kconfig. Who's going to review it? Who's going
> to accept it?

Again, this is all things that we all know how to do and aren't new.
Get someone in a subsystem relevant to it to review a patch series that
adds the needed files to the kernel in a way that they can actually be
reviewed.

As it seems that we do have other tracing/perf developers already, start
with them and go forward to see what they say.

good luck!

greg k-h

On Mon, Jul 14, 2025 at 6:28 AM Greg KH <gregkh@linuxfoundation.org> wrote:
 > So he could just send a thousand patch series that doesn't touch any other
> > subsystem? It would have a diffstat that touches nothing but kernel/lttng,
> > kernel/Makefile and kernel/Kconfig. Who's going to review it? Who's going
> > to accept it?
>
> Again, this is all things that we all know how to do and aren't new.
> Get someone in a subsystem relevant to it to review a patch series that
> adds the needed files to the kernel in a way that they can actually be
> reviewed.
>
> As it seems that we do have other tracing/perf developers already, start
> with them and go forward to see what they say.


and a good best practice before posting 1000 series is .. to identify
the most likely reviews and ask them upfront what their preferred way
to do this is...

On Mon, 14 Jul 2025 15:26:47 +0200
Greg KH <gregkh@linuxfoundation.org> wrote:

> As it seems that we do have other tracing/perf developers already, start
> with them and go forward to see what they say.

We tried that. I don't have the time and I'm sure the perf folks don't
care. I advocate for LTTng because of the support that Mathieu has given
us.

The only other option is to allow LTTng to have access to a couple of
functions that Mathieu helped develop. Otherwise it's forcing me to say
"Thank you Mathieu for all your work, now go F*** off!".

Which appears to be the only option :-(

-- Steve

On Mon, Jul 14, 2025 at 09:35:47AM -0400, Steven Rostedt wrote:
> On Mon, 14 Jul 2025 15:26:47 +0200
> Greg KH <gregkh@linuxfoundation.org> wrote:
> 
> > As it seems that we do have other tracing/perf developers already, start
> > with them and go forward to see what they say.
> 
> We tried that. I don't have the time and I'm sure the perf folks don't
> care. I advocate for LTTng because of the support that Mathieu has given
> us.

When was this last tried?  If you don't have time, then the fault is on
you, not us :)

> The only other option is to allow LTTng to have access to a couple of
> functions that Mathieu helped develop. Otherwise it's forcing me to say
> "Thank you Mathieu for all your work, now go F*** off!".
> 
> Which appears to be the only option :-(

Stop making this false argument please, we aren't telling anyone
anything different from what we have always been saying for decades now.

Again, we don't export symbols for when there are no in-tree users.
That's it.

lttng not getting merged because you don't have time to review it should
not make the above rule somehow invalid.

sorry,

greg k-h

On Mon, 14 Jul 2025 15:56:14 +0200
Greg KH <gregkh@linuxfoundation.org> wrote:

> lttng not getting merged because you don't have time to review it should
> not make the above rule somehow invalid.

Well, it's because LTTng is a separate entity that handles different
users than I worry about.

I'm having enough trouble keeping up with what is already in the
kernel. If we can find someone else to help out, I'm all for it.
Currently the tracing subsystem has two maintainers (Masami and
myself), and there's still not enough time to review the current code.
Mathieu is listed as a reviewer on the tracing subsystem. Could he
review his own code?

-- Steve

On Mon, 14 Jul 2025 08:20:33 -0400
Steven Rostedt <rostedt@goodmis.org> wrote:

> Just like perf and ftrace. They are stand alone subsystems where other

I should clarify, I'm using "ftrace" to reference the "tracing" subsystem,
as "ftrace" technically is the function hook mechanism that is tightly
coupled with every architecture that uses it.

So here, when I say "ftrace" I mean kernel/trace/*, and I really should say
"the tracing subsystem"

-- Steve

On Fri, Jul 11, 2025 at 06:57:42AM -0400, Steven Rostedt wrote:
> 
> [ Adding the TAB to this as well ]
> 
> On Fri, 11 Jul 2025 00:36:28 -0700
> Christoph Hellwig <hch@infradead.org> wrote:
> 
> > On Wed, Jul 09, 2025 at 05:25:49PM -0400, Mathieu Desnoyers wrote:
> > > Allow the unwind_user symbol to be used by GPL modules, for instance
> > > LTTng.  
> > 
> > I don't see a LTTng submission or any other user in this series.
> > So the usual prohibition against adding unused exports applies here
> > as usual.
> 
> I want to bring up this discussion. I understand there's a policy not to
> add EXPORT_SYMBOL_GPL() unless there's a current user of it in the kernel
> proper. My question is, does this policy have to be draconian?

It's not "draconian", it is "we do not add exports for stuff that is
not in our kernel tree."  Simple, direct, and obvious.  We have no idea
how, or if, external modules do anything with apis that we export and do
not use internally, so we can't change them without breaking anything,
so it's simpler and more obvious to not even attempt to care about them.

If external users want to use a symbol, they should get merged into the
tree.

Yes, lttng is a "good citizen" of the kernel community, and yes, it's
not merged into the tree, but that's not our issue, and living outside
of the tree has it's penalities, both economic and technical.  This is
one of those penalities, sorry.

So no, we shouldn't change this at all.  And I think we need to start
doing more of the "only export this symbol for this subsystem/module"
trimming as well.  I see the horrors that out-of-tree kernel code does
and by removing the ability of them to even get access to the things
they do have access to today, would be good for everyone involved (i.e.
our community, AND the users of Linux to allow them to have a more
stable and trustworthy base to rely on.)

thanks,

greg k-h

On Fri, 2025-07-11 at 13:38 +0200, Greg KH wrote:
> On Fri, Jul 11, 2025 at 06:57:42AM -0400, Steven Rostedt wrote:
> > 
> > [ Adding the TAB to this as well ]
> > 
> > On Fri, 11 Jul 2025 00:36:28 -0700
> > Christoph Hellwig <hch@infradead.org> wrote:
> > 
> > > On Wed, Jul 09, 2025 at 05:25:49PM -0400, Mathieu Desnoyers
> > > wrote:
> > > > Allow the unwind_user symbol to be used by GPL modules, for
> > > > instance LTTng.  
> > > 
> > > I don't see a LTTng submission or any other user in this series.
> > > So the usual prohibition against adding unused exports applies
> > > here as usual.
> > 
> > I want to bring up this discussion. I understand there's a policy
> > not to add EXPORT_SYMBOL_GPL() unless there's a current user of it
> > in the kernel proper. My question is, does this policy have to be
> > draconian?
> 
> It's not "draconian", it is "we do not add exports for stuff that is
> not in our kernel tree."  Simple, direct, and obvious.  We have no
> idea how, or if, external modules do anything with apis that we
> export and do not use internally, so we can't change them without
> breaking anything, so it's simpler and more obvious to not even
> attempt to care about
> them.

Since the argument is still going on, I've got to say I agree with
Greg: it's not draconian it's a simple technical rule which is easily
enforceable and gets us out of making political decisions like what's a
"good" use of a symbol and what isn't.  Once we start making political
decisions about kernel symbol exports we open ourselves up to
accusations of partisanship and favouritism ... since we have a clear
technical criterion, it's just way easier not to descend that slippery
slope.

Regards,

James

On Fri, 11 Jul 2025 13:38:07 +0200
Greg KH <gregkh@linuxfoundation.org> wrote:

> Yes, lttng is a "good citizen" of the kernel community, and yes, it's
> not merged into the tree, but that's not our issue, and living outside
> of the tree has it's penalities, both economic and technical.  This is
> one of those penalities, sorry.

BTW, when it comes to tracers, being out of tree is actually a huge
advantage. Tracers, unlike drivers, are only monitoring the kernel. Which
means it doesn't really rely on the workings of the kernel, so it really
doesn't get much help from changes made by other kernel developers. Like
the BPF folks have said keeping BPF programs up to date, isn't that much of
a burden.

The huge advantage that LTTng has over perf and ftrace for being out of
tree is that it controls the ABI between the LTTng kernel side and the user
space side. LTTng can experiment with new interfaces, and if something
breaks, it can simply change it and deliver a new tool that includes the
new module with the update.

One thing that perf and ftrace struggle with is getting the API correct the
first time. That's because they have no control over what other tools may use
their interface. If I expose something that another tool starts to use
(like powertop) and then I change it to enhance tracing and it breaks that
tool, that change gets reverted. To make any new update, I have to guess at
what are all the ways a tool could abuse this interface and also make sure
it works well into the future when new things come up. There's a lot of
interfaces I would love to change but can't, simply because they are now
ABI.

LTTng doesn't have the burden of worrying about "what other tool may use
this kernel interface?", and can even change it much later if Mathieu finds
a better way to do it.

-- Steve

Steven Rostedt wrote:
> On Fri, 11 Jul 2025 13:38:07 +0200
> Greg KH <gregkh@linuxfoundation.org> wrote:
> 
> > Yes, lttng is a "good citizen" of the kernel community, and yes, it's
> > not merged into the tree, but that's not our issue, and living outside
> > of the tree has it's penalities, both economic and technical.  This is
> > one of those penalities, sorry.
> 
> BTW, when it comes to tracers, being out of tree is actually a huge
> advantage. Tracers, unlike drivers, are only monitoring the kernel. Which
> means it doesn't really rely on the workings of the kernel, so it really
> doesn't get much help from changes made by other kernel developers. Like
> the BPF folks have said keeping BPF programs up to date, isn't that much of
> a burden.
> 
> The huge advantage that LTTng has over perf and ftrace for being out of
> tree is that it controls the ABI between the LTTng kernel side and the user
> space side. LTTng can experiment with new interfaces, and if something
> breaks, it can simply change it and deliver a new tool that includes the
> new module with the update.

It is odd to read this claimed benefit when viewing it from the wider
Linux kernel project. Upstream maintenance of ABI contracts is the
fundamental struggle of subsystems. The request, "can we get the kernel
out of the way and maintain our own ABI to our users?" is a consistent
refrain, and it consistently receives a qualified "no" for regression,
security, and other interface evolution concerns.

On Fri, Jul 11, 2025 at 11:10:06AM -0700, dan.j.williams@intel.com wrote:
> It is odd to read this claimed benefit when viewing it from the wider
> Linux kernel project. Upstream maintenance of ABI contracts is the
> fundamental struggle of subsystems. The request, "can we get the kernel
> out of the way and maintain our own ABI to our users?" is a consistent
> refrain, and it consistently receives a qualified "no" for regression,
> security, and other interface evolution concerns.

Exactly.

On Fri, 11 Jul 2025 11:10:06 -0700
<dan.j.williams@intel.com> wrote:

> > The huge advantage that LTTng has over perf and ftrace for being out of
> > tree is that it controls the ABI between the LTTng kernel side and the user
> > space side. LTTng can experiment with new interfaces, and if something
> > breaks, it can simply change it and deliver a new tool that includes the
> > new module with the update.  
> 
> It is odd to read this claimed benefit when viewing it from the wider
> Linux kernel project. Upstream maintenance of ABI contracts is the
> fundamental struggle of subsystems. The request, "can we get the kernel
> out of the way and maintain our own ABI to our users?" is a consistent
> refrain, and it consistently receives a qualified "no" for regression,
> security, and other interface evolution concerns.

Well I think the history matters a bit here. Mathieu tried to get LTTng
upstream several times. There was never a "Can I keep it out of tree so
I have this flexibility". Only after he gave up trying to get it
upstream, did this advantage play a role. There's been a few times
LTTng was able to take advantage of this "feature".

Note, as Mathieu has stated, LTTng now has a "stable ABI" similar to
what Linux has to make it easier for distributions to maintain LTTng.
But he also said that he can deprecate older interfaces much easier
than he would if it was upstream in Linux.

My comment wasn't to advocate such a approach, it was to counter Greg's
comment of "and living outside of the tree has it's penalities, both
economic and technical.

The main penalty that LTTng has being out of tree is the lack of access
to these functions. But it's been doing very well outside of that.

-- Steve

On 2025-07-11 10:02, Steven Rostedt wrote:
> On Fri, 11 Jul 2025 13:38:07 +0200
> Greg KH <gregkh@linuxfoundation.org> wrote:
> 
>> Yes, lttng is a "good citizen" of the kernel community, and yes, it's
>> not merged into the tree, but that's not our issue, and living outside
>> of the tree has it's penalities, both economic and technical.  This is
>> one of those penalities, sorry.
> 
> BTW, when it comes to tracers, being out of tree is actually a huge
> advantage. Tracers, unlike drivers, are only monitoring the kernel. Which
> means it doesn't really rely on the workings of the kernel, so it really
> doesn't get much help from changes made by other kernel developers. Like
> the BPF folks have said keeping BPF programs up to date, isn't that much of
> a burden.
> 
> The huge advantage that LTTng has over perf and ftrace for being out of
> tree is that it controls the ABI between the LTTng kernel side and the user
> space side. LTTng can experiment with new interfaces, and if something
> breaks, it can simply change it and deliver a new tool that includes the
> new module with the update.
> 
> One thing that perf and ftrace struggle with is getting the API correct the
> first time. That's because they have no control over what other tools may use
> their interface. If I expose something that another tool starts to use
> (like powertop) and then I change it to enhance tracing and it breaks that
> tool, that change gets reverted. To make any new update, I have to guess at
> what are all the ways a tool could abuse this interface and also make sure
> it works well into the future when new things come up. There's a lot of
> interfaces I would love to change but can't, simply because they are now
> ABI.
> 
> LTTng doesn't have the burden of worrying about "what other tool may use
> this kernel interface?", and can even change it much later if Mathieu finds
> a better way to do it.

This is indeed one advantage LTTng has: it controls both the kernel
and userspace sides of the software.

I should point out however that since about LTTng 2.7 (2015) we've made
it a project policy to never break the LTTng-modules userspace ABI.
Those ABIs are effectively append-only (with possible gradual
deprecation) to make life easier for distro vendors for which the
userspace vs kernel modules packages are not in sync sometimes.
ABI-breaking changes to LTTng-modules makes packaging of kernel + OOT
modules + userspace tricky for distros. Similarly to the Linux kernel,
our "do not break userspace ABI" policy takes care of this.

So even though LTTng does not _break_ ABIs, it benefits from controlling
what userspace is allowed to interact with it. The fact that we have
an aligned release cycle across LTTng-Tools, the LTTng userspace tracer,
and the LTTng kernel tracer allows us to deprecate ABIs at a faster pace
than what typical kernel ABIs with arbitrary userspace consumers can do.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
https://www.efficios.com

On Fri, 11 Jul 2025 13:38:07 +0200
Greg KH <gregkh@linuxfoundation.org> wrote:

> On Fri, Jul 11, 2025 at 06:57:42AM -0400, Steven Rostedt wrote:
> > 
> > [ Adding the TAB to this as well ]
> > 
> > On Fri, 11 Jul 2025 00:36:28 -0700
> > Christoph Hellwig <hch@infradead.org> wrote:
> >   
> > > On Wed, Jul 09, 2025 at 05:25:49PM -0400, Mathieu Desnoyers wrote:  
> > > > Allow the unwind_user symbol to be used by GPL modules, for instance
> > > > LTTng.    
> > > 
> > > I don't see a LTTng submission or any other user in this series.
> > > So the usual prohibition against adding unused exports applies here
> > > as usual.  
> > 
> > I want to bring up this discussion. I understand there's a policy not to
> > add EXPORT_SYMBOL_GPL() unless there's a current user of it in the kernel
> > proper. My question is, does this policy have to be draconian?  
> 
> It's not "draconian", it is "we do not add exports for stuff that is
> not in our kernel tree."  Simple, direct, and obvious.  We have no idea
> how, or if, external modules do anything with apis that we export and do
> not use internally, so we can't change them without breaking anything,
> so it's simpler and more obvious to not even attempt to care about them.

Let me clear up a misunderstanding on my part. That is, this *will* be used
internally when an EXPORT_SYMBOL_GPL() is added to it. Ftrace, perf and
probably BPF will be using the function. But all of these are not modules,
but the only one that is a module would be LTTng, and it is adaptable to
change. Just like a BPF program would be.

I mean, if I made a part of ftrace into a loadable module that used this
feature, then we could make this EXPORT_SYMBOL_GPL()?

> 
> If external users want to use a symbol, they should get merged into the
> tree.
> 
> Yes, lttng is a "good citizen" of the kernel community, and yes, it's
> not merged into the tree, but that's not our issue, and living outside
> of the tree has it's penalities, both economic and technical.  This is
> one of those penalities, sorry.

But it is our issue. We are the ones keeping it out. We can try to get it
in again, but it will be a long and tedious process. Mathieu is at a
disadvantage here because he has no leverage as most of his customers are
non-kernel developers. So the decisions he makes is for them, not us. And
this has caused tension when it came to the way he did things compared to
what we wanted. His keeping it out of tree has worked so far. And now he's
been an active contributor on a new feature that will benefit perf, ftrace
and BPF but because we've been keeping him out of tree, he doesn't get to
use the work he's contributed to?

> 
> So no, we shouldn't change this at all.  And I think we need to start
> doing more of the "only export this symbol for this subsystem/module"
> trimming as well.  I see the horrors that out-of-tree kernel code does
> and by removing the ability of them to even get access to the things
> they do have access to today, would be good for everyone involved (i.e.
> our community, AND the users of Linux to allow them to have a more
> stable and trustworthy base to rely on.)

Yes, but you mostly see drivers and side functions that really do nothing
for the kernel. LTTng has contributed tracepoints, several RCU
enhancements, restartable sequences and more.

The user stack unwinding that we are adding is a new feature to do stack
unwinding without frame pointers that Fedora is working on adding. There's
two functions that LTTng would need to get this, and both will be used by
ftrace and perf. It only gives the module a user stack unwind. It's
information only and not something that tweaks the kernel in any way.

-- Steve

On Fri, Jul 11, 2025 at 09:17:34AM -0400, Steven Rostedt wrote:
> On Fri, 11 Jul 2025 13:38:07 +0200
> Greg KH <gregkh@linuxfoundation.org> wrote:
> 
> > On Fri, Jul 11, 2025 at 06:57:42AM -0400, Steven Rostedt wrote:
> > > 
> > > [ Adding the TAB to this as well ]
> > > 
> > > On Fri, 11 Jul 2025 00:36:28 -0700
> > > Christoph Hellwig <hch@infradead.org> wrote:
> > >   
> > > > On Wed, Jul 09, 2025 at 05:25:49PM -0400, Mathieu Desnoyers wrote:  
> > > > > Allow the unwind_user symbol to be used by GPL modules, for instance
> > > > > LTTng.    
> > > > 
> > > > I don't see a LTTng submission or any other user in this series.
> > > > So the usual prohibition against adding unused exports applies here
> > > > as usual.  
> > > 
> > > I want to bring up this discussion. I understand there's a policy not to
> > > add EXPORT_SYMBOL_GPL() unless there's a current user of it in the kernel
> > > proper. My question is, does this policy have to be draconian?  
> > 
> > It's not "draconian", it is "we do not add exports for stuff that is
> > not in our kernel tree."  Simple, direct, and obvious.  We have no idea
> > how, or if, external modules do anything with apis that we export and do
> > not use internally, so we can't change them without breaking anything,
> > so it's simpler and more obvious to not even attempt to care about them.
> 
> Let me clear up a misunderstanding on my part. That is, this *will* be used
> internally when an EXPORT_SYMBOL_GPL() is added to it. Ftrace, perf and
> probably BPF will be using the function. But all of these are not modules,
> but the only one that is a module would be LTTng, and it is adaptable to
> change. Just like a BPF program would be.
> 
> I mean, if I made a part of ftrace into a loadable module that used this
> feature, then we could make this EXPORT_SYMBOL_GPL()?

Don't be silly, and don't attempt to "skirt the rules" like this.
Otherwise you will have a huge onflux of people/companies claiming that
they too deserve symbol exports for their long-out-of-tree-monstrosities
that they never worked to merge.

Again, let's keep the rule simple, exports are only for in-kernel users.

> > If external users want to use a symbol, they should get merged into the
> > tree.
> > 
> > Yes, lttng is a "good citizen" of the kernel community, and yes, it's
> > not merged into the tree, but that's not our issue, and living outside
> > of the tree has it's penalities, both economic and technical.  This is
> > one of those penalities, sorry.
> 
> But it is our issue. We are the ones keeping it out. We can try to get it
> in again, but it will be a long and tedious process. Mathieu is at a
> disadvantage here because he has no leverage as most of his customers are
> non-kernel developers. So the decisions he makes is for them, not us. And
> this has caused tension when it came to the way he did things compared to
> what we wanted. His keeping it out of tree has worked so far. And now he's
> been an active contributor on a new feature that will benefit perf, ftrace
> and BPF but because we've been keeping him out of tree, he doesn't get to
> use the work he's contributed to?

Sure he can, by virtue of it being in perf, ftrace, and bpf :)

Again, export stuff for in-tree users only.

> > So no, we shouldn't change this at all.  And I think we need to start
> > doing more of the "only export this symbol for this subsystem/module"
> > trimming as well.  I see the horrors that out-of-tree kernel code does
> > and by removing the ability of them to even get access to the things
> > they do have access to today, would be good for everyone involved (i.e.
> > our community, AND the users of Linux to allow them to have a more
> > stable and trustworthy base to rely on.)
> 
> Yes, but you mostly see drivers and side functions that really do nothing
> for the kernel. LTTng has contributed tracepoints, several RCU
> enhancements, restartable sequences and more.

Somehow those little drivers are "doing something" for the kernel by
virtue of you being able to use your keyboard and network card :)

Seriously, don't conflate tracing core code as some how being more
"important" than any existing driver in the tree, when both, in the end,
are necessary for a user to get their job done.  One could even say that
the tracing stuff is NOT needed, as if code works properly, no one
should be needing to use it...

Anyway, I see stuff all over the place, not just drivers, for "real"
things on a weekly basis.  Go poke in the Android kernel tree if you
want to see loads of examples of where vendors feel they can export/hook
into core parts of Linux just to get their work done.  I would argue
that most of them are NOT needed and are doing crazy things, but as
users have real hardware that sometimes requires it, it's a hard thing
to argue against.

I'll show one real-world example, the USB offload path code, that you
are using today on your Android phones that saves loads of battery life.
In older kernels, the hooks/exports needed for that were all over the
place, it took an engineer years to get this all working for lots of
different hardware types and merged upstream properly.  They knew that
they could not just get the upstream developers to export the needed
functions, they had to get their working code merged in order to be able
to have it happen.  And they did!

That was a non-trivial task, and while you might not feel it "does
nothing", the power savings you rely on daily would beg to differ.

So get the lttng code merged into the tree please, it seems that you are
the one objecting to the merge, not me :)

thanks,

greg k-h

On Fri, 11 Jul 2025 18:39:40 +0200
Greg KH <gregkh@linuxfoundation.org> wrote:

> I'll show one real-world example, the USB offload path code, that you
> are using today on your Android phones that saves loads of battery life.
> In older kernels, the hooks/exports needed for that were all over the
> place, it took an engineer years to get this all working for lots of
> different hardware types and merged upstream properly.  They knew that
> they could not just get the upstream developers to export the needed
> functions, they had to get their working code merged in order to be able
> to have it happen.  And they did!

So how did this work? Did they write infrastructure in the core kernel
code with EXPORT_SYMBOL_GPL() to that this new functionality could be
used by other USB drivers?

In doing that, the work is adding functionality to all users inside the
kernel as well as to out of tree modules.

When I said that tracers are different, it comes down to that the
in-tree tracers are never a module. Any new infrastructure they use
will not be exported. There's no way that LTTng can add infrastructure
for perf and ftrace and then use it. So how is it suppose to work to
integrate with the kernel if we do not allow it to access the shared
infrastructure?

If perf and ftrace were modules, there would likely be a lot of
infrastructure functions with EXPORT_SYMBOL_GPL() that LTTng could use
and work to change its infrastructure to use the same infrastructure as
perf and ftrace. But because perf and ftrace are never a module,
there's no way for LTTng to work with the same code.

I explain this in more detail with my reply to Linus in the other
thread that you were Cc'd on.

https://lore.kernel.org/all/20250715052459.0000e119@gandalf.local.home/

-- Steve

On Fri, 11 Jul 2025 18:39:40 +0200
Greg KH <gregkh@linuxfoundation.org> wrote:

> > Yes, but you mostly see drivers and side functions that really do nothing
> > for the kernel. LTTng has contributed tracepoints, several RCU
> > enhancements, restartable sequences and more.  
> 
> Somehow those little drivers are "doing something" for the kernel by
> virtue of you being able to use your keyboard and network card :)

I meant contributions to other parts of the kernel than just tracing.
This is more like PREEMPT_RT, where improvements to the core kernel was
made to help both LTTng and the kernel proper.

If you want to talk about precedent, we have it. raw_spin_lock() was
used *only* for PREEMPT_RT years before it was ever added to the kernel.

> 
> Seriously, don't conflate tracing core code as some how being more
> "important" than any existing driver in the tree, when both, in the end,
> are necessary for a user to get their job done.  One could even say that
> the tracing stuff is NOT needed, as if code works properly, no one
> should be needing to use it...

I'll even argue that this is fine for EXPORT_SYMBOL_GPL() because it
*isn't important*! I'm not saying it's more important. I'll agree with
you and say its less important. The point that the function I want to
export gives you information only and doesn't give you any operational
hooks into the kernel, means it's just a outside observer of the
system. It's not changing how the system works in any way.

I always tell people that tracing is a second class citizen when it
comes to the kernel. It should try it's damn best to not effect the
system it is trying to observe. If there's a decision to be made that
can improve tracing with the cost of hurting the system performance
when tracing is off, I'll make the decision not to do it. This is why
we have jump_label. To have nops when tracing is disabled and jmps when
it is, so that tracing doesn't affect normal operations. Not to mention
all the hacks done for function tracing to keep its overhead down to a
bare minimum.

> 
> Anyway, I see stuff all over the place, not just drivers, for "real"
> things on a weekly basis.  Go poke in the Android kernel tree if you
> want to see loads of examples of where vendors feel they can export/hook
> into core parts of Linux just to get their work done.  I would argue
> that most of them are NOT needed and are doing crazy things, but as
> users have real hardware that sometimes requires it, it's a hard thing
> to argue against.

I know you see real things. I think you misunderstood my comment. I
didn't mean to say what you see is anyway less than what happens in
core kernel. I find drivers are a key part of the kernel. The core
kernel is just a tool to have applications interact with the hardware,
and without drivers, the kernel is useless. My point was merely stating
that you and I have different view points of the kernel. I in no way
was saying mine was more important. In fact, I would say it's less
important.

> 
> I'll show one real-world example, the USB offload path code, that you
> are using today on your Android phones that saves loads of battery life.
> In older kernels, the hooks/exports needed for that were all over the
> place, it took an engineer years to get this all working for lots of
> different hardware types and merged upstream properly.  They knew that
> they could not just get the upstream developers to export the needed
> functions, they had to get their working code merged in order to be able
> to have it happen.  And they did!

I'm not sure how that is comparable? It sounds like they needed to
clean up a bunch of other code for other hardware so they could use
this feature.

The feature I'm talking about is fully available. It's just a way to
get a user space stack trace from the kernel. Nothing more. The work
being done is to make it generic so all the tracers could use it. All
that's upstream. The two functions I would like to export is "give me a
user space stack trace now" and "defer the user space stack trace
from NMI to when the task goes back to user space".

> 
> That was a non-trivial task, and while you might not feel it "does
> nothing", the power savings you rely on daily would beg to differ.
> 
> So get the lttng code merged into the tree please, it seems that you are
> the one objecting to the merge, not me :)

It was actually others, and it was years ago. Perhaps we could just
accept the LTTng kernel module as is, as a module, and leave it at that?

Then these functions would still have to be EXPORT_SYMBOL_GPL() but at
least it would be for an in-tree module.

-- Steve

On Fri, Jul 11, 2025 at 01:32:40PM -0400, Steven Rostedt wrote:
> > So get the lttng code merged into the tree please, it seems that you are
> > the one objecting to the merge, not me :)
> 
> It was actually others, and it was years ago. Perhaps we could just
> accept the LTTng kernel module as is, as a module, and leave it at that?

That's up to the maintainers of the subsystem where it would be, not me.

And that would solve this problem, so I do recommend it :)

thanks,

greg k-h

On Fri, 11 Jul 2025 20:24:37 +0200
Greg KH <gregkh@linuxfoundation.org> wrote:

> On Fri, Jul 11, 2025 at 01:32:40PM -0400, Steven Rostedt wrote:
> > > So get the lttng code merged into the tree please, it seems that you are
> > > the one objecting to the merge, not me :)  
> > 
> > It was actually others, and it was years ago. Perhaps we could just
> > accept the LTTng kernel module as is, as a module, and leave it at that?  
> 
> That's up to the maintainers of the subsystem where it would be, not me.

It would likely be its own subsystem (under kernel/lttng?), so I guess
it would be Linus's decision.

-- Steve