Kernel-doc has an obscure logic that uses an external file
to map files via a .tmp_filelist.txt file stored at the current
directory. The rationale for such code predates git time,
as it was added on Kernel v2.4.5.5, with the following description:

	# 26/05/2001 -         Support for separate source and object trees.
	#              Return error code.
	#              Keith Owens <kaos@ocs.com.au>

from commit 396a6123577d ("v2.4.5.4 -> v2.4.5.5") at the historic
tree:
	https://git.kernel.org/pub/scm/linux/kernel/git/history/history.git/

Support for separate source and object trees is now done on a different
way via make O=<object>.

There's no logic to create such file, so it sounds to me that this is
just dead code.

So, drop it.

Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 scripts/kernel-doc | 19 +------------------
 1 file changed, 1 insertion(+), 18 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index e57c5e989a0a..70da9a3369c6 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -179,7 +179,7 @@ my ($function, %function_table, %parametertypes, $declaration_purpose);
 my %nosymbol_table = ();
 my $declaration_start_line;
 my ($type, $declaration_name, $return_type);
-my ($newsection, $newcontents, $prototype, $brcount, %source_map);
+my ($newsection, $newcontents, $prototype, $brcount);
 
 if (defined($ENV{'KBUILD_VERBOSE'}) && $ENV{'KBUILD_VERBOSE'} =~ '1') {
     $verbose = 1;
@@ -2005,10 +2005,6 @@ sub map_filename($) {
         $file = $orig_file;
     }
 
-    if (defined($source_map{$file})) {
-        $file = $source_map{$file};
-    }
-
     return $file;
 }
 
@@ -2403,19 +2399,6 @@ for (my $k = 0; $k < @highlights; $k++) {
     $dohighlight .=  "\$contents =~ s:$pattern:$result:gs;\n";
 }
 
-# Read the file that maps relative names to absolute names for
-# separate source and object directories and for shadow trees.
-if (open(SOURCE_MAP, "<.tmp_filelist.txt")) {
-    my ($relname, $absname);
-    while(<SOURCE_MAP>) {
-        chop();
-        ($relname, $absname) = (split())[0..1];
-        $relname =~ s:^/+::;
-        $source_map{$relname} = $absname;
-    }
-    close(SOURCE_MAP);
-}
-
 if ($output_selection == OUTPUT_EXPORTED ||
     $output_selection == OUTPUT_INTERNAL) {
 
-- 
2.48.1

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> Kernel-doc has an obscure logic that uses an external file
> to map files via a .tmp_filelist.txt file stored at the current
> directory. The rationale for such code predates git time,
> as it was added on Kernel v2.4.5.5, with the following description:
>
> 	# 26/05/2001 -         Support for separate source and object trees.
> 	#              Return error code.
> 	#              Keith Owens <kaos@ocs.com.au>
>
> from commit 396a6123577d ("v2.4.5.4 -> v2.4.5.5") at the historic
> tree:
> 	https://git.kernel.org/pub/scm/linux/kernel/git/history/history.git/
>
> Support for separate source and object trees is now done on a different
> way via make O=<object>.
>
> There's no logic to create such file, so it sounds to me that this is
> just dead code.
>
> So, drop it.
>
> Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> ---
>  scripts/kernel-doc | 19 +------------------
>  1 file changed, 1 insertion(+), 18 deletions(-)

Weird ... I went and looked, and can't find anything that ever created
that tmp_filelist.txt file; I wonder if this code ever did anything?

Don't put that functionality into the Python version :)

Applied, thanks.

jon

Em Thu, 13 Feb 2025 09:35:58 -0700
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > Kernel-doc has an obscure logic that uses an external file
> > to map files via a .tmp_filelist.txt file stored at the current
> > directory. The rationale for such code predates git time,
> > as it was added on Kernel v2.4.5.5, with the following description:
> >
> > 	# 26/05/2001 -         Support for separate source and object trees.
> > 	#              Return error code.
> > 	#              Keith Owens <kaos@ocs.com.au>
> >
> > from commit 396a6123577d ("v2.4.5.4 -> v2.4.5.5") at the historic
> > tree:
> > 	https://git.kernel.org/pub/scm/linux/kernel/git/history/history.git/
> >
> > Support for separate source and object trees is now done on a different
> > way via make O=<object>.
> >
> > There's no logic to create such file, so it sounds to me that this is
> > just dead code.
> >
> > So, drop it.
> >
> > Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> > ---
> >  scripts/kernel-doc | 19 +------------------
> >  1 file changed, 1 insertion(+), 18 deletions(-)  
> 
> Weird ... I went and looked, and can't find anything that ever created
> that tmp_filelist.txt file; I wonder if this code ever did anything?

I wonder the same ;-) Anyway, better to remove this now, as, if people
complain, it would be easier to revert than after switching to the
Python version.

> Don't put that functionality into the Python version :)

Yeah, I started implementing it, but it sounded a waste of time, so
I dropped it from the RFC versions. It sounded too complex for people
to maintain a separate tmp file when make O=dir would do it on a much
better and automated way.

-

With regards to the Python transition, since our Makefile allows
switching to a different script since ever[1], I'm playing with 
the idea of sending a patch series with:

Patch 1: 
  - drops Sphinx version check from both kerneldoc 
    (-sphinx-version parameter) and the corresponding Sphinx extension;

patch 2: 
  - renames kerneldoc to kerneldoc.pl
  - creates a symlink:
	kerneldoc.pl -> kerneldoc

patch 3:
  - adds kerneldoc.py:

patch 4:
  - add info messages on both versions related to the transition,
    and instructions about using KERNELDOC=<script> makefile and ask
    people to report eventual regressions with new script.

patch 5:
  - change kerneldoc symlink to point to kerneldoc.py

We can then keep both for maybe one Kernel cycle and see how it goes,
stop accepting patches to the Perl version, in favor of doing the needed
changes at the Python one.

If everything goes well, we can remove the venerable Perl version on the 
upcoming merge window, and change the Sphinx extension to use the Python
classes directly instead of running an external executable code.

What do you think?

-

I'm in doubt if I should split the Kernel classes for the Python version
into a scripts/lib/kdoc directory on this series or doing such change
only after we drop the Perl version.

Keeping it on a single file helps to do more complex code adjustments 
on a single place, specially if we end renaming/shifting stuff[2].

[1] I didn't remember about that - it is a very welcomed feature,
    probably thanks to Markus.

[2] the currently global const regex macros is something that I want
    to rename and place them inside a class (or on multiple classes).
    Also, Python coding style is to use uppercase for const. There is
    currently a Pylint disabled warning about that. So, I do plan to
    do such changes in the future to make it more compatible with
    Python coding style.

-

On a separate but related issue, perhaps we should start talking about
coding style. We don't have anything defined at the Kernel, and
different scripts follow different conventions (or most likely
don't follow any convention at all). We should probably think having 
something defined in the future.

From my side, I like Pylint warnings, except for the (IMHO) useless "Rxxx"
warnings that complains about too many/too few things. They have
been useful to detect hidden bugs at scripts, and it allows inlined
exceptions to the coding style.

autopep8 autoformatter is also nice (and, up to some point, black),
as it auto-corrects whitespace issues, but there's two things I don't 
like on them:

1. its coding style on lines, creating function calls with open 
   parenthesis:

	-    parser.add_argument("-n", "-nosymbol", "--nosymbol", action='append',
	-                         help=NOSYMBOL_DESC)
	+    parser.add_argument(
	+        "-n", "-nosymbol", "--nosymbol", action="append", help=NOSYMBOL_DESC
	+    )

2. whitespace removal on aligned consts:

	-    STATE_INLINE_NA     = 0 # not applicable ($state != STATE_INLINE)
	-    STATE_INLINE_NAME   = 1 # looking for member name (@foo:)
	-    STATE_INLINE_TEXT   = 2 # looking for member documentation
	-    STATE_INLINE_END    = 3 # done
	-    STATE_INLINE_ERROR  = 4 # error - Comment without header was found.
	-                            # Spit a warning as it's not
	-                            # proper kernel-doc and ignore the rest.
	+    STATE_INLINE_NA = 0  # not applicable ($state != STATE_INLINE)
	+    STATE_INLINE_NAME = 1  # looking for member name (@foo:)
	+    STATE_INLINE_TEXT = 2  # looking for member documentation	
	+    STATE_INLINE_END = 3  # done
	+    STATE_INLINE_ERROR = 4  # error - Comment without header was found.

What I do here is from time to time manually run them and cherry-pick
only changes that sounds good to my personal taste.

It is probably a good idea to define a coding style and perhaps add
some config files (like a .pep8 file) to have a single coding style for
future code, letting scripts/checkpatch.pl to run pylint and/or some
other coding style tool(s).

Thanks,
Mauro

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> With regards to the Python transition, since our Makefile allows
> switching to a different script since ever[1], I'm playing with 
> the idea of sending a patch series with:
>
> Patch 1: 
>   - drops Sphinx version check from both kerneldoc 
>     (-sphinx-version parameter) and the corresponding Sphinx extension;
>
> patch 2: 
>   - renames kerneldoc to kerneldoc.pl
>   - creates a symlink:
> 	kerneldoc.pl -> kerneldoc
>
> patch 3:
>   - adds kerneldoc.py:
>
> patch 4:
>   - add info messages on both versions related to the transition,
>     and instructions about using KERNELDOC=<script> makefile and ask
>     people to report eventual regressions with new script.
>
> patch 5:
>   - change kerneldoc symlink to point to kerneldoc.py
>
> We can then keep both for maybe one Kernel cycle and see how it goes,
> stop accepting patches to the Perl version, in favor of doing the needed
> changes at the Python one.
>
> If everything goes well, we can remove the venerable Perl version on the 
> upcoming merge window, and change the Sphinx extension to use the Python
> classes directly instead of running an external executable code.
>
> What do you think?

Seems like a fine plan in general.  I wonder if we might want to keep
the old kernel-doc a bit longer just in case, but we can decide that as
we go.

> I'm in doubt if I should split the Kernel classes for the Python version
> into a scripts/lib/kdoc directory on this series or doing such change
> only after we drop the Perl version.
>
> Keeping it on a single file helps to do more complex code adjustments 
> on a single place, specially if we end renaming/shifting stuff[2].

Do whatever makes it easiest for you at this point, I'd say.

> On a separate but related issue, perhaps we should start talking about
> coding style. We don't have anything defined at the Kernel, and
> different scripts follow different conventions (or most likely
> don't follow any convention at all). We should probably think having 
> something defined in the future.

I've generally tried to stick to something that looks as close to the C
coding style as possible.  Formalizing that might not be a bad idea at
all.

jon

Em Tue, 18 Feb 2025 13:59:10 -0700
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > With regards to the Python transition, since our Makefile allows
> > switching to a different script since ever[1], I'm playing with 
> > the idea of sending a patch series with:
> >
> > Patch 1: 
> >   - drops Sphinx version check from both kerneldoc 
> >     (-sphinx-version parameter) and the corresponding Sphinx extension;
> >
> > patch 2: 
> >   - renames kerneldoc to kerneldoc.pl
> >   - creates a symlink:
> > 	kerneldoc.pl -> kerneldoc
> >
> > patch 3:
> >   - adds kerneldoc.py:
> >
> > patch 4:
> >   - add info messages on both versions related to the transition,
> >     and instructions about using KERNELDOC=<script> makefile and ask
> >     people to report eventual regressions with new script.
> >
> > patch 5:
> >   - change kerneldoc symlink to point to kerneldoc.py
> >
> > We can then keep both for maybe one Kernel cycle and see how it goes,
> > stop accepting patches to the Perl version, in favor of doing the needed
> > changes at the Python one.
> >
> > If everything goes well, we can remove the venerable Perl version on the 
> > upcoming merge window, and change the Sphinx extension to use the Python
> > classes directly instead of running an external executable code.
> >
> > What do you think?  
> 
> Seems like a fine plan in general.  I wonder if we might want to keep
> the old kernel-doc a bit longer just in case, but we can decide that as
> we go.

Yeah, I'm unsure about keeping the old kernel-doc or not. Anyway, I'll
send a patch series without dropping the old kernel-doc script. 

We can remove it when we feel it is time for it.

> 
> > I'm in doubt if I should split the Kernel classes for the Python version
> > into a scripts/lib/kdoc directory on this series or doing such change
> > only after we drop the Perl version.
> >
> > Keeping it on a single file helps to do more complex code adjustments 
> > on a single place, specially if we end renaming/shifting stuff[2].  
> 
> Do whatever makes it easiest for you at this point, I'd say.

Ok, I'm opting to split it on multiple files, on separate patches. This
way, it is easier to review/compare the new script with the old one, in
case someone wants to do that.

For now, I opted to keep the two output classes (for rest and man output),
plus a base class used by both altogether. We may split it further in the
future, as needed.

With that, the final patchset will contain those files:

scripts/kernel-doc.py		- Command line tool
scripts/lib/kdoc/kdoc_re.py     - Regex ancillary classes
scripts/lib/kdoc/kdoc_parser.py - kernel-doc single file parser
scripts/lib/kdoc/kdoc_output.py - output classes
scripts/lib/kdoc/kdoc_files.py  - kernel-doc dispatcher for multiple 
				  files, providing a glue between the
				  parser and the output classes.

Please notice that the new tool can now parse multiple files at the
same time. For instance, if you want to generate a ReST file with
everything inside the Kernel tree, you could run:

	./scripts/kernel-doc.py .

(on my machine, this takes about 1 minute to run)

Such extra functionality is given by the class inside kdoc_files.py.

> > On a separate but related issue, perhaps we should start talking about
> > coding style. We don't have anything defined at the Kernel, and
> > different scripts follow different conventions (or most likely
> > don't follow any convention at all). We should probably think having 
> > something defined in the future.  
> 
> I've generally tried to stick to something that looks as close to the C
> coding style as possible.  Formalizing that might not be a bad idea at
> all.

Ok, I'll prepare something in the future to formalize a coding style.

For now, I'll add a .pylintrc file just to include the scripts/lib/*
directories we're currently using.

Thanks,
Mauro

Hi--

On 2/13/25 6:24 PM, Mauro Carvalho Chehab wrote:
> Em Thu, 13 Feb 2025 09:35:58 -0700
> Jonathan Corbet <corbet@lwn.net> escreveu:
> 
>> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
>>
>>> Kernel-doc has an obscure logic that uses an external file
>>> to map files via a .tmp_filelist.txt file stored at the current
>>> directory. The rationale for such code predates git time,
>>> as it was added on Kernel v2.4.5.5, with the following description:
>>>
>>> 	# 26/05/2001 -         Support for separate source and object trees.
>>> 	#              Return error code.
>>> 	#              Keith Owens <kaos@ocs.com.au>
>>>
>>> from commit 396a6123577d ("v2.4.5.4 -> v2.4.5.5") at the historic
>>> tree:
>>> 	https://git.kernel.org/pub/scm/linux/kernel/git/history/history.git/
>>>
>>> Support for separate source and object trees is now done on a different
>>> way via make O=<object>.
>>>
>>> There's no logic to create such file, so it sounds to me that this is
>>> just dead code.
>>>
>>> So, drop it.
>>>
>>> Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
>>> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
>>> ---
>>>  scripts/kernel-doc | 19 +------------------
>>>  1 file changed, 1 insertion(+), 18 deletions(-)  
>>
>> Weird ... I went and looked, and can't find anything that ever created
>> that tmp_filelist.txt file; I wonder if this code ever did anything?
> 
> I wonder the same ;-) Anyway, better to remove this now, as, if people
> complain, it would be easier to revert than after switching to the
> Python version.
> 
>> Don't put that functionality into the Python version :)
> 
> Yeah, I started implementing it, but it sounded a waste of time, so
> I dropped it from the RFC versions. It sounded too complex for people
> to maintain a separate tmp file when make O=dir would do it on a much
> better and automated way.
> 
> -
> 
> With regards to the Python transition, since our Makefile allows
> switching to a different script since ever[1], I'm playing with 
> the idea of sending a patch series with:
> 
> Patch 1: 
>   - drops Sphinx version check from both kerneldoc 
>     (-sphinx-version parameter) and the corresponding Sphinx extension;
> 

It's currently scripts/kernel-doc. Are you planning to change it to
scripts/kerneldoc and break other scripts and makefiles?

> patch 2: 
>   - renames kerneldoc to kerneldoc.pl
>   - creates a symlink:
> 	kerneldoc.pl -> kerneldoc
> 
> patch 3:
>   - adds kerneldoc.py:
> 
> patch 4:
>   - add info messages on both versions related to the transition,
>     and instructions about using KERNELDOC=<script> makefile and ask
>     people to report eventual regressions with new script.
> 
> patch 5:
>   - change kerneldoc symlink to point to kerneldoc.py
> 
> We can then keep both for maybe one Kernel cycle and see how it goes,
> stop accepting patches to the Perl version, in favor of doing the needed
> changes at the Python one.
> 
> If everything goes well, we can remove the venerable Perl version on the 
> upcoming merge window, and change the Sphinx extension to use the Python
> classes directly instead of running an external executable code.
> 
> What do you think?
> 
> -
> 
> I'm in doubt if I should split the Kernel classes for the Python version
> into a scripts/lib/kdoc directory on this series or doing such change
> only after we drop the Perl version.
> 
> Keeping it on a single file helps to do more complex code adjustments 
> on a single place, specially if we end renaming/shifting stuff[2].
> 
> [1] I didn't remember about that - it is a very welcomed feature,
>     probably thanks to Markus.
> 
> [2] the currently global const regex macros is something that I want
>     to rename and place them inside a class (or on multiple classes).
>     Also, Python coding style is to use uppercase for const. There is
>     currently a Pylint disabled warning about that. So, I do plan to
>     do such changes in the future to make it more compatible with
>     Python coding style.
> 
> -
> 
> On a separate but related issue, perhaps we should start talking about
> coding style. We don't have anything defined at the Kernel, and
> different scripts follow different conventions (or most likely
> don't follow any convention at all). We should probably think having 
> something defined in the future.
> 
> From my side, I like Pylint warnings, except for the (IMHO) useless "Rxxx"
> warnings that complains about too many/too few things. They have
> been useful to detect hidden bugs at scripts, and it allows inlined
> exceptions to the coding style.
> 
> autopep8 autoformatter is also nice (and, up to some point, black),
> as it auto-corrects whitespace issues, but there's two things I don't 
> like on them:
> 
> 1. its coding style on lines, creating function calls with open 
>    parenthesis:
> 
> 	-    parser.add_argument("-n", "-nosymbol", "--nosymbol", action='append',
> 	-                         help=NOSYMBOL_DESC)
> 	+    parser.add_argument(
> 	+        "-n", "-nosymbol", "--nosymbol", action="append", help=NOSYMBOL_DESC
> 	+    )
> 
> 2. whitespace removal on aligned consts:
> 
> 	-    STATE_INLINE_NA     = 0 # not applicable ($state != STATE_INLINE)
> 	-    STATE_INLINE_NAME   = 1 # looking for member name (@foo:)
> 	-    STATE_INLINE_TEXT   = 2 # looking for member documentation
> 	-    STATE_INLINE_END    = 3 # done
> 	-    STATE_INLINE_ERROR  = 4 # error - Comment without header was found.
> 	-                            # Spit a warning as it's not
> 	-                            # proper kernel-doc and ignore the rest.
> 	+    STATE_INLINE_NA = 0  # not applicable ($state != STATE_INLINE)
> 	+    STATE_INLINE_NAME = 1  # looking for member name (@foo:)
> 	+    STATE_INLINE_TEXT = 2  # looking for member documentation	
> 	+    STATE_INLINE_END = 3  # done
> 	+    STATE_INLINE_ERROR = 4  # error - Comment without header was found.
> 
> What I do here is from time to time manually run them and cherry-pick
> only changes that sounds good to my personal taste.
> 
> It is probably a good idea to define a coding style and perhaps add
> some config files (like a .pep8 file) to have a single coding style for
> future code, letting scripts/checkpatch.pl to run pylint and/or some
> other coding style tool(s).
> 
> Thanks,
> Mauro
> 

-- 
~Randy

Em Thu, 13 Feb 2025 18:38:47 -0800
Randy Dunlap <rdunlap@infradead.org> escreveu:

> Hi--
> 
> On 2/13/25 6:24 PM, Mauro Carvalho Chehab wrote:
> > Em Thu, 13 Feb 2025 09:35:58 -0700
> > Jonathan Corbet <corbet@lwn.net> escreveu:
> >   
> >> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> >>  
> >>> Kernel-doc has an obscure logic that uses an external file
> >>> to map files via a .tmp_filelist.txt file stored at the current
> >>> directory. The rationale for such code predates git time,
> >>> as it was added on Kernel v2.4.5.5, with the following description:
> >>>
> >>> 	# 26/05/2001 -         Support for separate source and object trees.
> >>> 	#              Return error code.
> >>> 	#              Keith Owens <kaos@ocs.com.au>
> >>>
> >>> from commit 396a6123577d ("v2.4.5.4 -> v2.4.5.5") at the historic
> >>> tree:
> >>> 	https://git.kernel.org/pub/scm/linux/kernel/git/history/history.git/
> >>>
> >>> Support for separate source and object trees is now done on a different
> >>> way via make O=<object>.
> >>>
> >>> There's no logic to create such file, so it sounds to me that this is
> >>> just dead code.
> >>>
> >>> So, drop it.
> >>>
> >>> Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> >>> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> >>> ---
> >>>  scripts/kernel-doc | 19 +------------------
> >>>  1 file changed, 1 insertion(+), 18 deletions(-)    
> >>
> >> Weird ... I went and looked, and can't find anything that ever created
> >> that tmp_filelist.txt file; I wonder if this code ever did anything?  
> > 
> > I wonder the same ;-) Anyway, better to remove this now, as, if people
> > complain, it would be easier to revert than after switching to the
> > Python version.
> >   
> >> Don't put that functionality into the Python version :)  
> > 
> > Yeah, I started implementing it, but it sounded a waste of time, so
> > I dropped it from the RFC versions. It sounded too complex for people
> > to maintain a separate tmp file when make O=dir would do it on a much
> > better and automated way.
> > 
> > -
> > 
> > With regards to the Python transition, since our Makefile allows
> > switching to a different script since ever[1], I'm playing with 
> > the idea of sending a patch series with:
> > 
> > Patch 1: 
> >   - drops Sphinx version check from both kerneldoc 
> >     (-sphinx-version parameter) and the corresponding Sphinx extension;
> >   
> 
> It's currently scripts/kernel-doc. Are you planning to change it to
> scripts/kerneldoc and break other scripts and makefiles?

No, the idea is to keep it as kernel-doc.

I always confuse the names as we have both, depending on where you
look at:
	
	- scripts/kernel-doc
	- Documentation/sphinx/kerneldoc.py

The Python version was written to support all command-line parameters
as the original one - although I introduced both a single line and a
two dash alternative.

It also expects the file name(s) to be after the parameters, just like
kernel-doc.

I just changed the logger formatter to be similar to what we have
on kernel-doc:

	Warning: <msg>

So, I expect that such change will cause minimal impact on existing
scripts.

Thanks,
Mauro