Hi Jon,

This is the splitted version of my RFC to use sphinx.ext.autodoc
to document kernel-doc modules. It documents only kernel-doc
modules, but, once we merge it, we can document other things.

This series comes after:
    https://lore.kernel.org/linux-doc/cover.1768395332.git.mchehab+huawei@kernel.org/T/#t

patch 1 is actually not directly related: if fixes a bug I noticed
on Sphinx indexes with a large maxdepth;

patch 2 adds sphinx.ext.autodoc and place the directories where
python modules can be found. It doesn't use the extension, though.
This will happen only at the final patch;

patches 3 to 12 are basically documentation cleanups and fixes that
ensure that the documentation will be complete.

patch 13 creates new *.rst files that points to python.lib.kdoc
files.

With this version, python files inside tools/ or scripts/
can be documented, but there is a catch (probably due to PEP8):

    - files must end with ".py"
    - file names can't have "-".

So, unfortunately, we can't document kernel-doc.py, except if
we rename it to kernel_doc.py.

As I mentioned at RFC, we could use:

    $ sphinx-apidoc scripts tools -o foobar
    $ sphinx-apidoc tools/docs -o foobar
    ...

to generate .rst skeletons with the modules to be added, but
manual work is needed afterwards.

Mauro Carvalho Chehab (13):
  docs: custom.css: prevent li marker to override text
  docs: enable Sphinx autodoc extension to allow documenting python
  docs: custom.css: add CSS for python
  docs: kdoc: latex_fonts: Improve docstrings and comments
  docs: kdoc_files: Improve docstrings and comments
  docs: kdoc_item: Improve docstrings and comments
  docs: kdoc_parser: Improve docstrings and comments
  docs: kdoc_output: Improve docstrings and comments
  docs: kdoc_re: Improve docstrings and comments
  docs: kdoc: parse_data_structs: Improve docstrings and comments
  docs: kdoc: enrich_formatter: Improve docstrings and comments
  docs: kdoc: python_version: Improve docstrings and comments
  docs: add kernel-doc modules documentation

 Documentation/conf.py                       |  11 +-
 Documentation/sphinx-static/custom.css      |  12 ++
 Documentation/tools/index.rst               |   1 +
 Documentation/tools/kdoc.rst                |  12 ++
 Documentation/tools/kdoc_ancillary.rst      |  46 +++++
 Documentation/tools/kdoc_output.rst         |  14 ++
 Documentation/tools/kdoc_parser.rst         |  29 ++++
 Documentation/tools/python.rst              |  10 ++
 tools/lib/python/kdoc/enrich_formatter.py   |  20 ++-
 tools/lib/python/kdoc/kdoc_files.py         |  23 +--
 tools/lib/python/kdoc/kdoc_item.py          |  18 ++
 tools/lib/python/kdoc/kdoc_output.py        |  60 ++++---
 tools/lib/python/kdoc/kdoc_parser.py        | 175 +++++++++++---------
 tools/lib/python/kdoc/kdoc_re.py            |  18 +-
 tools/lib/python/kdoc/latex_fonts.py        |  95 ++++++-----
 tools/lib/python/kdoc/parse_data_structs.py |  62 ++++---
 tools/lib/python/kdoc/python_version.py     |  20 ++-
 17 files changed, 430 insertions(+), 196 deletions(-)
 create mode 100644 Documentation/tools/kdoc.rst
 create mode 100644 Documentation/tools/kdoc_ancillary.rst
 create mode 100644 Documentation/tools/kdoc_output.rst
 create mode 100644 Documentation/tools/kdoc_parser.rst
 create mode 100644 Documentation/tools/python.rst

-- 
2.52.0

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> With this version, python files inside tools/ or scripts/
> can be documented, but there is a catch (probably due to PEP8):
>
>     - files must end with ".py"
>     - file names can't have "-".
>
> So, unfortunately, we can't document kernel-doc.py, except if
> we rename it to kernel_doc.py.

That is ... irritating ...

You've probably seen my other series to finally ... rename
kernel-doc.py.  We could certainly rename it to something different.
But I really dislike having language extensions on files meant to be
executed as commands; you shouldn't care what language it's written in
when you run it.

In the end, I'd say let's not worry about running autodoc on that file.
All there is to document is its command line, and we can certainly
maintain that separately.

jon

Em Wed, 14 Jan 2026 11:01:45 -0700
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > With this version, python files inside tools/ or scripts/
> > can be documented, but there is a catch (probably due to PEP8):
> >
> >     - files must end with ".py"
> >     - file names can't have "-".
> >
> > So, unfortunately, we can't document kernel-doc.py, except if
> > we rename it to kernel_doc.py.  
> 
> That is ... irritating ...

Agreed, but this is not really a problem with Sphinx but with
Python itself: you can't include any module that has "-" on it, or
doesn't end with ".py"[1].

[1] I guess it is actually possible, but only using low-level APIs,
    e.g. using importlib.util, importlib.util.module_from_spec(spec)
    and other dirty tricks, but this is certainly something we should
    avoid to do.

> You've probably seen my other series to finally ... rename
> kernel-doc.py. 

Just saw it ;-)

> We could certainly rename it to something different.
> But I really dislike having language extensions on files meant to be
> executed as commands; you shouldn't care what language it's written in
> when you run it.

I don't like it either, but Python is really picky on some things.

The problem here is that this is a Python policy violation. To change
that, one needs to write a PEP and convince Python maintainers to merge
it, together with changes on python "import" directive.

Alternatively, assuming that some magic words would be enough to
convince importlib to load a name without ".py" and with "-", it could be
easier to convince Sphinx autodoc maintainers to take a patch, as they're 
probably using importlib somewhere to dynamically import a file based 
at the string inside "automodule" directive. On a quick grep,
this seems to be the case, and such logic is inside:

	sphinx/ext/autodoc/importer.py


> In the end, I'd say let's not worry about running autodoc on that file.
> All there is to document is its command line, and we can certainly
> maintain that separately.

It may be true for kernel-doc, but:

- there are other scripts that we may want to document; and:
- we need to define a naming convention for python code

Also,

- keeping it in separate makes harder to maintain.

So, even if we don't actually add kernel-doc docstrings and
functions via autodoc, I think it is still worth having a
name convention that would allow that.

Thanks,
Mauro

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

>> We could certainly rename it to something different.
>> But I really dislike having language extensions on files meant to be
>> executed as commands; you shouldn't care what language it's written in
>> when you run it.
>
> I don't like it either, but Python is really picky on some things.
>
> The problem here is that this is a Python policy violation. To change
> that, one needs to write a PEP and convince Python maintainers to merge
> it, together with changes on python "import" directive.

...or just ignore it.  There is a reason that "pip" is called "pip"
rather than "pip.py" - the Python folks don't keep those extensions on
commands either.

> Alternatively, assuming that some magic words would be enough to
> convince importlib to load a name without ".py" and with "-", it could be
> easier to convince Sphinx autodoc maintainers to take a patch, as they're 
> probably using importlib somewhere to dynamically import a file based 
> at the string inside "automodule" directive. On a quick grep,
> this seems to be the case, and such logic is inside:
>
> 	sphinx/ext/autodoc/importer.py

No doubt we could do that.  But is it really worth the trouble?  There
is not much in kernel-doc that needs documenting, especially since you
did the work to move the actual functionality into separate modules. 

> So, even if we don't actually add kernel-doc docstrings and
> functions via autodoc, I think it is still worth having a
> name convention that would allow that.

Instead, I think you're trying to take a functionality meant to describe
APIs and use it to document command-line stuff.  I'm happy to live by
the import rules for stuff that is actually imported; I think it makes
less sense to let them drive the naming of files that are outside of
their intended scope.

jon

Em Wed, 14 Jan 2026 13:46:31 -0700
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> >> We could certainly rename it to something different.
> >> But I really dislike having language extensions on files meant to be
> >> executed as commands; you shouldn't care what language it's written in
> >> when you run it.  
> >
> > I don't like it either, but Python is really picky on some things.
> >
> > The problem here is that this is a Python policy violation. To change
> > that, one needs to write a PEP and convince Python maintainers to merge
> > it, together with changes on python "import" directive.  
> 
> ...or just ignore it. 

Indeed this is an option.

> There is a reason that "pip" is called "pip"
> rather than "pip.py" - the Python folks don't keep those extensions on
> commands either.

True, but see what pip has:

	$ more /usr/bin/pip
	#! /usr/bin/python3 -P
	# -*- coding: utf-8 -*-
	import re
	import sys
	from pip._internal.cli.main import main
	if __name__ == '__main__':
	    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
	    sys.exit(main())
	/usr/bin/pip (END)

Everything, including main are outside it. Btw, this code is almost
identical to sphinx-build:

	$ more /usr/bin/sphinx-build
	#! /usr/bin/python3 -P
	# -*- coding: utf-8 -*-
	import re
	import sys
	from sphinx.cmd.build import main
	if __name__ == '__main__':
	    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
	    sys.exit(main())

This would be equivalent of having a tools/docs/kernel-doc like this
(untested):

	#!/usr/bin/env python3
	from .kdoc.main import main
	if __name__ == '__main__':
	    sys.exit(main())

where the actual argparse code would be inside tools/docs/kdoc/main.py

> > Alternatively, assuming that some magic words would be enough to
> > convince importlib to load a name without ".py" and with "-", it could be
> > easier to convince Sphinx autodoc maintainers to take a patch, as they're 
> > probably using importlib somewhere to dynamically import a file based 
> > at the string inside "automodule" directive. On a quick grep,
> > this seems to be the case, and such logic is inside:
> >
> > 	sphinx/ext/autodoc/importer.py  
> 
> No doubt we could do that.  But is it really worth the trouble?  There
> is not much in kernel-doc that needs documenting, especially since you
> did the work to move the actual functionality into separate modules. 

I'm not particularly concerned about kernel-doc here. I'm more
concerned on defining how things like that are expected to be
documented.

Yet, if you add this:

	.. automodule:: docs.kernel_doc
	   :members:
	   :show-inheritance:
	   :undoc-members:

The generated documentation sounds somewhat relevant to me - specially
if placed together with the kernel-doc module API documentation:

	kernel-doc module documentation
	===============================

	kernel_doc

	Print formatted kernel documentation to stdout

	Read C language source or header FILEs, extract embedded documentation comments,
	and print formatted documentation to standard output.

	The documentation comments are identified by the /** opening comment mark.

	See Documentation/doc-guide/kernel-doc.rst for the documentation comment syntax.

	class docs.kernel_doc.MsgFormatter(fmt=None, datefmt=None, style='%', validate=True, *, defaults=None)

	    Bases: Formatter

	    Helper class to format warnings in a similar way to kernel-doc.pl.

	    format(record)

	        Format the specified record as text.

	        The record’s attribute dictionary is used as the operand to a 
		string formatting operation which yields the returned string. 
		Before formatting the dictionary, a couple of preparatory
		steps are carried out. The message attribute of the record
		is computed using LogRecord.getMessage(). If the formatting
		string uses the time (as determined by a call to usesTime(),
		formatTime() is called to format the event time. If there is 
		exception information, it is formatted using formatException()
		and appended to the message.

	docs.kernel_doc.main()

	    Main program By default, the return value is:

	        0: success or Python version is not compatible with
		   kernel-doc. If -Werror is not used, it will also
        	   return 0 if there are issues at kernel-doc markups;

	        1: an abnormal condition happened;

	        2: argparse issued an error;

	        3: -Werror is used, and one or more unfiltered parse warnings
	           happened.

> > So, even if we don't actually add kernel-doc docstrings and
> > functions via autodoc, I think it is still worth having a
> > name convention that would allow that.  
> 
> Instead, I think you're trying to take a functionality meant to describe
> APIs and use it to document command-line stuff.  I'm happy to live by
> the import rules for stuff that is actually imported; I think it makes
> less sense to let them drive the naming of files that are outside of
> their intended scope.

Yeah, `MsgFormatter` doesn't belong to doc-guide. main return values
does, however. IMO it is important to keep it documented together with
the code.

It should be noticed that autodoc has support for selecting symbols.
So:

	.. automodule:: docs.kernel_doc
	   :members: main

would pick only module description + main

and if we add, instead:

	.. autofunction:: docs.kernel_doc.main

it would just pick docstrings for main, e.g. it would place just
this:

	docs.kernel_doc.main()

	    Main program By default, the return value is:

	        0: success or Python version is not compatible with
		   kernel-doc. If -Werror is not used, it will also
        	   return 0 if there are issues at kernel-doc markups;

	        1: an abnormal condition happened;

	        2: argparse issued an error;

	        3: -Werror is used, and one or more unfiltered parse warnings
	           happened.

---

In summary, all I'm saying is that, if we stick to PEP8 names, we
can opt to import a documentation directly from the script instead
of writing it twice: at the code and on a rst file.

Btw, if you want to test it, you need just one patch to enable
it:

	https://lore.kernel.org/linux-doc/6aa5a5b4a686f07c8f3e6cb04fe4c07ed9c1d071.1768396023.git.mchehab+huawei@kernel.org/T/#u

this basically allows using tools/ and scripts/ as the base for
documentation.

You may also want this css patch, as default format with alabaster
is very ugly:

	https://lore.kernel.org/linux-doc/8d66988f05d06d10938e062ed4465bf303c51441.1768396023.git.mchehab+huawei@kernel.org/T/#u
	
with that, you can experiment inserting autodoc stuff using:	

	.. automodule:: docs.name
	   :modules:

to document all public methods from "docs/name.py" file, or:

	.. autofunction:: docs.name.function

and/or:

	.. autoclass:: docs.name.class

for a single "function" (or "class") inside "docs/name.py".

Again, the limitation is that "name" ends with ".py" and only
have (lower case?) letters, numbers and underscores - e.g. it
shall be something that "import" and "from ... import" supports.


Thanks,
Mauro

On Thu, Jan 15, 2026 at 02:29:21AM +0100, Mauro Carvalho Chehab wrote:
> Em Wed, 14 Jan 2026 13:46:31 -0700
> Jonathan Corbet <corbet@lwn.net> escreveu:
> 
> > Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> > 
> > >> We could certainly rename it to something different.
> > >> But I really dislike having language extensions on files meant to be
> > >> executed as commands; you shouldn't care what language it's written in
> > >> when you run it.  
> > >
> > > I don't like it either, but Python is really picky on some things.
> > >
> > > The problem here is that this is a Python policy violation. To change
> > > that, one needs to write a PEP and convince Python maintainers to merge
> > > it, together with changes on python "import" directive.  

> > There is a reason that "pip" is called "pip"
> > rather than "pip.py" - the Python folks don't keep those extensions on
> > commands either.
> 
> This would be equivalent of having a tools/docs/kernel-doc like this
> (untested):
> 
> 	#!/usr/bin/env python3
> 	from .kdoc.main import main
> 	if __name__ == '__main__':
> 	    sys.exit(main())
> 
> where the actual argparse code would be inside tools/docs/kdoc/main.py

In time, I'm not suggesting doing that at all.

I don't like the idea of splitting main() into a separate file: at least 
in my head, the absolute minimum size of a script should be to be able 
to handle command line arguments by itself, without including a module 
for it.

> 	class docs.kernel_doc.MsgFormatter(fmt=None, datefmt=None, style='%', validate=True, *, defaults=None)
> 
> 	    Bases: Formatter
> 
> 	    Helper class to format warnings in a similar way to kernel-doc.pl.
> 
> 	    format(record)
> 
> 	        Format the specified record as text.
> 
> 	        The record’s attribute dictionary is used as the operand to a 
> 		string formatting operation which yields the returned string. 
> 		Before formatting the dictionary, a couple of preparatory
> 		steps are carried out. The message attribute of the record
> 		is computed using LogRecord.getMessage(). If the formatting
> 		string uses the time (as determined by a call to usesTime(),
> 		formatTime() is called to format the event time. If there is 
> 		exception information, it is formatted using formatException()
> 		and appended to the message.

I also think that we should move MsgFormatter to a separate file. What 
this class does is that it bolds arguments when argparse displays help. 
It has nothing to do with kernel-doc itself, and can be reused on all 
other places where we use argparse. So, placing it under 
tools/lib/python, outside kdoc dir, IMO makes sense.

The same applies to some other stuff currently under kdoc, like 
enrich_formatter.py. Once the series moving kernel-doc to tools/docs 
gets merged, IMHO the next step would be to do some cleanup at 
tools/lib/python, moving things that aren't independent on the actual 
parser to be outside kdoc directory.

Thanks,
Mauro