[PATCH 00/25] Add tools/lib/python modules to Documentation/tools

Mauro Carvalho Chehab posted 25 patches 2 weeks, 5 days ago
Documentation/conf.py                       |  23 +--
Documentation/sphinx-static/custom.css      |  12 ++
Documentation/tools/feat.rst                |  10 ++
Documentation/tools/index.rst               |   1 +
Documentation/tools/jobserver.rst           |  10 ++
Documentation/tools/kabi.rst                |  13 ++
Documentation/tools/kabi_helpers.rst        |  11 ++
Documentation/tools/kabi_parser.rst         |  10 ++
Documentation/tools/kabi_regex.rst          |  10 ++
Documentation/tools/kabi_symbols.rst        |  10 ++
Documentation/tools/kdoc.rst                |  12 ++
Documentation/tools/kdoc_ancillary.rst      |  46 ++++++
Documentation/tools/kdoc_output.rst         |  14 ++
Documentation/tools/kdoc_parser.rst         |  29 ++++
Documentation/tools/python.rst              |  13 ++
tools/lib/python/abi/abi_parser.py          |  33 ++--
tools/lib/python/abi/abi_regex.py           |  26 ++-
tools/lib/python/abi/helpers.py             |  42 ++---
tools/lib/python/abi/system_symbols.py      |  14 +-
tools/lib/python/feat/parse_features.py     |  27 +++-
tools/lib/python/jobserver.py               |  20 ++-
tools/lib/python/kdoc/enrich_formatter.py   |  20 ++-
tools/lib/python/kdoc/kdoc_files.py         |  23 +--
tools/lib/python/kdoc/kdoc_item.py          |  18 +++
tools/lib/python/kdoc/kdoc_output.py        |  60 ++++---
tools/lib/python/kdoc/kdoc_parser.py        | 169 +++++++++++---------
tools/lib/python/kdoc/kdoc_re.py            |  18 ++-
tools/lib/python/kdoc/latex_fonts.py        |  95 ++++++-----
tools/lib/python/kdoc/parse_data_structs.py |  62 ++++---
tools/lib/python/kdoc/python_version.py     |  20 ++-
30 files changed, 607 insertions(+), 264 deletions(-)
create mode 100644 Documentation/tools/feat.rst
create mode 100644 Documentation/tools/jobserver.rst
create mode 100644 Documentation/tools/kabi.rst
create mode 100644 Documentation/tools/kabi_helpers.rst
create mode 100644 Documentation/tools/kabi_parser.rst
create mode 100644 Documentation/tools/kabi_regex.rst
create mode 100644 Documentation/tools/kabi_symbols.rst
create mode 100644 Documentation/tools/kdoc.rst
create mode 100644 Documentation/tools/kdoc_ancillary.rst
create mode 100644 Documentation/tools/kdoc_output.rst
create mode 100644 Documentation/tools/kdoc_parser.rst
create mode 100644 Documentation/tools/python.rst
[PATCH 00/25] Add tools/lib/python modules to Documentation/tools
Posted by Mauro Carvalho Chehab 2 weeks, 5 days ago
Hi Jon,

This is an extended version of:
    https://lore.kernel.org/linux-doc/cover.1768488832.git.mchehab+huawei@kernel.org/

It basically adds everything we currently have inside libs/tool/python
to "tools" book inside documentation.

This version should be independent of the other series yet to be merged,
(including the jobserver one).

The vast amount of changes here are docstring cleanups and additions.
They mainly consists on:

- ensuring that every phrase will end with a period, making it uniform
  along all files;
- cleaning ups to better uniform docstrings;
- variable descriptions now use "#:" markup, as it allows autodoc to
  add them inside the documentation;
- added some missing docstrings;
- some new blank lines at comments to make ReST syntax parser happy;
- add a couple of sphinx markups (mainly, code blocks).

Most of those are minor changes, affecting only comments.

It also has one patch per libarary type, adding them to docs.

For kernel-doc, I did the cleanups first, as there is one code block
inside tools/lib/python/kdoc/latex_fonts.py that would cause a Sphinx
crash without such markups.

The series actually starts with 3 fixes:

- avoid "*" markups on indexes with deep> 3 to override text
- a variable rename to stop abusing doctree name
- don't rely on cwd to get Documentation/ location

patch 4 adds support to document scripts either at:
    - tools/
    - scripts/

patch 5 contains a CSS to better display autodoc html output.

For those who want to play with documentation, documenting a python
file is very simple. All it takes is to use:

    .. automodule:: lib.python.<dir+name>

Usually, we add a couple of control members to it to adjust
the desired documentation scope (add/remove members, showing class
inheritance, showing members that currently don't have
docstrings, etc). That's why we're using:

    .. automodule:: lib.python.kdoc.enrich_formatter
       :members:
       :show-inheritance:
       :undoc-members:

(and similar) inside tools/kdoc*.rst.

autodoc allows filtering in/out members, file docstrings, etc.

It also allows documenting just some members or functions with
directives like:

    ..autofunction:
    ..automember:

Sphinx also has a helper script to generate .rst files with
documentation:

    $ sphinx-apidoc -o foobar tools/lib/python/

which can be helpful to discover what should be documented,
although changes are needed to use what it produces.

Mauro Carvalho Chehab (25):
  docs: custom.css: prevent li marker to override text
  docs: conf.py: don't use doctree with a different meaning
  docs: conf: don't rely on cwd to get documentation location
  docs: enable Sphinx autodoc extension to allow documenting python
  docs: custom.css: add CSS for python
  docs: kdoc: latex_fonts: Improve docstrings and comments
  docs: kdoc_files: Improve docstrings and comments
  docs: kdoc_item: Improve docstrings and comments
  docs: kdoc_parser: Improve docstrings and comments
  docs: kdoc_output: Improve docstrings and comments
  docs: kdoc_re: Improve docstrings and comments
  docs: kdoc: parse_data_structs: Improve docstrings and comments
  docs: kdoc: enrich_formatter: Improve docstrings and comments
  docs: kdoc: python_version: Improve docstrings and comments
  docs: add kernel-doc modules documentation
  docs: add kabi modules documentation
  docs: python: abi_parser: do some improvements at documentation
  docs: python: abi_regex: do some improvements at documentation
  docs: kabi: system_symbols: end docstring phrases with a dot
  docs: kabi: helpers: add helper for debug bits 7 and 8
  docs: kabi: helpers: add documentation for each "enum" value
  docs: add jobserver module documentation
  docs: jobserver: do some documentation improvements
  docs: add parse_features module documentation
  docs: parse_features: make documentation more consistent

 Documentation/conf.py                       |  23 +--
 Documentation/sphinx-static/custom.css      |  12 ++
 Documentation/tools/feat.rst                |  10 ++
 Documentation/tools/index.rst               |   1 +
 Documentation/tools/jobserver.rst           |  10 ++
 Documentation/tools/kabi.rst                |  13 ++
 Documentation/tools/kabi_helpers.rst        |  11 ++
 Documentation/tools/kabi_parser.rst         |  10 ++
 Documentation/tools/kabi_regex.rst          |  10 ++
 Documentation/tools/kabi_symbols.rst        |  10 ++
 Documentation/tools/kdoc.rst                |  12 ++
 Documentation/tools/kdoc_ancillary.rst      |  46 ++++++
 Documentation/tools/kdoc_output.rst         |  14 ++
 Documentation/tools/kdoc_parser.rst         |  29 ++++
 Documentation/tools/python.rst              |  13 ++
 tools/lib/python/abi/abi_parser.py          |  33 ++--
 tools/lib/python/abi/abi_regex.py           |  26 ++-
 tools/lib/python/abi/helpers.py             |  42 ++---
 tools/lib/python/abi/system_symbols.py      |  14 +-
 tools/lib/python/feat/parse_features.py     |  27 +++-
 tools/lib/python/jobserver.py               |  20 ++-
 tools/lib/python/kdoc/enrich_formatter.py   |  20 ++-
 tools/lib/python/kdoc/kdoc_files.py         |  23 +--
 tools/lib/python/kdoc/kdoc_item.py          |  18 +++
 tools/lib/python/kdoc/kdoc_output.py        |  60 ++++---
 tools/lib/python/kdoc/kdoc_parser.py        | 169 +++++++++++---------
 tools/lib/python/kdoc/kdoc_re.py            |  18 ++-
 tools/lib/python/kdoc/latex_fonts.py        |  95 ++++++-----
 tools/lib/python/kdoc/parse_data_structs.py |  62 ++++---
 tools/lib/python/kdoc/python_version.py     |  20 ++-
 30 files changed, 607 insertions(+), 264 deletions(-)
 create mode 100644 Documentation/tools/feat.rst
 create mode 100644 Documentation/tools/jobserver.rst
 create mode 100644 Documentation/tools/kabi.rst
 create mode 100644 Documentation/tools/kabi_helpers.rst
 create mode 100644 Documentation/tools/kabi_parser.rst
 create mode 100644 Documentation/tools/kabi_regex.rst
 create mode 100644 Documentation/tools/kabi_symbols.rst
 create mode 100644 Documentation/tools/kdoc.rst
 create mode 100644 Documentation/tools/kdoc_ancillary.rst
 create mode 100644 Documentation/tools/kdoc_output.rst
 create mode 100644 Documentation/tools/kdoc_parser.rst
 create mode 100644 Documentation/tools/python.rst

-- 
2.52.0
Re: [PATCH 00/25] Add tools/lib/python modules to Documentation/tools
Posted by Jonathan Corbet 2 weeks ago
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> Hi Jon,
>
> This is an extended version of:
>     https://lore.kernel.org/linux-doc/cover.1768488832.git.mchehab+huawei@kernel.org/
>
> It basically adds everything we currently have inside libs/tool/python
> to "tools" book inside documentation.

OK, I have applied the set, thanks.

jon
Re: [PATCH 00/25] Add tools/lib/python modules to Documentation/tools
Posted by Akira Yokosawa 2 weeks ago
On Fri, 23 Jan 2026 11:47:30 -0700, Jonathan Corbet wrote:
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
>> Hi Jon,
>>
>> This is an extended version of:
>>     https://lore.kernel.org/linux-doc/cover.1768488832.git.mchehab+huawei@kernel.org/
>>
>> It basically adds everything we currently have inside libs/tool/python
>> to "tools" book inside documentation.
> 
> OK, I have applied the set, thanks.

I've not have time to test this set, so just a quick report of a build error
related to this set after the fact.

I'm getting this error from "make SPHINXDIRS=tools pdfdocs":

---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
------------
Running 'xelatex --no-pdf  -no-pdf -interaction=batchmode -no-shell-escape -recorder  "/<...>/Documentation/output/tools/latex/tools.tex"'
------------
This is XeTeX, Version 3.141592653-2.6-0.999997 (TeX Live 2025) (preloaded format=xelatex)
entering extended mode
Latexmk: Getting log file 'tools.log'
Latexmk: Examining 'tools.fls'
Latexmk: Examining 'tools.log'
Latexmk: References changed.
Latexmk: Missing input file 'tools.toc' message in .log file:
  No file tools.toc.
Latexmk: Missing input file 'tools.ind' message in .log file:
  No file tools.ind.
Latexmk: References changed.
Latexmk: References changed.
Latexmk: Log file says output to 'tools.xdv'
Latexmk: Index file 'tools.idx' was written
Latexmk: Using bibtex to make bibliography file(s).
Latexmk: Summary of warnings from last run of *latex:
  Latex failed to resolve 15 reference(s)
Latexmk: ====Undefined refs and citations with line #s in .tex file:
  Reference `kabi_parser:module-lib.python.abi.abi_parser' on page 75 undefined on input line 7092
  Reference `kabi_regex:module-lib.python.abi.abi_regex' on page 75 undefined on input line 7093
  Reference `kabi_helpers:module-lib.python.abi.helpers' on page 75 undefined on input line 7094
  Reference `kabi_symbols:module-lib.python.abi.system_symbols' on page 75 undefined on input line 7095
  Reference `feat:module-lib.python.feat.parse_features' on page 75 undefined on input line 7096
  Reference `jobserver:module-lib.python.jobserver' on page 75 undefined on input line 7097
  Reference `kdoc_ancillary:module-lib.python.kdoc.enrich_formatter' on page 75 undefined on input line 7098
 And 8 more --- see log file 'tools.log'
Latexmk: Errors, so I did not complete making targets
Collected error summary (may duplicate other messages):
  xelatex: Command for 'xelatex' gave return code 1
      Refer to 'tools.log' and/or above output for details

Latexmk: Sometimes, the -f option can be used to get latexmk
  to try to force complete processing.
  But normally, you will need to correct the file(s) that caused the
  error, and then rerun latexmk.
  In some cases, it is best to clean out generated files before rerunning
  latexmk after you've corrected the files.
Error: Can't build 1 PDF file(s): tools/pdf/tools.pdf
make[2]: *** [Documentation/Makefile:58: pdfdocs] Error 1
make[1]: *** [/<...>/Makefile:1842: pdfdocs] Error 2
make: *** [Makefile:248: __sub-make] Error 2
---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----

Excerpt from tools.log:

---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
! Text line contains an invalid character.
l.5865 if r’^^A
               ’ is used, it works just like re: it places there the
A funny symbol that I can't read has just been input.
Continue, and I'll forget that it ever happened.
---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----

, and from tools.tex:

--------
\sphinxAtStartPar
It matches a regex that it is followed by a delimiter,
replacing occurrences only if all delimiters are paired.

\sphinxAtStartPar
if r’’ is used, it works just like re: it places there the  <-- 5865
matched paired data with the delimiter stripped.
--------

Mauro, can you please have a look?

Thanks, Akira

> 
> jon

Re: [PATCH 00/25] Add tools/lib/python modules to Documentation/tools
Posted by Mauro Carvalho Chehab 2 weeks ago
On Sat, 24 Jan 2026 09:31:47 +0900
Akira Yokosawa <akiyks@gmail.com> wrote:

> On Fri, 23 Jan 2026 11:47:30 -0700, Jonathan Corbet wrote:
> > Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> >   
> >> Hi Jon,
> >>
> >> This is an extended version of:
> >>     https://lore.kernel.org/linux-doc/cover.1768488832.git.mchehab+huawei@kernel.org/
> >>
> >> It basically adds everything we currently have inside libs/tool/python
> >> to "tools" book inside documentation.  
> > 
> > OK, I have applied the set, thanks.  
> 
> I've not have time to test this set, so just a quick report of a build error
> related to this set after the fact.
> 
> I'm getting this error from "make SPHINXDIRS=tools pdfdocs":

Thanks for reporting it!

I ended forgetting to test PDF build on it. Funny enough, even for
html, this can be problematic. Fixing it is really trivial though: 
just add a single character to the beginning of a docstring:

	--- a/tools/lib/python/kdoc/kdoc_re.py
	+++ b/tools/lib/python/kdoc/kdoc_re.py
	@@ -228,7 +228,7 @@ class NestedMatch:
        	     yield line[t[0]:t[2]]
 
	     def sub(self, regex, sub, line, count=0):
	-        """
	+        r"""
	         This is similar to re.sub:
	 
	         It matches a regex that it is followed by a delimiter,

For it to not consider "\1" as a special character.

Patch sent:

	1a60aea3155a2e7c5e7bb004952739652407fb8c


Thanks,
Mauro