[PATCH v2 00/12] Thrash up the parser/output interface

Jonathan Corbet posted 12 patches 2 months, 4 weeks ago
scripts/lib/kdoc/kdoc_files.py  |   4 +-
scripts/lib/kdoc/kdoc_item.py   |  41 ++++++++
scripts/lib/kdoc/kdoc_output.py | 172 ++++++++++++--------------------
scripts/lib/kdoc/kdoc_parser.py | 171 +++++++------------------------
4 files changed, 144 insertions(+), 244 deletions(-)
create mode 100644 scripts/lib/kdoc/kdoc_item.py
[PATCH v2 00/12] Thrash up the parser/output interface
Posted by Jonathan Corbet 2 months, 4 weeks ago
The kerneldoc parsing phase gathers all of the information about the
declarations of interest, then passes it through to the output phase as a
dict that is an unstructured blob of information; this organization has its
origins in the Perl version of the program.  It results in an interface
that is difficult to reason about, dozen-parameter function calls, and
other ills.

Introduce a new class (KdocItem) to carry this information between the
parser and the output modules, and, step by step, modify the system to use
this class in a more structured way.  This could be taken further by
creating a subclass of KdocItem for each declaration type (function,
struct, ...), but that is probably more structure than we need.

The result is (I hope) clearer code, the removal of a bunch of boilerplate,
and no changes to the generated output.

Changes since v1:
- Coding-style tweaks requested by Mauro
- Drop the reworking of output-text accumulation for now
- Add a warning for prehistoric Python versions

Jonathan Corbet (12):
  docs: kdoc; Add a rudimentary class to represent output items
  docs: kdoc: simplify the output-item passing
  docs: kdoc: drop "sectionlist"
  docs: kdoc: Centralize handling of the item section list
  docs: kdoc: remove the "struct_actual" machinery
  docs: kdoc: use self.entry.parameterlist directly in check_sections()
  docs: kdoc: Coalesce parameter-list handling
  docs: kdoc: Regularize the use of the declaration name
  docs: kdoc: straighten up dump_declaration()
  docs: kdoc: directly access the always-there KdocItem fields
  docs: kdoc: clean up check_sections()
  docs: kdoc: emit a warning for ancient versions of Python

 scripts/lib/kdoc/kdoc_files.py  |   4 +-
 scripts/lib/kdoc/kdoc_item.py   |  41 ++++++++
 scripts/lib/kdoc/kdoc_output.py | 172 ++++++++++++--------------------
 scripts/lib/kdoc/kdoc_parser.py | 171 +++++++------------------------
 4 files changed, 144 insertions(+), 244 deletions(-)
 create mode 100644 scripts/lib/kdoc/kdoc_item.py

-- 
2.49.0
Re: [PATCH v2 00/12] Thrash up the parser/output interface
Posted by Akira Yokosawa 2 months, 4 weeks ago
On Thu, 10 Jul 2025 17:31:30 -0600, Jonathan Corbet wrote:
[...]

> Changes since v1:
> - Coding-style tweaks requested by Mauro
> - Drop the reworking of output-text accumulation for now
> - Add a warning for prehistoric Python versions

Serious review of python code is beyond my background, but I did a test
on this against opensuse/leap:15.6's python3-Sphinx_4_2_0, which comes with
python 3.6.15.

Running "./scripts/kernel-doc.py -none include/linux/rcupdate.h" emits this:

------------------------------------------------------------------------
Traceback (most recent call last):
  File "./scripts/kernel-doc.py", line 315, in <module>
    main()
  File "./scripts/kernel-doc.py", line 286, in main
    kfiles.parse(args.files, export_file=args.export_file)
  File "/linux/scripts/lib/kdoc/kdoc_files.py", line 222, in parse
    self.parse_file(fname)
  File "/linux/scripts/lib/kdoc/kdoc_files.py", line 119, in parse_file
    doc = KernelDoc(self.config, fname)
  File "/linux/scripts/lib/kdoc/kdoc_parser.py", line 247, in __init__
    self.emit_message(0,
AttributeError: 'KernelDoc' object has no attribute 'emit_message'
------------------------------------------------------------------------

This error appeared in 12/12.  No errors with python3 >=3.9.

I'm not sure but asking compatibility with python <3.9 increases
maintainers/testers' burden.  Obsoleting <3.9 all together would
make everyone's life easier, wouldn't it?

    Thanks, Akira
Re: [PATCH v2 00/12] Thrash up the parser/output interface
Posted by Jonathan Corbet 2 months, 3 weeks ago
Akira Yokosawa <akiyks@gmail.com> writes:

> On Thu, 10 Jul 2025 17:31:30 -0600, Jonathan Corbet wrote:
> [...]
>
>> Changes since v1:
>> - Coding-style tweaks requested by Mauro
>> - Drop the reworking of output-text accumulation for now
>> - Add a warning for prehistoric Python versions
>
> Serious review of python code is beyond my background, but I did a test
> on this against opensuse/leap:15.6's python3-Sphinx_4_2_0, which comes with
> python 3.6.15.
>
> Running "./scripts/kernel-doc.py -none include/linux/rcupdate.h" emits this:
>
> ------------------------------------------------------------------------
> Traceback (most recent call last):
>   File "./scripts/kernel-doc.py", line 315, in <module>
>     main()
>   File "./scripts/kernel-doc.py", line 286, in main
>     kfiles.parse(args.files, export_file=args.export_file)
>   File "/linux/scripts/lib/kdoc/kdoc_files.py", line 222, in parse
>     self.parse_file(fname)
>   File "/linux/scripts/lib/kdoc/kdoc_files.py", line 119, in parse_file
>     doc = KernelDoc(self.config, fname)
>   File "/linux/scripts/lib/kdoc/kdoc_parser.py", line 247, in __init__
>     self.emit_message(0,
> AttributeError: 'KernelDoc' object has no attribute 'emit_message'
> ------------------------------------------------------------------------
>
> This error appeared in 12/12.  No errors with python3 >=3.9.

Well, it does make it clear that things won't work properly with older
Python... :)

Seriously, though, that's embarrassing; I was clearly in too much of a
hurry when I tossed that last patch in.  Will fix.

> I'm not sure but asking compatibility with python <3.9 increases
> maintainers/testers' burden.  Obsoleting <3.9 all together would
> make everyone's life easier, wouldn't it?

That is pretty much what we have done, this was just intended to let
people know that they won't get the results they expect.

Thanks,

jon
Re: [PATCH v2 00/12] Thrash up the parser/output interface
Posted by Mauro Carvalho Chehab 2 months, 3 weeks ago
Em Fri, 11 Jul 2025 13:29:00 +0900
Akira Yokosawa <akiyks@gmail.com> escreveu:

> On Thu, 10 Jul 2025 17:31:30 -0600, Jonathan Corbet wrote:
> [...]
> 
> > Changes since v1:
> > - Coding-style tweaks requested by Mauro
> > - Drop the reworking of output-text accumulation for now
> > - Add a warning for prehistoric Python versions  
> 
> Serious review of python code is beyond my background, but I did a test
> on this against opensuse/leap:15.6's python3-Sphinx_4_2_0, which comes with
> python 3.6.15.
> 
> Running "./scripts/kernel-doc.py -none include/linux/rcupdate.h" emits this:
> 
> ------------------------------------------------------------------------
> Traceback (most recent call last):
>   File "./scripts/kernel-doc.py", line 315, in <module>
>     main()
>   File "./scripts/kernel-doc.py", line 286, in main
>     kfiles.parse(args.files, export_file=args.export_file)
>   File "/linux/scripts/lib/kdoc/kdoc_files.py", line 222, in parse
>     self.parse_file(fname)
>   File "/linux/scripts/lib/kdoc/kdoc_files.py", line 119, in parse_file
>     doc = KernelDoc(self.config, fname)
>   File "/linux/scripts/lib/kdoc/kdoc_parser.py", line 247, in __init__
>     self.emit_message(0,
> AttributeError: 'KernelDoc' object has no attribute 'emit_message'
> ------------------------------------------------------------------------
> 
> This error appeared in 12/12.  No errors with python3 >=3.9.

This is actually a bug. See:

	+        #
	+        # We need Python 3.7 for its "dicts remember the insertion
	+        # order" guarantee
	+        #
	+        if sys.version_info.major == 3 and sys.version_info.minor < 7:
	+            self.emit_message(0,
	+                              'Python 3.7 or later is required for correct results')
	+
	     def emit_msg(self, ln, msg, warning=True):
	         """Emit a message"""

The answer is just below the modified code: the function name is actually:
	self.emit_msg()

> I'm not sure but asking compatibility with python <3.9 increases
> maintainers/testers' burden.  Obsoleting <3.9 all together would
> make everyone's life easier, wouldn't it?

I'd say that the best is to have:

scripts/sphinx-pre-install:

- be compatible with Python 3.6, as it is needed to detect and
  request Python upgrades where needed.

- For the doc build, based on my tests with the pre-install tool,
  all distros on my testlist have at least Python >= 3.9 as optional
  packages.

Now, kernel-doc is a special case, as it is called during Kernel
builds, with "-none". In the specific case of this patchset, running
Python 3.6 would randomize the order of struct and function arguments.
Not a problem when "-none" is used. So, compilation should not
break.

Shifting kernel-doc minimal version to Python >= 3.9, will
require an extra logic at kerneldoc to abort early if "-none"
is used with too old Python, as otherwise it will break kernel
builds for RHEL8, OpenSUSE Leap, OpenMandriva Lx 4.3 and to 
other distros (those three comes with 3.6).

Jon,

I sent a two-patches series addressing such issues with kernel-doc
exec file.

I would keep patch 12/12, fixing it from:

	 self.emit_message -> self.emit_msg

as the warning is emitted when using kdoc classes, which is now the
default.

-

Btw, if you use Fedora, you can easily install python 3.6 there via 
dnf:

	$ dnf search python 3.6
	Updating and loading repositories:
	Repositories loaded.
	Matched fields: name, summary
	 python3.6.i686: Version 3.6 of the Python interpreter
	 python3.6.x86_64: Version 3.6 of the Python interpreter

	(Fedora 42 also have all python versions from 3.9 to 3.14 beta also
	 via dnf)

Regards,
Mauro