Hi Jon,

As promised, this series improves the Sphinx kerneldoc extension making it using the
kernel-doc.py classes directly if available.

The script still supports excecuting kernel-doc via shell, and, in verbose mode, it will
show the command line arguments to run kernel-doc manually, even when the Python
classes are used. The idea is that the command line verbose will help to eventually
debug issues if one needs to run kernel-doc.py manually. 

On other words, after this series, if one does:

	make htmldocs KERNELDOC=scripts/kernel-doc.py

Or, simply (as internally KERNELDOC is set to scripts/kernel-doc.py):

	make htmldocs

It will use the Python classes instead of running a subprocess.

If one uses, instead:

	make htmldocs KERNELDOC=scripts/kernel-doc
or:
	make htmldocs KERNELDOC=scripts/kernel-doc.pl

As the file extension doesn't end with .py, it will excecute the Python or the Perl
version of kernel-doc via a subprocess.

On this version, I opted to re-create the Python objects for every single kernel-doc
line, so no caches from past runs are used. I'm working on a version that will cache
results, but it is currently causing some regressions. So, let's do the changes
step-by-step, applying first this improvement patch series.

PS.: the first patches on this series are addressing some some bugs and one
improvement that I noticed while debugging the patch changing kerneldoc
Sphinx extension.

Mauro Carvalho Chehab (4):
  scripts/lib/kdoc/kdoc_files.py: don't try to join None
  scripts/lib/kdoc/kdoc_parser.py: move states to a separate class
  scripts:kdoc_files.py: use glob for export_file seek
  docs: sphinx: kerneldoc: Use python class if available

 Documentation/sphinx/kerneldoc.py | 138 ++++++++++++++++++++++++++++--
 scripts/lib/kdoc/kdoc_files.py    |  11 ++-
 scripts/lib/kdoc/kdoc_parser.py   | 119 ++++++++++++++------------
 3 files changed, 200 insertions(+), 68 deletions(-)

-- 
2.49.0

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> Hi Jon,
>
> As promised, this series improves the Sphinx kerneldoc extension making it using the
> kernel-doc.py classes directly if available.
>
> The script still supports excecuting kernel-doc via shell, and, in verbose mode, it will
> show the command line arguments to run kernel-doc manually, even when the Python
> classes are used. The idea is that the command line verbose will help to eventually
> debug issues if one needs to run kernel-doc.py manually. 
>
> On other words, after this series, if one does:
>
> 	make htmldocs KERNELDOC=scripts/kernel-doc.py
>
> Or, simply (as internally KERNELDOC is set to scripts/kernel-doc.py):
>
> 	make htmldocs
>
> It will use the Python classes instead of running a subprocess.
>
> If one uses, instead:
>
> 	make htmldocs KERNELDOC=scripts/kernel-doc
> or:
> 	make htmldocs KERNELDOC=scripts/kernel-doc.pl
>
> As the file extension doesn't end with .py, it will excecute the Python or the Perl
> version of kernel-doc via a subprocess.
>
> On this version, I opted to re-create the Python objects for every single kernel-doc
> line, so no caches from past runs are used. I'm working on a version that will cache
> results, but it is currently causing some regressions. So, let's do the changes
> step-by-step, applying first this improvement patch series.
>
> PS.: the first patches on this series are addressing some some bugs and one
> improvement that I noticed while debugging the patch changing kerneldoc
> Sphinx extension.
>
> Mauro Carvalho Chehab (4):
>   scripts/lib/kdoc/kdoc_files.py: don't try to join None
>   scripts/lib/kdoc/kdoc_parser.py: move states to a separate class
>   scripts:kdoc_files.py: use glob for export_file seek
>   docs: sphinx: kerneldoc: Use python class if available
>
>  Documentation/sphinx/kerneldoc.py | 138 ++++++++++++++++++++++++++++--
>  scripts/lib/kdoc/kdoc_files.py    |  11 ++-
>  scripts/lib/kdoc/kdoc_parser.py   | 119 ++++++++++++++------------
>  3 files changed, 200 insertions(+), 68 deletions(-)

I've applied the series.

I do note that the default "make htmldocs" build time is reliably slower
than with KERNELDOC=scripts/kernel-doc, I'd be curious to understand
why.

        External kdoc:  170s
        w/classes:	186s

jon

Em Mon, 21 Apr 2025 11:42:09 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > Hi Jon,
> >
> > As promised, this series improves the Sphinx kerneldoc extension making it using the
> > kernel-doc.py classes directly if available.
> >
> > The script still supports excecuting kernel-doc via shell, and, in verbose mode, it will
> > show the command line arguments to run kernel-doc manually, even when the Python
> > classes are used. The idea is that the command line verbose will help to eventually
> > debug issues if one needs to run kernel-doc.py manually. 
> >
> > On other words, after this series, if one does:
> >
> > 	make htmldocs KERNELDOC=scripts/kernel-doc.py
> >
> > Or, simply (as internally KERNELDOC is set to scripts/kernel-doc.py):
> >
> > 	make htmldocs
> >
> > It will use the Python classes instead of running a subprocess.
> >
> > If one uses, instead:
> >
> > 	make htmldocs KERNELDOC=scripts/kernel-doc
> > or:
> > 	make htmldocs KERNELDOC=scripts/kernel-doc.pl
> >
> > As the file extension doesn't end with .py, it will excecute the Python or the Perl
> > version of kernel-doc via a subprocess.
> >
> > On this version, I opted to re-create the Python objects for every single kernel-doc
> > line, so no caches from past runs are used. I'm working on a version that will cache
> > results, but it is currently causing some regressions. So, let's do the changes
> > step-by-step, applying first this improvement patch series.
> >
> > PS.: the first patches on this series are addressing some some bugs and one
> > improvement that I noticed while debugging the patch changing kerneldoc
> > Sphinx extension.
> >
> > Mauro Carvalho Chehab (4):
> >   scripts/lib/kdoc/kdoc_files.py: don't try to join None
> >   scripts/lib/kdoc/kdoc_parser.py: move states to a separate class
> >   scripts:kdoc_files.py: use glob for export_file seek
> >   docs: sphinx: kerneldoc: Use python class if available
> >
> >  Documentation/sphinx/kerneldoc.py | 138 ++++++++++++++++++++++++++++--
> >  scripts/lib/kdoc/kdoc_files.py    |  11 ++-
> >  scripts/lib/kdoc/kdoc_parser.py   | 119 ++++++++++++++------------
> >  3 files changed, 200 insertions(+), 68 deletions(-)  
> 
> I've applied the series.
> 
> I do note that the default "make htmldocs" build time is reliably slower
> than with KERNELDOC=scripts/kernel-doc, I'd be curious to understand
> why.
> 
>         External kdoc:  170s
>         w/classes:	186s

I noticed that. My guess is that this has to do with Python's big lock
(GIL). When it uses processes, there won't be any internal locks, as
kernel-doc will run independently. There is an effort for Python to get
rid of GIL, but it seems GIL-less threads won't be default on version 3.14.

-

That's said, please notice that I intend to work on some patches that
will optimize that. Right now, on some places, the same file is processed
multiple times. By using the classes, we can cache the processing results
the first time and then re-use them every time a kernel-doc tag for the
same file is found within Documentation.

This is easier said than done. I did one attempt to do that, but it ended
causing troubles. So, I'm working on a new version of it.

Regards,
Mauro