Hi Jon,

This series contain patches I made while working at the parser-yaml.
They aren't directly related to it. Instead, they address some issues
at the build system and provide test tools for building docs.
 
Most of the series is related to the new  test_doc_build.py.
This is a tool I wrote from scratch to help identifying regressions
for changes affecting the build system.

As described on its help page, it allows creating Python virtual
environments for different Sphinx versions that are supported 
by the Linux Kernel build system.

Besides creating the virtual environment, it can also test building
the documentation using "make htmldocs" (and/or other doc targets).

If called without "--versions" argument, it covers the versions shipped
on major distros, plus the lowest supported version.

A typical usage is to run:

$ time ./scripts/test_doc_build.py -m -a "SPHINXOPTS=-j8" -l distros.log
...
Summary:
        Sphinx 3.4.3 elapsed time: 00:07:22
        Sphinx 5.3.0 elapsed time: 00:07:30
        Sphinx 6.1.1 elapsed time: 00:23:43
        Sphinx 7.2.1 elapsed time: 00:07:34
        Sphinx 7.2.6 elapsed time: 00:07:43
        Sphinx 7.3.0 elapsed time: 00:07:54
        Sphinx 7.4.7 elapsed time: 00:04:04
        Sphinx 8.1.3 elapsed time: 00:03:14
        Sphinx 8.2.3 elapsed time: 00:03:12

real    72m30.037s
user    116m45.360s
sys     7m44.09

This should check the main backward-compatibility issues.

A more comprehensive test can be done with:

$ time ./scripts/test_doc_build.py -b -a "SPHINXOPTS=-j8" -l full.log --full

Summary:
        Sphinx 3.4.3 elapsed time: 00:07:15
        Sphinx 3.5.0 elapsed time: 00:07:05
        Sphinx 4.0.0 elapsed time: 00:07:10
        Sphinx 4.1.0 elapsed time: 00:07:20
        Sphinx 4.3.0 elapsed time: 00:07:22
        Sphinx 4.4.0 elapsed time: 00:07:24
        Sphinx 4.5.0 elapsed time: 00:07:13
        Sphinx 5.0.0 elapsed time: 00:07:34
        Sphinx 5.1.0 elapsed time: 00:07:32
        Sphinx 5.2.0 elapsed time: 00:07:29
        Sphinx 5.3.0 elapsed time: 00:07:35
        Sphinx 6.0.0 elapsed time: 00:22:34
        Sphinx 6.1.0 elapsed time: 00:23:57
        Sphinx 6.1.1 elapsed time: 00:23:41
        Sphinx 6.2.0 elapsed time: 00:07:26
        Sphinx 7.0.0 elapsed time: 00:07:29
        Sphinx 7.1.0 elapsed time: 00:07:22
        Sphinx 7.2.0 elapsed time: 00:07:24
        Sphinx 7.2.1 elapsed time: 00:07:31
        Sphinx 7.2.6 elapsed time: 00:07:47
        Sphinx 7.3.0 elapsed time: 00:07:44
        Sphinx 7.4.0 elapsed time: 00:04:16
        Sphinx 7.4.7 elapsed time: 00:04:12
        Sphinx 8.0.0 elapsed time: 00:03:11
        Sphinx 8.1.0 elapsed time: 00:03:17
        Sphinx 8.1.3 elapsed time: 00:03:17
        Sphinx 8.2.0 elapsed time: 00:03:12
        Sphinx 8.2.3 elapsed time: 00:03:14

real    229m13.749s
user    377m26.666s
sys     24m32.544s

Some notes:

1) on my machine, "-j8" is usually faster than "-jauto";
2) 6.x.x is problematic: Sphinx uses a lot of memory, being a friend
   of systemd-oomd killer, specially with -jauto. -j8 is a little better,
   but it still caused crashes on other apps, probably due to memory
   consumption.

Regards,
Mauro

- v2: Added patches 7 to 15.

Mauro Carvalho Chehab (15):
  docs: conf.py: properly handle include and exclude patterns
  docs: Makefile: disable check rules on make cleandocs
  scripts: scripts/test_doc_build.py: add script to test doc build
  scripts:y: make capture assynchronous
  scripts: test_doc_build.py: better control its output
  scripts: test_doc_build.py: better adjust to python version
  scripts: test_doc_build.py: improve dependency list
  scripts: test_doc_build.py: improve cmd.log logic
  scripts: test_doc_build.py: make the script smarter
  scripts: sphinx-pre-install: properly handle SPHINXBUILD
  scripts: sphinx-pre-install: fix release detection for Fedora
  scripts: test_doc_build.py: regroup and rename arguments
  docs: sphinx: add a file with the requirements for lowest version
  docs: conf.py: several coding style fixes
  docs: conf.py: Check Sphinx and docutils version

 Documentation/Makefile                    |   2 +
 Documentation/conf.py                     | 463 +++++++++++--------
 Documentation/doc-guide/sphinx.rst        |  23 +
 Documentation/sphinx/min_requirements.txt |  10 +
 scripts/sphinx-pre-install                |   6 +-
 scripts/test_doc_build.py                 | 513 ++++++++++++++++++++++
 6 files changed, 843 insertions(+), 174 deletions(-)
 create mode 100644 Documentation/sphinx/min_requirements.txt
 create mode 100755 scripts/test_doc_build.py

-- 
2.49.0

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> Hi Jon,
>
> This series contain patches I made while working at the parser-yaml.
> They aren't directly related to it. Instead, they address some issues
> at the build system and provide test tools for building docs.

So as you saw I'd applied the previous set - but I've undone that.
Something in the first patch (the conf.py changes) breaks the dependency
detection so that every build turns into a full build.  Just run
"make htmldocs" twice in a row to see what happens.  That somehow needs
to be fixed...

(this is with Sphinx 8.1.3)

Thanks,

jon

Em Sat, 21 Jun 2025 14:09:39 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > Hi Jon,
> >
> > This series contain patches I made while working at the parser-yaml.
> > They aren't directly related to it. Instead, they address some issues
> > at the build system and provide test tools for building docs.  
> 
> So as you saw I'd applied the previous set - but I've undone that.
> Something in the first patch (the conf.py changes) breaks the dependency
> detection so that every build turns into a full build.  Just run
> "make htmldocs" twice in a row to see what happens.  That somehow needs
> to be fixed...

Tricky. When I originally wrote it, on the top of 8.x, there is no need
to add a connect event. For older versions, this is a need.

I ended adding a 'builder-inited', but this is too late. Changing the
event to 'config-inited' address it.

I'll fold the patch below at patch 01/15.

Regards,
Mauro

diff --git a/Documentation/conf.py b/Documentation/conf.py
index 4ba4ee45e599..91ce2b1c33cc 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -41,7 +41,7 @@ dyn_exclude_patterns = ['output']
 # Properly handle include/exclude patterns
 # ----------------------------------------
 
-def update_patterns(app):
+def update_patterns(app, config):
 
     """
     On Sphinx, all directories are relative to what it is passed as
@@ -65,7 +65,7 @@ def update_patterns(app):
             if rel_path.startswith("../"):
                 continue
 
-            app.config.include_patterns.append(rel_path)
+            config.include_patterns.append(rel_path)
 
     # setup exclude_patterns dynamically
     for p in dyn_exclude_patterns:
@@ -75,7 +75,7 @@ def update_patterns(app):
         if rel_path.startswith("../"):
             continue
 
-        app.config.exclude_patterns.append(rel_path)
+        config.exclude_patterns.append(rel_path)
 
 # helper
 # ------
@@ -574,4 +574,4 @@ kerneldoc_srctree = '..'
 loadConfig(globals())
 
 def setup(app):
-	app.connect('builder-inited', update_patterns)
+    app.connect('config-inited', update_patterns)


Thanks,
Mauro