When one does:
	make SPHINXDIRS="foo" htmldocs

All patterns would be relative to Documentation/foo, which
causes the include/exclude patterns like:

	include_patterns = [
		...
		f'foo/*.{ext}',
	]

to break. This is not what it is expected. Address it by
adding a logic to dynamically adjust the pattern when
SPHINXDIRS is used.

That allows adding parsers for other file types.

Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
 Documentation/conf.py | 52 +++++++++++++++++++++++++++++++++++++++----
 1 file changed, 48 insertions(+), 4 deletions(-)

diff --git a/Documentation/conf.py b/Documentation/conf.py
index 12de52a2b17e..e887c1b786a4 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -17,6 +17,54 @@ import os
 import sphinx
 import shutil
 
+# Location of Documentation/ directory
+doctree = os.path.abspath('.')
+
+# List of patterns that don't contain directory names, in glob format.
+include_patterns = ['**.rst']
+exclude_patterns = []
+
+# List of patterns that contain directory names in glob format.
+dyn_include_patterns = []
+dyn_exclude_patterns = ['output']
+
+# Properly handle include/exclude patterns
+# ----------------------------------------
+
+def setup(app):
+    """
+    On Sphinx, all directories are relative to what it is passed as
+    SOURCEDIR parameter for sphinx-build. Due to that, all patterns
+    that have directory names on it need to be dynamically set, after
+    converting them to a relative patch.
+
+    As Sphinx doesn't include any patterns outside SOURCEDIR, we should
+    exclude relative patterns that start with "../".
+    """
+
+    sourcedir = app.srcdir  # full path to the source directory
+    builddir = os.environ.get("BUILDDIR")
+
+    # setup include_patterns dynamically
+    for p in dyn_include_patterns:
+        full = os.path.join(doctree, p)
+
+        rel_path = os.path.relpath(full, start = app.srcdir)
+        if rel_path.startswith("../"):
+            continue
+
+        app.config.include_patterns.append(rel_path)
+
+    # setup exclude_patterns dynamically
+    for p in dyn_exclude_patterns:
+        full = os.path.join(doctree, p)
+
+        rel_path = os.path.relpath(full, start = app.srcdir)
+        if rel_path.startswith("../"):
+            continue
+
+        app.config.exclude_patterns.append(rel_path)
+
 # helper
 # ------
 
@@ -219,10 +267,6 @@ language = 'en'
 # Else, today_fmt is used as the format for a strftime call.
 #today_fmt = '%B %d, %Y'
 
-# List of patterns, relative to source directory, that match files and
-# directories to ignore when looking for source files.
-exclude_patterns = ['output']
-
 # The reST default role (used for this markup: `text`) to use for all
 # documents.
 #default_role = None
-- 
2.49.0

Hi Mauro,

A comment on compatibility with earlier Sphinx.

On Tue, 17 Jun 2025 10:01:58 +0200, Mauro Carvalho Chehab wrote:
> When one does:
> 	make SPHINXDIRS="foo" htmldocs
> 
> All patterns would be relative to Documentation/foo, which
> causes the include/exclude patterns like:
> 
> 	include_patterns = [
> 		...
> 		f'foo/*.{ext}',
> 	]
> 
> to break. This is not what it is expected. Address it by
> adding a logic to dynamically adjust the pattern when
> SPHINXDIRS is used.
> 
> That allows adding parsers for other file types.
> 
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> ---
>  Documentation/conf.py | 52 +++++++++++++++++++++++++++++++++++++++----
>  1 file changed, 48 insertions(+), 4 deletions(-)
> 
> diff --git a/Documentation/conf.py b/Documentation/conf.py
> index 12de52a2b17e..e887c1b786a4 100644
> --- a/Documentation/conf.py
> +++ b/Documentation/conf.py
> @@ -17,6 +17,54 @@ import os
>  import sphinx
>  import shutil
>  
> +# Location of Documentation/ directory
> +doctree = os.path.abspath('.')
> +
> +# List of patterns that don't contain directory names, in glob format.
> +include_patterns = ['**.rst']
> +exclude_patterns = []
> +

Where "exclude_patterns" has been with us ever since Sphinx 1.0,
"include_patterns" was added fairly recently in Sphinx 5.1 [1].

[1]: https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-include_patterns

So, this breaks earlier Sphinx versions.

Also, after applying all of v5 on top of docs-next, I see these new
warnings with Sphinx 7.2.6 (of Ubuntu 24.04):

/<srcdir>/Documentation/output/ca.h.rst: WARNING: document isn't included in any toctree
/<srcdir>/Documentation/output/cec.h.rst: WARNING: document isn't included in any toctree
/<srcdir>/Documentation/output/dmx.h.rst: WARNING: document isn't included in any toctree
/<srcdir>/Documentation/output/frontend.h.rst: WARNING: document isn't included in any toctree
/<srcdir>/Documentation/output/lirc.h.rst: WARNING: document isn't included in any toctree
/<srcdir>/Documentation/output/media.h.rst: WARNING: document isn't included in any toctree
/<srcdir>/Documentation/output/net.h.rst: WARNING: document isn't included in any toctree
/<srcdir>/Documentation/output/videodev2.h.rst: WARNING: document isn't included in any toctree

Sphinx 7.3.7 and later are free of them.  I have no idea which change in
Sphinx 7.3 got rid of them.

Now that the parallel build performance regression has be resolved in
Sphinx 7.4, I don't think there is much demand for keeping Sphinx versions
compatible.
These build errors and extra warnings would encourage people to upgrade
there Sphinx.  So I'm not going to nack this.

Of course, getting rid of above warnings with < Sphinx 7.3 would be ideal.

        Thanks, Akira

> +# List of patterns that contain directory names in glob format.
> +dyn_include_patterns = []
> +dyn_exclude_patterns = ['output']
> +
[...]

Hi Akira,

Em Wed, 18 Jun 2025 11:42:14 +0900
Akira Yokosawa <akiyks@gmail.com> escreveu:

> Hi Mauro,
> 
> A comment on compatibility with earlier Sphinx.
> 
> On Tue, 17 Jun 2025 10:01:58 +0200, Mauro Carvalho Chehab wrote:
> > When one does:
> > 	make SPHINXDIRS="foo" htmldocs
> > 
> > All patterns would be relative to Documentation/foo, which
> > causes the include/exclude patterns like:
> > 
> > 	include_patterns = [
> > 		...
> > 		f'foo/*.{ext}',
> > 	]
> > 
> > to break. This is not what it is expected. Address it by
> > adding a logic to dynamically adjust the pattern when
> > SPHINXDIRS is used.
> > 
> > That allows adding parsers for other file types.
> > 
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> > ---
> >  Documentation/conf.py | 52 +++++++++++++++++++++++++++++++++++++++----
> >  1 file changed, 48 insertions(+), 4 deletions(-)
> > 
> > diff --git a/Documentation/conf.py b/Documentation/conf.py
> > index 12de52a2b17e..e887c1b786a4 100644
> > --- a/Documentation/conf.py
> > +++ b/Documentation/conf.py
> > @@ -17,6 +17,54 @@ import os
> >  import sphinx
> >  import shutil
> >  
> > +# Location of Documentation/ directory
> > +doctree = os.path.abspath('.')
> > +
> > +# List of patterns that don't contain directory names, in glob format.
> > +include_patterns = ['**.rst']
> > +exclude_patterns = []
> > +  
> 
> Where "exclude_patterns" has been with us ever since Sphinx 1.0,
> "include_patterns" was added fairly recently in Sphinx 5.1 [1].
> 
> [1]: https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-include_patterns
> 
> So, this breaks earlier Sphinx versions.

Heh, testing against old versions is harder with python 3.13 (Fedora
42 default), as one library used by older Sphinx versions were dropped.

I found a way to make it backward compatible up to 3.4.3, with a
backward-compatible logic at conf.py. I'll send the new version in a few.

> Also, after applying all of v5 on top of docs-next, I see these new
> warnings with Sphinx 7.2.6 (of Ubuntu 24.04):
> 
> /<srcdir>/Documentation/output/ca.h.rst: WARNING: document isn't included in any toctree
> /<srcdir>/Documentation/output/cec.h.rst: WARNING: document isn't included in any toctree
> /<srcdir>/Documentation/output/dmx.h.rst: WARNING: document isn't included in any toctree
> /<srcdir>/Documentation/output/frontend.h.rst: WARNING: document isn't included in any toctree
> /<srcdir>/Documentation/output/lirc.h.rst: WARNING: document isn't included in any toctree
> /<srcdir>/Documentation/output/media.h.rst: WARNING: document isn't included in any toctree
> /<srcdir>/Documentation/output/net.h.rst: WARNING: document isn't included in any toctree
> /<srcdir>/Documentation/output/videodev2.h.rst: WARNING: document isn't included in any toctree

We should likely use a Sphinx extension for those as well. Building those
are also made via some Makefile tricks that predates the time we start
adding our own extensions at the tree.

> Sphinx 7.3.7 and later are free of them.  I have no idea which change in
> Sphinx 7.3 got rid of them.
> 
> Now that the parallel build performance regression has be resolved in
> Sphinx 7.4, I don't think there is much demand for keeping Sphinx versions
> compatible.
> These build errors and extra warnings would encourage people to upgrade
> there Sphinx.  So I'm not going to nack this.
> 
> Of course, getting rid of above warnings with < Sphinx 7.3 would be ideal.

I'm all for using newer versions, but we need to check what LTS distros
are using those days.

On my machine, with -jauto, 3.4.3 is taking 11 minutes to build, which
is twice the time of 8.2.3. IMO, this is a very good reason for people
stop using legacy versions when possible :-)

Regards,
Mauro

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> When one does:
> 	make SPHINXDIRS="foo" htmldocs
>
> All patterns would be relative to Documentation/foo, which
> causes the include/exclude patterns like:
>
> 	include_patterns = [
> 		...
> 		f'foo/*.{ext}',
> 	]
>
> to break. This is not what it is expected. Address it by
> adding a logic to dynamically adjust the pattern when
> SPHINXDIRS is used.
>
> That allows adding parsers for other file types.
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>

Reviewed-by: Donald Hunter <donald.hunter@gmail.com>

Em Tue, 17 Jun 2025 11:38:06 +0100
Donald Hunter <donald.hunter@gmail.com> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > When one does:
> > 	make SPHINXDIRS="foo" htmldocs
> >
> > All patterns would be relative to Documentation/foo, which
> > causes the include/exclude patterns like:
> >
> > 	include_patterns = [
> > 		...
> > 		f'foo/*.{ext}',
> > 	]
> >
> > to break. This is not what it is expected. Address it by
> > adding a logic to dynamically adjust the pattern when
> > SPHINXDIRS is used.
> >
> > That allows adding parsers for other file types.
> >
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>  
> 
> Reviewed-by: Donald Hunter <donald.hunter@gmail.com>

Thanks for reviewing. At the next version, I'm placing some backward
compatible code for Sphinx 5.1, based on Akira's feedback.

As the basic logic is the same, I'm keeping your review there.


Thanks,
Mauro