Put this tool with the other documentation-related scripts.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 Documentation/Makefile                             | 14 +++++++-------
 Documentation/doc-guide/sphinx.rst                 |  4 ++--
 Documentation/sphinx/kerneldoc-preamble.sty        |  2 +-
 .../translations/it_IT/doc-guide/sphinx.rst        |  4 ++--
 .../translations/zh_CN/doc-guide/sphinx.rst        |  4 ++--
 Documentation/translations/zh_CN/how-to.rst        |  2 +-
 MAINTAINERS                                        |  2 --
 {scripts => tools/doc}/sphinx-pre-install          |  0
 8 files changed, 15 insertions(+), 17 deletions(-)
 rename {scripts => tools/doc}/sphinx-pre-install (100%)

diff --git a/Documentation/Makefile b/Documentation/Makefile
index eef5decb79b8..818d866756b0 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -46,7 +46,7 @@ ifeq ($(HAVE_SPHINX),0)
 .DEFAULT:
 	$(warning The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed and in PATH, or set the SPHINXBUILD make variable to point to the full path of the '$(SPHINXBUILD)' executable.)
 	@echo
-	@$(srctree)/scripts/sphinx-pre-install
+	@$(srctree)/tools/doc/sphinx-pre-install
 	@echo "  SKIP    Sphinx $@ target."
 
 else # HAVE_SPHINX
@@ -105,7 +105,7 @@ quiet_cmd_sphinx = SPHINX  $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
 	fi
 
 htmldocs:
-	@$(srctree)/scripts/sphinx-pre-install --version-check
+	@$(srctree)/tools/doc/sphinx-pre-install --version-check
 	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
 
 # If Rust support is available and .config exists, add rustdoc generated contents.
@@ -119,7 +119,7 @@ endif
 endif
 
 texinfodocs:
-	@$(srctree)/scripts/sphinx-pre-install --version-check
+	@$(srctree)/tools/doc/sphinx-pre-install --version-check
 	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,texinfo,$(var),texinfo,$(var)))
 
 # Note: the 'info' Make target is generated by sphinx itself when
@@ -131,7 +131,7 @@ linkcheckdocs:
 	@$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,linkcheck,$(var),,$(var)))
 
 latexdocs:
-	@$(srctree)/scripts/sphinx-pre-install --version-check
+	@$(srctree)/tools/doc/sphinx-pre-install --version-check
 	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$(var)))
 
 ifeq ($(HAVE_PDFLATEX),0)
@@ -144,7 +144,7 @@ else # HAVE_PDFLATEX
 
 pdfdocs: DENY_VF = XDG_CONFIG_HOME=$(FONTS_CONF_DENY_VF)
 pdfdocs: latexdocs
-	@$(srctree)/scripts/sphinx-pre-install --version-check
+	@$(srctree)/tools/doc/sphinx-pre-install --version-check
 	$(foreach var,$(SPHINXDIRS), \
 	   $(MAKE) PDFLATEX="$(PDFLATEX)" LATEXOPTS="$(LATEXOPTS)" $(DENY_VF) -C $(BUILDDIR)/$(var)/latex || sh $(srctree)/tools/doc/check-variable-fonts.sh || exit; \
 	   mkdir -p $(BUILDDIR)/$(var)/pdf; \
@@ -154,11 +154,11 @@ pdfdocs: latexdocs
 endif # HAVE_PDFLATEX
 
 epubdocs:
-	@$(srctree)/scripts/sphinx-pre-install --version-check
+	@$(srctree)/tools/doc/sphinx-pre-install --version-check
 	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var)))
 
 xmldocs:
-	@$(srctree)/scripts/sphinx-pre-install --version-check
+	@$(srctree)/tools/doc/sphinx-pre-install --version-check
 	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var)))
 
 endif # HAVE_SPHINX
diff --git a/Documentation/doc-guide/sphinx.rst b/Documentation/doc-guide/sphinx.rst
index 607589592bfb..2a0fc6c39cf4 100644
--- a/Documentation/doc-guide/sphinx.rst
+++ b/Documentation/doc-guide/sphinx.rst
@@ -106,7 +106,7 @@ There's a script that automatically checks for Sphinx dependencies. If it can
 recognize your distribution, it will also give a hint about the install
 command line options for your distro::
 
-	$ ./scripts/sphinx-pre-install
+	$ ./tools/doc/sphinx-pre-install
 	Checking if the needed tools for Fedora release 26 (Twenty Six) are available
 	Warning: better to also install "texlive-luatex85".
 	You should run:
@@ -116,7 +116,7 @@ command line options for your distro::
 		. sphinx_2.4.4/bin/activate
 		pip install -r Documentation/sphinx/requirements.txt
 
-	Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 468.
+	Can't build as 1 mandatory dependency is missing at ./tools/doc/sphinx-pre-install line 468.
 
 By default, it checks all the requirements for both html and PDF, including
 the requirements for images, math expressions and LaTeX build, and assumes
diff --git a/Documentation/sphinx/kerneldoc-preamble.sty b/Documentation/sphinx/kerneldoc-preamble.sty
index 5d68395539fe..736c2568377e 100644
--- a/Documentation/sphinx/kerneldoc-preamble.sty
+++ b/Documentation/sphinx/kerneldoc-preamble.sty
@@ -220,7 +220,7 @@
 	    If you want them, please install non-variable ``Noto Sans CJK''
 	    font families along with the texlive-xecjk package by following
 	    instructions from
-	    \sphinxcode{./scripts/sphinx-pre-install}.
+	    \sphinxcode{./tools/doc/sphinx-pre-install}.
 	    Having optional non-variable ``Noto Serif CJK'' font families will
 	    improve the looks of those translations.
 	\end{sphinxadmonition}}
diff --git a/Documentation/translations/it_IT/doc-guide/sphinx.rst b/Documentation/translations/it_IT/doc-guide/sphinx.rst
index 1f513bc33618..104aa159c1ce 100644
--- a/Documentation/translations/it_IT/doc-guide/sphinx.rst
+++ b/Documentation/translations/it_IT/doc-guide/sphinx.rst
@@ -109,7 +109,7 @@ Sphinx. Se lo script riesce a riconoscere la vostra distribuzione, allora
 sarà in grado di darvi dei suggerimenti su come procedere per completare
 l'installazione::
 
-	$ ./scripts/sphinx-pre-install
+	$ ./tools/doc/sphinx-pre-install
 	Checking if the needed tools for Fedora release 26 (Twenty Six) are available
 	Warning: better to also install "texlive-luatex85".
 	You should run:
@@ -119,7 +119,7 @@ l'installazione::
 		. sphinx_2.4.4/bin/activate
 		pip install -r Documentation/sphinx/requirements.txt
 
-	Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 468.
+	Can't build as 1 mandatory dependency is missing at ./tools/doc/sphinx-pre-install line 468.
 
 L'impostazione predefinita prevede il controllo dei requisiti per la generazione
 di documenti html e PDF, includendo anche il supporto per le immagini, le
diff --git a/Documentation/translations/zh_CN/doc-guide/sphinx.rst b/Documentation/translations/zh_CN/doc-guide/sphinx.rst
index 23eac67fbc30..3d2c4e262bb5 100644
--- a/Documentation/translations/zh_CN/doc-guide/sphinx.rst
+++ b/Documentation/translations/zh_CN/doc-guide/sphinx.rst
@@ -84,7 +84,7 @@ PDF和LaTeX构建
 这有一个脚本可以自动检查Sphinx依赖项。如果它认得您的发行版，还会提示您所用发行
 版的安装命令::
 
-	$ ./scripts/sphinx-pre-install
+	$ ./tools/doc/sphinx-pre-install
 	Checking if the needed tools for Fedora release 26 (Twenty Six) are available
 	Warning: better to also install "texlive-luatex85".
 	You should run:
@@ -94,7 +94,7 @@ PDF和LaTeX构建
 		. sphinx_2.4.4/bin/activate
 		pip install -r Documentation/sphinx/requirements.txt
 
-	Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 468.
+	Can't build as 1 mandatory dependency is missing at ./tools/doc/sphinx-pre-install line 468.
 
 默认情况下，它会检查html和PDF的所有依赖项，包括图像、数学表达式和LaTeX构建的
 需求，并假设将使用虚拟Python环境。html构建所需的依赖项被认为是必需的，其他依
diff --git a/Documentation/translations/zh_CN/how-to.rst b/Documentation/translations/zh_CN/how-to.rst
index cf66c72ee0c5..77da507d29cf 100644
--- a/Documentation/translations/zh_CN/how-to.rst
+++ b/Documentation/translations/zh_CN/how-to.rst
@@ -64,7 +64,7 @@ Linux 发行版和简单地使用 Linux 命令行，那么可以迅速开始了
 ::
 
 	cd linux
-	./scripts/sphinx-pre-install
+	./tools/doc/sphinx-pre-install
 
 以 Fedora 为例，它的输出是这样的::
 
diff --git a/MAINTAINERS b/MAINTAINERS
index b41b78215035..2f1374130240 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -7306,7 +7306,6 @@ F:	scripts/kernel-doc*
 F:	scripts/lib/abi/*
 F:	scripts/lib/kdoc/*
 F:	tools/net/ynl/pyynl/lib/doc_generator.py
-F:	scripts/sphinx-pre-install
 X:	Documentation/ABI/
 X:	Documentation/admin-guide/media/
 X:	Documentation/devicetree/
@@ -7341,7 +7340,6 @@ L:	linux-doc@vger.kernel.org
 S:	Maintained
 F:	Documentation/sphinx/parse-headers.pl
 F:	tools/doc/
-F:	scripts/sphinx-pre-install
 
 DOCUMENTATION/ITALIAN
 M:	Federico Vaga <federico.vaga@vaga.pv.it>
diff --git a/scripts/sphinx-pre-install b/tools/doc/sphinx-pre-install
similarity index 100%
rename from scripts/sphinx-pre-install
rename to tools/doc/sphinx-pre-install
-- 
2.50.1

Em Wed, 13 Aug 2025 15:32:06 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Put this tool with the other documentation-related scripts.

This one will be painful, as it will cause conflicts with my series
that clean up the docs Makefile.

> 
> Signed-off-by: Jonathan Corbet <corbet@lwn.net>
> ---
>  Documentation/Makefile                             | 14 +++++++-------
>  Documentation/doc-guide/sphinx.rst                 |  4 ++--
>  Documentation/sphinx/kerneldoc-preamble.sty        |  2 +-
>  .../translations/it_IT/doc-guide/sphinx.rst        |  4 ++--
>  .../translations/zh_CN/doc-guide/sphinx.rst        |  4 ++--
>  Documentation/translations/zh_CN/how-to.rst        |  2 +-
>  MAINTAINERS                                        |  2 --
>  {scripts => tools/doc}/sphinx-pre-install          |  0
>  8 files changed, 15 insertions(+), 17 deletions(-)
>  rename {scripts => tools/doc}/sphinx-pre-install (100%)
> 
> diff --git a/Documentation/Makefile b/Documentation/Makefile
> index eef5decb79b8..818d866756b0 100644
> --- a/Documentation/Makefile
> +++ b/Documentation/Makefile
> @@ -46,7 +46,7 @@ ifeq ($(HAVE_SPHINX),0)
>  .DEFAULT:
>  	$(warning The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed and in PATH, or set the SPHINXBUILD make variable to point to the full path of the '$(SPHINXBUILD)' executable.)
>  	@echo
> -	@$(srctree)/scripts/sphinx-pre-install
> +	@$(srctree)/tools/doc/sphinx-pre-install
>  	@echo "  SKIP    Sphinx $@ target."
>  
>  else # HAVE_SPHINX
> @@ -105,7 +105,7 @@ quiet_cmd_sphinx = SPHINX  $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
>  	fi
>  
>  htmldocs:
> -	@$(srctree)/scripts/sphinx-pre-install --version-check
> +	@$(srctree)/tools/doc/sphinx-pre-install --version-check
>  	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
>  
>  # If Rust support is available and .config exists, add rustdoc generated contents.
> @@ -119,7 +119,7 @@ endif
>  endif
>  
>  texinfodocs:
> -	@$(srctree)/scripts/sphinx-pre-install --version-check
> +	@$(srctree)/tools/doc/sphinx-pre-install --version-check
>  	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,texinfo,$(var),texinfo,$(var)))
>  
>  # Note: the 'info' Make target is generated by sphinx itself when
> @@ -131,7 +131,7 @@ linkcheckdocs:
>  	@$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,linkcheck,$(var),,$(var)))
>  
>  latexdocs:
> -	@$(srctree)/scripts/sphinx-pre-install --version-check
> +	@$(srctree)/tools/doc/sphinx-pre-install --version-check
>  	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$(var)))
>  
>  ifeq ($(HAVE_PDFLATEX),0)
> @@ -144,7 +144,7 @@ else # HAVE_PDFLATEX
>  
>  pdfdocs: DENY_VF = XDG_CONFIG_HOME=$(FONTS_CONF_DENY_VF)
>  pdfdocs: latexdocs
> -	@$(srctree)/scripts/sphinx-pre-install --version-check
> +	@$(srctree)/tools/doc/sphinx-pre-install --version-check
>  	$(foreach var,$(SPHINXDIRS), \
>  	   $(MAKE) PDFLATEX="$(PDFLATEX)" LATEXOPTS="$(LATEXOPTS)" $(DENY_VF) -C $(BUILDDIR)/$(var)/latex || sh $(srctree)/tools/doc/check-variable-fonts.sh || exit; \
>  	   mkdir -p $(BUILDDIR)/$(var)/pdf; \
> @@ -154,11 +154,11 @@ pdfdocs: latexdocs
>  endif # HAVE_PDFLATEX
>  
>  epubdocs:
> -	@$(srctree)/scripts/sphinx-pre-install --version-check
> +	@$(srctree)/tools/doc/sphinx-pre-install --version-check
>  	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var)))
>  
>  xmldocs:
> -	@$(srctree)/scripts/sphinx-pre-install --version-check
> +	@$(srctree)/tools/doc/sphinx-pre-install --version-check
>  	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var)))
>  
>  endif # HAVE_SPHINX
> diff --git a/Documentation/doc-guide/sphinx.rst b/Documentation/doc-guide/sphinx.rst
> index 607589592bfb..2a0fc6c39cf4 100644
> --- a/Documentation/doc-guide/sphinx.rst
> +++ b/Documentation/doc-guide/sphinx.rst
> @@ -106,7 +106,7 @@ There's a script that automatically checks for Sphinx dependencies. If it can
>  recognize your distribution, it will also give a hint about the install
>  command line options for your distro::
>  
> -	$ ./scripts/sphinx-pre-install
> +	$ ./tools/doc/sphinx-pre-install
>  	Checking if the needed tools for Fedora release 26 (Twenty Six) are available
>  	Warning: better to also install "texlive-luatex85".
>  	You should run:
> @@ -116,7 +116,7 @@ command line options for your distro::
>  		. sphinx_2.4.4/bin/activate
>  		pip install -r Documentation/sphinx/requirements.txt
>  
> -	Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 468.
> +	Can't build as 1 mandatory dependency is missing at ./tools/doc/sphinx-pre-install line 468.
>  
>  By default, it checks all the requirements for both html and PDF, including
>  the requirements for images, math expressions and LaTeX build, and assumes
> diff --git a/Documentation/sphinx/kerneldoc-preamble.sty b/Documentation/sphinx/kerneldoc-preamble.sty
> index 5d68395539fe..736c2568377e 100644
> --- a/Documentation/sphinx/kerneldoc-preamble.sty
> +++ b/Documentation/sphinx/kerneldoc-preamble.sty
> @@ -220,7 +220,7 @@
>  	    If you want them, please install non-variable ``Noto Sans CJK''
>  	    font families along with the texlive-xecjk package by following
>  	    instructions from
> -	    \sphinxcode{./scripts/sphinx-pre-install}.
> +	    \sphinxcode{./tools/doc/sphinx-pre-install}.
>  	    Having optional non-variable ``Noto Serif CJK'' font families will
>  	    improve the looks of those translations.
>  	\end{sphinxadmonition}}
> diff --git a/Documentation/translations/it_IT/doc-guide/sphinx.rst b/Documentation/translations/it_IT/doc-guide/sphinx.rst
> index 1f513bc33618..104aa159c1ce 100644
> --- a/Documentation/translations/it_IT/doc-guide/sphinx.rst
> +++ b/Documentation/translations/it_IT/doc-guide/sphinx.rst
> @@ -109,7 +109,7 @@ Sphinx. Se lo script riesce a riconoscere la vostra distribuzione, allora
>  sarà in grado di darvi dei suggerimenti su come procedere per completare
>  l'installazione::
>  
> -	$ ./scripts/sphinx-pre-install
> +	$ ./tools/doc/sphinx-pre-install
>  	Checking if the needed tools for Fedora release 26 (Twenty Six) are available
>  	Warning: better to also install "texlive-luatex85".
>  	You should run:
> @@ -119,7 +119,7 @@ l'installazione::
>  		. sphinx_2.4.4/bin/activate
>  		pip install -r Documentation/sphinx/requirements.txt
>  
> -	Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 468.
> +	Can't build as 1 mandatory dependency is missing at ./tools/doc/sphinx-pre-install line 468.
>  
>  L'impostazione predefinita prevede il controllo dei requisiti per la generazione
>  di documenti html e PDF, includendo anche il supporto per le immagini, le
> diff --git a/Documentation/translations/zh_CN/doc-guide/sphinx.rst b/Documentation/translations/zh_CN/doc-guide/sphinx.rst
> index 23eac67fbc30..3d2c4e262bb5 100644
> --- a/Documentation/translations/zh_CN/doc-guide/sphinx.rst
> +++ b/Documentation/translations/zh_CN/doc-guide/sphinx.rst
> @@ -84,7 +84,7 @@ PDF和LaTeX构建
>  这有一个脚本可以自动检查Sphinx依赖项。如果它认得您的发行版，还会提示您所用发行
>  版的安装命令::
>  
> -	$ ./scripts/sphinx-pre-install
> +	$ ./tools/doc/sphinx-pre-install
>  	Checking if the needed tools for Fedora release 26 (Twenty Six) are available
>  	Warning: better to also install "texlive-luatex85".
>  	You should run:
> @@ -94,7 +94,7 @@ PDF和LaTeX构建
>  		. sphinx_2.4.4/bin/activate
>  		pip install -r Documentation/sphinx/requirements.txt
>  
> -	Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 468.
> +	Can't build as 1 mandatory dependency is missing at ./tools/doc/sphinx-pre-install line 468.
>  
>  默认情况下，它会检查html和PDF的所有依赖项，包括图像、数学表达式和LaTeX构建的
>  需求，并假设将使用虚拟Python环境。html构建所需的依赖项被认为是必需的，其他依
> diff --git a/Documentation/translations/zh_CN/how-to.rst b/Documentation/translations/zh_CN/how-to.rst
> index cf66c72ee0c5..77da507d29cf 100644
> --- a/Documentation/translations/zh_CN/how-to.rst
> +++ b/Documentation/translations/zh_CN/how-to.rst
> @@ -64,7 +64,7 @@ Linux 发行版和简单地使用 Linux 命令行，那么可以迅速开始了
>  ::
>  
>  	cd linux
> -	./scripts/sphinx-pre-install
> +	./tools/doc/sphinx-pre-install
>  
>  以 Fedora 为例，它的输出是这样的::
>  
> diff --git a/MAINTAINERS b/MAINTAINERS
> index b41b78215035..2f1374130240 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -7306,7 +7306,6 @@ F:	scripts/kernel-doc*
>  F:	scripts/lib/abi/*
>  F:	scripts/lib/kdoc/*
>  F:	tools/net/ynl/pyynl/lib/doc_generator.py
> -F:	scripts/sphinx-pre-install
>  X:	Documentation/ABI/
>  X:	Documentation/admin-guide/media/
>  X:	Documentation/devicetree/
> @@ -7341,7 +7340,6 @@ L:	linux-doc@vger.kernel.org
>  S:	Maintained
>  F:	Documentation/sphinx/parse-headers.pl
>  F:	tools/doc/
> -F:	scripts/sphinx-pre-install
>  
>  DOCUMENTATION/ITALIAN
>  M:	Federico Vaga <federico.vaga@vaga.pv.it>
> diff --git a/scripts/sphinx-pre-install b/tools/doc/sphinx-pre-install
> similarity index 100%
> rename from scripts/sphinx-pre-install
> rename to tools/doc/sphinx-pre-install



Thanks,
Mauro

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> Em Wed, 13 Aug 2025 15:32:06 -0600
> Jonathan Corbet <corbet@lwn.net> escreveu:
>
>> Put this tool with the other documentation-related scripts.
>
> This one will be painful, as it will cause conflicts with my series
> that clean up the docs Makefile.

Just in general ... I'm more than happy to put this whole series on the
back burner until we've gotten that other stuff merged ... it's an RFC
after all, and there's no urgency here.

jon

Em Wed, 13 Aug 2025 20:14:42 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > Em Wed, 13 Aug 2025 15:32:06 -0600
> > Jonathan Corbet <corbet@lwn.net> escreveu:
> >  
> >> Put this tool with the other documentation-related scripts.  
> >
> > This one will be painful, as it will cause conflicts with my series
> > that clean up the docs Makefile.  
> 
> Just in general ... I'm more than happy to put this whole series on the
> back burner until we've gotten that other stuff merged ... it's an RFC
> after all, and there's no urgency here.

Ok, thanks! Yeah, this would help avoiding conflicts with the patches
I have piled here.

Btw, as I said I have a /51 patch series focused on parse-headers.pl
and kernel_include.py, which is currently used only by media docs.

For those who don't know, this was added at the very beginning of
Sphinx adoption, and had its own particular way to create *.rst
files from source code with cross references, via an specially crafted
Makefile that runs before Sphinx. This was written as a way to detect
uAPI documentation gaps, producing warnings for unsolved cross references.
Recent Sphinx versions broke it by disabling cross-reference warnings.

This series is big (51 patches) because it needs to fix thousands of
broken cross references on media. I may end splitting it on two series
to make easier for review, one for the script and another for media doc
fixes.

Such series affect this RFC as it is creating a tools/docs and placing 
there the parse-headers.py code as:

	 tools/docs/lib/parse_data_structs.py                                  |  230 +++++++++++++++--------------------
	 tools/docs/parse-headers.py                                           |    5 

Now, if you prefer tools/doc instead and/or place the libs elsewhere,
we have a couple of options:

1. rebase my series to do the changes. I suspect that there won't
   be much conflicts, but this may delay a little bit sending you
   what I have;

2. add a patch at the end moving stuff elsewhere;

3. on your series, move them elsewhere.

What do you prefer?

Thanks,
Mauro

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

> This series is big (51 patches) because it needs to fix thousands of
> broken cross references on media. I may end splitting it on two series
> to make easier for review, one for the script and another for media doc
> fixes.

That might help, yes.

> Such series affect this RFC as it is creating a tools/docs and placing 
> there the parse-headers.py code as:
>
> 	 tools/docs/lib/parse_data_structs.py                                  |  230 +++++++++++++++--------------------
> 	 tools/docs/parse-headers.py                                           |    5 
>
> Now, if you prefer tools/doc instead and/or place the libs elsewhere,
> we have a couple of options:
>
> 1. rebase my series to do the changes. I suspect that there won't
>    be much conflicts, but this may delay a little bit sending you
>    what I have;
>
> 2. add a patch at the end moving stuff elsewhere;
>
> 3. on your series, move them elsewhere.
>
> What do you prefer?

Between "tools/doc" and "tools/docs" I don't really have overly strong
feelings; if you work has the latter we can just stick with that.  If
you propose "tools/Documentation", though, expect resistance :)

As I said, my series was an RFC to see what it would look like; it did't
take all that long the first time around, and will be even quicker to
redo on top of a new base, whatever that turns out to be.

Thanks,

jon

Em Thu, 14 Aug 2025 07:52:22 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> > This series is big (51 patches) because it needs to fix thousands of
> > broken cross references on media. I may end splitting it on two series
> > to make easier for review, one for the script and another for media doc
> > fixes.  
> 
> That might help, yes.
> 
> > Such series affect this RFC as it is creating a tools/docs and placing 
> > there the parse-headers.py code as:
> >
> > 	 tools/docs/lib/parse_data_structs.py                                  |  230 +++++++++++++++--------------------
> > 	 tools/docs/parse-headers.py                                           |    5 
> >
> > Now, if you prefer tools/doc instead and/or place the libs elsewhere,
> > we have a couple of options:
> >
> > 1. rebase my series to do the changes. I suspect that there won't
> >    be much conflicts, but this may delay a little bit sending you
> >    what I have;
> >
> > 2. add a patch at the end moving stuff elsewhere;
> >
> > 3. on your series, move them elsewhere.
> >
> > What do you prefer?  
> 
> Between "tools/doc" and "tools/docs" I don't really have overly strong
> feelings; if you work has the latter we can just stick with that.  If
> you propose "tools/Documentation", though, expect resistance :)

<joke>
Heh, I'm tempted to propose:
	/Documentation -> /docs
or
	/Documentation -> /Docs
</joke>

Ok, so let's keep tools/docs then. We need to decide about python
lib. On my series, I'm placing at tools/docs/lib, but I guess we
can change it later.

From my side, I would prefer to have a single directory for tools,
as we may place there things that aren't specific to docs.

For instance, I have my own class that I use for command execution,
using asyncio. The rationale is that it allows output messages in
real time without needing to wait for the entire process to end(*).

(*) I recently discovered a way to do that without needing asyncio,
    which makes the code a little bit simpler.

Either using asyncio or not, a class for such purpose is something
that multiple tools could use. So, a generic dir like tools/lib, 
lib/python, ... IMO makes sense.

> As I said, my series was an RFC to see what it would look like; it did't
> take all that long the first time around, and will be even quicker to
> redo on top of a new base, whatever that turns out to be.

With regards to the RFC, IMO we still may need to discuss how we'll end 
placing libraries under a LIBDIR. IMO, your RFC should also propose
a directory structure. I mean, we could have something like:

	LIBDIR     # either tools/docs/lib, tools/lib, lib/python or whatever
	|
	+---> common
	\---> docs
		|
	    	+---> kdoc
	    	\---> abi

We could instead do:
	- flatten "common" to LIBDIR; or:
	- flatten "docs" to LIBDIR; or:
	- flatten both but keeping kdoc, abi, ... directories inside
	  LIBDIR; or:
	- have a completely flatten directory with everything
	  under LIBDIR.

 
> Thanks,
> 
> jon



Thanks,
Mauro

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

>> Between "tools/doc" and "tools/docs" I don't really have overly strong
>> feelings; if you work has the latter we can just stick with that.  If
>> you propose "tools/Documentation", though, expect resistance :)
>
> <joke>
> Heh, I'm tempted to propose:
> 	/Documentation -> /docs
> or
> 	/Documentation -> /Docs
> </joke>

I proposed the former at a maintainers summit a few years ago ... the
response was less than fully positive.  I guess the fact that docs have
the longest and only capitalized top-level directory name shows their
relative importance :)

> Ok, so let's keep tools/docs then. We need to decide about python
> lib. On my series, I'm placing at tools/docs/lib, but I guess we
> can change it later.
>
> From my side, I would prefer to have a single directory for tools,
> as we may place there things that aren't specific to docs.
>
> For instance, I have my own class that I use for command execution,
> using asyncio. The rationale is that it allows output messages in
> real time without needing to wait for the entire process to end(*).
>
> (*) I recently discovered a way to do that without needing asyncio,
>     which makes the code a little bit simpler.

This is just flushing the output buffer?  Asyncio seems like a heavy
tool for that; I guess I'm missing something.

> Either using asyncio or not, a class for such purpose is something
> that multiple tools could use. So, a generic dir like tools/lib, 
> lib/python, ... IMO makes sense.

I agree with this, anyway.  "tools/python" might end up as the winning
suggestion. 

>> As I said, my series was an RFC to see what it would look like; it did't
>> take all that long the first time around, and will be even quicker to
>> redo on top of a new base, whatever that turns out to be.
>
> With regards to the RFC, IMO we still may need to discuss how we'll end 
> placing libraries under a LIBDIR. IMO, your RFC should also propose
> a directory structure. I mean, we could have something like:
>
> 	LIBDIR     # either tools/docs/lib, tools/lib, lib/python or whatever
> 	|
> 	+---> common
> 	\---> docs
> 		|
> 	    	+---> kdoc
> 	    	\---> abi
>
> We could instead do:
> 	- flatten "common" to LIBDIR; or:
> 	- flatten "docs" to LIBDIR; or:
> 	- flatten both but keeping kdoc, abi, ... directories inside
> 	  LIBDIR; or:
> 	- have a completely flatten directory with everything
> 	  under LIBDIR.

I'm not sure we need the common/docs intermediate directory.

Meanwhile, I had a related, possibly unpopular idea...  Start with
.../tools/python/kernel and put a basic __init__.py file there;
everything else would go into that directory or before.  The imports
would then read something like:

  from kernel import abi_parser

That would make it clear which modules are ours and which come from
elsewhere.

I was planning to toss together another RFC with that scheme, again to
see what it would look like in practice.

Thoughts?

jon

Em Fri, 15 Aug 2025 07:18:51 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> >> Between "tools/doc" and "tools/docs" I don't really have overly strong
> >> feelings; if you work has the latter we can just stick with that.  If
> >> you propose "tools/Documentation", though, expect resistance :)  
> >
> > <joke>
> > Heh, I'm tempted to propose:
> > 	/Documentation -> /docs
> > or
> > 	/Documentation -> /Docs
> > </joke>  
> 
> I proposed the former at a maintainers summit a few years ago ... the
> response was less than fully positive.  I guess the fact that docs have
> the longest and only capitalized top-level directory name shows their
> relative importance :)

:-)

> > Ok, so let's keep tools/docs then. We need to decide about python
> > lib. On my series, I'm placing at tools/docs/lib, but I guess we
> > can change it later.
> >
> > From my side, I would prefer to have a single directory for tools,
> > as we may place there things that aren't specific to docs.
> >
> > For instance, I have my own class that I use for command execution,
> > using asyncio. The rationale is that it allows output messages in
> > real time without needing to wait for the entire process to end(*).
> >
> > (*) I recently discovered a way to do that without needing asyncio,
> >     which makes the code a little bit simpler.  
> 
> This is just flushing the output buffer?  Asyncio seems like a heavy
> tool for that; I guess I'm missing something.

Yes, flushing output as they arrive while storing results from stdout and
stderr and handling  "stdin", eventually dynamically printing stdin when
debug is enabled. The problem sounds simple, and this is really easy to
implement in Perl. However, on Python, this ends to be a very complex
problem that one can only get all the caveats after implementing unit 
tests and then discovering, for instance, that, if stdin is bigger than 
32KB, Python read logic freezes forever (or until gather() timeout, and
the stdin was incomplete.

To fix it, one needs to implement a different stdin function that can't
wait anymore for the end of a read operation, but instead needs to pick
buffers in small chunks. Plus, the subprocess execution method requires
some unusual buffering parameters.
 
> > Either using asyncio or not, a class for such purpose is something
> > that multiple tools could use. So, a generic dir like tools/lib, 
> > lib/python, ... IMO makes sense.  
> 
> I agree with this, anyway.  "tools/python" might end up as the winning
> suggestion. 
> 
> >> As I said, my series was an RFC to see what it would look like; it did't
> >> take all that long the first time around, and will be even quicker to
> >> redo on top of a new base, whatever that turns out to be.  
> >
> > With regards to the RFC, IMO we still may need to discuss how we'll end 
> > placing libraries under a LIBDIR. IMO, your RFC should also propose
> > a directory structure. I mean, we could have something like:
> >
> > 	LIBDIR     # either tools/docs/lib, tools/lib, lib/python or whatever
> > 	|
> > 	+---> common
> > 	\---> docs
> > 		|
> > 	    	+---> kdoc
> > 	    	\---> abi
> >
> > We could instead do:
> > 	- flatten "common" to LIBDIR; or:
> > 	- flatten "docs" to LIBDIR; or:
> > 	- flatten both but keeping kdoc, abi, ... directories inside
> > 	  LIBDIR; or:
> > 	- have a completely flatten directory with everything
> > 	  under LIBDIR.  
> 
> I'm not sure we need the common/docs intermediate directory.
> 
> Meanwhile, I had a related, possibly unpopular idea...  Start with
> .../tools/python/kernel and put a basic __init__.py file there;
> everything else would go into that directory or before.  The imports
> would then read something like:
> 
>   from kernel import abi_parser

Not against something similar to it, but IMO "kernel" is a bad
name as it sounds something that runs in kernel stace or for Kernel
build. It could be, instead:

	from lib import abi_parser

Yet, I guess it may still need to add something at PATH, depending from
where current working dir the script was called (but tests required).

For instance Documentation/sphinx would need to pick lib from
"../../tools/python". So, I guess Sphinx extensions need first to
do:

	sys.path.insert(0, "../../tools/python")

and then:

	from lib import abi_parser         # or:
	from lib.abi import abi_parser     # or:
	import .abi_parser


Btw, nothing prevents moving extensions from Documentation/sphinx
into tools/sphinx_extensions. We just need to add the path insert
at conf.py.

> That would make it clear which modules are ours and which come from
> elsewhere.
> 
> I was planning to toss together another RFC with that scheme, again to
> see what it would look like in practice.

> Thoughts?

Fine from my side.

Regards,
Mauro

Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:

>> I'm not sure we need the common/docs intermediate directory.
>> 
>> Meanwhile, I had a related, possibly unpopular idea...  Start with
>> .../tools/python/kernel and put a basic __init__.py file there;
>> everything else would go into that directory or before.  The imports
>> would then read something like:
>> 
>>   from kernel import abi_parser
>
> Not against something similar to it, but IMO "kernel" is a bad
> name as it sounds something that runs in kernel stace or for Kernel
> build. It could be, instead:
>
> 	from lib import abi_parser

Part of my purpose was to make it clear that the import was coming from
our own library - to distinguish it from all of the other imports that
these programs have.  "Kernel" seems good to me, but we could call it
"kernel_lib" or some such if we really want.  "lib" seems too generic.

> Yet, I guess it may still need to add something at PATH, depending from
> where current working dir the script was called (but tests required).

That seems hard to avoid, yes.

Of course, we could require that all kernel tools run in a special
virtualenv :)

> Btw, nothing prevents moving extensions from Documentation/sphinx
> into tools/sphinx_extensions. We just need to add the path insert
> at conf.py.

I feel less of a need to do that; it seems that the Sphinx-specific
stuff can stay where it is.  Though I guess I wouldn't scream too loud
if people really wanted to do that move.

Thanks,

jon

Em Fri, 15 Aug 2025 13:31:45 -0600
Jonathan Corbet <corbet@lwn.net> escreveu:

> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> writes:
> 
> >> I'm not sure we need the common/docs intermediate directory.
> >> 
> >> Meanwhile, I had a related, possibly unpopular idea...  Start with
> >> .../tools/python/kernel and put a basic __init__.py file there;
> >> everything else would go into that directory or before.  The imports
> >> would then read something like:
> >> 
> >>   from kernel import abi_parser  
> >
> > Not against something similar to it, but IMO "kernel" is a bad
> > name as it sounds something that runs in kernel stace or for Kernel
> > build. It could be, instead:
> >
> > 	from lib import abi_parser  
> 
> Part of my purpose was to make it clear that the import was coming from
> our own library - to distinguish it from all of the other imports that
> these programs have.  "Kernel" seems good to me, but we could call it
> "kernel_lib" or some such if we really want.  "lib" seems too generic.

Ok. let's stick with "Kernel" then.

> > Yet, I guess it may still need to add something at PATH, depending from
> > where current working dir the script was called (but tests required).  
> 
> That seems hard to avoid, yes.
> 
> Of course, we could require that all kernel tools run in a special
> virtualenv :)

No need. Yet, if you look at sphinx-build-wrapper, it has a "-V"
command line. If used, it will either pick a venv name or seek for
sphinx_latest and run from there. I wrote it mainly to help me
on my main devel machine, where I opted to have several different
venvs instead of installing via OS.

I was tempted of making the tool autodetect venv, if sphinx-build
is not found. I opted to not do it for now, but keeping it in mind
for a possible future change. There is also a code that detect if
Python is too old, running a newer version of it if found at the
system if required by the tool to run. 

Maybe we can place both logic into a library and let most tools
use it, adjusting the main function call to something like:

	MIN_PYTHON_VERSION = (3, x, y)

	def main(...):
	   ...

	if __name__ == "__main__":
	     run_on_version(MIN_PYTHON_VERSION, run, use_venv_if_available=True)

> > Btw, nothing prevents moving extensions from Documentation/sphinx
> > into tools/sphinx_extensions. We just need to add the path insert
> > at conf.py.  
> 
> I feel less of a need to do that; it seems that the Sphinx-specific
> stuff can stay where it is.  Though I guess I wouldn't scream too loud
> if people really wanted to do that move.

Ok. As on extensions we always have srctree set (sphinx-build-wrapper
warrants that), the PATH will always be relative to the kernel
directory, e.g.:

	srctree = os.path.abspath(os.environ["srctree"])
	sys.path.insert(0, os.path.join(srctree, "tools/python"))

	from Kernel import abi_parser

The only advantage of moving them to tools/python would be to remove
the need of "srctree" env, using instead os.path.dirname(__file__), 
e.g.:

	sys.path.insert(0, os.path.dirname(os.path.realpath(__file__)))
	from Kernel import abi_parser
	
but still at lease some extensions still need srctree anyway. So,
not really necessary.

So, let's not touch it then.

Thanks,
Mauro