Sometimes, it is desired to run Sphinx from a virtual environment.
Add a command line parameter to automatically build Sphinx from
such environment.
Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
---
tools/docs/sphinx-build-wrapper | 30 +++++++++++++++++++++++++++---
1 file changed, 27 insertions(+), 3 deletions(-)
diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrapper
index ea9f8e17b0bc..cf7b30bc40ff 100755
--- a/tools/docs/sphinx-build-wrapper
+++ b/tools/docs/sphinx-build-wrapper
@@ -63,6 +63,7 @@ from jobserver import JobserverExec # pylint: disable=C0413,C0411,E0401
#
# Some constants
#
+VENV_DEFAULT = "sphinx_latest"
MIN_PYTHON_VERSION = PythonVersion("3.7").version
PAPER = ["", "a4", "letter"]
@@ -119,8 +120,9 @@ class SphinxBuilder:
return path
- def __init__(self, builddir, verbose=False, n_jobs=None):
+ def __init__(self, builddir, venv=None, verbose=False, n_jobs=None):
"""Initialize internal variables"""
+ self.venv = venv
self.verbose = None
#
@@ -195,6 +197,21 @@ class SphinxBuilder:
self.env = os.environ.copy()
+ #
+ # If venv command line argument is specified, run Sphinx from venv
+ #
+ if venv:
+ bin_dir = os.path.join(venv, "bin")
+ if not os.path.isfile(os.path.join(bin_dir, "activate")):
+ sys.exit(f"Venv {venv} not found.")
+
+ # "activate" virtual env
+ self.env["PATH"] = bin_dir + ":" + self.env["PATH"]
+ self.env["VIRTUAL_ENV"] = venv
+ if "PYTHONHOME" in self.env:
+ del self.env["PYTHONHOME"]
+ print(f"Setting venv to {venv}")
+
def run_sphinx(self, sphinx_build, build_args, *args, **pwargs):
"""
Executes sphinx-build using current python3 command and setting
@@ -209,7 +226,10 @@ class SphinxBuilder:
cmd = []
- cmd.append(sys.executable)
+ if self.venv:
+ cmd.append("python")
+ else:
+ cmd.append(sys.executable)
cmd.append(sphinx_build)
@@ -533,11 +553,15 @@ def main():
parser.add_argument('-j', '--jobs', type=jobs_type,
help="Sets number of jobs to use with sphinx-build")
+ parser.add_argument("-V", "--venv", nargs='?', const=f'{VENV_DEFAULT}',
+ default=None,
+ help=f'If used, run Sphinx from a venv dir (default dir: {VENV_DEFAULT})')
+
args = parser.parse_args()
PythonVersion.check_python(MIN_PYTHON_VERSION)
- builder = SphinxBuilder(builddir=args.builddir,
+ builder = SphinxBuilder(builddir=args.builddir, venv=args.venv,
verbose=args.verbose, n_jobs=args.jobs)
builder.build(args.target, sphinxdirs=args.sphinxdirs, conf=args.conf,
--
2.51.0On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> Sometimes, it is desired to run Sphinx from a virtual environment.
> Add a command line parameter to automatically build Sphinx from
> such environment.
Why?
If you want Sphinx from a virtual environment, you enter the
environment, and run the regular build, with sphinx-build from the PATH
that points at the venv.
We don't do this kind of extra magic for any other tools, I honestly
don't understand why we'd do this for Sphinx. This just adds complexity
for no good reason.
BR,
Jani.
>
> Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> ---
> tools/docs/sphinx-build-wrapper | 30 +++++++++++++++++++++++++++---
> 1 file changed, 27 insertions(+), 3 deletions(-)
>
> diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrapper
> index ea9f8e17b0bc..cf7b30bc40ff 100755
> --- a/tools/docs/sphinx-build-wrapper
> +++ b/tools/docs/sphinx-build-wrapper
> @@ -63,6 +63,7 @@ from jobserver import JobserverExec # pylint: disable=C0413,C0411,E0401
> #
> # Some constants
> #
> +VENV_DEFAULT = "sphinx_latest"
> MIN_PYTHON_VERSION = PythonVersion("3.7").version
> PAPER = ["", "a4", "letter"]
>
> @@ -119,8 +120,9 @@ class SphinxBuilder:
>
> return path
>
> - def __init__(self, builddir, verbose=False, n_jobs=None):
> + def __init__(self, builddir, venv=None, verbose=False, n_jobs=None):
> """Initialize internal variables"""
> + self.venv = venv
> self.verbose = None
>
> #
> @@ -195,6 +197,21 @@ class SphinxBuilder:
>
> self.env = os.environ.copy()
>
> + #
> + # If venv command line argument is specified, run Sphinx from venv
> + #
> + if venv:
> + bin_dir = os.path.join(venv, "bin")
> + if not os.path.isfile(os.path.join(bin_dir, "activate")):
> + sys.exit(f"Venv {venv} not found.")
> +
> + # "activate" virtual env
> + self.env["PATH"] = bin_dir + ":" + self.env["PATH"]
> + self.env["VIRTUAL_ENV"] = venv
> + if "PYTHONHOME" in self.env:
> + del self.env["PYTHONHOME"]
> + print(f"Setting venv to {venv}")
> +
> def run_sphinx(self, sphinx_build, build_args, *args, **pwargs):
> """
> Executes sphinx-build using current python3 command and setting
> @@ -209,7 +226,10 @@ class SphinxBuilder:
>
> cmd = []
>
> - cmd.append(sys.executable)
> + if self.venv:
> + cmd.append("python")
> + else:
> + cmd.append(sys.executable)
>
> cmd.append(sphinx_build)
>
> @@ -533,11 +553,15 @@ def main():
> parser.add_argument('-j', '--jobs', type=jobs_type,
> help="Sets number of jobs to use with sphinx-build")
>
> + parser.add_argument("-V", "--venv", nargs='?', const=f'{VENV_DEFAULT}',
> + default=None,
> + help=f'If used, run Sphinx from a venv dir (default dir: {VENV_DEFAULT})')
> +
> args = parser.parse_args()
>
> PythonVersion.check_python(MIN_PYTHON_VERSION)
>
> - builder = SphinxBuilder(builddir=args.builddir,
> + builder = SphinxBuilder(builddir=args.builddir, venv=args.venv,
> verbose=args.verbose, n_jobs=args.jobs)
>
> builder.build(args.target, sphinxdirs=args.sphinxdirs, conf=args.conf,
--
Jani Nikula, Intel
Em Wed, 10 Sep 2025 13:51:40 +0300
Jani Nikula <jani.nikula@linux.intel.com> escreveu:
> On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > Sometimes, it is desired to run Sphinx from a virtual environment.
> > Add a command line parameter to automatically build Sphinx from
> > such environment.
>
> Why?
In my case, to be able to test build with different Sphinx versions.
On some distros, only venv works.
> If you want Sphinx from a virtual environment, you enter the
> environment, and run the regular build, with sphinx-build from the PATH
> that points at the venv.
when you do that, ./scripts/spdxcheck.py breaks, affecting checkpatch.
>
> We don't do this kind of extra magic for any other tools, I honestly
> don't understand why we'd do this for Sphinx. This just adds complexity
> for no good reason.
> >
> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
> > ---
> > tools/docs/sphinx-build-wrapper | 30 +++++++++++++++++++++++++++---
> > 1 file changed, 27 insertions(+), 3 deletions(-)
> >
> > diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrapper
> > index ea9f8e17b0bc..cf7b30bc40ff 100755
> > --- a/tools/docs/sphinx-build-wrapper
> > +++ b/tools/docs/sphinx-build-wrapper
> > @@ -63,6 +63,7 @@ from jobserver import JobserverExec # pylint: disable=C0413,C0411,E0401
> > #
> > # Some constants
> > #
> > +VENV_DEFAULT = "sphinx_latest"
> > MIN_PYTHON_VERSION = PythonVersion("3.7").version
> > PAPER = ["", "a4", "letter"]
> >
> > @@ -119,8 +120,9 @@ class SphinxBuilder:
> >
> > return path
> >
> > - def __init__(self, builddir, verbose=False, n_jobs=None):
> > + def __init__(self, builddir, venv=None, verbose=False, n_jobs=None):
> > """Initialize internal variables"""
> > + self.venv = venv
> > self.verbose = None
> >
> > #
> > @@ -195,6 +197,21 @@ class SphinxBuilder:
> >
> > self.env = os.environ.copy()
> >
> > + #
> > + # If venv command line argument is specified, run Sphinx from venv
> > + #
> > + if venv:
> > + bin_dir = os.path.join(venv, "bin")
> > + if not os.path.isfile(os.path.join(bin_dir, "activate")):
> > + sys.exit(f"Venv {venv} not found.")
> > +
> > + # "activate" virtual env
> > + self.env["PATH"] = bin_dir + ":" + self.env["PATH"]
> > + self.env["VIRTUAL_ENV"] = venv
> > + if "PYTHONHOME" in self.env:
> > + del self.env["PYTHONHOME"]
> > + print(f"Setting venv to {venv}")
> > +
> > def run_sphinx(self, sphinx_build, build_args, *args, **pwargs):
> > """
> > Executes sphinx-build using current python3 command and setting
> > @@ -209,7 +226,10 @@ class SphinxBuilder:
> >
> > cmd = []
> >
> > - cmd.append(sys.executable)
> > + if self.venv:
> > + cmd.append("python")
> > + else:
> > + cmd.append(sys.executable)
> >
> > cmd.append(sphinx_build)
> >
> > @@ -533,11 +553,15 @@ def main():
> > parser.add_argument('-j', '--jobs', type=jobs_type,
> > help="Sets number of jobs to use with sphinx-build")
> >
> > + parser.add_argument("-V", "--venv", nargs='?', const=f'{VENV_DEFAULT}',
> > + default=None,
> > + help=f'If used, run Sphinx from a venv dir (default dir: {VENV_DEFAULT})')
> > +
> > args = parser.parse_args()
> >
> > PythonVersion.check_python(MIN_PYTHON_VERSION)
> >
> > - builder = SphinxBuilder(builddir=args.builddir,
> > + builder = SphinxBuilder(builddir=args.builddir, venv=args.venv,
> > verbose=args.verbose, n_jobs=args.jobs)
> >
> > builder.build(args.target, sphinxdirs=args.sphinxdirs, conf=args.conf,
>
Thanks,
Mauro
On Fri, 12 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> Em Wed, 10 Sep 2025 13:51:40 +0300
> Jani Nikula <jani.nikula@linux.intel.com> escreveu:
>
>> On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
>> > Sometimes, it is desired to run Sphinx from a virtual environment.
>> > Add a command line parameter to automatically build Sphinx from
>> > such environment.
>>
>> Why?
>
> In my case, to be able to test build with different Sphinx versions.
> On some distros, only venv works.
I mean why add the complexity of running inside a venv in the wrapper.
>> If you want Sphinx from a virtual environment, you enter the
>> environment, and run the regular build, with sphinx-build from the PATH
>> that points at the venv.
>
> when you do that, ./scripts/spdxcheck.py breaks, affecting checkpatch.
Then you could turn the whole argument around, and say spdxcheck.py
should jump through venv and dependency hoops instead of the docs build.
The point is, it should be the user's responsibility to deal with the
environment and the dependencies.
If they're setting up a virtual environment, and it affects checkpatch,
then they should also include the spdxcheck.py dependencies in the
virtual environment.
This feels like reinventing pipx in a Sphinx wrapper.
We should *reduce* the complexity, not increase it.
>> We don't do this kind of extra magic for any other tools, I honestly
>> don't understand why we'd do this for Sphinx. This just adds complexity
>> for no good reason.
>
>> >
>> > Signed-off-by: Mauro Carvalho Chehab <mchehab+huawei@kernel.org>
>> > ---
>> > tools/docs/sphinx-build-wrapper | 30 +++++++++++++++++++++++++++---
>> > 1 file changed, 27 insertions(+), 3 deletions(-)
>> >
>> > diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrapper
>> > index ea9f8e17b0bc..cf7b30bc40ff 100755
>> > --- a/tools/docs/sphinx-build-wrapper
>> > +++ b/tools/docs/sphinx-build-wrapper
>> > @@ -63,6 +63,7 @@ from jobserver import JobserverExec # pylint: disable=C0413,C0411,E0401
>> > #
>> > # Some constants
>> > #
>> > +VENV_DEFAULT = "sphinx_latest"
>> > MIN_PYTHON_VERSION = PythonVersion("3.7").version
>> > PAPER = ["", "a4", "letter"]
>> >
>> > @@ -119,8 +120,9 @@ class SphinxBuilder:
>> >
>> > return path
>> >
>> > - def __init__(self, builddir, verbose=False, n_jobs=None):
>> > + def __init__(self, builddir, venv=None, verbose=False, n_jobs=None):
>> > """Initialize internal variables"""
>> > + self.venv = venv
>> > self.verbose = None
>> >
>> > #
>> > @@ -195,6 +197,21 @@ class SphinxBuilder:
>> >
>> > self.env = os.environ.copy()
>> >
>> > + #
>> > + # If venv command line argument is specified, run Sphinx from venv
>> > + #
>> > + if venv:
>> > + bin_dir = os.path.join(venv, "bin")
>> > + if not os.path.isfile(os.path.join(bin_dir, "activate")):
>> > + sys.exit(f"Venv {venv} not found.")
>> > +
>> > + # "activate" virtual env
>> > + self.env["PATH"] = bin_dir + ":" + self.env["PATH"]
>> > + self.env["VIRTUAL_ENV"] = venv
>> > + if "PYTHONHOME" in self.env:
>> > + del self.env["PYTHONHOME"]
>> > + print(f"Setting venv to {venv}")
>> > +
>> > def run_sphinx(self, sphinx_build, build_args, *args, **pwargs):
>> > """
>> > Executes sphinx-build using current python3 command and setting
>> > @@ -209,7 +226,10 @@ class SphinxBuilder:
>> >
>> > cmd = []
>> >
>> > - cmd.append(sys.executable)
>> > + if self.venv:
>> > + cmd.append("python")
>> > + else:
>> > + cmd.append(sys.executable)
>> >
>> > cmd.append(sphinx_build)
>> >
>> > @@ -533,11 +553,15 @@ def main():
>> > parser.add_argument('-j', '--jobs', type=jobs_type,
>> > help="Sets number of jobs to use with sphinx-build")
>> >
>> > + parser.add_argument("-V", "--venv", nargs='?', const=f'{VENV_DEFAULT}',
>> > + default=None,
>> > + help=f'If used, run Sphinx from a venv dir (default dir: {VENV_DEFAULT})')
>> > +
>> > args = parser.parse_args()
>> >
>> > PythonVersion.check_python(MIN_PYTHON_VERSION)
>> >
>> > - builder = SphinxBuilder(builddir=args.builddir,
>> > + builder = SphinxBuilder(builddir=args.builddir, venv=args.venv,
>> > verbose=args.verbose, n_jobs=args.jobs)
>> >
>> > builder.build(args.target, sphinxdirs=args.sphinxdirs, conf=args.conf,
>>
>
>
>
> Thanks,
> Mauro
>
--
Jani Nikula, Intel
On Fri, Sep 12, 2025 at 12:22:42PM +0300, Jani Nikula wrote:
> On Fri, 12 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> > Em Wed, 10 Sep 2025 13:51:40 +0300
> > Jani Nikula <jani.nikula@linux.intel.com> escreveu:
> >
> >> On Thu, 04 Sep 2025, Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> >> > Sometimes, it is desired to run Sphinx from a virtual environment.
> >> > Add a command line parameter to automatically build Sphinx from
> >> > such environment.
> >>
> >> Why?
> >
> > In my case, to be able to test build with different Sphinx versions.
> > On some distros, only venv works.
>
> I mean why add the complexity of running inside a venv in the wrapper.
I don't think it venv support is complex.
>
> >> If you want Sphinx from a virtual environment, you enter the
> >> environment, and run the regular build, with sphinx-build from the PATH
> >> that points at the venv.
> >
> > when you do that, ./scripts/spdxcheck.py breaks, affecting checkpatch.
>
> Then you could turn the whole argument around, and say spdxcheck.py
> should jump through venv and dependency hoops instead of the docs build.
Neither spdxcheck.py nor checkpatch recommends venv; make docs targets do.
> The point is, it should be the user's responsibility to deal with the
> environment and the dependencies.
>
> If they're setting up a virtual environment, and it affects checkpatch,
> then they should also include the spdxcheck.py dependencies in the
> virtual environment.
They are because we're recommending it.
> This feels like reinventing pipx in a Sphinx wrapper.
>
> We should *reduce* the complexity, not increase it.
Complexity is not the issue: There are several things a the Kernel
tree that are complex. Here, the entire wrapper script (not counting
blank lines/comments) is not more than ~300 lines of code, splitted
on multiple functions. This is not complex.
The big issue is what we have now where docs makefile is full of
hacks:
- scripts to workaround issues;
- "|| exit" to fix broken latexmk/xelatex error outputs;
- "+" to use GNU make parallelism;
- complex call macros;
- ...
None of those documented.
Liking or not, this series as a whole makes a lot clearer what
is done to build preparation, Sphinx build and post-build steps
that are required to produce Kernel docs. Also, almost half of
it is documentation. IMHO, a lot better from what we have so
far.
--
Thanks,
Mauro
© 2016 - 2025 Red Hat, Inc.