From nobody Sat Oct 4 00:31:18 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E7E433126D0; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872389; cv=none; b=Wv4g0PdZC8QnYF9s5BUyy556FdFjLjTpoqPbKtrfAXLHQ0xB+ZO1U4cqsQC1zQ/uUPAqLEQ0MEcPRpBJyL6JnVJktYB2kdVgI5K5OwiI0pegJzEpBilDV258FBVzEXbBaP4h1tJzylvi1JaJs1ff/0W6Vk9Wh3Q2egW/nD5Fqcc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872389; c=relaxed/simple; bh=bIP/wRP3b422VCDjvrgxR9/Zg0ckPnkkNpui1ZjXpUU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DGAFGhNz1KwvxbYzZovZkrtv75lBxIoBZA/1cv9EOlNWx8MX9UPkzCDP1iSZ0ml8T5URCiSaHAGPWUsLmGupRXh2KaL+jl/m9KGDvieh6ojcsKLL/F1/YkwXiaSp26JAfEey7pNio+4PZYb/PTKoXRzC8rrZg4OTgh4AAz8v8IQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=AltfQXcs; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="AltfQXcs" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 71AB1C2BD01; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=bIP/wRP3b422VCDjvrgxR9/Zg0ckPnkkNpui1ZjXpUU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AltfQXcsWcqXeCL5zo+KzuqiIWeX1U+NQjsEqTgzJXkLe+dx+7GJznuVEH+IVGUnW W649ffBV7iJZe2d7pfYmOqrBZH4DSFsjk8t645yNDtfW1Z2EoH7/A1ZVy5Qb6xSWzg EbLl/DQNne0x7Xtya3hxLDVinkSFPgDsN3mDr2hCPgjGSzMpeM/a1QgnlmpdcuaPia uHP3IUgWYrbybESt4/vud9zEPeVDV3Rao67o8s6E1qWtpfTRlhgQGASr5IzR+1CSEW H+brM+rOGAwOdkUAFxmIhEvCOdU/ce3RRKmxUTGVSLYK7TFZ2HZ/EGtBzCsMj1mdS5 lh4tWumW4qPuw== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCsO-3068; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , "Mauro Carvalho Chehab" , Alex Gaynor , Alice Ryhl , Andreas Hindborg , Benjamin Gaignard , Benno Lossin , Boqun Feng , Danilo Krummrich , Erling Ljunggren , Gary Guo , Hans Verkuil , Hans de Goede , Miguel Ojeda , Ricardo Ribalda , Sean Young , Trevor Gross , Yunke Cao , linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, rust-for-linux@vger.kernel.org Subject: [PATCH v2 23/24] scripts: sphinx-build-wrapper: get rid of uapi/media Makefile Date: Fri, 22 Aug 2025 16:19:35 +0200 Message-ID: <5dbb257a4b283697271c9c7b8f4713857e8191c8.1755872208.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Now that kernel-include directive supports parsing data structs directly, we can finally get rid of the horrible hack we added to support parsing media uAPI symbols. As a side effect, Documentation/output doesn't have anymore media auto-generated .rst files on it. Signed-off-by: Mauro Carvalho Chehab --- Documentation/Makefile | 3 +- Documentation/userspace-api/media/Makefile | 64 -- .../userspace-api/media/cec/cec-header.rst | 5 +- .../media/{ =3D> cec}/cec.h.rst.exceptions | 0 .../media/{ =3D> dvb}/ca.h.rst.exceptions | 0 .../media/{ =3D> dvb}/dmx.h.rst.exceptions | 0 .../media/{ =3D> dvb}/frontend.h.rst.exceptions | 0 .../userspace-api/media/dvb/headers.rst | 17 +- .../media/{ =3D> dvb}/net.h.rst.exceptions | 0 .../media/mediactl/media-header.rst | 5 +- .../{ =3D> mediactl}/media.h.rst.exceptions | 0 .../userspace-api/media/rc/lirc-header.rst | 4 +- .../media/{ =3D> rc}/lirc.h.rst.exceptions | 0 .../userspace-api/media/v4l/videodev.rst | 4 +- .../{ =3D> v4l}/videodev2.h.rst.exceptions | 0 scripts/sphinx-build-wrapper | 719 ++++++++++++++++++ 16 files changed, 745 insertions(+), 76 deletions(-) delete mode 100644 Documentation/userspace-api/media/Makefile rename Documentation/userspace-api/media/{ =3D> cec}/cec.h.rst.exceptions = (100%) rename Documentation/userspace-api/media/{ =3D> dvb}/ca.h.rst.exceptions (= 100%) rename Documentation/userspace-api/media/{ =3D> dvb}/dmx.h.rst.exceptions = (100%) rename Documentation/userspace-api/media/{ =3D> dvb}/frontend.h.rst.except= ions (100%) rename Documentation/userspace-api/media/{ =3D> dvb}/net.h.rst.exceptions = (100%) rename Documentation/userspace-api/media/{ =3D> mediactl}/media.h.rst.exce= ptions (100%) rename Documentation/userspace-api/media/{ =3D> rc}/lirc.h.rst.exceptions = (100%) rename Documentation/userspace-api/media/{ =3D> v4l}/videodev2.h.rst.excep= tions (100%) create mode 100755 scripts/sphinx-build-wrapper diff --git a/Documentation/Makefile b/Documentation/Makefile index 2ed334971acd..5c20c68be89a 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -87,7 +87,7 @@ loop_cmd =3D $(echo-cmd) $(cmd_$(1)) || exit; PYTHONPYCACHEPREFIX ?=3D $(abspath $(BUILDDIR)/__pycache__) =20 quiet_cmd_sphinx =3D SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4) - cmd_sphinx =3D $(MAKE) BUILDDIR=3D$(abspath $(BUILDDIR)) $(build)=3D= Documentation/userspace-api/media $2 && \ + cmd_sphinx =3D \ PYTHONPYCACHEPREFIX=3D"$(PYTHONPYCACHEPREFIX)" \ BUILDDIR=3D$(abspath $(BUILDDIR)) SPHINX_CONF=3D$(abspath $(src)/$5/$(SPH= INX_CONF)) \ $(PYTHON3) $(srctree)/scripts/jobserver-exec \ @@ -171,7 +171,6 @@ refcheckdocs: =20 cleandocs: $(Q)rm -rf $(BUILDDIR) - $(Q)$(MAKE) BUILDDIR=3D$(abspath $(BUILDDIR)) $(build)=3DDocumentation/us= erspace-api/media clean =20 dochelp: @echo ' Linux kernel internal documentation in different formats from Re= ST:' diff --git a/Documentation/userspace-api/media/Makefile b/Documentation/use= rspace-api/media/Makefile deleted file mode 100644 index accc734d045a..000000000000 --- a/Documentation/userspace-api/media/Makefile +++ /dev/null @@ -1,64 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0 - -# Rules to convert a .h file to inline RST documentation - -SRC_DIR=3D$(srctree)/Documentation/userspace-api/media -PARSER =3D $(srctree)/tools/docs/parse-headers.py -UAPI =3D $(srctree)/include/uapi/linux -KAPI =3D $(srctree)/include/linux - -FILES =3D ca.h.rst dmx.h.rst frontend.h.rst net.h.rst \ - videodev2.h.rst media.h.rst cec.h.rst lirc.h.rst - -TARGETS :=3D $(addprefix $(BUILDDIR)/, $(FILES)) - -gen_rst =3D \ - echo ${PARSER} $< $@ $(SRC_DIR)/$(notdir $@).exceptions; \ - ${PARSER} $< $@ $(SRC_DIR)/$(notdir $@).exceptions - -quiet_gen_rst =3D echo ' PARSE $(patsubst $(srctree)/%,%,$<)'; \ - ${PARSER} $< $@ $(SRC_DIR)/$(notdir $@).exceptions - -silent_gen_rst =3D ${gen_rst} - -$(BUILDDIR)/ca.h.rst: ${UAPI}/dvb/ca.h ${PARSER} $(SRC_DIR)/ca.h.rst.excep= tions - @$($(quiet)gen_rst) - -$(BUILDDIR)/dmx.h.rst: ${UAPI}/dvb/dmx.h ${PARSER} $(SRC_DIR)/dmx.h.rst.ex= ceptions - @$($(quiet)gen_rst) - -$(BUILDDIR)/frontend.h.rst: ${UAPI}/dvb/frontend.h ${PARSER} $(SRC_DIR)/fr= ontend.h.rst.exceptions - @$($(quiet)gen_rst) - -$(BUILDDIR)/net.h.rst: ${UAPI}/dvb/net.h ${PARSER} $(SRC_DIR)/net.h.rst.ex= ceptions - @$($(quiet)gen_rst) - -$(BUILDDIR)/videodev2.h.rst: ${UAPI}/videodev2.h ${PARSER} $(SRC_DIR)/vide= odev2.h.rst.exceptions - @$($(quiet)gen_rst) - -$(BUILDDIR)/media.h.rst: ${UAPI}/media.h ${PARSER} $(SRC_DIR)/media.h.rst.= exceptions - @$($(quiet)gen_rst) - -$(BUILDDIR)/cec.h.rst: ${UAPI}/cec.h ${PARSER} $(SRC_DIR)/cec.h.rst.except= ions - @$($(quiet)gen_rst) - -$(BUILDDIR)/lirc.h.rst: ${UAPI}/lirc.h ${PARSER} $(SRC_DIR)/lirc.h.rst.exc= eptions - @$($(quiet)gen_rst) - -# Media build rules - -.PHONY: all html texinfo epub xml latex - -all: $(IMGDOT) $(BUILDDIR) ${TARGETS} -html: all -texinfo: all -epub: all -xml: all -latex: $(IMGPDF) all -linkcheck: - -clean: - -rm -f $(DOTTGT) $(IMGTGT) ${TARGETS} 2>/dev/null - -$(BUILDDIR): - $(Q)mkdir -p $@ diff --git a/Documentation/userspace-api/media/cec/cec-header.rst b/Documen= tation/userspace-api/media/cec/cec-header.rst index d70736ac2b1d..f67003bb8740 100644 --- a/Documentation/userspace-api/media/cec/cec-header.rst +++ b/Documentation/userspace-api/media/cec/cec-header.rst @@ -6,5 +6,6 @@ CEC Header File *************** =20 -.. kernel-include:: $BUILDDIR/cec.h.rst - +.. kernel-include:: include/uapi/linux/cec.h + :generate-cross-refs: + :exception-file: cec.h.rst.exceptions diff --git a/Documentation/userspace-api/media/cec.h.rst.exceptions b/Docum= entation/userspace-api/media/cec/cec.h.rst.exceptions similarity index 100% rename from Documentation/userspace-api/media/cec.h.rst.exceptions rename to Documentation/userspace-api/media/cec/cec.h.rst.exceptions diff --git a/Documentation/userspace-api/media/ca.h.rst.exceptions b/Docume= ntation/userspace-api/media/dvb/ca.h.rst.exceptions similarity index 100% rename from Documentation/userspace-api/media/ca.h.rst.exceptions rename to Documentation/userspace-api/media/dvb/ca.h.rst.exceptions diff --git a/Documentation/userspace-api/media/dmx.h.rst.exceptions b/Docum= entation/userspace-api/media/dvb/dmx.h.rst.exceptions similarity index 100% rename from Documentation/userspace-api/media/dmx.h.rst.exceptions rename to Documentation/userspace-api/media/dvb/dmx.h.rst.exceptions diff --git a/Documentation/userspace-api/media/frontend.h.rst.exceptions b/= Documentation/userspace-api/media/dvb/frontend.h.rst.exceptions similarity index 100% rename from Documentation/userspace-api/media/frontend.h.rst.exceptions rename to Documentation/userspace-api/media/dvb/frontend.h.rst.exceptions diff --git a/Documentation/userspace-api/media/dvb/headers.rst b/Documentat= ion/userspace-api/media/dvb/headers.rst index 88c3eb33a89e..c75f64cf21d5 100644 --- a/Documentation/userspace-api/media/dvb/headers.rst +++ b/Documentation/userspace-api/media/dvb/headers.rst @@ -7,10 +7,19 @@ Digital TV uAPI header files Digital TV uAPI headers *********************** =20 -.. kernel-include:: $BUILDDIR/frontend.h.rst +.. kernel-include:: include/uapi/linux/dvb/frontend.h + :generate-cross-refs: + :exception-file: frontend.h.rst.exceptions =20 -.. kernel-include:: $BUILDDIR/dmx.h.rst +.. kernel-include:: include/uapi/linux/dvb/dmx.h + :generate-cross-refs: + :exception-file: dmx.h.rst.exceptions =20 -.. kernel-include:: $BUILDDIR/ca.h.rst +.. kernel-include:: include/uapi/linux/dvb/ca.h + :generate-cross-refs: + :exception-file: ca.h.rst.exceptions + +.. kernel-include:: include/uapi/linux/dvb/net.h + :generate-cross-refs: + :exception-file: net.h.rst.exceptions =20 -.. kernel-include:: $BUILDDIR/net.h.rst diff --git a/Documentation/userspace-api/media/net.h.rst.exceptions b/Docum= entation/userspace-api/media/dvb/net.h.rst.exceptions similarity index 100% rename from Documentation/userspace-api/media/net.h.rst.exceptions rename to Documentation/userspace-api/media/dvb/net.h.rst.exceptions diff --git a/Documentation/userspace-api/media/mediactl/media-header.rst b/= Documentation/userspace-api/media/mediactl/media-header.rst index c674271c93f5..d561d2845f3d 100644 --- a/Documentation/userspace-api/media/mediactl/media-header.rst +++ b/Documentation/userspace-api/media/mediactl/media-header.rst @@ -6,5 +6,6 @@ Media Controller Header File **************************** =20 -.. kernel-include:: $BUILDDIR/media.h.rst - +.. kernel-include:: include/uapi/linux/media.h + :generate-cross-refs: + :exception-file: media.h.rst.exceptions diff --git a/Documentation/userspace-api/media/media.h.rst.exceptions b/Doc= umentation/userspace-api/media/mediactl/media.h.rst.exceptions similarity index 100% rename from Documentation/userspace-api/media/media.h.rst.exceptions rename to Documentation/userspace-api/media/mediactl/media.h.rst.exceptions diff --git a/Documentation/userspace-api/media/rc/lirc-header.rst b/Documen= tation/userspace-api/media/rc/lirc-header.rst index 54cb40b8a065..a53328327847 100644 --- a/Documentation/userspace-api/media/rc/lirc-header.rst +++ b/Documentation/userspace-api/media/rc/lirc-header.rst @@ -6,5 +6,7 @@ LIRC Header File **************** =20 -.. kernel-include:: $BUILDDIR/lirc.h.rst +.. kernel-include:: include/uapi/linux/lirc.h + :generate-cross-refs: + :exception-file: lirc.h.rst.exceptions =20 diff --git a/Documentation/userspace-api/media/lirc.h.rst.exceptions b/Docu= mentation/userspace-api/media/rc/lirc.h.rst.exceptions similarity index 100% rename from Documentation/userspace-api/media/lirc.h.rst.exceptions rename to Documentation/userspace-api/media/rc/lirc.h.rst.exceptions diff --git a/Documentation/userspace-api/media/v4l/videodev.rst b/Documenta= tion/userspace-api/media/v4l/videodev.rst index c866fec417eb..cde485bc9a5f 100644 --- a/Documentation/userspace-api/media/v4l/videodev.rst +++ b/Documentation/userspace-api/media/v4l/videodev.rst @@ -6,4 +6,6 @@ Video For Linux Two Header File ******************************* =20 -.. kernel-include:: $BUILDDIR/videodev2.h.rst +.. kernel-include:: include/uapi/linux/videodev2.h + :generate-cross-refs: + :exception-file: videodev2.h.rst.exceptions diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b= /Documentation/userspace-api/media/v4l/videodev2.h.rst.exceptions similarity index 100% rename from Documentation/userspace-api/media/videodev2.h.rst.exceptions rename to Documentation/userspace-api/media/v4l/videodev2.h.rst.exceptions diff --git a/scripts/sphinx-build-wrapper b/scripts/sphinx-build-wrapper new file mode 100755 index 000000000000..abe8c26ae137 --- /dev/null +++ b/scripts/sphinx-build-wrapper @@ -0,0 +1,719 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2025 Mauro Carvalho Chehab +# +# pylint: disable=3DR0902, R0912, R0913, R0914, R0915, R0917, C0103 +# +# Converted from docs Makefile and parallel-wrapper.sh, both under +# GPLv2, copyrighted since 2008 by the following authors: +# +# Akira Yokosawa +# Arnd Bergmann +# Breno Leitao +# Carlos Bilbao +# Dave Young +# Donald Hunter +# Geert Uytterhoeven +# Jani Nikula +# Jan Stancek +# Jonathan Corbet +# Joshua Clayton +# Kees Cook +# Linus Torvalds +# Magnus Damm +# Masahiro Yamada +# Mauro Carvalho Chehab +# Maxim Cournoyer +# Peter Foley +# Randy Dunlap +# Rob Herring +# Shuah Khan +# Thorsten Blum +# Tomas Winkler + + +""" +Sphinx build wrapper that handles Kernel-specific business rules: + +- it gets the Kernel build environment vars; +- it determines what's the best parallelism; +- it handles SPHINXDIRS + +This tool ensures that MIN_PYTHON_VERSION is satisfied. If version is +below that, it seeks for a new Python version. If found, it re-runs using +the newer version. +""" + +import argparse +import locale +import os +import re +import shlex +import shutil +import subprocess +import sys + +from concurrent import futures +from glob import glob + +LIB_DIR =3D "lib" +SRC_DIR =3D os.path.dirname(os.path.realpath(__file__)) + +sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR)) + +from jobserver import JobserverExec # pylint: disable=3DC= 0413 + + +def parse_version(version): + """Convert a major.minor.patch version into a tuple""" + return tuple(int(x) for x in version.split(".")) + +def ver_str(version): + """Returns a version tuple as major.minor.patch""" + + return ".".join([str(x) for x in version]) + +# Minimal supported Python version needed by Sphinx and its extensions +MIN_PYTHON_VERSION =3D parse_version("3.7") + +# Default value for --venv parameter +VENV_DEFAULT =3D "sphinx_latest" + +# List of make targets and its corresponding builder and output directory +TARGETS =3D { + "cleandocs": { + "builder": "clean", + }, + "htmldocs": { + "builder": "html", + }, + "epubdocs": { + "builder": "epub", + "out_dir": "epub", + }, + "texinfodocs": { + "builder": "texinfo", + "out_dir": "texinfo", + }, + "infodocs": { + "builder": "texinfo", + "out_dir": "texinfo", + }, + "latexdocs": { + "builder": "latex", + "out_dir": "latex", + }, + "pdfdocs": { + "builder": "latex", + "out_dir": "latex", + }, + "xmldocs": { + "builder": "xml", + "out_dir": "xml", + }, + "linkcheckdocs": { + "builder": "linkcheck" + }, +} + +# Paper sizes. An empty value will pick the default +PAPER =3D ["", "a4", "letter"] + +class SphinxBuilder: + """ + Handles a sphinx-build target, adding needed arguments to build + with the Kernel. + """ + + def is_rust_enabled(self): + """Check if rust is enabled at .config""" + config_path =3D os.path.join(self.srctree, ".config") + if os.path.isfile(config_path): + with open(config_path, "r", encoding=3D"utf-8") as f: + return "CONFIG_RUST=3Dy" in f.read() + return False + + def get_path(self, path, abs_path=3DFalse): + """ + Ancillary routine to handle patches the right way, as shell does. + + It first expands "~" and "~user". Then, if patch is not absolute, + join self.srctree. Finally, if requested, convert to abspath. + """ + + path =3D os.path.expanduser(path) + if not path.startswith("/"): + path =3D os.path.join(self.srctree, path) + + if abs_path: + return os.path.abspath(path) + + return path + + def __init__(self, venv=3DNone, verbose=3DFalse, n_jobs=3DNone, intera= ctive=3DNone): + """Initialize internal variables""" + self.venv =3D venv + self.verbose =3D None + + # Normal variables passed from Kernel's makefile + self.kernelversion =3D os.environ.get("KERNELVERSION", "unknown") + self.kernelrelease =3D os.environ.get("KERNELRELEASE", "unknown") + self.pdflatex =3D os.environ.get("PDFLATEX", "xelatex") + + if not interactive: + self.latexopts =3D os.environ.get("LATEXOPTS", "-interaction= =3Dbatchmode -no-shell-escape") + else: + self.latexopts =3D os.environ.get("LATEXOPTS", "") + + if not verbose: + verbose =3D bool(os.environ.get("KBUILD_VERBOSE", "") !=3D "") + + # Handle SPHINXOPTS evironment + sphinxopts =3D shlex.split(os.environ.get("SPHINXOPTS", "")) + + # As we handle number of jobs and quiet in separate, we need to pi= ck + # it the same way as sphinx-build would pick, so let's use argparse + # do to the right argument expansion + parser =3D argparse.ArgumentParser() + parser.add_argument('-j', '--jobs', type=3Dint) + parser.add_argument('-q', '--quiet', type=3Dint) + + # Other sphinx-build arguments go as-is, so place them + # at self.sphinxopts + sphinx_args, self.sphinxopts =3D parser.parse_known_args(sphinxopt= s) + if sphinx_args.quiet =3D=3D True: + self.verbose =3D False + + if sphinx_args.jobs: + self.n_jobs =3D sphinx_args.jobs + + # Command line arguments was passed, override SPHINXOPTS + if verbose is not None: + self.verbose =3D verbose + + self.n_jobs =3D n_jobs + + # Source tree directory. This needs to be at os.environ, as + # Sphinx extensions and media uAPI makefile needs it + self.srctree =3D os.environ.get("srctree") + if not self.srctree: + self.srctree =3D "." + os.environ["srctree"] =3D self.srctree + + # Now that we can expand srctree, get other directories as well + self.sphinxbuild =3D os.environ.get("SPHINXBUILD", "sphinx-build") + self.kerneldoc =3D self.get_path(os.environ.get("KERNELDOC", + "scripts/kernel-doc.= py")) + self.obj =3D os.environ.get("obj", "Documentation") + self.builddir =3D self.get_path(os.path.join(self.obj, "output"), + abs_path=3DTrue) + + # Media uAPI needs it + os.environ["BUILDDIR"] =3D self.builddir + + # Detect if rust is enabled + self.config_rust =3D self.is_rust_enabled() + + # Get directory locations for LaTeX build toolchain + self.pdflatex_cmd =3D shutil.which(self.pdflatex) + self.latexmk_cmd =3D shutil.which("latexmk") + + self.env =3D os.environ.copy() + + # If venv parameter is specified, run Sphinx from venv + if venv: + bin_dir =3D os.path.join(venv, "bin") + if os.path.isfile(os.path.join(bin_dir, "activate")): + # "activate" virtual env + self.env["PATH"] =3D bin_dir + ":" + self.env["PATH"] + self.env["VIRTUAL_ENV"] =3D venv + if "PYTHONHOME" in self.env: + del self.env["PYTHONHOME"] + print(f"Setting venv to {venv}") + else: + sys.exit(f"Venv {venv} not found.") + + def run_sphinx(self, sphinx_build, build_args, *args, **pwargs): + """ + Executes sphinx-build using current python3 command and setting + -j parameter if possible to run the build in parallel. + """ + + with JobserverExec() as jobserver: + if jobserver.claim: + n_jobs =3D str(jobserver.claim) + else: + n_jobs =3D "auto" # Supported since Sphinx 1.7 + + cmd =3D [] + + if self.venv: + cmd.append("python") + else: + cmd.append(sys.executable) + + cmd.append(sphinx_build) + + # if present, SPHINXOPTS or command line --jobs overrides defa= ult + if self.n_jobs: + n_jobs =3D str(self.n_jobs) + + if n_jobs: + cmd +=3D [f"-j{n_jobs}"] + + if not self.verbose: + cmd.append("-q") + + cmd +=3D self.sphinxopts + + cmd +=3D build_args + + if self.verbose: + print(" ".join(cmd)) + + rc =3D subprocess.call(cmd, *args, **pwargs) + + def handle_html(self, css, output_dir): + """ + Extra steps for HTML and epub output. + + For such targets, we need to ensure that CSS will be properly + copied to the output _static directory + """ + + if not css: + return + + css =3D os.path.expanduser(css) + if not css.startswith("/"): + css =3D os.path.join(self.srctree, css) + + static_dir =3D os.path.join(output_dir, "_static") + os.makedirs(static_dir, exist_ok=3DTrue) + + try: + shutil.copy2(css, static_dir) + except (OSError, IOError) as e: + print(f"Warning: Failed to copy CSS: {e}", file=3Dsys.stderr) + + def build_pdf_file(self, latex_cmd, from_dir, path): + """Builds a single pdf file using latex_cmd""" + try: + subprocess.run(latex_cmd + [path], + cwd=3Dfrom_dir, check=3DTrue) + + return True + except subprocess.CalledProcessError: + # LaTeX PDF error code is almost useless: it returns + # error codes even when build succeeds but has warnings. + # So, we'll ignore the results + return False + + def pdf_parallel_build(self, tex_suffix, latex_cmd, tex_files, n_jobs): + """Build PDF files in parallel if possible""" + builds =3D {} + build_failed =3D False + max_len =3D 0 + has_tex =3D False + + # Process files in parallel + with futures.ThreadPoolExecutor(max_workers=3Dn_jobs) as executor: + jobs =3D {} + + for from_dir, pdf_dir, entry in tex_files: + name =3D entry.name + + if not name.endswith(tex_suffix): + continue + + name =3D name[:-len(tex_suffix)] + + max_len =3D max(max_len, len(name)) + + has_tex =3D True + + future =3D executor.submit(self.build_pdf_file, latex_cmd, + from_dir, entry.path) + jobs[future] =3D (from_dir, name, entry.path) + + for future in futures.as_completed(jobs): + from_dir, name, path =3D jobs[future] + + pdf_name =3D name + ".pdf" + pdf_from =3D os.path.join(from_dir, pdf_name) + + try: + success =3D future.result() + + if success and os.path.exists(pdf_from): + pdf_to =3D os.path.join(pdf_dir, pdf_name) + + os.rename(pdf_from, pdf_to) + builds[name] =3D os.path.relpath(pdf_to, self.buil= ddir) + else: + builds[name] =3D "FAILED" + build_failed =3D True + except Exception as e: + builds[name] =3D f"FAILED ({str(e)})" + build_failed =3D True + + # Handle case where no .tex files were found + if not has_tex: + name =3D "Sphinx LaTeX builder" + max_len =3D max(max_len, len(name)) + builds[name] =3D "FAILED (no .tex file was generated)" + build_failed =3D True + + return builds, build_failed, max_len + + def handle_pdf(self, output_dirs): + """ + Extra steps for PDF output. + + As PDF is handled via a LaTeX output, after building the .tex file, + a new build is needed to create the PDF output from the latex + directory. + """ + builds =3D {} + max_len =3D 0 + tex_suffix =3D ".tex" + + # Get all tex files that will be used for PDF build + tex_files =3D [] + for from_dir in output_dirs: + pdf_dir =3D os.path.join(from_dir, "../pdf") + os.makedirs(pdf_dir, exist_ok=3DTrue) + + if self.latexmk_cmd: + latex_cmd =3D [self.latexmk_cmd, f"-{self.pdflatex}"] + else: + latex_cmd =3D [self.pdflatex] + + latex_cmd.extend(shlex.split(self.latexopts)) + + # Get a list of tex files to process + with os.scandir(from_dir) as it: + for entry in it: + if entry.name.endswith(tex_suffix): + tex_files.append((from_dir, pdf_dir, entry)) + + # When using make, this won't be used, as the number of jobs comes + # from POSIX jobserver. So, this covers the case where build comes + # from command line. On such case, serialize by default, except if + # the user explicitly sets the number of jobs. + n_jobs =3D 1 + + # n_jobs is either an integer or "auto". Only use it if it is a nu= mber + if self.n_jobs: + try: + n_jobs =3D int(self.n_jobs) + except ValueError: + pass + + # When using make, jobserver.claim is the number of jobs that were + # used with "-j" and that aren't used by other make targets + with JobserverExec() as jobserver: + n_jobs =3D 1 + + # Handle the case when a parameter is passed via command line, + # using it as default, if jobserver doesn't claim anything + if self.n_jobs: + try: + n_jobs =3D int(self.n_jobs) + except ValueError: + pass + + if jobserver.claim: + n_jobs =3D jobserver.claim + + # Build files in parallel + builds, build_failed, max_len =3D self.pdf_parallel_build(tex_= suffix, + latex_= cmd, + tex_fi= les, + n_jobs) + + msg =3D "Summary" + msg +=3D "\n" + "=3D" * len(msg) + print() + print(msg) + + for pdf_name, pdf_file in builds.items(): + print(f"{pdf_name:<{max_len}}: {pdf_file}") + + print() + + # return an error if a PDF file is missing + + if build_failed: + sys.exit(f"PDF build failed: not all PDF files were created.") + else: + print("All PDF files were built.") + + def handle_info(self, output_dirs): + """ + Extra steps for Info output. + + For texinfo generation, an additional make is needed from the + texinfo directory. + """ + + for output_dir in output_dirs: + try: + subprocess.run(["make", "info"], cwd=3Doutput_dir, check= =3DTrue) + except subprocess.CalledProcessError as e: + sys.exit(f"Error generating info docs: {e}") + + def cleandocs(self, builder): + + shutil.rmtree(self.builddir, ignore_errors=3DTrue) + + def build(self, target, sphinxdirs=3DNone, conf=3D"conf.py", + theme=3DNone, css=3DNone, paper=3DNone): + """ + Build documentation using Sphinx. This is the core function of this + module. It prepares all arguments required by sphinx-build. + """ + + builder =3D TARGETS[target]["builder"] + out_dir =3D TARGETS[target].get("out_dir", "") + + # Cleandocs doesn't require sphinx-build + if target =3D=3D "cleandocs": + self.cleandocs(builder) + return + + # Other targets require sphinx-build + sphinxbuild =3D shutil.which(self.sphinxbuild, path=3Dself.env["PA= TH"]) + if not sphinxbuild: + sys.exit(f"Error: {self.sphinxbuild} not found in PATH.\n") + + if builder =3D=3D "latex": + if not self.pdflatex_cmd and not self.latexmk_cmd: + sys.exit("Error: pdflatex or latexmk required for PDF gene= ration") + + docs_dir =3D os.path.abspath(os.path.join(self.srctree, "Documenta= tion")) + + # Prepare base arguments for Sphinx build + kerneldoc =3D self.kerneldoc + if kerneldoc.startswith(self.srctree): + kerneldoc =3D os.path.relpath(kerneldoc, self.srctree) + + # Prepare common Sphinx options + args =3D [ + "-b", builder, + "-c", docs_dir, + ] + + if builder =3D=3D "latex": + if not paper: + paper =3D PAPER[1] + + args.extend(["-D", f"latex_elements.papersize=3D{paper}paper"]) + + if self.config_rust: + args.extend(["-t", "rustdoc"]) + + if conf: + self.env["SPHINX_CONF"] =3D self.get_path(conf, abs_path=3DTru= e) + + if not sphinxdirs: + sphinxdirs =3D os.environ.get("SPHINXDIRS", ".") + + # The sphinx-build tool has a bug: internally, it tries to set + # locale with locale.setlocale(locale.LC_ALL, ''). This causes a + # crash if language is not set. Detect and fix it. + try: + locale.setlocale(locale.LC_ALL, '') + except Exception: + self.env["LC_ALL"] =3D "C" + self.env["LANG"] =3D "C" + + # sphinxdirs can be a list or a whitespace-separated string + sphinxdirs_list =3D [] + for sphinxdir in sphinxdirs: + if isinstance(sphinxdir, list): + sphinxdirs_list +=3D sphinxdir + else: + for name in sphinxdir.split(" "): + sphinxdirs_list.append(name) + + # Build each directory + output_dirs =3D [] + for sphinxdir in sphinxdirs_list: + src_dir =3D os.path.join(docs_dir, sphinxdir) + doctree_dir =3D os.path.join(self.builddir, ".doctrees") + output_dir =3D os.path.join(self.builddir, sphinxdir, out_dir) + + # Make directory names canonical + src_dir =3D os.path.normpath(src_dir) + doctree_dir =3D os.path.normpath(doctree_dir) + output_dir =3D os.path.normpath(output_dir) + + os.makedirs(doctree_dir, exist_ok=3DTrue) + os.makedirs(output_dir, exist_ok=3DTrue) + + output_dirs.append(output_dir) + + build_args =3D args + [ + "-d", doctree_dir, + "-D", f"kerneldoc_bin=3D{kerneldoc}", + "-D", f"version=3D{self.kernelversion}", + "-D", f"release=3D{self.kernelrelease}", + "-D", f"kerneldoc_srctree=3D{self.srctree}", + src_dir, + output_dir, + ] + + # Execute sphinx-build + try: + self.run_sphinx(sphinxbuild, build_args, env=3Dself.env) + except Exception as e: + sys.exit(f"Build failed: {e}") + + # Ensure that html/epub will have needed static files + if target in ["htmldocs", "epubdocs"]: + self.handle_html(css, output_dir) + + # PDF and Info require a second build step + if target =3D=3D "pdfdocs": + self.handle_pdf(output_dirs) + elif target =3D=3D "infodocs": + self.handle_info(output_dirs) + + @staticmethod + def get_python_version(cmd): + """ + Get python version from a Python binary. As we need to detect if + are out there newer python binaries, we can't rely on sys.release = here. + """ + + result =3D subprocess.run([cmd, "--version"], check=3DTrue, + stdout=3Dsubprocess.PIPE, stderr=3Dsubproc= ess.PIPE, + universal_newlines=3DTrue) + version =3D result.stdout.strip() + + match =3D re.search(r"(\d+\.\d+\.\d+)", version) + if match: + return parse_version(match.group(1)) + + print(f"Can't parse version {version}") + return (0, 0, 0) + + @staticmethod + def find_python(): + """ + Detect if are out there any python 3.xy version newer than the + current one. + + Note: this routine is limited to up to 2 digits for python3. We + may need to update it one day, hopefully on a distant future. + """ + patterns =3D [ + "python3.[0-9]", + "python3.[0-9][0-9]", + ] + + # Seek for a python binary newer than MIN_PYTHON_VERSION + for path in os.getenv("PATH", "").split(":"): + for pattern in patterns: + for cmd in glob(os.path.join(path, pattern)): + if os.path.isfile(cmd) and os.access(cmd, os.X_OK): + version =3D SphinxBuilder.get_python_version(cmd) + if version >=3D MIN_PYTHON_VERSION: + return cmd + + return None + + @staticmethod + def check_python(): + """ + Check if the current python binary satisfies our minimal requireme= nt + for Sphinx build. If not, re-run with a newer version if found. + """ + cur_ver =3D sys.version_info[:3] + if cur_ver >=3D MIN_PYTHON_VERSION: + return + + python_ver =3D ver_str(cur_ver) + + new_python_cmd =3D SphinxBuilder.find_python() + if not new_python_cmd: + sys.exit(f"Python version {python_ver} is not supported anymor= e.") + + # Restart script using the newer version + script_path =3D os.path.abspath(sys.argv[0]) + args =3D [new_python_cmd, script_path] + sys.argv[1:] + + print(f"Python {python_ver} not supported. Changing to {new_python= _cmd}") + + try: + os.execv(new_python_cmd, args) + except OSError as e: + sys.exit(f"Failed to restart with {new_python_cmd}: {e}") + +def jobs_type(value): + """ + Handle valid values for -j. Accepts Sphinx "-jauto", plus a number + equal or bigger than one. + """ + if value is None: + return None + + if value.lower() =3D=3D 'auto': + return value.lower() + + try: + if int(value) >=3D 1: + return value + + raise argparse.ArgumentTypeError(f"Minimum jobs is 1, got {value}") + except ValueError: + raise argparse.ArgumentTypeError(f"Must be 'auto' or positive inte= ger, got {value}") + +def main(): + """ + Main function. The only mandatory argument is the target. If not + specified, the other arguments will use default values if not + specified at os.environ. + """ + parser =3D argparse.ArgumentParser(description=3D"Kernel documentation= builder") + + parser.add_argument("target", choices=3Dlist(TARGETS.keys()), + help=3D"Documentation target to build") + parser.add_argument("--sphinxdirs", nargs=3D"+", + help=3D"Specific directories to build") + parser.add_argument("--conf", default=3D"conf.py", + help=3D"Sphinx configuration file") + + parser.add_argument("--theme", help=3D"Sphinx theme to use") + + parser.add_argument("--css", help=3D"Custom CSS file for HTML/EPUB") + + parser.add_argument("--paper", choices=3DPAPER, default=3DPAPER[0], + help=3D"Paper size for LaTeX/PDF output") + + parser.add_argument("-v", "--verbose", action=3D'store_true', + help=3D"place build in verbose mode") + + parser.add_argument('-j', '--jobs', type=3Djobs_type, + help=3D"Sets number of jobs to use with sphinx-bui= ld") + + parser.add_argument('-i', '--interactive', action=3D'store_true', + help=3D"Change latex default to run in interactive= mode") + + parser.add_argument("-V", "--venv", nargs=3D'?', const=3Df'{VENV_DEFAU= LT}', + default=3DNone, + help=3Df'If used, run Sphinx from a venv dir (defa= ult dir: {VENV_DEFAULT})') + + args =3D parser.parse_args() + + SphinxBuilder.check_python() + + builder =3D SphinxBuilder(venv=3Dargs.venv, verbose=3Dargs.verbose, + n_jobs=3Dargs.jobs, interactive=3Dargs.interac= tive) + + builder.build(args.target, sphinxdirs=3Dargs.sphinxdirs, conf=3Dargs.c= onf, + theme=3Dargs.theme, css=3Dargs.css, paper=3Dargs.paper) + +if __name__ =3D=3D "__main__": + main() --=20 2.50.1