From nobody Fri Oct 3 20:22:42 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C4596301028; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; cv=none; b=mcmOxwe7ysUcrghps9XLoFlm5z1scgHBhLx9mVLEyCIIubrl7ljNQ+Zl8wqRx5YeAV8dbEJIZ1rRqzWxNLfQIuerFrwbgkGgeFOzvSHpZiyvK6Rq3AT0qtkJxSxAC52o18ty0HRQKm8GZ461jvwrnFvHIsGS7oQjeiGF5KTd8nM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; c=relaxed/simple; bh=AiE/HI6OagdGD9ZL2nK8qpA5ue5A1dovU9+3kLCIiqY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Xg4bce4QLuCgFxDqTzxoKAx+aubuwyEQibFsxJLVzOnswvxZVpJdUgjpbJSlms/llhl2aFRvS6aDHqu+SOtofbMblXiS40UrKb1Kgd9ZnmV31/hUNuIX9H7hSt9RdrwXgeBWsAtVqm4ca9GWt+R3xs59nvJyJB3Km9RqEWzHXZg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=RBKirJOv; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="RBKirJOv" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 198CCC116D0; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1756139449; bh=AiE/HI6OagdGD9ZL2nK8qpA5ue5A1dovU9+3kLCIiqY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RBKirJOvuIOKMA4LG+uVTfoxUxw0lOHz5quD3LULvoL0kaZu1eksKvWJj/J+UNueY 7kqijbvMTv4i3Z1HA5wC6FOWk9qHD+ee9tGx3r9IxkVhM5qQ/3FfVJOTX0+BnUceR6 584NDnd/HFr5t+gP+9GTzlZsAQSM8yPxDJsqAJwCfVmYKmvd2g9MifP+VGTWEUXoPy S2awwvWzETUxbJVHzGcj/DJHjcg0Qie6kA/8VXNoFn9udATKon2uJG9T2hBKZ9+JEa BZpmuVoAej0b31K0Mtiau4stVaYEVbv1duXF6UQ/Kz73cwjsTU8JK200JWXs9buDRY eWmGfEEPtXD9A== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uqa5n-0000000HALI-0hdR; Mon, 25 Aug 2025 18:30:47 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Jonathan Corbet" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v2 01/13] scripts/jobserver-exec: move the code to a class Date: Mon, 25 Aug 2025 18:30:28 +0200 Message-ID: <9228a9bb11dce3085988d3eac081071945aede94.1756138805.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Convert the code inside jobserver-exec to a class and properly document it. Using a class allows reusing the jobserver logic on other scripts. While the main code remains unchanged, being compatible with Python 2.6 and 3.0+, its coding style now follows a more modern standard, having tabs replaced by a 4-spaces indent, passing autopep8, black and pylint. The code now allows allows using a pythonic way to enter/exit a python code, e.g. it now supports: with JobserverExec() as jobserver: jobserver.run(sys.argv[1:]) With the new code, the __exit__() function should ensure that the jobserver slot will be closed at the end, even if something bad happens somewhere. Signed-off-by: Mauro Carvalho Chehab --- scripts/jobserver-exec | 218 ++++++++++++++++++++++++++++------------- 1 file changed, 151 insertions(+), 67 deletions(-) diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec index 7eca035472d3..b386b1a845de 100755 --- a/scripts/jobserver-exec +++ b/scripts/jobserver-exec @@ -1,77 +1,161 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0+ # +# pylint: disable=3DC0103,C0209 +# # This determines how many parallel tasks "make" is expecting, as it is # not exposed via an special variables, reserves them all, runs a subproce= ss # with PARALLELISM environment variable set, and releases the jobs back ag= ain. # # https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#= POSIX-Jobserver -from __future__ import print_function -import os, sys, errno + +""" +Interacts with the POSIX jobserver during the Kernel build time. + +A "normal" jobserver task, like the one initiated by a make subrocess woul= d do: + + - open read/write file descriptors to communicate with the job server; + - ask for one slot by calling: + claim =3D os.read(reader, 1) + - when the job finshes, call: + os.write(writer, b"+") # os.write(writer, claim) + +Here, the goal is different: This script aims to get the remaining number +of slots available, using all of them to run a command which handle tasks = in +parallel. To to that, it has a loop that ends only after there are no +slots left. It then increments the number by one, in order to allow a +call equivalent to make -j$((claim+1)), e.g. having a parent make creating +$claim child to do the actual work. + +The end goal here is to keep the total number of build tasks under the +limit established by the initial make -j$n_proc call. +""" + +import errno +import os import subprocess +import sys =20 -# Extract and prepare jobserver file descriptors from environment. -claim =3D 0 -jobs =3D b"" -try: - # Fetch the make environment options. - flags =3D os.environ['MAKEFLAGS'] - - # Look for "--jobserver=3DR,W" - # Note that GNU Make has used --jobserver-fds and --jobserver-auth - # so this handles all of them. - opts =3D [x for x in flags.split(" ") if x.startswith("--jobserver")] - - # Parse out R,W file descriptor numbers and set them nonblocking. - # If the MAKEFLAGS variable contains multiple instances of the - # --jobserver-auth=3D option, the last one is relevant. - fds =3D opts[-1].split("=3D", 1)[1] - - # Starting with GNU Make 4.4, named pipes are used for reader and writer. - # Example argument: --jobserver-auth=3Dfifo:/tmp/GMfifo8134 - _, _, path =3D fds.partition('fifo:') - - if path: - reader =3D os.open(path, os.O_RDONLY | os.O_NONBLOCK) - writer =3D os.open(path, os.O_WRONLY) - else: - reader, writer =3D [int(x) for x in fds.split(",", 1)] - # Open a private copy of reader to avoid setting nonblocking - # on an unexpecting process with the same reader fd. - reader =3D os.open("/proc/self/fd/%d" % (reader), - os.O_RDONLY | os.O_NONBLOCK) - - # Read out as many jobserver slots as possible. - while True: - try: - slot =3D os.read(reader, 8) - jobs +=3D slot - except (OSError, IOError) as e: - if e.errno =3D=3D errno.EWOULDBLOCK: - # Stop at the end of the jobserver queue. - break - # If something went wrong, give back the jobs. - if len(jobs): - os.write(writer, jobs) - raise e - # Add a bump for our caller's reserveration, since we're just going - # to sit here blocked on our child. - claim =3D len(jobs) + 1 -except (KeyError, IndexError, ValueError, OSError, IOError) as e: - # Any missing environment strings or bad fds should result in just - # not being parallel. - pass - -# We can only claim parallelism if there was a jobserver (i.e. a top-level -# "-jN" argument) and there were no other failures. Otherwise leave out the -# environment variable and let the child figure out what is best. -if claim > 0: - os.environ['PARALLELISM'] =3D '%d' % (claim) - -rc =3D subprocess.call(sys.argv[1:]) - -# Return all the reserved slots. -if len(jobs): - os.write(writer, jobs) - -sys.exit(rc) + +class JobserverExec: + """ + Claim all slots from make using POSIX Jobserver. + + The main methods here are: + - open(): reserves all slots; + - close(): method returns all used slots back to make; + - run(): executes a command setting PARALLELISM=3D + """ + + def __init__(self): + """Initialize internal vars""" + self.claim =3D 0 + self.jobs =3D b"" + self.reader =3D None + self.writer =3D None + self.is_open =3D False + + def open(self): + """Reserve all available slots to be claimed later on""" + + if self.is_open: + return + + try: + # Fetch the make environment options. + flags =3D os.environ["MAKEFLAGS"] + # Look for "--jobserver=3DR,W" + # Note that GNU Make has used --jobserver-fds and --jobserver-= auth + # so this handles all of them. + opts =3D [x for x in flags.split(" ") if x.startswith("--jobse= rver")] + + # Parse out R,W file descriptor numbers and set them nonblocki= ng. + # If the MAKEFLAGS variable contains multiple instances of the + # --jobserver-auth=3D option, the last one is relevant. + fds =3D opts[-1].split("=3D", 1)[1] + + # Starting with GNU Make 4.4, named pipes are used for reader + # and writer. + # Example argument: --jobserver-auth=3Dfifo:/tmp/GMfifo8134 + _, _, path =3D fds.partition("fifo:") + + if path: + self.reader =3D os.open(path, os.O_RDONLY | os.O_NONBLOCK) + self.writer =3D os.open(path, os.O_WRONLY) + else: + self.reader, self.writer =3D [int(x) for x in fds.split(",= ", 1)] + # Open a private copy of reader to avoid setting nonblocki= ng + # on an unexpecting process with the same reader fd. + self.reader =3D os.open("/proc/self/fd/%d" % (self.reader), + os.O_RDONLY | os.O_NONBLOCK) + + # Read out as many jobserver slots as possible + while True: + try: + slot =3D os.read(self.reader, 8) + self.jobs +=3D slot + except (OSError, IOError) as e: + if e.errno =3D=3D errno.EWOULDBLOCK: + # Stop at the end of the jobserver queue. + break + # If something went wrong, give back the jobs. + if self.jobs: + os.write(self.writer, self.jobs) + raise e + + # Add a bump for our caller's reserveration, since we're just = going + # to sit here blocked on our child. + self.claim =3D len(self.jobs) + 1 + + except (KeyError, IndexError, ValueError, OSError, IOError): + # Any missing environment strings or bad fds should result in = just + # not being parallel. + self.claim =3D None + + self.is_open =3D True + + def close(self): + """Return all reserved slots to Jobserver""" + + if not self.is_open: + return + + # Return all the reserved slots. + if len(self.jobs): + os.write(self.writer, self.jobs) + + self.is_open =3D False + + def __enter__(self): + self.open() + return self + + def __exit__(self, exc_type, exc_value, exc_traceback): + self.close() + + def run(self, cmd): + """ + Run a command setting PARALLELISM env variable to the number of + available job slots (claim) + 1, e.g. it will reserve claim slots + to do the actual build work, plus one to monitor its childs. + """ + self.open() # Ensure that self.claim is set + + # We can only claim parallelism if there was a jobserver (i.e. a + # top-level "-jN" argument) and there were no other failures. Othe= rwise + # leave out the environment variable and let the child figure out = what + # is best. + if self.claim: + os.environ["PARALLELISM"] =3D str(self.claim) + + return subprocess.call(cmd) + + +def main(): + """Main program""" + with JobserverExec() as jobserver: + jobserver.run(sys.argv[1:]) + + +if __name__ =3D=3D "__main__": + main() --=20 2.51.0 From nobody Fri Oct 3 20:22:42 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C25EC301022; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; cv=none; b=Ehca0RkIhLixVHZxMibaOY99lpd/S4MLexoDj6vXgTNz4+qL1NrNN1Gqj4crymdeD0OHPIfPZrnULf8q6cJcd93Gyo8NNtV3IoE/K9D6z4GtXcTdxv+HF+rGfbBFYrbQcws8lQMGEZSO3mGL7PywpIh3yXdWsFbIMAO2p/FCgDE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; c=relaxed/simple; bh=RnTLGaGUuMOKF6PxxwxnFcuRL48slw9WnQakpfhYXJ0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=kyRkYVLmlwmaTwEfJVDqPu47w812Zvzxx3Eh88V4E3DQRaVkWmObHbVGu++7DP8PUqSXeKw+4RsrVsQ8kaPAyf2pW5Oe6ecGY2qJZ5YJLtmN6wKxwr4qNuIZyY9EkD4XkMHcM2oNNLeaAazegRJzzRgnlVTFckw5DQNKWXC7y10= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=tog2T2DC; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="tog2T2DC" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 21298C4AF09; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1756139449; bh=RnTLGaGUuMOKF6PxxwxnFcuRL48slw9WnQakpfhYXJ0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tog2T2DCPTUzPCA6keEz8S/Ws08qqip3oSC6KGuspdbXsk1zuj5DmVn9DKY3KDIE8 nX5IFtnwYczokEmZv7vprKz6xh6f8+uFq7VG/7wpmkvrsNDtZD4e7Niw/7/IUW4ONb 5fvd3CFgDGBOheyCOkSWOIXfvq80FSRkGqP5fdvfpVcaX4+LNSESLdY6BsR8Vv//gQ BEsgqdo7F1rdIIBeelHLVWjBOH+CHI/06lSAr2XYJV0tmrvwNm4Pdy3QO0GRg7KmZN 9lCiZX1uQgIVJ8JN+vlZnXt6srAiczXydoyygzedzW5PaVkr6zo5NZplySc5aZfphc RXGBRMzj25xiQ== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uqa5n-0000000HALM-0oLk; Mon, 25 Aug 2025 18:30:47 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Jonathan Corbet" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v2 02/13] scripts/jobserver-exec: move its class to the lib directory Date: Mon, 25 Aug 2025 18:30:29 +0200 Message-ID: <275aff71243dcaf7fe07ec239c89ed9a8f529a34.1756138805.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" To make it easier to be re-used, move the JobserverExec class to the library directory. Signed-off-by: Mauro Carvalho Chehab --- scripts/jobserver-exec | 152 +++------------------------------------ scripts/lib/jobserver.py | 149 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 160 insertions(+), 141 deletions(-) create mode 100755 scripts/lib/jobserver.py diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec index b386b1a845de..40a0f0058733 100755 --- a/scripts/jobserver-exec +++ b/scripts/jobserver-exec @@ -1,155 +1,25 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0+ -# -# pylint: disable=3DC0103,C0209 -# -# This determines how many parallel tasks "make" is expecting, as it is -# not exposed via an special variables, reserves them all, runs a subproce= ss -# with PARALLELISM environment variable set, and releases the jobs back ag= ain. -# -# https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#= POSIX-Jobserver =20 -""" -Interacts with the POSIX jobserver during the Kernel build time. - -A "normal" jobserver task, like the one initiated by a make subrocess woul= d do: - - - open read/write file descriptors to communicate with the job server; - - ask for one slot by calling: - claim =3D os.read(reader, 1) - - when the job finshes, call: - os.write(writer, b"+") # os.write(writer, claim) - -Here, the goal is different: This script aims to get the remaining number -of slots available, using all of them to run a command which handle tasks = in -parallel. To to that, it has a loop that ends only after there are no -slots left. It then increments the number by one, in order to allow a -call equivalent to make -j$((claim+1)), e.g. having a parent make creating -$claim child to do the actual work. - -The end goal here is to keep the total number of build tasks under the -limit established by the initial make -j$n_proc call. -""" - -import errno import os -import subprocess import sys =20 +LIB_DIR =3D "lib" +SRC_DIR =3D os.path.dirname(os.path.realpath(__file__)) =20 -class JobserverExec: - """ - Claim all slots from make using POSIX Jobserver. +sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR)) =20 - The main methods here are: - - open(): reserves all slots; - - close(): method returns all used slots back to make; - - run(): executes a command setting PARALLELISM=3D - """ +from jobserver import JobserverExec # pylint: disable=3DC= 0415 =20 - def __init__(self): - """Initialize internal vars""" - self.claim =3D 0 - self.jobs =3D b"" - self.reader =3D None - self.writer =3D None - self.is_open =3D False =20 - def open(self): - """Reserve all available slots to be claimed later on""" - - if self.is_open: - return - - try: - # Fetch the make environment options. - flags =3D os.environ["MAKEFLAGS"] - # Look for "--jobserver=3DR,W" - # Note that GNU Make has used --jobserver-fds and --jobserver-= auth - # so this handles all of them. - opts =3D [x for x in flags.split(" ") if x.startswith("--jobse= rver")] - - # Parse out R,W file descriptor numbers and set them nonblocki= ng. - # If the MAKEFLAGS variable contains multiple instances of the - # --jobserver-auth=3D option, the last one is relevant. - fds =3D opts[-1].split("=3D", 1)[1] - - # Starting with GNU Make 4.4, named pipes are used for reader - # and writer. - # Example argument: --jobserver-auth=3Dfifo:/tmp/GMfifo8134 - _, _, path =3D fds.partition("fifo:") - - if path: - self.reader =3D os.open(path, os.O_RDONLY | os.O_NONBLOCK) - self.writer =3D os.open(path, os.O_WRONLY) - else: - self.reader, self.writer =3D [int(x) for x in fds.split(",= ", 1)] - # Open a private copy of reader to avoid setting nonblocki= ng - # on an unexpecting process with the same reader fd. - self.reader =3D os.open("/proc/self/fd/%d" % (self.reader), - os.O_RDONLY | os.O_NONBLOCK) - - # Read out as many jobserver slots as possible - while True: - try: - slot =3D os.read(self.reader, 8) - self.jobs +=3D slot - except (OSError, IOError) as e: - if e.errno =3D=3D errno.EWOULDBLOCK: - # Stop at the end of the jobserver queue. - break - # If something went wrong, give back the jobs. - if self.jobs: - os.write(self.writer, self.jobs) - raise e - - # Add a bump for our caller's reserveration, since we're just = going - # to sit here blocked on our child. - self.claim =3D len(self.jobs) + 1 - - except (KeyError, IndexError, ValueError, OSError, IOError): - # Any missing environment strings or bad fds should result in = just - # not being parallel. - self.claim =3D None - - self.is_open =3D True - - def close(self): - """Return all reserved slots to Jobserver""" - - if not self.is_open: - return - - # Return all the reserved slots. - if len(self.jobs): - os.write(self.writer, self.jobs) - - self.is_open =3D False - - def __enter__(self): - self.open() - return self - - def __exit__(self, exc_type, exc_value, exc_traceback): - self.close() - - def run(self, cmd): - """ - Run a command setting PARALLELISM env variable to the number of - available job slots (claim) + 1, e.g. it will reserve claim slots - to do the actual build work, plus one to monitor its childs. - """ - self.open() # Ensure that self.claim is set - - # We can only claim parallelism if there was a jobserver (i.e. a - # top-level "-jN" argument) and there were no other failures. Othe= rwise - # leave out the environment variable and let the child figure out = what - # is best. - if self.claim: - os.environ["PARALLELISM"] =3D str(self.claim) - - return subprocess.call(cmd) +""" +Determines how many parallel tasks "make" is expecting, as it is +not exposed via an special variables, reserves them all, runs a subprocess +with PARALLELISM environment variable set, and releases the jobs back agai= n. =20 +See: + https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.htm= l#POSIX-Jobserver +""" =20 def main(): """Main program""" diff --git a/scripts/lib/jobserver.py b/scripts/lib/jobserver.py new file mode 100755 index 000000000000..98d8b0ff0c89 --- /dev/null +++ b/scripts/lib/jobserver.py @@ -0,0 +1,149 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0+ +# +# pylint: disable=3DC0103,C0209 +# +# + +""" +Interacts with the POSIX jobserver during the Kernel build time. + +A "normal" jobserver task, like the one initiated by a make subrocess woul= d do: + + - open read/write file descriptors to communicate with the job server; + - ask for one slot by calling: + claim =3D os.read(reader, 1) + - when the job finshes, call: + os.write(writer, b"+") # os.write(writer, claim) + +Here, the goal is different: This script aims to get the remaining number +of slots available, using all of them to run a command which handle tasks = in +parallel. To to that, it has a loop that ends only after there are no +slots left. It then increments the number by one, in order to allow a +call equivalent to make -j$((claim+1)), e.g. having a parent make creating +$claim child to do the actual work. + +The end goal here is to keep the total number of build tasks under the +limit established by the initial make -j$n_proc call. + +See: + https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.htm= l#POSIX-Jobserver +""" + +import errno +import os +import subprocess +import sys + +class JobserverExec: + """ + Claim all slots from make using POSIX Jobserver. + + The main methods here are: + - open(): reserves all slots; + - close(): method returns all used slots back to make; + - run(): executes a command setting PARALLELISM=3D + """ + + def __init__(self): + """Initialize internal vars""" + self.claim =3D 0 + self.jobs =3D b"" + self.reader =3D None + self.writer =3D None + self.is_open =3D False + + def open(self): + """Reserve all available slots to be claimed later on""" + + if self.is_open: + return + + try: + # Fetch the make environment options. + flags =3D os.environ["MAKEFLAGS"] + # Look for "--jobserver=3DR,W" + # Note that GNU Make has used --jobserver-fds and --jobserver-= auth + # so this handles all of them. + opts =3D [x for x in flags.split(" ") if x.startswith("--jobse= rver")] + + # Parse out R,W file descriptor numbers and set them nonblocki= ng. + # If the MAKEFLAGS variable contains multiple instances of the + # --jobserver-auth=3D option, the last one is relevant. + fds =3D opts[-1].split("=3D", 1)[1] + + # Starting with GNU Make 4.4, named pipes are used for reader + # and writer. + # Example argument: --jobserver-auth=3Dfifo:/tmp/GMfifo8134 + _, _, path =3D fds.partition("fifo:") + + if path: + self.reader =3D os.open(path, os.O_RDONLY | os.O_NONBLOCK) + self.writer =3D os.open(path, os.O_WRONLY) + else: + self.reader, self.writer =3D [int(x) for x in fds.split(",= ", 1)] + # Open a private copy of reader to avoid setting nonblocki= ng + # on an unexpecting process with the same reader fd. + self.reader =3D os.open("/proc/self/fd/%d" % (self.reader), + os.O_RDONLY | os.O_NONBLOCK) + + # Read out as many jobserver slots as possible + while True: + try: + slot =3D os.read(self.reader, 8) + self.jobs +=3D slot + except (OSError, IOError) as e: + if e.errno =3D=3D errno.EWOULDBLOCK: + # Stop at the end of the jobserver queue. + break + # If something went wrong, give back the jobs. + if self.jobs: + os.write(self.writer, self.jobs) + raise e + + # Add a bump for our caller's reserveration, since we're just = going + # to sit here blocked on our child. + self.claim =3D len(self.jobs) + 1 + + except (KeyError, IndexError, ValueError, OSError, IOError): + # Any missing environment strings or bad fds should result in = just + # not being parallel. + self.claim =3D None + + self.is_open =3D True + + def close(self): + """Return all reserved slots to Jobserver""" + + if not self.is_open: + return + + # Return all the reserved slots. + if len(self.jobs): + os.write(self.writer, self.jobs) + + self.is_open =3D False + + def __enter__(self): + self.open() + return self + + def __exit__(self, exc_type, exc_value, exc_traceback): + self.close() + + def run(self, cmd, *args, **pwargs): + """ + Run a command setting PARALLELISM env variable to the number of + available job slots (claim) + 1, e.g. it will reserve claim slots + to do the actual build work, plus one to monitor its childs. + """ + self.open() # Ensure that self.claim is set + + # We can only claim parallelism if there was a jobserver (i.e. a + # top-level "-jN" argument) and there were no other failures. Othe= rwise + # leave out the environment variable and let the child figure out = what + # is best. + if self.claim: + os.environ["PARALLELISM"] =3D str(self.claim) + + return subprocess.call(cmd, *args, **pwargs) --=20 2.51.0 From nobody Fri Oct 3 20:22:42 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B0CC1FECAD; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; cv=none; b=NCYWL76RcxYhe/QsrWq+0l5Cd/XhmMi+xHz7jiy9ODF1YfUxIr0iQ+J/wxhlF22VJBQyamKAPGhHpxwpKBAXcpNmc3NDH2IaeV0eIZG9X71LZ6hIlT6CRp80/O85PdeygX2Efn7OVBy85pYQ5nscUNjIHg2pxVwu3ANTNXzb3dQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; c=relaxed/simple; bh=jlol1TuhVNcZDXlNqsLBvYWROc27W0iXgTQpgdIUTbU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=iQeSFK8A8w1pEpxJWGmI6nn6V29bcUrWlB8FQRmAkqGVud5qNPHz7b5g1gHrMdY7Z5fBbFVZQNqZoApn1h/j0z7/6ymmW2qnRAK6nmhsMRsHRCLJQRSqcy0GypOw6iKXZOS8cHE1dFh68sDKQmZLTvKy7/L4bmlle/Wpf37BpwI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=TA904OiL; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="TA904OiL" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1045DC4CEED; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1756139449; bh=jlol1TuhVNcZDXlNqsLBvYWROc27W0iXgTQpgdIUTbU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TA904OiL46jx1hTUpDx5EECjUx5Tcs3DkAxF1f44JJIrA/U8PpyqHG9GGQNj4iZuO kAKQ0EuWvSaCpBbkqAt1bTKRJjE7n6lsZ+MtFlmLeAsb9eXO8R59nRglhPGhVss3eY 4eE9CU17sYEOGTzwwG5UcMbu5nBnNZTa9iJmu+q8zcnxzdmcphc7XVTWLFplO/vzva wxt+ekNTp5ZxX1ZDdf9njGK8Q7H0qOrc/5GuPMZyMztqWN6rULe2Q/AETVwQtvWa4W KBqLzZJaIe7hwmVjxmU+aq8LylUY/W2TBWi8UVbVoB+VUK7YUoZJcT9gHyt0kVRXSs eT+yVIcf5fdHw== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uqa5n-0000000HALQ-0us9; Mon, 25 Aug 2025 18:30:47 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Jonathan Corbet" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v2 03/13] scripts/jobserver-exec: add a help message Date: Mon, 25 Aug 2025 18:30:30 +0200 Message-ID: <78712c238669ad9b3ac83ea3cc42fb600c2ace00.1756138805.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Currently, calling it without an argument shows an ugly error message. Instead, print a message using pythondoc as description. Signed-off-by: Mauro Carvalho Chehab --- scripts/jobserver-exec | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec index 40a0f0058733..ae23afd344ec 100755 --- a/scripts/jobserver-exec +++ b/scripts/jobserver-exec @@ -1,6 +1,15 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0+ =20 +""" +Determines how many parallel tasks "make" is expecting, as it is +not exposed via any special variables, reserves them all, runs a subprocess +with PARALLELISM environment variable set, and releases the jobs back agai= n. + +See: + https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.htm= l#POSIX-Jobserver +""" + import os import sys =20 @@ -12,17 +21,12 @@ sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR)) from jobserver import JobserverExec # pylint: disable=3DC= 0415 =20 =20 -""" -Determines how many parallel tasks "make" is expecting, as it is -not exposed via an special variables, reserves them all, runs a subprocess -with PARALLELISM environment variable set, and releases the jobs back agai= n. - -See: - https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.htm= l#POSIX-Jobserver -""" - def main(): """Main program""" + if len(sys.argv) < 2: + name =3D os.path.basename(__file__) + sys.exit("usage: " + name +" command [args ...]\n" + __doc__) + with JobserverExec() as jobserver: jobserver.run(sys.argv[1:]) =20 --=20 2.51.0 From nobody Fri Oct 3 20:22:42 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8A7E13002AC; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; cv=none; b=i4cIz1+7Rg1hjPfGT3O7CefcBYNhyim2vUHXrAz2ZxmusjVngw/fU2f0NC2tuOIf4YKz4SFHRd3KoKF00JaCJNHHOYmSp0X4gpvOm8diXPzA8sv37yybPfgw9i2DfOQLEDwy9b+geKYuc+lrSv2CbZ996nJPiE68+zlYYore00c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; c=relaxed/simple; bh=7GnO3HikSmm/j8N04JDdRzA73zbHZ6baYyVkORZ4ge8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lqWtWalRnoHHpViRJo5gb5TLh1ZVBbvkckIloksA3tpYIR/VGC6z9oh0XB2VgAU9t/H1CO/OBimbGyfjr0qXu3/thfKzYOCetY3xrC/XjHvrfgjsJr9ioJgRwfU3YNJWFVmARpW5qawrXgTXVevUih7bUQFJmK8Ddur3xDNb4k0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=TaigDscE; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="TaigDscE" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2800BC4AF0B; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1756139449; bh=7GnO3HikSmm/j8N04JDdRzA73zbHZ6baYyVkORZ4ge8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TaigDscE2hufRauQSbUki4a3vg7S8LeZWmD7Ershpweg7tEjOS7CE3FBQRGmGu5aJ EuZxXJGnkYP1oMMQBRBYhv5HLXoO5SguPfF9JAu1HeNPkCXaUGYK4bNv097jMHa7w3 hJpGsaWTB9NTyr/dfaBUY0N34FJ6DQmnuBnvtrzRhNt6SGDk+V55+jgaya8Tpw1qSg EbNM272yuLjE15dlL8Ev3C7QnjytpsnYJywjI5G382scD7EhTrL9BCJcvqPh3TNwEy cM+FtiF6dgT3a2k8KmWNSyiuUrEF5s5GvBGh2C8lcA9Ngml6a8xazHFV52B8XYqXHx gGSzH6B1a5d5w== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uqa5n-0000000HALU-11iz; Mon, 25 Aug 2025 18:30:47 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Jonathan Corbet" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v2 04/13] scripts: sphinx-pre-install: move it to tools/docs Date: Mon, 25 Aug 2025 18:30:31 +0200 Message-ID: X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" As we're reorganizing the place where doc scripts are located, move this one to tools/docs. No functional changes. Signed-off-by: Mauro Carvalho Chehab --- Documentation/Makefile | 14 +++++++------- {scripts =3D> tools/docs}/sphinx-pre-install | 0 2 files changed, 7 insertions(+), 7 deletions(-) rename {scripts =3D> tools/docs}/sphinx-pre-install (100%) diff --git a/Documentation/Makefile b/Documentation/Makefile index 5c20c68be89a..deb2029228ed 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -46,7 +46,7 @@ ifeq ($(HAVE_SPHINX),0) .DEFAULT: $(warning The '$(SPHINXBUILD)' command was not found. Make sure you have = Sphinx installed and in PATH, or set the SPHINXBUILD make variable to point= to the full path of the '$(SPHINXBUILD)' executable.) @echo - @$(srctree)/scripts/sphinx-pre-install + @$(srctree)/tools/docs/sphinx-pre-install @echo " SKIP Sphinx $@ target." =20 else # HAVE_SPHINX @@ -105,7 +105,7 @@ quiet_cmd_sphinx =3D SPHINX $@ --> file://$(abspath $(= BUILDDIR)/$3/$4) fi =20 htmldocs: - @$(srctree)/scripts/sphinx-pre-install --version-check + @$(srctree)/tools/docs/sphinx-pre-install --version-check @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var))) =20 # If Rust support is available and .config exists, add rustdoc generated c= ontents. @@ -119,7 +119,7 @@ endif endif =20 texinfodocs: - @$(srctree)/scripts/sphinx-pre-install --version-check + @$(srctree)/tools/docs/sphinx-pre-install --version-check @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,texinfo,$(var),texin= fo,$(var))) =20 # Note: the 'info' Make target is generated by sphinx itself when @@ -131,7 +131,7 @@ linkcheckdocs: @$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,linkcheck,$(var),,$(v= ar))) =20 latexdocs: - @$(srctree)/scripts/sphinx-pre-install --version-check + @$(srctree)/tools/docs/sphinx-pre-install --version-check @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$= (var))) =20 ifeq ($(HAVE_PDFLATEX),0) @@ -144,7 +144,7 @@ else # HAVE_PDFLATEX =20 pdfdocs: DENY_VF =3D XDG_CONFIG_HOME=3D$(FONTS_CONF_DENY_VF) pdfdocs: latexdocs - @$(srctree)/scripts/sphinx-pre-install --version-check + @$(srctree)/tools/docs/sphinx-pre-install --version-check $(foreach var,$(SPHINXDIRS), \ $(MAKE) PDFLATEX=3D"$(PDFLATEX)" LATEXOPTS=3D"$(LATEXOPTS)" $(DENY_VF)= -C $(BUILDDIR)/$(var)/latex || sh $(srctree)/scripts/check-variable-fonts.= sh || exit; \ mkdir -p $(BUILDDIR)/$(var)/pdf; \ @@ -154,11 +154,11 @@ pdfdocs: latexdocs endif # HAVE_PDFLATEX =20 epubdocs: - @$(srctree)/scripts/sphinx-pre-install --version-check + @$(srctree)/tools/docs/sphinx-pre-install --version-check @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(v= ar))) =20 xmldocs: - @$(srctree)/scripts/sphinx-pre-install --version-check + @$(srctree)/tools/docs/sphinx-pre-install --version-check @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var= ))) =20 endif # HAVE_SPHINX diff --git a/scripts/sphinx-pre-install b/tools/docs/sphinx-pre-install similarity index 100% rename from scripts/sphinx-pre-install rename to tools/docs/sphinx-pre-install --=20 2.51.0 From nobody Fri Oct 3 20:22:42 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C2E31301025; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; cv=none; b=UtZQ96900XT9FoEbdGDCik2BGEAzgTb/Nv2/WmllINE9lWatsT3KGGbkpCEgNskeuvMPyxCuyTcUIfDvj3X0qVkYkY7NTVo8CXIJs+akOhd5rXxzA0ZCkOJMDvxEmV54B9dkxvEdd+tNyzLDk29ufBdi4KvpFrM3LGj+xKUV1SE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; c=relaxed/simple; bh=MWdCYti9mYZYzPB0kxq4N3AXTer18uFYM3OxtDmMddc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=eeQUulv3HzIL2Isj6PJ2za1FT0JnonwoziWdt/r878YgTw9wj6ewxC37tJ9Wy2vP0y4PhH760AQlBpAT++k+JGS3CKd72f+je/Fe7kiJ9Fig8LtQ6LI2COOLQMelXPU0555AOR7J7D4PC/2arh9V+MS9BJRhykppNwDaH4pf17o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=EJcGruHI; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="EJcGruHI" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 14D50C4CEF4; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1756139449; bh=MWdCYti9mYZYzPB0kxq4N3AXTer18uFYM3OxtDmMddc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EJcGruHITfsJe5BPb6tsPiQn3SjVAXAro0LfxA42yOTVxdLr4TmIJe3/ipfJSLdoB mv9EognsiEvXcAU5WLuAHMjvqNOEJcKEU8mOQhdvO84xihA5cPBaPYSYJ8SqaA++UN fV0tNVp7Me8f9oePjxt9QkKLf8nC+DHGc/X/lhe2ITuqNFEs9MmU1niZEZtIcA2t+U cueDsp2sWdwE/erdWUPUq4NHzTqqCtmWfItKpg7OeXB6eFHMGoVNCKtaIclVLNtsuO qmommRt5jqAsLWRiO2TwjXoWcj0s/pdwnAly3IPCBJYWtHwyAm+VRh/XZf/TiWC68b 2UPIQB8MFXwcw== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uqa5n-0000000HALY-18Vl; Mon, 25 Aug 2025 18:30:47 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Jonathan Corbet" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v2 05/13] tools/docs: sphinx-pre-install: move Python version handling to lib Date: Mon, 25 Aug 2025 18:30:32 +0200 Message-ID: X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab The sphinx-pre-install code has some logic to deal with Python version, which ensures that a minimal version will be enforced for documentation build logic. Move it to a separate library to allow re-using its code. No functional changes. Signed-off-by: Mauro Carvalho Chehab --- tools/docs/lib/python_version.py | 133 +++++++++++++++++++++++++++++++ tools/docs/sphinx-pre-install | 120 +++------------------------- 2 files changed, 146 insertions(+), 107 deletions(-) create mode 100644 tools/docs/lib/python_version.py diff --git a/tools/docs/lib/python_version.py b/tools/docs/lib/python_versi= on.py new file mode 100644 index 000000000000..0519d524e547 --- /dev/null +++ b/tools/docs/lib/python_version.py @@ -0,0 +1,133 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2017-2025 Mauro Carvalho Chehab + +""" +Handle Python version check logic. + +Not all Python versions are supported by scripts. Yet, on some cases, +like during documentation build, a newer version of python could be +available. + +This class allows checking if the minimal requirements are followed. + +Better than that, PythonVersion.check_python() not only checks the minimal +requirements, but it automatically switches to a the newest available +Python version if present. + +""" + +import os +import re +import subprocess +import sys + +from glob import glob + +class PythonVersion: + """ + Ancillary methods that checks for missing dependencies for different + types of types, like binaries, python modules, rpm deps, etc. + """ + + def __init__(self, version): + """=C3=8Fnitialize self.version tuple from a version string""" + self.version =3D self.parse_version(version) + + @staticmethod + def parse_version(version): + """Convert a major.minor.patch version into a tuple""" + return tuple(int(x) for x in version.split(".")) + + @staticmethod + def ver_str(version): + """Returns a version tuple as major.minor.patch""" + return ".".join([str(x) for x in version]) + + def __str__(self): + """Returns a version tuple as major.minor.patch from self.version"= "" + return self.ver_str(self.version) + + @staticmethod + def get_python_version(cmd): + """ + Get python version from a Python binary. As we need to detect if + are out there newer python binaries, we can't rely on sys.release = here. + """ + + kwargs =3D {} + if sys.version_info < (3, 7): + kwargs['universal_newlines'] =3D True + else: + kwargs['text'] =3D True + + result =3D subprocess.run([cmd, "--version"], + stdout =3D subprocess.PIPE, + stderr =3D subprocess.PIPE, + **kwargs, check=3DFalse) + + version =3D result.stdout.strip() + + match =3D re.search(r"(\d+\.\d+\.\d+)", version) + if match: + return PythonVersion.parse_version(match.group(1)) + + print(f"Can't parse version {version}") + return (0, 0, 0) + + @staticmethod + def find_python(min_version): + """ + Detect if are out there any python 3.xy version newer than the + current one. + + Note: this routine is limited to up to 2 digits for python3. We + may need to update it one day, hopefully on a distant future. + """ + patterns =3D [ + "python3.[0-9]", + "python3.[0-9][0-9]", + ] + + # Seek for a python binary newer than min_version + for path in os.getenv("PATH", "").split(":"): + for pattern in patterns: + for cmd in glob(os.path.join(path, pattern)): + if os.path.isfile(cmd) and os.access(cmd, os.X_OK): + version =3D PythonVersion.get_python_version(cmd) + if version >=3D min_version: + return cmd + + return None + + @staticmethod + def check_python(min_version): + """ + Check if the current python binary satisfies our minimal requireme= nt + for Sphinx build. If not, re-run with a newer version if found. + """ + cur_ver =3D sys.version_info[:3] + if cur_ver >=3D min_version: + ver =3D PythonVersion.ver_str(cur_ver) + print(f"Python version: {ver}") + + return + + python_ver =3D PythonVersion.ver_str(cur_ver) + + new_python_cmd =3D PythonVersion.find_python(min_version) + if not new_python_cmd: + print(f"ERROR: Python version {python_ver} is not spported any= more\n") + print(" Can't find a new version. This script may fail") + return + + # Restart script using the newer version + script_path =3D os.path.abspath(sys.argv[0]) + args =3D [new_python_cmd, script_path] + sys.argv[1:] + + print(f"Python {python_ver} not supported. Changing to {new_python= _cmd}") + + try: + os.execv(new_python_cmd, args) + except OSError as e: + sys.exit(f"Failed to restart with {new_python_cmd}: {e}") diff --git a/tools/docs/sphinx-pre-install b/tools/docs/sphinx-pre-install index 954ed3dc0645..d6d673b7945c 100755 --- a/tools/docs/sphinx-pre-install +++ b/tools/docs/sphinx-pre-install @@ -32,20 +32,10 @@ import subprocess import sys from glob import glob =20 +from lib.python_version import PythonVersion =20 -def parse_version(version): - """Convert a major.minor.patch version into a tuple""" - return tuple(int(x) for x in version.split(".")) - - -def ver_str(version): - """Returns a version tuple as major.minor.patch""" - - return ".".join([str(x) for x in version]) - - -RECOMMENDED_VERSION =3D parse_version("3.4.3") -MIN_PYTHON_VERSION =3D parse_version("3.7") +RECOMMENDED_VERSION =3D PythonVersion("3.4.3").version +MIN_PYTHON_VERSION =3D PythonVersion("3.7").version =20 =20 class DepManager: @@ -235,95 +225,11 @@ class AncillaryMethods: =20 return None =20 - @staticmethod - def get_python_version(cmd): - """ - Get python version from a Python binary. As we need to detect if - are out there newer python binaries, we can't rely on sys.release = here. - """ - - result =3D SphinxDependencyChecker.run([cmd, "--version"], - capture_output=3DTrue, text=3D= True) - version =3D result.stdout.strip() - - match =3D re.search(r"(\d+\.\d+\.\d+)", version) - if match: - return parse_version(match.group(1)) - - print(f"Can't parse version {version}") - return (0, 0, 0) - - @staticmethod - def find_python(): - """ - Detect if are out there any python 3.xy version newer than the - current one. - - Note: this routine is limited to up to 2 digits for python3. We - may need to update it one day, hopefully on a distant future. - """ - patterns =3D [ - "python3.[0-9]", - "python3.[0-9][0-9]", - ] - - # Seek for a python binary newer than MIN_PYTHON_VERSION - for path in os.getenv("PATH", "").split(":"): - for pattern in patterns: - for cmd in glob(os.path.join(path, pattern)): - if os.path.isfile(cmd) and os.access(cmd, os.X_OK): - version =3D SphinxDependencyChecker.get_python_ver= sion(cmd) - if version >=3D MIN_PYTHON_VERSION: - return cmd - - @staticmethod - def check_python(): - """ - Check if the current python binary satisfies our minimal requireme= nt - for Sphinx build. If not, re-run with a newer version if found. - """ - cur_ver =3D sys.version_info[:3] - if cur_ver >=3D MIN_PYTHON_VERSION: - ver =3D ver_str(cur_ver) - print(f"Python version: {ver}") - - # This could be useful for debugging purposes - if SphinxDependencyChecker.which("docutils"): - result =3D SphinxDependencyChecker.run(["docutils", "--ver= sion"], - capture_output=3DTrue,= text=3DTrue) - ver =3D result.stdout.strip() - match =3D re.search(r"(\d+\.\d+\.\d+)", ver) - if match: - ver =3D match.group(1) - - print(f"Docutils version: {ver}") - - return - - python_ver =3D ver_str(cur_ver) - - new_python_cmd =3D SphinxDependencyChecker.find_python() - if not new_python_cmd: - print(f"ERROR: Python version {python_ver} is not spported any= more\n") - print(" Can't find a new version. This script may fail") - return - - # Restart script using the newer version - script_path =3D os.path.abspath(sys.argv[0]) - args =3D [new_python_cmd, script_path] + sys.argv[1:] - - print(f"Python {python_ver} not supported. Changing to {new_python= _cmd}") - - try: - os.execv(new_python_cmd, args) - except OSError as e: - sys.exit(f"Failed to restart with {new_python_cmd}: {e}") - @staticmethod def run(*args, **kwargs): """ Excecute a command, hiding its output by default. - Preserve comatibility with older Python versions. + Preserve compatibility with older Python versions. """ =20 capture_output =3D kwargs.pop('capture_output', False) @@ -527,11 +433,11 @@ class MissingCheckers(AncillaryMethods): for line in result.stdout.split("\n"): match =3D re.match(r"^sphinx-build\s+([\d\.]+)(?:\+(?:/[\da-f]= +)|b\d+)?\s*$", line) if match: - return parse_version(match.group(1)) + return PythonVersion.parse_version(match.group(1)) =20 match =3D re.match(r"^Sphinx.*\s+([\d\.]+)\s*$", line) if match: - return parse_version(match.group(1)) + return PythonVersion.parse_version(match.group(1)) =20 def check_sphinx(self, conf): """ @@ -542,7 +448,7 @@ class MissingCheckers(AncillaryMethods): for line in f: match =3D re.match(r"^\s*needs_sphinx\s*=3D\s*[\'\"]([= \d\.]+)[\'\"]", line) if match: - self.min_version =3D parse_version(match.group(1)) + self.min_version =3D PythonVersion.parse_version(m= atch.group(1)) break except IOError: sys.exit(f"Can't open {conf}") @@ -562,8 +468,8 @@ class MissingCheckers(AncillaryMethods): sys.exit(f"{sphinx} didn't return its version") =20 if self.cur_version < self.min_version: - curver =3D ver_str(self.cur_version) - minver =3D ver_str(self.min_version) + curver =3D PythonVersion.ver_str(self.cur_version) + minver =3D PythonVersion.ver_str(self.min_version) =20 print(f"ERROR: Sphinx version is {curver}. It should be >=3D {= minver}") self.need_sphinx =3D 1 @@ -1304,7 +1210,7 @@ class SphinxDependencyChecker(MissingCheckers): else: if self.need_sphinx and ver >=3D self.min_version: return (f, ver) - elif parse_version(ver) > self.cur_version: + elif PythonVersion.parse_version(ver) > self.cur_version: return (f, ver) =20 return ("", ver) @@ -1411,7 +1317,7 @@ class SphinxDependencyChecker(MissingCheckers): return =20 if self.latest_avail_ver: - latest_avail_ver =3D ver_str(self.latest_avail_ver) + latest_avail_ver =3D PythonVersion.ver_str(self.latest_avail_v= er) =20 if not self.need_sphinx: # sphinx-build is present and its version is >=3D $min_version @@ -1507,7 +1413,7 @@ class SphinxDependencyChecker(MissingCheckers): else: print("Unknown OS") if self.cur_version !=3D (0, 0, 0): - ver =3D ver_str(self.cur_version) + ver =3D PythonVersion.ver_str(self.cur_version) print(f"Sphinx version: {ver}\n") =20 # Check the type of virtual env, depending on Python version @@ -1613,7 +1519,7 @@ def main(): =20 checker =3D SphinxDependencyChecker(args) =20 - checker.check_python() + PythonVersion.check_python(MIN_PYTHON_VERSION) checker.check_needs() =20 # Call main if not used as module --=20 2.51.0 From nobody Fri Oct 3 20:22:42 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C3115301027; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; cv=none; b=J9BT7OSlfMkw+b4A3aYrmkqXH3mcEl1obij7Ev/mkUlU6dQXoh6bcNv5vqgbRuq5D1t+F2eC9SJBJbmPHhycsQOjaD3a5b3X9FU7oTMTm8Db4uL3VbmXMqpI/7AN1QT2o1KzopZnlx+ghTTBulYvOUiLQL7CBhfQrVFq5E/NCmA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; c=relaxed/simple; bh=lUKOh/++1d6UgJrnOisFTbfDPc1mGpsnI1gPehdySmo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=i8FX98PXFBKJuAT5t3jYz7vvLpugQSSeNPMnDRDuZ1pCLJDmx6oPpUTt9nGWTKkXVRKU00ZmYe+BVzDkMz0kl05hs7uf70Vak9MFWXcyQw9KIaue5vtNf2k6Hc1PHCAXSw6LYPE0DlVQ/WIAXR8lhtThhmN7VDyAngHV+0E5Dzg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oJmyC2SC; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oJmyC2SC" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1710EC116C6; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1756139449; bh=lUKOh/++1d6UgJrnOisFTbfDPc1mGpsnI1gPehdySmo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oJmyC2SCYBsnWCvyOSIG4VtLRKL+jcEf+WnuNsjY+6oiGsFUwZS1FVSinCVbjfPOz 6hjoxr4JpSWJOYxt2Jo7rPTdl5C3II2F3t/arTJ9K2TIbd1c5NLQsTyhaCTc3ewLdP Ve31VXG7lw0CGCBfcAQKQmwS69Gv+W4XLW3NZk5sm6sGv8NUqZin4VzYpadcq7EZFO N73+lVccFsUXvw65xwROmXYYcfCVYZPnPiFEVk1hQ+XpAIp8c9xiyyKprT4b2Nxjpq F9Z8idxvMoSqzM2A1O2jMe5+4ks9yPImplHhlZpxi9PvlOvWj+PKh8BWgF+14vk+Im uEFXSjA0eGqOg== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uqa5n-0000000HALc-1FqO; Mon, 25 Aug 2025 18:30:47 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List Cc: Mauro Carvalho Chehab , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , "Jonathan Corbet" , "Mauro Carvalho Chehab" , Alex Gaynor , Alice Ryhl , Andreas Hindborg , Benno Lossin , Boqun Feng , Danilo Krummrich , Gary Guo , Miguel Ojeda , Trevor Gross , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org Subject: [PATCH v2 06/13] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build Date: Mon, 25 Aug 2025 18:30:33 +0200 Message-ID: <3e8ffb127bec01bd30d150b3325579e0a9c45cbf.1756138805.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" There are too much magic inside docs Makefile to properly run sphinx-build. Create an ancillary script that contains all kernel-related sphinx-build call logic currently at Makefile. Such script is designed to work both as an standalone command and as part of a Makefile. As such, it properly handles POSIX jobserver used by GNU make. On a side note, there was a line number increase due to the conversion: $ git show|diffstat -p1 Documentation/Makefile | 129 +++------------- tools/docs/sphinx-build-wrapper | 288 +++++++++++++++++++++++++++++++++++= ++ 2 files changed, 316 insertions(+), 101 deletions(-) This is because some things are more verbosed on Python and because it requires reading env vars from Makefile. Besides it, this script has some extra features that don't exist at the Makefile: - It can be called directly from command line; - It properly return PDF build errors. When running the script alone, it will only take handle sphinx-build targets. On other words, it won't runn make rustdoc after building htmlfiles, nor it will run the extra check scripts. Signed-off-by: Mauro Carvalho Chehab --- Documentation/Makefile | 129 ++++---------- tools/docs/sphinx-build-wrapper | 288 ++++++++++++++++++++++++++++++++ 2 files changed, 316 insertions(+), 101 deletions(-) create mode 100755 tools/docs/sphinx-build-wrapper diff --git a/Documentation/Makefile b/Documentation/Makefile index deb2029228ed..2b0ed8cd5ea8 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -23,21 +23,22 @@ SPHINXOPTS =3D SPHINXDIRS =3D . DOCS_THEME =3D DOCS_CSS =3D -_SPHINXDIRS =3D $(sort $(patsubst $(srctree)/Documentation/%/index.rst,%= ,$(wildcard $(srctree)/Documentation/*/index.rst))) SPHINX_CONF =3D conf.py PAPER =3D BUILDDIR =3D $(obj)/output PDFLATEX =3D xelatex LATEXOPTS =3D -interaction=3Dbatchmode -no-shell-escape =20 +PYTHONPYCACHEPREFIX ?=3D $(abspath $(BUILDDIR)/__pycache__) + +# Wrapper for sphinx-build + +BUILD_WRAPPER =3D $(srctree)/tools/docs/sphinx-build-wrapper + # For denylisting "variable font" files # Can be overridden by setting as an env variable FONTS_CONF_DENY_VF ?=3D $(HOME)/deny-vf =20 -ifeq ($(findstring 1, $(KBUILD_VERBOSE)),) -SPHINXOPTS +=3D "-q" -endif - # User-friendly check for sphinx-build HAVE_SPHINX :=3D $(shell if which $(SPHINXBUILD) >/dev/null 2>&1; then ech= o 1; else echo 0; fi) =20 @@ -51,63 +52,29 @@ ifeq ($(HAVE_SPHINX),0) =20 else # HAVE_SPHINX =20 -# User-friendly check for pdflatex and latexmk -HAVE_PDFLATEX :=3D $(shell if which $(PDFLATEX) >/dev/null 2>&1; then echo= 1; else echo 0; fi) -HAVE_LATEXMK :=3D $(shell if which latexmk >/dev/null 2>&1; then echo 1; e= lse echo 0; fi) +# Common documentation targets +infodocs texinfodocs latexdocs epubdocs xmldocs pdfdocs linkcheckdocs: + $(Q)@$(srctree)/tools/docs/sphinx-pre-install --version-check + +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \ + --sphinxdirs=3D"$(SPHINXDIRS)" --conf=3D$(SPHINX_CONF) \ + --theme=3D$(DOCS_THEME) --css=3D$(DOCS_CSS) --paper=3D$(PAPER) =20 -ifeq ($(HAVE_LATEXMK),1) - PDFLATEX :=3D latexmk -$(PDFLATEX) -endif #HAVE_LATEXMK - -# Internal variables. -PAPEROPT_a4 =3D -D latex_elements.papersize=3Da4paper -PAPEROPT_letter =3D -D latex_elements.papersize=3Dletterpaper -ALLSPHINXOPTS =3D -D kerneldoc_srctree=3D$(srctree) -D kerneldoc_bin=3D$= (KERNELDOC) -ALLSPHINXOPTS +=3D $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) -ifneq ($(wildcard $(srctree)/.config),) -ifeq ($(CONFIG_RUST),y) - # Let Sphinx know we will include rustdoc - ALLSPHINXOPTS +=3D -t rustdoc -endif +# Special handling for pdfdocs +ifeq ($(shell which $(PDFLATEX) >/dev/null 2>&1; echo $$?),0) +pdfdocs: DENY_VF =3D XDG_CONFIG_HOME=3D$(FONTS_CONF_DENY_VF) +else +pdfdocs: + $(warning The '$(PDFLATEX)' command was not found. Make sure you have it = installed and in PATH to produce PDF output.) + @echo " SKIP Sphinx $@ target." endif -# the i18n builder cannot share the environment and doctrees with the othe= rs -I18NSPHINXOPTS =3D $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . - -# commands; the 'cmd' from scripts/Kbuild.include is not *loopable* -loop_cmd =3D $(echo-cmd) $(cmd_$(1)) || exit; - -# $2 sphinx builder e.g. "html" -# $3 name of the build subfolder / e.g. "userspace-api/media", used as: -# * dest folder relative to $(BUILDDIR) and -# * cache folder relative to $(BUILDDIR)/.doctrees -# $4 dest subfolder e.g. "man" for man pages at userspace-api/media/man -# $5 reST source folder relative to $(src), -# e.g. "userspace-api/media" for the linux-tv book-set at ./Documentati= on/userspace-api/media - -PYTHONPYCACHEPREFIX ?=3D $(abspath $(BUILDDIR)/__pycache__) - -quiet_cmd_sphinx =3D SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4) - cmd_sphinx =3D \ - PYTHONPYCACHEPREFIX=3D"$(PYTHONPYCACHEPREFIX)" \ - BUILDDIR=3D$(abspath $(BUILDDIR)) SPHINX_CONF=3D$(abspath $(src)/$5/$(SPH= INX_CONF)) \ - $(PYTHON3) $(srctree)/scripts/jobserver-exec \ - $(CONFIG_SHELL) $(srctree)/Documentation/sphinx/parallel-wrapper.sh \ - $(SPHINXBUILD) \ - -b $2 \ - -c $(abspath $(src)) \ - -d $(abspath $(BUILDDIR)/.doctrees/$3) \ - -D version=3D$(KERNELVERSION) -D release=3D$(KERNELRELEASE) \ - $(ALLSPHINXOPTS) \ - $(abspath $(src)/$5) \ - $(abspath $(BUILDDIR)/$3/$4) && \ - if [ "x$(DOCS_CSS)" !=3D "x" ]; then \ - cp $(if $(patsubst /%,,$(DOCS_CSS)),$(abspath $(srctree)/$(DOCS_CSS)),$(= DOCS_CSS)) $(BUILDDIR)/$3/_static/; \ - fi =20 +# HTML main logic is identical to other targets. However, if rust is enabl= ed, +# an extra step at the end is required to generate rustdoc. htmldocs: - @$(srctree)/tools/docs/sphinx-pre-install --version-check - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var))) - + $(Q)@$(srctree)/tools/docs/sphinx-pre-install --version-check + +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \ + --sphinxdirs=3D"$(SPHINXDIRS)" --conf=3D$(SPHINX_CONF) \ + --theme=3D$(DOCS_THEME) --css=3D$(DOCS_CSS) --paper=3D$(PAPER) # If Rust support is available and .config exists, add rustdoc generated c= ontents. # If there are any, the errors from this make rustdoc will be displayed but # won't stop the execution of htmldocs @@ -118,49 +85,6 @@ ifeq ($(CONFIG_RUST),y) endif endif =20 -texinfodocs: - @$(srctree)/tools/docs/sphinx-pre-install --version-check - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,texinfo,$(var),texin= fo,$(var))) - -# Note: the 'info' Make target is generated by sphinx itself when -# running the texinfodocs target define above. -infodocs: texinfodocs - $(MAKE) -C $(BUILDDIR)/texinfo info - -linkcheckdocs: - @$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,linkcheck,$(var),,$(v= ar))) - -latexdocs: - @$(srctree)/tools/docs/sphinx-pre-install --version-check - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$= (var))) - -ifeq ($(HAVE_PDFLATEX),0) - -pdfdocs: - $(warning The '$(PDFLATEX)' command was not found. Make sure you have it = installed and in PATH to produce PDF output.) - @echo " SKIP Sphinx $@ target." - -else # HAVE_PDFLATEX - -pdfdocs: DENY_VF =3D XDG_CONFIG_HOME=3D$(FONTS_CONF_DENY_VF) -pdfdocs: latexdocs - @$(srctree)/tools/docs/sphinx-pre-install --version-check - $(foreach var,$(SPHINXDIRS), \ - $(MAKE) PDFLATEX=3D"$(PDFLATEX)" LATEXOPTS=3D"$(LATEXOPTS)" $(DENY_VF)= -C $(BUILDDIR)/$(var)/latex || sh $(srctree)/scripts/check-variable-fonts.= sh || exit; \ - mkdir -p $(BUILDDIR)/$(var)/pdf; \ - mv $(subst .tex,.pdf,$(wildcard $(BUILDDIR)/$(var)/latex/*.tex)) $(BUI= LDDIR)/$(var)/pdf/; \ - ) - -endif # HAVE_PDFLATEX - -epubdocs: - @$(srctree)/tools/docs/sphinx-pre-install --version-check - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(v= ar))) - -xmldocs: - @$(srctree)/tools/docs/sphinx-pre-install --version-check - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var= ))) - endif # HAVE_SPHINX =20 # The following targets are independent of HAVE_SPHINX, and the rules shou= ld @@ -172,6 +96,9 @@ refcheckdocs: cleandocs: $(Q)rm -rf $(BUILDDIR) =20 +# Used only on help +_SPHINXDIRS =3D $(sort $(patsubst $(srctree)/Documentation/%/index.rst,%= ,$(wildcard $(srctree)/Documentation/*/index.rst))) + dochelp: @echo ' Linux kernel internal documentation in different formats from Re= ST:' @echo ' htmldocs - HTML' diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrap= per new file mode 100755 index 000000000000..df469af8a4ef --- /dev/null +++ b/tools/docs/sphinx-build-wrapper @@ -0,0 +1,288 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +import argparse +import os +import shlex +import shutil +import subprocess +import sys +from lib.python_version import PythonVersion + +LIB_DIR =3D "../../scripts/lib" +SRC_DIR =3D os.path.dirname(os.path.realpath(__file__)) +sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR)) + +from jobserver import JobserverExec + +MIN_PYTHON_VERSION =3D PythonVersion("3.7").version +PAPER =3D ["", "a4", "letter"] +TARGETS =3D { + "cleandocs": { "builder": "clean" }, + "linkcheckdocs": { "builder": "linkcheck" }, + "htmldocs": { "builder": "html" }, + "epubdocs": { "builder": "epub", "out_dir": "epub" }, + "texinfodocs": { "builder": "texinfo", "out_dir": "texinfo" }, + "infodocs": { "builder": "texinfo", "out_dir": "texinfo" }, + "latexdocs": { "builder": "latex", "out_dir": "latex" }, + "pdfdocs": { "builder": "latex", "out_dir": "latex" }, + "xmldocs": { "builder": "xml", "out_dir": "xml" }, +} + +class SphinxBuilder: + def is_rust_enabled(self): + config_path =3D os.path.join(self.srctree, ".config") + if os.path.isfile(config_path): + with open(config_path, "r", encoding=3D"utf-8") as f: + return "CONFIG_RUST=3Dy" in f.read() + return False + + def get_path(self, path, abs_path=3DFalse): + path =3D os.path.expanduser(path) + if not path.startswith("/"): + path =3D os.path.join(self.srctree, path) + if abs_path: + return os.path.abspath(path) + return path + + def __init__(self, verbose=3DFalse, n_jobs=3DNone): + self.verbose =3D None + self.kernelversion =3D os.environ.get("KERNELVERSION", "unknown") + self.kernelrelease =3D os.environ.get("KERNELRELEASE", "unknown") + self.pdflatex =3D os.environ.get("PDFLATEX", "xelatex") + self.latexopts =3D os.environ.get("LATEXOPTS", "-interaction=3Dbat= chmode -no-shell-escape") + if not verbose: + verbose =3D bool(os.environ.get("KBUILD_VERBOSE", "") !=3D "") + if verbose is not None: + self.verbose =3D verbose + parser =3D argparse.ArgumentParser() + parser.add_argument('-j', '--jobs', type=3Dint) + parser.add_argument('-q', '--quiet', type=3Dint) + sphinxopts =3D shlex.split(os.environ.get("SPHINXOPTS", "")) + sphinx_args, self.sphinxopts =3D parser.parse_known_args(sphinxopt= s) + if sphinx_args.quiet is True: + self.verbose =3D False + if sphinx_args.jobs: + self.n_jobs =3D sphinx_args.jobs + self.n_jobs =3D n_jobs + self.srctree =3D os.environ.get("srctree") + if not self.srctree: + self.srctree =3D "." + os.environ["srctree"] =3D self.srctree + self.sphinxbuild =3D os.environ.get("SPHINXBUILD", "sphinx-build") + self.kerneldoc =3D self.get_path(os.environ.get("KERNELDOC", + "scripts/kernel-doc.= py")) + self.obj =3D os.environ.get("obj", "Documentation") + self.builddir =3D self.get_path(os.path.join(self.obj, "output"), + abs_path=3DTrue) + + self.config_rust =3D self.is_rust_enabled() + + self.pdflatex_cmd =3D shutil.which(self.pdflatex) + self.latexmk_cmd =3D shutil.which("latexmk") + + self.env =3D os.environ.copy() + + def run_sphinx(self, sphinx_build, build_args, *args, **pwargs): + with JobserverExec() as jobserver: + if jobserver.claim: + n_jobs =3D str(jobserver.claim) + else: + n_jobs =3D "auto" # Supported since Sphinx 1.7 + cmd =3D [] + cmd.append(sys.executable) + cmd.append(sphinx_build) + if self.n_jobs: + n_jobs =3D str(self.n_jobs) + + if n_jobs: + cmd +=3D [f"-j{n_jobs}"] + + if not self.verbose: + cmd.append("-q") + cmd +=3D self.sphinxopts + cmd +=3D build_args + if self.verbose: + print(" ".join(cmd)) + return subprocess.call(cmd, *args, **pwargs) + + def handle_html(self, css, output_dir): + if not css: + return + css =3D os.path.expanduser(css) + if not css.startswith("/"): + css =3D os.path.join(self.srctree, css) + static_dir =3D os.path.join(output_dir, "_static") + os.makedirs(static_dir, exist_ok=3DTrue) + try: + shutil.copy2(css, static_dir) + except (OSError, IOError) as e: + print(f"Warning: Failed to copy CSS: {e}", file=3Dsys.stderr) + + def handle_pdf(self, output_dirs): + builds =3D {} + max_len =3D 0 + for from_dir in output_dirs: + pdf_dir =3D os.path.join(from_dir, "../pdf") + os.makedirs(pdf_dir, exist_ok=3DTrue) + if self.latexmk_cmd: + latex_cmd =3D [self.latexmk_cmd, f"-{self.pdflatex}"] + else: + latex_cmd =3D [self.pdflatex] + latex_cmd.extend(shlex.split(self.latexopts)) + tex_suffix =3D ".tex" + has_tex =3D False + build_failed =3D False + with os.scandir(from_dir) as it: + for entry in it: + if not entry.name.endswith(tex_suffix): + continue + name =3D entry.name[:-len(tex_suffix)] + has_tex =3D True + try: + subprocess.run(latex_cmd + [entry.path], + cwd=3Dfrom_dir, check=3DTrue) + except subprocess.CalledProcessError: + pass + pdf_name =3D name + ".pdf" + pdf_from =3D os.path.join(from_dir, pdf_name) + pdf_to =3D os.path.join(pdf_dir, pdf_name) + if os.path.exists(pdf_from): + os.rename(pdf_from, pdf_to) + builds[name] =3D os.path.relpath(pdf_to, self.buil= ddir) + else: + builds[name] =3D "FAILED" + build_failed =3D True + name =3D entry.name.removesuffix(".tex") + max_len =3D max(max_len, len(name)) + + if not has_tex: + name =3D os.path.basename(from_dir) + max_len =3D max(max_len, len(name)) + builds[name] =3D "FAILED (no .tex)" + build_failed =3D True + msg =3D "Summary" + msg +=3D "\n" + "=3D" * len(msg) + print() + print(msg) + for pdf_name, pdf_file in builds.items(): + print(f"{pdf_name:<{max_len}}: {pdf_file}") + print() + if build_failed: + sys.exit("PDF build failed: not all PDF files were created.") + else: + print("All PDF files were built.") + + def handle_info(self, output_dirs): + for output_dir in output_dirs: + try: + subprocess.run(["make", "info"], cwd=3Doutput_dir, check= =3DTrue) + except subprocess.CalledProcessError as e: + sys.exit(f"Error generating info docs: {e}") + + def cleandocs(self, builder): + shutil.rmtree(self.builddir, ignore_errors=3DTrue) + + def build(self, target, sphinxdirs=3DNone, conf=3D"conf.py", + theme=3DNone, css=3DNone, paper=3DNone): + builder =3D TARGETS[target]["builder"] + out_dir =3D TARGETS[target].get("out_dir", "") + if target =3D=3D "cleandocs": + self.cleandocs(builder) + return + if theme: + os.environ["DOCS_THEME"] =3D theme + sphinxbuild =3D shutil.which(self.sphinxbuild, path=3Dself.env["PA= TH"]) + if not sphinxbuild: + sys.exit(f"Error: {self.sphinxbuild} not found in PATH.\n") + if builder =3D=3D "latex": + if not self.pdflatex_cmd and not self.latexmk_cmd: + sys.exit("Error: pdflatex or latexmk required for PDF gene= ration") + docs_dir =3D os.path.abspath(os.path.join(self.srctree, "Documenta= tion")) + kerneldoc =3D self.kerneldoc + if kerneldoc.startswith(self.srctree): + kerneldoc =3D os.path.relpath(kerneldoc, self.srctree) + args =3D [ "-b", builder, "-c", docs_dir ] + if builder =3D=3D "latex": + if not paper: + paper =3D PAPER[1] + args.extend(["-D", f"latex_elements.papersize=3D{paper}paper"]) + if self.config_rust: + args.extend(["-t", "rustdoc"]) + if conf: + self.env["SPHINX_CONF"] =3D self.get_path(conf, abs_path=3DTru= e) + if not sphinxdirs: + sphinxdirs =3D os.environ.get("SPHINXDIRS", ".") + sphinxdirs_list =3D [] + for sphinxdir in sphinxdirs: + if isinstance(sphinxdir, list): + sphinxdirs_list +=3D sphinxdir + else: + for name in sphinxdir.split(" "): + sphinxdirs_list.append(name) + output_dirs =3D [] + for sphinxdir in sphinxdirs_list: + src_dir =3D os.path.join(docs_dir, sphinxdir) + doctree_dir =3D os.path.join(self.builddir, ".doctrees") + output_dir =3D os.path.join(self.builddir, sphinxdir, out_dir) + src_dir =3D os.path.normpath(src_dir) + doctree_dir =3D os.path.normpath(doctree_dir) + output_dir =3D os.path.normpath(output_dir) + os.makedirs(doctree_dir, exist_ok=3DTrue) + os.makedirs(output_dir, exist_ok=3DTrue) + output_dirs.append(output_dir) + build_args =3D args + [ + "-d", doctree_dir, + "-D", f"kerneldoc_bin=3D{kerneldoc}", + "-D", f"version=3D{self.kernelversion}", + "-D", f"release=3D{self.kernelrelease}", + "-D", f"kerneldoc_srctree=3D{self.srctree}", + src_dir, + output_dir, + ] + try: + self.run_sphinx(sphinxbuild, build_args, env=3Dself.env) + except (OSError, ValueError, subprocess.SubprocessError) as e: + sys.exit(f"Build failed: {repr(e)}") + if target in ["htmldocs", "epubdocs"]: + self.handle_html(css, output_dir) + if target =3D=3D "pdfdocs": + self.handle_pdf(output_dirs) + elif target =3D=3D "infodocs": + self.handle_info(output_dirs) + +def jobs_type(value): + if value is None: + return None + if value.lower() =3D=3D 'auto': + return value.lower() + try: + if int(value) >=3D 1: + return value + raise argparse.ArgumentTypeError(f"Minimum jobs is 1, got {value}") + except ValueError: + raise argparse.ArgumentTypeError(f"Must be 'auto' or positive inte= ger, got {value}") + +def main(): + parser =3D argparse.ArgumentParser(description=3D"Kernel documentation= builder") + parser.add_argument("target", choices=3Dlist(TARGETS.keys()), + help=3D"Documentation target to build") + parser.add_argument("--sphinxdirs", nargs=3D"+", + help=3D"Specific directories to build") + parser.add_argument("--conf", default=3D"conf.py", + help=3D"Sphinx configuration file") + parser.add_argument("--theme", help=3D"Sphinx theme to use") + parser.add_argument("--css", help=3D"Custom CSS file for HTML/EPUB") + parser.add_argument("--paper", choices=3DPAPER, default=3DPAPER[0], + help=3D"Paper size for LaTeX/PDF output") + parser.add_argument("-v", "--verbose", action=3D'store_true', + help=3D"place build in verbose mode") + parser.add_argument('-j', '--jobs', type=3Djobs_type, + help=3D"Sets number of jobs to use with sphinx-bui= ld") + args =3D parser.parse_args() + PythonVersion.check_python(MIN_PYTHON_VERSION) + builder =3D SphinxBuilder(verbose=3Dargs.verbose, n_jobs=3Dargs.jobs) + builder.build(args.target, sphinxdirs=3Dargs.sphinxdirs, conf=3Dargs.c= onf, + theme=3Dargs.theme, css=3Dargs.css, paper=3Dargs.paper) + +if __name__ =3D=3D "__main__": + main() --=20 2.51.0 From nobody Fri Oct 3 20:22:42 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C6120301029; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; cv=none; b=te1K3e4cIvxu6zRzcnrM2VWqqwAhSsaZFgOVf7DGTbuAMjo8EKhurMgN/h8joiP54aw6Qd2nJvOWL4W8I+5xqQBSdgN5OzDh+eJswkc6EZCrFMFwchU7FO1iV203kUa963brYM1AouKM9CH9NaBPUDukFoLCmShFgvyRkApSYU4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; c=relaxed/simple; bh=zgcBwTYzYpYoguSl9SjIJy1h20wgmzRq08xK3oGy2bM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LQr7KcVtgYUM1d0kFMC2LKnfIM1yy+7jrR4F+vDQGgrYl1ZZkwlZ3oFvU9dMFcbOaC8NmQhSex9gPQNsNA90Cf90CHV6TY8uB5GU8oRYuQ9WCJAdPmJl0djEy/msIbi2OYovpb/vYzd8izZxQSdYWEGuD+inXxdbo0rGnLAzTTM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=SdCp4AHS; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="SdCp4AHS" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 234D9C16AAE; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1756139449; bh=zgcBwTYzYpYoguSl9SjIJy1h20wgmzRq08xK3oGy2bM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SdCp4AHSS3pmRSWfMDVI2xkReQZlCxaxXCN1dQckwO00Kw9MNCgm5QspCDmeDjoei 47kwtgcM0v/ofYprSzhMt1ueBhAWoJ0D/2P6tX+ZYkaE1N935IIM9aspYB83O2+m/3 AnLsXdlhe4olFveXNdxF+T1kzCTLJH7ShRdEB4pntSRT84f9WHDtsil5RJ9LdlZsEa MHGOlxAQ8rwEeACJq3K1R5Mvu6B14rM+HOxkhj4RcFMsugwUt2ULqOMiryCSQ52W/j abHqJwwWhqZ1xGhR+4rHYlWjdMWfe6+FTx9/RHYzzAzOXtoZyIF0oLVZ/DyYv32KFq K+Rzvdvgy3B5w== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uqa5n-0000000HALg-1NAh; Mon, 25 Aug 2025 18:30:47 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List Cc: Mauro Carvalho Chehab , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , "Jonathan Corbet" , "Mauro Carvalho Chehab" , Alex Gaynor , Alice Ryhl , Andreas Hindborg , Benno Lossin , Boqun Feng , Danilo Krummrich , Gary Guo , Miguel Ojeda , Trevor Gross , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org Subject: [PATCH v2 07/13] tools/docs: sphinx-build-wrapper: add comments and blank lines Date: Mon, 25 Aug 2025 18:30:34 +0200 Message-ID: <42688d491785370d83b670edc9d32a7a6a9304ff.1756138805.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" To help seing the actual size of the script when it was added, I opted to strip out all comments from the original script. Add them here: tools/docs/sphinx-build-wrapper | 261 +++++++++++++++++++++++++++++++- 1 file changed, 257 insertions(+), 4 deletion(-) As the code from the script has 288 lines of code, it means that about half of the script are comments. Also ensure pylint won't report any warnings. No functional changes. Signed-off-by: Mauro Carvalho Chehab --- tools/docs/sphinx-build-wrapper | 261 +++++++++++++++++++++++++++++++- 1 file changed, 257 insertions(+), 4 deletions(-) diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrap= per index df469af8a4ef..e9c522794fbe 100755 --- a/tools/docs/sphinx-build-wrapper +++ b/tools/docs/sphinx-build-wrapper @@ -1,21 +1,71 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2025 Mauro Carvalho Chehab +# +# pylint: disable=3DR0902, R0912, R0913, R0914, R0915, R0917, C0103 +# +# Converted from docs Makefile and parallel-wrapper.sh, both under +# GPLv2, copyrighted since 2008 by the following authors: +# +# Akira Yokosawa +# Arnd Bergmann +# Breno Leitao +# Carlos Bilbao +# Dave Young +# Donald Hunter +# Geert Uytterhoeven +# Jani Nikula +# Jan Stancek +# Jonathan Corbet +# Joshua Clayton +# Kees Cook +# Linus Torvalds +# Magnus Damm +# Masahiro Yamada +# Mauro Carvalho Chehab +# Maxim Cournoyer +# Peter Foley +# Randy Dunlap +# Rob Herring +# Shuah Khan +# Thorsten Blum +# Tomas Winkler + + +""" +Sphinx build wrapper that handles Kernel-specific business rules: + +- it gets the Kernel build environment vars; +- it determines what's the best parallelism; +- it handles SPHINXDIRS + +This tool ensures that MIN_PYTHON_VERSION is satisfied. If version is +below that, it seeks for a new Python version. If found, it re-runs using +the newer version. +""" + import argparse import os import shlex import shutil import subprocess import sys + from lib.python_version import PythonVersion =20 LIB_DIR =3D "../../scripts/lib" SRC_DIR =3D os.path.dirname(os.path.realpath(__file__)) + sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR)) =20 -from jobserver import JobserverExec +from jobserver import JobserverExec # pylint: disable=3DC0413,C041= 1,E0401 =20 +# +# Some constants +# MIN_PYTHON_VERSION =3D PythonVersion("3.7").version PAPER =3D ["", "a4", "letter"] + TARGETS =3D { "cleandocs": { "builder": "clean" }, "linkcheckdocs": { "builder": "linkcheck" }, @@ -28,8 +78,19 @@ TARGETS =3D { "xmldocs": { "builder": "xml", "out_dir": "xml" }, } =20 + +# +# SphinxBuilder class +# + class SphinxBuilder: + """ + Handles a sphinx-build target, adding needed arguments to build + with the Kernel. + """ + def is_rust_enabled(self): + """Check if rust is enabled at .config""" config_path =3D os.path.join(self.srctree, ".config") if os.path.isfile(config_path): with open(config_path, "r", encoding=3D"utf-8") as f: @@ -37,37 +98,83 @@ class SphinxBuilder: return False =20 def get_path(self, path, abs_path=3DFalse): + """ + Ancillary routine to handle patches the right way, as shell does. + + It first expands "~" and "~user". Then, if patch is not absolute, + join self.srctree. Finally, if requested, convert to abspath. + """ + path =3D os.path.expanduser(path) if not path.startswith("/"): path =3D os.path.join(self.srctree, path) + if abs_path: return os.path.abspath(path) + return path =20 def __init__(self, verbose=3DFalse, n_jobs=3DNone): + """Initialize internal variables""" self.verbose =3D None + + # + # Normal variables passed from Kernel's makefile + # self.kernelversion =3D os.environ.get("KERNELVERSION", "unknown") self.kernelrelease =3D os.environ.get("KERNELRELEASE", "unknown") self.pdflatex =3D os.environ.get("PDFLATEX", "xelatex") self.latexopts =3D os.environ.get("LATEXOPTS", "-interaction=3Dbat= chmode -no-shell-escape") + if not verbose: verbose =3D bool(os.environ.get("KBUILD_VERBOSE", "") !=3D "") + if verbose is not None: self.verbose =3D verbose + + # + # As we handle number of jobs and quiet in separate, we need to pi= ck + # both the same way as sphinx-build would pick, optionally accepts + # whitespaces or not. So let's use argparse to handle argument exp= ansion + # parser =3D argparse.ArgumentParser() parser.add_argument('-j', '--jobs', type=3Dint) parser.add_argument('-q', '--quiet', type=3Dint) + + # + # Other sphinx-build arguments go as-is, so place them + # at self.sphinxopts, using shell parser + # sphinxopts =3D shlex.split(os.environ.get("SPHINXOPTS", "")) + + # + # Build a list of sphinx args + # sphinx_args, self.sphinxopts =3D parser.parse_known_args(sphinxopt= s) if sphinx_args.quiet is True: self.verbose =3D False + if sphinx_args.jobs: self.n_jobs =3D sphinx_args.jobs + + # + # If the command line argument "-j" is used override SPHINXOPTS + # + self.n_jobs =3D n_jobs + + # + # Source tree directory. This needs to be at os.environ, as + # Sphinx extensions use it + # self.srctree =3D os.environ.get("srctree") if not self.srctree: self.srctree =3D "." os.environ["srctree"] =3D self.srctree + + # + # Now that we can expand srctree, get other directories as well + # self.sphinxbuild =3D os.environ.get("SPHINXBUILD", "sphinx-build") self.kerneldoc =3D self.get_path(os.environ.get("KERNELDOC", "scripts/kernel-doc.= py")) @@ -77,20 +184,36 @@ class SphinxBuilder: =20 self.config_rust =3D self.is_rust_enabled() =20 + # + # Get directory locations for LaTeX build toolchain + # self.pdflatex_cmd =3D shutil.which(self.pdflatex) self.latexmk_cmd =3D shutil.which("latexmk") =20 self.env =3D os.environ.copy() =20 def run_sphinx(self, sphinx_build, build_args, *args, **pwargs): + """ + Executes sphinx-build using current python3 command and setting + -j parameter if possible to run the build in parallel. + """ + with JobserverExec() as jobserver: if jobserver.claim: n_jobs =3D str(jobserver.claim) else: n_jobs =3D "auto" # Supported since Sphinx 1.7 + cmd =3D [] + cmd.append(sys.executable) + cmd.append(sphinx_build) + + # + # Override auto setting, if explicitly passed from command line + # or via SPHINXOPTS + # if self.n_jobs: n_jobs =3D str(self.n_jobs) =20 @@ -99,59 +222,100 @@ class SphinxBuilder: =20 if not self.verbose: cmd.append("-q") + cmd +=3D self.sphinxopts cmd +=3D build_args + if self.verbose: print(" ".join(cmd)) return subprocess.call(cmd, *args, **pwargs) =20 def handle_html(self, css, output_dir): + """ + Extra steps for HTML and epub output. + + For such targets, we need to ensure that CSS will be properly + copied to the output _static directory + """ + if not css: return + css =3D os.path.expanduser(css) if not css.startswith("/"): css =3D os.path.join(self.srctree, css) + static_dir =3D os.path.join(output_dir, "_static") os.makedirs(static_dir, exist_ok=3DTrue) + try: shutil.copy2(css, static_dir) except (OSError, IOError) as e: print(f"Warning: Failed to copy CSS: {e}", file=3Dsys.stderr) =20 def handle_pdf(self, output_dirs): + """ + Extra steps for PDF output. + + As PDF is handled via a LaTeX output, after building the .tex file, + a new build is needed to create the PDF output from the latex + directory. + """ builds =3D {} max_len =3D 0 + for from_dir in output_dirs: pdf_dir =3D os.path.join(from_dir, "../pdf") os.makedirs(pdf_dir, exist_ok=3DTrue) + if self.latexmk_cmd: latex_cmd =3D [self.latexmk_cmd, f"-{self.pdflatex}"] else: latex_cmd =3D [self.pdflatex] + latex_cmd.extend(shlex.split(self.latexopts)) + tex_suffix =3D ".tex" + + # + # Process each .tex file + # + has_tex =3D False build_failed =3D False with os.scandir(from_dir) as it: for entry in it: if not entry.name.endswith(tex_suffix): continue + name =3D entry.name[:-len(tex_suffix)] has_tex =3D True + + # + # LaTeX PDF error code is almost useless for us: + # any warning makes it non-zero. For kernel doc builds= it + # always return non-zero even when build succeeds. + # So, let's do the best next thing: check if all PDF + # files were built. If they're, print a summary and + # return 0 at the end of this function + # try: subprocess.run(latex_cmd + [entry.path], cwd=3Dfrom_dir, check=3DTrue) except subprocess.CalledProcessError: pass + pdf_name =3D name + ".pdf" pdf_from =3D os.path.join(from_dir, pdf_name) pdf_to =3D os.path.join(pdf_dir, pdf_name) + if os.path.exists(pdf_from): os.rename(pdf_from, pdf_to) builds[name] =3D os.path.relpath(pdf_to, self.buil= ddir) else: builds[name] =3D "FAILED" build_failed =3D True + name =3D entry.name.removesuffix(".tex") max_len =3D max(max_len, len(name)) =20 @@ -160,58 +324,100 @@ class SphinxBuilder: max_len =3D max(max_len, len(name)) builds[name] =3D "FAILED (no .tex)" build_failed =3D True + msg =3D "Summary" msg +=3D "\n" + "=3D" * len(msg) print() print(msg) + for pdf_name, pdf_file in builds.items(): print(f"{pdf_name:<{max_len}}: {pdf_file}") + print() + if build_failed: sys.exit("PDF build failed: not all PDF files were created.") else: print("All PDF files were built.") =20 def handle_info(self, output_dirs): + """ + Extra steps for Info output. + + For texinfo generation, an additional make is needed from the + texinfo directory. + """ + for output_dir in output_dirs: try: subprocess.run(["make", "info"], cwd=3Doutput_dir, check= =3DTrue) except subprocess.CalledProcessError as e: sys.exit(f"Error generating info docs: {e}") =20 - def cleandocs(self, builder): + def cleandocs(self, builder): # pylint: disable=3DW0613 + """Remove documentation output directory""" shutil.rmtree(self.builddir, ignore_errors=3DTrue) =20 def build(self, target, sphinxdirs=3DNone, conf=3D"conf.py", theme=3DNone, css=3DNone, paper=3DNone): + """ + Build documentation using Sphinx. This is the core function of this + module. It prepares all arguments required by sphinx-build. + """ + builder =3D TARGETS[target]["builder"] out_dir =3D TARGETS[target].get("out_dir", "") + + # + # Cleandocs doesn't require sphinx-build + # if target =3D=3D "cleandocs": self.cleandocs(builder) return + if theme: - os.environ["DOCS_THEME"] =3D theme + os.environ["DOCS_THEME"] =3D theme + + # + # Other targets require sphinx-build, so check if it exists + # sphinxbuild =3D shutil.which(self.sphinxbuild, path=3Dself.env["PA= TH"]) if not sphinxbuild: sys.exit(f"Error: {self.sphinxbuild} not found in PATH.\n") + if builder =3D=3D "latex": if not self.pdflatex_cmd and not self.latexmk_cmd: sys.exit("Error: pdflatex or latexmk required for PDF gene= ration") + docs_dir =3D os.path.abspath(os.path.join(self.srctree, "Documenta= tion")) + + # + # Fill in base arguments for Sphinx build + # kerneldoc =3D self.kerneldoc if kerneldoc.startswith(self.srctree): kerneldoc =3D os.path.relpath(kerneldoc, self.srctree) + args =3D [ "-b", builder, "-c", docs_dir ] + if builder =3D=3D "latex": if not paper: paper =3D PAPER[1] + args.extend(["-D", f"latex_elements.papersize=3D{paper}paper"]) + if self.config_rust: args.extend(["-t", "rustdoc"]) + if conf: self.env["SPHINX_CONF"] =3D self.get_path(conf, abs_path=3DTru= e) + if not sphinxdirs: sphinxdirs =3D os.environ.get("SPHINXDIRS", ".") + + # + # sphinxdirs can be a list or a whitespace-separated string + # sphinxdirs_list =3D [] for sphinxdir in sphinxdirs: if isinstance(sphinxdir, list): @@ -219,17 +425,32 @@ class SphinxBuilder: else: for name in sphinxdir.split(" "): sphinxdirs_list.append(name) + + # + # Step 1: Build each directory in separate. + # + # This is not the best way of handling it, as cross-references bet= ween + # them will be broken, but this is what we've been doing since + # the beginning. + # output_dirs =3D [] for sphinxdir in sphinxdirs_list: src_dir =3D os.path.join(docs_dir, sphinxdir) doctree_dir =3D os.path.join(self.builddir, ".doctrees") output_dir =3D os.path.join(self.builddir, sphinxdir, out_dir) + + # + # Make directory names canonical + # src_dir =3D os.path.normpath(src_dir) doctree_dir =3D os.path.normpath(doctree_dir) output_dir =3D os.path.normpath(output_dir) + os.makedirs(doctree_dir, exist_ok=3DTrue) os.makedirs(output_dir, exist_ok=3DTrue) + output_dirs.append(output_dir) + build_args =3D args + [ "-d", doctree_dir, "-D", f"kerneldoc_bin=3D{kerneldoc}", @@ -239,48 +460,80 @@ class SphinxBuilder: src_dir, output_dir, ] + try: self.run_sphinx(sphinxbuild, build_args, env=3Dself.env) except (OSError, ValueError, subprocess.SubprocessError) as e: sys.exit(f"Build failed: {repr(e)}") + + # + # Ensure that each html/epub output will have needed static fi= les + # if target in ["htmldocs", "epubdocs"]: self.handle_html(css, output_dir) + + # + # Step 2: Some targets (PDF and info) require an extra step once + # sphinx-build finishes + # if target =3D=3D "pdfdocs": self.handle_pdf(output_dirs) elif target =3D=3D "infodocs": self.handle_info(output_dirs) =20 def jobs_type(value): + """ + Handle valid values for -j. Accepts Sphinx "-jauto", plus a number + equal or bigger than one. + """ if value is None: return None + if value.lower() =3D=3D 'auto': return value.lower() + try: if int(value) >=3D 1: return value + raise argparse.ArgumentTypeError(f"Minimum jobs is 1, got {value}") except ValueError: - raise argparse.ArgumentTypeError(f"Must be 'auto' or positive inte= ger, got {value}") + raise argparse.ArgumentTypeError(f"Must be 'auto' or positive inte= ger, got {value}") # pylint: disable=3DW0707 =20 def main(): + """ + Main function. The only mandatory argument is the target. If not + specified, the other arguments will use default values if not + specified at os.environ. + """ parser =3D argparse.ArgumentParser(description=3D"Kernel documentation= builder") + parser.add_argument("target", choices=3Dlist(TARGETS.keys()), help=3D"Documentation target to build") parser.add_argument("--sphinxdirs", nargs=3D"+", help=3D"Specific directories to build") parser.add_argument("--conf", default=3D"conf.py", help=3D"Sphinx configuration file") + parser.add_argument("--theme", help=3D"Sphinx theme to use") + parser.add_argument("--css", help=3D"Custom CSS file for HTML/EPUB") + parser.add_argument("--paper", choices=3DPAPER, default=3DPAPER[0], help=3D"Paper size for LaTeX/PDF output") + parser.add_argument("-v", "--verbose", action=3D'store_true', help=3D"place build in verbose mode") + parser.add_argument('-j', '--jobs', type=3Djobs_type, help=3D"Sets number of jobs to use with sphinx-bui= ld") + args =3D parser.parse_args() + PythonVersion.check_python(MIN_PYTHON_VERSION) + builder =3D SphinxBuilder(verbose=3Dargs.verbose, n_jobs=3Dargs.jobs) + builder.build(args.target, sphinxdirs=3Dargs.sphinxdirs, conf=3Dargs.c= onf, theme=3Dargs.theme, css=3Dargs.css, paper=3Dargs.paper) =20 --=20 2.51.0 From nobody Fri Oct 3 20:22:42 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 814F32FE58F; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; cv=none; b=q++jePTTc5KBbX/U+e+YhZqs/p/hyz25Sk5Ez/RRsOsKTPbbuKEPeo4gAKSD9FS/oAazd81fWj2ZqJ+9PrS08oqsG3XAYKH1qZkQH3JArWFpxE3Kfmw+A9cePD0CQ1NilRV60rW4/PDGlUGZdiyZfShNNhVmcfYlx6M7msHbnWY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; c=relaxed/simple; bh=07ltFNs1ZVDrRsHyXMC8vS/Q/NnwAG3htK5IbjLMyV0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=An+JmY91DvZqA4SEs9r7LDDcSnuWf+xXvKWw3ScQ2RXJO3oYeViwFY2xSbsaAtvxjFw+5PwE8C8k5r5GS4ffblyNgYW61MFinhp5Hq/KRXyySZWhYM/l6b1GCyBgCZTgzZ7mRP6CAbCNr6RC2MNV/mPuUJ26qYYNTnJp3dqpqGc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=lZ8bjumF; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="lZ8bjumF" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1E84EC19421; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1756139449; bh=07ltFNs1ZVDrRsHyXMC8vS/Q/NnwAG3htK5IbjLMyV0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lZ8bjumF8lLHC46ZcGpjlepVHJJsc9/irk/sVw3HIaBH8sAqnCHuxjFwR/L/dYOQ7 fBp4Hk171PBrzVlDLdM1tNgxVkbsDmarXWGmLu1Z0mD0hyAA3DrZqhMeYFdojH1sDG oa+iDgnmZlhMPmbCprAaEVxuJ98Z0EM58/fLoXQC2MDlqItw9buphyPmVY22qiiC74 7hxgDeOcBt9defB96anKaPUrIcd2w+JrodgwOhKNGwXQc3WOtmRCjT7D28tKePlg3h H/zWcO9wCd2m7Pl2rKNijv0DSWjLn9xU19CS5CxoIO92pw1t3I2A8G91FtkZiJvzUK jcZw25JsC+2GA== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uqa5n-0000000HALk-1TrC; Mon, 25 Aug 2025 18:30:47 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Jonathan Corbet" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v2 08/13] tools/docs: sphinx-build-wrapper: add support to run inside venv Date: Mon, 25 Aug 2025 18:30:35 +0200 Message-ID: X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Sometimes, it is desired to run Sphinx from a virtual environment. Add a command line parameter to automatically build Sphinx from such environment. Signed-off-by: Mauro Carvalho Chehab --- tools/docs/sphinx-build-wrapper | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrap= per index e9c522794fbe..9c5df50fb99d 100755 --- a/tools/docs/sphinx-build-wrapper +++ b/tools/docs/sphinx-build-wrapper @@ -63,6 +63,7 @@ from jobserver import JobserverExec # pylint: dis= able=3DC0413,C0411,E0401 # # Some constants # +VENV_DEFAULT =3D "sphinx_latest" MIN_PYTHON_VERSION =3D PythonVersion("3.7").version PAPER =3D ["", "a4", "letter"] =20 @@ -114,8 +115,9 @@ class SphinxBuilder: =20 return path =20 - def __init__(self, verbose=3DFalse, n_jobs=3DNone): + def __init__(self, venv=3DNone, verbose=3DFalse, n_jobs=3DNone): """Initialize internal variables""" + self.venv =3D venv self.verbose =3D None =20 # @@ -192,6 +194,21 @@ class SphinxBuilder: =20 self.env =3D os.environ.copy() =20 + # + # If venv command line argument is specified, run Sphinx from venv + # + if venv: + bin_dir =3D os.path.join(venv, "bin") + if not os.path.isfile(os.path.join(bin_dir, "activate")): + sys.exit(f"Venv {venv} not found.") + + # "activate" virtual env + self.env["PATH"] =3D bin_dir + ":" + self.env["PATH"] + self.env["VIRTUAL_ENV"] =3D venv + if "PYTHONHOME" in self.env: + del self.env["PYTHONHOME"] + print(f"Setting venv to {venv}") + def run_sphinx(self, sphinx_build, build_args, *args, **pwargs): """ Executes sphinx-build using current python3 command and setting @@ -206,7 +223,10 @@ class SphinxBuilder: =20 cmd =3D [] =20 - cmd.append(sys.executable) + if self.venv: + cmd.append("python") + else: + cmd.append(sys.executable) =20 cmd.append(sphinx_build) =20 @@ -528,11 +548,16 @@ def main(): parser.add_argument('-j', '--jobs', type=3Djobs_type, help=3D"Sets number of jobs to use with sphinx-bui= ld") =20 + parser.add_argument("-V", "--venv", nargs=3D'?', const=3Df'{VENV_DEFAU= LT}', + default=3DNone, + help=3Df'If used, run Sphinx from a venv dir (defa= ult dir: {VENV_DEFAULT})') + args =3D parser.parse_args() =20 PythonVersion.check_python(MIN_PYTHON_VERSION) =20 - builder =3D SphinxBuilder(verbose=3Dargs.verbose, n_jobs=3Dargs.jobs) + builder =3D SphinxBuilder(venv=3Dargs.venv, verbose=3Dargs.verbose, + n_jobs=3Dargs.jobs) =20 builder.build(args.target, sphinxdirs=3Dargs.sphinxdirs, conf=3Dargs.c= onf, theme=3Dargs.theme, css=3Dargs.css, paper=3Dargs.paper) --=20 2.51.0 From nobody Fri Oct 3 20:22:42 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E83C2F9C39; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; cv=none; b=NE4vrMjMxR/jKNKJw+0ipr5h5Q7nBDC8znFz8buxPq0LxC2T9VUI3JWKh4to1KGshudzNwmjT/0UCArh4LzN7BqztyyUXDkeQ26xviEKbqA6oy8qJXWxjQWd/UhourZNpWZKzA+03GhEE6IJcAc59oIHejvibsrw0SNzvwJFFz0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; c=relaxed/simple; bh=ZLbhSMLrYrG412X2oMrDythT1CiWtmLDdQQNSbEgmmI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DGJPTimXZrTPGQfHumsKZfzofkHI7oirNEgcNHOxsF4zMWP7tryJvikpDK6yMWjP730HX6XsTEEaf9Jyf5ciSUYwYKesQ/vEFfmKBXLsX2EvkSJn7kVqnpJJ7Lboy4CxO8cWgO8tDpKm+SupQlt7jwTla2N6/5ZiBijB6pZqnDU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=j8+g6R4q; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="j8+g6R4q" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1BE7CC113D0; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1756139449; bh=ZLbhSMLrYrG412X2oMrDythT1CiWtmLDdQQNSbEgmmI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=j8+g6R4qWqiOPi1kXfYRrWQzD/YoUuizd9FmCqnkh+bO0ulgH9xvxOAFjrkYseXkK QAwKBi3QKR3LRPmF/eLLObR0uF5MiH4YUUrkQQFWvugrPI+s0hKL7uqiG/GWR/ntuV Be9rz0noLNgCN6k8S/EIpbG+V+XZ+q4wdDdShLz9GZSZ8dqO2B3TKoJy5SV+D89fH7 F7F/x1pFCqsJD4TvbmokJyCfCC9jVOGU/6OGmPp4xWqkaM70xSSFBsva7gM4CRu9u/ 1QhyTFHRxSs2I8kmDYFsff3EiAMUdrx+S/5uK+WnK6WkBaCQxkPvuAdrxFtkv/YKnH +KUMxdNhj4gnw== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uqa5n-0000000HALo-1aWZ; Mon, 25 Aug 2025 18:30:47 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Jonathan Corbet" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v2 09/13] docs: parallel-wrapper.sh: remove script Date: Mon, 25 Aug 2025 18:30:36 +0200 Message-ID: <7a2d57ae8fddcfdf0498d5dcdab09741df26d222.1756138805.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" The only usage of this script was docs Makefile. Now that it is using the new sphinx-build-wrapper, which has inside the code from parallel-wrapper.sh, we can drop this script. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/parallel-wrapper.sh | 33 ------------------------ 1 file changed, 33 deletions(-) delete mode 100644 Documentation/sphinx/parallel-wrapper.sh diff --git a/Documentation/sphinx/parallel-wrapper.sh b/Documentation/sphin= x/parallel-wrapper.sh deleted file mode 100644 index e54c44ce117d..000000000000 --- a/Documentation/sphinx/parallel-wrapper.sh +++ /dev/null @@ -1,33 +0,0 @@ -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0+ -# -# Figure out if we should follow a specific parallelism from the make -# environment (as exported by scripts/jobserver-exec), or fall back to -# the "auto" parallelism when "-jN" is not specified at the top-level -# "make" invocation. - -sphinx=3D"$1" -shift || true - -parallel=3D"$PARALLELISM" -if [ -z "$parallel" ] ; then - # If no parallelism is specified at the top-level make, then - # fall back to the expected "-jauto" mode that the "htmldocs" - # target has had. - auto=3D$(perl -e 'open IN,"'"$sphinx"' --version 2>&1 |"; - while () { - if (m/([\d\.]+)/) { - print "auto" if ($1 >=3D "1.7") - } - } - close IN') - if [ -n "$auto" ] ; then - parallel=3D"$auto" - fi -fi -# Only if some parallelism has been determined do we add the -jN option. -if [ -n "$parallel" ] ; then - parallel=3D"-j$parallel" -fi - -exec "$sphinx" $parallel "$@" --=20 2.51.0 From nobody Fri Oct 3 20:22:42 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8C1043002AD; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; cv=none; b=sp4QXu5oWsQ0jpLk97YtMjbKiEAy2jZYaW+tXP5cKk5x7NXkhlB3ghm5lJDZ+XSp9dc5xAKnVsp35jHnURV+F/VrckrReZaVYttvYdRRAjDQaKaOxhOOs4xU+TjGRrPkiRg7n+xSujdtuIZf+I9bDG7v8xJz03NAUqalgE7tNEc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; c=relaxed/simple; bh=q5NJvNQyCP9iSlId/hbG6e0du+t6sh6fZ2TXKRoeDBU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ikCEbGUvHUX74mzlKAQkxaGeZ/2AcLNLA94a488MjdVEkkJzwovg8OWnqudAx3v7MXLCb2O4y41bNmecEgzfB08a2ClXC5eawzJnsbxEodEiAF8WZFUHpCuG9KTId8hFx1DjJ4uViYSBj90RDpV7RbDD4WwOU72heGvSxBpdIhI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Fzele6Xx; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Fzele6Xx" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3103AC19424; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1756139449; bh=q5NJvNQyCP9iSlId/hbG6e0du+t6sh6fZ2TXKRoeDBU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Fzele6XxKhjb0c37JX4A0urG5aPmRpkxgKm1o51xLME/jBz6iujImtF1LPvGO89sR aMpfJ3LB+Ce9VMjP8upQBSPC9u6EsXQJHF54owlRLE1K2E5Rh8d3XM+NSmjTxJ40Uo tAx+dLWuHR/n0DqdHIdZwcH11wdU1Jfc/k/L99UOnHKwEgQGQkxQjG+SYDQLQWyWVy 8oKH+txSLFRuxGS0j6gy4cSw6fRdB9H6N0rxsMByJBssWC4j0gV3icpRa7JIGQyQE4 itLA1Emf31X/mj0/m/QGKnoLrL32qlR4O7aA+jp5fOR5ZaHf8B9xRHDCauVprZDUAE 6uHBt9XkiLYLw== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uqa5n-0000000HALs-1h7I; Mon, 25 Aug 2025 18:30:47 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Jonathan Corbet" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v2 10/13] docs: Makefile: document latex/PDF PAPER= parameter Date: Mon, 25 Aug 2025 18:30:37 +0200 Message-ID: <565ceecafde7345153f5b7aeb1483e6cb22fec73.1756138805.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" While the build system supports this for a long time, this was never documented. Add a documentation for it. Signed-off-by: Mauro Carvalho Chehab --- Documentation/Makefile | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Documentation/Makefile b/Documentation/Makefile index 2b0ed8cd5ea8..3e1cb44a5fbb 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -124,4 +124,6 @@ dochelp: @echo @echo ' make DOCS_CSS=3D{a .css file} adds a DOCS_CSS override file for= html/epub output.' @echo + @echo ' make PAPER=3D{a4|letter} Specifies the paper size used for LaTe= X/PDF output.' + @echo @echo ' Default location for the generated documents is Documentation/o= utput' --=20 2.51.0 From nobody Fri Oct 3 20:22:42 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A328E301003; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; cv=none; b=pKzIyX6WBB7HUlcWbT0k/n3FtOBRYoZac10+M3GdBIbB9W+lV8oiAH3eXqi5LnY89FrKIHAO5qyEMFJkVOyUeoOniZfHxdzN4GFZjNuyaQwOv0q32AU+l8b21CjBvtBYGo+UftOeJYWF4sQyQna/T32E8Lfx8rEDGnlPi7piskI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; c=relaxed/simple; bh=DUnDSGE1B+eDLTJGGuJp1airL8xfT3+GXJbkMbRzrzw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LGJYrjstrlAnLOKgmyaodJjJx3h1HgafMvA99g6ah8hMYqJIJAVt3Z90XRHq/dfS9fS0x9eT6uCU7mCrun8rnoszDYBWs11BYv5nJ1XyaNYO3KVCwOGDk3nmS/J3bKHZtdq5+gftHP9cIfOPspnnczRMRXvry6XwjHgk4+6CF54= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mL7Tge5U; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mL7Tge5U" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 385E5C2BCAF; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1756139449; bh=DUnDSGE1B+eDLTJGGuJp1airL8xfT3+GXJbkMbRzrzw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mL7Tge5UDOXr94ffaXCo6zSd4Xz0TTlVTFbF7QVaiGgF2fFm4xew84fVEx2aYv6OS 2CdqOG3EcF2R1GNvbJCYOVAdCDh+w+X1Zjvltb+Jm9h/Z5kHE2Y2TjQuwldQMLLt8L sZBTOCf5lnvpkqxZOMpSztmaQ9KdwJhfc5/nQOGY8/7h/8ur84tyQUN7V+Jc+/G0q+ sF8glHFt/ITeKdlyvsIVSpLwnfw2luEk/iB4CccB65pHyBlgazyRK0AH9IA3FaoC3k HF3g4nZ8YvuIppFBgsd3BIL8M0t3QiljLeLa6IAoIC0t1B9G9mq/nSCtb24zh/U5gH ecBkyuEwe9dLQ== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uqa5n-0000000HALw-1nla; Mon, 25 Aug 2025 18:30:47 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Jonathan Corbet" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v2 11/13] tools/docs: sphinx-build-wrapper: add an argument for LaTeX interactive mode Date: Mon, 25 Aug 2025 18:30:38 +0200 Message-ID: X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" By default, we use LaTeX batch mode to build docs. This way, when an error happens, the build fails. This is good for normal builds, but when debugging problems with pdf generation, the best is to use interactive mode. We already support it via LATEXOPTS, but having a command line argument makes it easier: Interactive mode: ./scripts/sphinx-build-wrapper pdfdocs --sphinxdirs peci -v -i ... Running 'xelatex --no-pdf -no-pdf -recorder ".../Documentation/output/pe= ci/latex/peci.tex"' ... Default batch mode: ./scripts/sphinx-build-wrapper pdfdocs --sphinxdirs peci -v ... Running 'xelatex --no-pdf -no-pdf -interaction=3Dbatchmode -no-shell-esca= pe -recorder ".../Documentation/output/peci/latex/peci.tex"' ... Signed-off-by: Mauro Carvalho Chehab --- tools/docs/sphinx-build-wrapper | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrap= per index 9c5df50fb99d..4e87584a92cd 100755 --- a/tools/docs/sphinx-build-wrapper +++ b/tools/docs/sphinx-build-wrapper @@ -115,7 +115,7 @@ class SphinxBuilder: =20 return path =20 - def __init__(self, venv=3DNone, verbose=3DFalse, n_jobs=3DNone): + def __init__(self, venv=3DNone, verbose=3DFalse, n_jobs=3DNone, intera= ctive=3DNone): """Initialize internal variables""" self.venv =3D venv self.verbose =3D None @@ -126,7 +126,11 @@ class SphinxBuilder: self.kernelversion =3D os.environ.get("KERNELVERSION", "unknown") self.kernelrelease =3D os.environ.get("KERNELRELEASE", "unknown") self.pdflatex =3D os.environ.get("PDFLATEX", "xelatex") - self.latexopts =3D os.environ.get("LATEXOPTS", "-interaction=3Dbat= chmode -no-shell-escape") + + if not interactive: + self.latexopts =3D os.environ.get("LATEXOPTS", "-interaction= =3Dbatchmode -no-shell-escape") + else: + self.latexopts =3D os.environ.get("LATEXOPTS", "") =20 if not verbose: verbose =3D bool(os.environ.get("KBUILD_VERBOSE", "") !=3D "") @@ -548,6 +552,9 @@ def main(): parser.add_argument('-j', '--jobs', type=3Djobs_type, help=3D"Sets number of jobs to use with sphinx-bui= ld") =20 + parser.add_argument('-i', '--interactive', action=3D'store_true', + help=3D"Change latex default to run in interactive= mode") + parser.add_argument("-V", "--venv", nargs=3D'?', const=3Df'{VENV_DEFAU= LT}', default=3DNone, help=3Df'If used, run Sphinx from a venv dir (defa= ult dir: {VENV_DEFAULT})') @@ -557,7 +564,7 @@ def main(): PythonVersion.check_python(MIN_PYTHON_VERSION) =20 builder =3D SphinxBuilder(venv=3Dargs.venv, verbose=3Dargs.verbose, - n_jobs=3Dargs.jobs) + n_jobs=3Dargs.jobs, interactive=3Dargs.interac= tive) =20 builder.build(args.target, sphinxdirs=3Dargs.sphinxdirs, conf=3Dargs.c= onf, theme=3Dargs.theme, css=3Dargs.css, paper=3Dargs.paper) --=20 2.51.0 From nobody Fri Oct 3 20:22:42 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 76C362561AE; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; cv=none; b=fjO2uUIpWFHmwZaBttrPUR/y07v4dMZzPju4IhUOT5fOSk+ktHW56pPtel/hV38wI4buBmbhy6kIwyuWj0W7zEO06fuh/0gbyH4gYTonm0KVE+ISOr/IDpZHLU+IuxmWb9mFN3b7SgLYzwImbEI+dRZS5d79NbIM1iSqe+5cSp8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; c=relaxed/simple; bh=nyxVw48XE87zy6IhgQPKNGycuGi899OK6HHKPI4XVM8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=i2CFoE0LZJ0zSzccP/GCdTkhRdUHZC82EX3M+v0E8wjyd282TSoijUJYX1jWkr6fIygASKx8mSAX5MGLyGx75OTsImYBtdHOP0paXIkpy09Sel/18H0V6QRo/jRQks1JdY0Fa4nAkdusoyktVGFwH4PodnHJ+yRKpmSRUjmioSo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=mqGv7swu; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="mqGv7swu" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 34A1CC19425; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1756139449; bh=nyxVw48XE87zy6IhgQPKNGycuGi899OK6HHKPI4XVM8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=mqGv7swu45/lj5Wj6MPygUJZ5944aWDKzQbrnCnhNUJGOWb4s0GitCnUa5ZqwNMK+ L57ZsBOSWrRtq/w8LDpVI+ZJ2ufSKXcfHDzjwCyxDpLfyFnHbKir4A4/f5fJhZ9bSG xT/7+dlC4nCQ1kDANbwwOYjhsndgxC6Dr7EwA3onPjFa8g0fN1nL/1sHaQMkHS9/LQ WB0BgAsB75OY46eNl5vH+yS+6ohOvUcEt6vXsQCXrboyzI4L5lMVUpvKPvrx7INcrL ZxmgQQ3rs4UPra5UMt/BQ/Ly9U1vVxYOHMmJh9YwRPkhHy2xBAuk8798M3CO+gd8Sq KgwoqaGiarn2w== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uqa5n-0000000HAM0-1uT3; Mon, 25 Aug 2025 18:30:47 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Jonathan Corbet" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v2 12/13] tools/docs,scripts: sphinx-*: prevent sphinx-build crashes Date: Mon, 25 Aug 2025 18:30:39 +0200 Message-ID: <1779b6a3ec1d320e872883381fe1b725c2140ffc.1756138805.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" On a properly set system, LANG and LC_ALL is always defined. However, some distros like Debian, Gentoo and their variants start with those undefioned. When Sphinx tries to set a locale with: locale.setlocale(locale.LC_ALL, '') It raises an exception, making Sphinx fail. This is more likely to happen with test containers. Add a logic to detect and workaround such issue by setting locale to C. Signed-off-by: Mauro Carvalho Chehab --- tools/docs/sphinx-build-wrapper | 11 +++++++++++ tools/docs/sphinx-pre-install | 14 +++++++++++++- 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrap= per index 4e87584a92cd..580582a76c93 100755 --- a/tools/docs/sphinx-build-wrapper +++ b/tools/docs/sphinx-build-wrapper @@ -45,6 +45,7 @@ the newer version. """ =20 import argparse +import locale import os import shlex import shutil @@ -439,6 +440,16 @@ class SphinxBuilder: if not sphinxdirs: sphinxdirs =3D os.environ.get("SPHINXDIRS", ".") =20 + # + # The sphinx-build tool has a bug: internally, it tries to set + # locale with locale.setlocale(locale.LC_ALL, ''). This causes a + # crash if language is not set. Detect and fix it. + # + try: + locale.setlocale(locale.LC_ALL, '') + except locale.Error: + self.env["LC_ALL"] =3D "C" + # # sphinxdirs can be a list or a whitespace-separated string # diff --git a/tools/docs/sphinx-pre-install b/tools/docs/sphinx-pre-install index d6d673b7945c..663d4e2a3f57 100755 --- a/tools/docs/sphinx-pre-install +++ b/tools/docs/sphinx-pre-install @@ -26,6 +26,7 @@ system pacage install is recommended. """ =20 import argparse +import locale import os import re import subprocess @@ -422,8 +423,19 @@ class MissingCheckers(AncillaryMethods): """ Gets sphinx-build version. """ + env =3D os.environ.copy() + + # The sphinx-build tool has a bug: internally, it tries to set + # locale with locale.setlocale(locale.LC_ALL, ''). This causes a + # crash if language is not set. Detect and fix it. try: - result =3D self.run([cmd, "--version"], + locale.setlocale(locale.LC_ALL, '') + except Exception: + env["LC_ALL"] =3D "C" + env["LANG"] =3D "C" + + try: + result =3D self.run([cmd, "--version"], env=3Denv, stdout=3Dsubprocess.PIPE, stderr=3Dsubprocess.STDOUT, text=3DTrue, check=3DTrue) --=20 2.51.0 From nobody Fri Oct 3 20:22:42 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC7D7301017; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; cv=none; b=MIHCN7BsAdNFpNEhUfJwiP6lrkC3B4GJ8wF1E1g3W51oWkehcx0D0Oqw/54sILdzInhPUVpk/oergc3aD6XJ/Ofa6TZ9klBJ9kkNOl18OeQY3jPZdlwEcljmrHTbnODw12Ok2YO8qgA5iby9i4Y1vU9Xtx6in+L+jv50Qjc0sek= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756139449; c=relaxed/simple; bh=yZuejI+7kMNbMkVJynlYhnA8shfEcPMt1jbLRCzsGuw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Q0fvhBUT4EAc3WVq6TvONzli5k8q7QF1ZiGxBMxbOecUchd78kFUOYuPlS5XPDJTfABgKWpnVOK29oVQVODs3GF59sFREAxRSpc90YtxQnlAzEodoVd341A5GKSz73e068tffWDUaMEOQrTBMg5WN2hTy1QmptcxhxM3hAv33fA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=TdiQDSOm; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="TdiQDSOm" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 35B42C2BC87; Mon, 25 Aug 2025 16:30:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1756139449; bh=yZuejI+7kMNbMkVJynlYhnA8shfEcPMt1jbLRCzsGuw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TdiQDSOmSg1XQkkMPpJR2N4uS5LSKYTpxgCprQJy/sagPQlzMzboEtwG4rvtkF4PP 3JTxRyvhIO15DIzeMCMrYSmo7iPbvuPGDye8au+ZoAc5j6OrtDuIInDVB59Wc0/fMX K/A9TAw+6uVGrZ9QjakXcvAfcAtPiX1qc6Jy/SfuOAMQcB9N2lLzjsQOJg0P6Ps62h VnTR6UHe36CZcpnD/ubmNR1f3dkemkG+oMOn2eIrtpOWs0KS/2sLgtIbGC2EiiHZMW mmZQB+qGYQtWwtfz/YnwdGkSnkYbDpmLI4yjWej9MOeQUYSDrklzuC3+hvW1jaT7po KmTkLBQsP3TDQ== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uqa5n-0000000HAM4-21Ev; Mon, 25 Aug 2025 18:30:47 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Jonathan Corbet" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v2 13/13] tools/docs: sphinx-build-wrapper: allow building PDF files in parallel Date: Mon, 25 Aug 2025 18:30:40 +0200 Message-ID: <1b5a3721b7aad0efbf9cc6fde30328c244160783.1756138805.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Use POSIX jobserver when available or -j to run PDF builds in parallel, restoring pdf build performance. Yet, running it when debugging troubles is a bad idea, so, when calling directly via command line, except if "-j" is splicitly requested, it will serialize the build. With such change, a PDF doc builds now takes around 5 minutes on a Ryzen 9 machine with 32 cpu threads: # Explicitly paralelize both Sphinx and LaTeX pdf builds $ make cleandocs; time scripts/sphinx-build-wrapper pdfdocs -j 33 real 5m17.901s user 15m1.499s sys 2m31.482s # Use POSIX jobserver to paralelize both sphinx-build and LaTeX $ make cleandocs; time make pdfdocs real 5m22.369s user 15m9.076s sys 2m31.419s # Serializes PDF build, while keeping Sphinx parallelized. # it is equivalent of passing -jauto via command line $ make cleandocs; time scripts/sphinx-build-wrapper pdfdocs real 11m20.901s user 13m2.910s sys 1m44.553s Signed-off-by: Mauro Carvalho Chehab --- tools/docs/sphinx-build-wrapper | 158 +++++++++++++++++++++++--------- 1 file changed, 117 insertions(+), 41 deletions(-) diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrap= per index 580582a76c93..c884022ad733 100755 --- a/tools/docs/sphinx-build-wrapper +++ b/tools/docs/sphinx-build-wrapper @@ -52,6 +52,8 @@ import shutil import subprocess import sys =20 +from concurrent import futures + from lib.python_version import PythonVersion =20 LIB_DIR =3D "../../scripts/lib" @@ -278,6 +280,82 @@ class SphinxBuilder: except (OSError, IOError) as e: print(f"Warning: Failed to copy CSS: {e}", file=3Dsys.stderr) =20 + def build_pdf_file(self, latex_cmd, from_dir, path): + """Builds a single pdf file using latex_cmd""" + try: + subprocess.run(latex_cmd + [path], + cwd=3Dfrom_dir, check=3DTrue) + + return True + except subprocess.CalledProcessError: + return False + + def pdf_parallel_build(self, tex_suffix, latex_cmd, tex_files, n_jobs): + """Build PDF files in parallel if possible""" + builds =3D {} + build_failed =3D False + max_len =3D 0 + has_tex =3D False + + # + # LaTeX PDF error code is almost useless for us: + # any warning makes it non-zero. For kernel doc builds it always r= eturn + # non-zero even when build succeeds. So, let's do the best next th= ing: + # Ignore build errors. At the end, check if all PDF files were bui= lt, + # printing a summary with the built ones and returning 0 if all of + # them were actually built. + # + with futures.ThreadPoolExecutor(max_workers=3Dn_jobs) as executor: + jobs =3D {} + + for from_dir, pdf_dir, entry in tex_files: + name =3D entry.name + + if not name.endswith(tex_suffix): + continue + + name =3D name[:-len(tex_suffix)] + + max_len =3D max(max_len, len(name)) + + has_tex =3D True + + future =3D executor.submit(self.build_pdf_file, latex_cmd, + from_dir, entry.path) + jobs[future] =3D (from_dir, pdf_dir, name) + + for future in futures.as_completed(jobs): + from_dir, pdf_dir, name =3D jobs[future] + + pdf_name =3D name + ".pdf" + pdf_from =3D os.path.join(from_dir, pdf_name) + + try: + success =3D future.result() + + if success and os.path.exists(pdf_from): + pdf_to =3D os.path.join(pdf_dir, pdf_name) + + os.rename(pdf_from, pdf_to) + builds[name] =3D os.path.relpath(pdf_to, self.buil= ddir) + else: + builds[name] =3D "FAILED" + build_failed =3D True + except futures.Error as e: + builds[name] =3D f"FAILED ({repr(e)})" + build_failed =3D True + + # + # Handle case where no .tex files were found + # + if not has_tex: + name =3D "Sphinx LaTeX builder" + max_len =3D max(max_len, len(name)) + builds[name] =3D "FAILED (no .tex file was generated)" + build_failed =3D True + + return builds, build_failed, max_len + def handle_pdf(self, output_dirs): """ Extra steps for PDF output. @@ -288,7 +366,9 @@ class SphinxBuilder: """ builds =3D {} max_len =3D 0 + tex_suffix =3D ".tex" =20 + tex_files =3D [] for from_dir in output_dirs: pdf_dir =3D os.path.join(from_dir, "../pdf") os.makedirs(pdf_dir, exist_ok=3DTrue) @@ -300,55 +380,51 @@ class SphinxBuilder: =20 latex_cmd.extend(shlex.split(self.latexopts)) =20 - tex_suffix =3D ".tex" - - # - # Process each .tex file - # - - has_tex =3D False - build_failed =3D False + # Get a list of tex files to process with os.scandir(from_dir) as it: for entry in it: - if not entry.name.endswith(tex_suffix): - continue + if entry.name.endswith(tex_suffix): + tex_files.append((from_dir, pdf_dir, entry)) =20 - name =3D entry.name[:-len(tex_suffix)] - has_tex =3D True + # + # When using make, this won't be used, as the number of jobs comes + # from POSIX jobserver. So, this covers the case where build comes + # from command line. On such case, serialize by default, except if + # the user explicitly sets the number of jobs. + # + n_jobs =3D 1 =20 - # - # LaTeX PDF error code is almost useless for us: - # any warning makes it non-zero. For kernel doc builds= it - # always return non-zero even when build succeeds. - # So, let's do the best next thing: check if all PDF - # files were built. If they're, print a summary and - # return 0 at the end of this function - # - try: - subprocess.run(latex_cmd + [entry.path], - cwd=3Dfrom_dir, check=3DTrue) - except subprocess.CalledProcessError: - pass + # n_jobs is either an integer or "auto". Only use it if it is a nu= mber + if self.n_jobs: + try: + n_jobs =3D int(self.n_jobs) + except ValueError: + pass =20 - pdf_name =3D name + ".pdf" - pdf_from =3D os.path.join(from_dir, pdf_name) - pdf_to =3D os.path.join(pdf_dir, pdf_name) + # + # When using make, jobserver.claim is the number of jobs that were + # used with "-j" and that aren't used by other make targets + # + with JobserverExec() as jobserver: + n_jobs =3D 1 =20 - if os.path.exists(pdf_from): - os.rename(pdf_from, pdf_to) - builds[name] =3D os.path.relpath(pdf_to, self.buil= ddir) - else: - builds[name] =3D "FAILED" - build_failed =3D True + # + # Handle the case when a parameter is passed via command line, + # using it as default, if jobserver doesn't claim anything + # + if self.n_jobs: + try: + n_jobs =3D int(self.n_jobs) + except ValueError: + pass =20 - name =3D entry.name.removesuffix(".tex") - max_len =3D max(max_len, len(name)) + if jobserver.claim: + n_jobs =3D jobserver.claim =20 - if not has_tex: - name =3D os.path.basename(from_dir) - max_len =3D max(max_len, len(name)) - builds[name] =3D "FAILED (no .tex)" - build_failed =3D True + builds, build_failed, max_len =3D self.pdf_parallel_build(tex_= suffix, + latex_= cmd, + tex_fi= les, + n_jobs) =20 msg =3D "Summary" msg +=3D "\n" + "=3D" * len(msg) --=20 2.51.0