From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B4872BF012; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; cv=none; b=srx74jg/nklgIpr+UH8CetVDZAY4xlVFQxsIY3T3Yt2TYVvhcWfk4VXV3fM9lBCGKLngkL4zOCG4RXJXb+ubfHFmcdXb7DAbdhQnVZEuutZIOxDRpS/8QJJJCjfvBjW0KZGrmR4FJfbWd78P5E4n30THvvZ23CuhXkI1Ae4TKWc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; c=relaxed/simple; bh=9mDPdBlDyrg/aeHqaYVYg/sDx78PrFwDonVBGcuS+OM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=B41rSbBXy4A+MdgwKUCZcwNhs9aK9We5VmifNUuLwXIbspPB8I0kvjrU5VZdf9dNVEbTwCk2E7OtYcMWbVvgSgJGHsva8ByPaByGRRwUa+cFtz1Nb7/3Ic3U8M3VE7VOoIyCfJ5q1RNuvBu8J6v0zioXIen1jeNfleXdaa565jU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=aEhSktRZ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="aEhSktRZ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BAA07C4AF0B; Thu, 18 Sep 2025 11:55:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196505; bh=9mDPdBlDyrg/aeHqaYVYg/sDx78PrFwDonVBGcuS+OM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=aEhSktRZDQG63xHJX41+kgYmOD+fpXC5tVl/bv39UD7+wCbkWFX8ji1MG9ZHsYYLb pPPlCEap+NIza9nRGGb03RNaYvQe18CS6jtbFOFPn4VoeVM87tEL5R9TK0j0Bu2jYz yyJfao1vz6HlSnlbVTr1dE8Lt3IUwOfBSBgqa9/GmbzR9AuNRgfrwvfMf5p2iI8LmH ZTxkU83y60H6bpIoqZBeG5qO8LjPNM+FA3zE2RPMmD2AETFesx0VDN/V6W8bnZONBy EBgiHge9MsAowrtg3kT+LVHWd9cV1J5VAZq3DXh9du6K8YjoTF1tkycx8DvR52mCze +gwmWzpI5NDOA== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE7-0000000CrkL-35aZ; Thu, 18 Sep 2025 13:55:03 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v8 01/24] scripts/jobserver-exec: move the code to a class Date: Thu, 18 Sep 2025 13:54:35 +0200 Message-ID: <4749921b75d4e0bd85a25d4d94aa2c940fad084e.1758196090.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Convert the code inside jobserver-exec to a class and properly document it. Using a class allows reusing the jobserver logic on other scripts. While the main code remains unchanged, being compatible with Python 2.6 and 3.0+, its coding style now follows a more modern standard, having tabs replaced by a 4-spaces indent, passing autopep8, black and pylint. The code allows using a pythonic way to enter/exit a python code, e.g. it now supports: with JobserverExec() as jobserver: jobserver.run(sys.argv[1:]) With the new code, the __exit__() function should ensure that the jobserver slot will be closed at the end, even if something bad happens somewhere. Signed-off-by: Mauro Carvalho Chehab --- scripts/jobserver-exec | 218 ++++++++++++++++++++++++++++------------- 1 file changed, 151 insertions(+), 67 deletions(-) diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec index 7eca035472d3..897c0cca9e6e 100755 --- a/scripts/jobserver-exec +++ b/scripts/jobserver-exec @@ -1,77 +1,161 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0+ # +# pylint: disable=3DC0103,C0209 +# # This determines how many parallel tasks "make" is expecting, as it is # not exposed via an special variables, reserves them all, runs a subproce= ss # with PARALLELISM environment variable set, and releases the jobs back ag= ain. # # https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#= POSIX-Jobserver -from __future__ import print_function -import os, sys, errno + +""" +Interacts with the POSIX jobserver during the Kernel build time. + +A "normal" jobserver task, like the one initiated by a make subrocess woul= d do: + + - open read/write file descriptors to communicate with the job server; + - ask for one slot by calling: + claim =3D os.read(reader, 1) + - when the job finshes, call: + os.write(writer, b"+") # os.write(writer, claim) + +Here, the goal is different: This script aims to get the remaining number +of slots available, using all of them to run a command which handle tasks = in +parallel. To to that, it has a loop that ends only after there are no +slots left. It then increments the number by one, in order to allow a +call equivalent to make -j$((claim+1)), e.g. having a parent make creating +$claim child to do the actual work. + +The end goal here is to keep the total number of build tasks under the +limit established by the initial make -j$n_proc call. +""" + +import errno +import os import subprocess +import sys =20 -# Extract and prepare jobserver file descriptors from environment. -claim =3D 0 -jobs =3D b"" -try: - # Fetch the make environment options. - flags =3D os.environ['MAKEFLAGS'] - - # Look for "--jobserver=3DR,W" - # Note that GNU Make has used --jobserver-fds and --jobserver-auth - # so this handles all of them. - opts =3D [x for x in flags.split(" ") if x.startswith("--jobserver")] - - # Parse out R,W file descriptor numbers and set them nonblocking. - # If the MAKEFLAGS variable contains multiple instances of the - # --jobserver-auth=3D option, the last one is relevant. - fds =3D opts[-1].split("=3D", 1)[1] - - # Starting with GNU Make 4.4, named pipes are used for reader and writer. - # Example argument: --jobserver-auth=3Dfifo:/tmp/GMfifo8134 - _, _, path =3D fds.partition('fifo:') - - if path: - reader =3D os.open(path, os.O_RDONLY | os.O_NONBLOCK) - writer =3D os.open(path, os.O_WRONLY) - else: - reader, writer =3D [int(x) for x in fds.split(",", 1)] - # Open a private copy of reader to avoid setting nonblocking - # on an unexpecting process with the same reader fd. - reader =3D os.open("/proc/self/fd/%d" % (reader), - os.O_RDONLY | os.O_NONBLOCK) - - # Read out as many jobserver slots as possible. - while True: - try: - slot =3D os.read(reader, 8) - jobs +=3D slot - except (OSError, IOError) as e: - if e.errno =3D=3D errno.EWOULDBLOCK: - # Stop at the end of the jobserver queue. - break - # If something went wrong, give back the jobs. - if len(jobs): - os.write(writer, jobs) - raise e - # Add a bump for our caller's reserveration, since we're just going - # to sit here blocked on our child. - claim =3D len(jobs) + 1 -except (KeyError, IndexError, ValueError, OSError, IOError) as e: - # Any missing environment strings or bad fds should result in just - # not being parallel. - pass - -# We can only claim parallelism if there was a jobserver (i.e. a top-level -# "-jN" argument) and there were no other failures. Otherwise leave out the -# environment variable and let the child figure out what is best. -if claim > 0: - os.environ['PARALLELISM'] =3D '%d' % (claim) - -rc =3D subprocess.call(sys.argv[1:]) - -# Return all the reserved slots. -if len(jobs): - os.write(writer, jobs) - -sys.exit(rc) + +class JobserverExec: + """ + Claim all slots from make using POSIX Jobserver. + + The main methods here are: + - open(): reserves all slots; + - close(): method returns all used slots back to make; + - run(): executes a command setting PARALLELISM=3D + """ + + def __init__(self): + """Initialize internal vars""" + self.claim =3D 0 + self.jobs =3D b"" + self.reader =3D None + self.writer =3D None + self.is_open =3D False + + def open(self): + """Reserve all available slots to be claimed later on""" + + if self.is_open: + return + + try: + # Fetch the make environment options. + flags =3D os.environ["MAKEFLAGS"] + # Look for "--jobserver=3DR,W" + # Note that GNU Make has used --jobserver-fds and --jobserver-= auth + # so this handles all of them. + opts =3D [x for x in flags.split(" ") if x.startswith("--jobse= rver")] + + # Parse out R,W file descriptor numbers and set them nonblocki= ng. + # If the MAKEFLAGS variable contains multiple instances of the + # --jobserver-auth=3D option, the last one is relevant. + fds =3D opts[-1].split("=3D", 1)[1] + + # Starting with GNU Make 4.4, named pipes are used for reader + # and writer. + # Example argument: --jobserver-auth=3Dfifo:/tmp/GMfifo8134 + _, _, path =3D fds.partition("fifo:") + + if path: + self.reader =3D os.open(path, os.O_RDONLY | os.O_NONBLOCK) + self.writer =3D os.open(path, os.O_WRONLY) + else: + self.reader, self.writer =3D [int(x) for x in fds.split(",= ", 1)] + # Open a private copy of reader to avoid setting nonblocki= ng + # on an unexpecting process with the same reader fd. + self.reader =3D os.open("/proc/self/fd/%d" % (self.reader), + os.O_RDONLY | os.O_NONBLOCK) + + # Read out as many jobserver slots as possible + while True: + try: + slot =3D os.read(self.reader, 8) + self.jobs +=3D slot + except (OSError, IOError) as e: + if e.errno =3D=3D errno.EWOULDBLOCK: + # Stop at the end of the jobserver queue. + break + # If something went wrong, give back the jobs. + if self.jobs: + os.write(self.writer, self.jobs) + raise e + + # Add a bump for our caller's reserveration, since we're just = going + # to sit here blocked on our child. + self.claim =3D len(self.jobs) + 1 + + except (KeyError, IndexError, ValueError, OSError, IOError): + # Any missing environment strings or bad fds should result in = just + # not being parallel. + self.claim =3D None + + self.is_open =3D True + + def close(self): + """Return all reserved slots to Jobserver""" + + if not self.is_open: + return + + # Return all the reserved slots. + if len(self.jobs): + os.write(self.writer, self.jobs) + + self.is_open =3D False + + def __enter__(self): + self.open() + return self + + def __exit__(self, exc_type, exc_value, exc_traceback): + self.close() + + def run(self, cmd): + """ + Run a command setting PARALLELISM env variable to the number of + available job slots (claim) + 1, e.g. it will reserve claim slots + to do the actual build work, plus one to monitor its children. + """ + self.open() # Ensure that self.claim is set + + # We can only claim parallelism if there was a jobserver (i.e. a + # top-level "-jN" argument) and there were no other failures. Othe= rwise + # leave out the environment variable and let the child figure out = what + # is best. + if self.claim: + os.environ["PARALLELISM"] =3D str(self.claim) + + return subprocess.call(cmd) + + +def main(): + """Main program""" + with JobserverExec() as jobserver: + jobserver.run(sys.argv[1:]) + + +if __name__ =3D=3D "__main__": + main() --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B4F62C0263; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; cv=none; b=JRe0jHLLIZDG7Ps7ShpKvgOqsN/++DuO4vJ4eQr+AOWILiWr6Jo8PesR1dzs+GOl6nCzHtqrT2Aamp+vcjeA/+1vwcchdRRzZjmQajYoYIOZmIMExBhSw4TD4CxV5WqvOzBSQiKIlHBRtTih6y9MU2zvsYI7kQKBHvS/ZSTcR1g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; c=relaxed/simple; bh=kdzw8bKSlWfCx7XdnNlorMPC4frNn6vVlI1VZc7nnA8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bTVsTMQX94W+EzuJHatyXqDQKwLB8kHxsSP/FIxsTxYCjHRekaD+BabhQn7GItPv0UG0POI9650WrWE33VhMk7ViU0Va6vTPuvusw3w2h/weHfsGu5LW09D2S5WXgwZP98ZbDTrbmyOrYUWVUV3h+6b0Qv5edvq93nMRoRFrjzI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=dD+ZNBhg; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="dD+ZNBhg" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B3703C4CEFA; Thu, 18 Sep 2025 11:55:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196505; bh=kdzw8bKSlWfCx7XdnNlorMPC4frNn6vVlI1VZc7nnA8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dD+ZNBhgmWBrLWtfUrNXdI9FreVY0CT4sXq2iwN8gLuoDcxxFT/17yB+kIzG2W1yT lSkpLMal3mY78+jprUi2oyPc2Av5irsZlGSWwIqM0LEWtrsepj1ixPrni2oEqBw6iB 3LE4uvVWzNTLAuS/sZ3uJjtyXWAbjMmXh1x8PL5lE9qMYoCfUjmDrMhfpc3gNL8XtH IEOX3PJIJW0no2UDr7R0imdm20KcePDuUgU6w9ovy+BczMzS/Nc5v9UwMtoOPp6ohU CgOtzqwpLep0YB6GqwKkFLNB4rwOQnTLyDeN7tO+PHBd+U3bn61w4ibKDJh6OWETmb UwsQ753DFugEQ== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE7-0000000Crkv-3CUx; Thu, 18 Sep 2025 13:55:03 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v8 02/24] scripts/jobserver-exec: move its class to the lib directory Date: Thu, 18 Sep 2025 13:54:36 +0200 Message-ID: <6be7b161b6c005a9807162ebfd239af6a4e6fa47.1758196090.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" To make it easier to be re-used, move the JobserverExec class to the library directory. Signed-off-by: Mauro Carvalho Chehab --- scripts/jobserver-exec | 152 +++------------------------------------ scripts/lib/jobserver.py | 149 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 160 insertions(+), 141 deletions(-) create mode 100755 scripts/lib/jobserver.py diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec index 897c0cca9e6e..40a0f0058733 100755 --- a/scripts/jobserver-exec +++ b/scripts/jobserver-exec @@ -1,155 +1,25 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0+ -# -# pylint: disable=3DC0103,C0209 -# -# This determines how many parallel tasks "make" is expecting, as it is -# not exposed via an special variables, reserves them all, runs a subproce= ss -# with PARALLELISM environment variable set, and releases the jobs back ag= ain. -# -# https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#= POSIX-Jobserver =20 -""" -Interacts with the POSIX jobserver during the Kernel build time. - -A "normal" jobserver task, like the one initiated by a make subrocess woul= d do: - - - open read/write file descriptors to communicate with the job server; - - ask for one slot by calling: - claim =3D os.read(reader, 1) - - when the job finshes, call: - os.write(writer, b"+") # os.write(writer, claim) - -Here, the goal is different: This script aims to get the remaining number -of slots available, using all of them to run a command which handle tasks = in -parallel. To to that, it has a loop that ends only after there are no -slots left. It then increments the number by one, in order to allow a -call equivalent to make -j$((claim+1)), e.g. having a parent make creating -$claim child to do the actual work. - -The end goal here is to keep the total number of build tasks under the -limit established by the initial make -j$n_proc call. -""" - -import errno import os -import subprocess import sys =20 +LIB_DIR =3D "lib" +SRC_DIR =3D os.path.dirname(os.path.realpath(__file__)) =20 -class JobserverExec: - """ - Claim all slots from make using POSIX Jobserver. +sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR)) =20 - The main methods here are: - - open(): reserves all slots; - - close(): method returns all used slots back to make; - - run(): executes a command setting PARALLELISM=3D - """ +from jobserver import JobserverExec # pylint: disable=3DC= 0415 =20 - def __init__(self): - """Initialize internal vars""" - self.claim =3D 0 - self.jobs =3D b"" - self.reader =3D None - self.writer =3D None - self.is_open =3D False =20 - def open(self): - """Reserve all available slots to be claimed later on""" - - if self.is_open: - return - - try: - # Fetch the make environment options. - flags =3D os.environ["MAKEFLAGS"] - # Look for "--jobserver=3DR,W" - # Note that GNU Make has used --jobserver-fds and --jobserver-= auth - # so this handles all of them. - opts =3D [x for x in flags.split(" ") if x.startswith("--jobse= rver")] - - # Parse out R,W file descriptor numbers and set them nonblocki= ng. - # If the MAKEFLAGS variable contains multiple instances of the - # --jobserver-auth=3D option, the last one is relevant. - fds =3D opts[-1].split("=3D", 1)[1] - - # Starting with GNU Make 4.4, named pipes are used for reader - # and writer. - # Example argument: --jobserver-auth=3Dfifo:/tmp/GMfifo8134 - _, _, path =3D fds.partition("fifo:") - - if path: - self.reader =3D os.open(path, os.O_RDONLY | os.O_NONBLOCK) - self.writer =3D os.open(path, os.O_WRONLY) - else: - self.reader, self.writer =3D [int(x) for x in fds.split(",= ", 1)] - # Open a private copy of reader to avoid setting nonblocki= ng - # on an unexpecting process with the same reader fd. - self.reader =3D os.open("/proc/self/fd/%d" % (self.reader), - os.O_RDONLY | os.O_NONBLOCK) - - # Read out as many jobserver slots as possible - while True: - try: - slot =3D os.read(self.reader, 8) - self.jobs +=3D slot - except (OSError, IOError) as e: - if e.errno =3D=3D errno.EWOULDBLOCK: - # Stop at the end of the jobserver queue. - break - # If something went wrong, give back the jobs. - if self.jobs: - os.write(self.writer, self.jobs) - raise e - - # Add a bump for our caller's reserveration, since we're just = going - # to sit here blocked on our child. - self.claim =3D len(self.jobs) + 1 - - except (KeyError, IndexError, ValueError, OSError, IOError): - # Any missing environment strings or bad fds should result in = just - # not being parallel. - self.claim =3D None - - self.is_open =3D True - - def close(self): - """Return all reserved slots to Jobserver""" - - if not self.is_open: - return - - # Return all the reserved slots. - if len(self.jobs): - os.write(self.writer, self.jobs) - - self.is_open =3D False - - def __enter__(self): - self.open() - return self - - def __exit__(self, exc_type, exc_value, exc_traceback): - self.close() - - def run(self, cmd): - """ - Run a command setting PARALLELISM env variable to the number of - available job slots (claim) + 1, e.g. it will reserve claim slots - to do the actual build work, plus one to monitor its children. - """ - self.open() # Ensure that self.claim is set - - # We can only claim parallelism if there was a jobserver (i.e. a - # top-level "-jN" argument) and there were no other failures. Othe= rwise - # leave out the environment variable and let the child figure out = what - # is best. - if self.claim: - os.environ["PARALLELISM"] =3D str(self.claim) - - return subprocess.call(cmd) +""" +Determines how many parallel tasks "make" is expecting, as it is +not exposed via an special variables, reserves them all, runs a subprocess +with PARALLELISM environment variable set, and releases the jobs back agai= n. =20 +See: + https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.htm= l#POSIX-Jobserver +""" =20 def main(): """Main program""" diff --git a/scripts/lib/jobserver.py b/scripts/lib/jobserver.py new file mode 100755 index 000000000000..a24f30ef4fa8 --- /dev/null +++ b/scripts/lib/jobserver.py @@ -0,0 +1,149 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0+ +# +# pylint: disable=3DC0103,C0209 +# +# + +""" +Interacts with the POSIX jobserver during the Kernel build time. + +A "normal" jobserver task, like the one initiated by a make subrocess woul= d do: + + - open read/write file descriptors to communicate with the job server; + - ask for one slot by calling: + claim =3D os.read(reader, 1) + - when the job finshes, call: + os.write(writer, b"+") # os.write(writer, claim) + +Here, the goal is different: This script aims to get the remaining number +of slots available, using all of them to run a command which handle tasks = in +parallel. To to that, it has a loop that ends only after there are no +slots left. It then increments the number by one, in order to allow a +call equivalent to make -j$((claim+1)), e.g. having a parent make creating +$claim child to do the actual work. + +The end goal here is to keep the total number of build tasks under the +limit established by the initial make -j$n_proc call. + +See: + https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.htm= l#POSIX-Jobserver +""" + +import errno +import os +import subprocess +import sys + +class JobserverExec: + """ + Claim all slots from make using POSIX Jobserver. + + The main methods here are: + - open(): reserves all slots; + - close(): method returns all used slots back to make; + - run(): executes a command setting PARALLELISM=3D + """ + + def __init__(self): + """Initialize internal vars""" + self.claim =3D 0 + self.jobs =3D b"" + self.reader =3D None + self.writer =3D None + self.is_open =3D False + + def open(self): + """Reserve all available slots to be claimed later on""" + + if self.is_open: + return + + try: + # Fetch the make environment options. + flags =3D os.environ["MAKEFLAGS"] + # Look for "--jobserver=3DR,W" + # Note that GNU Make has used --jobserver-fds and --jobserver-= auth + # so this handles all of them. + opts =3D [x for x in flags.split(" ") if x.startswith("--jobse= rver")] + + # Parse out R,W file descriptor numbers and set them nonblocki= ng. + # If the MAKEFLAGS variable contains multiple instances of the + # --jobserver-auth=3D option, the last one is relevant. + fds =3D opts[-1].split("=3D", 1)[1] + + # Starting with GNU Make 4.4, named pipes are used for reader + # and writer. + # Example argument: --jobserver-auth=3Dfifo:/tmp/GMfifo8134 + _, _, path =3D fds.partition("fifo:") + + if path: + self.reader =3D os.open(path, os.O_RDONLY | os.O_NONBLOCK) + self.writer =3D os.open(path, os.O_WRONLY) + else: + self.reader, self.writer =3D [int(x) for x in fds.split(",= ", 1)] + # Open a private copy of reader to avoid setting nonblocki= ng + # on an unexpecting process with the same reader fd. + self.reader =3D os.open("/proc/self/fd/%d" % (self.reader), + os.O_RDONLY | os.O_NONBLOCK) + + # Read out as many jobserver slots as possible + while True: + try: + slot =3D os.read(self.reader, 8) + self.jobs +=3D slot + except (OSError, IOError) as e: + if e.errno =3D=3D errno.EWOULDBLOCK: + # Stop at the end of the jobserver queue. + break + # If something went wrong, give back the jobs. + if self.jobs: + os.write(self.writer, self.jobs) + raise e + + # Add a bump for our caller's reserveration, since we're just = going + # to sit here blocked on our child. + self.claim =3D len(self.jobs) + 1 + + except (KeyError, IndexError, ValueError, OSError, IOError): + # Any missing environment strings or bad fds should result in = just + # not being parallel. + self.claim =3D None + + self.is_open =3D True + + def close(self): + """Return all reserved slots to Jobserver""" + + if not self.is_open: + return + + # Return all the reserved slots. + if len(self.jobs): + os.write(self.writer, self.jobs) + + self.is_open =3D False + + def __enter__(self): + self.open() + return self + + def __exit__(self, exc_type, exc_value, exc_traceback): + self.close() + + def run(self, cmd, *args, **pwargs): + """ + Run a command setting PARALLELISM env variable to the number of + available job slots (claim) + 1, e.g. it will reserve claim slots + to do the actual build work, plus one to monitor its children. + """ + self.open() # Ensure that self.claim is set + + # We can only claim parallelism if there was a jobserver (i.e. a + # top-level "-jN" argument) and there were no other failures. Othe= rwise + # leave out the environment variable and let the child figure out = what + # is best. + if self.claim: + os.environ["PARALLELISM"] =3D str(self.claim) + + return subprocess.call(cmd, *args, **pwargs) --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 21303241696; Thu, 18 Sep 2025 11:55:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; cv=none; b=CejpaBv7PSc3Y/kxw+8xB4dLO5MsigJ5AsVxTenq/xPMBnH+8p7lWe0cPaO/UskXmpAI/E/ZVQYv0Ak2hnw0u0Aky33UqBiCXoTBzDWgza9GjLlRr/NACQbv1myF9BkXVc9NhkQ3Hd2dA2LWkMIZDRICRwA+gwV3otVx1bPBCFE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; c=relaxed/simple; bh=jlol1TuhVNcZDXlNqsLBvYWROc27W0iXgTQpgdIUTbU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lmRlWkpObYrRZT0JRuUvolHD3SbkRrGpjRAFLGTnQlLW4mSbdWD2CRhn36r12HvvOH8qQLoLhR4HQZkCCfVOZg+YSSVPrHwTiYsh/mpIKhv+kxU91OMMFNCwz6v+tlFGc7NhcmPiVDDau+4DF0ueJ0lWycyyRdStNszc3vqu3Fk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=eU4nKEuj; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="eU4nKEuj" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B09C7C4CEEB; Thu, 18 Sep 2025 11:55:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196505; bh=jlol1TuhVNcZDXlNqsLBvYWROc27W0iXgTQpgdIUTbU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eU4nKEujOkGza7zI9NVXrOrMjS8/xIaX1IqMt/mLaPZ2MoJQpakuYI+VC06hxEPeb 3VNhOAMKZORuKrGeTdOaYKn2iRxvIKa2Uql2yuj3KAOjRTtTvrjdaaR5rkXJZPzOZT U3yXLTjeyexWCMvMRPZE9DyqR8IJuVwq5rfteHx94JjxHHuTBd3zGGfKLqQcQWdOyp XDZNxSqIsaVLzyJHnKtbK2a0dDPhme/8fiNNOFff03UAhYpMiBahJs25Nolh9ruEkN j98mIx0FVuTvzKB/tyemyXbN0IaPWdaMq0pTJJuOasFYcLkumnKWlI/CPFsc++d49K AJ7b0KMjsnnyg== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE7-0000000CrlR-3JIW; Thu, 18 Sep 2025 13:55:03 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v8 03/24] scripts/jobserver-exec: add a help message Date: Thu, 18 Sep 2025 13:54:37 +0200 Message-ID: <64b0339eac54ac0f2b3de3667a7f4f5becb1c6ae.1758196090.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Currently, calling it without an argument shows an ugly error message. Instead, print a message using pythondoc as description. Signed-off-by: Mauro Carvalho Chehab --- scripts/jobserver-exec | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec index 40a0f0058733..ae23afd344ec 100755 --- a/scripts/jobserver-exec +++ b/scripts/jobserver-exec @@ -1,6 +1,15 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0+ =20 +""" +Determines how many parallel tasks "make" is expecting, as it is +not exposed via any special variables, reserves them all, runs a subprocess +with PARALLELISM environment variable set, and releases the jobs back agai= n. + +See: + https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.htm= l#POSIX-Jobserver +""" + import os import sys =20 @@ -12,17 +21,12 @@ sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR)) from jobserver import JobserverExec # pylint: disable=3DC= 0415 =20 =20 -""" -Determines how many parallel tasks "make" is expecting, as it is -not exposed via an special variables, reserves them all, runs a subprocess -with PARALLELISM environment variable set, and releases the jobs back agai= n. - -See: - https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.htm= l#POSIX-Jobserver -""" - def main(): """Main program""" + if len(sys.argv) < 2: + name =3D os.path.basename(__file__) + sys.exit("usage: " + name +" command [args ...]\n" + __doc__) + with JobserverExec() as jobserver: jobserver.run(sys.argv[1:]) =20 --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2129E221F29; Thu, 18 Sep 2025 11:55:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; cv=none; b=PhD8AOCHW8MjDEebM9Jz9Sph+wwsZ5rXilDP9CExAZXnvs4lwh/+v7FRyIq3tODTT1RZrvvLp3h8D7WtiG0njTbkw0Wpgq4hCZCCPo1OP11gSHAqOAN/jVKMY+VzKOHMOFkunPPF7a13GFGlfBxydtFTjVAGIM+PgwZoKYgsaio= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; c=relaxed/simple; bh=fJpsAtJ92MxZvD5oc+zobQhN2JKwDEF9zPLyP1fCyLQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=b9nY3aIQyG4cPDMExpsS+DGZ/Im6gKanU3o62HOJrnwqLgqCESq7gTfITcR7Er5MKSLhs4LXgLjaVNtTmMNcsK8Yawc3r3zBTqLQJanyOCM7b06XGtEftNCcgG3kFmPUQcK1YkJEQLsaba9dMaw2Q+VSkc7I1UM/HKFKzJ1k1xo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=MxHhn/GM; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="MxHhn/GM" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B832EC4AF09; Thu, 18 Sep 2025 11:55:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196505; bh=fJpsAtJ92MxZvD5oc+zobQhN2JKwDEF9zPLyP1fCyLQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MxHhn/GMdKawhvKBilWREK7SzxI9E/gmHlQ5j0punTXANvW5i2dAMFgVSlCwKEDI4 D3OpMA1rrVqwG+w5+98l7VYi/PBOTjH2vunSMsNrMaK2UfVVLOV/yyhO+ViQm7Gjhq E2LA8RYfkojG07Dhw/+Rnof/7yraKMyWfep0ir5VyDh+X/WdtixRwZgyE5pN1Jf+SV FRlheAdgCymEAwPmf6YvgG0+HNMekeoiv1plvCNESVTf3dVd7Ib2qtpsf4qSi148SF mrnDjW15k+DV+wAMJibKb9gTTweanZucv6X1S8pBMEDkma1M/al8kn8waLgKp0V5bI tMOG8QDRzhXDA== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE7-0000000Crlw-3QCk; Thu, 18 Sep 2025 13:55:03 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v8 04/24] scripts: check-variable-fonts.sh: convert to Python Date: Thu, 18 Sep 2025 13:54:38 +0200 Message-ID: X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" This script handle errors when trying to build translations with make pdfdocs. As part of our cleanup work to remove hacks from docs Makefile, convert this to python, preparing it to be part of a library to be called by sphinx-build-wrapper. Signed-off-by: Mauro Carvalho Chehab --- Documentation/Makefile | 2 +- MAINTAINERS | 2 +- ...iable-fonts.sh =3D> check-variable-fonts.py} | 104 +++++++++++++----- 3 files changed, 79 insertions(+), 29 deletions(-) rename scripts/{check-variable-fonts.sh =3D> check-variable-fonts.py} (61%) diff --git a/Documentation/Makefile b/Documentation/Makefile index 5c20c68be89a..d5e436435eab 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -146,7 +146,7 @@ pdfdocs: DENY_VF =3D XDG_CONFIG_HOME=3D$(FONTS_CONF_DEN= Y_VF) pdfdocs: latexdocs @$(srctree)/scripts/sphinx-pre-install --version-check $(foreach var,$(SPHINXDIRS), \ - $(MAKE) PDFLATEX=3D"$(PDFLATEX)" LATEXOPTS=3D"$(LATEXOPTS)" $(DENY_VF)= -C $(BUILDDIR)/$(var)/latex || sh $(srctree)/scripts/check-variable-fonts.= sh || exit; \ + $(MAKE) PDFLATEX=3D"$(PDFLATEX)" LATEXOPTS=3D"$(LATEXOPTS)" $(DENY_VF)= -C $(BUILDDIR)/$(var)/latex || $(PYTHON3) $(srctree)/scripts/check-variabl= e-fonts.py || exit; \ mkdir -p $(BUILDDIR)/$(var)/pdf; \ mv $(subst .tex,.pdf,$(wildcard $(BUILDDIR)/$(var)/latex/*.tex)) $(BUI= LDDIR)/$(var)/pdf/; \ ) diff --git a/MAINTAINERS b/MAINTAINERS index ef87548b8f88..88d8f7435e6d 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -7301,7 +7301,7 @@ S: Maintained P: Documentation/doc-guide/maintainer-profile.rst T: git git://git.lwn.net/linux.git docs-next F: Documentation/ -F: scripts/check-variable-fonts.sh +F: scripts/check-variable-fonts.py F: scripts/checktransupdate.py F: scripts/documentation-file-ref-check F: scripts/get_abi.py diff --git a/scripts/check-variable-fonts.sh b/scripts/check-variable-fonts= .py similarity index 61% rename from scripts/check-variable-fonts.sh rename to scripts/check-variable-fonts.py index ce63f0acea5f..8be1c0f39588 100755 --- a/scripts/check-variable-fonts.sh +++ b/scripts/check-variable-fonts.py @@ -1,7 +1,9 @@ -#!/bin/sh +#!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0-only # Copyright (C) Akira Yokosawa, 2024 # +# Ported to Python by (c) Mauro Carvalho Chehab, 2025 +# # For "make pdfdocs", reports of build errors of translations.pdf started # arriving early 2024 [1, 2]. It turned out that Fedora and openSUSE # tumbleweed have started deploying variable-font [3] format of "Noto CJK" @@ -87,29 +89,77 @@ # Denylisting should be less invasive, as it is effective only while # XeLaTeX runs in "make pdfdocs". =20 -# Default per-user fontconfig path (overridden by env variable) -: ${FONTS_CONF_DENY_VF:=3D$HOME/deny-vf} - -export XDG_CONFIG_HOME=3D${FONTS_CONF_DENY_VF} - -notocjkvffonts=3D`fc-list : file family variable | \ - grep 'variable=3DTrue' | \ - grep -E -e 'Noto (Sans|Sans Mono|Serif) CJK' | \ - sed -e 's/^/ /' -e 's/: Noto S.*$//' | sort | uniq` - -if [ "x$notocjkvffonts" !=3D "x" ] ; then - echo '=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D' - echo 'XeTeX is confused by "variable font" files listed below:' - echo "$notocjkvffonts" - echo - echo 'For CJK pages in PDF, they need to be hidden from XeTeX by denylist= ing.' - echo 'Or, CJK pages can be skipped by uninstalling texlive-xecjk.' - echo - echo 'For more info on denylisting, other options, and variable font, see= header' - echo 'comments of scripts/check-variable-fonts.sh.' - echo '=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D' -fi - -# As this script is invoked from Makefile's error path, always error exit -# regardless of whether any variable font is discovered or not. -exit 1 +import os +import re +import subprocess +import sys +import textwrap + +class LatexFontChecker: + """ + Detect problems with CJK variable fonts that affect PDF builds for + translations. + """ + + def __init__(self): + deny_vf =3D os.environ.get('FONTS_CONF_DENY_VF', "~/deny-vf") + + self.environ =3D os.environ.copy() + self.environ['XDG_CONFIG_HOME'] =3D os.path.expanduser(deny_vf) + + self.re_cjk =3D re.compile(r"([^:]+):\s*Noto\s+(Sans|Sans Mono|Ser= if) CJK") + + def get_noto_cjk_vf_fonts(self): + """Get Noto CJK fonts""" + + cjk_fonts =3D set() + cmd =3D ["fc-list", ":", "file", "family", "variable"] + try: + result =3D subprocess.run(cmd,stdout=3Dsubprocess.PIPE, + stderr=3Dsubprocess.PIPE, + universal_newlines=3DTrue, + env=3Dself.environ, + check=3DTrue) + + except subprocess.CalledProcessError as exc: + sys.exit(f"Error running fc-list: {repr(exc)}") + + for line in result.stdout.splitlines(): + if 'variable=3DTrue' not in line: + continue + + match =3D self.re_cjk.search(line) + if match: + cjk_fonts.add(match.group(1)) + + return sorted(cjk_fonts) + + def check(self): + """Check for problems with CJK fonts""" + + fonts =3D textwrap.indent("\n".join(self.get_noto_cjk_vf_fonts()),= " ") + if not fonts: + return None + + rel_file =3D os.path.relpath(__file__, os.getcwd()) + + msg =3D "=3D" * 77 + "\n" + msg +=3D 'XeTeX is confused by "variable font" files listed below:= \n' + msg +=3D fonts + "\n" + msg +=3D textwrap.dedent(f""" + For CJK pages in PDF, they need to be hidden from XeTeX by= denylisting. + Or, CJK pages can be skipped by uninstalling texlive-xecjk. + + For more info on denylisting, other options, and variable = font, see header + comments of {rel_file}. + """) + msg +=3D "=3D" * 77 + + return msg + +if __name__ =3D=3D "__main__": + msg =3D LatexFontChecker().check() + if msg: + print(msg) + + sys.exit(1) --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6338D2D7DD0; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; cv=none; b=ZC5vOfhOjMA4Jj3dG+pl2MWHDavInfpl0HEciBwx+tytsvg99Tf3EKxUMlu1ASHvjmsnC7h1P99wGn7xuUaQSBKg4UEqBLjKcGvEQd1aMUDVw9Zm0CBAveK4dfOAgFovqNEW5w9SHvXLVj4fvtaObWe4NlAwIHEDvAmxISNYch8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; c=relaxed/simple; bh=tvVO87MptHKvNiX/IGd4PofEFJGIlaeoGyYFXrabYqQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lXFHG9FkjmaxTIz+FKgyPPbv1oMjdp4DVZ6gyzufKMzUWnPC4sadDJIfwp7sUlcPshZASjsv1YmGupylGSNw/qdQvXmf7ZRLbVnx7gosNr+iQnjCNy0vExasSdn8cTHwGjP7zI/5EU3t5n/l/bVquPxw9kGHYeuXnXqyKggVb0Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DMwVlBV0; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DMwVlBV0" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B5A56C4CEFB; Thu, 18 Sep 2025 11:55:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196506; bh=tvVO87MptHKvNiX/IGd4PofEFJGIlaeoGyYFXrabYqQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DMwVlBV0JytDQkmxXr8C4eBYUNO+JgS1qrNmRAqXFcrfgcRRz9/qlRvzFWLxC+SSv EqulEdI+Z1sXePYrfzDFsW2tQAzPxpw45RYJGgmvPLbQvR6XOb9J9UJIVtC66qrCGU 1SCd+Glz2qRqt/8WRLErnJY5/9LHB/YBcGrIzDydcgj4+89q2Zq0MlfOYwa2VeX8kn /6wwcvUbFI3BLq5ChLZdVzYsVK2fhEmFD04qzSCZ1LrxFdMqbaVdHNyWxan2Ami0AN d4T1WhfKYPs6V/njjMxvm7igezyiDgdyvpWvBFjTOf7zDwSi3eQQv5SULVJQNiSQT5 tf9VOLZnLhWqg== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE7-0000000CrmZ-3YTj; Thu, 18 Sep 2025 13:55:03 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v8 05/24] tools/docs: check-variable-fonts.py: split into a lib and an exec file Date: Thu, 18 Sep 2025 13:54:39 +0200 Message-ID: <8adbc22df1d43b1c5a673799d2333cc429ffe9fc.1758196090.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" As we'll be using the actual code inside sphinx-build-wrapper, split the library from the executable, placing the exec at the new place we've been using: tools/docs No functional changes. Signed-off-by: Mauro Carvalho Chehab --- Documentation/Makefile | 2 +- MAINTAINERS | 1 - scripts/check-variable-fonts.py | 165 ----------------------------- tools/docs/check-variable-fonts.py | 23 ++++ tools/docs/lib/latex_fonts.py | 162 ++++++++++++++++++++++++++++ 5 files changed, 186 insertions(+), 167 deletions(-) delete mode 100755 scripts/check-variable-fonts.py create mode 100755 tools/docs/check-variable-fonts.py create mode 100755 tools/docs/lib/latex_fonts.py diff --git a/Documentation/Makefile b/Documentation/Makefile index d5e436435eab..6e11d0593742 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -146,7 +146,7 @@ pdfdocs: DENY_VF =3D XDG_CONFIG_HOME=3D$(FONTS_CONF_DEN= Y_VF) pdfdocs: latexdocs @$(srctree)/scripts/sphinx-pre-install --version-check $(foreach var,$(SPHINXDIRS), \ - $(MAKE) PDFLATEX=3D"$(PDFLATEX)" LATEXOPTS=3D"$(LATEXOPTS)" $(DENY_VF)= -C $(BUILDDIR)/$(var)/latex || $(PYTHON3) $(srctree)/scripts/check-variabl= e-fonts.py || exit; \ + $(MAKE) PDFLATEX=3D"$(PDFLATEX)" LATEXOPTS=3D"$(LATEXOPTS)" $(DENY_VF)= -C $(BUILDDIR)/$(var)/latex || $(PYTHON3) $(srctree)/tools/docs/check-vari= able-fonts.py || exit; \ mkdir -p $(BUILDDIR)/$(var)/pdf; \ mv $(subst .tex,.pdf,$(wildcard $(BUILDDIR)/$(var)/latex/*.tex)) $(BUI= LDDIR)/$(var)/pdf/; \ ) diff --git a/MAINTAINERS b/MAINTAINERS index 88d8f7435e6d..76dd823bfcc4 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -7301,7 +7301,6 @@ S: Maintained P: Documentation/doc-guide/maintainer-profile.rst T: git git://git.lwn.net/linux.git docs-next F: Documentation/ -F: scripts/check-variable-fonts.py F: scripts/checktransupdate.py F: scripts/documentation-file-ref-check F: scripts/get_abi.py diff --git a/scripts/check-variable-fonts.py b/scripts/check-variable-fonts= .py deleted file mode 100755 index 8be1c0f39588..000000000000 --- a/scripts/check-variable-fonts.py +++ /dev/null @@ -1,165 +0,0 @@ -#!/usr/bin/env python3 -# SPDX-License-Identifier: GPL-2.0-only -# Copyright (C) Akira Yokosawa, 2024 -# -# Ported to Python by (c) Mauro Carvalho Chehab, 2025 -# -# For "make pdfdocs", reports of build errors of translations.pdf started -# arriving early 2024 [1, 2]. It turned out that Fedora and openSUSE -# tumbleweed have started deploying variable-font [3] format of "Noto CJK" -# fonts [4, 5]. For PDF, a LaTeX package named xeCJK is used for CJK -# (Chinese, Japanese, Korean) pages. xeCJK requires XeLaTeX/XeTeX, which -# does not (and likely never will) understand variable fonts for historical -# reasons. -# -# The build error happens even when both of variable- and non-variable-for= mat -# fonts are found on the build system. To make matters worse, Fedora enli= sts -# variable "Noto CJK" fonts in the requirements of langpacks-ja, -ko, -zh_= CN, -# -zh_TW, etc. Hence developers who have interest in CJK pages are more -# likely to encounter the build errors. -# -# This script is invoked from the error path of "make pdfdocs" and emits -# suggestions if variable-font files of "Noto CJK" fonts are in the list of -# fonts accessible from XeTeX. -# -# References: -# [1]: https://lore.kernel.org/r/8734tqsrt7.fsf@meer.lwn.net/ -# [2]: https://lore.kernel.org/r/1708585803.600323099@f111.i.mail.ru/ -# [3]: https://en.wikipedia.org/wiki/Variable_font -# [4]: https://fedoraproject.org/wiki/Changes/Noto_CJK_Variable_Fonts -# [5]: https://build.opensuse.org/request/show/1157217 -# -#=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D -# Workarounds for building translations.pdf -#=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D -# -# * Denylist "variable font" Noto CJK fonts. -# - Create $HOME/deny-vf/fontconfig/fonts.conf from template below, with -# tweaks if necessary. Remove leading "# ". -# - Path of fontconfig/fonts.conf can be overridden by setting an env -# variable FONTS_CONF_DENY_VF. -# -# * Template: -# ----------------------------------------------------------------- -# -# -# -# -# -# -# -# /usr/share/fonts/google-noto-*-cjk-vf-fonts -# -# /usr/share/fonts/truetype/Noto*CJK*-VF.otf -# -# -# -# ----------------------------------------------------------------- -# -# The denylisting is activated for "make pdfdocs". -# -# * For skipping CJK pages in PDF -# - Uninstall texlive-xecjk. -# Denylisting is not needed in this case. -# -# * For printing CJK pages in PDF -# - Need non-variable "Noto CJK" fonts. -# * Fedora -# - google-noto-sans-cjk-fonts -# - google-noto-serif-cjk-fonts -# * openSUSE tumbleweed -# - Non-variable "Noto CJK" fonts are not available as distro packag= es -# as of April, 2024. Fetch a set of font files from upstream Noto -# CJK Font released at: -# https://github.com/notofonts/noto-cjk/tree/main/Sans#super-otc -# and at: -# https://github.com/notofonts/noto-cjk/tree/main/Serif#super-otc -# , then uncompress and deploy them. -# - Remember to update fontconfig cache by running fc-cache. -# -# !!! Caution !!! -# Uninstalling "variable font" packages can be dangerous. -# They might be depended upon by other packages important for your wor= k. -# Denylisting should be less invasive, as it is effective only while -# XeLaTeX runs in "make pdfdocs". - -import os -import re -import subprocess -import sys -import textwrap - -class LatexFontChecker: - """ - Detect problems with CJK variable fonts that affect PDF builds for - translations. - """ - - def __init__(self): - deny_vf =3D os.environ.get('FONTS_CONF_DENY_VF', "~/deny-vf") - - self.environ =3D os.environ.copy() - self.environ['XDG_CONFIG_HOME'] =3D os.path.expanduser(deny_vf) - - self.re_cjk =3D re.compile(r"([^:]+):\s*Noto\s+(Sans|Sans Mono|Ser= if) CJK") - - def get_noto_cjk_vf_fonts(self): - """Get Noto CJK fonts""" - - cjk_fonts =3D set() - cmd =3D ["fc-list", ":", "file", "family", "variable"] - try: - result =3D subprocess.run(cmd,stdout=3Dsubprocess.PIPE, - stderr=3Dsubprocess.PIPE, - universal_newlines=3DTrue, - env=3Dself.environ, - check=3DTrue) - - except subprocess.CalledProcessError as exc: - sys.exit(f"Error running fc-list: {repr(exc)}") - - for line in result.stdout.splitlines(): - if 'variable=3DTrue' not in line: - continue - - match =3D self.re_cjk.search(line) - if match: - cjk_fonts.add(match.group(1)) - - return sorted(cjk_fonts) - - def check(self): - """Check for problems with CJK fonts""" - - fonts =3D textwrap.indent("\n".join(self.get_noto_cjk_vf_fonts()),= " ") - if not fonts: - return None - - rel_file =3D os.path.relpath(__file__, os.getcwd()) - - msg =3D "=3D" * 77 + "\n" - msg +=3D 'XeTeX is confused by "variable font" files listed below:= \n' - msg +=3D fonts + "\n" - msg +=3D textwrap.dedent(f""" - For CJK pages in PDF, they need to be hidden from XeTeX by= denylisting. - Or, CJK pages can be skipped by uninstalling texlive-xecjk. - - For more info on denylisting, other options, and variable = font, see header - comments of {rel_file}. - """) - msg +=3D "=3D" * 77 - - return msg - -if __name__ =3D=3D "__main__": - msg =3D LatexFontChecker().check() - if msg: - print(msg) - - sys.exit(1) diff --git a/tools/docs/check-variable-fonts.py b/tools/docs/check-variable= -fonts.py new file mode 100755 index 000000000000..79b28f0f7d85 --- /dev/null +++ b/tools/docs/check-variable-fonts.py @@ -0,0 +1,23 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-only +# Copyright (C) Akira Yokosawa, 2024 +# +# Ported to Python by (c) Mauro Carvalho Chehab, 2025 +# +# pylint: disable=3DC0103 + +""" +Detect problematic Noto CJK variable fonts. + +or more details, see lib/latex_fonts.py. +""" + +import sys + +from lib.latex_fonts import LatexFontChecker + +msg =3D LatexFontChecker().check() +if msg: + print(msg) + +sys.exit(1) diff --git a/tools/docs/lib/latex_fonts.py b/tools/docs/lib/latex_fonts.py new file mode 100755 index 000000000000..81358a70f320 --- /dev/null +++ b/tools/docs/lib/latex_fonts.py @@ -0,0 +1,162 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-only +# Copyright (C) Akira Yokosawa, 2024 +# +# Ported to Python by (c) Mauro Carvalho Chehab, 2025 + +""" +Detect problematic Noto CJK variable fonts. + +For "make pdfdocs", reports of build errors of translations.pdf started +arriving early 2024 [1, 2]. It turned out that Fedora and openSUSE +tumbleweed have started deploying variable-font [3] format of "Noto CJK" +fonts [4, 5]. For PDF, a LaTeX package named xeCJK is used for CJK +(Chinese, Japanese, Korean) pages. xeCJK requires XeLaTeX/XeTeX, which +does not (and likely never will) understand variable fonts for historical +reasons. + +The build error happens even when both of variable- and non-variable-format +fonts are found on the build system. To make matters worse, Fedora enlists +variable "Noto CJK" fonts in the requirements of langpacks-ja, -ko, -zh_CN, +-zh_TW, etc. Hence developers who have interest in CJK pages are more +likely to encounter the build errors. + +This script is invoked from the error path of "make pdfdocs" and emits +suggestions if variable-font files of "Noto CJK" fonts are in the list of +fonts accessible from XeTeX. + +References: +[1]: https://lore.kernel.org/r/8734tqsrt7.fsf@meer.lwn.net/ +[2]: https://lore.kernel.org/r/1708585803.600323099@f111.i.mail.ru/ +[3]: https://en.wikipedia.org/wiki/Variable_font +[4]: https://fedoraproject.org/wiki/Changes/Noto_CJK_Variable_Fonts +[5]: https://build.opensuse.org/request/show/1157217 + +#=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D +Workarounds for building translations.pdf +#=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D + +* Denylist "variable font" Noto CJK fonts. + - Create $HOME/deny-vf/fontconfig/fonts.conf from template below, with + tweaks if necessary. Remove leading "". + - Path of fontconfig/fonts.conf can be overridden by setting an env + variable FONTS_CONF_DENY_VF. + + * Template: +----------------------------------------------------------------- + + + + + + + + /usr/share/fonts/google-noto-*-cjk-vf-fonts + + /usr/share/fonts/truetype/Noto*CJK*-VF.otf + + + +----------------------------------------------------------------- + + The denylisting is activated for "make pdfdocs". + +* For skipping CJK pages in PDF + - Uninstall texlive-xecjk. + Denylisting is not needed in this case. + +* For printing CJK pages in PDF + - Need non-variable "Noto CJK" fonts. + * Fedora + - google-noto-sans-cjk-fonts + - google-noto-serif-cjk-fonts + * openSUSE tumbleweed + - Non-variable "Noto CJK" fonts are not available as distro packages + as of April, 2024. Fetch a set of font files from upstream Noto + CJK Font released at: + https://github.com/notofonts/noto-cjk/tree/main/Sans#super-otc + and at: + https://github.com/notofonts/noto-cjk/tree/main/Serif#super-otc + , then uncompress and deploy them. + - Remember to update fontconfig cache by running fc-cache. + +!!! Caution !!! + Uninstalling "variable font" packages can be dangerous. + They might be depended upon by other packages important for your work. + Denylisting should be less invasive, as it is effective only while + XeLaTeX runs in "make pdfdocs". +""" + +import os +import re +import subprocess +import textwrap +import sys + +class LatexFontChecker: + """ + Detect problems with CJK variable fonts that affect PDF builds for + translations. + """ + + def __init__(self): + deny_vf =3D os.environ.get('FONTS_CONF_DENY_VF', "~/deny-vf") + + self.environ =3D os.environ.copy() + self.environ['XDG_CONFIG_HOME'] =3D os.path.expanduser(deny_vf) + + self.re_cjk =3D re.compile(r"([^:]+):\s*Noto\s+(Sans|Sans Mono|Ser= if) CJK") + + def get_noto_cjk_vf_fonts(self): + """Get Noto CJK fonts""" + + cjk_fonts =3D set() + cmd =3D ["fc-list", ":", "file", "family", "variable"] + try: + result =3D subprocess.run(cmd,stdout=3Dsubprocess.PIPE, + stderr=3Dsubprocess.PIPE, + universal_newlines=3DTrue, + env=3Dself.environ, + check=3DTrue) + + except subprocess.CalledProcessError as exc: + sys.exit(f"Error running fc-list: {repr(exc)}") + + for line in result.stdout.splitlines(): + if 'variable=3DTrue' not in line: + continue + + match =3D self.re_cjk.search(line) + if match: + cjk_fonts.add(match.group(1)) + + return sorted(cjk_fonts) + + def check(self): + """Check for problems with CJK fonts""" + + fonts =3D textwrap.indent("\n".join(self.get_noto_cjk_vf_fonts()),= " ") + if not fonts: + return None + + rel_file =3D os.path.relpath(__file__, os.getcwd()) + + msg =3D "=3D" * 77 + "\n" + msg +=3D 'XeTeX is confused by "variable font" files listed below:= \n' + msg +=3D fonts + "\n" + msg +=3D textwrap.dedent(f""" + For CJK pages in PDF, they need to be hidden from XeTeX by= denylisting. + Or, CJK pages can be skipped by uninstalling texlive-xecjk. + + For more info on denylisting, other options, and variable = font, see header + comments of {rel_file}. + """) + msg +=3D "=3D" * 77 + + return msg --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 64F772D9491; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; cv=none; b=kFxtgmHTzvGmRLWxmX9jP+miFpqCpUPKO89/aqoj7n7RmidBisWbXtkWfF8uaWYQo8HIsNdtM5/IRWp6kLIvGRuc+JtV2hvxFxbM5RBxU/ywduL1xcYyDoPEt+bqKJe3Y+slXbNS3CVA4xFNHrdGl9LSXem92RXgSjVgbr7cOnI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; c=relaxed/simple; bh=wgeju9sHn9sGYAdF8T8ePAcnZc8uoFBBK8ROWiPi7/c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=QGxokk+E1P6X//Dxt2HN7Hil6G0ZF2c2/O8WzBabQqeXlRP17oVZ/hDUMVP7q/ZOaJfbGejCgLRKgoSNo5FfjXYXC5rbjclSXSP0SVkC69cYLfhnkRhCwzmmdJEsJEcABD7rW/TdNyAzrIjNyHcr3XUU0CP27dtIBjewEJ9ah3M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=vDOJiC6P; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="vDOJiC6P" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 278BBC113D0; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196506; bh=wgeju9sHn9sGYAdF8T8ePAcnZc8uoFBBK8ROWiPi7/c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vDOJiC6PXNj3WVhh5kDe4ugwRU8W2+eEU5jldQybysSRGT9rwucc5/9mDc1hmCZ8J Au0BWA0WaL4UcySjQlu6Igm+FfGFWrrMpRRbasYJMecy7fXaAZ5PiGj8zy7nf2dX2r QoimRqwjjbXwVvKCV03k4fFmWXCsQW4NinwkhHow6T69uGtnRrPg+xSTZeoaSMFD3R +o53E1jyjlQSXjnMkF+h1uYMSl64NOKqtqvjwi8GMP9pNlbSHsbL7oqqy72qwHCSAw HutSkUDrDZiA/Yj4oMHLbjVxJ13LGqasPFRbP1Lqs21+8RI5rqP60lGXuzuexN+LKq In8U0jSN909Ug== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE7-0000000CrnA-3fLb; Thu, 18 Sep 2025 13:55:03 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v8 06/24] check-variable-fonts.py: add a helper to display instructions Date: Thu, 18 Sep 2025 13:54:40 +0200 Message-ID: <577162cf4e07de74c4a783f16e3404f0040e5e0a.1758196090.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Use lib docstring to output the comments via --help/-h. With that, update the default instructions to recomment it instead of asking the user to read the source code. Signed-off-by: Mauro Carvalho Chehab --- tools/docs/check-variable-fonts.py | 12 +++++++++++- tools/docs/lib/latex_fonts.py | 13 +++++++++---- 2 files changed, 20 insertions(+), 5 deletions(-) diff --git a/tools/docs/check-variable-fonts.py b/tools/docs/check-variable= -fonts.py index 79b28f0f7d85..c0997d6861dc 100755 --- a/tools/docs/check-variable-fonts.py +++ b/tools/docs/check-variable-fonts.py @@ -12,11 +12,21 @@ Detect problematic Noto CJK variable fonts. or more details, see lib/latex_fonts.py. """ =20 +import argparse import sys =20 from lib.latex_fonts import LatexFontChecker =20 -msg =3D LatexFontChecker().check() +checker =3D LatexFontChecker() + +parser=3Dargparse.ArgumentParser(description=3Dchecker.description(), + formatter_class=3Dargparse.RawTextHelpForma= tter) +parser.add_argument("--deny-vf", + help=3D"XDG_CONFIG_HOME dir containing fontconfig/font= s.conf file") + +args=3Dparser.parse_args() + +msg =3D LatexFontChecker(args.deny_vf).check() if msg: print(msg) =20 diff --git a/tools/docs/lib/latex_fonts.py b/tools/docs/lib/latex_fonts.py index 81358a70f320..29317f8006ea 100755 --- a/tools/docs/lib/latex_fonts.py +++ b/tools/docs/lib/latex_fonts.py @@ -105,14 +105,18 @@ class LatexFontChecker: translations. """ =20 - def __init__(self): - deny_vf =3D os.environ.get('FONTS_CONF_DENY_VF', "~/deny-vf") + def __init__(self, deny_vf=3DNone): + if not deny_vf: + deny_vf =3D os.environ.get('FONTS_CONF_DENY_VF', "~/deny-vf") =20 self.environ =3D os.environ.copy() self.environ['XDG_CONFIG_HOME'] =3D os.path.expanduser(deny_vf) =20 self.re_cjk =3D re.compile(r"([^:]+):\s*Noto\s+(Sans|Sans Mono|Ser= if) CJK") =20 + def description(self): + return __doc__ + def get_noto_cjk_vf_fonts(self): """Get Noto CJK fonts""" =20 @@ -154,8 +158,9 @@ class LatexFontChecker: For CJK pages in PDF, they need to be hidden from XeTeX by= denylisting. Or, CJK pages can be skipped by uninstalling texlive-xecjk. =20 - For more info on denylisting, other options, and variable = font, see header - comments of {rel_file}. + For more info on denylisting, other options, and variable = font, run: + + tools/docs/check-variable-fonts.py -h """) msg +=3D "=3D" * 77 =20 --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 756252FFF9F; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; cv=none; b=sRfR67IaxnkhaUkMtHwOTW4qZAlS3Few9jwCw+f2zDSvEN+D0FkDZbnUXyLUwC82RL4kT8ZOhPAVCiMTAfPDUg2XFKAAOpRGIrov64R+U2bLPxFpxqb+Erg4LSEOhsTg6mI6wZ+aMYJ95OYOaQgFmrZJ0W2RS0u8YeDP0UQ0Qso= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; c=relaxed/simple; bh=tyTIqAnvcS4UX09dEZuAJBcwACv9raN6toXoVwiSVjI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=qOwvyPOMfvNcekvT9g/UO4sI4z1E1nz3EemQlwrIGyP9jcJUsRVqcqRuSmWDAfvruQkH0yGSiJX+r5LL1P9MY/ElZ8fuRPpqyJVUrUkKlijipstypFCKwjkKxQ2E2ZDlKSsIARAcT20RnDSDP0TjSHjEorWPHP4JfFls37Ihst4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oE6DtlyD; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oE6DtlyD" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 07FCEC113CF; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196506; bh=tyTIqAnvcS4UX09dEZuAJBcwACv9raN6toXoVwiSVjI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oE6DtlyDuV/9azvaFjPIzDpvIYqE7tsYg7vpK7Lo+oRDvHkCF8YuIOG/TvEeeLF7E mcUHsjDzVpQbEjAOV95jHC3+D7H/BY1jIri4xyn4X9UCCmryFTvxebfWGOaY5Fisvm 4VvRn2Q3df/6eepLMS4X4i73eGmAjvFR29MhauFJoQ23p8hJ6x1xEARemQNw59xUJj +K5sDMEIRwTuZxcM/lS1wQwLmSK+b3V8/PMejwkkyV9fxBQWH1ywf9Cuq2ccUeREAy RvROogsOKUB1DW0sneHNkRBg/CBrsXW7PRs1RAB98bqg+Gd3F3kmRnQ+YcN9EAFZAo 0RME5DVWqBPSg== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE7-0000000Crna-3mTB; Thu, 18 Sep 2025 13:55:03 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , Alex Shi , Dongliang Mu , Federico Vaga , Randy Dunlap , Yanteng Si , linux-kernel@vger.kernel.org Subject: [PATCH v8 07/24] scripts: sphinx-pre-install: move it to tools/docs Date: Thu, 18 Sep 2025 13:54:41 +0200 Message-ID: <5e2c40d3aebfd67b7ac7817f548bd1fa4ff661a8.1758196090.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab As we're reorganizing the place where doc scripts are located, move this one to tools/docs. No functional changes. Signed-off-by: Mauro Carvalho Chehab --- Documentation/Makefile | 14 +++++++------- Documentation/doc-guide/sphinx.rst | 4 ++-- Documentation/sphinx/kerneldoc-preamble.sty | 2 +- .../translations/it_IT/doc-guide/sphinx.rst | 4 ++-- .../translations/zh_CN/doc-guide/sphinx.rst | 4 ++-- Documentation/translations/zh_CN/how-to.rst | 2 +- MAINTAINERS | 3 +-- {scripts =3D> tools/docs}/sphinx-pre-install | 0 8 files changed, 16 insertions(+), 17 deletions(-) rename {scripts =3D> tools/docs}/sphinx-pre-install (100%) diff --git a/Documentation/Makefile b/Documentation/Makefile index 6e11d0593742..fd6399c79fab 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -46,7 +46,7 @@ ifeq ($(HAVE_SPHINX),0) .DEFAULT: $(warning The '$(SPHINXBUILD)' command was not found. Make sure you have = Sphinx installed and in PATH, or set the SPHINXBUILD make variable to point= to the full path of the '$(SPHINXBUILD)' executable.) @echo - @$(srctree)/scripts/sphinx-pre-install + @$(srctree)/tools/docs/sphinx-pre-install @echo " SKIP Sphinx $@ target." =20 else # HAVE_SPHINX @@ -105,7 +105,7 @@ quiet_cmd_sphinx =3D SPHINX $@ --> file://$(abspath $(= BUILDDIR)/$3/$4) fi =20 htmldocs: - @$(srctree)/scripts/sphinx-pre-install --version-check + @$(srctree)/tools/docs/sphinx-pre-install --version-check @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var))) =20 # If Rust support is available and .config exists, add rustdoc generated c= ontents. @@ -119,7 +119,7 @@ endif endif =20 texinfodocs: - @$(srctree)/scripts/sphinx-pre-install --version-check + @$(srctree)/tools/docs/sphinx-pre-install --version-check @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,texinfo,$(var),texin= fo,$(var))) =20 # Note: the 'info' Make target is generated by sphinx itself when @@ -131,7 +131,7 @@ linkcheckdocs: @$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,linkcheck,$(var),,$(v= ar))) =20 latexdocs: - @$(srctree)/scripts/sphinx-pre-install --version-check + @$(srctree)/tools/docs/sphinx-pre-install --version-check @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$= (var))) =20 ifeq ($(HAVE_PDFLATEX),0) @@ -144,7 +144,7 @@ else # HAVE_PDFLATEX =20 pdfdocs: DENY_VF =3D XDG_CONFIG_HOME=3D$(FONTS_CONF_DENY_VF) pdfdocs: latexdocs - @$(srctree)/scripts/sphinx-pre-install --version-check + @$(srctree)/tools/docs/sphinx-pre-install --version-check $(foreach var,$(SPHINXDIRS), \ $(MAKE) PDFLATEX=3D"$(PDFLATEX)" LATEXOPTS=3D"$(LATEXOPTS)" $(DENY_VF)= -C $(BUILDDIR)/$(var)/latex || $(PYTHON3) $(srctree)/tools/docs/check-vari= able-fonts.py || exit; \ mkdir -p $(BUILDDIR)/$(var)/pdf; \ @@ -154,11 +154,11 @@ pdfdocs: latexdocs endif # HAVE_PDFLATEX =20 epubdocs: - @$(srctree)/scripts/sphinx-pre-install --version-check + @$(srctree)/tools/docs/sphinx-pre-install --version-check @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(v= ar))) =20 xmldocs: - @$(srctree)/scripts/sphinx-pre-install --version-check + @$(srctree)/tools/docs/sphinx-pre-install --version-check @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var= ))) =20 endif # HAVE_SPHINX diff --git a/Documentation/doc-guide/sphinx.rst b/Documentation/doc-guide/s= phinx.rst index 607589592bfb..932f68c53075 100644 --- a/Documentation/doc-guide/sphinx.rst +++ b/Documentation/doc-guide/sphinx.rst @@ -106,7 +106,7 @@ There's a script that automatically checks for Sphinx d= ependencies. If it can recognize your distribution, it will also give a hint about the install command line options for your distro:: =20 - $ ./scripts/sphinx-pre-install + $ ./tools/docs/sphinx-pre-install Checking if the needed tools for Fedora release 26 (Twenty Six) are avail= able Warning: better to also install "texlive-luatex85". You should run: @@ -116,7 +116,7 @@ command line options for your distro:: . sphinx_2.4.4/bin/activate pip install -r Documentation/sphinx/requirements.txt =20 - Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-= install line 468. + Can't build as 1 mandatory dependency is missing at ./tools/docs/sphinx-p= re-install line 468. =20 By default, it checks all the requirements for both html and PDF, including the requirements for images, math expressions and LaTeX build, and assumes diff --git a/Documentation/sphinx/kerneldoc-preamble.sty b/Documentation/sp= hinx/kerneldoc-preamble.sty index 5d68395539fe..16d9ff46fdf6 100644 --- a/Documentation/sphinx/kerneldoc-preamble.sty +++ b/Documentation/sphinx/kerneldoc-preamble.sty @@ -220,7 +220,7 @@ If you want them, please install non-variable ``Noto Sans CJK'' font families along with the texlive-xecjk package by following instructions from - \sphinxcode{./scripts/sphinx-pre-install}. + \sphinxcode{./tools/docs/sphinx-pre-install}. Having optional non-variable ``Noto Serif CJK'' font families will improve the looks of those translations. \end{sphinxadmonition}} diff --git a/Documentation/translations/it_IT/doc-guide/sphinx.rst b/Docume= ntation/translations/it_IT/doc-guide/sphinx.rst index 1f513bc33618..a5c5d935febf 100644 --- a/Documentation/translations/it_IT/doc-guide/sphinx.rst +++ b/Documentation/translations/it_IT/doc-guide/sphinx.rst @@ -109,7 +109,7 @@ Sphinx. Se lo script riesce a riconoscere la vostra dis= tribuzione, allora sar=C3=A0 in grado di darvi dei suggerimenti su come procedere per complet= are l'installazione:: =20 - $ ./scripts/sphinx-pre-install + $ ./tools/docs/sphinx-pre-install Checking if the needed tools for Fedora release 26 (Twenty Six) are avail= able Warning: better to also install "texlive-luatex85". You should run: @@ -119,7 +119,7 @@ l'installazione:: . sphinx_2.4.4/bin/activate pip install -r Documentation/sphinx/requirements.txt =20 - Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-= install line 468. + Can't build as 1 mandatory dependency is missing at ./tools/docs/sphinx-p= re-install line 468. =20 L'impostazione predefinita prevede il controllo dei requisiti per la gener= azione di documenti html e PDF, includendo anche il supporto per le immagini, le diff --git a/Documentation/translations/zh_CN/doc-guide/sphinx.rst b/Docume= ntation/translations/zh_CN/doc-guide/sphinx.rst index 23eac67fbc30..3375c6f3a811 100644 --- a/Documentation/translations/zh_CN/doc-guide/sphinx.rst +++ b/Documentation/translations/zh_CN/doc-guide/sphinx.rst @@ -84,7 +84,7 @@ PDF=E5=92=8CLaTeX=E6=9E=84=E5=BB=BA =E8=BF=99=E6=9C=89=E4=B8=80=E4=B8=AA=E8=84=9A=E6=9C=AC=E5=8F=AF=E4=BB=A5= =E8=87=AA=E5=8A=A8=E6=A3=80=E6=9F=A5Sphinx=E4=BE=9D=E8=B5=96=E9=A1=B9=E3=80= =82=E5=A6=82=E6=9E=9C=E5=AE=83=E8=AE=A4=E5=BE=97=E6=82=A8=E7=9A=84=E5=8F=91= =E8=A1=8C=E7=89=88=EF=BC=8C=E8=BF=98=E4=BC=9A=E6=8F=90=E7=A4=BA=E6=82=A8=E6= =89=80=E7=94=A8=E5=8F=91=E8=A1=8C =E7=89=88=E7=9A=84=E5=AE=89=E8=A3=85=E5=91=BD=E4=BB=A4:: =20 - $ ./scripts/sphinx-pre-install + $ ./tools/docs/sphinx-pre-install Checking if the needed tools for Fedora release 26 (Twenty Six) are avail= able Warning: better to also install "texlive-luatex85". You should run: @@ -94,7 +94,7 @@ PDF=E5=92=8CLaTeX=E6=9E=84=E5=BB=BA . sphinx_2.4.4/bin/activate pip install -r Documentation/sphinx/requirements.txt =20 - Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-= install line 468. + Can't build as 1 mandatory dependency is missing at ./tools/docs/sphinx-p= re-install line 468. =20 =E9=BB=98=E8=AE=A4=E6=83=85=E5=86=B5=E4=B8=8B=EF=BC=8C=E5=AE=83=E4=BC=9A= =E6=A3=80=E6=9F=A5html=E5=92=8CPDF=E7=9A=84=E6=89=80=E6=9C=89=E4=BE=9D=E8= =B5=96=E9=A1=B9=EF=BC=8C=E5=8C=85=E6=8B=AC=E5=9B=BE=E5=83=8F=E3=80=81=E6=95= =B0=E5=AD=A6=E8=A1=A8=E8=BE=BE=E5=BC=8F=E5=92=8CLaTeX=E6=9E=84=E5=BB=BA=E7= =9A=84 =E9=9C=80=E6=B1=82=EF=BC=8C=E5=B9=B6=E5=81=87=E8=AE=BE=E5=B0=86=E4=BD=BF= =E7=94=A8=E8=99=9A=E6=8B=9FPython=E7=8E=AF=E5=A2=83=E3=80=82html=E6=9E=84= =E5=BB=BA=E6=89=80=E9=9C=80=E7=9A=84=E4=BE=9D=E8=B5=96=E9=A1=B9=E8=A2=AB=E8= =AE=A4=E4=B8=BA=E6=98=AF=E5=BF=85=E9=9C=80=E7=9A=84=EF=BC=8C=E5=85=B6=E4=BB= =96=E4=BE=9D diff --git a/Documentation/translations/zh_CN/how-to.rst b/Documentation/tr= anslations/zh_CN/how-to.rst index ddd99c0f9b4d..714664fec308 100644 --- a/Documentation/translations/zh_CN/how-to.rst +++ b/Documentation/translations/zh_CN/how-to.rst @@ -64,7 +64,7 @@ Linux =E5=8F=91=E8=A1=8C=E7=89=88=E5=92=8C=E7=AE=80=E5=8D= =95=E5=9C=B0=E4=BD=BF=E7=94=A8 Linux =E5=91=BD=E4=BB=A4=E8=A1=8C=EF=BC=8C= =E9=82=A3=E4=B9=88=E5=8F=AF=E4=BB=A5=E8=BF=85=E9=80=9F=E5=BC=80=E5=A7=8B=E4= =BA=86 :: =20 cd linux - ./scripts/sphinx-pre-install + ./tools/docs/sphinx-pre-install =20 =E4=BB=A5 Fedora =E4=B8=BA=E4=BE=8B=EF=BC=8C=E5=AE=83=E7=9A=84=E8=BE=93=E5= =87=BA=E6=98=AF=E8=BF=99=E6=A0=B7=E7=9A=84:: =20 diff --git a/MAINTAINERS b/MAINTAINERS index 76dd823bfcc4..16a5d6ab627d 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -7309,7 +7309,6 @@ F: scripts/lib/abi/* F: scripts/lib/kdoc/* F: tools/docs/* F: tools/net/ynl/pyynl/lib/doc_generator.py -F: scripts/sphinx-pre-install X: Documentation/ABI/ X: Documentation/admin-guide/media/ X: Documentation/devicetree/ @@ -7344,7 +7343,7 @@ L: linux-doc@vger.kernel.org S: Maintained F: Documentation/sphinx/parse-headers.pl F: scripts/documentation-file-ref-check -F: scripts/sphinx-pre-install +F: tools/docs/sphinx-pre-install =20 DOCUMENTATION/ITALIAN M: Federico Vaga diff --git a/scripts/sphinx-pre-install b/tools/docs/sphinx-pre-install similarity index 100% rename from scripts/sphinx-pre-install rename to tools/docs/sphinx-pre-install --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 739EA2FBDFD; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; cv=none; b=jkXN/+hk8mwqtgysZfV7P/EbZwU7XG0mrycfXtwkrckGkGg22CUkXcugG1zl3AeoTPFClM9kNlbuqDmyXMLgJN1Y4Q4WcJPtUrLWAtfrtrsHK+ns5aVUe2/WyItJwUQyz+CFOjIOajeIKyGirP7z1vtTcNnPB6LlGlwPvW53yd0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; c=relaxed/simple; bh=AH2lL83LG/MCK6GW00LCQjiXUczLYQf3wfFtphqsZ6o=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=M1zmmvLGp23VpKbkW1EmfLhb4ed94P/IrQ8s78c7/pAY34dd1rgmEdSdp5KT4dSElT1SKC/pnNpUV/H3s3XzkO0XkyyZjDKzpYS2jLETleN5ipxnZhMrq9ubicQ1hAI3tIbeYHJ1rk0Azb5TaNZzOb/z+czgSnauBaRCcwAaT6M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oFy9/200; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oFy9/200" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2B45AC4CEFF; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196506; bh=AH2lL83LG/MCK6GW00LCQjiXUczLYQf3wfFtphqsZ6o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oFy9/2009MSF9lVwB/c4IGmRbx5k6u06iBl+gT4pj/Y5KLN7agcfAl8aEcWFEkJy9 sIx1LBjcoaNWMJiHZyIevKcOJj6WyvYN7Oy2mGcrl56bkNzr9idPsBM36a5jLfZnbe rxRzS8rf+wKTAz1HwnRtdCoeR73H8kWvgRVsDfLFI0NnDYygil1C50yxkC70kp9P73 1JJbNZ/0rQ0kte8O4qpuQ13nKOkqW3WkPhjZUaCtiVxqU7kXi7JRDgg+Rr7uDnhUW0 vnJhrZuAeZYc4S8vh/8+6V6jjPgnZIBiuL/w+h2RnkvQ50p7VXVUCKa467TDeNCDdD tvtjSOYPtFDug== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE7-0000000CroN-3vuf; Thu, 18 Sep 2025 13:55:03 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v8 08/24] tools/docs: sphinx-pre-install: drop a debug print Date: Thu, 18 Sep 2025 13:54:42 +0200 Message-ID: <27f76a4df2b80c38d277d58a92c85c614544e013.1758196090.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" The version print at the lib was added for debugging purposes. Get rid of it. Signed-off-by: Mauro Carvalho Chehab --- tools/docs/sphinx-pre-install | 1 - 1 file changed, 1 deletion(-) diff --git a/tools/docs/sphinx-pre-install b/tools/docs/sphinx-pre-install index 954ed3dc0645..fa12e01fc7fe 100755 --- a/tools/docs/sphinx-pre-install +++ b/tools/docs/sphinx-pre-install @@ -285,7 +285,6 @@ class AncillaryMethods: cur_ver =3D sys.version_info[:3] if cur_ver >=3D MIN_PYTHON_VERSION: ver =3D ver_str(cur_ver) - print(f"Python version: {ver}") =20 # This could be useful for debugging purposes if SphinxDependencyChecker.which("docutils"): --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7EEAE30171F; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; cv=none; b=B7IWOl8NSQQ0pb6aWe7/fd7UBFyGUzhQ1lQzxF5xI/3iyGqxusm5ObfWa8vS45Po0trGcJFDRoxc+PwzMgnPl3RiIsu8hGg+lLCCVs8eOhojAp9VNwh8vKRba2Q+ky82JcucDn9T13y51ouN52PgvTevqPA+q1FVlr8Gq3rImwg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; c=relaxed/simple; bh=+6TvP4G2cDhLKPN/soKTsxtFrnu65L1eQKi7+8h+GhI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tghorNZS5TtsYa2uqb39c7xEaZZ1yqfOvXxbmZ+AXYywOFjH+rKgOAyQ/Eh1veB0bXXEHsJzTKglMNq6gB4Zx5uPtTXx8/BXfOc+zqXWYMT3i06wK/4X7JYxLJ9sT7B6AUf6i6AL5FD3NAqSh2RLtDzcbsLGu6gfPNZ3hQb27Rs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=QP/GE0vO; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="QP/GE0vO" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5B4FCC4CEFD; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196506; bh=+6TvP4G2cDhLKPN/soKTsxtFrnu65L1eQKi7+8h+GhI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QP/GE0vO33UptbFtdEio/BtO+A2ZJhZRr1H9NpwYJCL6AmTlzSN2Ual6rBgY+LOug HnhDIKof1ZN9d9RERzKsLVDOUwotjPuaFUMfLTnnsQdktDQW1hdSfmGrfuQVynjSqE SCsFqq04O7QMhOtzyK5t16Mctlz+uWRQvxAYG/xkw7afjeRRPdVgZtdTJD3aenx3Eo Y0SWhPt5r2HR2UYWDIE6qme+43m9MbM9W76jyfWcMedeejjCRKLbGZ0SfxTOHDayuf bKOZ+fVCJd/o0NdvlxhnLFmsTdPV/SkYiGOe5R6yGRHPFaRrn8IjA0rr7wGZrpMRDJ YWYep7WWyOA6w== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE7-0000000CrpA-45Pe; Thu, 18 Sep 2025 13:55:03 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v8 09/24] tools/docs: sphinx-pre-install: allow check for alternatives and bail out Date: Thu, 18 Sep 2025 13:54:43 +0200 Message-ID: <19777bc710bf901ffbb0ad0f1bb57b18fc01b163.1758196090.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" The caller script may not want an automatic execution of the new version. Add two parameters to allow showing alternatives and to bail out if version is incompatible. Signed-off-by: Mauro Carvalho Chehab --- tools/docs/sphinx-pre-install | 48 +++++++++++++++++++++++++++-------- 1 file changed, 38 insertions(+), 10 deletions(-) diff --git a/tools/docs/sphinx-pre-install b/tools/docs/sphinx-pre-install index fa12e01fc7fe..895220c6d4f8 100755 --- a/tools/docs/sphinx-pre-install +++ b/tools/docs/sphinx-pre-install @@ -254,7 +254,7 @@ class AncillaryMethods: return (0, 0, 0) =20 @staticmethod - def find_python(): + def find_python(min_version): """ Detect if are out there any python 3.xy version newer than the current one. @@ -263,21 +263,25 @@ class AncillaryMethods: may need to update it one day, hopefully on a distant future. """ patterns =3D [ - "python3.[0-9]", "python3.[0-9][0-9]", + "python3.[0-9]", ] =20 # Seek for a python binary newer than MIN_PYTHON_VERSION + python_cmd =3D [] for path in os.getenv("PATH", "").split(":"): for pattern in patterns: for cmd in glob(os.path.join(path, pattern)): if os.path.isfile(cmd) and os.access(cmd, os.X_OK): version =3D SphinxDependencyChecker.get_python_ver= sion(cmd) - if version >=3D MIN_PYTHON_VERSION: - return cmd + if version >=3D min_version: + python_cmd.append((version, cmd)) + + return sorted(python_cmd, reverse=3DTrue) =20 @staticmethod - def check_python(): + def check_python(min_version, show_alternatives=3DFalse, bail_out=3DFa= lse, + success_on_error=3DFalse): """ Check if the current python binary satisfies our minimal requireme= nt for Sphinx build. If not, re-run with a newer version if found. @@ -301,18 +305,42 @@ class AncillaryMethods: =20 python_ver =3D ver_str(cur_ver) =20 - new_python_cmd =3D SphinxDependencyChecker.find_python() - if not new_python_cmd: + available_versions =3D SphinxDependencyChecker.find_python(min_ver= sion) + if not available_versions: print(f"ERROR: Python version {python_ver} is not spported any= more\n") print(" Can't find a new version. This script may fail") return =20 - # Restart script using the newer version script_path =3D os.path.abspath(sys.argv[0]) - args =3D [new_python_cmd, script_path] + sys.argv[1:] + + # Check possible alternatives + if available_versions: + new_python_cmd =3D available_versions[0][1] + else: + new_python_cmd =3D None + + if show_alternatives: + print("You could run, instead:") + for _, cmd in available_versions: + args =3D [cmd, script_path] + sys.argv[1:] + + cmd_str =3D " ".join(args) + print(f" {cmd_str}") + print() + + if bail_out: + msg =3D f"Python {python_ver} not supported. Bailing out" + if success_on_error: + print(msg, file=3Dsys.stderr) + sys.exit(0) + else: + sys.exit(msg) =20 print(f"Python {python_ver} not supported. Changing to {new_python= _cmd}") =20 + # Restart script using the newer version + args =3D [new_python_cmd, script_path] + sys.argv[1:] + try: os.execv(new_python_cmd, args) except OSError as e: @@ -1612,7 +1640,7 @@ def main(): =20 checker =3D SphinxDependencyChecker(args) =20 - checker.check_python() + checker.check_python(MIN_PYTHON_VERSION) checker.check_needs() =20 # Call main if not used as module --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A1281303A01; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; cv=none; b=VqGCPJ88A9fHkKM18xAYw/0X5euwZgwH2Y9pxWJIPVUR5dmFVMFAfwRPxi8F5BQOhVgY5O6ulgpOWHQ7jKUBhPbcIPU3VLuZ0x+WoG7obPE4uMJY9aqmqh2Ge6nxR1iDk5ysrKjz3juRtCBmjAYJQcIU8iWjuF4AZUZUHMgoMVE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; c=relaxed/simple; bh=uFmDzg+Gwc2jpPZcX4mIfP+EFkRf+cZmvTJP34ibSMs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=aNAOjQTlnEYAhOydPhFYWp3osLybR9a7KJd2WI5mzRRDMy8hym0CPCRmTShzHrncoIDhRqKGAqQapRqQm97V4Fyyc61AR6ovMKHVXUyv+0gq/po5czo3NA0TCh898Bcb8UbL4ndUeDueUP81JP6pORhgaIKZPB1oQjhiXVCEUXc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Tiy0Z478; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Tiy0Z478" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3092BC116B1; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196506; bh=uFmDzg+Gwc2jpPZcX4mIfP+EFkRf+cZmvTJP34ibSMs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Tiy0Z478OO3qAbIjT3Zz/MRtaBqZeuOoCrEKQAYGjK1/w9rJWfe7W0J64sl+j4But fNkJvWxcBhe2NVRpiToHV1ELq0Lg0a6VcwrolcM2P739l+OlDw8OtBJWD9S+EjDIT0 sCp9LI/wf4EIliEcFu2Vap10AUqLIPvy28vybmp10tPqzaTpZcLglyo0GULLrxivdi kr+12z/+cNVXenwtn0flgkGMguhxaRxz4NXpacfakakRor0+VbddR76iyjtCf6vrZT AjgayMxutIZy/BVP3WeOS7N2FhNDCTp7jWqSwdVhAcS5R8iA+8X8cB8yX5zMzO+RqM nYTbmBjRM4uEg== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE8-0000000Crq1-02bw; Thu, 18 Sep 2025 13:55:04 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v8 10/24] tools/docs: python_version: move version check from sphinx-pre-install Date: Thu, 18 Sep 2025 13:54:44 +0200 Message-ID: X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab The sphinx-pre-install code has some logic to deal with Python version, which ensures that a minimal version will be enforced for documentation build logic. Move it to a separate library to allow re-using its code. No functional changes. Signed-off-by: Mauro Carvalho Chehab --- tools/docs/lib/python_version.py | 158 +++++++++++++++++++++++++++++++ tools/docs/sphinx-pre-install | 147 +++------------------------- 2 files changed, 171 insertions(+), 134 deletions(-) create mode 100644 tools/docs/lib/python_version.py diff --git a/tools/docs/lib/python_version.py b/tools/docs/lib/python_versi= on.py new file mode 100644 index 000000000000..a9fda2470a26 --- /dev/null +++ b/tools/docs/lib/python_version.py @@ -0,0 +1,158 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2017-2025 Mauro Carvalho Chehab + +""" +Handle Python version check logic. + +Not all Python versions are supported by scripts. Yet, on some cases, +like during documentation build, a newer version of python could be +available. + +This class allows checking if the minimal requirements are followed. + +Better than that, PythonVersion.check_python() not only checks the minimal +requirements, but it automatically switches to a the newest available +Python version if present. + +""" + +import os +import re +import subprocess +import sys + +from glob import glob + +class PythonVersion: + """ + Ancillary methods that checks for missing dependencies for different + types of types, like binaries, python modules, rpm deps, etc. + """ + + def __init__(self, version): + """=C3=8Fnitialize self.version tuple from a version string""" + self.version =3D self.parse_version(version) + + @staticmethod + def parse_version(version): + """Convert a major.minor.patch version into a tuple""" + return tuple(int(x) for x in version.split(".")) + + @staticmethod + def ver_str(version): + """Returns a version tuple as major.minor.patch""" + return ".".join([str(x) for x in version]) + + def __str__(self): + """Returns a version tuple as major.minor.patch from self.version"= "" + return self.ver_str(self.version) + + @staticmethod + def get_python_version(cmd): + """ + Get python version from a Python binary. As we need to detect if + are out there newer python binaries, we can't rely on sys.release = here. + """ + + kwargs =3D {} + if sys.version_info < (3, 7): + kwargs['universal_newlines'] =3D True + else: + kwargs['text'] =3D True + + result =3D subprocess.run([cmd, "--version"], + stdout =3D subprocess.PIPE, + stderr =3D subprocess.PIPE, + **kwargs, check=3DFalse) + + version =3D result.stdout.strip() + + match =3D re.search(r"(\d+\.\d+\.\d+)", version) + if match: + return PythonVersion.parse_version(match.group(1)) + + print(f"Can't parse version {version}") + return (0, 0, 0) + + @staticmethod + def find_python(min_version): + """ + Detect if are out there any python 3.xy version newer than the + current one. + + Note: this routine is limited to up to 2 digits for python3. We + may need to update it one day, hopefully on a distant future. + """ + patterns =3D [ + "python3.[0-9][0-9]", + "python3.[0-9]", + ] + + python_cmd =3D [] + + # Seek for a python binary newer than min_version + for path in os.getenv("PATH", "").split(":"): + for pattern in patterns: + for cmd in glob(os.path.join(path, pattern)): + if os.path.isfile(cmd) and os.access(cmd, os.X_OK): + version =3D PythonVersion.get_python_version(cmd) + if version >=3D min_version: + python_cmd.append((version, cmd)) + + return sorted(python_cmd, reverse=3DTrue) + + @staticmethod + def check_python(min_version, show_alternatives=3DFalse, bail_out=3DFa= lse, + success_on_error=3DFalse): + """ + Check if the current python binary satisfies our minimal requireme= nt + for Sphinx build. If not, re-run with a newer version if found. + """ + cur_ver =3D sys.version_info[:3] + if cur_ver >=3D min_version: + ver =3D PythonVersion.ver_str(cur_ver) + return + + python_ver =3D PythonVersion.ver_str(cur_ver) + + available_versions =3D PythonVersion.find_python(min_version) + if not available_versions: + print(f"ERROR: Python version {python_ver} is not spported any= more\n") + print(" Can't find a new version. This script may fail") + return + + script_path =3D os.path.abspath(sys.argv[0]) + + # Check possible alternatives + if available_versions: + new_python_cmd =3D available_versions[0][1] + else: + new_python_cmd =3D None + + if show_alternatives: + print("You could run, instead:") + for _, cmd in available_versions: + args =3D [cmd, script_path] + sys.argv[1:] + + cmd_str =3D " ".join(args) + print(f" {cmd_str}") + print() + + if bail_out: + msg =3D f"Python {python_ver} not supported. Bailing out" + if success_on_error: + print(msg, file=3Dsys.stderr) + sys.exit(0) + else: + sys.exit(msg) + + print(f"Python {python_ver} not supported. Changing to {new_python= _cmd}") + + # Restart script using the newer version + args =3D [new_python_cmd, script_path] + sys.argv[1:] + + try: + os.execv(new_python_cmd, args) + except OSError as e: + sys.exit(f"Failed to restart with {new_python_cmd}: {e}") diff --git a/tools/docs/sphinx-pre-install b/tools/docs/sphinx-pre-install index 895220c6d4f8..d6d673b7945c 100755 --- a/tools/docs/sphinx-pre-install +++ b/tools/docs/sphinx-pre-install @@ -32,20 +32,10 @@ import subprocess import sys from glob import glob =20 +from lib.python_version import PythonVersion =20 -def parse_version(version): - """Convert a major.minor.patch version into a tuple""" - return tuple(int(x) for x in version.split(".")) - - -def ver_str(version): - """Returns a version tuple as major.minor.patch""" - - return ".".join([str(x) for x in version]) - - -RECOMMENDED_VERSION =3D parse_version("3.4.3") -MIN_PYTHON_VERSION =3D parse_version("3.7") +RECOMMENDED_VERSION =3D PythonVersion("3.4.3").version +MIN_PYTHON_VERSION =3D PythonVersion("3.7").version =20 =20 class DepManager: @@ -235,122 +225,11 @@ class AncillaryMethods: =20 return None =20 - @staticmethod - def get_python_version(cmd): - """ - Get python version from a Python binary. As we need to detect if - are out there newer python binaries, we can't rely on sys.release = here. - """ - - result =3D SphinxDependencyChecker.run([cmd, "--version"], - capture_output=3DTrue, text=3D= True) - version =3D result.stdout.strip() - - match =3D re.search(r"(\d+\.\d+\.\d+)", version) - if match: - return parse_version(match.group(1)) - - print(f"Can't parse version {version}") - return (0, 0, 0) - - @staticmethod - def find_python(min_version): - """ - Detect if are out there any python 3.xy version newer than the - current one. - - Note: this routine is limited to up to 2 digits for python3. We - may need to update it one day, hopefully on a distant future. - """ - patterns =3D [ - "python3.[0-9][0-9]", - "python3.[0-9]", - ] - - # Seek for a python binary newer than MIN_PYTHON_VERSION - python_cmd =3D [] - for path in os.getenv("PATH", "").split(":"): - for pattern in patterns: - for cmd in glob(os.path.join(path, pattern)): - if os.path.isfile(cmd) and os.access(cmd, os.X_OK): - version =3D SphinxDependencyChecker.get_python_ver= sion(cmd) - if version >=3D min_version: - python_cmd.append((version, cmd)) - - return sorted(python_cmd, reverse=3DTrue) - - @staticmethod - def check_python(min_version, show_alternatives=3DFalse, bail_out=3DFa= lse, - success_on_error=3DFalse): - """ - Check if the current python binary satisfies our minimal requireme= nt - for Sphinx build. If not, re-run with a newer version if found. - """ - cur_ver =3D sys.version_info[:3] - if cur_ver >=3D MIN_PYTHON_VERSION: - ver =3D ver_str(cur_ver) - - # This could be useful for debugging purposes - if SphinxDependencyChecker.which("docutils"): - result =3D SphinxDependencyChecker.run(["docutils", "--ver= sion"], - capture_output=3DTrue,= text=3DTrue) - ver =3D result.stdout.strip() - match =3D re.search(r"(\d+\.\d+\.\d+)", ver) - if match: - ver =3D match.group(1) - - print(f"Docutils version: {ver}") - - return - - python_ver =3D ver_str(cur_ver) - - available_versions =3D SphinxDependencyChecker.find_python(min_ver= sion) - if not available_versions: - print(f"ERROR: Python version {python_ver} is not spported any= more\n") - print(" Can't find a new version. This script may fail") - return - - script_path =3D os.path.abspath(sys.argv[0]) - - # Check possible alternatives - if available_versions: - new_python_cmd =3D available_versions[0][1] - else: - new_python_cmd =3D None - - if show_alternatives: - print("You could run, instead:") - for _, cmd in available_versions: - args =3D [cmd, script_path] + sys.argv[1:] - - cmd_str =3D " ".join(args) - print(f" {cmd_str}") - print() - - if bail_out: - msg =3D f"Python {python_ver} not supported. Bailing out" - if success_on_error: - print(msg, file=3Dsys.stderr) - sys.exit(0) - else: - sys.exit(msg) - - print(f"Python {python_ver} not supported. Changing to {new_python= _cmd}") - - # Restart script using the newer version - args =3D [new_python_cmd, script_path] + sys.argv[1:] - - try: - os.execv(new_python_cmd, args) - except OSError as e: - sys.exit(f"Failed to restart with {new_python_cmd}: {e}") - @staticmethod def run(*args, **kwargs): """ Excecute a command, hiding its output by default. - Preserve comatibility with older Python versions. + Preserve compatibility with older Python versions. """ =20 capture_output =3D kwargs.pop('capture_output', False) @@ -554,11 +433,11 @@ class MissingCheckers(AncillaryMethods): for line in result.stdout.split("\n"): match =3D re.match(r"^sphinx-build\s+([\d\.]+)(?:\+(?:/[\da-f]= +)|b\d+)?\s*$", line) if match: - return parse_version(match.group(1)) + return PythonVersion.parse_version(match.group(1)) =20 match =3D re.match(r"^Sphinx.*\s+([\d\.]+)\s*$", line) if match: - return parse_version(match.group(1)) + return PythonVersion.parse_version(match.group(1)) =20 def check_sphinx(self, conf): """ @@ -569,7 +448,7 @@ class MissingCheckers(AncillaryMethods): for line in f: match =3D re.match(r"^\s*needs_sphinx\s*=3D\s*[\'\"]([= \d\.]+)[\'\"]", line) if match: - self.min_version =3D parse_version(match.group(1)) + self.min_version =3D PythonVersion.parse_version(m= atch.group(1)) break except IOError: sys.exit(f"Can't open {conf}") @@ -589,8 +468,8 @@ class MissingCheckers(AncillaryMethods): sys.exit(f"{sphinx} didn't return its version") =20 if self.cur_version < self.min_version: - curver =3D ver_str(self.cur_version) - minver =3D ver_str(self.min_version) + curver =3D PythonVersion.ver_str(self.cur_version) + minver =3D PythonVersion.ver_str(self.min_version) =20 print(f"ERROR: Sphinx version is {curver}. It should be >=3D {= minver}") self.need_sphinx =3D 1 @@ -1331,7 +1210,7 @@ class SphinxDependencyChecker(MissingCheckers): else: if self.need_sphinx and ver >=3D self.min_version: return (f, ver) - elif parse_version(ver) > self.cur_version: + elif PythonVersion.parse_version(ver) > self.cur_version: return (f, ver) =20 return ("", ver) @@ -1438,7 +1317,7 @@ class SphinxDependencyChecker(MissingCheckers): return =20 if self.latest_avail_ver: - latest_avail_ver =3D ver_str(self.latest_avail_ver) + latest_avail_ver =3D PythonVersion.ver_str(self.latest_avail_v= er) =20 if not self.need_sphinx: # sphinx-build is present and its version is >=3D $min_version @@ -1534,7 +1413,7 @@ class SphinxDependencyChecker(MissingCheckers): else: print("Unknown OS") if self.cur_version !=3D (0, 0, 0): - ver =3D ver_str(self.cur_version) + ver =3D PythonVersion.ver_str(self.cur_version) print(f"Sphinx version: {ver}\n") =20 # Check the type of virtual env, depending on Python version @@ -1640,7 +1519,7 @@ def main(): =20 checker =3D SphinxDependencyChecker(args) =20 - checker.check_python(MIN_PYTHON_VERSION) + PythonVersion.check_python(MIN_PYTHON_VERSION) checker.check_needs() =20 # Call main if not used as module --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE270303A31; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; cv=none; b=lEeWiMRQHB8d9jTsBITqTB0KXMw8RBb6wHkOIevCgbMWor6U3bvVROKV9cH1AZlu5WOw8ta/az5PLqq1tqM7YF3v6XUyVbVxCsAbcOtSyS9RyRdSxhr/B9a93rSmZhDXCVYmt1wGjxm93yLSjyrptLIixu3bmJEUVMKe620DLVE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; c=relaxed/simple; bh=0Y7xHKz4h3eGEkKm5XfGE40A/GthvIC7Xyzd3QVPaDw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=clrnRQ+me61/dubRyATRVGD0V5+w+bjkQuNYtMt9OW09WloyAqv3DRBcDYxsPCxHIzp+h5SUXs5vGYqV+o328voFbXV1Cgq9x3JGLH6AmXvuCkPNaM4sjzU4OkDkgbeBOA/RjBoCDdRUEr1kwbSZtv/Z/pVxDCxu778g7h/4YPQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oRDIVvCO; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oRDIVvCO" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3628EC16AAE; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196506; bh=0Y7xHKz4h3eGEkKm5XfGE40A/GthvIC7Xyzd3QVPaDw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oRDIVvCO7FbMxiUwJXdJ49l4ucmXYxcya3SoD+tO99X/fOa4TMOx3hjbNWQBCjS8p Ph8CciOCBQCcyTLKMJqMuMbwDJA9Bn/A5STO5zeLjka3RJOhIl6FVy6pso07NvyGK3 s2C1pX+EMfvy04RQdgHVV55W3KrxpPMihFTmcWs29PGCrfvhSXhDs8mfZSM/KwS7fM 8w65IkEIdmPtIo3javecW2SySTY/3Txlwc+SCQZvShUCv1UMKflAgaOml87LVtraG3 N0/RbMpWcdb7JdzqRZQEUMDJuGtex5FAu+XYmKyVvavJA4PjUaavJVJ/BIJ5E2I+en tUUDh8jgFBwnQ== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE8-0000000Crqk-0CBO; Thu, 18 Sep 2025 13:55:04 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , "Mauro Carvalho Chehab" , Alex Gaynor , Alice Ryhl , Andreas Hindborg , Benno Lossin , Boqun Feng , Danilo Krummrich , Gary Guo , Miguel Ojeda , Trevor Gross , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org Subject: [PATCH v8 11/24] tools/docs: sphinx-build-wrapper: add a wrapper for sphinx-build Date: Thu, 18 Sep 2025 13:54:45 +0200 Message-ID: <80ae57b01fcfb1d338d93b8f8e26e57b69b5f16b.1758196090.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" There are too much magic inside docs Makefile to properly run sphinx-build. Create an ancillary script that contains all kernel-related sphinx-build call logic currently at Makefile. Such script is designed to work both as an standalone command and as part of a Makefile. As such, it properly handles POSIX jobserver used by GNU make. On a side note, there was a line number increase due to the conversion (ignoring comments) is: Documentation/Makefile | 131 +++---------- tools/docs/sphinx-build-wrapper | 293 +++++++++++++++++++++++++++++++ 2 files changed, 323 insertions(+), 101 deletions(-) Comments and descriptions adds: tools/docs/sphinx-build-wrapper | 261 +++++++++++++++++++++++++++++++- So, about half of the script are comments/descriptions. This is because some things are more verbosed on Python and because it requires reading env vars from Makefile. Besides it, this script has some extra features that don't exist at the Makefile: - It can be called directly from command line; - It properly return PDF build errors. When running the script alone, it will only take handle sphinx-build targets. On other words, it won't runn make rustdoc after building htmlfiles, nor it will run the extra check scripts. Signed-off-by: Mauro Carvalho Chehab --- Documentation/Makefile | 129 ++----- tools/docs/sphinx-build-wrapper | 599 ++++++++++++++++++++++++++++++++ 2 files changed, 627 insertions(+), 101 deletions(-) create mode 100755 tools/docs/sphinx-build-wrapper diff --git a/Documentation/Makefile b/Documentation/Makefile index fd6399c79fab..380284026c13 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -23,21 +23,22 @@ SPHINXOPTS =3D SPHINXDIRS =3D . DOCS_THEME =3D DOCS_CSS =3D -_SPHINXDIRS =3D $(sort $(patsubst $(srctree)/Documentation/%/index.rst,%= ,$(wildcard $(srctree)/Documentation/*/index.rst))) SPHINX_CONF =3D conf.py PAPER =3D BUILDDIR =3D $(obj)/output PDFLATEX =3D xelatex LATEXOPTS =3D -interaction=3Dbatchmode -no-shell-escape =20 +PYTHONPYCACHEPREFIX ?=3D $(abspath $(BUILDDIR)/__pycache__) + +# Wrapper for sphinx-build + +BUILD_WRAPPER =3D $(srctree)/tools/docs/sphinx-build-wrapper + # For denylisting "variable font" files # Can be overridden by setting as an env variable FONTS_CONF_DENY_VF ?=3D $(HOME)/deny-vf =20 -ifeq ($(findstring 1, $(KBUILD_VERBOSE)),) -SPHINXOPTS +=3D "-q" -endif - # User-friendly check for sphinx-build HAVE_SPHINX :=3D $(shell if which $(SPHINXBUILD) >/dev/null 2>&1; then ech= o 1; else echo 0; fi) =20 @@ -51,63 +52,29 @@ ifeq ($(HAVE_SPHINX),0) =20 else # HAVE_SPHINX =20 -# User-friendly check for pdflatex and latexmk -HAVE_PDFLATEX :=3D $(shell if which $(PDFLATEX) >/dev/null 2>&1; then echo= 1; else echo 0; fi) -HAVE_LATEXMK :=3D $(shell if which latexmk >/dev/null 2>&1; then echo 1; e= lse echo 0; fi) +# Common documentation targets +infodocs texinfodocs latexdocs epubdocs xmldocs pdfdocs linkcheckdocs: + $(Q)@$(srctree)/tools/docs/sphinx-pre-install --version-check + +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \ + --sphinxdirs=3D"$(SPHINXDIRS)" --conf=3D"$(SPHINX_CONF)" \ + --builddir=3D"$(BUILDDIR)" --deny-vf=3D$(FONTS_CONF_DENY_VF) \ + --theme=3D$(DOCS_THEME) --css=3D$(DOCS_CSS) --paper=3D$(PAPER) =20 -ifeq ($(HAVE_LATEXMK),1) - PDFLATEX :=3D latexmk -$(PDFLATEX) -endif #HAVE_LATEXMK - -# Internal variables. -PAPEROPT_a4 =3D -D latex_elements.papersize=3Da4paper -PAPEROPT_letter =3D -D latex_elements.papersize=3Dletterpaper -ALLSPHINXOPTS =3D -D kerneldoc_srctree=3D$(srctree) -D kerneldoc_bin=3D$= (KERNELDOC) -ALLSPHINXOPTS +=3D $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) -ifneq ($(wildcard $(srctree)/.config),) -ifeq ($(CONFIG_RUST),y) - # Let Sphinx know we will include rustdoc - ALLSPHINXOPTS +=3D -t rustdoc -endif +# Special handling for pdfdocs +ifneq ($(shell which $(PDFLATEX) >/dev/null 2>&1; echo $$?),0) +pdfdocs: + $(warning The '$(PDFLATEX)' command was not found. Make sure you have it = installed and in PATH to produce PDF output.) + @echo " SKIP Sphinx $@ target." endif -# the i18n builder cannot share the environment and doctrees with the othe= rs -I18NSPHINXOPTS =3D $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . - -# commands; the 'cmd' from scripts/Kbuild.include is not *loopable* -loop_cmd =3D $(echo-cmd) $(cmd_$(1)) || exit; - -# $2 sphinx builder e.g. "html" -# $3 name of the build subfolder / e.g. "userspace-api/media", used as: -# * dest folder relative to $(BUILDDIR) and -# * cache folder relative to $(BUILDDIR)/.doctrees -# $4 dest subfolder e.g. "man" for man pages at userspace-api/media/man -# $5 reST source folder relative to $(src), -# e.g. "userspace-api/media" for the linux-tv book-set at ./Documentati= on/userspace-api/media - -PYTHONPYCACHEPREFIX ?=3D $(abspath $(BUILDDIR)/__pycache__) - -quiet_cmd_sphinx =3D SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4) - cmd_sphinx =3D \ - PYTHONPYCACHEPREFIX=3D"$(PYTHONPYCACHEPREFIX)" \ - BUILDDIR=3D$(abspath $(BUILDDIR)) SPHINX_CONF=3D$(abspath $(src)/$5/$(SPH= INX_CONF)) \ - $(PYTHON3) $(srctree)/scripts/jobserver-exec \ - $(CONFIG_SHELL) $(srctree)/Documentation/sphinx/parallel-wrapper.sh \ - $(SPHINXBUILD) \ - -b $2 \ - -c $(abspath $(src)) \ - -d $(abspath $(BUILDDIR)/.doctrees/$3) \ - -D version=3D$(KERNELVERSION) -D release=3D$(KERNELRELEASE) \ - $(ALLSPHINXOPTS) \ - $(abspath $(src)/$5) \ - $(abspath $(BUILDDIR)/$3/$4) && \ - if [ "x$(DOCS_CSS)" !=3D "x" ]; then \ - cp $(if $(patsubst /%,,$(DOCS_CSS)),$(abspath $(srctree)/$(DOCS_CSS)),$(= DOCS_CSS)) $(BUILDDIR)/$3/_static/; \ - fi =20 +# HTML main logic is identical to other targets. However, if rust is enabl= ed, +# an extra step at the end is required to generate rustdoc. htmldocs: - @$(srctree)/tools/docs/sphinx-pre-install --version-check - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var))) - + $(Q)@$(srctree)/tools/docs/sphinx-pre-install --version-check + +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \ + --sphinxdirs=3D"$(SPHINXDIRS)" --conf=3D"$(SPHINX_CONF)" \ + --builddir=3D"$(BUILDDIR)" \ + --theme=3D$(DOCS_THEME) --css=3D$(DOCS_CSS) --paper=3D$(PAPER) # If Rust support is available and .config exists, add rustdoc generated c= ontents. # If there are any, the errors from this make rustdoc will be displayed but # won't stop the execution of htmldocs @@ -118,49 +85,6 @@ ifeq ($(CONFIG_RUST),y) endif endif =20 -texinfodocs: - @$(srctree)/tools/docs/sphinx-pre-install --version-check - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,texinfo,$(var),texin= fo,$(var))) - -# Note: the 'info' Make target is generated by sphinx itself when -# running the texinfodocs target define above. -infodocs: texinfodocs - $(MAKE) -C $(BUILDDIR)/texinfo info - -linkcheckdocs: - @$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,linkcheck,$(var),,$(v= ar))) - -latexdocs: - @$(srctree)/tools/docs/sphinx-pre-install --version-check - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$= (var))) - -ifeq ($(HAVE_PDFLATEX),0) - -pdfdocs: - $(warning The '$(PDFLATEX)' command was not found. Make sure you have it = installed and in PATH to produce PDF output.) - @echo " SKIP Sphinx $@ target." - -else # HAVE_PDFLATEX - -pdfdocs: DENY_VF =3D XDG_CONFIG_HOME=3D$(FONTS_CONF_DENY_VF) -pdfdocs: latexdocs - @$(srctree)/tools/docs/sphinx-pre-install --version-check - $(foreach var,$(SPHINXDIRS), \ - $(MAKE) PDFLATEX=3D"$(PDFLATEX)" LATEXOPTS=3D"$(LATEXOPTS)" $(DENY_VF)= -C $(BUILDDIR)/$(var)/latex || $(PYTHON3) $(srctree)/tools/docs/check-vari= able-fonts.py || exit; \ - mkdir -p $(BUILDDIR)/$(var)/pdf; \ - mv $(subst .tex,.pdf,$(wildcard $(BUILDDIR)/$(var)/latex/*.tex)) $(BUI= LDDIR)/$(var)/pdf/; \ - ) - -endif # HAVE_PDFLATEX - -epubdocs: - @$(srctree)/tools/docs/sphinx-pre-install --version-check - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(v= ar))) - -xmldocs: - @$(srctree)/tools/docs/sphinx-pre-install --version-check - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var= ))) - endif # HAVE_SPHINX =20 # The following targets are independent of HAVE_SPHINX, and the rules shou= ld @@ -172,6 +96,9 @@ refcheckdocs: cleandocs: $(Q)rm -rf $(BUILDDIR) =20 +# Used only on help +_SPHINXDIRS =3D $(sort $(patsubst $(srctree)/Documentation/%/index.rst,%= ,$(wildcard $(srctree)/Documentation/*/index.rst))) + dochelp: @echo ' Linux kernel internal documentation in different formats from Re= ST:' @echo ' htmldocs - HTML' diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrap= per new file mode 100755 index 000000000000..c57c732b879c --- /dev/null +++ b/tools/docs/sphinx-build-wrapper @@ -0,0 +1,599 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2025 Mauro Carvalho Chehab +# +# pylint: disable=3DR0902, R0912, R0913, R0914, R0915, R0917, C0103 +# +# Converted from docs Makefile and parallel-wrapper.sh, both under +# GPLv2, copyrighted since 2008 by the following authors: +# +# Akira Yokosawa +# Arnd Bergmann +# Breno Leitao +# Carlos Bilbao +# Dave Young +# Donald Hunter +# Geert Uytterhoeven +# Jani Nikula +# Jan Stancek +# Jonathan Corbet +# Joshua Clayton +# Kees Cook +# Linus Torvalds +# Magnus Damm +# Masahiro Yamada +# Mauro Carvalho Chehab +# Maxim Cournoyer +# Peter Foley +# Randy Dunlap +# Rob Herring +# Shuah Khan +# Thorsten Blum +# Tomas Winkler + + +""" +Sphinx build wrapper that handles Kernel-specific business rules: + +- it gets the Kernel build environment vars; +- it determines what's the best parallelism; +- it handles SPHINXDIRS + +This tool ensures that MIN_PYTHON_VERSION is satisfied. If version is +below that, it seeks for a new Python version. If found, it re-runs using +the newer version. +""" + +import argparse +import os +import shlex +import shutil +import subprocess +import sys + +from lib.python_version import PythonVersion +from lib.latex_fonts import LatexFontChecker + +LIB_DIR =3D "../../scripts/lib" +SRC_DIR =3D os.path.dirname(os.path.realpath(__file__)) + +sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR)) + +from jobserver import JobserverExec # pylint: disable=3DC0413,C041= 1,E0401 + +# +# Some constants +# +MIN_PYTHON_VERSION =3D PythonVersion("3.7").version +PAPER =3D ["", "a4", "letter"] + +TARGETS =3D { + "cleandocs": { "builder": "clean" }, + "linkcheckdocs": { "builder": "linkcheck" }, + "htmldocs": { "builder": "html" }, + "epubdocs": { "builder": "epub", "out_dir": "epub" }, + "texinfodocs": { "builder": "texinfo", "out_dir": "texinfo" }, + "infodocs": { "builder": "texinfo", "out_dir": "texinfo" }, + "latexdocs": { "builder": "latex", "out_dir": "latex" }, + "pdfdocs": { "builder": "latex", "out_dir": "latex" }, + "xmldocs": { "builder": "xml", "out_dir": "xml" }, +} + + +# +# SphinxBuilder class +# + +class SphinxBuilder: + """ + Handles a sphinx-build target, adding needed arguments to build + with the Kernel. + """ + + def is_rust_enabled(self): + """Check if rust is enabled at .config""" + config_path =3D os.path.join(self.srctree, ".config") + if os.path.isfile(config_path): + with open(config_path, "r", encoding=3D"utf-8") as f: + return "CONFIG_RUST=3Dy" in f.read() + return False + + def get_path(self, path, use_cwd=3DFalse, abs_path=3DFalse): + """ + Ancillary routine to handle patches the right way, as shell does. + + It first expands "~" and "~user". Then, if patch is not absolute, + join self.srctree. Finally, if requested, convert to abspath. + """ + + path =3D os.path.expanduser(path) + if not path.startswith("/"): + if use_cwd: + base =3D os.getcwd() + else: + base =3D self.srctree + + path =3D os.path.join(base, path) + + if abs_path: + return os.path.abspath(path) + + return path + + def get_sphinx_extra_opts(self, n_jobs): + """ + Get the number of jobs to be used for docs build passed via command + line and desired sphinx verbosity. + + The number of jobs can be on different places: + + 1) It can be passed via "-j" argument; + 2) The SPHINXOPTS=3D"-j8" env var may have "-j"; + 3) if called via GNU make, -j specifies the desired number of jobs. + with GNU makefile, this number is available via POSIX jobserver; + 4) if none of the above is available, it should default to "-jauto= ", + and let sphinx decide the best value. + """ + + # + # SPHINXOPTS env var, if used, contains extra arguments to be used + # by sphinx-build time. Among them, it may contain sphinx verbosity + # and desired number of parallel jobs. + # + parser =3D argparse.ArgumentParser() + parser.add_argument('-j', '--jobs', type=3Dint) + parser.add_argument('-q', '--quiet', type=3Dint) + + # + # Other sphinx-build arguments go as-is, so place them + # at self.sphinxopts, using shell parser + # + sphinxopts =3D shlex.split(os.environ.get("SPHINXOPTS", "")) + + # + # Build a list of sphinx args, honoring verbosity here if specified + # + + verbose =3D self.verbose + sphinx_args, self.sphinxopts =3D parser.parse_known_args(sphinxopt= s) + if sphinx_args.quiet is True: + verbose =3D False + + # + # If the user explicitly sets "-j" at command line, use it. + # Otherwise, pick it from SPHINXOPTS args + # + if n_jobs: + self.n_jobs =3D n_jobs + elif sphinx_args.jobs: + self.n_jobs =3D sphinx_args.jobs + else: + self.n_jobs =3D None + + if not verbose: + self.sphinxopts +=3D ["-q"] + + def __init__(self, builddir, verbose=3DFalse, n_jobs=3DNone): + """Initialize internal variables""" + self.verbose =3D None + + # + # Normal variables passed from Kernel's makefile + # + self.kernelversion =3D os.environ.get("KERNELVERSION", "unknown") + self.kernelrelease =3D os.environ.get("KERNELRELEASE", "unknown") + self.pdflatex =3D os.environ.get("PDFLATEX", "xelatex") + self.latexopts =3D os.environ.get("LATEXOPTS", "-interaction=3Dbat= chmode -no-shell-escape") + + if not verbose: + verbose =3D bool(os.environ.get("KBUILD_VERBOSE", "") !=3D "") + + if verbose is not None: + self.verbose =3D verbose + + # + # Source tree directory. This needs to be at os.environ, as + # Sphinx extensions use it + # + self.srctree =3D os.environ.get("srctree") + if not self.srctree: + self.srctree =3D "." + os.environ["srctree"] =3D self.srctree + + # + # Now that we can expand srctree, get other directories as well + # + self.sphinxbuild =3D os.environ.get("SPHINXBUILD", "sphinx-build") + self.kerneldoc =3D self.get_path(os.environ.get("KERNELDOC", + "scripts/kernel-doc.= py")) + self.builddir =3D self.get_path(builddir, use_cwd=3DTrue, abs_path= =3DTrue) + + self.config_rust =3D self.is_rust_enabled() + + # + # Get directory locations for LaTeX build toolchain + # + self.pdflatex_cmd =3D shutil.which(self.pdflatex) + self.latexmk_cmd =3D shutil.which("latexmk") + + self.env =3D os.environ.copy() + + self.get_sphinx_extra_opts(n_jobs) + + def run_sphinx(self, sphinx_build, build_args, *args, **pwargs): + """ + Executes sphinx-build using current python3 command. + + When calling via GNU make, POSIX jobserver is used to tell how + many jobs are still available from a job pool. claim all remaining + jobs, as we don't want sphinx-build to run in parallel with other + jobs. + + Despite that, the user may actually force a different value than + the number of available jobs via command line. + + The "with" logic here is used to ensure that the claimed jobs will + be freed once subprocess finishes + """ + + with JobserverExec() as jobserver: + if jobserver.claim: + # + # when GNU make is used, claim available jobs from jobserv= er + # + n_jobs =3D str(jobserver.claim) + else: + # + # Otherwise, let sphinx decide by default + # + n_jobs =3D "auto" + + # + # If explicitly requested via command line, override default + # + if self.n_jobs: + n_jobs =3D str(self.n_jobs) + + cmd =3D [sys.executable, sphinx_build] + cmd +=3D [f"-j{n_jobs}"] + cmd +=3D self.sphinxopts + cmd +=3D build_args + + if self.verbose: + print(" ".join(cmd)) + + return subprocess.call(cmd, *args, **pwargs) + + def handle_html(self, css, output_dir): + """ + Extra steps for HTML and epub output. + + For such targets, we need to ensure that CSS will be properly + copied to the output _static directory + """ + + if not css: + return + + css =3D os.path.expanduser(css) + if not css.startswith("/"): + css =3D os.path.join(self.srctree, css) + + static_dir =3D os.path.join(output_dir, "_static") + os.makedirs(static_dir, exist_ok=3DTrue) + + try: + shutil.copy2(css, static_dir) + except (OSError, IOError) as e: + print(f"Warning: Failed to copy CSS: {e}", file=3Dsys.stderr) + + def handle_pdf(self, output_dirs, deny_vf): + """ + Extra steps for PDF output. + + As PDF is handled via a LaTeX output, after building the .tex file, + a new build is needed to create the PDF output from the latex + directory. + """ + builds =3D {} + max_len =3D 0 + + # + # Since early 2024, Fedora and openSUSE tumbleweed have started + # deploying variable-font format of "Noto CJK", causing LaTeX + # to break with CJK. Work around it, by denying the variable font + # usage during xelatex build by passing the location of a config + # file with a deny list. + # + # See tools/docs/lib/latex_fonts.py for more details. + # + if deny_vf: + deny_vf =3D os.path.expanduser(deny_vf) + if os.path.isdir(deny_vf): + self.env["XDG_CONFIG_HOME"] =3D deny_vf + + for from_dir in output_dirs: + pdf_dir =3D os.path.join(from_dir, "../pdf") + os.makedirs(pdf_dir, exist_ok=3DTrue) + + if self.latexmk_cmd: + latex_cmd =3D [self.latexmk_cmd, f"-{self.pdflatex}"] + else: + latex_cmd =3D [self.pdflatex] + + latex_cmd.extend(shlex.split(self.latexopts)) + + tex_suffix =3D ".tex" + + # + # Process each .tex file + # + + has_tex =3D False + build_failed =3D False + with os.scandir(from_dir) as it: + for entry in it: + if not entry.name.endswith(tex_suffix): + continue + + name =3D entry.name[:-len(tex_suffix)] + has_tex =3D True + + # + # LaTeX PDF error code is almost useless for us: + # any warning makes it non-zero. For kernel doc builds= it + # always return non-zero even when build succeeds. + # So, let's do the best next thing: check if all PDF + # files were built. If they're, print a summary and + # return 0 at the end of this function + # + try: + subprocess.run(latex_cmd + [entry.path], + cwd=3Dfrom_dir, check=3DTrue, env= =3Dself.env) + except subprocess.CalledProcessError: + pass + + pdf_name =3D name + ".pdf" + pdf_from =3D os.path.join(from_dir, pdf_name) + pdf_to =3D os.path.join(pdf_dir, pdf_name) + + if os.path.exists(pdf_from): + os.rename(pdf_from, pdf_to) + builds[name] =3D os.path.relpath(pdf_to, self.buil= ddir) + else: + builds[name] =3D "FAILED" + build_failed =3D True + + name =3D entry.name.removesuffix(".tex") + max_len =3D max(max_len, len(name)) + + if not has_tex: + name =3D os.path.basename(from_dir) + max_len =3D max(max_len, len(name)) + builds[name] =3D "FAILED (no .tex)" + build_failed =3D True + + msg =3D "Summary" + msg +=3D "\n" + "=3D" * len(msg) + print() + print(msg) + + for pdf_name, pdf_file in builds.items(): + print(f"{pdf_name:<{max_len}}: {pdf_file}") + + print() + + if build_failed: + msg =3D LatexFontChecker().check() + if msg: + print(msg) + + sys.exit("PDF build failed: not all PDF files were created.") + else: + print("All PDF files were built.") + + def handle_info(self, output_dirs): + """ + Extra steps for Info output. + + For texinfo generation, an additional make is needed from the + texinfo directory. + """ + + for output_dir in output_dirs: + try: + subprocess.run(["make", "info"], cwd=3Doutput_dir, check= =3DTrue) + except subprocess.CalledProcessError as e: + sys.exit(f"Error generating info docs: {e}") + + def cleandocs(self, builder): # pylint: disable=3DW0613 + """Remove documentation output directory""" + shutil.rmtree(self.builddir, ignore_errors=3DTrue) + + def build(self, target, sphinxdirs=3DNone, conf=3D"conf.py", + theme=3DNone, css=3DNone, paper=3DNone, deny_vf=3DNone): + """ + Build documentation using Sphinx. This is the core function of this + module. It prepares all arguments required by sphinx-build. + """ + + builder =3D TARGETS[target]["builder"] + out_dir =3D TARGETS[target].get("out_dir", "") + + # + # Cleandocs doesn't require sphinx-build + # + if target =3D=3D "cleandocs": + self.cleandocs(builder) + return + + if theme: + os.environ["DOCS_THEME"] =3D theme + + # + # Other targets require sphinx-build, so check if it exists + # + sphinxbuild =3D shutil.which(self.sphinxbuild, path=3Dself.env["PA= TH"]) + if not sphinxbuild: + sys.exit(f"Error: {self.sphinxbuild} not found in PATH.\n") + + if builder =3D=3D "latex": + if not self.pdflatex_cmd and not self.latexmk_cmd: + sys.exit("Error: pdflatex or latexmk required for PDF gene= ration") + + docs_dir =3D os.path.abspath(os.path.join(self.srctree, "Documenta= tion")) + + # + # Fill in base arguments for Sphinx build + # + kerneldoc =3D self.kerneldoc + if kerneldoc.startswith(self.srctree): + kerneldoc =3D os.path.relpath(kerneldoc, self.srctree) + + args =3D [ "-b", builder, "-c", docs_dir ] + + if builder =3D=3D "latex": + if not paper: + paper =3D PAPER[1] + + args.extend(["-D", f"latex_elements.papersize=3D{paper}paper"]) + + if self.config_rust: + args.extend(["-t", "rustdoc"]) + + if conf: + self.env["SPHINX_CONF"] =3D self.get_path(conf, abs_path=3DTru= e) + + if not sphinxdirs: + sphinxdirs =3D os.environ.get("SPHINXDIRS", ".") + + # + # sphinxdirs can be a list or a whitespace-separated string + # + sphinxdirs_list =3D [] + for sphinxdir in sphinxdirs: + if isinstance(sphinxdir, list): + sphinxdirs_list +=3D sphinxdir + else: + sphinxdirs_list +=3D sphinxdir.split() + + # + # Step 1: Build each directory in separate. + # + # This is not the best way of handling it, as cross-references bet= ween + # them will be broken, but this is what we've been doing since + # the beginning. + # + output_dirs =3D [] + for sphinxdir in sphinxdirs_list: + src_dir =3D os.path.join(docs_dir, sphinxdir) + doctree_dir =3D os.path.join(self.builddir, ".doctrees") + output_dir =3D os.path.join(self.builddir, sphinxdir, out_dir) + + # + # Make directory names canonical + # + src_dir =3D os.path.normpath(src_dir) + doctree_dir =3D os.path.normpath(doctree_dir) + output_dir =3D os.path.normpath(output_dir) + + os.makedirs(doctree_dir, exist_ok=3DTrue) + os.makedirs(output_dir, exist_ok=3DTrue) + + output_dirs.append(output_dir) + + build_args =3D args + [ + "-d", doctree_dir, + "-D", f"kerneldoc_bin=3D{kerneldoc}", + "-D", f"version=3D{self.kernelversion}", + "-D", f"release=3D{self.kernelrelease}", + "-D", f"kerneldoc_srctree=3D{self.srctree}", + src_dir, + output_dir, + ] + + try: + self.run_sphinx(sphinxbuild, build_args, env=3Dself.env) + except (OSError, ValueError, subprocess.SubprocessError) as e: + sys.exit(f"Build failed: {repr(e)}") + + # + # Ensure that each html/epub output will have needed static fi= les + # + if target in ["htmldocs", "epubdocs"]: + self.handle_html(css, output_dir) + + # + # Step 2: Some targets (PDF and info) require an extra step once + # sphinx-build finishes + # + if target =3D=3D "pdfdocs": + self.handle_pdf(output_dirs, deny_vf) + elif target =3D=3D "infodocs": + self.handle_info(output_dirs) + +def jobs_type(value): + """ + Handle valid values for -j. Accepts Sphinx "-jauto", plus a number + equal or bigger than one. + """ + if value is None: + return None + + if value.lower() =3D=3D 'auto': + return value.lower() + + try: + if int(value) >=3D 1: + return value + + raise argparse.ArgumentTypeError(f"Minimum jobs is 1, got {value}") + except ValueError: + raise argparse.ArgumentTypeError(f"Must be 'auto' or positive inte= ger, got {value}") # pylint: disable=3DW0707 + +def main(): + """ + Main function. The only mandatory argument is the target. If not + specified, the other arguments will use default values if not + specified at os.environ. + """ + parser =3D argparse.ArgumentParser(description=3D"Kernel documentation= builder") + + parser.add_argument("target", choices=3Dlist(TARGETS.keys()), + help=3D"Documentation target to build") + parser.add_argument("--sphinxdirs", nargs=3D"+", + help=3D"Specific directories to build") + parser.add_argument("--conf", default=3D"conf.py", + help=3D"Sphinx configuration file") + parser.add_argument("--builddir", default=3D"output", + help=3D"Sphinx configuration file") + + parser.add_argument("--theme", help=3D"Sphinx theme to use") + + parser.add_argument("--css", help=3D"Custom CSS file for HTML/EPUB") + + parser.add_argument("--paper", choices=3DPAPER, default=3DPAPER[0], + help=3D"Paper size for LaTeX/PDF output") + + parser.add_argument('--deny-vf', + help=3D"Configuration to deny variable fonts on pd= f builds") + + parser.add_argument("-v", "--verbose", action=3D'store_true', + help=3D"place build in verbose mode") + + parser.add_argument('-j', '--jobs', type=3Djobs_type, + help=3D"Sets number of jobs to use with sphinx-bui= ld") + + args =3D parser.parse_args() + + PythonVersion.check_python(MIN_PYTHON_VERSION) + + builder =3D SphinxBuilder(builddir=3Dargs.builddir, + verbose=3Dargs.verbose, n_jobs=3Dargs.jobs) + + builder.build(args.target, sphinxdirs=3Dargs.sphinxdirs, conf=3Dargs.c= onf, + theme=3Dargs.theme, css=3Dargs.css, paper=3Dargs.paper, + deny_vf=3Dargs.deny_vf) + +if __name__ =3D=3D "__main__": + main() --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9378B30277A; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; cv=none; b=uXFYbfF09IQ1QeNYY0iCL0c6G4w1kJGM9bMvLjvTaNcZe8UigXDINSGIP9+5SmtnW5CsxZ/i3plsK7X2dFxROxwLqgOIX8dam9BGWngYmD/e50tVhMImYPSa8husydR8DT3OmSZGXdmCFHJyUsWp67EFDnqvacbtc0E4SBSp+co= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; c=relaxed/simple; bh=ZLbhSMLrYrG412X2oMrDythT1CiWtmLDdQQNSbEgmmI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PnO98p8p9UzISRFRG6sg0bvp5qLSxDauwLzZY2nSiVQNkT+fcw7c6XFMTuv7HlqVYdtoso5hJziCcF0SyTnnZSFWXLJ+qdrvu8IVT4l1lvvgIcb2y+NMUOkcSCbh5K4pS0dWDVHAzdjR4aVEf+K7IfNwROamqJTtZLlz7c3traY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Eg+UMlDD; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Eg+UMlDD" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 75F41C4CEFA; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196506; bh=ZLbhSMLrYrG412X2oMrDythT1CiWtmLDdQQNSbEgmmI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Eg+UMlDDJOfm7n5PKt6CgcXgfPPhzrybopuRXYko+f0Wk9hwwF+Kv6SyHCqp1yCcv N8Ip9GZ0ylAuz+6G5kDgh9XAVGVEeTY6z3JLRDmFx+570bTTZYs4E7aDYTcb7PYmBn BnAArogJApj8dNBbmTIkFXD6pWbZ1jNk62cwK2BsqBKoeKRahWnEVbSTtlcs1z924U fat+nk7dhrmye6FOIJ0K/OOrjEsiX4sngh7KlroBQmLK0hyyZ+i4kOXbUz7bb83s9o HR0vmAXgv1nUqKbHJIqLW2VDblkijUTPfMUW3P8zv3mj2a//1HdtFzkOizJ/s5oOZ9 K7lobiAqozGqA== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE8-0000000Crr2-0KdJ; Thu, 18 Sep 2025 13:55:04 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v8 12/24] docs: parallel-wrapper.sh: remove script Date: Thu, 18 Sep 2025 13:54:46 +0200 Message-ID: X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" The only usage of this script was docs Makefile. Now that it is using the new sphinx-build-wrapper, which has inside the code from parallel-wrapper.sh, we can drop this script. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/parallel-wrapper.sh | 33 ------------------------ 1 file changed, 33 deletions(-) delete mode 100644 Documentation/sphinx/parallel-wrapper.sh diff --git a/Documentation/sphinx/parallel-wrapper.sh b/Documentation/sphin= x/parallel-wrapper.sh deleted file mode 100644 index e54c44ce117d..000000000000 --- a/Documentation/sphinx/parallel-wrapper.sh +++ /dev/null @@ -1,33 +0,0 @@ -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0+ -# -# Figure out if we should follow a specific parallelism from the make -# environment (as exported by scripts/jobserver-exec), or fall back to -# the "auto" parallelism when "-jN" is not specified at the top-level -# "make" invocation. - -sphinx=3D"$1" -shift || true - -parallel=3D"$PARALLELISM" -if [ -z "$parallel" ] ; then - # If no parallelism is specified at the top-level make, then - # fall back to the expected "-jauto" mode that the "htmldocs" - # target has had. - auto=3D$(perl -e 'open IN,"'"$sphinx"' --version 2>&1 |"; - while () { - if (m/([\d\.]+)/) { - print "auto" if ($1 >=3D "1.7") - } - } - close IN') - if [ -n "$auto" ] ; then - parallel=3D"$auto" - fi -fi -# Only if some parallelism has been determined do we add the -jN option. -if [ -n "$parallel" ] ; then - parallel=3D"-j$parallel" -fi - -exec "$sphinx" $parallel "$@" --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B1E5D303C91; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; cv=none; b=Ru9ZSLecemwLAYg54SHNbRsCed+3gpZrbV3O7QMwwZ5BaX4Zb02/kID8Z/u3ZPVkCb5GViL9thLeYb2F3DnRV3ggXAthRF8vAbwT5ZOicdomiQ2jQbIka+aYCleoo9w8FxOuhZ+tLi1PesI5+n4hhfCbxAVBf29nxJDmgZYiM2c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; c=relaxed/simple; bh=0ypUcNGg9z6BJTKB3g3MJi0xHG6c16cyioz2ve+I/iQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=keEcZZUEJQy+oTKeD6FFBz5O1S9Z4+1Ez3PC1ZuVAqyDiArClrOer6lAnz99ES7Qz9lbTdJiIArm5ML4Cr4P9N28Sj8uNjB5sDf+tRJr2pYmq25iClgnjft+NUre9aDrA/FVhOU6E05H7sWVUFB6h2cpFf3+7lA9Jn0z9ljstVk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=BHE6ZxU0; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="BHE6ZxU0" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 909ADC4CEF7; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196506; bh=0ypUcNGg9z6BJTKB3g3MJi0xHG6c16cyioz2ve+I/iQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BHE6ZxU0ddOMfUSS863CWmDMeaG8NkUrH4b3PsnEGakWddytqBwbfVQrAfm7BMhN9 wYC1DsoZxoo9jOVuegwKI152o7S5GRzjiyXB8yMnrVuAk4gFuBkBE+lW9v3skYFXcq iOa37fd13wY7/QlXFAlmfWKpQiYrYEm21WqgbZVXg87nEXa510nXVtu/uD98jZWA9H gMl6lm5Sm8Q52Main/EuLZhwTme5ThMbXUspQjV/InMcAVuTdQ4IKgQG9Q5t467P0B hVinEnxyBXp9qQpc2MDeSAhdV/Z8m0/WCn+k9vwkBg+1ed8PfXjFvoEgwcFOmtmgVb e7gVik1SPIqSQ== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE8-0000000Crrk-0Twq; Thu, 18 Sep 2025 13:55:04 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v8 13/24] docs: Makefile: document latex/PDF PAPER= parameter Date: Thu, 18 Sep 2025 13:54:47 +0200 Message-ID: <9c7b34db18642081d22c36a4203f341c1100341e.1758196090.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" While the build system supports this for a long time, this was never documented. Add a documentation for it. Signed-off-by: Mauro Carvalho Chehab --- Documentation/Makefile | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Documentation/Makefile b/Documentation/Makefile index 380284026c13..a52b311cdadc 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -124,4 +124,6 @@ dochelp: @echo @echo ' make DOCS_CSS=3D{a .css file} adds a DOCS_CSS override file for= html/epub output.' @echo + @echo ' make PAPER=3D{a4|letter} Specifies the paper size used for LaTe= X/PDF output.' + @echo @echo ' Default location for the generated documents is Documentation/o= utput' --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA176303C9F; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; cv=none; b=RyXYQ/04DTcgCtzk4hvoSk8Vxl6PMONhCC53CyTTOPwnGX/fR3wY/j5VUmSu81DN++Bha/OGIPZDTP/DfkX9OPVIeRp9zNCYkJmQKqaQLSz7Dwbob0p0E/GoaJpdWTteDBjQNiDATGqoafU0ksblGc5oAZNO7KMc5J+26z8T/dw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196506; c=relaxed/simple; bh=mOGPpjFInnJF5ficZgf3c7TZFdWiehGB3ZFVn3w7fEg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=TAuP4zk7osbDT57elMwYEijpg1GDxYLi/5n+qWG6QvrxDYRy5U1/fkuBeaLNlAExNW6CcIiaQda0Keubkb15J56w+WqDG/OoUNaQTEz1V7R/++KpJN3LhvG0QDugg5GYOdlej73juZiLcJ9MaYaj2nGZSQIAZgkMODGtpg5hP6U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=bCgRYetp; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="bCgRYetp" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 94D3CC4CEFC; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196506; bh=mOGPpjFInnJF5ficZgf3c7TZFdWiehGB3ZFVn3w7fEg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bCgRYetpxQj5ircu7SffS0RZu478L9Xl2HMnqnncjBLuvUJP0o70y8UTz91AT8B/h cJwxPoDhFx0KielP/ZfAK43Xm+JcjmM03kg1Wp0y1wlY1dXDhCLsbIZtULXX1OLelf tcbphH353m6dAyO1CIPgZTZRhU8i5YNCG8EYOvA3plwjyhb5nGoNpkPy2I+AnFfAos PTMhuej/Qa5ufiIhKR+3UMirHtb0NNO1uVTUADLYJ4ZB9Mqe4sMOZQ8PFrr9Mi4uhv rzdetRo0hoECqYTk6HINBn0MzVC6VFIl7tV4z/CnW+qLJkDU5B6jIV7n5yPWhbShPR dLWyAZpNFCuig== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE8-0000000CrsT-0dGB; Thu, 18 Sep 2025 13:55:04 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v8 14/24] docs: Makefile: document FONTS_CONF_DENY_VF= parameter Date: Thu, 18 Sep 2025 13:54:48 +0200 Message-ID: X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" This parameter is there for some time, but it doesn't have anything documenting it at make help. Add some documentation, pointing to the place where one can find more details. Signed-off-by: Mauro Carvalho Chehab --- Documentation/Makefile | 3 +++ 1 file changed, 3 insertions(+) diff --git a/Documentation/Makefile b/Documentation/Makefile index a52b311cdadc..05397b9d844d 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -126,4 +126,7 @@ dochelp: @echo @echo ' make PAPER=3D{a4|letter} Specifies the paper size used for LaTe= X/PDF output.' @echo + @echo ' make FONTS_CONF_DENY_VF=3D{path} sets a deny list to block vari= able Noto CJK fonts' + @echo ' for PDF build. See tools/docs/lib/latex_fonts.py for more detai= ls' + @echo @echo ' Default location for the generated documents is Documentation/o= utput' --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E1A8C304BAB; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; cv=none; b=liyForztzLwcFX6ucemt2IzqhSScp0vLDQGTEhZBn+DJNlwoz8Wj6RV5vxiA22Mv2AkVXwAQm1ed8y3DMlUWSMMdVYFl7oCMwy9HYaFW5nBIPAx0/tSAq9CBjOjn+DlUYUoiICwDbIupV7gunJ0ZtpSHbufTWvL87zzIuK4PlV0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; c=relaxed/simple; bh=PUUaTyw8rlc0Hc2z+x0BdPRC+RKe/Sri73mjZjht+w8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EtzGX3q2XN17sS0wfiZNKJKsp9FnmA6OLnOLFrnPMAhPolliajEockJyqfQotXhMG/r4wiRUffJ9dkvE+74YhTEFi6GlKZjQJ8MaMn8DjL/ooOOXx/rrtp6hmUbyZNV1i4gAYkhlygyY5Kp9xYxqkMMjO2IjH/cvLvIF63yXFuM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=TciCRFNb; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="TciCRFNb" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C256EC4AF0B; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196506; bh=PUUaTyw8rlc0Hc2z+x0BdPRC+RKe/Sri73mjZjht+w8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TciCRFNbAjmoTG9R/zfKSvON4u6mU+PjQ5bAY+3QiWkmXyQ7Jx1YSfcFWJC7AGYAM M3uRFn09WzdMIIiiuIFh/5lZBM2QoejnUgSJUnmFZoHbTZKDfJO0loYykzw0TijhKr AQMWIpz2RpTH9hXGTaJLcfv5MqBXYHPfXRzL6S25eCr606ZNrvERn742fFc24OITMX oamxYIt63y6nK131McDFMYtDNxYxIMIRijLTr7Uq3Ro429ULW9vYwXkDzLToJGOS19 CWogoEvteuGJNQQ2C2GFMVmoCPHbALw6fCzfVQPLze4wNa8ZhByKNRlJ4pbJGZBVcV t5yd4E2prtIBw== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE8-0000000Crt5-0ma2; Thu, 18 Sep 2025 13:55:04 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v8 15/24] tools/docs: sphinx-build-wrapper: add an argument for LaTeX interactive mode Date: Thu, 18 Sep 2025 13:54:49 +0200 Message-ID: <9e5b9a8becc981b47ca3bf3ddce034f273400738.1758196090.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" By default, we use LaTeX batch mode to build docs. This way, when an error happens, the build fails. This is good for normal builds, but when debugging problems with pdf generation, the best is to use interactive mode. We already support it via LATEXOPTS, but having a command line argument makes it easier: Interactive mode: ./scripts/sphinx-build-wrapper pdfdocs --sphinxdirs peci -v -i ... Running 'xelatex --no-pdf -no-pdf -recorder ".../Documentation/output/pe= ci/latex/peci.tex"' ... Default batch mode: ./scripts/sphinx-build-wrapper pdfdocs --sphinxdirs peci -v ... Running 'xelatex --no-pdf -no-pdf -interaction=3Dbatchmode -no-shell-esca= pe -recorder ".../Documentation/output/peci/latex/peci.tex"' ... Signed-off-by: Mauro Carvalho Chehab --- tools/docs/sphinx-build-wrapper | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrap= per index c57c732b879c..50761f125bc9 100755 --- a/tools/docs/sphinx-build-wrapper +++ b/tools/docs/sphinx-build-wrapper @@ -173,7 +173,7 @@ class SphinxBuilder: if not verbose: self.sphinxopts +=3D ["-q"] =20 - def __init__(self, builddir, verbose=3DFalse, n_jobs=3DNone): + def __init__(self, builddir, verbose=3DFalse, n_jobs=3DNone, interacti= ve=3DNone): """Initialize internal variables""" self.verbose =3D None =20 @@ -183,7 +183,11 @@ class SphinxBuilder: self.kernelversion =3D os.environ.get("KERNELVERSION", "unknown") self.kernelrelease =3D os.environ.get("KERNELRELEASE", "unknown") self.pdflatex =3D os.environ.get("PDFLATEX", "xelatex") - self.latexopts =3D os.environ.get("LATEXOPTS", "-interaction=3Dbat= chmode -no-shell-escape") + + if not interactive: + self.latexopts =3D os.environ.get("LATEXOPTS", "-interaction= =3Dbatchmode -no-shell-escape") + else: + self.latexopts =3D os.environ.get("LATEXOPTS", "") =20 if not verbose: verbose =3D bool(os.environ.get("KBUILD_VERBOSE", "") !=3D "") @@ -584,12 +588,16 @@ def main(): parser.add_argument('-j', '--jobs', type=3Djobs_type, help=3D"Sets number of jobs to use with sphinx-bui= ld") =20 + parser.add_argument('-i', '--interactive', action=3D'store_true', + help=3D"Change latex default to run in interactive= mode") + args =3D parser.parse_args() =20 PythonVersion.check_python(MIN_PYTHON_VERSION) =20 builder =3D SphinxBuilder(builddir=3Dargs.builddir, - verbose=3Dargs.verbose, n_jobs=3Dargs.jobs) + verbose=3Dargs.verbose, n_jobs=3Dargs.jobs, + interactive=3Dargs.interactive) =20 builder.build(args.target, sphinxdirs=3Dargs.sphinxdirs, conf=3Dargs.c= onf, theme=3Dargs.theme, css=3Dargs.css, paper=3Dargs.paper, --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 16F063054F2; Thu, 18 Sep 2025 11:55:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; cv=none; b=IimwAlycsc05dfsya3rlCsj1sq4ebTAYS+woUMRl7enMTJgfc+XjHZXdWpgyHgGSJxRE/1gwrQ+6G1GxB4RLI26wM8o5VG2UZU/LKpVGmwzrSy1Isu9vK6a97JsCiVOF7FJJNSChqROUFBkfK5o6EAs1vNmJFpQoo5SCVQgZu5g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; c=relaxed/simple; bh=Pbt/uH3/UgM6ew/nk/FSpaBLbop272L8YPcfdZN28/U=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=d5fldqk/zAiXcHp8smtcnuG1U4KLU+UDV0pcDmztRAuShNrQwPm4zmKlfW6hRd53wAZkHw33dmWLA4G2/YagW4VRyrEyFctpZHcuPDsW7cuG/lQ1w0MGg++LuxwJ/4aluTWL/J3I+TbSB4bMUAzT6hLJYsE4cr77+cWW+iM5cfc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=VCN/EBrQ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="VCN/EBrQ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C8733C4CEFA; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196507; bh=Pbt/uH3/UgM6ew/nk/FSpaBLbop272L8YPcfdZN28/U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VCN/EBrQDn6/Kf6UB8bd2F88BB2hhstUfQuGwcRNSMtPDIs6hI2m6i1bzLPrS3JRZ xYyU/9vriz7ihSBkxHm51NeUchj6EjfBz2rFtUTYI5G3EwFUYubwz+9dfqqax+97Gn WF15ORnSBY5aXUn34YN0agU1CWB5CibPLpAydbioTLDsZRUIvYx9aHbQ7pVr9fKScM 3OZXSpTd3Z3PAKPWbLzEo57kPytlB5ElZVrF6p3A8Miuev2Sb7BZ/yRfhSP3ixhUFl MkC8gZg6sfMKAMO3b4l9JJhKtFY+rkZXIc4jgMBfM8KNWeHPHkt2qLUdBKH7CgYQzl aPLeqFJ8tGQzA== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE8-0000000CrtA-0vo7; Thu, 18 Sep 2025 13:55:04 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v8 16/24] tools/docs: sphinx-build-wrapper: allow building PDF files in parallel Date: Thu, 18 Sep 2025 13:54:50 +0200 Message-ID: <42eef319f9af6f9feb12bcd74ca6392c8119929d.1758196090.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Use POSIX jobserver when available or -j to run PDF builds in parallel, restoring pdf build performance. Yet, running it when debugging troubles is a bad idea, so, when calling directly via command line, except if "-j" is splicitly requested, it will serialize the build. With such change, a PDF doc builds now takes around 5 minutes on a Ryzen 9 machine with 32 cpu threads: # Explicitly paralelize both Sphinx and LaTeX pdf builds $ make cleandocs; time scripts/sphinx-build-wrapper pdfdocs -j 33 real 5m17.901s user 15m1.499s sys 2m31.482s # Use POSIX jobserver to paralelize both sphinx-build and LaTeX $ make cleandocs; time make pdfdocs real 5m22.369s user 15m9.076s sys 2m31.419s # Serializes PDF build, while keeping Sphinx parallelized. # it is equivalent of passing -jauto via command line $ make cleandocs; time scripts/sphinx-build-wrapper pdfdocs real 11m20.901s user 13m2.910s sys 1m44.553s Signed-off-by: Mauro Carvalho Chehab --- tools/docs/sphinx-build-wrapper | 218 +++++++++++++++++++++++--------- 1 file changed, 156 insertions(+), 62 deletions(-) diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrap= per index 50761f125bc9..0fe21b343e7a 100755 --- a/tools/docs/sphinx-build-wrapper +++ b/tools/docs/sphinx-build-wrapper @@ -51,6 +51,8 @@ import shutil import subprocess import sys =20 +from concurrent import futures + from lib.python_version import PythonVersion from lib.latex_fonts import LatexFontChecker =20 @@ -291,6 +293,87 @@ class SphinxBuilder: except (OSError, IOError) as e: print(f"Warning: Failed to copy CSS: {e}", file=3Dsys.stderr) =20 + def build_pdf_file(self, latex_cmd, from_dir, path): + """Builds a single pdf file using latex_cmd""" + try: + subprocess.run(latex_cmd + [path], + cwd=3Dfrom_dir, check=3DTrue, env=3Dself.env) + + return True + except subprocess.CalledProcessError: + return False + + def pdf_parallel_build(self, tex_suffix, latex_cmd, tex_files, n_jobs): + """Build PDF files in parallel if possible""" + builds =3D {} + build_failed =3D False + max_len =3D 0 + has_tex =3D False + + # + # LaTeX PDF error code is almost useless for us: + # any warning makes it non-zero. For kernel doc builds it always r= eturn + # non-zero even when build succeeds. So, let's do the best next th= ing: + # Ignore build errors. At the end, check if all PDF files were bui= lt, + # printing a summary with the built ones and returning 0 if all of + # them were actually built. + # + with futures.ThreadPoolExecutor(max_workers=3Dn_jobs) as executor: + jobs =3D {} + + for from_dir, pdf_dir, entry in tex_files: + name =3D entry.name + + if not name.endswith(tex_suffix): + continue + + name =3D name[:-len(tex_suffix)] + + max_len =3D max(max_len, len(name)) + + has_tex =3D True + + future =3D executor.submit(self.build_pdf_file, latex_cmd, + from_dir, entry.path) + jobs[future] =3D (from_dir, pdf_dir, name) + + for future in futures.as_completed(jobs): + from_dir, pdf_dir, name =3D jobs[future] + + pdf_name =3D name + ".pdf" + pdf_from =3D os.path.join(from_dir, pdf_name) + + try: + success =3D future.result() + + if success and os.path.exists(pdf_from): + pdf_to =3D os.path.join(pdf_dir, pdf_name) + + os.rename(pdf_from, pdf_to) + + # + # if verbose, get the name of built PDF file + # + if self.verbose: + builds[name] =3D os.path.relpath(pdf_to, self.b= uilddir) + else: + builds[name] =3D "FAILED" + build_failed =3D True + except futures.Error as e: + builds[name] =3D f"FAILED ({repr(e)})" + build_failed =3D True + + # + # Handle case where no .tex files were found + # + if not has_tex: + name =3D "Sphinx LaTeX builder" + max_len =3D max(max_len, len(name)) + builds[name] =3D "FAILED (no .tex file was generated)" + build_failed =3D True + + return builds, build_failed, max_len + def handle_pdf(self, output_dirs, deny_vf): """ Extra steps for PDF output. @@ -301,6 +384,8 @@ class SphinxBuilder: """ builds =3D {} max_len =3D 0 + tex_suffix =3D ".tex" + tex_files =3D [] =20 # # Since early 2024, Fedora and openSUSE tumbleweed have started @@ -327,74 +412,83 @@ class SphinxBuilder: =20 latex_cmd.extend(shlex.split(self.latexopts)) =20 - tex_suffix =3D ".tex" - - # - # Process each .tex file - # - - has_tex =3D False - build_failed =3D False + # Get a list of tex files to process with os.scandir(from_dir) as it: for entry in it: - if not entry.name.endswith(tex_suffix): - continue - - name =3D entry.name[:-len(tex_suffix)] - has_tex =3D True - - # - # LaTeX PDF error code is almost useless for us: - # any warning makes it non-zero. For kernel doc builds= it - # always return non-zero even when build succeeds. - # So, let's do the best next thing: check if all PDF - # files were built. If they're, print a summary and - # return 0 at the end of this function - # - try: - subprocess.run(latex_cmd + [entry.path], - cwd=3Dfrom_dir, check=3DTrue, env= =3Dself.env) - except subprocess.CalledProcessError: - pass - - pdf_name =3D name + ".pdf" - pdf_from =3D os.path.join(from_dir, pdf_name) - pdf_to =3D os.path.join(pdf_dir, pdf_name) - - if os.path.exists(pdf_from): - os.rename(pdf_from, pdf_to) - builds[name] =3D os.path.relpath(pdf_to, self.buil= ddir) - else: - builds[name] =3D "FAILED" - build_failed =3D True - - name =3D entry.name.removesuffix(".tex") - max_len =3D max(max_len, len(name)) - - if not has_tex: - name =3D os.path.basename(from_dir) - max_len =3D max(max_len, len(name)) - builds[name] =3D "FAILED (no .tex)" - build_failed =3D True - - msg =3D "Summary" - msg +=3D "\n" + "=3D" * len(msg) - print() - print(msg) - - for pdf_name, pdf_file in builds.items(): - print(f"{pdf_name:<{max_len}}: {pdf_file}") - - print() - - if build_failed: + if entry.name.endswith(tex_suffix): + tex_files.append((from_dir, pdf_dir, entry)) + + # + # When using make, this won't be used, as the number of jobs comes + # from POSIX jobserver. So, this covers the case where build comes + # from command line. On such case, serialize by default, except if + # the user explicitly sets the number of jobs. + # + n_jobs =3D 1 + + # n_jobs is either an integer or "auto". Only use it if it is a nu= mber + if self.n_jobs: + try: + n_jobs =3D int(self.n_jobs) + except ValueError: + pass + + # + # When using make, jobserver.claim is the number of jobs that were + # used with "-j" and that aren't used by other make targets + # + with JobserverExec() as jobserver: + n_jobs =3D 1 + + # + # Handle the case when a parameter is passed via command line, + # using it as default, if jobserver doesn't claim anything + # + if self.n_jobs: + try: + n_jobs =3D int(self.n_jobs) + except ValueError: + pass + + if jobserver.claim: + n_jobs =3D jobserver.claim + + builds, build_failed, max_len =3D self.pdf_parallel_build(tex_= suffix, + latex_= cmd, + tex_fi= les, + n_jobs) + + # + # In verbose mode, print a summary with the build results per file. + # Otherwise, print a single line with all failures, if any. + # On both cases, return code 1 indicates build failures, + # + if self.verbose: + msg =3D "Summary" + msg +=3D "\n" + "=3D" * len(msg) + print() + print(msg) + + for pdf_name, pdf_file in builds.items(): + print(f"{pdf_name:<{max_len}}: {pdf_file}") + + print() + if build_failed: + msg =3D LatexFontChecker().check() + if msg: + print(msg) + + sys.exit("Error: not all PDF files were created.") + + elif build_failed: + n_failures =3D len(builds) + failures =3D ", ".join(builds.keys()) + msg =3D LatexFontChecker().check() if msg: print(msg) =20 - sys.exit("PDF build failed: not all PDF files were created.") - else: - print("All PDF files were built.") + sys.exit(f"Error: Can't build {n_failures} PDF file(s): {failu= res}") =20 def handle_info(self, output_dirs): """ --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F370B30507F; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; cv=none; b=R37Jw4JoHSkP9eKLkB/1I+7fOEiRDAJsnMmvePn0NXoMD+fcIA48BDtfkydmkqoi7+Ri9X7cv7bqL/UiNaubbzNZqEK7sVbt/MFYFQyzPtnkr+kmvwex5PRSDZT3INvxsq5s8uO0plx2M9tBqgJM9Q3l63ni4BnGqyC7W+B5S6g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; c=relaxed/simple; bh=WDtXWJRYNGux0wtm1BAyWaADemyId01OLfJz/YWYDuE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EP439z9DqAmlTBwrS6oZuRIG8UYZN2f6wAqM2HGeYXyfOLMs3cxqg8pj2wHIubuEg5EuzP+aA2Qx9D1fS6svk4FwHQP15eOp98Ur3IAvp+cAWMQqEJm7ZHjZH/9i/NqC47BlriaQEM7aLdAj/BGW4/6ltPrNdcy2x3d7ehkT71c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=i9HwUcC8; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="i9HwUcC8" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D487AC4AF0C; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196506; bh=WDtXWJRYNGux0wtm1BAyWaADemyId01OLfJz/YWYDuE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=i9HwUcC8S4myQ3uK5hVbP+BgkoxMBw9lONzsssuAWnwkXKxLsvpueRpRMuRzKyj5B 9pefEFMHKh8DjCbTt+vCCrpzTHZo93TBsUkkMrnPdCYJpnLwjoIgKUziMbwv7npKgf KeFOAy9e5b2FUoyVv11fFIcyE10oWoY1d0iFqRALQw4ljgz58tCTdIlkd2cHIle/Uo yO3N5OfE8wI1GWwDZSF1YmaHW/wLg39X7lwZEW6RdaN8ST/VR6cGJTjRYgmAZWpm0o 4jWBWf62N75xYcJRb/GNMssyChY0K4JtwrhrSI6hdvkj6BMbl3ajV5kFc9x8wCtXMw eRJZjN9vz3Tsg== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE8-0000000CrtF-1521; Thu, 18 Sep 2025 13:55:04 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v8 17/24] tools/docs,scripts: sphinx-*: prevent sphinx-build crashes Date: Thu, 18 Sep 2025 13:54:51 +0200 Message-ID: <1d0afad8fe3d83182be3a08eb00dd71322e23e69.1758196090.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" On a properly set system, LANG and LC_ALL is always defined. However, some distros like Debian, Gentoo and their variants start with those undefioned. When Sphinx tries to set a locale with: locale.setlocale(locale.LC_ALL, '') It raises an exception, making Sphinx fail. This is more likely to happen with test containers. Add a logic to detect and workaround such issue by setting locale to C. Signed-off-by: Mauro Carvalho Chehab --- tools/docs/sphinx-build-wrapper | 11 +++++++++++ tools/docs/sphinx-pre-install | 14 +++++++++++++- 2 files changed, 24 insertions(+), 1 deletion(-) diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrap= per index 0fe21b343e7a..6c2580303e8e 100755 --- a/tools/docs/sphinx-build-wrapper +++ b/tools/docs/sphinx-build-wrapper @@ -45,6 +45,7 @@ the newer version. """ =20 import argparse +import locale import os import shlex import shutil @@ -565,6 +566,16 @@ class SphinxBuilder: if not sphinxdirs: sphinxdirs =3D os.environ.get("SPHINXDIRS", ".") =20 + # + # The sphinx-build tool has a bug: internally, it tries to set + # locale with locale.setlocale(locale.LC_ALL, ''). This causes a + # crash if language is not set. Detect and fix it. + # + try: + locale.setlocale(locale.LC_ALL, '') + except locale.Error: + self.env["LC_ALL"] =3D "C" + # # sphinxdirs can be a list or a whitespace-separated string # diff --git a/tools/docs/sphinx-pre-install b/tools/docs/sphinx-pre-install index d6d673b7945c..663d4e2a3f57 100755 --- a/tools/docs/sphinx-pre-install +++ b/tools/docs/sphinx-pre-install @@ -26,6 +26,7 @@ system pacage install is recommended. """ =20 import argparse +import locale import os import re import subprocess @@ -422,8 +423,19 @@ class MissingCheckers(AncillaryMethods): """ Gets sphinx-build version. """ + env =3D os.environ.copy() + + # The sphinx-build tool has a bug: internally, it tries to set + # locale with locale.setlocale(locale.LC_ALL, ''). This causes a + # crash if language is not set. Detect and fix it. try: - result =3D self.run([cmd, "--version"], + locale.setlocale(locale.LC_ALL, '') + except Exception: + env["LC_ALL"] =3D "C" + env["LANG"] =3D "C" + + try: + result =3D self.run([cmd, "--version"], env=3Denv, stdout=3Dsubprocess.PIPE, stderr=3Dsubprocess.STDOUT, text=3DTrue, check=3DTrue) --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0477B3054C7; Thu, 18 Sep 2025 11:55:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; cv=none; b=gPQE1K2lwrf1PHCZNPq+G2luJfLUwRn0TjU1w6VyEaCogcOKWiiKjq3u0DdqGoCi6vSHbRRUdQEvdwvXHH0sKEP9S4EMmDmqGosD+kfGqoIB9UWyQ2SV6R5Ly/gQG7pQtJcRm0y9jkLeGQj3+AXcviDgP0/5vK3mW12tnFsxY7g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; c=relaxed/simple; bh=k6YcbtXwsJKesw6XBa4evKMXBMHtuUFMe++T/iQChqE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DDXens9UZGpaTNuz4lksIjNUIdvHstykHp40ORXlknDW7Scz7doyOuAm5Ss1O+jC4KovIuNvEyiRdDGWemsJ4t8j0SgkM6L7n1I538DqmbsWPjQzbF+YCTw5Bo4kq5kyqzjnJcaU6On3+5eyf9eL1GafW8fThcr+o9m5vB9ggZ0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=LVv/HXV5; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="LVv/HXV5" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DBACBC4CEE7; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196506; bh=k6YcbtXwsJKesw6XBa4evKMXBMHtuUFMe++T/iQChqE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LVv/HXV5lJeEQx/4YKkpJbhFEJ/P8kAUlZmUu6jB2i4zMFVJ5pNBMVy5KIYuQf5ML EupaPCu1Jq66TubtXjKDB0zYlO8HDgICRXeO6prILWlvG+T7mVw67M+jgMwQAwg5lx fPgeiSX/bnEJdBwA6G6tC2ubAnTXV61beMv+8UTJuDx+EgRfcqRmWeANhdbvkAW8La +45ns09YOnrjImSUyXt/8yzaRN2mFHSVyjT8sILAeXXDYhOSug8dCwcezEGThlehpy 9iVR7YzMSDRchRoRNFgT/Pv0KUqwJ/JQOWMIyRMknmEiCwizodp0tjyPtWWk4KnBvh 0H28n//IsHIFQ== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE8-0000000CrtK-1EIb; Thu, 18 Sep 2025 13:55:04 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v8 18/24] tools/docs: sphinx-build-wrapper: Fix output for duplicated names Date: Thu, 18 Sep 2025 13:54:52 +0200 Message-ID: X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" When SPHINXDIRS is used, basename may be identical for different files. If this happens, the summary and error detection won't be accurate. Fix it by using relative names from builddir. While here, don't duplicate names. Report, instead: - SUCCESS output PDF file was built - FAILED latexmk/xelatex didn't build any PDF output - FAILED: no .tex files were generated Sphinx didn't build any tex file for SPHINXDIRS directories - FAILED ({python exception}) When a concurrent.futures is catched. Usually indicates an internal error at the build logic. With that, building multiple dirs with the same name is reported properly: $ make V=3D1 SPHINXDIRS=3D"admin-guide/media driver-api/media userspace= -api/media" pdfdocs Summary =3D=3D=3D=3D=3D=3D=3D admin-guide/media/pdf/media.pdf : SUCCESS driver-api/media/pdf/media.pdf : SUCCESS userspace-api/media/pdf/media.pdf: SUCCESS And if at least one of them fails, return code will be 1. Signed-off-by: Mauro Carvalho Chehab --- tools/docs/sphinx-build-wrapper | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrap= per index 6c2580303e8e..8d1f77c4a880 100755 --- a/tools/docs/sphinx-build-wrapper +++ b/tools/docs/sphinx-build-wrapper @@ -329,9 +329,6 @@ class SphinxBuilder: continue =20 name =3D name[:-len(tex_suffix)] - - max_len =3D max(max_len, len(name)) - has_tex =3D True =20 future =3D executor.submit(self.build_pdf_file, latex_cmd, @@ -343,34 +340,35 @@ class SphinxBuilder: =20 pdf_name =3D name + ".pdf" pdf_from =3D os.path.join(from_dir, pdf_name) + pdf_to =3D os.path.join(pdf_dir, pdf_name) + out_name =3D os.path.relpath(pdf_to, self.builddir) + max_len =3D max(max_len, len(out_name)) =20 try: success =3D future.result() =20 if success and os.path.exists(pdf_from): - pdf_to =3D os.path.join(pdf_dir, pdf_name) - os.rename(pdf_from, pdf_to) =20 # # if verbose, get the name of built PDF file # if self.verbose: - builds[name] =3D os.path.relpath(pdf_to, self.b= uilddir) + builds[out_name] =3D "SUCCESS" else: - builds[name] =3D "FAILED" + builds[out_name] =3D "FAILED" build_failed =3D True except futures.Error as e: - builds[name] =3D f"FAILED ({repr(e)})" + builds[out_name] =3D f"FAILED ({repr(e)})" build_failed =3D True =20 # # Handle case where no .tex files were found # if not has_tex: - name =3D "Sphinx LaTeX builder" - max_len =3D max(max_len, len(name)) - builds[name] =3D "FAILED (no .tex file was generated)" + out_name =3D "LaTeX files" + max_len =3D max(max_len, len(out_name)) + builds[out_name] =3D "FAILED: no .tex files were generated" build_failed =3D True =20 return builds, build_failed, max_len --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2AF54305955; Thu, 18 Sep 2025 11:55:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; cv=none; b=tvhCcZktByqP3XV2tepaM3Dr5BxFO/uWsqIQybUq/1+b3D6z6CSIVO2w8qZKm0VVZPVJERexRLFIwJWhRd4tV7hoj14OWGLgdDzHp924Fhd2fJ9yA+XcpcuuS+cWT9k3mXbqFX/uM0LON1rcgMbI4qFYpJXbPcgyGZHxBoOjFtA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; c=relaxed/simple; bh=KDZQ/WnN5vw7Bl6a4chb+TcehefEfRpozxecUjS1tVU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=i5kVBjgG3ZAb19KLWIPcLB8RaFDw5yJoscvTI6p2uW/GiekAxhGgRxz6vlbX0Ym9ZDrKoTdVS/LIxupbIGhsen9UNlnJPkpBGSeJa+yxhxD/lr3/tSsz0PA+fQgjnS8sp0Ouyn1fWDJBWq+jgwmn+ZkPEAYOMe7mFPbtQf6+/Cw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=UIoAbqlu; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="UIoAbqlu" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DD548C113D0; Thu, 18 Sep 2025 11:55:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196507; bh=KDZQ/WnN5vw7Bl6a4chb+TcehefEfRpozxecUjS1tVU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UIoAbqlur3zL/fLLqxQ568kKmQRSMV0renDu3v0Q/HE2qYKFVPWeus2WrH6ui7qR7 5A5OKoMvYVoj0Bn8skXKMS2fJcj6n+PSQwn+LuZoQLEGDkJjr4hFP3SqflD8lR8u6x ycBgA/L81zlMYNPq08gthrsbCx0xy3w0Tb6vSUFyoZh4m+C7tVnuYiNkyuf5MukhO7 veKohplinc9hRH53of3B6zQg4uBimuriwmKq6gkNAPgqdtUHWu32xSytafZmQuActV sf4Itd8T+HyCF2Pr2cm/BN8C0BRqlkpkMlTE8zHOKpDm71WPzfi3NtaQ18tF5JXISn YHN9ZAer0eimw== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE8-0000000CrtP-1Nkx; Thu, 18 Sep 2025 13:55:04 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , =?UTF-8?q?Thomas=20Wei=C3=9Fschuh?= , Alice Ryhl , Masahiro Yamada , Miguel Ojeda , Nathan Chancellor , Nicolas Schier , Randy Dunlap , Tamir Duberstein , linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v8 19/24] docs: add support to build manpages from kerneldoc output Date: Thu, 18 Sep 2025 13:54:53 +0200 Message-ID: <3d248d724e7f3154f6e3a227e5923d7360201de9.1758196090.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Generating man files currently requires running a separate script. The target also doesn't appear at the docs Makefile. Add support for mandocs at the Makefile, adding the build logic inside sphinx-build-wrapper, updating documentation and dropping the ancillary script. Signed-off-by: Mauro Carvalho Chehab Acked-by: Nicolas Schier --- Documentation/Makefile | 3 +- Documentation/doc-guide/kernel-doc.rst | 29 ++++----- Makefile | 2 +- scripts/split-man.pl | 28 --------- tools/docs/sphinx-build-wrapper | 81 ++++++++++++++++++++++++-- 5 files changed, 95 insertions(+), 48 deletions(-) delete mode 100755 scripts/split-man.pl diff --git a/Documentation/Makefile b/Documentation/Makefile index 05397b9d844d..aa42b2cb7030 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -53,7 +53,7 @@ ifeq ($(HAVE_SPHINX),0) else # HAVE_SPHINX =20 # Common documentation targets -infodocs texinfodocs latexdocs epubdocs xmldocs pdfdocs linkcheckdocs: +mandocs infodocs texinfodocs latexdocs epubdocs xmldocs pdfdocs linkcheckd= ocs: $(Q)@$(srctree)/tools/docs/sphinx-pre-install --version-check +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \ --sphinxdirs=3D"$(SPHINXDIRS)" --conf=3D"$(SPHINX_CONF)" \ @@ -104,6 +104,7 @@ dochelp: @echo ' htmldocs - HTML' @echo ' texinfodocs - Texinfo' @echo ' infodocs - Info' + @echo ' mandocs - Man pages' @echo ' latexdocs - LaTeX' @echo ' pdfdocs - PDF' @echo ' epubdocs - EPUB' diff --git a/Documentation/doc-guide/kernel-doc.rst b/Documentation/doc-gui= de/kernel-doc.rst index af9697e60165..4370cc8fbcf5 100644 --- a/Documentation/doc-guide/kernel-doc.rst +++ b/Documentation/doc-guide/kernel-doc.rst @@ -579,20 +579,23 @@ source. How to use kernel-doc to generate man pages ------------------------------------------- =20 -If you just want to use kernel-doc to generate man pages you can do this -from the kernel git tree:: +To generate man pages for all files that contain kernel-doc markups, run:: =20 - $ scripts/kernel-doc -man \ - $(git grep -l '/\*\*' -- :^Documentation :^tools) \ - | scripts/split-man.pl /tmp/man + $ make mandocs =20 -Some older versions of git do not support some of the variants of syntax f= or -path exclusion. One of the following commands may work for those versions= :: +Or calling ``script-build-wrapper`` directly:: =20 - $ scripts/kernel-doc -man \ - $(git grep -l '/\*\*' -- . ':!Documentation' ':!tools') \ - | scripts/split-man.pl /tmp/man + $ ./tools/docs/sphinx-build-wrapper mandocs =20 - $ scripts/kernel-doc -man \ - $(git grep -l '/\*\*' -- . ":(exclude)Documentation" ":(exclude)tools"= ) \ - | scripts/split-man.pl /tmp/man +The output will be at ``/man`` directory inside the output directory +(by default: ``Documentation/output``). + +Optionally, it is possible to generate a partial set of man pages by +using SPHINXDIRS: + + $ make SPHINXDIRS=3Ddriver-api/media mandocs + +.. note:: + + When SPHINXDIRS=3D{subdir} is used, it will only generate man pages for + the files explicitly inside a ``Documentation/{subdir}/.../*.rst`` file. diff --git a/Makefile b/Makefile index 6bfe776bf3c5..9bd44afeda26 100644 --- a/Makefile +++ b/Makefile @@ -1800,7 +1800,7 @@ $(help-board-dirs): help-%: # Documentation targets # ------------------------------------------------------------------------= --- DOC_TARGETS :=3D xmldocs latexdocs pdfdocs htmldocs epubdocs cleandocs \ - linkcheckdocs dochelp refcheckdocs texinfodocs infodocs + linkcheckdocs dochelp refcheckdocs texinfodocs infodocs mandocs PHONY +=3D $(DOC_TARGETS) $(DOC_TARGETS): $(Q)$(MAKE) $(build)=3DDocumentation $@ diff --git a/scripts/split-man.pl b/scripts/split-man.pl deleted file mode 100755 index 96bd99dc977a..000000000000 --- a/scripts/split-man.pl +++ /dev/null @@ -1,28 +0,0 @@ -#!/usr/bin/env perl -# SPDX-License-Identifier: GPL-2.0 -# -# Author: Mauro Carvalho Chehab -# -# Produce manpages from kernel-doc. -# See Documentation/doc-guide/kernel-doc.rst for instructions - -if ($#ARGV < 0) { - die "where do I put the results?\n"; -} - -mkdir $ARGV[0],0777; -$state =3D 0; -while () { - if (/^\.TH \"[^\"]*\" 9 \"([^\"]*)\"/) { - if ($state =3D=3D 1) { close OUT } - $state =3D 1; - $fn =3D "$ARGV[0]/$1.9"; - print STDERR "Creating $fn\n"; - open OUT, ">$fn" or die "can't open $fn: $!\n"; - print OUT $_; - } elsif ($state !=3D 0) { - print OUT $_; - } -} - -close OUT; diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrap= per index 8d1f77c4a880..7a6eb41837e6 100755 --- a/tools/docs/sphinx-build-wrapper +++ b/tools/docs/sphinx-build-wrapper @@ -47,12 +47,14 @@ the newer version. import argparse import locale import os +import re import shlex import shutil import subprocess import sys =20 from concurrent import futures +from glob import glob =20 from lib.python_version import PythonVersion from lib.latex_fonts import LatexFontChecker @@ -77,6 +79,7 @@ TARGETS =3D { "epubdocs": { "builder": "epub", "out_dir": "epub" }, "texinfodocs": { "builder": "texinfo", "out_dir": "texinfo" }, "infodocs": { "builder": "texinfo", "out_dir": "texinfo" }, + "mandocs": { "builder": "man", "out_dir": "man" }, "latexdocs": { "builder": "latex", "out_dir": "latex" }, "pdfdocs": { "builder": "latex", "out_dir": "latex" }, "xmldocs": { "builder": "xml", "out_dir": "xml" }, @@ -503,6 +506,71 @@ class SphinxBuilder: except subprocess.CalledProcessError as e: sys.exit(f"Error generating info docs: {e}") =20 + def handle_man(self, kerneldoc, docs_dir, src_dir, output_dir): + """ + Create man pages from kernel-doc output + """ + + re_kernel_doc =3D re.compile(r"^\.\.\s+kernel-doc::\s*(\S+)") + re_man =3D re.compile(r'^\.TH "[^"]*" (\d+) "([^"]*)"') + + if docs_dir =3D=3D src_dir: + # + # Pick the entire set of kernel-doc markups from the entire tr= ee + # + kdoc_files =3D set([self.srctree]) + else: + kdoc_files =3D set() + + for fname in glob(os.path.join(src_dir, "**"), recursive=3DTru= e): + if os.path.isfile(fname) and fname.endswith(".rst"): + with open(fname, "r", encoding=3D"utf-8") as in_fp: + data =3D in_fp.read() + + for line in data.split("\n"): + match =3D re_kernel_doc.match(line) + if match: + if os.path.isfile(match.group(1)): + kdoc_files.add(match.group(1)) + + if not kdoc_files: + sys.exit(f"Directory {src_dir} doesn't contain kernel-doc = tags") + + cmd =3D [ kerneldoc, "-m" ] + sorted(kdoc_files) + try: + if self.verbose: + print(" ".join(cmd)) + + result =3D subprocess.run(cmd, stdout=3Dsubprocess.PIPE, text= =3D True) + + if result.returncode: + print(f"Warning: kernel-doc returned {result.returncode} w= arnings") + + except (OSError, ValueError, subprocess.SubprocessError) as e: + sys.exit(f"Failed to create man pages for {src_dir}: {repr(e)}= ") + + fp =3D None + try: + for line in result.stdout.split("\n"): + match =3D re_man.match(line) + if not match: + if fp: + fp.write(line + '\n') + continue + + if fp: + fp.close() + + fname =3D f"{output_dir}/{match.group(2)}.{match.group(1)}" + + if self.verbose: + print(f"Creating {fname}") + fp =3D open(fname, "w", encoding=3D"utf-8") + fp.write(line + '\n') + finally: + if fp: + fp.close() + def cleandocs(self, builder): # pylint: disable=3DW0613 """Remove documentation output directory""" shutil.rmtree(self.builddir, ignore_errors=3DTrue) @@ -531,7 +599,7 @@ class SphinxBuilder: # Other targets require sphinx-build, so check if it exists # sphinxbuild =3D shutil.which(self.sphinxbuild, path=3Dself.env["PA= TH"]) - if not sphinxbuild: + if not sphinxbuild and target !=3D "mandocs": sys.exit(f"Error: {self.sphinxbuild} not found in PATH.\n") =20 if builder =3D=3D "latex": @@ -619,10 +687,13 @@ class SphinxBuilder: output_dir, ] =20 - try: - self.run_sphinx(sphinxbuild, build_args, env=3Dself.env) - except (OSError, ValueError, subprocess.SubprocessError) as e: - sys.exit(f"Build failed: {repr(e)}") + if target =3D=3D "mandocs": + self.handle_man(kerneldoc, docs_dir, src_dir, output_dir) + else: + try: + self.run_sphinx(sphinxbuild, build_args, env=3Dself.en= v) + except (OSError, ValueError, subprocess.SubprocessError) a= s e: + sys.exit(f"Build failed: {repr(e)}") =20 # # Ensure that each html/epub output will have needed static fi= les --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A70A30596E; Thu, 18 Sep 2025 11:55:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; cv=none; b=dtWLwE2AumcfVcxY3eAFsecyQVBvFnVcz7F+xEMaUVnmkPuS94YJEjHHP2DcdoXwoQ+l3kdxC5F2W2TWDPYyzNO/IH7IA2Z4NvdQVMiCM+De8XFR5nLkInty+i3coXLCRP2rcW8y6S/2pTY2C6imVbPwe3EKERLPILelZ29bMck= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; c=relaxed/simple; bh=bNOmw1tzFH8Sl7kR3p+wUYP6U5+a6FHYIcsXQNkIydI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=MZ8TqfOqjihsbOFvZDzDOdGJGX6hJxWiGjAEd7ucuVmsgTLeBjDt9uENEzNQRBVhsYbfNpqOfqNvFYnl4h5XpUy5GaZB/YZAwjYF35BnAxKOWLcl7Yjjg2vd1SiAKFhrvpULSYv088h0WalK61FwIOerVBI5m8hw5mldoMkHtZg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Xh5Kw9Bz; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Xh5Kw9Bz" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 08AB6C4CEEB; Thu, 18 Sep 2025 11:55:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196507; bh=bNOmw1tzFH8Sl7kR3p+wUYP6U5+a6FHYIcsXQNkIydI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Xh5Kw9BzPzpIE/0dFJ/i0tZQOf1qY1IFX6jqsh4+EwCAFk0jnYib7pPSBp5Br1MlP CVrR0iROfUKhBnMUWvatnGJ5RnGhca74WXFllzlDORFeBE+1acKNYEz5OUqYr257JT Owf/tB3QLmKNClke128wkIpjki3Yty5rIBkAKEMwQ4GzSmrfcClAFH/YWi+qjVQVoJ p/0WxcyNdf26YaG6zNaeLSGSwR9mnU25i6dvIkVHh8UQ1sIQ1klBCZXtXlgTD7Q3KW 1rqolE+PPu6Gouydk4aHahT47a9UsvOIzNWFHAbSQXmVQKKhddDwnan0Q+YNM/SGB3 cRdXHYhmNu7Vw== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE8-0000000CrtU-1WcV; Thu, 18 Sep 2025 13:55:04 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v8 20/24] tools: kernel-doc: add a see also section at man pages Date: Thu, 18 Sep 2025 13:54:54 +0200 Message-ID: X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab While cross-references are complex, as related ones can be on different files, we can at least correlate the ones that belong to the same file, adding a SEE ALSO section for them. The result is not bad. See for instance: $ tools/docs/sphinx-build-wrapper --sphinxdirs driver-api/media -- mandocs $ man Documentation/output/driver-api/man/edac_pci_add_device.9 edac_pci_add_device(9) Kernel Hacker's Manual edac_pci_add_device(9) NAME edac_pci_add_device - Insert the 'edac_dev' structure into the edac_pci global list and create sysfs entries associated with edac_pci structure. SYNOPSIS int edac_pci_add_device (struct edac_pci_ctl_info *pci , int edac_idx ); ARGUMENTS pci pointer to the edac_device structure to be added to the list edac_idx A unique numeric identifier to be assigned to the RETURN 0 on Success, or an error code on failure SEE ALSO edac_pci_alloc_ctl_info(9), edac_pci_free_ctl_info(9), edac_pci_alloc_index(9), edac_pci_del_device(9), edac_pci_cre=E2= =80=90 ate_generic_ctl(9), edac_pci_release_generic_ctl(9), edac_pci_create_sysfs(9), edac_pci_remove_sysfs(9) August 2025 edac_pci_add_device edac_pci_add_device(9) Signed-off-by: Mauro Carvalho Chehab --- scripts/lib/kdoc/kdoc_files.py | 5 +- scripts/lib/kdoc/kdoc_output.py | 84 +++++++++++++++++++++++++++++++-- 2 files changed, 83 insertions(+), 6 deletions(-) diff --git a/scripts/lib/kdoc/kdoc_files.py b/scripts/lib/kdoc/kdoc_files.py index 9e09b45b02fa..061c033f32da 100644 --- a/scripts/lib/kdoc/kdoc_files.py +++ b/scripts/lib/kdoc/kdoc_files.py @@ -275,7 +275,10 @@ class KernelFiles(): self.config.log.warning("No kernel-doc for file %s", fname) continue =20 - for arg in self.results[fname]: + symbols =3D self.results[fname] + self.out_style.set_symbols(symbols) + + for arg in symbols: m =3D self.out_msg(fname, arg.name, arg) =20 if m is None: diff --git a/scripts/lib/kdoc/kdoc_output.py b/scripts/lib/kdoc/kdoc_output= .py index ea8914537ba0..1eca9a918558 100644 --- a/scripts/lib/kdoc/kdoc_output.py +++ b/scripts/lib/kdoc/kdoc_output.py @@ -215,6 +215,9 @@ class OutputFormat: =20 # Virtual methods to be overridden by inherited classes # At the base class, those do nothing. + def set_symbols(self, symbols): + """Get a list of all symbols from kernel_doc""" + def out_doc(self, fname, name, args): """Outputs a DOC block""" =20 @@ -577,6 +580,7 @@ class ManFormat(OutputFormat): =20 super().__init__() self.modulename =3D modulename + self.symbols =3D [] =20 dt =3D None tstamp =3D os.environ.get("KBUILD_BUILD_TIMESTAMP") @@ -593,6 +597,68 @@ class ManFormat(OutputFormat): =20 self.man_date =3D dt.strftime("%B %Y") =20 + def arg_name(self, args, name): + """ + Return the name that will be used for the man page. + + As we may have the same name on different namespaces, + prepend the data type for all types except functions and typedefs. + + The doc section is special: it uses the modulename. + """ + + dtype =3D args.type + + if dtype =3D=3D "doc": + return self.modulename + + if dtype in ["function", "typedef"]: + return name + + return f"{dtype} {name}" + + def set_symbols(self, symbols): + """ + Get a list of all symbols from kernel_doc. + + Man pages will uses it to add a SEE ALSO section with other + symbols at the same file. + """ + self.symbols =3D symbols + + def out_tail(self, fname, name, args): + """Adds a tail for all man pages""" + + # SEE ALSO section + if len(self.symbols) >=3D 2: + cur_name =3D self.arg_name(args, name) + + self.data +=3D f'.SH "SEE ALSO"' + "\n.PP\n" + related =3D [] + for arg in self.symbols: + out_name =3D self.arg_name(arg, arg.name) + + if cur_name =3D=3D out_name: + continue + + related.append(f"\\fB{out_name}\\fR(9)") + + self.data +=3D ",\n".join(related) + "\n" + + # TODO: does it make sense to add other sections? Maybe + # REPORTING ISSUES? LICENSE? + + def msg(self, fname, name, args): + """ + Handles a single entry from kernel-doc parser. + + Add a tail at the end of man pages output. + """ + super().msg(fname, name, args) + self.out_tail(fname, name, args) + + return self.data + def output_highlight(self, block): """ Outputs a C symbol that may require being highlighted with @@ -618,7 +684,9 @@ class ManFormat(OutputFormat): if not self.check_doc(name, args): return =20 - self.data +=3D f'.TH "{self.modulename}" 9 "{self.modulename}" "{s= elf.man_date}" "API Manual" LINUX' + "\n" + out_name =3D self.arg_name(args, name) + + self.data +=3D f'.TH "{self.modulename}" 9 "{out_name}" "{self.man= _date}" "API Manual" LINUX' + "\n" =20 for section, text in args.sections.items(): self.data +=3D f'.SH "{section}"' + "\n" @@ -627,7 +695,9 @@ class ManFormat(OutputFormat): def out_function(self, fname, name, args): """output function in man""" =20 - self.data +=3D f'.TH "{name}" 9 "{name}" "{self.man_date}" "Kernel= Hacker\'s Manual" LINUX' + "\n" + out_name =3D self.arg_name(args, name) + + self.data +=3D f'.TH "{name}" 9 "{out_name}" "{self.man_date}" "Ke= rnel Hacker\'s Manual" LINUX' + "\n" =20 self.data +=3D ".SH NAME\n" self.data +=3D f"{name} \\- {args['purpose']}\n" @@ -671,7 +741,9 @@ class ManFormat(OutputFormat): self.output_highlight(text) =20 def out_enum(self, fname, name, args): - self.data +=3D f'.TH "{self.modulename}" 9 "enum {name}" "{self.ma= n_date}" "API Manual" LINUX' + "\n" + out_name =3D self.arg_name(args, name) + + self.data +=3D f'.TH "{self.modulename}" 9 "{out_name}" "{self.man= _date}" "API Manual" LINUX' + "\n" =20 self.data +=3D ".SH NAME\n" self.data +=3D f"enum {name} \\- {args['purpose']}\n" @@ -703,8 +775,9 @@ class ManFormat(OutputFormat): def out_typedef(self, fname, name, args): module =3D self.modulename purpose =3D args.get('purpose') + out_name =3D self.arg_name(args, name) =20 - self.data +=3D f'.TH "{module}" 9 "{name}" "{self.man_date}" "API = Manual" LINUX' + "\n" + self.data +=3D f'.TH "{module}" 9 "{out_name}" "{self.man_date}" "= API Manual" LINUX' + "\n" =20 self.data +=3D ".SH NAME\n" self.data +=3D f"typedef {name} \\- {purpose}\n" @@ -717,8 +790,9 @@ class ManFormat(OutputFormat): module =3D self.modulename purpose =3D args.get('purpose') definition =3D args.get('definition') + out_name =3D self.arg_name(args, name) =20 - self.data +=3D f'.TH "{module}" 9 "{args.type} {name}" "{self.man_= date}" "API Manual" LINUX' + "\n" + self.data +=3D f'.TH "{module}" 9 "{out_name}" "{self.man_date}" "= API Manual" LINUX' + "\n" =20 self.data +=3D ".SH NAME\n" self.data +=3D f"{args.type} {name} \\- {purpose}\n" --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 532E9305E08; Thu, 18 Sep 2025 11:55:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; cv=none; b=KPnDfIYtSVr11BeR8VrGG6JpU6SdG2r7a6KRgh/XRY518YY2Ng9wyZWjWK/dmj55lyUPJr1Vkmd4GioEgCVymUVqg+NFyvBaKiA/BRNkCYDIcHPonGxXaldVgWy9awa8v68aBYKQ88ajXklFc6rQZXb54L0lWF7PBRnrasE+lXM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; c=relaxed/simple; bh=4Yu1gA5/uZ+OPBNEREkXam+9wN35SUbcDazI/2U7I9g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=qahrFpndzZUsQE7vkSUWu+vMCM9lLsNXhKsGbJafq57ZCuGZSu9M9SRLRhJ5rp+jTKC1pDhO6cVxyI/WAXgxYjXwtRCS+rufE/zU7dfCspv6DndRgP4ltSKYHxu8kMAEf8M0/YDP4jTfhEQFcztAY1vzxIzGzL8FCO54P6Hgft0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=quE5BdD1; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="quE5BdD1" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 37A7DC4CEFD; Thu, 18 Sep 2025 11:55:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196507; bh=4Yu1gA5/uZ+OPBNEREkXam+9wN35SUbcDazI/2U7I9g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=quE5BdD1RvB/1PFWlnVLrgB3LtNghauPk5RqZOdlhWs3TfKznk5lR7KK65kso/aKz 7I/jmClNvTT55OtMiKQNVwjrNLrm6FgiaztDkGvL1UtPq33zS4j0idnRm/LpQGMFHa fOYqRlJ6cWs1DLbjbBtVHTpZiERIjtLQN9VbLkTef/7nw36FikiJQYDAVNtQEK2nml jrKib3SQaJ+zc+UkeU6doqis7KEQvPLNs/vLubjq8I/zf+UYEgVY8sgQZ0NiZW0aB7 NYSfIR4WKm/bzXX1QYMhpgeVjGXI9oJTNZ6BC1wzlAAHehSFRbrDFPg1TPLZR5H6yS vWrSc+hZrI2AA== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE8-0000000CrtZ-1fyX; Thu, 18 Sep 2025 13:55:04 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v8 21/24] scripts: kdoc_parser.py: warn about Python version only once Date: Thu, 18 Sep 2025 13:54:55 +0200 Message-ID: <68e54cf8b1201d1f683aad9bc710a99421910356.1758196090.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" When running kernel-doc over multiple documents, it emits one error message per file with is not what we want: $ python3.6 scripts/kernel-doc.py . --none ... Warning: ./include/trace/events/swiotlb.h:0 Python 3.7 or later is require= d for correct results Warning: ./include/trace/events/iommu.h:0 Python 3.7 or later is required = for correct results Warning: ./include/trace/events/sock.h:0 Python 3.7 or later is required f= or correct results ... Change the logic to warn it only once at the library: $ python3.6 scripts/kernel-doc.py . --none Warning: Python 3.7 or later is required for correct results Warning: ./include/cxl/features.h:0 Python 3.7 or later is required for co= rrect results When running from command line, it warns twice, but that sounds ok. Signed-off-by: Mauro Carvalho Chehab --- scripts/lib/kdoc/kdoc_parser.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser= .py index a560546c1867..574972e1f741 100644 --- a/scripts/lib/kdoc/kdoc_parser.py +++ b/scripts/lib/kdoc/kdoc_parser.py @@ -314,6 +314,7 @@ class KernelEntry: self.section =3D SECTION_DEFAULT self._contents =3D [] =20 +python_warning =3D False =20 class KernelDoc: """ @@ -347,9 +348,13 @@ class KernelDoc: # We need Python 3.7 for its "dicts remember the insertion # order" guarantee # - if sys.version_info.major =3D=3D 3 and sys.version_info.minor < 7: + global python_warning + if (not python_warning and + sys.version_info.major =3D=3D 3 and sys.version_info.minor < 7= ): + self.emit_msg(0, 'Python 3.7 or later is required for correct res= ults') + python_warning =3D True =20 def emit_msg(self, ln, msg, warning=3DTrue): """Emit a message""" --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 830F1306484; Thu, 18 Sep 2025 11:55:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; cv=none; b=m5gAg6dThIC9FC80nhGq12d6gMvN65Q/x0R/zunC7nlyZmql9NohNBbnAe+FTA+7dHxYrNFWt3Q+2qXPArBSEz3xz+fzB7s9GvagOqfJELchQ9UXH6rBV4iIWyESQHS9JW+yvCqMx9PacIGPKyYVLNM3MlEfuPfpdHZvPx8yrzs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; c=relaxed/simple; bh=lrdZawkRIwStzU2yfc2G77xAk+2MIGzDv7SZmR0L4QE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=n2un+ISlOrakoJ5jpJxWqTyaq+REYaYRA5YWXoRRXSnXYHJfje8vj3H+KQXXmOa13hihPJ5GUXLvtfu3N5QEjOLIy0ipH+hlNBCc9sXYv0U1paRYfBiZUC0yt2rv8R7iEG2DRHTHx4O3IFQ3mWB8ghZ3zeMu833t9e2/KLk3L34= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=drjKcERv; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="drjKcERv" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3F8D7C4CEFF; Thu, 18 Sep 2025 11:55:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196507; bh=lrdZawkRIwStzU2yfc2G77xAk+2MIGzDv7SZmR0L4QE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=drjKcERv3ds/ll+B1cKCoCPs+B3qKzaHO85s/sTpjkRMOFt4nwNKaG7kVkg8Y5DPx aO0q0ng3EJeg4OleEkrOUgTESugQ5v/ry/Q6XzEHyLfLImzrq5KGVfsx3l1gRlFJQ+ HnDsKtlUvZSDPMPIKRzdEl3xXKQI2rw+fXBF0wEbyBzZyT9Q0z+8noNS2aCaC3vwuy wpnzcAAQp6I4ExAUxaGmgMPBZlTJ/jpFhVuh7Wo0LHpGP/5avvOqxGUZzTI98W0wqy W8y5GixDKD54LDh/Lme9vROqRzN/gql7+1fXIlHzmwpT1+q0mzTYdK2dUntVKJfyl4 TGz6nmZDB54Dg== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE8-0000000Crte-1pvJ; Thu, 18 Sep 2025 13:55:04 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , "Mauro Carvalho Chehab" , Alex Gaynor , Alice Ryhl , Andreas Hindborg , Benno Lossin , Bill Wendling , Boqun Feng , Danilo Krummrich , Gary Guo , Justin Stitt , Miguel Ojeda , Nathan Chancellor , Nick Desaulniers , Trevor Gross , linux-kernel@vger.kernel.org, llvm@lists.linux.dev, rust-for-linux@vger.kernel.org Subject: [PATCH v8 22/24] tools/docs: sphinx-build-wrapper: move rust doc builder to wrapper Date: Thu, 18 Sep 2025 13:54:56 +0200 Message-ID: X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Simplify even further the docs Makefile by moving rust build logic to the wrapper. After this change, running make on an environment with rust enabled works as expected. With CONFIG_RUST: $ make O=3D/tmp/foo LLVM=3D1 SPHINXDIRS=3Dpeci htmldocs make[1]: Entrando no diret=C3=B3rio '/tmp/foo' Using alabaster theme Using Python kernel-doc GEN Makefile DESCEND objtool CC arch/x86/kernel/asm-offsets.s INSTALL libsubcmd_headers CALL /new_devel/docs/scripts/checksyscalls.sh RUSTC L rust/core.o BINDGEN rust/bindings/bindings_generated.rs BINDGEN rust/bindings/bindings_helpers_generated.rs ... Without it: $ make SPHINXDIRS=3Dpeci htmldocs Using alabaster theme Using Python kernel-doc Both work as it is it is supposed to do. After the change, it is also possible to build directly with the script by passing "--rustodoc". if CONFIG_RUST, this works fine: $ ./tools/docs/sphinx-build-wrapper --sphinxdirs peci --rustdoc -- html= docs Using alabaster theme Using Python kernel-doc SYNC include/config/auto.conf ... RUSTC L rust/core.o ... If not, it will produce a warning that RUST may be disabled: $ ./tools/docs/sphinx-build-wrapper --sphinxdirs peci --rustdoc -- html= docs Using alabaster theme Using Python kernel-doc *** *** Configuration file ".config" not found! *** *** Please run some configurator (e.g. "make oldconfig" or *** "make menuconfig" or "make xconfig"). *** make[1]: *** [/new_devel/docs/Makefile:829: .config] Error 1 make: *** [Makefile:248: __sub-make] Error 2 Ignored errors when building rustdoc: Command '['make', 'LLVM=3D1', 'ru= stdoc']' returned non-zero exit status 2.. Is RUST enabled? Signed-off-by: Mauro Carvalho Chehab --- Documentation/Makefile | 29 +++++----------- tools/docs/sphinx-build-wrapper | 59 ++++++++++++++++++--------------- 2 files changed, 41 insertions(+), 47 deletions(-) diff --git a/Documentation/Makefile b/Documentation/Makefile index aa42b2cb7030..b32f68387dfc 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -23,6 +23,7 @@ SPHINXOPTS =3D SPHINXDIRS =3D . DOCS_THEME =3D DOCS_CSS =3D +RUSTDOC =3D SPHINX_CONF =3D conf.py PAPER =3D BUILDDIR =3D $(obj)/output @@ -42,6 +43,10 @@ FONTS_CONF_DENY_VF ?=3D $(HOME)/deny-vf # User-friendly check for sphinx-build HAVE_SPHINX :=3D $(shell if which $(SPHINXBUILD) >/dev/null 2>&1; then ech= o 1; else echo 0; fi) =20 +ifeq ($(CONFIG_RUST),y) + RUSTDOC=3D"--rustdoc" +endif + ifeq ($(HAVE_SPHINX),0) =20 .DEFAULT: @@ -53,10 +58,10 @@ ifeq ($(HAVE_SPHINX),0) else # HAVE_SPHINX =20 # Common documentation targets -mandocs infodocs texinfodocs latexdocs epubdocs xmldocs pdfdocs linkcheckd= ocs: +htmldocs mandocs infodocs texinfodocs latexdocs epubdocs xmldocs pdfdocs l= inkcheckdocs: $(Q)@$(srctree)/tools/docs/sphinx-pre-install --version-check - +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \ - --sphinxdirs=3D"$(SPHINXDIRS)" --conf=3D"$(SPHINX_CONF)" \ + +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ $(RUSTDOC)\ + --sphinxdirs=3D"$(SPHINXDIRS)" --conf=3D"$(SPHINX_CONF)" $(RUSTDOC)\ --builddir=3D"$(BUILDDIR)" --deny-vf=3D$(FONTS_CONF_DENY_VF) \ --theme=3D$(DOCS_THEME) --css=3D$(DOCS_CSS) --paper=3D$(PAPER) =20 @@ -67,24 +72,6 @@ pdfdocs: @echo " SKIP Sphinx $@ target." endif =20 -# HTML main logic is identical to other targets. However, if rust is enabl= ed, -# an extra step at the end is required to generate rustdoc. -htmldocs: - $(Q)@$(srctree)/tools/docs/sphinx-pre-install --version-check - +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \ - --sphinxdirs=3D"$(SPHINXDIRS)" --conf=3D"$(SPHINX_CONF)" \ - --builddir=3D"$(BUILDDIR)" \ - --theme=3D$(DOCS_THEME) --css=3D$(DOCS_CSS) --paper=3D$(PAPER) -# If Rust support is available and .config exists, add rustdoc generated c= ontents. -# If there are any, the errors from this make rustdoc will be displayed but -# won't stop the execution of htmldocs - -ifneq ($(wildcard $(srctree)/.config),) -ifeq ($(CONFIG_RUST),y) - $(Q)$(MAKE) rustdoc || true -endif -endif - endif # HAVE_SPHINX =20 # The following targets are independent of HAVE_SPHINX, and the rules shou= ld diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrap= per index 7a6eb41837e6..e24486dede76 100755 --- a/tools/docs/sphinx-build-wrapper +++ b/tools/docs/sphinx-build-wrapper @@ -96,14 +96,6 @@ class SphinxBuilder: with the Kernel. """ =20 - def is_rust_enabled(self): - """Check if rust is enabled at .config""" - config_path =3D os.path.join(self.srctree, ".config") - if os.path.isfile(config_path): - with open(config_path, "r", encoding=3D"utf-8") as f: - return "CONFIG_RUST=3Dy" in f.read() - return False - def get_path(self, path, use_cwd=3DFalse, abs_path=3DFalse): """ Ancillary routine to handle patches the right way, as shell does. @@ -218,8 +210,6 @@ class SphinxBuilder: "scripts/kernel-doc.= py")) self.builddir =3D self.get_path(builddir, use_cwd=3DTrue, abs_path= =3DTrue) =20 - self.config_rust =3D self.is_rust_enabled() - # # Get directory locations for LaTeX build toolchain # @@ -274,7 +264,7 @@ class SphinxBuilder: =20 return subprocess.call(cmd, *args, **pwargs) =20 - def handle_html(self, css, output_dir): + def handle_html(self, css, output_dir, rustdoc): """ Extra steps for HTML and epub output. =20 @@ -282,20 +272,34 @@ class SphinxBuilder: copied to the output _static directory """ =20 - if not css: - return + if css: + css =3D os.path.expanduser(css) + if not css.startswith("/"): + css =3D os.path.join(self.srctree, css) =20 - css =3D os.path.expanduser(css) - if not css.startswith("/"): - css =3D os.path.join(self.srctree, css) + static_dir =3D os.path.join(output_dir, "_static") + os.makedirs(static_dir, exist_ok=3DTrue) =20 - static_dir =3D os.path.join(output_dir, "_static") - os.makedirs(static_dir, exist_ok=3DTrue) + try: + shutil.copy2(css, static_dir) + except (OSError, IOError) as e: + print(f"Warning: Failed to copy CSS: {e}", file=3Dsys.stde= rr) =20 - try: - shutil.copy2(css, static_dir) - except (OSError, IOError) as e: - print(f"Warning: Failed to copy CSS: {e}", file=3Dsys.stderr) + if rustdoc: + if "MAKE" in self.env: + cmd =3D [self.env["MAKE"]] + else: + cmd =3D ["make", "LLVM=3D1"] + + cmd +=3D [ "rustdoc"] + if self.verbose: + print(" ".join(cmd)) + + try: + subprocess.run(cmd, check=3DTrue) + except subprocess.CalledProcessError as e: + print(f"Ignored errors when building rustdoc: {e}. Is RUST= enabled?", + file=3Dsys.stderr) =20 def build_pdf_file(self, latex_cmd, from_dir, path): """Builds a single pdf file using latex_cmd""" @@ -576,7 +580,7 @@ class SphinxBuilder: shutil.rmtree(self.builddir, ignore_errors=3DTrue) =20 def build(self, target, sphinxdirs=3DNone, conf=3D"conf.py", - theme=3DNone, css=3DNone, paper=3DNone, deny_vf=3DNone): + theme=3DNone, css=3DNone, paper=3DNone, deny_vf=3DNone, rust= doc=3DFalse): """ Build documentation using Sphinx. This is the core function of this module. It prepares all arguments required by sphinx-build. @@ -623,7 +627,7 @@ class SphinxBuilder: =20 args.extend(["-D", f"latex_elements.papersize=3D{paper}paper"]) =20 - if self.config_rust: + if rustdoc: args.extend(["-t", "rustdoc"]) =20 if conf: @@ -699,7 +703,7 @@ class SphinxBuilder: # Ensure that each html/epub output will have needed static fi= les # if target in ["htmldocs", "epubdocs"]: - self.handle_html(css, output_dir) + self.handle_html(css, output_dir, rustdoc) =20 # # Step 2: Some targets (PDF and info) require an extra step once @@ -756,6 +760,9 @@ def main(): parser.add_argument('--deny-vf', help=3D"Configuration to deny variable fonts on pd= f builds") =20 + parser.add_argument('--rustdoc', action=3D"store_true", + help=3D"Enable rustdoc build. Requires CONFIG_RUST= ") + parser.add_argument("-v", "--verbose", action=3D'store_true', help=3D"place build in verbose mode") =20 @@ -775,7 +782,7 @@ def main(): =20 builder.build(args.target, sphinxdirs=3Dargs.sphinxdirs, conf=3Dargs.c= onf, theme=3Dargs.theme, css=3Dargs.css, paper=3Dargs.paper, - deny_vf=3Dargs.deny_vf) + rustdoc=3Dargs.rustdoc, deny_vf=3Dargs.deny_vf) =20 if __name__ =3D=3D "__main__": main() --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 67886305E1E; Thu, 18 Sep 2025 11:55:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; cv=none; b=T4S6iFlbOZGuViOq1P2KUYObdqBVHOK15DNNXceIbZGWN4i7Tknr7RyU3yONUIOwNqDzTagAGvT/foX6VK+JpHBoXVZwwrj6vPiUI8mPWjrGFDQvJT9pYsV5VUdb/+WfAMpCq1ZV1VATe6i115K7gZ8hqlekLefWxU9oaM9/p7Y= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; c=relaxed/simple; bh=xi6aaKetpX4fDfqLRLgmvyJ++GISVlfrcSwACNlUxeU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Gz035z2BgaxukRKyw2me345zcGU+f3wCMFoNWrlpGsCv4S0b4wFnyhppubQGrxM2XYPPXaI48Ho3YDtck9sN/t00q4EJncoIrhqb3K5n4aaTwY4n6o6xS+N3FwuQdx1Oif0RLrIJkvq5uNTbqJ6xHdSEWqS5R5EyklKuaOcQa8w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=o32wMz4L; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="o32wMz4L" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4A464C4CEFA; Thu, 18 Sep 2025 11:55:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196507; bh=xi6aaKetpX4fDfqLRLgmvyJ++GISVlfrcSwACNlUxeU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=o32wMz4LSITpu8V6PpbPHsch6YkRhm6HnLwziF62iNeOvMDxlXdRX5B5Ms7h6BRHI UfPFZ1nF6+MrxMHkg/ZKbMbuuLZYBYcAiPS9B7aAKcc8iDhG9KIg5rfn+s7gAI2Czb 0jzWnhOOG7CKt67pEUPlH7iSLl8P8I/bumMoWHL612uoSyslaaCfg+DUhZFmGBr1P7 o2fCGtVbLKVn6xiIRmEi65uuIMsjBwt2ltiPEFzYuHTE2uiaaleyEjC0+ZRNeue/MJ xqsKx4y+Ctum13Iy4M7ICBzirYZtRLCdspXjCUP7xdneLmPKvWN08RbpIE60L61ShV o6RKbTJZwpymQ== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE8-0000000Crtj-1zAs; Thu, 18 Sep 2025 13:55:04 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , "Mauro Carvalho Chehab" , Alex Gaynor , Alice Ryhl , Andreas Hindborg , Benno Lossin , Boqun Feng , Danilo Krummrich , Gary Guo , Miguel Ojeda , Trevor Gross , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org Subject: [PATCH v8 23/24] tools/docs: sphinx-* break documentation bulds on openSUSE Date: Thu, 18 Sep 2025 13:54:57 +0200 Message-ID: <0635c311295300e9fb48c0ea607e2408910036e3.1758196090.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Before this patch, building htmldocs on opensuseLEAP works fine: # make htmldocs Available Python versions: /usr/bin/python3.11 Python 3.6.15 not supported. Changing to /usr/bin/python3.11 Python 3.6.15 not supported. Changing to /usr/bin/python3.11 Using alabaster theme Using Python kernel-doc ... As the logic detects that Python 3.6 is too old and recommends intalling python311-Sphinx. If installed, documentation builds work like a charm. Yet, some develpers complained that running python3.11 instead of python3 should not happen. So, let's break the build to make them happier: $ make htmldocs Python 3.6.15 not supported. Bailing out You could run, instead: /usr/bin/python3.11 tools/docs/sphinx-build-wrapper htmldocs \ --sphinxdirs=3D. --conf=3Dconf.py --builddir=3DDocumentation/output= --theme=3D --css=3D \ --paper=3D Python 3.6.15 not supported. Bailing out make[2]: *** [Documentation/Makefile:76: htmldocs] Error 1 make[1]: *** [Makefile:1806: htmldocs] Error 2 make: *** [Makefile:248: __sub-make] Error 2 It should be noticed that: 1. after this change, sphinx-pre-install needs to be called by hand: $ /usr/bin/python3.11 tools/docs/sphinx-pre-install Detected OS: openSUSE Leap 15.6. Sphinx version: 7.2.6 All optional dependencies are met. Needed package dependencies are met. 2. sphinx-build-wrapper will auto-detect python3.11 and suggest a way to build the docs using the parameters passed via make variables. In this specific example: /usr/bin/python3.11 tools/docs/sphinx-build-wrapper htmldocs --sphinxdir= s=3D. --conf=3Dconf.py --theme=3D --css=3D --paper=3D 3. As this needs to be executed outside docs Makefile, it won't run the validation check scripts nor build Rust documentation if enabled, as the extra scripts are part of the docs Makefile. Signed-off-by: Mauro Carvalho Chehab --- tools/docs/lib/python_version.py | 28 ++++++++++++++++++++++++---- tools/docs/sphinx-build-wrapper | 3 ++- tools/docs/sphinx-pre-install | 3 ++- 3 files changed, 28 insertions(+), 6 deletions(-) diff --git a/tools/docs/lib/python_version.py b/tools/docs/lib/python_versi= on.py index a9fda2470a26..4fde1b882164 100644 --- a/tools/docs/lib/python_version.py +++ b/tools/docs/lib/python_version.py @@ -20,9 +20,11 @@ Python version if present. import os import re import subprocess +import shlex import sys =20 from glob import glob +from textwrap import indent =20 class PythonVersion: """ @@ -44,6 +46,25 @@ class PythonVersion: """Returns a version tuple as major.minor.patch""" return ".".join([str(x) for x in version]) =20 + @staticmethod + def cmd_print(cmd, max_len=3D80): + cmd_line =3D [] + + for w in cmd: + w =3D shlex.quote(w) + + if cmd_line: + if not max_len or len(cmd_line[-1]) + len(w) < max_len: + cmd_line[-1] +=3D " " + w + continue + else: + cmd_line[-1] +=3D " \\" + cmd_line.append(w) + else: + cmd_line.append(w) + + return "\n ".join(cmd_line) + def __str__(self): """Returns a version tuple as major.minor.patch from self.version"= "" return self.ver_str(self.version) @@ -130,14 +151,13 @@ class PythonVersion: else: new_python_cmd =3D None =20 - if show_alternatives: + if show_alternatives and available_versions: print("You could run, instead:") for _, cmd in available_versions: args =3D [cmd, script_path] + sys.argv[1:] =20 - cmd_str =3D " ".join(args) - print(f" {cmd_str}") - print() + cmd_str =3D indent(PythonVersion.cmd_print(args), " ") + print(f"{cmd_str}\n") =20 if bail_out: msg =3D f"Python {python_ver} not supported. Bailing out" diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrap= per index e24486dede76..103b0044bb0b 100755 --- a/tools/docs/sphinx-build-wrapper +++ b/tools/docs/sphinx-build-wrapper @@ -774,7 +774,8 @@ def main(): =20 args =3D parser.parse_args() =20 - PythonVersion.check_python(MIN_PYTHON_VERSION) + PythonVersion.check_python(MIN_PYTHON_VERSION, show_alternatives=3DTru= e, + bail_out=3DTrue) =20 builder =3D SphinxBuilder(builddir=3Dargs.builddir, verbose=3Dargs.verbose, n_jobs=3Dargs.jobs, diff --git a/tools/docs/sphinx-pre-install b/tools/docs/sphinx-pre-install index 663d4e2a3f57..698989584b6a 100755 --- a/tools/docs/sphinx-pre-install +++ b/tools/docs/sphinx-pre-install @@ -1531,7 +1531,8 @@ def main(): =20 checker =3D SphinxDependencyChecker(args) =20 - PythonVersion.check_python(MIN_PYTHON_VERSION) + PythonVersion.check_python(MIN_PYTHON_VERSION, + bail_out=3DTrue, success_on_error=3DTrue) checker.check_needs() =20 # Call main if not used as module --=20 2.51.0 From nobody Thu Oct 2 09:16:53 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6CB33305E29; Thu, 18 Sep 2025 11:55:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; cv=none; b=KFS7j5p25NeulfAwycnGNVB9XJHI/kvOaXwMmsIHs8kaENBAKhsZ7jhA1EwPHJOMvyheHEFSWYGhU/Jx0A90dbhyPPt3cIU111ZiSAXWVVrdQeg2eaPpk4QZCSFdd5fnYs6DHspZgasi4kLjdvZiHiNJcuL4Ag2qihSg7RT+WT4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1758196507; c=relaxed/simple; bh=5h0Y2l0VICf3uSdAhL346MOGEeZeAoHZKXVIcjeYSFk=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tO0yAn4ijTDpJ4PLup8EoyWJ43tQ+s1kx0Gx4pGv4oLJQvLxuIHSPjpR2qqhIaJINRHWUgWXGk+eu14F3P2mbI9q3AeG44BCo9zvdEKi0bFCUoKy9TBTrX34LT/uyD6icHZo11T9dLdKdY6A8FkiiLOj9YCMO1nuYHyn8te36dg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=TCMrCqG7; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="TCMrCqG7" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4F872C116D0; Thu, 18 Sep 2025 11:55:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1758196507; bh=5h0Y2l0VICf3uSdAhL346MOGEeZeAoHZKXVIcjeYSFk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TCMrCqG7ctX+pVvP6a/KXeUHdrHQq+lo+vQf0AiCROyz9+D7XnSXsZ1rcf3JRi43+ EFW6+W4SasJrXPvbtwoSd47DGAOVVeHNV2weriIBV4dh7lkasigtz30AIL4Pw0WIWW 27qquDem5WqvyTXakF7/dINwECygPHQ15sYwIJKXf9ucIvVm8K2l8GeHwaQtaEQ3ew FGilNrFz7X1YLB/obTMOTKGV5vPoMOPLbRBvFOM76hKEqiXihGqdayu9msrvGEdDNc QyMQ8/sT4TUvzxtmKzj7Map9p6TO53QyrzaWSpUfMIkgHWzoTxD+q+thR0o8tWkUv4 7MaE+Ou+Q7VFg== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1uzDE8-0000000Crto-286H; Thu, 18 Sep 2025 13:55:04 +0200 From: Mauro Carvalho Chehab To: Linux Doc Mailing List , Jonathan Corbet Cc: Mauro Carvalho Chehab , "Akira Yokosawa" , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v8 24/24] tools/docs: sphinx-build-wrapper: add support to run inside venv Date: Thu, 18 Sep 2025 13:54:58 +0200 Message-ID: X-Mailer: git-send-email 2.51.0 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Sometimes, it is desired to run Sphinx from a virtual environment. Add a command line parameter to automatically build Sphinx from such environment. Signed-off-by: Mauro Carvalho Chehab --- tools/docs/sphinx-build-wrapper | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/tools/docs/sphinx-build-wrapper b/tools/docs/sphinx-build-wrap= per index 103b0044bb0b..6ed3d58ec277 100755 --- a/tools/docs/sphinx-build-wrapper +++ b/tools/docs/sphinx-build-wrapper @@ -69,6 +69,7 @@ from jobserver import JobserverExec # pylint: dis= able=3DC0413,C0411,E0401 # # Some constants # +VENV_DEFAULT =3D "sphinx_latest" MIN_PYTHON_VERSION =3D PythonVersion("3.7").version PAPER =3D ["", "a4", "letter"] =20 @@ -171,8 +172,10 @@ class SphinxBuilder: if not verbose: self.sphinxopts +=3D ["-q"] =20 - def __init__(self, builddir, verbose=3DFalse, n_jobs=3DNone, interacti= ve=3DNone): + def __init__(self, builddir, venv=3DNone, verbose=3DFalse, n_jobs=3DNo= ne, + interactive=3DNone): """Initialize internal variables""" + self.venv =3D venv self.verbose =3D None =20 # @@ -220,6 +223,21 @@ class SphinxBuilder: =20 self.get_sphinx_extra_opts(n_jobs) =20 + # + # If venv command line argument is specified, run Sphinx from venv + # + if venv: + bin_dir =3D os.path.join(venv, "bin") + if not os.path.isfile(os.path.join(bin_dir, "activate")): + sys.exit(f"Venv {venv} not found.") + + # "activate" virtual env + self.env["PATH"] =3D bin_dir + ":" + self.env["PATH"] + self.env["VIRTUAL_ENV"] =3D venv + if "PYTHONHOME" in self.env: + del self.env["PYTHONHOME"] + print(f"Setting venv to {venv}") + def run_sphinx(self, sphinx_build, build_args, *args, **pwargs): """ Executes sphinx-build using current python3 command. @@ -254,7 +272,12 @@ class SphinxBuilder: if self.n_jobs: n_jobs =3D str(self.n_jobs) =20 - cmd =3D [sys.executable, sphinx_build] + if self.venv: + cmd =3D ["python"] + else: + cmd =3D [sys.executable,] + + cmd +=3D [sphinx_build] cmd +=3D [f"-j{n_jobs}"] cmd +=3D self.sphinxopts cmd +=3D build_args @@ -772,12 +795,16 @@ def main(): parser.add_argument('-i', '--interactive', action=3D'store_true', help=3D"Change latex default to run in interactive= mode") =20 + parser.add_argument("-V", "--venv", nargs=3D'?', const=3Df'{VENV_DEFAU= LT}', + default=3DNone, + help=3Df'If used, run Sphinx from a venv dir (defa= ult dir: {VENV_DEFAULT})') + args =3D parser.parse_args() =20 PythonVersion.check_python(MIN_PYTHON_VERSION, show_alternatives=3DTru= e, bail_out=3DTrue) =20 - builder =3D SphinxBuilder(builddir=3Dargs.builddir, + builder =3D SphinxBuilder(builddir=3Dargs.builddir, venv=3Dargs.venv, verbose=3Dargs.verbose, n_jobs=3Dargs.jobs, interactive=3Dargs.interactive) =20 --=20 2.51.0