From nobody Sat Oct 4 14:32:49 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 362BA2882A1; Fri, 15 Aug 2025 11:51:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; cv=none; b=hCvBodnr8LrUTeKHbX71yoqUEHRrurr5vMDEPddl5BBbWqFmkOn2R8cGmewNIhJiVvC7CJe/kEZN7m/iv3HXf0+RC+GabAX8HoY1awU7JMuJOWNJANh+r+1xWaIa3EeY0wpIzmC6HewEgjXmCXCg+ci9AsvFDDAeodvDJV1XtvM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; c=relaxed/simple; bh=ztQX4EfYtVNL/Ti82jJ338U7XQtnB7XlW3/TuIKgFuo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZUkLcghPkBzweqf8ny837dowzxQJlEIFHl8QZzcLdt/5/mHT5UZfz+P/cmCT5j5cEfsKB+0a8sgm6YR+gosTYMYqqVneX6l7FSNUy9u5GW40A0QG2KB3LRGiyAf5kv0DxZk+3HMh+jbgcqYTH67fBNKhZoYW5rS3XBtaZ9kW2uE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=N9tYCbz6; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="N9tYCbz6" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B5BF5C4CEEB; Fri, 15 Aug 2025 11:51:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755258665; bh=ztQX4EfYtVNL/Ti82jJ338U7XQtnB7XlW3/TuIKgFuo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=N9tYCbz6jc+QHr5RON840Fb+ko/vQ++JWptY51RJl5e/6ao6QuwZgfClpXPQ9E+kr smjQcRvq3HrKvD3GAxdG+ECh6bMN/c6q0j3urGw7AurQ0ATTpRzKDmIZ+Y6TlpStS/ vlcOtncwGthbg/BcS5ANd1TUIk+ncRtW5S0PjH/9MFMUWOM54F2B2XESgFTtBtA/6t QGRkJPBETpJT7C1N5anKfTXkQUWeDlcPM5CVjg6zAbXAdMtKuceD6SfXne6Z6WUZF/ RXemCUKnpLuhCaiCB0QEG1ArF9T2WZBrI7ugKyyyv5nKvsOANdpVFEEkpxn7T97jSZ 7ND9VfIxJ89dw== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1umsxb-000000042oG-4AJk; Fri, 15 Aug 2025 13:51:03 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH 01/11] scripts/jobserver-exec: move the code to a class Date: Fri, 15 Aug 2025 13:50:29 +0200 Message-ID: <79a98d50aa4bea322821e4c7418f09b9eca0c591.1755258303.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Convert the code inside jobserver-exec to a class and properly document it. Using a class allows reusing the jobserver logic on other scripts. While the main code remains unchanged, being compatible with Python 2.6 and 3.0+, its coding style now follows a more modern standard, having tabs replaced by a 4-spaces indent, passing autopep8, black and pylint. The code now allows allows using a pythonic way to enter/exit a python code, e.g. it now supports: with JobserverExec() as jobserver: jobserver.run(sys.argv[1:]) With the new code, the __exit__() function should ensure that the jobserver slot will be closed at the end, even if something bad happens somewhere. Signed-off-by: Mauro Carvalho Chehab --- scripts/jobserver-exec | 218 ++++++++++++++++++++++++++++------------- 1 file changed, 151 insertions(+), 67 deletions(-) diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec index 7eca035472d3..b386b1a845de 100755 --- a/scripts/jobserver-exec +++ b/scripts/jobserver-exec @@ -1,77 +1,161 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0+ # +# pylint: disable=3DC0103,C0209 +# # This determines how many parallel tasks "make" is expecting, as it is # not exposed via an special variables, reserves them all, runs a subproce= ss # with PARALLELISM environment variable set, and releases the jobs back ag= ain. # # https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#= POSIX-Jobserver -from __future__ import print_function -import os, sys, errno + +""" +Interacts with the POSIX jobserver during the Kernel build time. + +A "normal" jobserver task, like the one initiated by a make subrocess woul= d do: + + - open read/write file descriptors to communicate with the job server; + - ask for one slot by calling: + claim =3D os.read(reader, 1) + - when the job finshes, call: + os.write(writer, b"+") # os.write(writer, claim) + +Here, the goal is different: This script aims to get the remaining number +of slots available, using all of them to run a command which handle tasks = in +parallel. To to that, it has a loop that ends only after there are no +slots left. It then increments the number by one, in order to allow a +call equivalent to make -j$((claim+1)), e.g. having a parent make creating +$claim child to do the actual work. + +The end goal here is to keep the total number of build tasks under the +limit established by the initial make -j$n_proc call. +""" + +import errno +import os import subprocess +import sys =20 -# Extract and prepare jobserver file descriptors from environment. -claim =3D 0 -jobs =3D b"" -try: - # Fetch the make environment options. - flags =3D os.environ['MAKEFLAGS'] - - # Look for "--jobserver=3DR,W" - # Note that GNU Make has used --jobserver-fds and --jobserver-auth - # so this handles all of them. - opts =3D [x for x in flags.split(" ") if x.startswith("--jobserver")] - - # Parse out R,W file descriptor numbers and set them nonblocking. - # If the MAKEFLAGS variable contains multiple instances of the - # --jobserver-auth=3D option, the last one is relevant. - fds =3D opts[-1].split("=3D", 1)[1] - - # Starting with GNU Make 4.4, named pipes are used for reader and writer. - # Example argument: --jobserver-auth=3Dfifo:/tmp/GMfifo8134 - _, _, path =3D fds.partition('fifo:') - - if path: - reader =3D os.open(path, os.O_RDONLY | os.O_NONBLOCK) - writer =3D os.open(path, os.O_WRONLY) - else: - reader, writer =3D [int(x) for x in fds.split(",", 1)] - # Open a private copy of reader to avoid setting nonblocking - # on an unexpecting process with the same reader fd. - reader =3D os.open("/proc/self/fd/%d" % (reader), - os.O_RDONLY | os.O_NONBLOCK) - - # Read out as many jobserver slots as possible. - while True: - try: - slot =3D os.read(reader, 8) - jobs +=3D slot - except (OSError, IOError) as e: - if e.errno =3D=3D errno.EWOULDBLOCK: - # Stop at the end of the jobserver queue. - break - # If something went wrong, give back the jobs. - if len(jobs): - os.write(writer, jobs) - raise e - # Add a bump for our caller's reserveration, since we're just going - # to sit here blocked on our child. - claim =3D len(jobs) + 1 -except (KeyError, IndexError, ValueError, OSError, IOError) as e: - # Any missing environment strings or bad fds should result in just - # not being parallel. - pass - -# We can only claim parallelism if there was a jobserver (i.e. a top-level -# "-jN" argument) and there were no other failures. Otherwise leave out the -# environment variable and let the child figure out what is best. -if claim > 0: - os.environ['PARALLELISM'] =3D '%d' % (claim) - -rc =3D subprocess.call(sys.argv[1:]) - -# Return all the reserved slots. -if len(jobs): - os.write(writer, jobs) - -sys.exit(rc) + +class JobserverExec: + """ + Claim all slots from make using POSIX Jobserver. + + The main methods here are: + - open(): reserves all slots; + - close(): method returns all used slots back to make; + - run(): executes a command setting PARALLELISM=3D + """ + + def __init__(self): + """Initialize internal vars""" + self.claim =3D 0 + self.jobs =3D b"" + self.reader =3D None + self.writer =3D None + self.is_open =3D False + + def open(self): + """Reserve all available slots to be claimed later on""" + + if self.is_open: + return + + try: + # Fetch the make environment options. + flags =3D os.environ["MAKEFLAGS"] + # Look for "--jobserver=3DR,W" + # Note that GNU Make has used --jobserver-fds and --jobserver-= auth + # so this handles all of them. + opts =3D [x for x in flags.split(" ") if x.startswith("--jobse= rver")] + + # Parse out R,W file descriptor numbers and set them nonblocki= ng. + # If the MAKEFLAGS variable contains multiple instances of the + # --jobserver-auth=3D option, the last one is relevant. + fds =3D opts[-1].split("=3D", 1)[1] + + # Starting with GNU Make 4.4, named pipes are used for reader + # and writer. + # Example argument: --jobserver-auth=3Dfifo:/tmp/GMfifo8134 + _, _, path =3D fds.partition("fifo:") + + if path: + self.reader =3D os.open(path, os.O_RDONLY | os.O_NONBLOCK) + self.writer =3D os.open(path, os.O_WRONLY) + else: + self.reader, self.writer =3D [int(x) for x in fds.split(",= ", 1)] + # Open a private copy of reader to avoid setting nonblocki= ng + # on an unexpecting process with the same reader fd. + self.reader =3D os.open("/proc/self/fd/%d" % (self.reader), + os.O_RDONLY | os.O_NONBLOCK) + + # Read out as many jobserver slots as possible + while True: + try: + slot =3D os.read(self.reader, 8) + self.jobs +=3D slot + except (OSError, IOError) as e: + if e.errno =3D=3D errno.EWOULDBLOCK: + # Stop at the end of the jobserver queue. + break + # If something went wrong, give back the jobs. + if self.jobs: + os.write(self.writer, self.jobs) + raise e + + # Add a bump for our caller's reserveration, since we're just = going + # to sit here blocked on our child. + self.claim =3D len(self.jobs) + 1 + + except (KeyError, IndexError, ValueError, OSError, IOError): + # Any missing environment strings or bad fds should result in = just + # not being parallel. + self.claim =3D None + + self.is_open =3D True + + def close(self): + """Return all reserved slots to Jobserver""" + + if not self.is_open: + return + + # Return all the reserved slots. + if len(self.jobs): + os.write(self.writer, self.jobs) + + self.is_open =3D False + + def __enter__(self): + self.open() + return self + + def __exit__(self, exc_type, exc_value, exc_traceback): + self.close() + + def run(self, cmd): + """ + Run a command setting PARALLELISM env variable to the number of + available job slots (claim) + 1, e.g. it will reserve claim slots + to do the actual build work, plus one to monitor its childs. + """ + self.open() # Ensure that self.claim is set + + # We can only claim parallelism if there was a jobserver (i.e. a + # top-level "-jN" argument) and there were no other failures. Othe= rwise + # leave out the environment variable and let the child figure out = what + # is best. + if self.claim: + os.environ["PARALLELISM"] =3D str(self.claim) + + return subprocess.call(cmd) + + +def main(): + """Main program""" + with JobserverExec() as jobserver: + jobserver.run(sys.argv[1:]) + + +if __name__ =3D=3D "__main__": + main() --=20 2.50.1 From nobody Sat Oct 4 14:32:49 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 583062FB96A; Fri, 15 Aug 2025 11:51:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; cv=none; b=Hgj9tOBey0Jb+lVXc0aeYQIhLcs0tli+iHNaLTNCLbr7U/AOryf4TpiS8IMVVbdWQFJcjYKRNTN6lGgcK7O2g7a7F3QzTHjRT3RBMcXJyylUFEaBX+hRdr4jTP/fqMLckkzVm312WrN3E2getvfXIi1ZopD8VznePMu5kJAVhjA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; c=relaxed/simple; bh=J+qWJJVTYcSvzpVZS39x9lWxV/iv7PYOZfnHXHy1oUM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=cTm/V+4/lTglbEGcuF7sjoCKoKl3+xuaU1D+CINwryTy17y/JU8FEeSI3p+8g/oPanlIjJrT5zoI4Icqnpkojx7ddjqnxforseyjX+kwel9t/g4/NStl52xICUquS2Qxg5L7RnsBln56YN0TOLVFu9igM/TdO4IcGAaM8+q2ArI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=bXYhasgl; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="bXYhasgl" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B99FBC4CEF5; Fri, 15 Aug 2025 11:51:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755258665; bh=J+qWJJVTYcSvzpVZS39x9lWxV/iv7PYOZfnHXHy1oUM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bXYhasglZb2FQE5i1bLt3rdyFkliju4805Y6+HJL4tjD2HL91xWSyie7aAik2ymAL O+MJkrzFEk6PDnoknm+EdIFHADZkpYaFOqU58uW2F0SodAKI99uO+5x8YgoCYPWKjy 6KH1Vfh/b0ALSorBSmahwCw0o4yuyHka5UKpewhJ+tKNcYspFzrkPXLL1gvFH5eivV vF71VgAJe9bKB5+qmP743Y+FtSed/PcoS3jf0HUYogmeFcY/tBedM2VqhPwpwm6Mt+ s14Wmzld94yLOWFGy4QNcfo2kNSz7GtwFbLB2qEJcm10t1Sy/IoXt0Uo2+RAX7uCuV UuujX2y/IPEWA== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1umsxc-000000042oK-05Iq; Fri, 15 Aug 2025 13:51:04 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH 02/11] scripts/jobserver-exec: move its class to the lib directory Date: Fri, 15 Aug 2025 13:50:30 +0200 Message-ID: <209861470d5a806265a054a8cb111000bbb703e2.1755258303.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" To make it easier to be re-used, move the JobserverExec class to the library directory. Signed-off-by: Mauro Carvalho Chehab --- scripts/jobserver-exec | 152 +++------------------------------------ scripts/lib/jobserver.py | 149 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 160 insertions(+), 141 deletions(-) create mode 100755 scripts/lib/jobserver.py diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec index b386b1a845de..40a0f0058733 100755 --- a/scripts/jobserver-exec +++ b/scripts/jobserver-exec @@ -1,155 +1,25 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0+ -# -# pylint: disable=3DC0103,C0209 -# -# This determines how many parallel tasks "make" is expecting, as it is -# not exposed via an special variables, reserves them all, runs a subproce= ss -# with PARALLELISM environment variable set, and releases the jobs back ag= ain. -# -# https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html#= POSIX-Jobserver =20 -""" -Interacts with the POSIX jobserver during the Kernel build time. - -A "normal" jobserver task, like the one initiated by a make subrocess woul= d do: - - - open read/write file descriptors to communicate with the job server; - - ask for one slot by calling: - claim =3D os.read(reader, 1) - - when the job finshes, call: - os.write(writer, b"+") # os.write(writer, claim) - -Here, the goal is different: This script aims to get the remaining number -of slots available, using all of them to run a command which handle tasks = in -parallel. To to that, it has a loop that ends only after there are no -slots left. It then increments the number by one, in order to allow a -call equivalent to make -j$((claim+1)), e.g. having a parent make creating -$claim child to do the actual work. - -The end goal here is to keep the total number of build tasks under the -limit established by the initial make -j$n_proc call. -""" - -import errno import os -import subprocess import sys =20 +LIB_DIR =3D "lib" +SRC_DIR =3D os.path.dirname(os.path.realpath(__file__)) =20 -class JobserverExec: - """ - Claim all slots from make using POSIX Jobserver. +sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR)) =20 - The main methods here are: - - open(): reserves all slots; - - close(): method returns all used slots back to make; - - run(): executes a command setting PARALLELISM=3D - """ +from jobserver import JobserverExec # pylint: disable=3DC= 0415 =20 - def __init__(self): - """Initialize internal vars""" - self.claim =3D 0 - self.jobs =3D b"" - self.reader =3D None - self.writer =3D None - self.is_open =3D False =20 - def open(self): - """Reserve all available slots to be claimed later on""" - - if self.is_open: - return - - try: - # Fetch the make environment options. - flags =3D os.environ["MAKEFLAGS"] - # Look for "--jobserver=3DR,W" - # Note that GNU Make has used --jobserver-fds and --jobserver-= auth - # so this handles all of them. - opts =3D [x for x in flags.split(" ") if x.startswith("--jobse= rver")] - - # Parse out R,W file descriptor numbers and set them nonblocki= ng. - # If the MAKEFLAGS variable contains multiple instances of the - # --jobserver-auth=3D option, the last one is relevant. - fds =3D opts[-1].split("=3D", 1)[1] - - # Starting with GNU Make 4.4, named pipes are used for reader - # and writer. - # Example argument: --jobserver-auth=3Dfifo:/tmp/GMfifo8134 - _, _, path =3D fds.partition("fifo:") - - if path: - self.reader =3D os.open(path, os.O_RDONLY | os.O_NONBLOCK) - self.writer =3D os.open(path, os.O_WRONLY) - else: - self.reader, self.writer =3D [int(x) for x in fds.split(",= ", 1)] - # Open a private copy of reader to avoid setting nonblocki= ng - # on an unexpecting process with the same reader fd. - self.reader =3D os.open("/proc/self/fd/%d" % (self.reader), - os.O_RDONLY | os.O_NONBLOCK) - - # Read out as many jobserver slots as possible - while True: - try: - slot =3D os.read(self.reader, 8) - self.jobs +=3D slot - except (OSError, IOError) as e: - if e.errno =3D=3D errno.EWOULDBLOCK: - # Stop at the end of the jobserver queue. - break - # If something went wrong, give back the jobs. - if self.jobs: - os.write(self.writer, self.jobs) - raise e - - # Add a bump for our caller's reserveration, since we're just = going - # to sit here blocked on our child. - self.claim =3D len(self.jobs) + 1 - - except (KeyError, IndexError, ValueError, OSError, IOError): - # Any missing environment strings or bad fds should result in = just - # not being parallel. - self.claim =3D None - - self.is_open =3D True - - def close(self): - """Return all reserved slots to Jobserver""" - - if not self.is_open: - return - - # Return all the reserved slots. - if len(self.jobs): - os.write(self.writer, self.jobs) - - self.is_open =3D False - - def __enter__(self): - self.open() - return self - - def __exit__(self, exc_type, exc_value, exc_traceback): - self.close() - - def run(self, cmd): - """ - Run a command setting PARALLELISM env variable to the number of - available job slots (claim) + 1, e.g. it will reserve claim slots - to do the actual build work, plus one to monitor its childs. - """ - self.open() # Ensure that self.claim is set - - # We can only claim parallelism if there was a jobserver (i.e. a - # top-level "-jN" argument) and there were no other failures. Othe= rwise - # leave out the environment variable and let the child figure out = what - # is best. - if self.claim: - os.environ["PARALLELISM"] =3D str(self.claim) - - return subprocess.call(cmd) +""" +Determines how many parallel tasks "make" is expecting, as it is +not exposed via an special variables, reserves them all, runs a subprocess +with PARALLELISM environment variable set, and releases the jobs back agai= n. =20 +See: + https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.htm= l#POSIX-Jobserver +""" =20 def main(): """Main program""" diff --git a/scripts/lib/jobserver.py b/scripts/lib/jobserver.py new file mode 100755 index 000000000000..98d8b0ff0c89 --- /dev/null +++ b/scripts/lib/jobserver.py @@ -0,0 +1,149 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0+ +# +# pylint: disable=3DC0103,C0209 +# +# + +""" +Interacts with the POSIX jobserver during the Kernel build time. + +A "normal" jobserver task, like the one initiated by a make subrocess woul= d do: + + - open read/write file descriptors to communicate with the job server; + - ask for one slot by calling: + claim =3D os.read(reader, 1) + - when the job finshes, call: + os.write(writer, b"+") # os.write(writer, claim) + +Here, the goal is different: This script aims to get the remaining number +of slots available, using all of them to run a command which handle tasks = in +parallel. To to that, it has a loop that ends only after there are no +slots left. It then increments the number by one, in order to allow a +call equivalent to make -j$((claim+1)), e.g. having a parent make creating +$claim child to do the actual work. + +The end goal here is to keep the total number of build tasks under the +limit established by the initial make -j$n_proc call. + +See: + https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.htm= l#POSIX-Jobserver +""" + +import errno +import os +import subprocess +import sys + +class JobserverExec: + """ + Claim all slots from make using POSIX Jobserver. + + The main methods here are: + - open(): reserves all slots; + - close(): method returns all used slots back to make; + - run(): executes a command setting PARALLELISM=3D + """ + + def __init__(self): + """Initialize internal vars""" + self.claim =3D 0 + self.jobs =3D b"" + self.reader =3D None + self.writer =3D None + self.is_open =3D False + + def open(self): + """Reserve all available slots to be claimed later on""" + + if self.is_open: + return + + try: + # Fetch the make environment options. + flags =3D os.environ["MAKEFLAGS"] + # Look for "--jobserver=3DR,W" + # Note that GNU Make has used --jobserver-fds and --jobserver-= auth + # so this handles all of them. + opts =3D [x for x in flags.split(" ") if x.startswith("--jobse= rver")] + + # Parse out R,W file descriptor numbers and set them nonblocki= ng. + # If the MAKEFLAGS variable contains multiple instances of the + # --jobserver-auth=3D option, the last one is relevant. + fds =3D opts[-1].split("=3D", 1)[1] + + # Starting with GNU Make 4.4, named pipes are used for reader + # and writer. + # Example argument: --jobserver-auth=3Dfifo:/tmp/GMfifo8134 + _, _, path =3D fds.partition("fifo:") + + if path: + self.reader =3D os.open(path, os.O_RDONLY | os.O_NONBLOCK) + self.writer =3D os.open(path, os.O_WRONLY) + else: + self.reader, self.writer =3D [int(x) for x in fds.split(",= ", 1)] + # Open a private copy of reader to avoid setting nonblocki= ng + # on an unexpecting process with the same reader fd. + self.reader =3D os.open("/proc/self/fd/%d" % (self.reader), + os.O_RDONLY | os.O_NONBLOCK) + + # Read out as many jobserver slots as possible + while True: + try: + slot =3D os.read(self.reader, 8) + self.jobs +=3D slot + except (OSError, IOError) as e: + if e.errno =3D=3D errno.EWOULDBLOCK: + # Stop at the end of the jobserver queue. + break + # If something went wrong, give back the jobs. + if self.jobs: + os.write(self.writer, self.jobs) + raise e + + # Add a bump for our caller's reserveration, since we're just = going + # to sit here blocked on our child. + self.claim =3D len(self.jobs) + 1 + + except (KeyError, IndexError, ValueError, OSError, IOError): + # Any missing environment strings or bad fds should result in = just + # not being parallel. + self.claim =3D None + + self.is_open =3D True + + def close(self): + """Return all reserved slots to Jobserver""" + + if not self.is_open: + return + + # Return all the reserved slots. + if len(self.jobs): + os.write(self.writer, self.jobs) + + self.is_open =3D False + + def __enter__(self): + self.open() + return self + + def __exit__(self, exc_type, exc_value, exc_traceback): + self.close() + + def run(self, cmd, *args, **pwargs): + """ + Run a command setting PARALLELISM env variable to the number of + available job slots (claim) + 1, e.g. it will reserve claim slots + to do the actual build work, plus one to monitor its childs. + """ + self.open() # Ensure that self.claim is set + + # We can only claim parallelism if there was a jobserver (i.e. a + # top-level "-jN" argument) and there were no other failures. Othe= rwise + # leave out the environment variable and let the child figure out = what + # is best. + if self.claim: + os.environ["PARALLELISM"] =3D str(self.claim) + + return subprocess.call(cmd, *args, **pwargs) --=20 2.50.1 From nobody Sat Oct 4 14:32:49 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1662DA935; Fri, 15 Aug 2025 11:51:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; cv=none; b=CphFP4qBw/opGj89KnBSxMceOGniuIQz1eKWSg7RvE8OEwbFuWYwC5ohMCSoRzlNyK7IqDZ3NtbBJaR7xsT7FKbux6CI7nUdzeQCVCvYWSphagzsmxmZ4IVMEVeLP6iPyHx8U2TcrEvbtVWuJQbrXFJMxJATmckRrMb65MGNZ4w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; c=relaxed/simple; bh=xHd5yELJPH27y6gsrGTyhvKGeyhmNEsyTEL9ELtOnwI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NX54kxlwLk2bzpQrpWwzVleZvI3h2Sx8PfGFEyMwGad4MhKHtTqmvggeD8Uez7tOaA/C3BSnmAR+UkvJ37nIA/uA5MQfR9ikaUB4/fAGzFPMyeLyxOMAf4hnAL3SXdo/hBuEnSeP2i/kME7+WhcNr8xEcF/OY5fby89/NX5L2kI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=nkb0Qwen; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="nkb0Qwen" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BB8D2C4CEF6; Fri, 15 Aug 2025 11:51:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755258665; bh=xHd5yELJPH27y6gsrGTyhvKGeyhmNEsyTEL9ELtOnwI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nkb0Qwen0NWcT2Pe9q3xTjpsyfmD4oZoMrU9BfA4DkOuC+xy2+/4iwviO3s06/rnY yjLx3BAjQmr5JFSJGj46hBa0AnfNx56nTu2OPmPmiGS0D6S/6a5ry0GeqFGGwI8CrX wptFNXPX+N7HKaHZPojIs+2ZpBz/WYTtu561YqHhFH4Uu/2yWjj3yA6zjKqUvb8xut Dmbt/5PCmq9KIWBSrFveIrc0F96el0UnykMdMA3PkPYccwfrmbV/LbtIQGl41YK1EG bIiP2p1jRzrDVtM07DEp52SLNCMgORudaDK8by5Ji9Yf/Xv5CTC4Ah7iE4xR7V8pQz QUM+NSt6HVrXA== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1umsxc-000000042oO-0CWI; Fri, 15 Aug 2025 13:51:04 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH 03/11] scripts/jobserver-exec: add a help message Date: Fri, 15 Aug 2025 13:50:31 +0200 Message-ID: <2d08ff8644a78256ebb427f40988d04c536f018e.1755258303.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Currently, calling it without an argument shows an ugly error message. Instead, print a message using pythondoc as description. Signed-off-by: Mauro Carvalho Chehab --- scripts/jobserver-exec | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/scripts/jobserver-exec b/scripts/jobserver-exec index 40a0f0058733..ae23afd344ec 100755 --- a/scripts/jobserver-exec +++ b/scripts/jobserver-exec @@ -1,6 +1,15 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0+ =20 +""" +Determines how many parallel tasks "make" is expecting, as it is +not exposed via any special variables, reserves them all, runs a subprocess +with PARALLELISM environment variable set, and releases the jobs back agai= n. + +See: + https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.htm= l#POSIX-Jobserver +""" + import os import sys =20 @@ -12,17 +21,12 @@ sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR)) from jobserver import JobserverExec # pylint: disable=3DC= 0415 =20 =20 -""" -Determines how many parallel tasks "make" is expecting, as it is -not exposed via an special variables, reserves them all, runs a subprocess -with PARALLELISM environment variable set, and releases the jobs back agai= n. - -See: - https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.htm= l#POSIX-Jobserver -""" - def main(): """Main program""" + if len(sys.argv) < 2: + name =3D os.path.basename(__file__) + sys.exit("usage: " + name +" command [args ...]\n" + __doc__) + with JobserverExec() as jobserver: jobserver.run(sys.argv[1:]) =20 --=20 2.50.1 From nobody Sat Oct 4 14:32:49 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B1313009ED; Fri, 15 Aug 2025 11:51:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; cv=none; b=WOYXu/r5mL5YX8Cc5qyiFqPt07+j+fLMtPStEkwV2h1HMaPSiAu6V2gmlSM7eGnUbGa7lAyKSur6Nlwupu3XO02QbEg3WjfXEmDmb7gSlBdfGzg17yEwVMWVBM5fK1hcoRTBzS9Ypxtx0JeDkNIdeorFdT19NrRGbwE8rQEjN50= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; c=relaxed/simple; bh=zumhMRghLThqkpHf4QwjSWnxlXD8mXBUhoe7SLsig1I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=o4ACOw4Fkvu5MqM8G6xZX71OKDycjFaudEUKMfa2tLsOqYHKpXP2lyTTAqfcJXoSBcCuqj13WkZlGw3FaXsE0UMqhaLkeKnI/bvoenN9dyucT0Gj0xZROnifpC4s25BccqWeTXSryqfFKjaV3MY16flajum8w2TbfdTKsN4GzyM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=iRGZFgqS; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="iRGZFgqS" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C21BCC4CEF8; Fri, 15 Aug 2025 11:51:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755258666; bh=zumhMRghLThqkpHf4QwjSWnxlXD8mXBUhoe7SLsig1I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iRGZFgqSQixi8wGJlByklO+ePIz27GiRzUoLm1djZgzEY+bcZRg+zVW2DH4IwgLjZ sv7AZFdUcTZyodzNZX1/JGPkYWuZThk+o3f2v6VI98cn1Vs2nCJ0UjT7Jc0z+NbfIQ UWzC/N9xe7N15fH7NwK9c9fLkmCOdmQ/TDgO1OlF4iCHlfQTPxIR2nTuXX0R8bzNRn 5huQRR4kBx/snn1z7zE5O0e6vTdcxIEezVY14ZtMLDSJ5VDALpwQCes0NPhsIuxUMH Nlqg3zcTnmAohNDBfluRvuxiA0byAWHaL7vF5KNd7TeDzS8MYZdaIAB5A54RMgxfMW aZbR4GA/l+9Fg== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1umsxc-000000042oS-0K5P; Fri, 15 Aug 2025 13:51:04 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , "Mauro Carvalho Chehab" , Alex Gaynor , Alice Ryhl , Andreas Hindborg , Benno Lossin , Boqun Feng , Danilo Krummrich , Gary Guo , Miguel Ojeda , Trevor Gross , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org Subject: [PATCH 04/11] scripts: sphinx-build-wrapper: add a wrapper for sphinx-build Date: Fri, 15 Aug 2025 13:50:32 +0200 Message-ID: <88a95c7f6996cafb247d6706060173b17a46d570.1755258303.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" There are too much magic inside docs Makefile to properly run sphinx-build. Create an ancillary script that contains all kernel-related sphinx-build call logic currently at Makefile. Such script is designed to work both as an standalone command and as part of a Makefile. As such, it properly handles POSIX jobserver used by GNU make. It should be noticed that, when running the script alone, it will only take care of sphinx-build and cleandocs target. As such: - it won't run "make rustdoc"; - no extra checks. Signed-off-by: Mauro Carvalho Chehab --- .pylintrc | 2 +- scripts/sphinx-build-wrapper | 627 +++++++++++++++++++++++++++++++++++ 2 files changed, 628 insertions(+), 1 deletion(-) create mode 100755 scripts/sphinx-build-wrapper diff --git a/.pylintrc b/.pylintrc index 30b8ae1659f8..f1d21379254b 100644 --- a/.pylintrc +++ b/.pylintrc @@ -1,2 +1,2 @@ [MASTER] -init-hook=3D'import sys; sys.path +=3D ["scripts/lib/kdoc", "scripts/lib/a= bi"]' +init-hook=3D'import sys; sys.path +=3D ["scripts/lib", "scripts/lib/kdoc",= "scripts/lib/abi"]' diff --git a/scripts/sphinx-build-wrapper b/scripts/sphinx-build-wrapper new file mode 100755 index 000000000000..5c728956b53c --- /dev/null +++ b/scripts/sphinx-build-wrapper @@ -0,0 +1,627 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2025 Mauro Carvalho Chehab +# +# pylint: disable=3DR0902, R0912, R0913, R0914, R0915, R0917, C0103 +# +# Converted from docs Makefile and parallel-wrapper.sh, both under +# GPLv2, copyrighted since 2008 by the following authors: +# +# Akira Yokosawa +# Arnd Bergmann +# Breno Leitao +# Carlos Bilbao +# Dave Young +# Donald Hunter +# Geert Uytterhoeven +# Jani Nikula +# Jan Stancek +# Jonathan Corbet +# Joshua Clayton +# Kees Cook +# Linus Torvalds +# Magnus Damm +# Masahiro Yamada +# Mauro Carvalho Chehab +# Maxim Cournoyer +# Peter Foley +# Randy Dunlap +# Rob Herring +# Shuah Khan +# Thorsten Blum +# Tomas Winkler + + +""" +Sphinx build wrapper that handles Kernel-specific business rules: + +- it gets the Kernel build environment vars; +- it determines what's the best parallelism; +- it handles SPHINXDIRS + +This tool ensures that MIN_PYTHON_VERSION is satisfied. If version is +below that, it seeks for a new Python version. If found, it re-runs using +the newer version. +""" + +import argparse +import os +import re +import shlex +import shutil +import subprocess +import sys + +from glob import glob + +LIB_DIR =3D "lib" +SRC_DIR =3D os.path.dirname(os.path.realpath(__file__)) + +sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR)) + +from jobserver import JobserverExec # pylint: disable=3DC= 0413 + + +def parse_version(version): + """Convert a major.minor.patch version into a tuple""" + return tuple(int(x) for x in version.split(".")) + +def ver_str(version): + """Returns a version tuple as major.minor.patch""" + + return ".".join([str(x) for x in version]) + +# Minimal supported Python version needed by Sphinx and its extensions +MIN_PYTHON_VERSION =3D parse_version("3.7") + +# Default value for --venv parameter +VENV_DEFAULT =3D "sphinx_latest" + +# List of make targets and its corresponding builder and output directory +TARGETS =3D { + "cleandocs": { + "builder": "clean", + }, + "htmldocs": { + "builder": "html", + }, + "epubdocs": { + "builder": "epub", + "out_dir": "epub", + }, + "texinfodocs": { + "builder": "texinfo", + "out_dir": "texinfo", + }, + "infodocs": { + "builder": "texinfo", + "out_dir": "texinfo", + }, + "latexdocs": { + "builder": "latex", + "out_dir": "latex", + }, + "pdfdocs": { + "builder": "latex", + "out_dir": "latex", + }, + "xmldocs": { + "builder": "xml", + "out_dir": "xml", + }, + "linkcheckdocs": { + "builder": "linkcheck" + }, +} + +# Paper sizes. An empty value will pick the default +PAPER =3D ["", "a4", "letter"] + +class SphinxBuilder: + """ + Handles a sphinx-build target, adding needed arguments to build + with the Kernel. + """ + + def is_rust_enabled(self): + """Check if rust is enabled at .config""" + config_path =3D os.path.join(self.srctree, ".config") + if os.path.isfile(config_path): + with open(config_path, "r", encoding=3D"utf-8") as f: + return "CONFIG_RUST=3Dy" in f.read() + return False + + def get_path(self, path, abs_path=3DFalse): + """ + Ancillary routine to handle patches the right way, as shell does. + + It first expands "~" and "~user". Then, if patch is not absolute, + join self.srctree. Finally, if requested, convert to abspath. + """ + + path =3D os.path.expanduser(path) + if not path.startswith("/"): + path =3D os.path.join(self.srctree, path) + + if abs_path: + return os.path.abspath(path) + + return path + + def __init__(self, venv=3DNone, verbose=3DFalse): + """Initialize internal variables""" + self.venv =3D venv + self.verbose =3D verbose + + # Normal variables passed from Kernel's makefile + self.kernelversion =3D os.environ.get("KERNELVERSION", "unknown") + self.kernelrelease =3D os.environ.get("KERNELRELEASE", "unknown") + self.pdflatex =3D os.environ.get("PDFLATEX", "xelatex") + self.latexopts =3D os.environ.get("LATEXOPTS", "-interaction=3Dbat= chmode -no-shell-escape") + + # Source tree directory. This needs to be at os.environ, as + # Sphinx extensions and media uAPI makefile needs it + self.srctree =3D os.environ.get("srctree") + if not self.srctree: + self.srctree =3D "." + os.environ["srctree"] =3D self.srctree + + # Now that we can expand srctree, get other directories as well + self.sphinxbuild =3D os.environ.get("SPHINXBUILD", "sphinx-build") + self.kerneldoc =3D self.get_path(os.environ.get("KERNELDOC", + "scripts/kernel-doc.= py")) + self.obj =3D os.environ.get("obj", "Documentation") + self.builddir =3D self.get_path(os.path.join(self.obj, "output"), + abs_path=3DTrue) + + # Media uAPI needs it + os.environ["BUILDDIR"] =3D self.builddir + + # Detect if rust is enabled + self.config_rust =3D self.is_rust_enabled() + + # Get directory locations for LaTeX build toolchain + self.pdflatex_cmd =3D shutil.which(self.pdflatex) + self.latexmk_cmd =3D shutil.which("latexmk") + + self.env =3D os.environ.copy() + + # If venv parameter is specified, run Sphinx from venv + if venv: + bin_dir =3D os.path.join(venv, "bin") + if os.path.isfile(os.path.join(bin_dir, "activate")): + # "activate" virtual env + self.env["PATH"] =3D bin_dir + ":" + self.env["PATH"] + self.env["VIRTUAL_ENV"] =3D venv + if "PYTHONHOME" in self.env: + del self.env["PYTHONHOME"] + print(f"Setting venv to {venv}") + else: + sys.exit(f"Venv {venv} not found.") + + def run_sphinx(self, sphinx_build, sphinx_args, *args, **pwargs): + """ + Executes sphinx-build using current python3 command and setting + -j parameter if possible to run the build in parallel. + """ + + with JobserverExec() as jobserver: + if jobserver.claim: + parallelism =3D str(jobserver.claim) + else: + # As Sphinx has parallelism since version 1.7, we don't ne= ed + # any check here. + parallelism =3D "auto" + + cmd =3D [] + + if self.venv: + cmd.append("python") + else: + cmd.append(sys.executable) + + cmd.append(sphinx_build) + + if parallelism: + cmd.append("-j" + parallelism) + + cmd +=3D sphinx_args + + if self.verbose: + print(" ".join(cmd)) + + rc =3D subprocess.call(cmd, *args, **pwargs) + + def handle_html(self, css, output_dir): + """ + Extra steps for HTML and epub output. + + For such targets, we need to ensure that CSS will be properly + copied to the output _static directory + """ + + if not css: + return + + css =3D os.path.expanduser(css) + if not css.startswith("/"): + css =3D os.path.join(self.srctree, css) + + static_dir =3D os.path.join(output_dir, "_static") + os.makedirs(static_dir, exist_ok=3DTrue) + + try: + shutil.copy2(css, static_dir) + except (OSError, IOError) as e: + print(f"Warning: Failed to copy CSS: {e}", file=3Dsys.stderr) + + def handle_pdf(self, output_dirs): + """ + Extra steps for PDF output. + + As PDF is handled via a LaTeX output, after building the .tex file, + a new build is needed to create the PDF output from the latex + directory. + """ + builds =3D {} + max_len =3D 0 + + for from_dir in output_dirs: + pdf_dir =3D os.path.join(from_dir, "../pdf") + os.makedirs(pdf_dir, exist_ok=3DTrue) + + if self.latexmk_cmd: + latex_cmd =3D [self.latexmk_cmd, f"-{self.pdflatex}"] + else: + latex_cmd =3D [self.pdflatex] + + latex_cmd.extend(shlex.split(self.latexopts)) + + tex_suffix =3D ".tex" + + # Process each .tex file + has_tex =3D False + build_failed =3D False + with os.scandir(from_dir) as it: + for entry in it: + if not entry.name.endswith(tex_suffix): + continue + + name =3D entry.name[:-len(tex_suffix)] + has_tex =3D True + + try: + subprocess.run(latex_cmd + [entry.path], + cwd=3Dfrom_dir, check=3DTrue) + except subprocess.CalledProcessError: + # LaTeX PDF error code is almost useless: it retur= ns + # error codes even when build succeeds but has war= nings. + pass + + # Instead of checking errors, let's do the next best t= hing: + # check if the PDF file was actually created. + + pdf_name =3D name + ".pdf" + pdf_from =3D os.path.join(from_dir, pdf_name) + pdf_to =3D os.path.join(pdf_dir, pdf_name) + + if os.path.exists(pdf_from): + os.rename(pdf_from, pdf_to) + builds[name] =3D os.path.relpath(pdf_to, self.buil= ddir) + else: + builds[name] =3D "FAILED" + build_failed =3D True + + name =3D entry.name.removesuffix(".tex") + max_len =3D max(max_len, len(name)) + + if not has_tex: + name =3D os.path.basename(from_dir) + max_len =3D max(max_len, len(name)) + builds[name] =3D "FAILED (no .tex)" + build_failed =3D True + + msg =3D "Summary" + msg +=3D "\n" + "=3D" * len(msg) + print() + print(msg) + + for pdf_name, pdf_file in builds.items(): + print(f"{pdf_name:<{max_len}}: {pdf_file}") + + print() + + # return an error if a PDF file is missing + + if build_failed: + sys.exit(f"PDF build failed: not all PDF files were created.") + else: + print("All PDF files were built.") + + def handle_info(self, output_dirs): + """ + Extra steps for Info output. + + For texinfo generation, an additional make is needed from the + texinfo directory. + """ + + for output_dir in output_dirs: + try: + subprocess.run(["make", "info"], cwd=3Doutput_dir, check= =3DTrue) + except subprocess.CalledProcessError as e: + sys.exit(f"Error generating info docs: {e}") + + def get_make_media(self): + """ + The media uAPI requires an additional Makefile target. + """ + + mediadir =3D f"{self.obj}/userspace-api/media" + + make =3D os.environ.get("MAKE", "make") + build =3D os.environ.get("build", "-f $(srctree)/scripts/Makefile.= build obj") + + # Check if the script was started outside docs Makefile + if not os.environ.get("obj"): + mediadir =3D os.path.abspath(mediadir) + + # the build makefile var contains macros that require expand + make_media =3D f"{make} {build}=3D{mediadir}" + make_media =3D make_media.replace("$(", "${").replace(")", "}") + make_media =3D os.path.expandvars(make_media) + + # As it also contains multiple arguments, use shlex to split it + return shlex.split(make_media) + + def prepare_media(self, builder): + """ + Run userspace-api/media Makefile. + + The logic behind it are from the initial ports to Sphinx. + They're old and need to be replaced by a proper Sphinx extension. + While we don't do that, we need to explicitly call media Makefile + to build some files. + """ + + cmd =3D self.get_make_media() + [builder] + + if self.verbose: + print(" ".join(cmd)) + + with JobserverExec() as jobserver: + rc =3D jobserver.run(cmd, env=3Dself.env) + + if rc: + cmd_str =3D " ".join(cmd) + sys.exit(f"Failed to run {cmd_str}") + + def cleandocs(self, builder): + + shutil.rmtree(self.builddir, ignore_errors=3DTrue) + + self.prepare_media(builder) + + def build(self, target, sphinxdirs=3DNone, conf=3D"conf.py", + theme=3DNone, css=3DNone, paper=3DNone): + """ + Build documentation using Sphinx. This is the core function of this + module. It prepares all arguments required by sphinx-build. + """ + + builder =3D TARGETS[target]["builder"] + out_dir =3D TARGETS[target].get("out_dir", "") + + # Cleandocs doesn't require sphinx-build + if target =3D=3D "cleandocs": + self.cleandocs(builder) + return + + # Other targets require sphinx-build + sphinxbuild =3D shutil.which(self.sphinxbuild, path=3Dself.env["PA= TH"]) + if not sphinxbuild: + sys.exit(f"Error: {self.sphinxbuild} not found in PATH.\n") + + self.prepare_media(builder) + + if builder =3D=3D "latex": + if not self.pdflatex_cmd and not self.latexmk_cmd: + sys.exit("Error: pdflatex or latexmk required for PDF gene= ration") + + docs_dir =3D os.path.abspath(os.path.join(self.srctree, "Documenta= tion")) + + # Prepare base arguments for Sphinx build + kerneldoc =3D self.kerneldoc + if kerneldoc.startswith(self.srctree): + kerneldoc =3D os.path.relpath(kerneldoc, self.srctree) + + # Prepare common Sphinx options + args =3D [ + "-b", builder, + "-c", docs_dir, + ] + + if builder =3D=3D "latex": + if not paper: + paper =3D PAPER[1] + + args.extend(["-D", f"latex_elements.papersize=3D{paper}paper"]) + + if not self.verbose: + args.append("-q") + + if self.config_rust: + args.extend(["-t", "rustdoc"]) + + if conf: + self.env["SPHINX_CONF"] =3D self.get_path(conf, abs_path=3DTru= e) + + if not sphinxdirs: + sphinxdirs =3D os.environ.get("SPHINXDIRS", ".") + + # sphinxdirs can be a list or a whitespace-separated string + sphinxdirs_list =3D [] + for sphinxdir in sphinxdirs: + if isinstance(sphinxdir, list): + sphinxdirs_list +=3D sphinxdir + else: + for name in sphinxdir.split(" "): + sphinxdirs_list.append(name) + + # Build each directory + output_dirs =3D [] + for sphinxdir in sphinxdirs_list: + src_dir =3D os.path.join(docs_dir, sphinxdir) + doctree_dir =3D os.path.join(self.builddir, ".doctrees") + output_dir =3D os.path.join(self.builddir, sphinxdir, out_dir) + + # Make directory names canonical + src_dir =3D os.path.normpath(src_dir) + doctree_dir =3D os.path.normpath(doctree_dir) + output_dir =3D os.path.normpath(output_dir) + + os.makedirs(doctree_dir, exist_ok=3DTrue) + os.makedirs(output_dir, exist_ok=3DTrue) + + output_dirs.append(output_dir) + + build_args =3D args + [ + "-d", doctree_dir, + "-D", f"kerneldoc_bin=3D{kerneldoc}", + "-D", f"version=3D{self.kernelversion}", + "-D", f"release=3D{self.kernelrelease}", + "-D", f"kerneldoc_srctree=3D{self.srctree}", + src_dir, + output_dir, + ] + + # Execute sphinx-build + try: + self.run_sphinx(sphinxbuild, build_args, env=3Dself.env) + except Exception as e: + sys.exit(f"Build failed: {e}") + + # Ensure that html/epub will have needed static files + if target in ["htmldocs", "epubdocs"]: + self.handle_html(css, output_dir) + + # PDF and Info require a second build step + if target =3D=3D "pdfdocs": + self.handle_pdf(output_dirs) + elif target =3D=3D "infodocs": + self.handle_info(output_dirs) + + @staticmethod + def get_python_version(cmd): + """ + Get python version from a Python binary. As we need to detect if + are out there newer python binaries, we can't rely on sys.release = here. + """ + + result =3D subprocess.run([cmd, "--version"], check=3DTrue, + stdout=3Dsubprocess.PIPE, stderr=3Dsubproc= ess.PIPE, + universal_newlines=3DTrue) + version =3D result.stdout.strip() + + match =3D re.search(r"(\d+\.\d+\.\d+)", version) + if match: + return parse_version(match.group(1)) + + print(f"Can't parse version {version}") + return (0, 0, 0) + + @staticmethod + def find_python(): + """ + Detect if are out there any python 3.xy version newer than the + current one. + + Note: this routine is limited to up to 2 digits for python3. We + may need to update it one day, hopefully on a distant future. + """ + patterns =3D [ + "python3.[0-9]", + "python3.[0-9][0-9]", + ] + + # Seek for a python binary newer than MIN_PYTHON_VERSION + for path in os.getenv("PATH", "").split(":"): + for pattern in patterns: + for cmd in glob(os.path.join(path, pattern)): + if os.path.isfile(cmd) and os.access(cmd, os.X_OK): + version =3D SphinxBuilder.get_python_version(cmd) + if version >=3D MIN_PYTHON_VERSION: + return cmd + + return None + + @staticmethod + def check_python(): + """ + Check if the current python binary satisfies our minimal requireme= nt + for Sphinx build. If not, re-run with a newer version if found. + """ + cur_ver =3D sys.version_info[:3] + if cur_ver >=3D MIN_PYTHON_VERSION: + return + + python_ver =3D ver_str(cur_ver) + + new_python_cmd =3D SphinxBuilder.find_python() + if not new_python_cmd: + sys.exit(f"Python version {python_ver} is not supported anymor= e.") + + # Restart script using the newer version + script_path =3D os.path.abspath(sys.argv[0]) + args =3D [new_python_cmd, script_path] + sys.argv[1:] + + print(f"Python {python_ver} not supported. Changing to {new_python= _cmd}") + + try: + os.execv(new_python_cmd, args) + except OSError as e: + sys.exit(f"Failed to restart with {new_python_cmd}: {e}") + +def main(): + """ + Main function. The only mandatory argument is the target. If not + specified, the other arguments will use default values if not + specified at os.environ. + """ + parser =3D argparse.ArgumentParser(description=3D"Kernel documentation= builder") + + parser.add_argument("target", choices=3Dlist(TARGETS.keys()), + help=3D"Documentation target to build") + parser.add_argument("--sphinxdirs", nargs=3D"+", + help=3D"Specific directories to build") + parser.add_argument("--conf", default=3D"conf.py", + help=3D"Sphinx configuration file") + + parser.add_argument("--theme", help=3D"Sphinx theme to use") + + parser.add_argument("--css", help=3D"Custom CSS file for HTML/EPUB") + + parser.add_argument("--paper", choices=3DPAPER, default=3DPAPER[0], + help=3D"Paper size for LaTeX/PDF output") + + parser.add_argument("-v", "--verbose", action=3D'store_true', + help=3D"place build in verbose mode") + + parser.add_argument("-V", "--venv", nargs=3D'?', const=3Df'{VENV_DEFAU= LT}', + default=3DNone, + help=3Df'If used, run Sphinx from a venv dir (defa= ult dir: {VENV_DEFAULT})') + + args =3D parser.parse_args() + + if not args.verbose: + args.verbose =3D bool(os.environ.get("KBUILD_VERBOSE", "") !=3D "") + + SphinxBuilder.check_python() + + builder =3D SphinxBuilder(venv=3Dargs.venv, verbose=3Dargs.verbose) + + builder.build(args.target, sphinxdirs=3Dargs.sphinxdirs, conf=3Dargs.c= onf, + theme=3Dargs.theme, css=3Dargs.css, paper=3Dargs.paper) + +if __name__ =3D=3D "__main__": + main() --=20 2.50.1 From nobody Sat Oct 4 14:32:49 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 62CA62FCBE1; Fri, 15 Aug 2025 11:51:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; cv=none; b=pdunVbrezGR4DIkkZywB9sqOk0IgkJm0Id5a+neMvPnJVh2ST91bUNmJpd7Upg7bShQHXHuXyuh68fVMEUMPM7EvQEPOEIFOkJEiqcLixJGrZqAEtdxNVmUGLRHzKwiihSD94Pt7Jljuv5oveZqUiJmxj2oC8XCmY+imo0QYXwk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; c=relaxed/simple; bh=GYr8tmb4fF6RXpmQJmwNpnesZAm+FKFpIMMt22zUWek=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ERYhz0+qNFOg6NBblDKNFvbZNA2wg7WKefhktowoHqyA1DG8XTn88VwPIlKeqdVLcbJfN4OddWGIBJpWF26NlCznEmVlRlsnbLsKgYeZdvz2neoKr228iTZgBMDifxEgjKabgCt/XrGvdGBFrOq8HVTZsAbjDYCwbst9GBP+t0o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Pz2ke7K1; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Pz2ke7K1" Received: by smtp.kernel.org (Postfix) with ESMTPSA id CB17FC4CEF7; Fri, 15 Aug 2025 11:51:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755258666; bh=GYr8tmb4fF6RXpmQJmwNpnesZAm+FKFpIMMt22zUWek=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Pz2ke7K1JrytdVdkFRYd5y6b7sTYFT+c70cN4MoXzaSP/KAqidpInRrAdKsYMyAYd DYhcBdf4l5lcZetXKlZZaWKOM5cK3izEsxvQXJhldYV1Qez57Z0Q4f7upOXCfaaXYr hGzNOxn5SCaleubtPX9Qaj99SvfjNAB2oeQGmd0DQ4Tkzc0wrCVy9Wt9Lmkt4LUEVb lveu373kP/gmvjlqvFAOdI7UjxNt1XrhIfcM9psgmyDk16rxFdjvgHidZiFtnq84Od 6FdPyNm0NtyfKilirzxuOEE8a+26fIVJWgPEqpb6d0TnPnmp5uLvGDWXBUMK4Q0F3x P4GkfLp+ND43g== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1umsxc-000000042oW-0RQD; Fri, 15 Aug 2025 13:51:04 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , "Mauro Carvalho Chehab" , Alex Gaynor , Alice Ryhl , Andreas Hindborg , Benno Lossin , Boqun Feng , Danilo Krummrich , Gary Guo , Miguel Ojeda , Trevor Gross , linux-kernel@vger.kernel.org, rust-for-linux@vger.kernel.org Subject: [PATCH 05/11] docs: Makefile: cleanup the logic by using sphinx-build-wrapper Date: Fri, 15 Aug 2025 13:50:33 +0200 Message-ID: <3a62c847a812905f3a4c85f6b984d021ae88188d.1755258303.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Now that we have a sphinx-build-wrapper capable of handling all the needed step to build the supported build targets, cleanup the Makefile. Signed-off-by: Mauro Carvalho Chehab --- Documentation/Makefile | 127 ++++++++++------------------------------- 1 file changed, 29 insertions(+), 98 deletions(-) diff --git a/Documentation/Makefile b/Documentation/Makefile index 2ed334971acd..4013286bef04 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -19,7 +19,6 @@ endif =20 # You can set these variables from the command line. SPHINXBUILD =3D sphinx-build -SPHINXOPTS =3D SPHINXDIRS =3D . DOCS_THEME =3D DOCS_CSS =3D @@ -29,14 +28,14 @@ PAPER =3D BUILDDIR =3D $(obj)/output PDFLATEX =3D xelatex LATEXOPTS =3D -interaction=3Dbatchmode -no-shell-escape +BUILD_WRAPPER =3D $(srctree)/scripts/sphinx-build-wrapper + +PYTHONPYCACHEPREFIX ?=3D $(abspath $(BUILDDIR)/__pycache__) =20 # For denylisting "variable font" files # Can be overridden by setting as an env variable FONTS_CONF_DENY_VF ?=3D $(HOME)/deny-vf =20 -ifeq ($(findstring 1, $(KBUILD_VERBOSE)),) -SPHINXOPTS +=3D "-q" -endif =20 # User-friendly check for sphinx-build HAVE_SPHINX :=3D $(shell if which $(SPHINXBUILD) >/dev/null 2>&1; then ech= o 1; else echo 0; fi) @@ -51,62 +50,37 @@ ifeq ($(HAVE_SPHINX),0) =20 else # HAVE_SPHINX =20 -# User-friendly check for pdflatex and latexmk -HAVE_PDFLATEX :=3D $(shell if which $(PDFLATEX) >/dev/null 2>&1; then echo= 1; else echo 0; fi) -HAVE_LATEXMK :=3D $(shell if which latexmk >/dev/null 2>&1; then echo 1; e= lse echo 0; fi) +# Common documentation targets +infodocs texinfodocs latexdocs epubdocs xmldocs pdfdocs linkcheckdocs: + $(Q)@$(srctree)/scripts/sphinx-pre-install --version-check + +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \ + --sphinxdirs=3D"$(SPHINXDIRS)" \ + --conf=3D$(SPHINX_CONF) \ + --theme=3D$(DOCS_THEME) \ + --css=3D$(DOCS_CSS) \ + --paper=3D$(PAPER) =20 -ifeq ($(HAVE_LATEXMK),1) - PDFLATEX :=3D latexmk -$(PDFLATEX) -endif #HAVE_LATEXMK - -# Internal variables. -PAPEROPT_a4 =3D -D latex_elements.papersize=3Da4paper -PAPEROPT_letter =3D -D latex_elements.papersize=3Dletterpaper -ALLSPHINXOPTS =3D -D kerneldoc_srctree=3D$(srctree) -D kerneldoc_bin=3D$= (KERNELDOC) -ALLSPHINXOPTS +=3D $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) -ifneq ($(wildcard $(srctree)/.config),) -ifeq ($(CONFIG_RUST),y) - # Let Sphinx know we will include rustdoc - ALLSPHINXOPTS +=3D -t rustdoc -endif +# Special handling for pdfdocs +ifeq ($(shell which $(PDFLATEX) >/dev/null 2>&1; echo $$?),0) +pdfdocs: DENY_VF =3D XDG_CONFIG_HOME=3D$(FONTS_CONF_DENY_VF) +else +pdfdocs: + $(warning The '$(PDFLATEX)' command was not found. Make sure you have it = installed and in PATH to produce PDF output.) + @echo " SKIP Sphinx $@ target." endif -# the i18n builder cannot share the environment and doctrees with the othe= rs -I18NSPHINXOPTS =3D $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . - -# commands; the 'cmd' from scripts/Kbuild.include is not *loopable* -loop_cmd =3D $(echo-cmd) $(cmd_$(1)) || exit; - -# $2 sphinx builder e.g. "html" -# $3 name of the build subfolder / e.g. "userspace-api/media", used as: -# * dest folder relative to $(BUILDDIR) and -# * cache folder relative to $(BUILDDIR)/.doctrees -# $4 dest subfolder e.g. "man" for man pages at userspace-api/media/man -# $5 reST source folder relative to $(src), -# e.g. "userspace-api/media" for the linux-tv book-set at ./Documentati= on/userspace-api/media - -PYTHONPYCACHEPREFIX ?=3D $(abspath $(BUILDDIR)/__pycache__) =20 -quiet_cmd_sphinx =3D SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4) - cmd_sphinx =3D $(MAKE) BUILDDIR=3D$(abspath $(BUILDDIR)) $(build)=3D= Documentation/userspace-api/media $2 && \ - PYTHONPYCACHEPREFIX=3D"$(PYTHONPYCACHEPREFIX)" \ - BUILDDIR=3D$(abspath $(BUILDDIR)) SPHINX_CONF=3D$(abspath $(src)/$5/$(SPH= INX_CONF)) \ - $(PYTHON3) $(srctree)/scripts/jobserver-exec \ - $(CONFIG_SHELL) $(srctree)/Documentation/sphinx/parallel-wrapper.sh \ - $(SPHINXBUILD) \ - -b $2 \ - -c $(abspath $(src)) \ - -d $(abspath $(BUILDDIR)/.doctrees/$3) \ - -D version=3D$(KERNELVERSION) -D release=3D$(KERNELRELEASE) \ - $(ALLSPHINXOPTS) \ - $(abspath $(src)/$5) \ - $(abspath $(BUILDDIR)/$3/$4) && \ - if [ "x$(DOCS_CSS)" !=3D "x" ]; then \ - cp $(if $(patsubst /%,,$(DOCS_CSS)),$(abspath $(srctree)/$(DOCS_CSS)),$(= DOCS_CSS)) $(BUILDDIR)/$3/_static/; \ - fi +infodocs: texinfodocs =20 +# HTML main logic is identical to other targets. However, if rust is enabl= ed, +# an extra step at the end is required to generate rustdoc. htmldocs: - @$(srctree)/scripts/sphinx-pre-install --version-check - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var))) + $(Q)@$(srctree)/scripts/sphinx-pre-install --version-check + +$(Q)$(PYTHON3) $(BUILD_WRAPPER) $@ \ + --sphinxdirs=3D"$(SPHINXDIRS)" \ + --conf=3D$(SPHINX_CONF) \ + --theme=3D$(DOCS_THEME) \ + --css=3D$(DOCS_CSS) \ + --paper=3D$(PAPER) =20 # If Rust support is available and .config exists, add rustdoc generated c= ontents. # If there are any, the errors from this make rustdoc will be displayed but @@ -118,49 +92,6 @@ ifeq ($(CONFIG_RUST),y) endif endif =20 -texinfodocs: - @$(srctree)/scripts/sphinx-pre-install --version-check - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,texinfo,$(var),texin= fo,$(var))) - -# Note: the 'info' Make target is generated by sphinx itself when -# running the texinfodocs target define above. -infodocs: texinfodocs - $(MAKE) -C $(BUILDDIR)/texinfo info - -linkcheckdocs: - @$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,linkcheck,$(var),,$(v= ar))) - -latexdocs: - @$(srctree)/scripts/sphinx-pre-install --version-check - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$= (var))) - -ifeq ($(HAVE_PDFLATEX),0) - -pdfdocs: - $(warning The '$(PDFLATEX)' command was not found. Make sure you have it = installed and in PATH to produce PDF output.) - @echo " SKIP Sphinx $@ target." - -else # HAVE_PDFLATEX - -pdfdocs: DENY_VF =3D XDG_CONFIG_HOME=3D$(FONTS_CONF_DENY_VF) -pdfdocs: latexdocs - @$(srctree)/scripts/sphinx-pre-install --version-check - $(foreach var,$(SPHINXDIRS), \ - $(MAKE) PDFLATEX=3D"$(PDFLATEX)" LATEXOPTS=3D"$(LATEXOPTS)" $(DENY_VF)= -C $(BUILDDIR)/$(var)/latex || sh $(srctree)/scripts/check-variable-fonts.= sh || exit; \ - mkdir -p $(BUILDDIR)/$(var)/pdf; \ - mv $(subst .tex,.pdf,$(wildcard $(BUILDDIR)/$(var)/latex/*.tex)) $(BUI= LDDIR)/$(var)/pdf/; \ - ) - -endif # HAVE_PDFLATEX - -epubdocs: - @$(srctree)/scripts/sphinx-pre-install --version-check - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(v= ar))) - -xmldocs: - @$(srctree)/scripts/sphinx-pre-install --version-check - @+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var= ))) - endif # HAVE_SPHINX =20 # The following targets are independent of HAVE_SPHINX, and the rules shou= ld --=20 2.50.1 From nobody Sat Oct 4 14:32:50 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3624D287279; Fri, 15 Aug 2025 11:51:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; cv=none; b=fM0dOWlyX83uQFjVwebvarE3Xmv0W8KT8+EPhhLFybl0Xv0WRl7l38SVkOSaoE5753YS6INOzUzsCMH8poiaUt42IP9JLe9nrDqRaRtYQsCGxrOZ75n2puYkihSFmuoaoUmOxCTdcl7mP3BG2opLHuTVxjr1CVSydllOu75ec0s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; c=relaxed/simple; bh=sYk4mJd+UWrLz9Ei9ZPyoYmT4cWG5qlsc0OeM9aNUz8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lhxpKC5scaEXSkTj8S70lWffMla6KSuVx7+aq2QE/rsVrbWt+5BmMPUlUCjHgD7OtUFSNySmUBPFTnVwCreufRUND1ARDGDYtBrgJQe3kNtVBmkmIbNyWUuddMDwrFDbyjp0v0a/UTu1HvRzqLls1xJot0N/HLP7SfDxVuv7f9w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=j25J8fVx; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="j25J8fVx" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D41E7C4CEF9; Fri, 15 Aug 2025 11:51:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755258665; bh=sYk4mJd+UWrLz9Ei9ZPyoYmT4cWG5qlsc0OeM9aNUz8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=j25J8fVxjsK4GpVhTMfuFJSURqWFBKvVoxk1POR/DNfXU8Bz7X8i+spGhVPlG2++u v4KcXVzNMacLYDoeB4A4ARH4pRPnDjRfb98kB63zcO5ZTEeiG0Rhooz0OTPwDONYER NwuVGiy9q0HtUaaik6i+nJ3EV6G1aq5gKbDDbxk7vBMqXr3W8vs9vBxnRGKBswoEsV 8PTUZ9FQYzlSsqdp8og5ilIGslKUu8mIdgYXmz0BMOswZzT4jueF7p5jUSjDdJ8kaF W31BQhwbKZYViO43q8C1/DOfn46xtq79nBjq+ZDx41gejJ7+ew4rnilHcDsHutTdK+ POGfgiAm4+7uw== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1umsxc-000000042oa-0YFn; Fri, 15 Aug 2025 13:51:04 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH 06/11] docs: parallel-wrapper.sh: remove script Date: Fri, 15 Aug 2025 13:50:34 +0200 Message-ID: X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" The only usage of this script was docs Makefile. Now that it is using the new sphinx-build-wrapper, which has inside the code from parallel-wrapper.sh, we can drop this script. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/parallel-wrapper.sh | 33 ------------------------ 1 file changed, 33 deletions(-) delete mode 100644 Documentation/sphinx/parallel-wrapper.sh diff --git a/Documentation/sphinx/parallel-wrapper.sh b/Documentation/sphin= x/parallel-wrapper.sh deleted file mode 100644 index e54c44ce117d..000000000000 --- a/Documentation/sphinx/parallel-wrapper.sh +++ /dev/null @@ -1,33 +0,0 @@ -#!/bin/sh -# SPDX-License-Identifier: GPL-2.0+ -# -# Figure out if we should follow a specific parallelism from the make -# environment (as exported by scripts/jobserver-exec), or fall back to -# the "auto" parallelism when "-jN" is not specified at the top-level -# "make" invocation. - -sphinx=3D"$1" -shift || true - -parallel=3D"$PARALLELISM" -if [ -z "$parallel" ] ; then - # If no parallelism is specified at the top-level make, then - # fall back to the expected "-jauto" mode that the "htmldocs" - # target has had. - auto=3D$(perl -e 'open IN,"'"$sphinx"' --version 2>&1 |"; - while () { - if (m/([\d\.]+)/) { - print "auto" if ($1 >=3D "1.7") - } - } - close IN') - if [ -n "$auto" ] ; then - parallel=3D"$auto" - fi -fi -# Only if some parallelism has been determined do we add the -jN option. -if [ -n "$parallel" ] ; then - parallel=3D"-j$parallel" -fi - -exec "$sphinx" $parallel "$@" --=20 2.50.1 From nobody Sat Oct 4 14:32:50 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4B62C2C374E; Fri, 15 Aug 2025 11:51:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; cv=none; b=MqjkVAxkAcCuCKCx7t/ONv+Y70E9z36NAFfeoEVQN1E1hihqI0KpZdXd7sqCpPsymycQYjrCZyfi8K6YKX7fuj1WpfAeyfKtcY/4tyHoWiNoNElrCA1CZC/pmh0KEF55/GhBgXkSLbPMznZ1nARjrTsNRGkDIwSNdYOQcREl9eA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; c=relaxed/simple; bh=okAkyWYXR3GzaUm2kaZCuiY/fD0sE+ZIr5TU73IPSTg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=uT/liVpgd+DcW8yZnFLlL576TFKxPN89c/aAokaP7Jw2mR0Px6Lyy+1olZ+kPKVjDe72gSMCz7Bgl0Pbt9cAiRyxw/Q7NXmW6RKSPSq/JPUjRhY0fNyRPF2rQuhiyUT81A12o3qc8nL812W9H6/qKKYs6bgGrL0oK4Sws1XY5WM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=eheRVI45; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="eheRVI45" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D9AA2C4CEFD; Fri, 15 Aug 2025 11:51:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755258665; bh=okAkyWYXR3GzaUm2kaZCuiY/fD0sE+ZIr5TU73IPSTg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eheRVI45uZJxUcqr6HUgFEMqDTGPqz/T4XpWpCSly5bGpzdc7eEy7tAskXGJI2Ple A5HGZqtuNcBNlTIwJbQ1n3UrWo9BeBAvK8kEza9yMD8p8jBPX9G3oV1jidKFyo05Br 00rjGrm2qD535eUPQoNdIE4bnJJc6CRN//p3/rSfB/34M4B4KaainHoyC8lyl+Ut9s zAMrqLkypcFi57fqSWbhilvhsKLb2KdpdO7aqAqt8hZ8Y2+dS26SQpmUZlfPeCh943 R2cJxyya/PKBcNWp1ho7wxZv48c/55PAiGp6Ctt4+DBnmsldjNkbCxnpnaW+aOOOzg Corps6CE9PPiw== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1umsxc-000000042oe-0euP; Fri, 15 Aug 2025 13:51:04 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH 07/11] docs: Makefile: document latex/PDF PAPER= parameter Date: Fri, 15 Aug 2025 13:50:35 +0200 Message-ID: <70f03a706a01d1493ed3fe085b394cc15683fe51.1755258303.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" While the build system supports this for a long time, this was never documented. Add a documentation for it. Signed-off-by: Mauro Carvalho Chehab --- Documentation/Makefile | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Documentation/Makefile b/Documentation/Makefile index 4013286bef04..280728cf78b9 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -129,4 +129,6 @@ dochelp: @echo @echo ' make DOCS_CSS=3D{a .css file} adds a DOCS_CSS override file for= html/epub output.' @echo + @echo ' make PAPER=3D{a4|letter} Specifies the paper size used for LaTe= X/PDF output.' + @echo @echo ' Default location for the generated documents is Documentation/o= utput' --=20 2.50.1 From nobody Sat Oct 4 14:32:50 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 62D172FCC0B; Fri, 15 Aug 2025 11:51:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; cv=none; b=GKAFPpUaLKIWir78JfRMYVHzVycFLGVXgcI+SQ9RMEBWLXYEkGt7uoMmwxB6fntVerftuUFVozQ+unpAw5RcSLegz18H9DBK4TPCHcXDrTuTVUpC8nXzktjQ4KsruKAm+tTVkLH6dvKz+Yn9OsY5j+PGOEdrJQXyDtbNu9qmkHE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; c=relaxed/simple; bh=ZoOtF8id0ICXUTMlmq1Pqq/MM5WeKdvyH7/glTLcTBs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AnJX8yFIWKLMCXUbj/4P207wlk8gvQnwBLBTKRO5CzSSdUdaMMTzNW79fGfCTpyxl1e1NJZyz5UPH25lC1S+qohpJtoJs4P4KsbIGYGUDBZlqildo2QSahBk1ksoJxEWjwtChxD23+wIqJqfUfJlQkij+xmr4EKXho18D71iTMc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oJqbPr+o; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oJqbPr+o" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E5371C4CEFC; Fri, 15 Aug 2025 11:51:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755258665; bh=ZoOtF8id0ICXUTMlmq1Pqq/MM5WeKdvyH7/glTLcTBs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oJqbPr+okVXndDrRx0wc0NynxAK9Hf7ZjlKzaD06dTFia2tJP1ubqqMKadBBiKRfP 6rfoH36bWO1DDRyyn1fiBpcJunXb1MwxA3cV6mBIRxatgkI/rxbzUJ1Dpjpx7u9Iiy Z/qR+2vcKy45Fy/FCV1qJClvrKfvUyfL1XT/RcK4Rlw2BOjsGxqVXt1mr1fp0ouoo+ cyyz06PZ7KjxqF76LJgYR0Mq/8ocBsNefCEgzwD9En2N7m/eMpqpW3pFV1GRyx2CMJ pTJQft2rsqLjh+YZWEKpYkavxgUMdlIULRSdp8yk6N6GWMssksYU6uBCBSACBM33jL 616iVHowj+ObA== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1umsxc-000000042oi-0m79; Fri, 15 Aug 2025 13:51:04 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH 08/11] scripts/sphinx-build-wrapper: restore SPHINXOPTS parsing Date: Fri, 15 Aug 2025 13:50:36 +0200 Message-ID: X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" While rewriting the wrapper, incidentally support for SPHINXOPTS was dropped. Restore it and better handle them. Signed-off-by: Mauro Carvalho Chehab --- Documentation/Makefile | 1 + scripts/sphinx-build-wrapper | 86 +++++++++++++++++++++++++++++------- 2 files changed, 70 insertions(+), 17 deletions(-) diff --git a/Documentation/Makefile b/Documentation/Makefile index 280728cf78b9..d2e626627ee6 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -19,6 +19,7 @@ endif =20 # You can set these variables from the command line. SPHINXBUILD =3D sphinx-build +SPHINXOPTS =3D SPHINXDIRS =3D . DOCS_THEME =3D DOCS_CSS =3D diff --git a/scripts/sphinx-build-wrapper b/scripts/sphinx-build-wrapper index 5c728956b53c..b9d40f4a6573 100755 --- a/scripts/sphinx-build-wrapper +++ b/scripts/sphinx-build-wrapper @@ -148,10 +148,10 @@ class SphinxBuilder: =20 return path =20 - def __init__(self, venv=3DNone, verbose=3DFalse): + def __init__(self, venv=3DNone, verbose=3DFalse, n_jobs=3DNone): """Initialize internal variables""" self.venv =3D venv - self.verbose =3D verbose + self.verbose =3D None =20 # Normal variables passed from Kernel's makefile self.kernelversion =3D os.environ.get("KERNELVERSION", "unknown") @@ -159,6 +159,34 @@ class SphinxBuilder: self.pdflatex =3D os.environ.get("PDFLATEX", "xelatex") self.latexopts =3D os.environ.get("LATEXOPTS", "-interaction=3Dbat= chmode -no-shell-escape") =20 + if not verbose: + verbose =3D bool(os.environ.get("KBUILD_VERBOSE", "") !=3D "") + + # Handle SPHINXOPTS evironment + sphinxopts =3D shlex.split(os.environ.get("SPHINXOPTS", "")) + + # As we handle number of jobs and quiet in separate, we need to pi= ck + # it the same way as sphinx-build would pick, so let's use argparse + # do to the right argument expansion + parser =3D argparse.ArgumentParser() + parser.add_argument('-j', '--jobs', type=3Dint) + parser.add_argument('-q', '--quiet', type=3Dint) + + # Other sphinx-build arguments go as-is, so place them + # at self.sphinxopts + sphinx_args, self.sphinxopts =3D parser.parse_known_args(sphinxopt= s) + if sphinx_args.quiet =3D=3D True: + self.verbose =3D False + + if sphinx_args.jobs: + self.n_jobs =3D sphinx_args.jobs + + # Command line arguments was passed, override SPHINXOPTS + if verbose is not None: + self.verbose =3D verbose + + self.n_jobs =3D n_jobs + # Source tree directory. This needs to be at os.environ, as # Sphinx extensions and media uAPI makefile needs it self.srctree =3D os.environ.get("srctree") @@ -199,7 +227,7 @@ class SphinxBuilder: else: sys.exit(f"Venv {venv} not found.") =20 - def run_sphinx(self, sphinx_build, sphinx_args, *args, **pwargs): + def run_sphinx(self, sphinx_build, build_args, *args, **pwargs): """ Executes sphinx-build using current python3 command and setting -j parameter if possible to run the build in parallel. @@ -207,11 +235,9 @@ class SphinxBuilder: =20 with JobserverExec() as jobserver: if jobserver.claim: - parallelism =3D str(jobserver.claim) + n_jobs =3D str(jobserver.claim) else: - # As Sphinx has parallelism since version 1.7, we don't ne= ed - # any check here. - parallelism =3D "auto" + n_jobs =3D "auto" # Supported since Sphinx 1.7 =20 cmd =3D [] =20 @@ -222,10 +248,19 @@ class SphinxBuilder: =20 cmd.append(sphinx_build) =20 - if parallelism: - cmd.append("-j" + parallelism) + # if present, SPHINXOPTS or command line --jobs overrides defa= ult + if self.n_jobs: + n_jobs =3D str(self.n_jobs) =20 - cmd +=3D sphinx_args + if n_jobs: + cmd +=3D [f"-j{n_jobs}"] + + if not self.verbose: + cmd.append("-q") + + cmd +=3D self.sphinxopts + + cmd +=3D build_args =20 if self.verbose: print(" ".join(cmd)) @@ -447,9 +482,6 @@ class SphinxBuilder: =20 args.extend(["-D", f"latex_elements.papersize=3D{paper}paper"]) =20 - if not self.verbose: - args.append("-q") - if self.config_rust: args.extend(["-t", "rustdoc"]) =20 @@ -582,6 +614,25 @@ class SphinxBuilder: except OSError as e: sys.exit(f"Failed to restart with {new_python_cmd}: {e}") =20 +def jobs_type(value): + """ + Handle valid values for -j. Accepts Sphinx "-jauto", plus a number + equal or bigger than one. + """ + if value is None: + return None + + if value.lower() =3D=3D 'auto': + return value.lower() + + try: + if int(value) >=3D 1: + return value + + raise argparse.ArgumentTypeError(f"Minimum jobs is 1, got {value}") + except ValueError: + raise argparse.ArgumentTypeError(f"Must be 'auto' or positive inte= ger, got {value}") + def main(): """ Main function. The only mandatory argument is the target. If not @@ -607,18 +658,19 @@ def main(): parser.add_argument("-v", "--verbose", action=3D'store_true', help=3D"place build in verbose mode") =20 + parser.add_argument('-j', '--jobs', type=3Djobs_type, + help=3D"Sets number of jobs to use with sphinx-bui= ld") + parser.add_argument("-V", "--venv", nargs=3D'?', const=3Df'{VENV_DEFAU= LT}', default=3DNone, help=3Df'If used, run Sphinx from a venv dir (defa= ult dir: {VENV_DEFAULT})') =20 args =3D parser.parse_args() =20 - if not args.verbose: - args.verbose =3D bool(os.environ.get("KBUILD_VERBOSE", "") !=3D "") - SphinxBuilder.check_python() =20 - builder =3D SphinxBuilder(venv=3Dargs.venv, verbose=3Dargs.verbose) + builder =3D SphinxBuilder(venv=3Dargs.venv, verbose=3Dargs.verbose, + n_jobs=3Dargs.jobs) =20 builder.build(args.target, sphinxdirs=3Dargs.sphinxdirs, conf=3Dargs.c= onf, theme=3Dargs.theme, css=3Dargs.css, paper=3Dargs.paper) --=20 2.50.1 From nobody Sat Oct 4 14:32:50 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7DDAA3002DC; Fri, 15 Aug 2025 11:51:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; cv=none; b=IDYbaNzmPEmUJ+rQfqebXEPb5vN7mBm/7+7iJk21UVm7tIxFojlILQXrgIkA86larIK9NKExJlRhWxDl1IZYNr3+e77dCirQXoZq+g1Fox/ITcWADKFoO5JK9Fxe7wxD/bGiRZyJ/vgxcwBubA37ond60DfXw42TzubYkyhYt2U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; c=relaxed/simple; bh=muQZDm3HB4OxC2K9pGxjzcvqGF5LqYzQdGjLcQnqVE8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Tw0zQ07HU/e2J52o2wwGG9B6fbvOOj1pILFpYcJDV2y9B0yjw2v8W4Ff3v32nt1D15lIOdp5w1G++L2+lWz4rYi3ZLC3n8rzY0xqvRiarN2AOSR6l73Xa4PXCSz5VEb359cO+bAtLvNHAw0jjGC2CLgwT50U8vt40LETwfj9reQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DzmeQFUr; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DzmeQFUr" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F34D9C4CEFE; Fri, 15 Aug 2025 11:51:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755258666; bh=muQZDm3HB4OxC2K9pGxjzcvqGF5LqYzQdGjLcQnqVE8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DzmeQFUrnWmwtmy/iTqbo6QyGc+nSn/XEaDytqBMN+Eb9pIU4Kos5ev6IWO5U+8rp rH3DFry4IizN5Ql75FnNiTcY8DwXr33T5IOtGB3M/u9TnTWt1xrZNre8b8vQCshqw0 9zapCDw5nHc94o0DAJjuIzdUELPEXnAvQ8rZ3IUCfcxik8+89O82rJjE8NVk7M73nG 2rXcg9OTKUIVUEfV0YQ4ekpwRLpT0aer9z4l78zSlRAdkywdISA2iMFsG2uBd/IK63 7y+Dp1GHExhRofcOH+JZ5LO0XUGHgJyJ3a4hnUFT8mMUNglgPWHuLczemM2NOmITf8 gwqsllBoadvSQ== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1umsxc-000000042om-0tB6; Fri, 15 Aug 2025 13:51:04 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH 09/11] scripts: sphinx-build-wrapper: add an argument for LaTeX interactive mode Date: Fri, 15 Aug 2025 13:50:37 +0200 Message-ID: <6ad508ef010240665c8a8ce98c8ffe38a7be67fe.1755258303.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" By default, we use LaTeX batch mode to build docs. This way, when an error happens, the build fails. This is good for normal builds, but when debugging problems with pdf generation, the best is to use interactive mode. We already support it via LATEXOPTS, but having a command line argument makes it easier: Interactive mode: ./scripts/sphinx-build-wrapper pdfdocs --sphinxdirs peci -v -i ... Running 'xelatex --no-pdf -no-pdf -recorder ".../Documentation/output/pe= ci/latex/peci.tex"' ... Default batch mode: ./scripts/sphinx-build-wrapper pdfdocs --sphinxdirs peci -v ... Running 'xelatex --no-pdf -no-pdf -interaction=3Dbatchmode -no-shell-esca= pe -recorder ".../Documentation/output/peci/latex/peci.tex"' ... Signed-off-by: Mauro Carvalho Chehab --- scripts/sphinx-build-wrapper | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/scripts/sphinx-build-wrapper b/scripts/sphinx-build-wrapper index b9d40f4a6573..f6fec766c3e6 100755 --- a/scripts/sphinx-build-wrapper +++ b/scripts/sphinx-build-wrapper @@ -148,7 +148,7 @@ class SphinxBuilder: =20 return path =20 - def __init__(self, venv=3DNone, verbose=3DFalse, n_jobs=3DNone): + def __init__(self, venv=3DNone, verbose=3DFalse, n_jobs=3DNone, intera= ctive=3DNone): """Initialize internal variables""" self.venv =3D venv self.verbose =3D None @@ -157,7 +157,11 @@ class SphinxBuilder: self.kernelversion =3D os.environ.get("KERNELVERSION", "unknown") self.kernelrelease =3D os.environ.get("KERNELRELEASE", "unknown") self.pdflatex =3D os.environ.get("PDFLATEX", "xelatex") - self.latexopts =3D os.environ.get("LATEXOPTS", "-interaction=3Dbat= chmode -no-shell-escape") + + if not interactive: + self.latexopts =3D os.environ.get("LATEXOPTS", "-interaction= =3Dbatchmode -no-shell-escape") + else: + self.latexopts =3D os.environ.get("LATEXOPTS", "") =20 if not verbose: verbose =3D bool(os.environ.get("KBUILD_VERBOSE", "") !=3D "") @@ -661,6 +665,9 @@ def main(): parser.add_argument('-j', '--jobs', type=3Djobs_type, help=3D"Sets number of jobs to use with sphinx-bui= ld") =20 + parser.add_argument('-i', '--interactive', action=3D'store_true', + help=3D"Change latex default to run in interactive= mode") + parser.add_argument("-V", "--venv", nargs=3D'?', const=3Df'{VENV_DEFAU= LT}', default=3DNone, help=3Df'If used, run Sphinx from a venv dir (defa= ult dir: {VENV_DEFAULT})') @@ -670,7 +677,7 @@ def main(): SphinxBuilder.check_python() =20 builder =3D SphinxBuilder(venv=3Dargs.venv, verbose=3Dargs.verbose, - n_jobs=3Dargs.jobs) + n_jobs=3Dargs.jobs, interactive=3Dargs.interac= tive) =20 builder.build(args.target, sphinxdirs=3Dargs.sphinxdirs, conf=3Dargs.c= onf, theme=3Dargs.theme, css=3Dargs.css, paper=3Dargs.paper) --=20 2.50.1 From nobody Sat Oct 4 14:32:50 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4F50A2D7389; Fri, 15 Aug 2025 11:51:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; cv=none; b=ROlVZNsmqE/4vpS2kbUhgLJNQsXpSBP4H8hwGvV8og/epzwdaNxKimefu4VPRy9FNMKTfiKBuQ0hKAgtaiVdgqj41tGkccwkZOKHjQbIwpwp7arY0Wfd9aDD+WrBhA76ciwwcY4GWosme2XwN6Wc+xoT+Khl5LBhzcfFOOgVyw0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; c=relaxed/simple; bh=gQ6kk5WPc72Dq2AQd/n6nSscjReWsPfB0/s1IlVKE+c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NeL2IGo6R9Xgrvzo1RMsY3+Ps4jpqbA11WZ95uwakfhwgVD+mf6tGfPvAXw/l8mB67dS3ns7ccYXhNlv/lSqFhbWLNnhW7QpFsjgzPtVXhGi0cJLIDC2iQzOBE3xnzLo4VaC712uqSiddgEwiyF+Ao2maVGWpntr174l/uhwvsw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=h+M+Tq/d; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="h+M+Tq/d" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F193AC113CF; Fri, 15 Aug 2025 11:51:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755258666; bh=gQ6kk5WPc72Dq2AQd/n6nSscjReWsPfB0/s1IlVKE+c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=h+M+Tq/d+hBeMlPVTjHZo2vkzeO1qLH30Cde8nS9nSbkTp19fczQNfeHW2yUt503f 2aI7K0JnvENo9IK8lO9CY990Erpn8d0X4FcIzKy7Wl5oO0iB3JgFLl/lsoE03bUAi2 EBPYYo+YR6jifsd6/lqxUx4LjYjA+vTWkWF/SRtGOtPtByWUNPWsI586XoLmAkTCEg PJOXbsHFgvSuwx4gBO/cB524XswwN5Pv5Qr9JI6vpJZVwXfawEyBOIuYlRRN19cOPL LL7vkfgSeB42lDhXLBysGkJWdM+wpCRfjwC26z8kn9EKvYHmYJS/8xFB27Qnc+Fk0O z8F5YOQadHx1w== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1umsxc-000000042oq-0zxq; Fri, 15 Aug 2025 13:51:04 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH 10/11] scripts: sphinx-*: prevent sphinx-build crashes Date: Fri, 15 Aug 2025 13:50:38 +0200 Message-ID: <461e00fe7ce75eaac90d98572bb93910b39361e2.1755258303.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" On a properly set system, LANG and LC_ALL is always defined. However, some distros like Debian, Gentoo and their variants start with those undefioned. When Sphinx tries to set a locale with: locale.setlocale(locale.LC_ALL, '') It raises an exception, making Sphinx fail. This is more likely to happen with test containers. Add a logic to detect and workaround such issue by setting locale to C. Signed-off-by: Mauro Carvalho Chehab --- scripts/sphinx-build-wrapper | 10 ++++++++++ scripts/sphinx-pre-install | 14 +++++++++++++- 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/scripts/sphinx-build-wrapper b/scripts/sphinx-build-wrapper index f6fec766c3e6..f21701d34552 100755 --- a/scripts/sphinx-build-wrapper +++ b/scripts/sphinx-build-wrapper @@ -45,6 +45,7 @@ the newer version. """ =20 import argparse +import locale import os import re import shlex @@ -495,6 +496,15 @@ class SphinxBuilder: if not sphinxdirs: sphinxdirs =3D os.environ.get("SPHINXDIRS", ".") =20 + # The sphinx-build tool has a bug: internally, it tries to set + # locale with locale.setlocale(locale.LC_ALL, ''). This causes a + # crash if language is not set. Detect and fix it. + try: + locale.setlocale(locale.LC_ALL, '') + except Exception: + self.env["LC_ALL"] =3D "C" + self.env["LANG"] =3D "C" + # sphinxdirs can be a list or a whitespace-separated string sphinxdirs_list =3D [] for sphinxdir in sphinxdirs: diff --git a/scripts/sphinx-pre-install b/scripts/sphinx-pre-install index c46d7b76f93c..50ae9cfcb3d7 100755 --- a/scripts/sphinx-pre-install +++ b/scripts/sphinx-pre-install @@ -26,6 +26,7 @@ system pacage install is recommended. """ =20 import argparse +import locale import os import re import subprocess @@ -516,8 +517,19 @@ class MissingCheckers(AncillaryMethods): """ Gets sphinx-build version. """ + env =3D os.environ.copy() + + # The sphinx-build tool has a bug: internally, it tries to set + # locale with locale.setlocale(locale.LC_ALL, ''). This causes a + # crash if language is not set. Detect and fix it. try: - result =3D self.run([cmd, "--version"], + locale.setlocale(locale.LC_ALL, '') + except Exception: + env["LC_ALL"] =3D "C" + env["LANG"] =3D "C" + + try: + result =3D self.run([cmd, "--version"], env=3Denv, stdout=3Dsubprocess.PIPE, stderr=3Dsubprocess.STDOUT, text=3DTrue, check=3DTrue) --=20 2.50.1 From nobody Sat Oct 4 14:32:50 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 648212FE06D; Fri, 15 Aug 2025 11:51:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; cv=none; b=IfI1FAytyzqI66T4KPMJ/PlOl6tgun/90apq0MAIMLnZiseTol/ZB0bjLDcnuwhzidsHi0FV94+9K4KWDf7nSOhswNmFfhPQWs9nd0ieXXgTjjGV5Grbm7TnWYJWcvSI8DiOWYK6jxiyEQWi22KwiKLUtoj3Y/mteOuftLaEcxs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755258666; c=relaxed/simple; bh=M/Xid9UCECaZaplNDPUeVWw4gAtONrcmsCGcddE9UyY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=R8/uXIT7j8qfh7mSOlwlrf+kyx0+7qxNUrl5n4NhmUS37G4cUCoz2cNaGmCaAYVG3lwrghuLndVppYEA3HXqzUWmyGhDpD9t4WRyEwDBShWkJ1GF/ar7MMD2mKxGL+9zNLf84ugDJ3FtQtFBBdoy+3DjxekpWy9uvxsSk1ZV/10= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=acIb1pi5; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="acIb1pi5" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 11480C16AAE; Fri, 15 Aug 2025 11:51:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755258666; bh=M/Xid9UCECaZaplNDPUeVWw4gAtONrcmsCGcddE9UyY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=acIb1pi5M02DmAniDHj6wI5v+F8YZtGF27mThCk+gdRjQXJmDJ3fpEDtnU/PTL17c xPlnDdYzmF3/GJsRpXOXQkqI453QHS2giSd8qgZT7eT65KBkSY061N13AzJjgnlnB5 i4RB2PnJI0UqJY/LIXE+khYYrSYq1KM46iDjhe+jCgrUmwUPtM2gOfeEE0I2d/OYZI GDI7XaXyAZEzcr4QxvNl4/csxoC+9H+xqpTG18oBRt1vb/in0r5aP3oCjhVh4PoESA YKBY3KEBrLOxx6/Ix/OZcfubsrfkcjXXs1aGj4XsxtomaA7ooYstZ4Je5sy4YaJhC6 VQfdBKooD3SsA== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1umsxc-000000042ou-16aQ; Fri, 15 Aug 2025 13:51:04 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH 11/11] docs: Makefile: cleanup the logic by using sphinx-build-wrapper Date: Fri, 15 Aug 2025 13:50:39 +0200 Message-ID: <1dbef234504de4a661baaebcc79c9ae3accdb23f.1755258303.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Now that we have a sphinx-build-wrapper capable of handling all the needed step to build the supported build targets, cleanup the Makefile. Signed-off-by: Mauro Carvalho Chehab --- Documentation/Makefile | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/Documentation/Makefile b/Documentation/Makefile index d2e626627ee6..5a3dc9e5b578 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -19,11 +19,9 @@ endif =20 # You can set these variables from the command line. SPHINXBUILD =3D sphinx-build -SPHINXOPTS =3D SPHINXDIRS =3D . DOCS_THEME =3D DOCS_CSS =3D -_SPHINXDIRS =3D $(sort $(patsubst $(srctree)/Documentation/%/index.rst,%= ,$(wildcard $(srctree)/Documentation/*/index.rst))) SPHINX_CONF =3D conf.py PAPER =3D BUILDDIR =3D $(obj)/output @@ -33,6 +31,12 @@ BUILD_WRAPPER =3D $(srctree)/scripts/sphinx-build-wrapper =20 PYTHONPYCACHEPREFIX ?=3D $(abspath $(BUILDDIR)/__pycache__) =20 +PYTHONPYCACHEPREFIX ?=3D $(abspath $(BUILDDIR)/__pycache__) + +# Wrapper for sphinx-build + +BUILD_WRAPPER =3D $(srctree)/scripts/sphinx-build-wrapper + # For denylisting "variable font" files # Can be overridden by setting as an env variable FONTS_CONF_DENY_VF ?=3D $(HOME)/deny-vf @@ -103,7 +107,9 @@ refcheckdocs: =20 cleandocs: $(Q)rm -rf $(BUILDDIR) - $(Q)$(MAKE) BUILDDIR=3D$(abspath $(BUILDDIR)) $(build)=3DDocumentation/us= erspace-api/media clean + +# Used only on help +_SPHINXDIRS =3D $(sort $(patsubst $(srctree)/Documentation/%/index.rst,%= ,$(wildcard $(srctree)/Documentation/*/index.rst))) =20 dochelp: @echo ' Linux kernel internal documentation in different formats from Re= ST:' --=20 2.50.1