From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6022730748F; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=T8M6l2Xxtlsyc+T5UykSbXuQmjh5APYJgdUM+IuU4pwmZRhFxEbuoQZDO88/wOFa5DLCOXK59Dxf26oRSixCwcDbTNn6c22tkIHRGzmfAOCbkQ4n3KQTHYhxciwu6iQAr/gSk681jLeB3YJWNghLkRP/Uhqge5pfwoscc2ymAIk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=XDxTPT2c6W8eHR7BT6Ec6vb15gsGsFJ8CPPzepOj7LY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DtyvvABDfvBYhTjjeGdQBHp0LVPNXSTD16dmDWwl9+xGCbBstgEJqZ5bryADrywO/7kNJKQZl1yLIeZOkWTH8bCwUbw+1MvAGgKKR6Kh5hzIk++96gq1e6IkcDORL1Lv4fyRUWIQYYv+LSZtVUdFeeP3O9IIpfZCKP/HQz/yg2Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=iKa5TAxQ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="iKa5TAxQ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 08C15C4CEED; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=XDxTPT2c6W8eHR7BT6Ec6vb15gsGsFJ8CPPzepOj7LY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=iKa5TAxQ4kuKEqZ7waaDQnnY0ZlPBzABftighqhaZ7U19xSAhr7AvsZrOv404D8i/ lMgNFTcTSllxoFAkeyuoS6JcnJnIpQ8/F2sD3wyy2AYvmk9fNz7D/6jIi61z82KugT zm0fOLHPsYsU89KdAn4/bjy91vYmmiV0KERJMaI5aB1spxet6cDQsxUB4pqQiFdIpM FZtm/+jN4U446VHNm1532YE+XIvQVnAtyWRcA9ipphxm4AsNceG65FMNblBLYEnvJa T6nAffR3+TZQW+Wifq63UJ0pvZlzu7DArK7lg31miyWu7F/ESyfmN/XzbtMq83CRnt qBXnnRjKfU9eg== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCqv-0YfL; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v2 01/24] docs: parse-headers.pl: improve its debug output format Date: Fri, 22 Aug 2025 16:19:13 +0200 Message-ID: <1064011717951eac257889a3032303c9d4440711.1755872208.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Change the --debug logic to help comparing its results with a new python script. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/parse-headers.pl | 31 ++++++++++++++++++++------- 1 file changed, 23 insertions(+), 8 deletions(-) diff --git a/Documentation/sphinx/parse-headers.pl b/Documentation/sphinx/p= arse-headers.pl index 7b1458544e2e..560685926cdb 100755 --- a/Documentation/sphinx/parse-headers.pl +++ b/Documentation/sphinx/parse-headers.pl @@ -31,8 +31,6 @@ my %enums; my %enum_symbols; my %structs; =20 -require Data::Dumper if ($debug); - # # read the file and get identifiers # @@ -197,6 +195,9 @@ if ($file_exceptions) { } else { $reftype =3D $def_reftype{$type}; } + if (!$reftype) { + print STDERR "Warning: can't find ref type for $type"; + } $new =3D "$reftype:`$old <$new>`"; =20 if ($type eq "ioctl") { @@ -229,12 +230,26 @@ if ($file_exceptions) { } =20 if ($debug) { - print Data::Dumper->Dump([\%ioctls], [qw(*ioctls)]) if (%ioctls); - print Data::Dumper->Dump([\%typedefs], [qw(*typedefs)]) if (%typedefs); - print Data::Dumper->Dump([\%enums], [qw(*enums)]) if (%enums); - print Data::Dumper->Dump([\%structs], [qw(*structs)]) if (%structs); - print Data::Dumper->Dump([\%defines], [qw(*defines)]) if (%defines); - print Data::Dumper->Dump([\%enum_symbols], [qw(*enum_symbols)]) if (%enum= _symbols); + my @all_hashes =3D ( + {ioctl =3D> \%ioctls}, + {typedef =3D> \%typedefs}, + {enum =3D> \%enums}, + {struct =3D> \%structs}, + {define =3D> \%defines}, + {symbol =3D> \%enum_symbols} + ); + + foreach my $hash (@all_hashes) { + while (my ($name, $hash_ref) =3D each %$hash) { + next unless %$hash_ref; # Skip empty hashes + + print "$name:\n"; + for my $key (sort keys %$hash_ref) { + print " $key -> $hash_ref->{$key}\n"; + } + print "\n"; + } + } } =20 # --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CCCB03126B3; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=rnvaPTWD+/BuwC3CSM/EbEa7HT2vPI15bCatnQ/v013wzq1ZfNXzYJKTzvGJkGzlVrzGRFTnMKrG80axbcDyVaV1abp3FDuhdC1iQBsN8nte6hThjo+cQIgxnsTKsMAP2y4c3zyCCZl7pomE7juty8pNY0UFwpI3PMXjQcfZK38= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=IkR2tDIMqOGFWIt+/VpJUQvrkHkeqcw1eRxVumPctes=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=t+cVsIxFV34BQE0+Rxrt2sRPXtEGzhbhIYxdQhlVP20an4xne0zC/zoAG1nx7Xvpr3Xjvm0qbVAG+kIkFSbKYyJWwUGLDqXWNm4fN0SbcVWeqYRVtYU1LoXg3zkpzZ97cFGATZF1lnh+j9mCEEjKI1pHaAe6Vng+1e4wwoiaV+s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oU65SJP/; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oU65SJP/" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 10E97C116D0; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=IkR2tDIMqOGFWIt+/VpJUQvrkHkeqcw1eRxVumPctes=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oU65SJP/+jTEHTdqTyCfM+UFLSFdFufzlQG6yrIQibvMjeyaPCDZMHGhNUprTFotx hHNAGLEW3JhEzU/91ob0zp9BVM4ppQav7TMPy5ktxJZLEvz2xePoGeFyx7rhlehf3A SZq4/NQm4sFnCj3zJH64K8BkFWdv0n9f2W6bwlI5InXEjLNyQAKsrpBNreIK8RAvPZ SnxY8VlJDnE7fIAR29afATlqu9Nw1H30hSznzWEzYg/mhdgtE5QF7MAZrl3Ax4EAMV CkUSYjfgJFkiOaVq+UZQCKOIIyUfSErtMC9ENCE5fzHd3TzH3Ht4YawQbDAkcrTlyq rA5pK9xaIXJVQ== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCqz-0fiZ; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v2 02/24] docs: parse-headers.py: convert parse-headers.pl Date: Fri, 22 Aug 2025 16:19:14 +0200 Message-ID: X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" When the Kernel started to use Sphinx, we had to come up with a solution to parse media headers. On that time, we didn't have much experience with Sphinx extensions. So, we came up with our own script-based solution that were basically implementing a set of rules we used to have at the Makefile. Convert it to Python, keeping it bug-compatible with the original script. While here, try to better document it. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/parse-headers.py | 429 ++++++++++++++++++++++++++ 1 file changed, 429 insertions(+) create mode 100755 Documentation/sphinx/parse-headers.py diff --git a/Documentation/sphinx/parse-headers.py b/Documentation/sphinx/p= arse-headers.py new file mode 100755 index 000000000000..b39284d21090 --- /dev/null +++ b/Documentation/sphinx/parse-headers.py @@ -0,0 +1,429 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2016 by Mauro Carvalho Chehab . +# pylint: disable=3DC0103,R0902,R0912,R0914,R0915 + +""" +Convert a C header or source file (C_FILE), into a ReStructured Text +included via ..parsed-literal block with cross-references for the +documentation files that describe the API. It accepts an optional +EXCEPTIONS_FILE with describes what elements will be either ignored or +be pointed to a non-default reference. + +The output is written at the (OUT_FILE). + +It is capable of identifying defines, functions, structs, typedefs, +enums and enum symbols and create cross-references for all of them. +It is also capable of distinguish #define used for specifying a Linux +ioctl. + +The EXCEPTIONS_FILE contains a set of rules like: + + ignore ioctl VIDIOC_ENUM_FMT + replace ioctl VIDIOC_DQBUF vidioc_qbuf + replace define V4L2_EVENT_MD_FL_HAVE_FRAME_SEQ :c:type:`v4l2_event_mot= ion_det` +""" + +import argparse +import os +import re +import sys + + +class ParseHeader: + """ + Creates an enriched version of a Kernel header file with cross-links + to each C data structure type. + + It is meant to allow having a more comprehensive documentation, where + uAPI headers will create cross-reference links to the code. + + It is capable of identifying defines, functions, structs, typedefs, + enums and enum symbols and create cross-references for all of them. + It is also capable of distinguish #define used for specifying a Linux + ioctl. + + By default, it create rules for all symbols and defines, but it also + allows parsing an exception file. Such file contains a set of rules + using the syntax below: + + 1. Ignore rules: + + ignore ` + + Removes the symbol from reference generation. + + 2. Replace rules: + + replace + + Replaces how old_symbol with a new reference. The new_reference can be: + - A simple symbol name; + - A full Sphinx reference. + + On both cases, can be: + - ioctl: for defines that end with _IO*, e.g. ioctl definitions + - define: for other defines + - symbol: for symbols defined within enums; + - typedef: for typedefs; + - enum: for the name of a non-anonymous enum; + - struct: for structs. + + Examples: + + ignore define __LINUX_MEDIA_H + ignore ioctl VIDIOC_ENUM_FMT + replace ioctl VIDIOC_DQBUF vidioc_qbuf + replace define V4L2_EVENT_MD_FL_HAVE_FRAME_SEQ :c:type:`v4l2_event= _motion_det` + """ + + # Parser regexes with multiple ways to capture enums and structs + RE_ENUMS =3D [ + re.compile(r"^\s*enum\s+([\w_]+)\s*\{"), + re.compile(r"^\s*enum\s+([\w_]+)\s*$"), + re.compile(r"^\s*typedef\s*enum\s+([\w_]+)\s*\{"), + re.compile(r"^\s*typedef\s*enum\s+([\w_]+)\s*$"), + ] + RE_STRUCTS =3D [ + re.compile(r"^\s*struct\s+([_\w][\w\d_]+)\s*\{"), + re.compile(r"^\s*struct\s+([_\w][\w\d_]+)$"), + re.compile(r"^\s*typedef\s*struct\s+([_\w][\w\d_]+)\s*\{"), + re.compile(r"^\s*typedef\s*struct\s+([_\w][\w\d_]+)$"), + ] + + # FIXME: the original code was written a long time before Sphinx C + # domain to have multiple namespaces. To avoid to much turn at the + # existing hyperlinks, the code kept using "c:type" instead of the + # right types. To change that, we need to change the types not only + # here, but also at the uAPI media documentation. + DEF_SYMBOL_TYPES =3D { + "ioctl": { + "prefix": "\\ ", + "suffix": "\\ ", + "ref_type": ":ref", + }, + "define": { + "prefix": "\\ ", + "suffix": "\\ ", + "ref_type": ":ref", + }, + # We're calling each definition inside an enum as "symbol" + "symbol": { + "prefix": "\\ ", + "suffix": "\\ ", + "ref_type": ":ref", + }, + "typedef": { + "prefix": "\\ ", + "suffix": "\\ ", + "ref_type": ":c:type", + }, + # This is the name of the enum itself + "enum": { + "prefix": "", + "suffix": "\\ ", + "ref_type": ":c:type", + }, + "struct": { + "prefix": "", + "suffix": "\\ ", + "ref_type": ":c:type", + }, + } + + def __init__(self, debug: bool =3D False): + """Initialize internal vars""" + self.debug =3D debug + self.data =3D "" + + self.symbols =3D {} + + for symbol_type in self.DEF_SYMBOL_TYPES: + self.symbols[symbol_type] =3D {} + + def store_type(self, symbol_type: str, symbol: str, + ref_name: str =3D None, replace_underscores: bool =3D T= rue): + """ + Stores a new symbol at self.symbols under symbol_type. + + By default, underscores are replaced by "-" + """ + defs =3D self.DEF_SYMBOL_TYPES[symbol_type] + + prefix =3D defs.get("prefix", "") + suffix =3D defs.get("suffix", "") + ref_type =3D defs.get("ref_type") + + # Determine ref_link based on symbol type + if ref_type: + if symbol_type =3D=3D "enum": + ref_link =3D f"{ref_type}:`{symbol}`" + else: + if not ref_name: + ref_name =3D symbol.lower() + + if replace_underscores: + ref_name =3D ref_name.replace("_", "-") + + ref_link =3D f"{ref_type}:`{symbol} <{ref_name}>`" + else: + ref_link =3D symbol + + self.symbols[symbol_type][symbol] =3D f"{prefix}{ref_link}{suffix}" + + def store_line(self, line): + """Stores a line at self.data, properly indented""" + line =3D " " + line.expandtabs() + self.data +=3D line.rstrip(" ") + + def parse_file(self, file_in: str): + """Reads a C source file and get identifiers""" + self.data =3D "" + is_enum =3D False + is_comment =3D False + multiline =3D "" + + with open(file_in, "r", + encoding=3D"utf-8", errors=3D"backslashreplace") as f: + for line_no, line in enumerate(f): + self.store_line(line) + line =3D line.strip("\n") + + # Handle continuation lines + if line.endswith(r"\\"): + multiline +=3D line[-1] + continue + + if multiline: + line =3D multiline + line + multiline =3D "" + + # Handle comments. They can be multilined + if not is_comment: + if re.search(r"/\*.*", line): + is_comment =3D True + else: + # Strip C99-style comments + line =3D re.sub(r"(//.*)", "", line) + + if is_comment: + if re.search(r".*\*/", line): + is_comment =3D False + else: + multiline =3D line + continue + + # At this point, line variable may be a multilined stateme= nt, + # if lines end with \ or if they have multi-line comments + # With that, it can safely remove the entire comments, + # and there's no need to use re.DOTALL for the logic below + + line =3D re.sub(r"(/\*.*\*/)", "", line) + if not line.strip(): + continue + + # It can be useful for debug purposes to print the file af= ter + # having comments stripped and multi-lines grouped. + if self.debug > 1: + print(f"line {line_no + 1}: {line}") + + # Now the fun begins: parse each type and store it. + + # We opted for a two parsing logic here due to: + # 1. it makes easier to debug issues not-parsed symbols; + # 2. we want symbol replacement at the entire content, not + # just when the symbol is detected. + + if is_enum: + match =3D re.match(r"^\s*([_\w][\w\d_]+)\s*[\,=3D]?", = line) + if match: + self.store_type("symbol", match.group(1)) + if "}" in line: + is_enum =3D False + continue + + match =3D re.match(r"^\s*#\s*define\s+([\w_]+)\s+_IO", lin= e) + if match: + self.store_type("ioctl", match.group(1), + replace_underscores=3DFalse) + continue + + match =3D re.match(r"^\s*#\s*define\s+([\w_]+)(\s+|$)", li= ne) + if match: + self.store_type("define", match.group(1)) + continue + + match =3D re.match(r"^\s*typedef\s+([_\w][\w\d_]+)\s+(.*)\= s+([_\w][\w\d_]+);", + line) + if match: + name =3D match.group(2).strip() + symbol =3D match.group(3) + self.store_type("typedef", symbol, ref_name=3Dname, + replace_underscores=3DFalse) + continue + + for re_enum in self.RE_ENUMS: + match =3D re_enum.match(line) + if match: + self.store_type("enum", match.group(1)) + is_enum =3D True + break + + for re_struct in self.RE_STRUCTS: + match =3D re_struct.match(line) + if match: + self.store_type("struct", match.group(1), + replace_underscores=3DFalse) + break + + def process_exceptions(self, fname: str): + """ + Process exceptions file with rules to ignore or replace references. + """ + if not fname: + return + + name =3D os.path.basename(fname) + + with open(fname, "r", encoding=3D"utf-8", errors=3D"backslashrepla= ce") as f: + for ln, line in enumerate(f): + ln +=3D 1 + line =3D line.strip() + if not line or line.startswith("#"): + continue + + # Handle ignore rules + match =3D re.match(r"^ignore\s+(\w+)\s+(\S+)", line) + if match: + c_type =3D match.group(1) + symbol =3D match.group(2) + + if c_type not in self.DEF_SYMBOL_TYPES: + sys.exit(f"{name}:{ln}: {c_type} is invalid") + + d =3D self.symbols[c_type] + if symbol in d: + del d[symbol] + + continue + + # Handle replace rules + match =3D re.match(r"^replace\s+(\S+)\s+(\S+)\s+(\S+)", li= ne) + if not match: + sys.exit(f"{name}:{ln}: invalid line: {line}") + + c_type, old, new =3D match.groups() + + if c_type not in self.DEF_SYMBOL_TYPES: + sys.exit(f"{name}:{ln}: {c_type} is invalid") + + reftype =3D None + + # Parse reference type when the type is specified + + match =3D re.match(r"^\:c\:(data|func|macro|type)\:\`(.+)\= `", new) + if match: + reftype =3D f":c:{match.group(1)}" + new =3D match.group(2) + else: + match =3D re.search(r"(\:ref)\:\`(.+)\`", new) + if match: + reftype =3D match.group(1) + new =3D match.group(2) + + # If the replacement rule doesn't have a type, get default + if not reftype: + reftype =3D self.DEF_SYMBOL_TYPES[c_type].get("ref_typ= e") + if not reftype: + reftype =3D self.DEF_SYMBOL_TYPES[c_type].get("rea= l_type") + + new_ref =3D f"{reftype}:`{old} <{new}>`" + + # Change self.symbols to use the replacement rule + if old in self.symbols[c_type]: + self.symbols[c_type][old] =3D new_ref + else: + print(f"{name}:{ln}: Warning: can't find {old} {c_type= }") + + def debug_print(self): + """ + Print debug information containing the replacement rules per symbo= l. + To make easier to check, group them per type. + """ + if not self.debug: + return + + for c_type, refs in self.symbols.items(): + if not refs: # Skip empty dictionaries + continue + + print(f"{c_type}:") + + for symbol, ref in sorted(refs.items()): + print(f" {symbol} -> {ref}") + + print() + + def write_output(self, file_in: str, file_out: str): + """Write the formatted output to a file.""" + + # Avoid extra blank lines + text =3D re.sub(r"\s+$", "", self.data) + "\n" + text =3D re.sub(r"\n\s+\n", "\n\n", text) + + # Escape Sphinx special characters + text =3D re.sub(r"([\_\`\*\<\>\&\\\\:\/\|\%\$\#\{\}\~\^])", r"\\\1= ", text) + + # Source uAPI files may have special notes. Use bold font for them + text =3D re.sub(r"DEPRECATED", "**DEPRECATED**", text) + + # Delimiters to catch the entire symbol after escaped + start_delim =3D r"([ \n\t\(=3D\*\@])" + end_delim =3D r"(\s|,|\\=3D|\\:|\;|\)|\}|\{)" + + # Process all reference types + for ref_dict in self.symbols.values(): + for symbol, replacement in ref_dict.items(): + symbol =3D re.escape(re.sub(r"([\_\`\*\<\>\&\\\\:\/])", r"= \\\1", symbol)) + text =3D re.sub(fr'{start_delim}{symbol}{end_delim}', + fr'\1{replacement}\2', text) + + # Remove "\ " where not needed: before spaces and at the end of li= nes + text =3D re.sub(r"\\ ([\n ])", r"\1", text) + + title =3D os.path.basename(file_in) + + with open(file_out, "w", encoding=3D"utf-8", errors=3D"backslashre= place") as f: + f.write(".. -*- coding: utf-8; mode: rst -*-\n\n") + f.write(f"{title}\n") + f.write("=3D" * len(title)) + f.write("\n\n.. parsed-literal::\n\n") + f.write(text) + + +def main(): + """Main function""" + parser =3D argparse.ArgumentParser(description=3D__doc__, + formatter_class=3Dargparse.RawDescrip= tionHelpFormatter) + + parser.add_argument("-d", "--debug", action=3D"count", default=3D0, + help=3D"Increase debug level. Can be used multiple= times") + parser.add_argument("file_in", help=3D"Input C file") + parser.add_argument("file_out", help=3D"Output RST file") + parser.add_argument("file_exceptions", nargs=3D"?", + help=3D"Exceptions file (optional)") + + args =3D parser.parse_args() + + parser =3D ParseHeader(debug=3Dargs.debug) + parser.parse_file(args.file_in) + + if args.file_exceptions: + parser.process_exceptions(args.file_exceptions) + + parser.debug_print() + parser.write_output(args.file_in, args.file_out) + + +if __name__ =3D=3D "__main__": + main() --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 602B0308F16; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=aTG0Mu8XNRb1RYKls8GIb2S/gu1p34zK1JOOAqemmPCqHkoAwYN1DbkkQZCj/IEEpf9AjOO6CHioskYllLnTz3nWyavfocSSH86HcW1U+XNKmIQeJY8HMJq1Y3xGDdOypTmPbJAhRPt4TALtrFXyaDsupRCLNXpFyuYIN8JgCw0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=/wRz/bm5opGP5gUAlYf6YzB0TWc+kftZ82g5lfHklv8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Xg/owHZs2cG89HPf8+2cvJdp2QdgoHZL/a0A+EaB4h409T49zrktdxwkV1Sk+n2ue7hr1vlCG79gSn5RdBhfULgKPEpBbKJw7u/nBfCzE4KM7ngh1qE/cYpkpSXkBnw6zGjzlWO5uVPUFcq86mSRWo1v3MmE7wWZyyjLzrtPidM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=YkCJONWv; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="YkCJONWv" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 139C1C116C6; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=/wRz/bm5opGP5gUAlYf6YzB0TWc+kftZ82g5lfHklv8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YkCJONWvnEedaT3kPPHzebvGJ3os/UfQ9WIWR9eJfOpKVgLKFAm5472X0mUGk/Wnj ZVmVnuAqBzcunlqKGkjrzsO+sZKjWJSRIy38oBqS0c5T/XUUxlVVy7t9oiEnt/gyI7 o5RvrqFvdu7EG1y/DscD/BOVutxM/hmx+nOYn7A8+lQJI+7ZP+VODgauwY3R5R94/j 3QyPYJqgBG4qRRekMmLJDQuKzqggPt9iXTjphEtfjjJbqCjZlkrrx+nkA1sqdsRiFI rFNHGc0AH0q1YRXuGOU+18sQCHGo7FMid1dAqYn+Zl+JG0uvSIpGVbd/Tep50p83Tq DzrHHr9RuH+nA== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCr3-0mR0; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v2 03/24] docs: parse-headers.py: improve --help logic Date: Fri, 22 Aug 2025 16:19:15 +0200 Message-ID: <2c1e61d1fb1b2a2838b443beee89c1528831997f.1755872208.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" When printing --help, we'd like the name of the files from __doc__ to match the displayed positional arguments at both usage and argument description lines. Use a custom formatter class to convert ``foo`` into ANSI SGR code to bold the argument, if is TTY, and adjust the help text to match the argument names. Here on Plasma, that makes it display it colored, wich is really cool. Yet, I opted for SGR, as the best is to follow the terminal color schema for bold. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/parse-headers.py | 67 +++++++++++++++++++++++---- 1 file changed, 58 insertions(+), 9 deletions(-) diff --git a/Documentation/sphinx/parse-headers.py b/Documentation/sphinx/p= arse-headers.py index b39284d21090..650f9c9a68d1 100755 --- a/Documentation/sphinx/parse-headers.py +++ b/Documentation/sphinx/parse-headers.py @@ -4,20 +4,20 @@ # pylint: disable=3DC0103,R0902,R0912,R0914,R0915 =20 """ -Convert a C header or source file (C_FILE), into a ReStructured Text +Convert a C header or source file ``FILE_IN``, into a ReStructured Text included via ..parsed-literal block with cross-references for the documentation files that describe the API. It accepts an optional -EXCEPTIONS_FILE with describes what elements will be either ignored or -be pointed to a non-default reference. +``FILE_RULES`` file to describes what elements will be either ignored or +be pointed to a non-default reference type/name. =20 -The output is written at the (OUT_FILE). +The output is written at ``FILE_OUT``. =20 It is capable of identifying defines, functions, structs, typedefs, enums and enum symbols and create cross-references for all of them. It is also capable of distinguish #define used for specifying a Linux ioctl. =20 -The EXCEPTIONS_FILE contains a set of rules like: +The optional ``FILE_RULES`` contains a set of rules like: =20 ignore ioctl VIDIOC_ENUM_FMT replace ioctl VIDIOC_DQBUF vidioc_qbuf @@ -400,17 +400,66 @@ class ParseHeader: f.write("\n\n.. parsed-literal::\n\n") f.write(text) =20 +class EnrichFormatter(argparse.HelpFormatter): + """ + Better format the output, making easier to identify the positional args + and how they're used at the __doc__ description. + """ + def __init__(self, *args, **kwargs): + """Initialize class and check if is TTY""" + super().__init__(*args, **kwargs) + self._tty =3D sys.stdout.isatty() + + def enrich_text(self, text): + """Handle ReST markups (currently, only ``foo``)""" + if self._tty and text: + # Replace ``text`` with ANSI bold + return re.sub(r'\`\`(.+?)\`\`', + lambda m: f'\033[1m{m.group(1)}\033[0m', text) + return text + + def _fill_text(self, text, width, indent): + """Enrich descriptions with markups on it""" + enriched =3D self.enrich_text(text) + return "\n".join(indent + line for line in enriched.splitlines()) + + def _format_usage(self, usage, actions, groups, prefix): + """Enrich positional arguments at usage: line""" + + prog =3D self._prog + parts =3D [] + + for action in actions: + if action.option_strings: + opt =3D action.option_strings[0] + if action.nargs !=3D 0: + opt +=3D f" {action.dest.upper()}" + parts.append(f"[{opt}]") + else: + # Positional argument + parts.append(self.enrich_text(f"``{action.dest.upper()}``"= )) + + usage_text =3D f"{prefix or 'usage: '} {prog} {' '.join(parts)}\n" + return usage_text + + def _format_action_invocation(self, action): + """Enrich argument names""" + if not action.option_strings: + return self.enrich_text(f"``{action.dest.upper()}``") + else: + return ", ".join(action.option_strings) + =20 def main(): """Main function""" parser =3D argparse.ArgumentParser(description=3D__doc__, - formatter_class=3Dargparse.RawDescrip= tionHelpFormatter) + formatter_class=3DEnrichFormatter) =20 parser.add_argument("-d", "--debug", action=3D"count", default=3D0, help=3D"Increase debug level. Can be used multiple= times") parser.add_argument("file_in", help=3D"Input C file") parser.add_argument("file_out", help=3D"Output RST file") - parser.add_argument("file_exceptions", nargs=3D"?", + parser.add_argument("file_rules", nargs=3D"?", help=3D"Exceptions file (optional)") =20 args =3D parser.parse_args() @@ -418,8 +467,8 @@ def main(): parser =3D ParseHeader(debug=3Dargs.debug) parser.parse_file(args.file_in) =20 - if args.file_exceptions: - parser.process_exceptions(args.file_exceptions) + if args.file_rules: + parser.process_exceptions(args.file_rules) =20 parser.debug_print() parser.write_output(args.file_in, args.file_out) --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 902D53112C2; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=f7casIFIac2wlqB6Gmp8v1IX4e8C0DWcJ/xmOxiSF0HLm/RQzJI9yt58HZDdTaZrgr+HJaBG5t4MvzWuZVevgGaHGcPW96ZkZ+CF6PGJm0ZRe8yU1U66LRzb84i46RtUnLSeEY7iVEgArPqckaMfGkCQwpf4ojwigZXbWXWPrcU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=HiN7NqutO6hirOm4EnCnsuZhXt+MMY3adGpEmCAC7NY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=tSi0TIbIGwmTxrdwYGOAH7sTmEINx6BhfCZa8T4bSehXsbb9yBcdoDpo6No+4slYxcoartzNPnFLhuh+YNaoyNf8asP8gXXY1GOMS0cDeiUFB/YkaJkg/DGyYr5L98NMdGfQcgTGNf8fWXjFyS8KKweM4QoahKKYOWNMlEwDzQY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oEEVSqQz; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oEEVSqQz" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1DC11C19423; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=HiN7NqutO6hirOm4EnCnsuZhXt+MMY3adGpEmCAC7NY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oEEVSqQz141kP3LMW9d5m0XBcOP2hUjvdJSzYCNaGBMzYEvn5sdUjpdzOy3bY6hyl HWeU8lxXxyzpZC9xCyN2r0YjI8NlVMRB3iSlmvlZPjNUJMspTE9QLLDzO5P3Hxdfoq C4dTq9+wYFjSGOMSvoW16chvgXjvYgnln5JBVEH7XYnNb6Z88REAikr/6eZo3KwDy0 EFjU0CUUGCN9zj0SVL9uOkg2WK5RrNXaLjfV7kFCSLAh1YAwhL4+cY2HLSchS8EJCb DsOsBye9c5xvkrH6B4SJZGtDJYBgOTKkSIJypNBxwv+h0VowmDu1vVzx/55CpVxNfe rebz35QLGpnjg== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCr7-0t4P; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v2 04/24] docs: parse-headers.py: better handle @var arguments Date: Fri, 22 Aug 2025 16:19:16 +0200 Message-ID: <8d06bb713d6ec8de65179dd93defe479715409b6.1755872208.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" The kernel-doc markups inside headers may contain @var markups. With the current rule, this would be converted into: \* @:c:type:`DMX_BUFFER_FLAG_DISCONTINUITY_INDICATOR `\: Fix it adding a non-printed space if needed. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/parse-headers.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/Documentation/sphinx/parse-headers.py b/Documentation/sphinx/p= arse-headers.py index 650f9c9a68d1..f4ab9c49d2f5 100755 --- a/Documentation/sphinx/parse-headers.py +++ b/Documentation/sphinx/parse-headers.py @@ -120,12 +120,12 @@ class ParseHeader: }, # This is the name of the enum itself "enum": { - "prefix": "", + "prefix": "\\ ", "suffix": "\\ ", "ref_type": ":c:type", }, "struct": { - "prefix": "", + "prefix": "\\ ", "suffix": "\\ ", "ref_type": ":c:type", }, @@ -390,6 +390,8 @@ class ParseHeader: =20 # Remove "\ " where not needed: before spaces and at the end of li= nes text =3D re.sub(r"\\ ([\n ])", r"\1", text) + text =3D re.sub(r" \\ ", " ", text) + =20 title =3D os.path.basename(file_in) =20 --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6013A2FE564; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=CjF1jBPR9640pDqwZL5jY+oMZgPGimGqPRmCTKAJQYOp6cLpRsdxAPXCkK9GIgB7BEaXoroJkt3SGpqKlnAHyXydCk/yCTQBvK7h/LXgJHFKUOgu1UoH/t3OCXfxTcA3VUhj2d0JBssbkNZZKLG2Q3qLiJmS3dCo7VWLvfHlUrs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=xaVYPkTp3UCaFGLlpkZHV5Y3yGQI0qRfBiGX9ydW30c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Ltm+hbG7NGxuUVfp/FSKpO4/5G4sJNkoiqBx1Oh9lpZfUdg05iX0clmP7K9sfBHyWG05ckSfsSB/gtHeQr9kfUBK8c+E/irplrPN4hkHhYDjmuF8Pt9bl++0q7kjYtKkfoJsr3iN4ODCZEB/F+1NOA/7D4JdvrAaBc3UudWDa/o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ZPDDhWkt; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ZPDDhWkt" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1B03BC19421; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=xaVYPkTp3UCaFGLlpkZHV5Y3yGQI0qRfBiGX9ydW30c=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ZPDDhWktb4G+LquyVzJFUAueM1W0FMp3s2jOTA2CUSJFGdOfKE8/6iy//dlqvoTLh WrpIrRbxqnfj9gntTOgJsx9gaI12gZDTYdEp+zlSvzhZeeGfqHfkKnVPSCrqyIO2YA nsfBrEdMrxFuNP1/5f6TLeOIAB/Wd5DrSsx/d+VumIKqJvTpOWf4zPoIGuA4bJuEZ0 DyNHP7+WFX0Q5FFPdiNnTcniubti+Oys12ag3XfaqzIjBarRXBPhXAe3MNjUv8JZJM XZat/DItWzK1WZ/3RgOfIHYVk52EAkUOndsWLX5VcjCofKU//QnxrJxdgr1O/Yshp/ U5GpqeGd7vpLg== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCrB-0zmo; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v2 05/24] docs: parse-headers.py: simplify the rules for hashes Date: Fri, 22 Aug 2025 16:19:17 +0200 Message-ID: <0c011090272f7a1068545409222f970ddb1ed431.1755872208.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Normal :ref domain accept either hashes or underscores, but c-domain ones don't. Fix it and remove unneeded places where we opt to disable underscore transformation. Ideally, we should have a rule about the default, or change the way media docs have their references. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/parse-headers.py | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/Documentation/sphinx/parse-headers.py b/Documentation/sphinx/p= arse-headers.py index f4ab9c49d2f5..344090ef259c 100755 --- a/Documentation/sphinx/parse-headers.py +++ b/Documentation/sphinx/parse-headers.py @@ -162,7 +162,8 @@ class ParseHeader: if not ref_name: ref_name =3D symbol.lower() =20 - if replace_underscores: + # c-type references don't support hash + if ref_type =3D=3D ":ref" and replace_underscores: ref_name =3D ref_name.replace("_", "-") =20 ref_link =3D f"{ref_type}:`{symbol} <{ref_name}>`" @@ -258,8 +259,7 @@ class ParseHeader: if match: name =3D match.group(2).strip() symbol =3D match.group(3) - self.store_type("typedef", symbol, ref_name=3Dname, - replace_underscores=3DFalse) + self.store_type("typedef", symbol, ref_name=3Dname) continue =20 for re_enum in self.RE_ENUMS: @@ -272,8 +272,7 @@ class ParseHeader: for re_struct in self.RE_STRUCTS: match =3D re_struct.match(line) if match: - self.store_type("struct", match.group(1), - replace_underscores=3DFalse) + self.store_type("struct", match.group(1)) break =20 def process_exceptions(self, fname: str): --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9066D311595; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=Sy/KGDsEoRRyvrfvhqWjRJP1eblSM7Wr9VRTURUEiM2ynYbZmVI/oynwAo7qY8DRG6RU0qSAPQUUwnGeWy2pJbzX0flB6g6pAtwBHkB4MFejQsJPmRH3G291zWdjHZRZ0WyR5vRH0eXZZc0fVmxvuUWGHrhQsTvsZHwpi+eoI9k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=BpY1ojw/Qkk4wSUHKK5IpeGejLVULJug09/IqB3x+xs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=LqkVnPFAH93QYr1S6Pdj1fAPozpXmsjTn557p7zTa5XbP+b280FDVEn8fo1n41tqUnOlXZP7NWDeLFlHtbraresQmqqCZ/nns5GutszGix4EtsbkI4zqF0B9Pp/yGTlxmB1lI+UCMsf8cpmpZwAcG1n5YPlu7N34UmVl3prN+XE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=msHiZmle; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="msHiZmle" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 08CF2C113CF; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=BpY1ojw/Qkk4wSUHKK5IpeGejLVULJug09/IqB3x+xs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=msHiZmleMzqwSGeStt9vpUZHhh6OQsviXKX5h7h8e9WdwZDjpp9qf675vGp8Q9Joe bEJFsATLAIwwXvq+J8gsCX9yymdn0u9Evy+3PKe31c//8tNY5ieRled1yVCWHfEiad i6Yq4r9kcE7+TTvZbMACEOWCuZWEcIawAXayrdQ4Q3VdXToaPWuFe8tgaejjBmBQMm Xn+OYNKRU9gik6DUG5OaRT+d52qt3nVoxVeE1MsQj/iWsMawD2m3RPUqmblXWgj9Un q1a2JJw6x6Lynz79HiUD12MNY9jN+qla0BTYb/o0ok/yvyMhs7x7JXng2GMY0BX+3p XFdBsOe3rwCZw== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCrF-16ia; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , Donald Hunter , Jakub Kicinski , Jan Stancek , linux-kernel@vger.kernel.org Subject: [PATCH v2 06/24] tools: docs: parse-headers.py: move it from sphinx dir Date: Fri, 22 Aug 2025 16:19:18 +0200 Message-ID: <0f5ac2d704cffe9834e589b39549d2393e1237ef.1755872208.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" As suggested by Jon, we should start having a tools/docs directory, instead of placing everything under scripts. In the specific case of parse-headers.py, the previous location is where we're placing Sphinx extensions, which is not the right place for execs. Move it to tools/docs/parse-headers.py. Signed-off-by: Mauro Carvalho Chehab --- .pylintrc | 2 +- tools/docs/lib/__init__.py | 0 tools/docs/lib/enrich_formatter.py | 70 ++++++++++++++ .../docs/lib/parse_data_structs.py | 95 ++----------------- tools/docs/parse-headers.py | 57 +++++++++++ 5 files changed, 135 insertions(+), 89 deletions(-) create mode 100644 tools/docs/lib/__init__.py create mode 100644 tools/docs/lib/enrich_formatter.py rename Documentation/sphinx/parse-headers.py =3D> tools/docs/lib/parse_dat= a_structs.py (80%) create mode 100755 tools/docs/parse-headers.py diff --git a/.pylintrc b/.pylintrc index 30b8ae1659f8..89eaf2100edd 100644 --- a/.pylintrc +++ b/.pylintrc @@ -1,2 +1,2 @@ [MASTER] -init-hook=3D'import sys; sys.path +=3D ["scripts/lib/kdoc", "scripts/lib/a= bi"]' +init-hook=3D'import sys; sys.path +=3D ["scripts/lib/kdoc", "scripts/lib/a= bi", "tools/docs/lib"]' diff --git a/tools/docs/lib/__init__.py b/tools/docs/lib/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/tools/docs/lib/enrich_formatter.py b/tools/docs/lib/enrich_for= matter.py new file mode 100644 index 000000000000..bb171567a4ca --- /dev/null +++ b/tools/docs/lib/enrich_formatter.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 by Mauro Carvalho Chehab . + +""" +Ancillary argparse HelpFormatter class that works on a similar way as +argparse.RawDescriptionHelpFormatter, e.g. description maintains line +breaks, but it also implement transformations to the help text. The +actual transformations ar given by enrich_text(), if the output is tty. + +Currently, the follow transformations are done: + + - Positional arguments are shown in upper cases; + - if output is TTY, ``var`` and positional arguments are shown prepend= ed + by an ANSI SGR code. This is usually translated to bold. On some + terminals, like, konsole, this is translated into a colored bold tex= t. +""" + +import argparse +import re +import sys + +class EnrichFormatter(argparse.HelpFormatter): + """ + Better format the output, making easier to identify the positional args + and how they're used at the __doc__ description. + """ + def __init__(self, *args, **kwargs): + """Initialize class and check if is TTY""" + super().__init__(*args, **kwargs) + self._tty =3D sys.stdout.isatty() + + def enrich_text(self, text): + """Handle ReST markups (currently, only ``foo``)""" + if self._tty and text: + # Replace ``text`` with ANSI SGR (bold) + return re.sub(r'\`\`(.+?)\`\`', + lambda m: f'\033[1m{m.group(1)}\033[0m', text) + return text + + def _fill_text(self, text, width, indent): + """Enrich descriptions with markups on it""" + enriched =3D self.enrich_text(text) + return "\n".join(indent + line for line in enriched.splitlines()) + + def _format_usage(self, usage, actions, groups, prefix): + """Enrich positional arguments at usage: line""" + + prog =3D self._prog + parts =3D [] + + for action in actions: + if action.option_strings: + opt =3D action.option_strings[0] + if action.nargs !=3D 0: + opt +=3D f" {action.dest.upper()}" + parts.append(f"[{opt}]") + else: + # Positional argument + parts.append(self.enrich_text(f"``{action.dest.upper()}``"= )) + + usage_text =3D f"{prefix or 'usage: '} {prog} {' '.join(parts)}\n" + return usage_text + + def _format_action_invocation(self, action): + """Enrich argument names""" + if not action.option_strings: + return self.enrich_text(f"``{action.dest.upper()}``") + + return ", ".join(action.option_strings) diff --git a/Documentation/sphinx/parse-headers.py b/tools/docs/lib/parse_d= ata_structs.py similarity index 80% rename from Documentation/sphinx/parse-headers.py rename to tools/docs/lib/parse_data_structs.py index 344090ef259c..2b7fa6bd8321 100755 --- a/Documentation/sphinx/parse-headers.py +++ b/tools/docs/lib/parse_data_structs.py @@ -1,36 +1,32 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0 -# Copyright (c) 2016 by Mauro Carvalho Chehab . -# pylint: disable=3DC0103,R0902,R0912,R0914,R0915 +# Copyright (c) 2016-2025 by Mauro Carvalho Chehab . +# pylint: disable=3DR0912,R0915 =20 """ -Convert a C header or source file ``FILE_IN``, into a ReStructured Text -included via ..parsed-literal block with cross-references for the -documentation files that describe the API. It accepts an optional -``FILE_RULES`` file to describes what elements will be either ignored or -be pointed to a non-default reference type/name. +Parse a source file or header, creating ReStructured Text cross references. =20 -The output is written at ``FILE_OUT``. +It accepts an optional file to change the default symbol reference or to +suppress symbols from the output. =20 It is capable of identifying defines, functions, structs, typedefs, enums and enum symbols and create cross-references for all of them. It is also capable of distinguish #define used for specifying a Linux ioctl. =20 -The optional ``FILE_RULES`` contains a set of rules like: +The optional rules file contains a set of rules like: =20 ignore ioctl VIDIOC_ENUM_FMT replace ioctl VIDIOC_DQBUF vidioc_qbuf replace define V4L2_EVENT_MD_FL_HAVE_FRAME_SEQ :c:type:`v4l2_event_mot= ion_det` """ =20 -import argparse import os import re import sys =20 =20 -class ParseHeader: +class ParseDataStructs: """ Creates an enriched version of a Kernel header file with cross-links to each C data structure type. @@ -400,80 +396,3 @@ class ParseHeader: f.write("=3D" * len(title)) f.write("\n\n.. parsed-literal::\n\n") f.write(text) - -class EnrichFormatter(argparse.HelpFormatter): - """ - Better format the output, making easier to identify the positional args - and how they're used at the __doc__ description. - """ - def __init__(self, *args, **kwargs): - """Initialize class and check if is TTY""" - super().__init__(*args, **kwargs) - self._tty =3D sys.stdout.isatty() - - def enrich_text(self, text): - """Handle ReST markups (currently, only ``foo``)""" - if self._tty and text: - # Replace ``text`` with ANSI bold - return re.sub(r'\`\`(.+?)\`\`', - lambda m: f'\033[1m{m.group(1)}\033[0m', text) - return text - - def _fill_text(self, text, width, indent): - """Enrich descriptions with markups on it""" - enriched =3D self.enrich_text(text) - return "\n".join(indent + line for line in enriched.splitlines()) - - def _format_usage(self, usage, actions, groups, prefix): - """Enrich positional arguments at usage: line""" - - prog =3D self._prog - parts =3D [] - - for action in actions: - if action.option_strings: - opt =3D action.option_strings[0] - if action.nargs !=3D 0: - opt +=3D f" {action.dest.upper()}" - parts.append(f"[{opt}]") - else: - # Positional argument - parts.append(self.enrich_text(f"``{action.dest.upper()}``"= )) - - usage_text =3D f"{prefix or 'usage: '} {prog} {' '.join(parts)}\n" - return usage_text - - def _format_action_invocation(self, action): - """Enrich argument names""" - if not action.option_strings: - return self.enrich_text(f"``{action.dest.upper()}``") - else: - return ", ".join(action.option_strings) - - -def main(): - """Main function""" - parser =3D argparse.ArgumentParser(description=3D__doc__, - formatter_class=3DEnrichFormatter) - - parser.add_argument("-d", "--debug", action=3D"count", default=3D0, - help=3D"Increase debug level. Can be used multiple= times") - parser.add_argument("file_in", help=3D"Input C file") - parser.add_argument("file_out", help=3D"Output RST file") - parser.add_argument("file_rules", nargs=3D"?", - help=3D"Exceptions file (optional)") - - args =3D parser.parse_args() - - parser =3D ParseHeader(debug=3Dargs.debug) - parser.parse_file(args.file_in) - - if args.file_rules: - parser.process_exceptions(args.file_rules) - - parser.debug_print() - parser.write_output(args.file_in, args.file_out) - - -if __name__ =3D=3D "__main__": - main() diff --git a/tools/docs/parse-headers.py b/tools/docs/parse-headers.py new file mode 100755 index 000000000000..07d3b47c4834 --- /dev/null +++ b/tools/docs/parse-headers.py @@ -0,0 +1,57 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2016, 2025 by Mauro Carvalho Chehab . +# pylint: disable=3DC0103 + +""" +Convert a C header or source file ``FILE_IN``, into a ReStructured Text +included via ..parsed-literal block with cross-references for the +documentation files that describe the API. It accepts an optional +``FILE_RULES`` file to describes what elements will be either ignored or +be pointed to a non-default reference type/name. + +The output is written at ``FILE_OUT``. + +It is capable of identifying defines, functions, structs, typedefs, +enums and enum symbols and create cross-references for all of them. +It is also capable of distinguish #define used for specifying a Linux +ioctl. + +The optional ``FILE_RULES`` contains a set of rules like: + + ignore ioctl VIDIOC_ENUM_FMT + replace ioctl VIDIOC_DQBUF vidioc_qbuf + replace define V4L2_EVENT_MD_FL_HAVE_FRAME_SEQ :c:type:`v4l2_event_mot= ion_det` +""" + +import argparse + +from lib.parse_data_structs import ParseDataStructs +from lib.enrich_formatter import EnrichFormatter + +def main(): + """Main function""" + parser =3D argparse.ArgumentParser(description=3D__doc__, + formatter_class=3DEnrichFormatter) + + parser.add_argument("-d", "--debug", action=3D"count", default=3D0, + help=3D"Increase debug level. Can be used multiple= times") + parser.add_argument("file_in", help=3D"Input C file") + parser.add_argument("file_out", help=3D"Output RST file") + parser.add_argument("file_rules", nargs=3D"?", + help=3D"Exceptions file (optional)") + + args =3D parser.parse_args() + + parser =3D ParseDataStructs(debug=3Dargs.debug) + parser.parse_file(args.file_in) + + if args.file_rules: + parser.process_exceptions(args.file_rules) + + parser.debug_print() + parser.write_output(args.file_in, args.file_out) + + +if __name__ =3D=3D "__main__": + main() --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 601B1301476; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=Zlk7+8MiYc/WhiUjxEdKcVkaE2R5bgg1KAHEM5pqbVOwf5puNYGn8EPF0OnS6DtM2HqeCujamulb3gHUFmvskfUHDv5nhLQZ+Wo6+LfAykmwEyqHg7W8thRju2lCROieGO/qIDsyC6gKSLwC0gFRPsNKR+BBeB84+a8C56QaXr0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=uE8fFpGnhMJcK5GXfCzqeiiea156Y1/8xBUIdOSHKgI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gu9U4mGr9QKzPrkygxrpVUyAW/4tbVVaeLhq+1CRPKX9acEs6lRGvk0Z8TLoG6zd24Hm6T9k4p4EDoDbifrJNSphwuiuy3BVPT4iLhuJrFLu8Ja30Sdon/4UKkxDAOvJ1eqIZpOVTCVhPE8hnH+Qq/QVEgz0m5soGqwIKVjeSpk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=J28/AtaW; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="J28/AtaW" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 167CAC16AAE; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=uE8fFpGnhMJcK5GXfCzqeiiea156Y1/8xBUIdOSHKgI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=J28/AtaWZHBdvAsacEffPQkK3sB0xJ44okSEHhpklKZ1OEurf+d1Ea+nOq4pAqdfQ vVSkW29ZSAjAcAIW/vA2nkXj9DRmpKuc33i01NSOI0OzudhcRQghmYKNZEqKurBgbJ fKwNviSheveADFSW8WTdbweDzWDMqlois6fhBnRY+Wap+kTqIRQW+kTtAS4yFoh/Cp QbkNwCksg/zDJK8PTOxfzDu0EVEZ7buAbKO46+Tn/pTyjRGXEl90KJ0Tr7+rPRje6I QORkTU1EJ2R1yWv39/E+6xbLkfyIVMmz5SU9fN3x+y7e/AfcrvxbfhtAdsWkxxhy3m TkgwRQDz/TUpg== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCrJ-1DXT; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v2 07/24] tools: docs: parse_data_structs.py: add methods to return output Date: Fri, 22 Aug 2025 16:19:19 +0200 Message-ID: X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" When running it from command line, we want to write an output file, but when used as a class, one may just want the output content returned as a string. Split write_output() on two methods to allow both usecases. Also add an extra method to produce a TOC. Signed-off-by: Mauro Carvalho Chehab --- tools/docs/lib/parse_data_structs.py | 62 ++++++++++++++++++++++++++-- tools/docs/parse-headers.py | 5 ++- 2 files changed, 62 insertions(+), 5 deletions(-) diff --git a/tools/docs/lib/parse_data_structs.py b/tools/docs/lib/parse_da= ta_structs.py index 2b7fa6bd8321..a5aa2e182052 100755 --- a/tools/docs/lib/parse_data_structs.py +++ b/tools/docs/lib/parse_data_structs.py @@ -97,33 +97,39 @@ class ParseDataStructs: "prefix": "\\ ", "suffix": "\\ ", "ref_type": ":ref", + "description": "IOCTL Commands", }, "define": { "prefix": "\\ ", "suffix": "\\ ", "ref_type": ":ref", + "description": "Macros and Definitions", }, # We're calling each definition inside an enum as "symbol" "symbol": { "prefix": "\\ ", "suffix": "\\ ", "ref_type": ":ref", + "description": "Enumeration values", }, "typedef": { "prefix": "\\ ", "suffix": "\\ ", "ref_type": ":c:type", + "description": "Type Definitions", }, - # This is the name of the enum itself + # This is the description of the enum itself "enum": { "prefix": "\\ ", "suffix": "\\ ", "ref_type": ":c:type", + "description": "Enumerations", }, "struct": { "prefix": "\\ ", "suffix": "\\ ", "ref_type": ":c:type", + "description": "Structures", }, } =20 @@ -359,7 +365,7 @@ class ParseDataStructs: =20 print() =20 - def write_output(self, file_in: str, file_out: str): + def gen_output(self): """Write the formatted output to a file.""" =20 # Avoid extra blank lines @@ -387,12 +393,60 @@ class ParseDataStructs: text =3D re.sub(r"\\ ([\n ])", r"\1", text) text =3D re.sub(r" \\ ", " ", text) =20 + return text =20 + def gen_toc(self): + """ + Create a TOC table pointing to each symbol from the header + """ + text =3D [] + + # Add header + text.append(".. contents:: Table of Contents") + text.append(" :depth: 2") + text.append(" :local:") + text.append("") + + # Sort symbol types per description + symbol_descriptions =3D [] + for k, v in self.DEF_SYMBOL_TYPES.items(): + symbol_descriptions.append((v['description'], k)) + + symbol_descriptions.sort() + + # Process each category + for description, c_type in symbol_descriptions: + + refs =3D self.symbols[c_type] + if not refs: # Skip empty categories + continue + + text.append(f"{description}") + text.append("-" * len(description)) + text.append("") + + # Sort symbols alphabetically + for symbol, ref in sorted(refs.items()): + text.append(f"* :{ref}:") + + text.append("") # Add empty line between categories + + return "\n".join(text) + + def write_output(self, file_in: str, file_out: str, toc: bool): title =3D os.path.basename(file_in) =20 + if toc: + text =3D self.gen_toc() + else: + text =3D self.gen_output() + with open(file_out, "w", encoding=3D"utf-8", errors=3D"backslashre= place") as f: f.write(".. -*- coding: utf-8; mode: rst -*-\n\n") f.write(f"{title}\n") - f.write("=3D" * len(title)) - f.write("\n\n.. parsed-literal::\n\n") + f.write("=3D" * len(title) + "\n\n") + + if not toc: + f.write(".. parsed-literal::\n\n") + f.write(text) diff --git a/tools/docs/parse-headers.py b/tools/docs/parse-headers.py index 07d3b47c4834..bfa4e46a53e3 100755 --- a/tools/docs/parse-headers.py +++ b/tools/docs/parse-headers.py @@ -36,6 +36,9 @@ def main(): =20 parser.add_argument("-d", "--debug", action=3D"count", default=3D0, help=3D"Increase debug level. Can be used multiple= times") + parser.add_argument("-t", "--toc", action=3D"store_true", + help=3D"instead of a literal block, outputs a TOC = table at the RST file") + parser.add_argument("file_in", help=3D"Input C file") parser.add_argument("file_out", help=3D"Output RST file") parser.add_argument("file_rules", nargs=3D"?", @@ -50,7 +53,7 @@ def main(): parser.process_exceptions(args.file_rules) =20 parser.debug_print() - parser.write_output(args.file_in, args.file_out) + parser.write_output(args.file_in, args.file_out, args.toc) =20 =20 if __name__ =3D=3D "__main__": --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A7351311C12; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=c1KjvMo6O9WroKMXEpT1Ks/kqXLKjQq0vqzl6ggU1BTRtUlwJas8x3luDiXuNVEQsN6gYvxpZpovmc7oGPKPkWNGgXxtSsjX/Nvo2EVRn3BdjoioVP3UZhosEPTj6YwhiSt+7dID23xrRagjjYdMLlFAmENlfcjmd5Vk1ls8Iv0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=RsM+00sJ/1A8GHlMF3o/XgMw54KHkYT1Q+rNZSrXntI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=SK/MCVPFpRJHNzYLHUTfUl8Jm4L1IHH00Tg9wJeXP75bTMGAaOUEIwK02ok6jELETk2VN/yITOjdlvjjNZrJNAcg2jjfueoBG5umIW21yJJz1dQVJWdPgShQybu54MF8PbYDYFNVDawgzpoCcwNgrtcO5FWcBuDeRls8TXf5Ogw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=AzcnIl2n; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="AzcnIl2n" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 20353C19424; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=RsM+00sJ/1A8GHlMF3o/XgMw54KHkYT1Q+rNZSrXntI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AzcnIl2n8lSxnrpuxK5ZsmOjQFgBoyGiGSSc/CHbeXizoJKZbva+5Xo/63tYgL2xv 3uYkcgKWjmFKVaB7vOEU/mNsdqIUxZ5WT/z0bdvZ9DFl+MMUJdw9S5STLUeSIb6CVN fITzBmJVKBHYemHCftxjGxWOtswLkVAvaJs8e1izCJNXerIPLF6Xsnb3Vnh4uwiBSL Xf6H76jbXvp5soQqPPoduv9BL9Net6skrxhTWtMHuraszTLdDqo3dMJGmruE3r3zMl Lmf1jMf8zfQeg6gbenmxSxBXoUXUvx9MkXd7igliRa+Y90NvXYPUK+FCSjkPhi49Lz D26bZF03UPk5A== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCrN-1LLz; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v2 08/24] MAINTAINERS: add files from tools/docs to documentation entry Date: Fri, 22 Aug 2025 16:19:20 +0200 Message-ID: <87bbb94e442fe747e24f801d7685856b392b2568.1755872208.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" As we now have a tools directory for docs, add it to its corresponding entry. Signed-off-by: Mauro Carvalho Chehab --- MAINTAINERS | 1 + 1 file changed, 1 insertion(+) diff --git a/MAINTAINERS b/MAINTAINERS index dafc11712544..ef87548b8f88 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -7308,6 +7308,7 @@ F: scripts/get_abi.py F: scripts/kernel-doc* F: scripts/lib/abi/* F: scripts/lib/kdoc/* +F: tools/docs/* F: tools/net/ynl/pyynl/lib/doc_generator.py F: scripts/sphinx-pre-install X: Documentation/ABI/ --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A56CA31197D; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=F//AVQtH4fcPehIUxjNrO3oMqAdmYMFB9ViSBiH6OiE6bba1rfCLAFgr+5TVjeBKrgZKegaiwg+MbugvtsCvYVweUvaISN2XmLFIW6UR77gm7ihR/zqw2a9f8KNAvFDOB71wn0UC8iK6YQrbDEDQljvStSEh0XS8mu8qt8e7kBc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=6KNyilChtfEfKAJL2KTz7S3+EmJ+UL4s3OTBvprovpQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Y68VDYw/nomqRSEOub7Cx5CcJ9yCLS8faowqiHd+BMOi2t5dRgyMUJkF1ZtCPycfpXfQHLqqdgKVh7+imfoLPudKvE+JyYiMEBMy7mWPgapqHRu8EFwX4CWzTl88X/+5XYwypIRtxA+i5YHZ28f+pTAk8w8txWwFBfs5vCR1/iY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=miAKsje/; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="miAKsje/" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2BACEC2BC87; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=6KNyilChtfEfKAJL2KTz7S3+EmJ+UL4s3OTBvprovpQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=miAKsje/nfOyunqaqtj4aa3jGRjwdsh3CVfqAUM0XNA7e0ukqTN/HJRKYVxST7EKs t9pvMB9/OHsSZdQz45SC4OmGD0xlWXWqE+jZ+qMTps+sItb44S3lgN8P/lS47ZlxVX X4Ew8ZEUoS5/7QS8bLEGVWi6ZxR8nVUwW8fa5755zdn6LMHYVmc9klkFyweHuzld+r Ff5G2Xybd23mxysIMlDrEONskhMZguzx63hxuYVCmMMdWGl6jO9XkWfX5PsXyJyai0 5rut41JyxHalBAXtdAbgzJqjolHU++Bo6vrIXx0IedNx+hTeJwuJWPXRH2pyeei0BX kqzRlP4qFg3Iw== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCrR-1RxK; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org, linux-media@vger.kernel.org Subject: [PATCH v2 09/24] docs: uapi: media: Makefile: use parse-headers.py Date: Fri, 22 Aug 2025 16:19:21 +0200 Message-ID: <7025759744f74058eee55c35e8cd8cb5a2953fca.1755872208.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Now that we have a new parser, use it. Signed-off-by: Mauro Carvalho Chehab --- Documentation/userspace-api/media/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Documentation/userspace-api/media/Makefile b/Documentation/use= rspace-api/media/Makefile index 3d8aaf5c253b..accc734d045a 100644 --- a/Documentation/userspace-api/media/Makefile +++ b/Documentation/userspace-api/media/Makefile @@ -3,7 +3,7 @@ # Rules to convert a .h file to inline RST documentation =20 SRC_DIR=3D$(srctree)/Documentation/userspace-api/media -PARSER =3D $(srctree)/Documentation/sphinx/parse-headers.pl +PARSER =3D $(srctree)/tools/docs/parse-headers.py UAPI =3D $(srctree)/include/uapi/linux KAPI =3D $(srctree)/include/linux =20 --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA1FC3126B0; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=sOLX1+VKJdU1Kblk2zUBGvw6IM5RDQE4LD/5L1Oxtnqrb+elzJvTgfIOKF10Tgbhv2CvUtmZhEqZx4+1XU4pNmZsdJD7wY9W0qZrX+sHH4LHg+bmd1d0/KW8CkZhIk4gc96/p8Cs1k56hBqDHevqvMSPZ9wg2v5KgK9BMeXWKuc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=QHi7tmIl6nunuTAnM3ij6+YulhA5kpLqxpviwyDbWAA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RkLNEYDelZHJBiAu9xW5z0NLNKSYIScaR17okbZ6mqGBEA9Hfwa2Olb98SsYShJ8Z/zVR39YyHP1KPtP8yd59HnSYgK7euAKYjOZiXjvmvUfe+UWXCeS5BlJsEJERUneTswasYeQUIPOrqBN5E06BYPJRpebFm9kgArtWYCh4H4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=TeGepcH5; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="TeGepcH5" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 231EAC19422; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=QHi7tmIl6nunuTAnM3ij6+YulhA5kpLqxpviwyDbWAA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TeGepcH5/5qli8litD44Aq0Yhvo7gooOdf6YxyWLW14sag1XnYh9sXlY5+rNfENRG k/5uhjfW8odFPXY5jpTmW8p+O5j+9jIqIRc36zxprsRN2RV1P5LE9fK4gCE3o+wl+u pHllJS+D3v/XpGrxcIkdsCIzJLCEZWToimYSt7VKAJ1SDCAuToP2+XCTLHcQgk+ebh Lryuxo0EHAwm9FaluZ8RO5GuIuAbvzDujETiRNeKG5ShxMwXf3GQE5D0Y1Lfpu0Cew moKCbkodchFj16iHEz9WLUa901oOmlyNGX7lhZBaTeqScmt5UlfMqu8kyEvNHYcwTb uFIyH9QBh5zBQ== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCrV-1Yej; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , Kees Cook , linux-kernel@vger.kernel.org Subject: [PATCH v2 10/24] docs: kernel_include.py: Update its coding style Date: Fri, 22 Aug 2025 16:19:22 +0200 Message-ID: X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" With the help of tools like black, pylint, autopep8 and flake, improve the code style in preparation for further changes. No functional changes. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/kernel_include.py | 100 ++++++++++++------------- 1 file changed, 47 insertions(+), 53 deletions(-) diff --git a/Documentation/sphinx/kernel_include.py b/Documentation/sphinx/= kernel_include.py index 1e566e87ebcd..1212786ac516 100755 --- a/Documentation/sphinx/kernel_include.py +++ b/Documentation/sphinx/kernel_include.py @@ -1,7 +1,6 @@ #!/usr/bin/env python3 -# -*- coding: utf-8; mode: python -*- # SPDX-License-Identifier: GPL-2.0 -# pylint: disable=3DR0903, C0330, R0914, R0912, E0401 +# pylint: disable=3DR0903, R0912, R0914, R0915, C0209,W0707 =20 """ kernel-include @@ -40,41 +39,38 @@ from docutils.parsers.rst import directives from docutils.parsers.rst.directives.body import CodeBlock, NumberLines from docutils.parsers.rst.directives.misc import Include =20 -__version__ =3D '1.0' +__version__ =3D "1.0" + =20 # =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D def setup(app): -# =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D - + """Setup Sphinx exension""" app.add_directive("kernel-include", KernelInclude) - return dict( - version =3D __version__, - parallel_read_safe =3D True, - parallel_write_safe =3D True - ) + return { + "version": __version__, + "parallel_read_safe": True, + "parallel_write_safe": True, + } + =20 # =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D class KernelInclude(Include): -# =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D - """KernelInclude (``kernel-include``) directive""" =20 def run(self): env =3D self.state.document.settings.env - path =3D os.path.realpath( - os.path.expandvars(self.arguments[0])) + path =3D os.path.realpath(os.path.expandvars(self.arguments[0])) =20 # to get a bit security back, prohibit /etc: if path.startswith(os.sep + "etc"): - raise self.severe( - 'Problems with "%s" directive, prohibited path: %s' - % (self.name, path)) + raise self.severe('Problems with "%s" directive, prohibited pa= th: %s' % + (self.name, path)) =20 self.arguments[0] =3D path =20 env.note_dependency(os.path.abspath(path)) =20 - #return super(KernelInclude, self).run() # won't work, see HINTs i= n _run() + # return super(KernelInclude, self).run() # won't work, see HINTs = in _run() return self._run() =20 def _run(self): @@ -87,41 +83,39 @@ class KernelInclude(Include): =20 if not self.state.document.settings.file_insertion_enabled: raise self.warning('"%s" directive disabled.' % self.name) - source =3D self.state_machine.input_lines.source( - self.lineno - self.state_machine.input_offset - 1) + source =3D self.state_machine.input_lines.source(self.lineno - + self.state_machine.= input_offset - 1) source_dir =3D os.path.dirname(os.path.abspath(source)) path =3D directives.path(self.arguments[0]) - if path.startswith('<') and path.endswith('>'): + if path.startswith("<") and path.endswith(">"): path =3D os.path.join(self.standard_include_path, path[1:-1]) path =3D os.path.normpath(os.path.join(source_dir, path)) =20 # HINT: this is the only line I had to change / commented out: - #path =3D utils.relative_path(None, path) + # path =3D utils.relative_path(None, path) =20 - encoding =3D self.options.get( - 'encoding', self.state.document.settings.input_encoding) - e_handler=3Dself.state.document.settings.input_encoding_error_hand= ler - tab_width =3D self.options.get( - 'tab-width', self.state.document.settings.tab_width) + encoding =3D self.options.get("encoding", + self.state.document.settings.input_enc= oding) + e_handler =3D self.state.document.settings.input_encoding_error_ha= ndler + tab_width =3D self.options.get("tab-width", + self.state.document.settings.tab_widt= h) try: self.state.document.settings.record_dependencies.add(path) - include_file =3D io.FileInput(source_path=3Dpath, - encoding=3Dencoding, + include_file =3D io.FileInput(source_path=3Dpath, encoding=3De= ncoding, error_handler=3De_handler) - except UnicodeEncodeError as error: + except UnicodeEncodeError: raise self.severe('Problems with "%s" directive path:\n' 'Cannot encode input file path "%s" ' - '(wrong locale?).' % - (self.name, SafeString(path))) + "(wrong locale?)." % (self.name, SafeString(= path))) except IOError as error: - raise self.severe('Problems with "%s" directive path:\n%s.' % - (self.name, ErrorString(error))) - startline =3D self.options.get('start-line', None) - endline =3D self.options.get('end-line', None) + raise self.severe('Problems with "%s" directive path:\n%s.' + % (self.name, ErrorString(error))) + startline =3D self.options.get("start-line", None) + endline =3D self.options.get("end-line", None) try: if startline or (endline is not None): lines =3D include_file.readlines() - rawtext =3D ''.join(lines[startline:endline]) + rawtext =3D "".join(lines[startline:endline]) else: rawtext =3D include_file.read() except UnicodeError as error: @@ -129,43 +123,43 @@ class KernelInclude(Include): (self.name, ErrorString(error))) # start-after/end-before: no restrictions on newlines in match-tex= t, # and no restrictions on matching inside lines vs. line boundaries - after_text =3D self.options.get('start-after', None) + after_text =3D self.options.get("start-after", None) if after_text: # skip content in rawtext before *and incl.* a matching text after_index =3D rawtext.find(after_text) if after_index < 0: raise self.severe('Problem with "start-after" option of "%= s" ' - 'directive:\nText not found.' % self.nam= e) - rawtext =3D rawtext[after_index + len(after_text):] - before_text =3D self.options.get('end-before', None) + "directive:\nText not found." % self.nam= e) + rawtext =3D rawtext[after_index + len(after_text) :] + before_text =3D self.options.get("end-before", None) if before_text: # skip content in rawtext after *and incl.* a matching text before_index =3D rawtext.find(before_text) if before_index < 0: raise self.severe('Problem with "end-before" option of "%s= " ' - 'directive:\nText not found.' % self.nam= e) + "directive:\nText not found." % self.nam= e) rawtext =3D rawtext[:before_index] =20 include_lines =3D statemachine.string2lines(rawtext, tab_width, convert_whitespace=3DTru= e) - if 'literal' in self.options: + if "literal" in self.options: # Convert tabs to spaces, if `tab_width` is positive. if tab_width >=3D 0: text =3D rawtext.expandtabs(tab_width) else: text =3D rawtext literal_block =3D nodes.literal_block(rawtext, source=3Dpath, - classes=3Dself.options.get('class', []= )) + classes=3Dself.options.get= ("class", []) + ) literal_block.line =3D 1 self.add_name(literal_block) - if 'number-lines' in self.options: + if "number-lines" in self.options: try: - startline =3D int(self.options['number-lines'] or 1) + startline =3D int(self.options["number-lines"] or 1) except ValueError: - raise self.error(':number-lines: with non-integer ' - 'start value') + raise self.error(":number-lines: with non-integer star= t value") endline =3D startline + len(include_lines) - if text.endswith('\n'): + if text.endswith("\n"): text =3D text[:-1] tokens =3D NumberLines([([], text)], startline, endline) for classes, value in tokens: @@ -177,12 +171,12 @@ class KernelInclude(Include): else: literal_block +=3D nodes.Text(text, text) return [literal_block] - if 'code' in self.options: - self.options['source'] =3D path + if "code" in self.options: + self.options["source"] =3D path codeblock =3D CodeBlock(self.name, - [self.options.pop('code')], # arguments + [self.options.pop("code")], # arguments self.options, - include_lines, # content + include_lines, # content self.lineno, self.content_offset, self.block_text, --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BF8563126B1; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=Q1WbwRKw9q4dt0JRqD9H2vsqRBB/XtOHG01q9GwsPDFT5AvtnOcbJSKzapKTbCo+gH4lkARKhiAMjL0O1puBu1FfxYTIb1SwY6S85dxZpo/cjF22PEyzYoQheL8e9z+VckeD6JsA/TQPxsbwCmPh5bIqBz3oIZ8XwtI8KrSXM48= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=lRH9SEB4iDb1LfHInxrSMa8aPsx5ggsWJbqLQRJAoqQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=CQ1c0Kig6TsI7xkObiYUUYrRviFuuEGjeTbCNp+Mgs3dpEVhgvA571QQqEd8tuudwgAnc6dpHyBBRX+mlqbSe35qUdL4NsIHjqR1Za23dbYGlSSx84TOXa6yk3OtOlPE1tU+Sh+lu5qetVgFgx2Kv8RoHRM0QeABH1ol1p58lXs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=QYYkXOU3; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="QYYkXOU3" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2BF0DC2BC9E; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=lRH9SEB4iDb1LfHInxrSMa8aPsx5ggsWJbqLQRJAoqQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QYYkXOU3/ipHgWgtXTfBSbsdEnwC0+UoegWmTxdTXZQOpSYSzrLpdKU4f3lhpusYn lSnbym7pyKO3LtuYqcYYe4XzIud6e8FLAxGIhgq+LRAVhbyzSTBjN4QIYkfs5XVnjC cz5B3MyuMFnZfMDYoOn4MC4GPFmp0RhcgaNmxUKRuYw8BvAh+crlrMLLwQsXx6jhlu f9JaYTDhyJ2ZEItjf8VRSF4uhFfOwIxH7c195ejBIPJQY3zGLXkktFTNnjGgowKP1I yA2Ut026HP8Fxcb8I6irFIj1XSM310Qg20EkG5Rt/Frm/yYs6Ukg0bzP/m7vK0w2K2 0iUtRLNrtt3iQ== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCrZ-1fRm; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , Kees Cook , linux-kernel@vger.kernel.org Subject: [PATCH v2 11/24] docs: kernel_include.py: allow cross-reference generation Date: Fri, 22 Aug 2025 16:19:23 +0200 Message-ID: X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" kernel_include extension was originally designed to be used by the media comprehensive uAPI documentation, where, instead of simpler kernel-doc markups, the uAPI documentation is enriched with a larger text, with images, complex tables, graphs, etc. There, we wanted to include the much simpler yet documented .h file. This extension is needed to include files from other parts of the Kernel tree outside Documentation, because the original Sphinx include tag doesn't allow going outside of the directory passed via sphinx-build command line. Yet, the cross-references themselves to the full documentation were using a perl script to create cross-references against the comprehensive documentation. As the perl script is now converted to Phython and there is a Python class producing an include-compatible output with cross references, add two optional arguments to kernel_include.py: 1. :generate-cross-refs: If present, instead of reading the file, it calls ParseDataStructs() class, which converts C data structures into cross-references to be linked to ReST files containing a more comprehensive documentati= on; Don't use it together with :start-line: and/or :end-line:, as filtering input file line range is currently not supported. 2. :exception-file: Used together with :generate-cross-refs:. Points to a file containi= ng rules to ignore C data structs or to use a different reference name, optionally using a different reference type. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/kernel_include.py | 94 ++++++++++++++++++++------ 1 file changed, 74 insertions(+), 20 deletions(-) diff --git a/Documentation/sphinx/kernel_include.py b/Documentation/sphinx/= kernel_include.py index 1212786ac516..fc37e6fa9d96 100755 --- a/Documentation/sphinx/kernel_include.py +++ b/Documentation/sphinx/kernel_include.py @@ -25,6 +25,24 @@ Substrings of the form $name or ${name} are replaced by the value of environment variable name. Malformed variable names and references to non-existing variables are left unchanged. + + This extension overrides Sphinx include directory, adding two extra + arguments: + + 1. :generate-cross-refs: + + If present, instead of reading the file, it calls ParseDataStructs= () + class, which converts C data structures into cross-references to + be linked to ReST files containing a more comprehensive documentat= ion; + + Don't use it together with :start-line: and/or :end-line:, as + filtering input file line range is currently not supported. + + 2. :exception-file: + + Used together with :generate-cross-refs:. Points to a file contain= ing + rules to ignore C data structs or to use a different reference nam= e, + optionally using a different reference type. """ =20 # =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D @@ -32,6 +50,7 @@ # =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D =20 import os.path +import sys =20 from docutils import io, nodes, statemachine from docutils.utils.error_reporting import SafeString, ErrorString @@ -39,6 +58,11 @@ from docutils.parsers.rst import directives from docutils.parsers.rst.directives.body import CodeBlock, NumberLines from docutils.parsers.rst.directives.misc import Include =20 +srctree =3D os.path.abspath(os.environ["srctree"]) +sys.path.insert(0, os.path.join(srctree, "tools/docs/lib")) + +from parse_data_structs import ParseDataStructs + __version__ =3D "1.0" =20 =20 @@ -57,6 +81,14 @@ def setup(app): class KernelInclude(Include): """KernelInclude (``kernel-include``) directive""" =20 + # Add extra options + option_spec =3D Include.option_spec.copy() + + option_spec.update({ + 'generate-cross-refs': directives.flag, + 'exception-file': directives.unchanged, + }) + def run(self): env =3D self.state.document.settings.env path =3D os.path.realpath(os.path.expandvars(self.arguments[0])) @@ -99,28 +131,49 @@ class KernelInclude(Include): e_handler =3D self.state.document.settings.input_encoding_error_ha= ndler tab_width =3D self.options.get("tab-width", self.state.document.settings.tab_widt= h) - try: - self.state.document.settings.record_dependencies.add(path) - include_file =3D io.FileInput(source_path=3Dpath, encoding=3De= ncoding, - error_handler=3De_handler) - except UnicodeEncodeError: - raise self.severe('Problems with "%s" directive path:\n' - 'Cannot encode input file path "%s" ' - "(wrong locale?)." % (self.name, SafeString(= path))) - except IOError as error: - raise self.severe('Problems with "%s" directive path:\n%s.' - % (self.name, ErrorString(error))) startline =3D self.options.get("start-line", None) endline =3D self.options.get("end-line", None) - try: - if startline or (endline is not None): - lines =3D include_file.readlines() - rawtext =3D "".join(lines[startline:endline]) - else: - rawtext =3D include_file.read() - except UnicodeError as error: - raise self.severe('Problem with "%s" directive:\n%s' % - (self.name, ErrorString(error))) + + # Get optional arguments to related to cross-references generation + if 'generate-cross-refs' in self.options: + parser =3D ParseDataStructs() + parser.parse_file(path) + + exceptions_file =3D self.options.get('exception-file') + if exceptions_file: + exceptions_file =3D os.path.join(source_dir, exceptions_fi= le) + parser.process_exceptions(exceptions_file) + + title =3D os.path.basename(path) + rawtext =3D parser.gen_output() + if startline or endline: + raise self.severe('generate-cross-refs can\'t be used toge= ther with "start-line" or "end-line"') + + if "code" not in self.options: + rawtext =3D ".. parsed-literal::\n\n" + rawtext + else: + try: + self.state.document.settings.record_dependencies.add(path) + include_file =3D io.FileInput(source_path=3Dpath, encoding= =3Dencoding, + error_handler=3De_handler) + except UnicodeEncodeError: + raise self.severe('Problems with "%s" directive path:\n' + 'Cannot encode input file path "%s" ' + "(wrong locale?)." % (self.name, SafeStrin= g(path))) + except IOError as error: + raise self.severe('Problems with "%s" directive path:\n%s.' + % (self.name, ErrorString(error))) + + try: + if startline or (endline is not None): + lines =3D include_file.readlines() + rawtext =3D "".join(lines[startline:endline]) + else: + rawtext =3D include_file.read() + except UnicodeError as error: + raise self.severe('Problem with "%s" directive:\n%s' % + (self.name, ErrorString(error))) + # start-after/end-before: no restrictions on newlines in match-tex= t, # and no restrictions on matching inside lines vs. line boundaries after_text =3D self.options.get("start-after", None) @@ -171,6 +224,7 @@ class KernelInclude(Include): else: literal_block +=3D nodes.Text(text, text) return [literal_block] + if "code" in self.options: self.options["source"] =3D path codeblock =3D CodeBlock(self.name, --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A7876311C21; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=oiJKpzPlrIb0pMWNKCY/FiRKqnajV4GYNfPID0A4VF69YCW6sKbgjgj1NjzNPHO3efQvOdB3xHowM/RT9Gre/J2v1w21Op0TsyIDGnOPy3VQ17iiArIPJ6qTW1/KDXW3dp7WZ3BrZdtp1tLrgPBiYN7QoauBLgK4yTabf0WHtCg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=UW2xelo4TnzX16OxyxtwPo8LfwSKWCjViRI3KVdm6R4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=DnAT5ufSU6x8p1hWBm/O60KHY2As637uzYyefY4moSoGK5xjILQusSZT4GgdHkoYt6vHaERC3xlPDkmR0ayuZPimdUuIRK5id4uPj0VsXzZpJKX1J1U7L6XTJkIihRnP1LQ6ODAsEiG/aQMrPfA2L/k/4lE2drZxviOS9jweWz8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Lu/d3ZW4; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Lu/d3ZW4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 35F26C2BCB3; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=UW2xelo4TnzX16OxyxtwPo8LfwSKWCjViRI3KVdm6R4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Lu/d3ZW4YxqP5K5h0db1aASen4oTPyXhAKssWGNSB9hYF0X2zKftJAbYuzdLI/QL7 qCrE7p9Ccl+MKlWAgZ0ZmknLKlGe7uEkzgdmyJrEjUvLreh3b+w7I7jEH7dV0L5XpO YgAReR7HZlGPgBidfp00g03da5IGsEEWe5TBmnCsKafU5T3+ybpmOnTBGJRfFhtmba lTUnMfwLhq882ezmYeqocAW8rGIsUb0eWSpeIo2e3m062DCEySs0vVm/O3oleoa5yM +GVxGk8YOWSJ3wTsHIGrMuau5kAtceiIHcii9W07rWhEdJUJf5tvrtW6gWTDAvLULK wQiNly3ci8XkA== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCrd-1m7g; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , Kees Cook , linux-kernel@vger.kernel.org Subject: [PATCH v2 12/24] docs: kernel_include.py: generate warnings for broken refs Date: Fri, 22 Aug 2025 16:19:24 +0200 Message-ID: <73be9a198746421687e2eee916ccf8bf67980b7d.1755872208.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab In the past, Sphinx used to warn about broken references. That's basically the rationale for adding media uAPI files: to get warnings about missed symbols. This is not true anymore. So, we need to explicitly check them after doctree-resolved event. While here, move setup() to the end, to make it closer to what we do on other extensions. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/kernel_include.py | 108 ++++++++++++++++++++----- 1 file changed, 89 insertions(+), 19 deletions(-) diff --git a/Documentation/sphinx/kernel_include.py b/Documentation/sphinx/= kernel_include.py index fc37e6fa9d96..0a3e5377dd1e 100755 --- a/Documentation/sphinx/kernel_include.py +++ b/Documentation/sphinx/kernel_include.py @@ -26,7 +26,7 @@ environment variable name. Malformed variable names and references to non-existing variables are left unchanged. =20 - This extension overrides Sphinx include directory, adding two extra + This extension overrides Sphinx include directory, adding some extra arguments: =20 1. :generate-cross-refs: @@ -35,14 +35,20 @@ class, which converts C data structures into cross-references to be linked to ReST files containing a more comprehensive documentat= ion; =20 - Don't use it together with :start-line: and/or :end-line:, as - filtering input file line range is currently not supported. - 2. :exception-file: =20 - Used together with :generate-cross-refs:. Points to a file contain= ing - rules to ignore C data structs or to use a different reference nam= e, - optionally using a different reference type. + Used together with :generate-cross-refs + + Points to a file containing rules to ignore C data structs or to + use a different reference name, optionally using a different + reference type. + + 3. :warn-broken: + + Used together with :generate-cross-refs: + + Detect if the auto-generated cross references doesn't exist. + """ =20 # =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D @@ -50,6 +56,7 @@ # =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D =20 import os.path +import re import sys =20 from docutils import io, nodes, statemachine @@ -58,23 +65,18 @@ from docutils.parsers.rst import directives from docutils.parsers.rst.directives.body import CodeBlock, NumberLines from docutils.parsers.rst.directives.misc import Include =20 +from sphinx.util import logging + srctree =3D os.path.abspath(os.environ["srctree"]) sys.path.insert(0, os.path.join(srctree, "tools/docs/lib")) =20 from parse_data_structs import ParseDataStructs =20 __version__ =3D "1.0" +logger =3D logging.getLogger(__name__) =20 - -# =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D -def setup(app): - """Setup Sphinx exension""" - app.add_directive("kernel-include", KernelInclude) - return { - "version": __version__, - "parallel_read_safe": True, - "parallel_write_safe": True, - } +RE_DOMAIN_REF =3D re.compile(r'\\ :(ref|c:type|c:func):`([^<`]+)(?:<([^>]+= )>)?`\\') +RE_SIMPLE_REF =3D re.compile(r'`([^`]+)`') =20 =20 # =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D @@ -86,6 +88,7 @@ class KernelInclude(Include): =20 option_spec.update({ 'generate-cross-refs': directives.flag, + 'warn-broken': directives.flag, 'exception-file': directives.unchanged, }) =20 @@ -103,9 +106,9 @@ class KernelInclude(Include): env.note_dependency(os.path.abspath(path)) =20 # return super(KernelInclude, self).run() # won't work, see HINTs = in _run() - return self._run() + return self._run(env) =20 - def _run(self): + def _run(self, env): """Include a file as part of the content of this reST file.""" =20 # HINT: I had to copy&paste the whole Include.run method. I'am not= happy @@ -151,6 +154,10 @@ class KernelInclude(Include): =20 if "code" not in self.options: rawtext =3D ".. parsed-literal::\n\n" + rawtext + + # Store references on a symbol dict to be used at check time + if 'warn-broken' in self.options: + env._xref_files.add(path) else: try: self.state.document.settings.record_dependencies.add(path) @@ -239,3 +246,66 @@ class KernelInclude(Include): return codeblock.run() self.state_machine.insert_input(include_lines, path) return [] + +# =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D + +reported =3D set() + +def check_missing_refs(app, env, node, contnode): + """Check broken refs for the files it creates xrefs""" + if not node.source: + return None + + try: + xref_files =3D env._xref_files + except AttributeError: + logger.critical("FATAL: _xref_files not initialized!") + raise + + # Only show missing references for kernel-include reference-parsed fil= es + if node.source not in xref_files: + return None + + target =3D node.get('reftarget', '') + domain =3D node.get('refdomain', 'std') + reftype =3D node.get('reftype', '') + + msg =3D f"can't link to: {domain}:{reftype}:: {target}" + + # Don't duplicate warnings + data =3D (node.source, msg) + if data in reported: + return None + reported.add(data) + + logger.warning(msg, location=3Dnode, type=3D'ref', subtype=3D'missing') + + return None + +def merge_xref_info(app, env, docnames, other): + """ + As each process modify env._xref_files, we need to merge them back. + """ + if not hasattr(other, "_xref_files"): + return + env._xref_files.update(getattr(other, "_xref_files", set())) + +def init_xref_docs(app, env, docnames): + """Initialize a list of files that we're generating cross references= =C2=A8""" + app.env._xref_files =3D set() + +# =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D + +def setup(app): + """Setup Sphinx exension""" + + app.connect("env-before-read-docs", init_xref_docs) + app.connect("env-merge-info", merge_xref_info) + app.add_directive("kernel-include", KernelInclude) + app.connect("missing-reference", check_missing_refs) + + return { + "version": __version__, + "parallel_read_safe": True, + "parallel_write_safe": True, + } --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8FE9B3101BD; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=QUKZemv+RFYjz6TidXlp8TQiBlPe0BJTE7Sp6RIM2MszJh+Sn1a4yLrBrOeqYaFMek4ZrTzZQucgN+NnGDWSb3GsXzpc2ZYPlrkWi/ZnZFQO52FD4SslONs9ucy4ypQuFug/jH+uXzwxLIJ1FCIfvhxH6MHaBbw5yP17oiyfipM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=DL+ejH7HzWPaC/HHDV1RfIxKyreXW8+q4TsBZyqpgb0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hp6kFhikDSgX1D+E/xx1k+mhl7PA3dGL8sSiqYQvLmgqFekxYyHleOdtx/R1le0VOv6EWwHKT6eREu/dwBYm/+GXaBYTGB2MMMzT/h8KOGEkvn+LfCTzU8ClohjcyYffa/NzkVYdIqBH2kvI+crt7HFhedwrUgBM8I95IVTo2IM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=fnsg4Icm; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="fnsg4Icm" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 37EECC2BCB4; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=DL+ejH7HzWPaC/HHDV1RfIxKyreXW8+q4TsBZyqpgb0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fnsg4IcmGUBrbAcFRfBOHL+qHjRbTSrbcokUy+8FlKuPHdztZm4JyQvI4CqZNt/OF fCYyI/sAxPU4jPwvZ0dcR4rlJoqXDdmvCK2PLu3Oc0fnOgl28+NWJEjqnNtH8a9unh v8c+xKzT8QSod1iwCJF2BMNXi/7N1KVFkzjLTLUgihSHDnk00GWuOnOHTtWkPIe5nF AxwPtre8Fv0qb5UPRiABQRxmxPCF0PUTu+LUEj6VvxEdEoUb/1SoS0ytsNXSnUK+6a EKdXVDu8XtgU8HJK/FO+DmCDlHDPRxPNhhtkJMKxlFsqHviO/LqZ7BCi+ruQkYIIpY sUPdCHBUun85w== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCrh-1slL; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , Kees Cook , linux-kernel@vger.kernel.org Subject: [PATCH v2 13/24] docs: kernel_include.py: move rawtext logic to separate functions Date: Fri, 22 Aug 2025 16:19:25 +0200 Message-ID: <04776a94c85b6c931c198a149f08b299c9f571a3.1755872208.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" The run function is too complex. merge run() and _run() into a single function and move the read logic to separate functions. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/kernel_include.py | 82 ++++++++++++++------------ 1 file changed, 43 insertions(+), 39 deletions(-) diff --git a/Documentation/sphinx/kernel_include.py b/Documentation/sphinx/= kernel_include.py index 0a3e5377dd1e..ef86ee9e79d6 100755 --- a/Documentation/sphinx/kernel_include.py +++ b/Documentation/sphinx/kernel_include.py @@ -92,7 +92,47 @@ class KernelInclude(Include): 'exception-file': directives.unchanged, }) =20 + def read_rawtext(self, path, encoding): + """Read and process file content with error handling""" + try: + self.state.document.settings.record_dependencies.add(path) + include_file =3D io.FileInput(source_path=3Dpath, + encoding=3Dencoding, + error_handler=3Dself.state.doc= ument.settings.input_encoding_error_handler) + except UnicodeEncodeError: + raise self.severe('Problems with directive path:\n' + 'Cannot encode input file path "%s" ' + '(wrong locale?).' % SafeString(path)) + except IOError as error: + raise self.severe('Problems with directive path:\n%s.' % E= rrorString(error)) + + try: + return include_file.read() + except UnicodeError as error: + raise self.severe('Problem with directive:\n%s' % ErrorStr= ing(error)) + + def read_rawtext_with_xrefs(self, env, path): + parser =3D ParseDataStructs() + parser.parse_file(path) + + if 'exception-file' in self.options: + source_dir =3D os.path.dirname(os.path.abspath( + self.state_machine.input_lines.source( + self.lineno - self.state_machine.input_offset - 1))) + exceptions_file =3D os.path.join(source_dir, self.options['exc= eption-file']) + parser.process_exceptions(exceptions_file) + + if self.options.get("start-line") or self.options.get("end-line"): + raise self.severe('generate-cross-refs can\'t be used with "st= art-line" or "end-line"') + + # Store references on a symbol dict to be used at check time + if 'warn-broken' in self.options: + env._xref_files.add(path) + + return parser.gen_output() + def run(self): + """Include a file as part of the content of this reST file.""" env =3D self.state.document.settings.env path =3D os.path.realpath(os.path.expandvars(self.arguments[0])) =20 @@ -105,12 +145,6 @@ class KernelInclude(Include): =20 env.note_dependency(os.path.abspath(path)) =20 - # return super(KernelInclude, self).run() # won't work, see HINTs = in _run() - return self._run(env) - - def _run(self, env): - """Include a file as part of the content of this reST file.""" - # HINT: I had to copy&paste the whole Include.run method. I'am not= happy # with this, but due to security reasons, the Include.run method d= oes # not allow absolute or relative pathnames pointing to locations *= above* @@ -139,47 +173,17 @@ class KernelInclude(Include): =20 # Get optional arguments to related to cross-references generation if 'generate-cross-refs' in self.options: - parser =3D ParseDataStructs() - parser.parse_file(path) - - exceptions_file =3D self.options.get('exception-file') - if exceptions_file: - exceptions_file =3D os.path.join(source_dir, exceptions_fi= le) - parser.process_exceptions(exceptions_file) + rawtext =3D self.read_rawtext_with_xrefs(env, path) =20 title =3D os.path.basename(path) - rawtext =3D parser.gen_output() + if startline or endline: raise self.severe('generate-cross-refs can\'t be used toge= ther with "start-line" or "end-line"') =20 if "code" not in self.options: rawtext =3D ".. parsed-literal::\n\n" + rawtext - - # Store references on a symbol dict to be used at check time - if 'warn-broken' in self.options: - env._xref_files.add(path) else: - try: - self.state.document.settings.record_dependencies.add(path) - include_file =3D io.FileInput(source_path=3Dpath, encoding= =3Dencoding, - error_handler=3De_handler) - except UnicodeEncodeError: - raise self.severe('Problems with "%s" directive path:\n' - 'Cannot encode input file path "%s" ' - "(wrong locale?)." % (self.name, SafeStrin= g(path))) - except IOError as error: - raise self.severe('Problems with "%s" directive path:\n%s.' - % (self.name, ErrorString(error))) - - try: - if startline or (endline is not None): - lines =3D include_file.readlines() - rawtext =3D "".join(lines[startline:endline]) - else: - rawtext =3D include_file.read() - except UnicodeError as error: - raise self.severe('Problem with "%s" directive:\n%s' % - (self.name, ErrorString(error))) + rawtext =3D self.read_rawtext(path, encoding) =20 # start-after/end-before: no restrictions on newlines in match-tex= t, # and no restrictions on matching inside lines vs. line boundaries --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9A851311979; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=oH+YUUVuzid+uE/C5kSduQSAHKgPvGnv9MjTYxuX/bpG/RBeKiRYF3/cLAcfWXYyq++rPf6qivwTVrBlTUamRdkZmnKljFCifIiLUIYTXQAg+jbUY+KgYfQ1sBEYgF5npjgJQc+v+BAG3A5hlY7aNkaqPwycjxEWNEDaK1Xqang= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=qu5eqKXlag7ALsP16SkP+d9fl76226iDf9IK5Sgc2pE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Y352w/Oh8E6x0vY9jzKi1bgMRoMcrS8pWYBtG6OuQGit9w3KK0GZEMVsIpL3v2iK79x/tKY1Vh/KiiNp4C/KhhGsE2yDa/oEfQUu+cPKPTMD0TOewiBntkgSoSgabwhetZHE6Up5eGGlqYKeTMQT0PBY4DTHE94gzy8J8hL11U8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=RdSwNKJt; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="RdSwNKJt" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3331AC19425; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=qu5eqKXlag7ALsP16SkP+d9fl76226iDf9IK5Sgc2pE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RdSwNKJtIWxqNfoqmY0cnxTHcYolGcM5WW/mwIh0w0awYgEDDtZboXnG+5DlqMEcI N/RjB868xkaXRXQEXQIA/VqG7KOXqEzDIyBRWiYJ9KnBMcmztF67RReWqh+UE9zTFD HaKm+zHeqU0+evOhe7SU0GzXFj6RozppGl4ifQF+Ay+Po3OLJzLwCegiSGMqJFtuAm N0yi1Aa3FV5i48XWN7496Rv5MKoHrsVCbbHsuzYBqGJ3CYcqZHuqrosEJAtTcPMHpb ObmxCrdVEAB8lTSv+qVV494IUANdgO6Vy8Xm/a7fLFPESbln1AGjBa/yneB809Poyg dEhBGOFIp6oaw== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCrl-1zot; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , Kees Cook , linux-kernel@vger.kernel.org Subject: [PATCH v2 14/24] docs: kernel_include.py: move range logic to a separate function Date: Fri, 22 Aug 2025 16:19:26 +0200 Message-ID: <12fa2204a9e7e309ae4b8694a37ebad9327ca634.1755872208.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Cleanup run() function by moving the range logic to a separate function. Here, I ended checking the current Sphinx implementation, as it has some extra logic for the range check. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/kernel_include.py | 51 +++++++++++++++++--------- 1 file changed, 33 insertions(+), 18 deletions(-) diff --git a/Documentation/sphinx/kernel_include.py b/Documentation/sphinx/= kernel_include.py index ef86ee9e79d6..c5f4f34e22cb 100755 --- a/Documentation/sphinx/kernel_include.py +++ b/Documentation/sphinx/kernel_include.py @@ -131,6 +131,38 @@ class KernelInclude(Include): =20 return parser.gen_output() =20 + def apply_range(self, rawtext): + # Get to-be-included content + startline =3D self.options.get('start-line', None) + endline =3D self.options.get('end-line', None) + try: + if startline or (endline is not None): + lines =3D rawtext.splitlines() + rawtext =3D '\n'.join(lines[startline:endline]) + except UnicodeError as error: + raise self.severe(f'Problem with "{self.name}" directive:\n' + + io.error_string(error)) + # start-after/end-before: no restrictions on newlines in match-tex= t, + # and no restrictions on matching inside lines vs. line boundaries + after_text =3D self.options.get("start-after", None) + if after_text: + # skip content in rawtext before *and incl.* a matching text + after_index =3D rawtext.find(after_text) + if after_index < 0: + raise self.severe('Problem with "start-after" option of "%= s" ' + "directive:\nText not found." % self.nam= e) + rawtext =3D rawtext[after_index + len(after_text) :] + before_text =3D self.options.get("end-before", None) + if before_text: + # skip content in rawtext after *and incl.* a matching text + before_index =3D rawtext.find(before_text) + if before_index < 0: + raise self.severe('Problem with "end-before" option of "%s= " ' + "directive:\nText not found." % self.nam= e) + rawtext =3D rawtext[:before_index] + + return rawtext + def run(self): """Include a file as part of the content of this reST file.""" env =3D self.state.document.settings.env @@ -185,24 +217,7 @@ class KernelInclude(Include): else: rawtext =3D self.read_rawtext(path, encoding) =20 - # start-after/end-before: no restrictions on newlines in match-tex= t, - # and no restrictions on matching inside lines vs. line boundaries - after_text =3D self.options.get("start-after", None) - if after_text: - # skip content in rawtext before *and incl.* a matching text - after_index =3D rawtext.find(after_text) - if after_index < 0: - raise self.severe('Problem with "start-after" option of "%= s" ' - "directive:\nText not found." % self.nam= e) - rawtext =3D rawtext[after_index + len(after_text) :] - before_text =3D self.options.get("end-before", None) - if before_text: - # skip content in rawtext after *and incl.* a matching text - before_index =3D rawtext.find(before_text) - if before_index < 0: - raise self.severe('Problem with "end-before" option of "%s= " ' - "directive:\nText not found." % self.nam= e) - rawtext =3D rawtext[:before_index] + rawtext =3D self.apply_range(rawtext) =20 include_lines =3D statemachine.string2lines(rawtext, tab_width, convert_whitespace=3DTru= e) --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 987C5311977; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=sBk3eapjgedg6EyLXr2NJIenzh+PhC/87gY6WbaNlO8XL4p9JGwGJjMXWFuyYaf+ivGCsCGs5jF2ZVwGI/HKEj2nHMNoLB+xXv5YjmjfcPGqSCjcbEFKCeOByaTz2ldggzXNRRNH+iqh2WgONvAN4PQxW0ctGTvjLAMYl5c/zWI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=wQmzELiyLlP6dYnGbR5xm0Wl2cXgsAK4Vx4o5Slpgnw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=b1BLz7ZyTVp4J8wvZojFyePONFcD9yZ9T5FzMjXtQxDihaTvozYPVMj0S+8yOIUEbbdk11zhPqR+Z/NIXWOCEgSva7+p+HsiMs420iOliFFOX+5wygFHtCUWyF5t/eXWcLXd0vdeH3Ha8rRg8j6Kqi298fyDGUyGX5l9l/afDJ4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Zgmhr3DZ; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Zgmhr3DZ" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4037BC2BCB7; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=wQmzELiyLlP6dYnGbR5xm0Wl2cXgsAK4Vx4o5Slpgnw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Zgmhr3DZt+Ed1uvBvlGY6a0RI3Qw7O7QBPDF2XQDJvwwuhtFW4YIVZnU2WiUdtnXy RZ6Gc1RJRbRACwbkhdZJ8UdE6Vv/0XMyicEl3jkF+wt80SQ1cf2nKRVj+p4kcg34G/ Wo2fKhy1W3zqrj9n9g18rwI9uj+3uYLg+7ms3Vr/2/hvhixcL4MIZdKlRMuGt15dzk SfFZw4BPyIVCyRhOdHxY6ZCPD3OrCKHvmntddpns1C4ViDTOgwRmMQ0JCvOBoJHvGe BQPk9Igq2xL9iHvh9Uhm1W2WpfNejvvj4Y2Iz72516mizzANBsm7K1f3ETrphzCKih nOjuuX2gRKaRA== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCrp-26gB; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , Kees Cook , linux-kernel@vger.kernel.org Subject: [PATCH v2 15/24] docs: kernel_include.py: remove range restriction for gen docs Date: Fri, 22 Aug 2025 16:19:27 +0200 Message-ID: <5dff693860a6a3faade15c24abdc380f09db468d.1755872208.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Originally, parse-readers were generating an output where the first two lines were setting a literal block. The script now gets only the actual parsed data without that, so it is now safe to allow start-line and end-line parameters to be handled. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/kernel_include.py | 6 ------ 1 file changed, 6 deletions(-) diff --git a/Documentation/sphinx/kernel_include.py b/Documentation/sphinx/= kernel_include.py index c5f4f34e22cb..4cdd1c77982e 100755 --- a/Documentation/sphinx/kernel_include.py +++ b/Documentation/sphinx/kernel_include.py @@ -122,9 +122,6 @@ class KernelInclude(Include): exceptions_file =3D os.path.join(source_dir, self.options['exc= eption-file']) parser.process_exceptions(exceptions_file) =20 - if self.options.get("start-line") or self.options.get("end-line"): - raise self.severe('generate-cross-refs can\'t be used with "st= art-line" or "end-line"') - # Store references on a symbol dict to be used at check time if 'warn-broken' in self.options: env._xref_files.add(path) @@ -209,9 +206,6 @@ class KernelInclude(Include): =20 title =3D os.path.basename(path) =20 - if startline or endline: - raise self.severe('generate-cross-refs can\'t be used toge= ther with "start-line" or "end-line"') - if "code" not in self.options: rawtext =3D ".. parsed-literal::\n\n" + rawtext else: --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B80B83126A9; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=Dl3jvL4OCbMIGR7+ZpHzO5R6ZRto7u2HQ4LokFbToTzS94o6AoabG1lgn90gvuDibLccK/kT2Wn5JzzsUFxTphMbPJPlVOSZk7LU33SXFsGys/Mq29wdT5N66rgGoe+czMSicGYw+oSY5M3pohIhpicZXXiyw2Ce4jsA2peRR/4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=XLpGjVmiz7KWSgkDl0zwCX9wJOnCvElrKCUuUORJrj4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=RSdzuA+0ow5XOBeB+KUGPCEqkX3aH0OQP/FE+HCxcK9li3h5xUA6jhS/VAj1g2E+dauAKIyUiGE3HBatoZxr4+T9dAc/MpUACvsxQwYiMMOv+KiWxFWz4rgHRP//ANePbgyUfhhchVdkwpHCX0WrnYc2aMI1xdYnfJpKecW+wO8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=F1sGOReh; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="F1sGOReh" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 44211C2BCB8; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=XLpGjVmiz7KWSgkDl0zwCX9wJOnCvElrKCUuUORJrj4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=F1sGORehtpcfx0NvTk0TWxy4b1/C9716/JIBnVQYA2XcZl6dwmLl/aAPKjbRF4Onl GOz6J9sz/mTkMTSYlkhyE4iASqOcerz/nKWw6v8J/0Pb0ZsLzmz5r8DFe16Rwd/+2B youByWaCw7uodBK2E3wY5ksN7z3hScnKBVZFeBueFA2XEapIewoMcRk5WINRUq+QJ+ nDyz5i96sNLUcLQk73kp+JTe/SKmjqYv3u/hqq/qmfpzDXZh93wYYFeh4ZRt4oJy1x OqOTkRxoO19j5nlVc86hekohRSO0GWW5BXj1W/wMAUJTGHuiVUYzqz5feWZLgj7C6d zvParDAEiN+LQ== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCru-2DaE; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , Kees Cook , linux-kernel@vger.kernel.org Subject: [PATCH v2 16/24] docs: kernel_include.py: move code and literal functions Date: Fri, 22 Aug 2025 16:19:28 +0200 Message-ID: <78d08dfa3f08adabc30bf93b8a1cde4e19b7bd41.1755872208.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Simplify run() even more by moving the code which handles with code and literal blocks to their own functions. No functional changes. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/kernel_include.py | 100 +++++++++++++++---------- 1 file changed, 59 insertions(+), 41 deletions(-) diff --git a/Documentation/sphinx/kernel_include.py b/Documentation/sphinx/= kernel_include.py index 4cdd1c77982e..0909eb3a07ea 100755 --- a/Documentation/sphinx/kernel_include.py +++ b/Documentation/sphinx/kernel_include.py @@ -160,6 +160,52 @@ class KernelInclude(Include): =20 return rawtext =20 + def literal(self, path, tab_width, rawtext): + """Output a literal block""" + + # Convert tabs to spaces, if `tab_width` is positive. + if tab_width >=3D 0: + text =3D rawtext.expandtabs(tab_width) + else: + text =3D rawtext + literal_block =3D nodes.literal_block(rawtext, source=3Dpath, + classes=3Dself.options.get("cl= ass", [])) + literal_block.line =3D 1 + self.add_name(literal_block) + if "number-lines" in self.options: + try: + startline =3D int(self.options["number-lines"] or 1) + except ValueError: + raise self.error(":number-lines: with non-integer start va= lue") + endline =3D startline + len(include_lines) + if text.endswith("\n"): + text =3D text[:-1] + tokens =3D NumberLines([([], text)], startline, endline) + for classes, value in tokens: + if classes: + literal_block +=3D nodes.inline(value, value, + classes=3Dclasses) + else: + literal_block +=3D nodes.Text(value, value) + else: + literal_block +=3D nodes.Text(text, text) + return [literal_block] + + def code(self, path, include_lines): + """Output a code block""" + + self.options["source"] =3D path + codeblock =3D CodeBlock(self.name, + [self.options.pop("code")], # arguments + self.options, + include_lines, + self.lineno, + self.content_offset, + self.block_text, + self.state, + self.state_machine) + return codeblock.run() + def run(self): """Include a file as part of the content of this reST file.""" env =3D self.state.document.settings.env @@ -200,6 +246,13 @@ class KernelInclude(Include): startline =3D self.options.get("start-line", None) endline =3D self.options.get("end-line", None) =20 + if "literal" in self.options: + ouptut_type =3D "literal" + elif "code" in self.options: + ouptut_type =3D "code" + else: + ouptut_type =3D "normal" + # Get optional arguments to related to cross-references generation if 'generate-cross-refs' in self.options: rawtext =3D self.read_rawtext_with_xrefs(env, path) @@ -213,50 +266,15 @@ class KernelInclude(Include): =20 rawtext =3D self.apply_range(rawtext) =20 + if ouptut_type =3D=3D "literal": + return self.literal(path, tab_width, rawtext) + include_lines =3D statemachine.string2lines(rawtext, tab_width, convert_whitespace=3DTru= e) - if "literal" in self.options: - # Convert tabs to spaces, if `tab_width` is positive. - if tab_width >=3D 0: - text =3D rawtext.expandtabs(tab_width) - else: - text =3D rawtext - literal_block =3D nodes.literal_block(rawtext, source=3Dpath, - classes=3Dself.options.get= ("class", []) - ) - literal_block.line =3D 1 - self.add_name(literal_block) - if "number-lines" in self.options: - try: - startline =3D int(self.options["number-lines"] or 1) - except ValueError: - raise self.error(":number-lines: with non-integer star= t value") - endline =3D startline + len(include_lines) - if text.endswith("\n"): - text =3D text[:-1] - tokens =3D NumberLines([([], text)], startline, endline) - for classes, value in tokens: - if classes: - literal_block +=3D nodes.inline(value, value, - classes=3Dclasses) - else: - literal_block +=3D nodes.Text(value, value) - else: - literal_block +=3D nodes.Text(text, text) - return [literal_block] =20 - if "code" in self.options: - self.options["source"] =3D path - codeblock =3D CodeBlock(self.name, - [self.options.pop("code")], # arguments - self.options, - include_lines, # content - self.lineno, - self.content_offset, - self.block_text, - self.state, - self.state_machine) - return codeblock.run() + if ouptut_type =3D=3D "code": + return self.code(path, include_lines) + self.state_machine.insert_input(include_lines, path) return [] =20 --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA08C3126AC; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=DwEAUXhj1F5+w2H8mDQY9uBZ+aicIRSukiQMuld0MSubyjvPlxniSNCCbcJU39II64p5I7do99C0LlSiC36sGKYqEDdEQLEcNVb0T6/6ruA3HQjESLkPNd9r7VrgkeWLZ/BdPKdUQNuJFBL5CJltWYuZ8zDJCHzDCzwnePx2Ch0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=R6Yr4duEC8arn5hYajMGz65rXfC8mfkSt1jOlvNU8yA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=a9+5m5hMqxr9L68+CNdwGnoamiR28Wm89KgzaLrWUOlJyaYuml6JXgFqx4HZ0/NdnFE3hwChQ8m350Vsm9EKIEeYlO4ylXRyFPyuFRzafrsDsPo+JT9bWdKoQFUNZoP/mN4On2crU7x9+xREhbCYfubRdIAdZPe4WBTlNUPaf1Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ePCvR+L7; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ePCvR+L7" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4EACAC2BCC4; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=R6Yr4duEC8arn5hYajMGz65rXfC8mfkSt1jOlvNU8yA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ePCvR+L7rN7B+iSOPEv1/U0vt/52uYLktgOFffaLMwGGvHc1j6sSfoFRaOL06vkKD mQseree+rD5dqsdkuT4aNoCPLR82MZdPSaW/MAq1s3yiJ/7+F+Yi0PS2VznA5XFScS bolHDFfq04eQFc3AADsjI7S9ZEKAfSY1Wmaw8lJl767MSktf6dQtqG0rB9QB1kl9Jy jlVs+AnSlrwZC0Ai6XkNgOact86pMye9AvNHl2/284FsmPhWuPrRpeHsznovStUAWO SIrpP/D2c05UNDpu62uFDKZ1zBZ6b/hXWQ+2MmJr4jZcE1SV8kJqAQGrHYQZf8Ehf2 LBCVNg69x0fBw== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCrz-2KH0; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , Kees Cook , linux-kernel@vger.kernel.org Subject: [PATCH v2 17/24] docs: kernel_include.py: add support to generate a TOC table Date: Fri, 22 Aug 2025 16:19:29 +0200 Message-ID: X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" When generate-cross-refs is used, instead of just implementing the default of generating a literal block, we can also generate a ReST file as a TOC. The advantage is that, by being a ReST file, missing references will point to the place inside the header file that has the broken link. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/kernel_include.py | 36 ++++++++++++++++---------- 1 file changed, 22 insertions(+), 14 deletions(-) diff --git a/Documentation/sphinx/kernel_include.py b/Documentation/sphinx/= kernel_include.py index 0909eb3a07ea..79682408105e 100755 --- a/Documentation/sphinx/kernel_include.py +++ b/Documentation/sphinx/kernel_include.py @@ -89,6 +89,7 @@ class KernelInclude(Include): option_spec.update({ 'generate-cross-refs': directives.flag, 'warn-broken': directives.flag, + 'toc': directives.flag, 'exception-file': directives.unchanged, }) =20 @@ -111,7 +112,7 @@ class KernelInclude(Include): except UnicodeError as error: raise self.severe('Problem with directive:\n%s' % ErrorStr= ing(error)) =20 - def read_rawtext_with_xrefs(self, env, path): + def read_rawtext_with_xrefs(self, env, path, output_type): parser =3D ParseDataStructs() parser.parse_file(path) =20 @@ -126,7 +127,10 @@ class KernelInclude(Include): if 'warn-broken' in self.options: env._xref_files.add(path) =20 - return parser.gen_output() + if output_type =3D=3D "toc": + return parser.gen_toc() + + return ".. parsed-literal::\n\n" + parser.gen_output() =20 def apply_range(self, rawtext): # Get to-be-included content @@ -243,39 +247,43 @@ class KernelInclude(Include): e_handler =3D self.state.document.settings.input_encoding_error_ha= ndler tab_width =3D self.options.get("tab-width", self.state.document.settings.tab_widt= h) - startline =3D self.options.get("start-line", None) - endline =3D self.options.get("end-line", None) =20 if "literal" in self.options: - ouptut_type =3D "literal" + output_type =3D "literal" elif "code" in self.options: - ouptut_type =3D "code" + output_type =3D "code" else: - ouptut_type =3D "normal" + output_type =3D "rst" =20 # Get optional arguments to related to cross-references generation - if 'generate-cross-refs' in self.options: - rawtext =3D self.read_rawtext_with_xrefs(env, path) + if "generate-cross-refs" in self.options: + if "toc" in self.options: + output_type =3D "toc" + + rawtext =3D self.read_rawtext_with_xrefs(env, path, output_typ= e) + + # When :generate-cross-refs: is used, the input is always a C + # file, so it has to be handled as a parsed-literal + if output_type =3D=3D "rst": + output_type =3D "literal" =20 title =3D os.path.basename(path) - - if "code" not in self.options: - rawtext =3D ".. parsed-literal::\n\n" + rawtext else: rawtext =3D self.read_rawtext(path, encoding) =20 rawtext =3D self.apply_range(rawtext) =20 - if ouptut_type =3D=3D "literal": + if output_type =3D=3D "literal": return self.literal(path, tab_width, rawtext) =20 include_lines =3D statemachine.string2lines(rawtext, tab_width, convert_whitespace=3DTru= e) =20 - if ouptut_type =3D=3D "code": + if output_type =3D=3D "code": return self.code(path, include_lines) =20 self.state_machine.insert_input(include_lines, path) + return [] =20 # =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BA1503126AF; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=b0GsjUDmTNNnDk81RcTNoyllu2mWWhHZ2TnBnpxAdEHYJVBG7BbMUY0QmIPdueTxBlBKliKc2DTYZ5dXwP1oDf2RK47vy3wisnYQqn4ytuclhGpHUiAtwaLGVeVLMKUprtTHzaFvR8XQBqQn4a3BVQWsKZmkYkPjQKDYu6S0F4o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=Ryz5fK1HBEzZjQk/ZMX2f1BMYW5Jfa7MSrJhKnqCZ5k=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lqn/XCLvQYelnQ9HGpMNx8tkrml7qsZkHTy8ZvHFQLi0ZsCr2eXqj5csqtLFqwZ+hon29ymcJ6wxL9/ItXEwVSb2UivGbQjxdrSWmQpTCo9/MKpvnI6XM7BN0iQEJIfLJh2MQQ4MCZV4FzTzRfF4U9LKHAb0UdymjQaZB3nSsXA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=nMMdXJGY; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="nMMdXJGY" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4C6ADC2BCC7; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=Ryz5fK1HBEzZjQk/ZMX2f1BMYW5Jfa7MSrJhKnqCZ5k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nMMdXJGYKSGznKAOoExjtZnUlnzsuvWOZ58YcVgDJwmaO5DgLCT3+BP9Mm8mmLMgl Ma1pcMrsYrT7/GwlK0f5J739qN/lNkyJR85DFVf0ZyapseGD/y/s0/TO+35f3YJgxi 4HiU4pXLEhwEY04fJHI8aiAV/LEzTHgM3CWWfAIoRjbSrUMqiO6IspVWDdlCikNLU4 XG77DfGxiWITgG9HyQ1xCuN9/EIq9Q/DY8MsRjudJ9UcqEOMpWnzbkQLtxYSEqN1PX P+ByQ1/AyT6DkntItl/lFSAK2PSgfK334goUwnz8GVeF+UpelsWPOfMgCRYNDvYHMp wppBktPGXCD6Q== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCs3-2R59; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , Kees Cook , linux-kernel@vger.kernel.org Subject: [PATCH v2 18/24] docs: kernel_include.py: append line numbers to better report errors Date: Fri, 22 Aug 2025 16:19:30 +0200 Message-ID: X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" It is best to point to the original line of code that generated an error than to point to the beginning of a directive. Add support for it. It should be noticed that this won't work for literal or code blocks, as Sphinx will ignore it, pointing to the beginning of the directive. Yet, when the output is known to be in ReST format, like on TOC, this makes the error a lot more easier to be handled. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/kernel_include.py | 81 ++++++++++++++------------ 1 file changed, 44 insertions(+), 37 deletions(-) diff --git a/Documentation/sphinx/kernel_include.py b/Documentation/sphinx/= kernel_include.py index 79682408105e..90ed8428f776 100755 --- a/Documentation/sphinx/kernel_include.py +++ b/Documentation/sphinx/kernel_include.py @@ -60,6 +60,7 @@ import re import sys =20 from docutils import io, nodes, statemachine +from docutils.statemachine import ViewList from docutils.utils.error_reporting import SafeString, ErrorString from docutils.parsers.rst import directives from docutils.parsers.rst.directives.body import CodeBlock, NumberLines @@ -112,7 +113,14 @@ class KernelInclude(Include): except UnicodeError as error: raise self.severe('Problem with directive:\n%s' % ErrorStr= ing(error)) =20 - def read_rawtext_with_xrefs(self, env, path, output_type): + def xref_text(self, env, path, tab_width): + """ + Read and add contents from a C file parsed to have cross reference= s. + + There are two types of supported output here: + - A C source code with cross-references; + - a TOC table containing cross references. + """ parser =3D ParseDataStructs() parser.parse_file(path) =20 @@ -127,10 +135,33 @@ class KernelInclude(Include): if 'warn-broken' in self.options: env._xref_files.add(path) =20 - if output_type =3D=3D "toc": - return parser.gen_toc() + if "toc" in self.options: + rawtext =3D parser.gen_toc() + else: + rawtext =3D ".. parsed-literal::\n\n" + parser.gen_output() + self.apply_range(rawtext) =20 - return ".. parsed-literal::\n\n" + parser.gen_output() + title =3D os.path.basename(path) + + include_lines =3D statemachine.string2lines(rawtext, tab_width, + convert_whitespace=3DTru= e) + + # Append line numbers data + + startline =3D self.options.get('start-line', None) + + result =3D ViewList() + if startline and startline > 0: + offset =3D startline - 1 + else: + offset =3D 0 + + for ln, line in enumerate(include_lines, start=3Doffset): + result.append(line, path, ln) + + self.state_machine.insert_input(result, path) + + return [] =20 def apply_range(self, rawtext): # Get to-be-included content @@ -195,9 +226,12 @@ class KernelInclude(Include): literal_block +=3D nodes.Text(text, text) return [literal_block] =20 - def code(self, path, include_lines): + def code(self, path, tab_width): """Output a code block""" =20 + include_lines =3D statemachine.string2lines(rawtext, tab_width, + convert_whitespace=3DTru= e) + self.options["source"] =3D path codeblock =3D CodeBlock(self.name, [self.options.pop("code")], # arguments @@ -244,47 +278,20 @@ class KernelInclude(Include): =20 encoding =3D self.options.get("encoding", self.state.document.settings.input_enc= oding) - e_handler =3D self.state.document.settings.input_encoding_error_ha= ndler tab_width =3D self.options.get("tab-width", self.state.document.settings.tab_widt= h) =20 - if "literal" in self.options: - output_type =3D "literal" - elif "code" in self.options: - output_type =3D "code" - else: - output_type =3D "rst" - # Get optional arguments to related to cross-references generation if "generate-cross-refs" in self.options: - if "toc" in self.options: - output_type =3D "toc" - - rawtext =3D self.read_rawtext_with_xrefs(env, path, output_typ= e) - - # When :generate-cross-refs: is used, the input is always a C - # file, so it has to be handled as a parsed-literal - if output_type =3D=3D "rst": - output_type =3D "literal" - - title =3D os.path.basename(path) - else: - rawtext =3D self.read_rawtext(path, encoding) + return self.xref_text(env, path, tab_width) =20 + rawtext =3D self.read_rawtext(path, encoding) rawtext =3D self.apply_range(rawtext) =20 - if output_type =3D=3D "literal": - return self.literal(path, tab_width, rawtext) + if "code" in self.options: + return self.code(path, tab_width, rawtext) =20 - include_lines =3D statemachine.string2lines(rawtext, tab_width, - convert_whitespace=3DTru= e) - - if output_type =3D=3D "code": - return self.code(path, include_lines) - - self.state_machine.insert_input(include_lines, path) - - return [] + return self.literal(path, tab_width, rawtext) =20 # =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D =20 --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CCDC53126B6; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=qLf2i4Dh4QFJCLqnTlQZ0IKI0VM7+PL8MOumJTlyAaymiGVrCK9oSS3sEPzZiipRjgLgAWr3UWQi4QoAiFRjFbSd3Qeq+6qtXKPaKjo5qXcYFUSGZZWNu6S6nR/rDsg7KWh/FlW/oRcV4bNCBnyRM6PPSBR7qf/RkEeWjyYoasA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=Ws1QXmr61TBgn6INp6mbk145NP3+6J63X8KSGxb87Js=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=h35OQ33KyF7+nPnjMP1jJT1Z5OiTHzF1iyBH7JkxYZXbRig9WQ/qDehPuPTur9yfcpetpShcVdtk1HJS7YMdXu+1AM8eKVBbqb/egxoyEogCYZbbp7sjEv+l3sousKYQITnT1z9H7k8NycBYD9fkkvz20Kv1EuvSH3Ro1RCiMko= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=E/UmZz7y; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="E/UmZz7y" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5EF21C2BCFB; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=Ws1QXmr61TBgn6INp6mbk145NP3+6J63X8KSGxb87Js=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=E/UmZz7yAOustPWvl4KdxNa10k9SKWpsVEgVCBHWWf2qeF9aIrTQJl7w7HcQAXpx2 lhK5ujvC58DtAbc3i3mwHico7q3QrXurJ00wVtE7oCYB6MWdzf8s40abNQdyogKjTo nyCJYRTZzy3O+8AIyOT2vX4RXmtnlm+BVE8cvKftd0TVeYfU0DnDDBjurqXhdqUeDf oFMKr1E8bq/SvcUYNN3fCvRpLafkkhIbCBpgCNQuraAoNAz20XcYPSd7+8wAB+0bwC n47qFXAld4ObKSRDYjBRDmdpX5pp8uIsbrTVMJ6rz9XZTIOdOnV6WgRUOykCWHk2Sy 8asOz4S2JOfwg== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCs7-2Xu1; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , Kees Cook , linux-kernel@vger.kernel.org Subject: [PATCH v2 19/24] docs: kernel_include.py: move apply_range() and add a docstring Date: Fri, 22 Aug 2025 16:19:31 +0200 Message-ID: X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" While not required, better to have caller functions at the end. As apply_range() is now called by xref_text(), move it to be before the latter. No functional changes. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/kernel_include.py | 68 ++++++++++++++------------ 1 file changed, 36 insertions(+), 32 deletions(-) diff --git a/Documentation/sphinx/kernel_include.py b/Documentation/sphinx/= kernel_include.py index 90ed8428f776..fd4887f80577 100755 --- a/Documentation/sphinx/kernel_include.py +++ b/Documentation/sphinx/kernel_include.py @@ -113,6 +113,42 @@ class KernelInclude(Include): except UnicodeError as error: raise self.severe('Problem with directive:\n%s' % ErrorStr= ing(error)) =20 + def apply_range(self, rawtext): + """ + Handles start-line, end-line, start-after and end-before parameters + """ + + # Get to-be-included content + startline =3D self.options.get('start-line', None) + endline =3D self.options.get('end-line', None) + try: + if startline or (endline is not None): + lines =3D rawtext.splitlines() + rawtext =3D '\n'.join(lines[startline:endline]) + except UnicodeError as error: + raise self.severe(f'Problem with "{self.name}" directive:\n' + + io.error_string(error)) + # start-after/end-before: no restrictions on newlines in match-tex= t, + # and no restrictions on matching inside lines vs. line boundaries + after_text =3D self.options.get("start-after", None) + if after_text: + # skip content in rawtext before *and incl.* a matching text + after_index =3D rawtext.find(after_text) + if after_index < 0: + raise self.severe('Problem with "start-after" option of "%= s" ' + "directive:\nText not found." % self.nam= e) + rawtext =3D rawtext[after_index + len(after_text) :] + before_text =3D self.options.get("end-before", None) + if before_text: + # skip content in rawtext after *and incl.* a matching text + before_index =3D rawtext.find(before_text) + if before_index < 0: + raise self.severe('Problem with "end-before" option of "%s= " ' + "directive:\nText not found." % self.nam= e) + rawtext =3D rawtext[:before_index] + + return rawtext + def xref_text(self, env, path, tab_width): """ Read and add contents from a C file parsed to have cross reference= s. @@ -163,38 +199,6 @@ class KernelInclude(Include): =20 return [] =20 - def apply_range(self, rawtext): - # Get to-be-included content - startline =3D self.options.get('start-line', None) - endline =3D self.options.get('end-line', None) - try: - if startline or (endline is not None): - lines =3D rawtext.splitlines() - rawtext =3D '\n'.join(lines[startline:endline]) - except UnicodeError as error: - raise self.severe(f'Problem with "{self.name}" directive:\n' - + io.error_string(error)) - # start-after/end-before: no restrictions on newlines in match-tex= t, - # and no restrictions on matching inside lines vs. line boundaries - after_text =3D self.options.get("start-after", None) - if after_text: - # skip content in rawtext before *and incl.* a matching text - after_index =3D rawtext.find(after_text) - if after_index < 0: - raise self.severe('Problem with "start-after" option of "%= s" ' - "directive:\nText not found." % self.nam= e) - rawtext =3D rawtext[after_index + len(after_text) :] - before_text =3D self.options.get("end-before", None) - if before_text: - # skip content in rawtext after *and incl.* a matching text - before_index =3D rawtext.find(before_text) - if before_index < 0: - raise self.severe('Problem with "end-before" option of "%s= " ' - "directive:\nText not found." % self.nam= e) - rawtext =3D rawtext[:before_index] - - return rawtext - def literal(self, path, tab_width, rawtext): """Output a literal block""" =20 --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 11694312806; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872389; cv=none; b=N7Nq4vZ0e23o3QDPeB5fn18MmhXjfWjKGlARjkPrAALj2BaF/m/Ja+N7CjmVLNVkKi2rv/DYnHPA9iGAh8u94eq0jINM2Ogl0lmrbFa/QdN7nwCE4dAJYxie7DnvQcHA0k+wEfH4n6XTR6GMIWAcE9AIb/QrJv9MNLUFjuHyqhQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872389; c=relaxed/simple; bh=TiBwJFe/nRO7mW633U4ruwZsiS7f5kmzwOmkA8jDJ8g=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=EzgrHagGB8MnaLjKZgHejJ945dr157sBl4vJZdQ8ws/KGLbvlrFtoHncd2EScQvK2lTNZwUHtQrP1Ab96PepeJ5lu5kjS2QIKpCYxprx922QBa1tkyNAC/mEaQqx6F4ImD6uNSe3Hpa5PcrOkTLvfg0FE6RHcz8xwAOrOSAE01o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=cnZoaR8m; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="cnZoaR8m" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6CF74C4AF0B; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=TiBwJFe/nRO7mW633U4ruwZsiS7f5kmzwOmkA8jDJ8g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cnZoaR8mkqWdkhXogG36OiOzzru0Q6F7fY9vMqdWMxvZ31bkmVwl52w7x7dWtmhaY JkbG2COElI/BIJMRBEwcjkL0vWWFJkuHvVpOhhtYS66oVh9Q730sNv1q3C6NAroF8N 4HuhUUIwa8/5Gpj1kdzENRtehmRX2VV9AxlepiaNx14v7uBuUBTWoNZBR9iCqOTuYM IcM7nOvztBIdGuxbbJ4uaPSv5BjUoV2mfML0QrF+atwB0sssFFHpy7VFzG/ub2O2nk TGxXySWuxgMNWLeM4guToi46M8saTRq0gSkN+OCyhEYLibHvIVa6UyeIVXJrbcPDdo ByYC5yMsFBJ8w== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCsB-2ebD; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , Kees Cook , linux-kernel@vger.kernel.org Subject: [PATCH v2 20/24] docs: kernel_include.py: remove line numbers from parsed-literal Date: Fri, 22 Aug 2025 16:19:32 +0200 Message-ID: X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" When parsed-literal directive is added to rawtext, while cross references will be properly displayed, Sphinx will ignore line numbers. So, it is not worth adding them. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/kernel_include.py | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/Documentation/sphinx/kernel_include.py b/Documentation/sphinx/= kernel_include.py index fd4887f80577..3a1753486319 100755 --- a/Documentation/sphinx/kernel_include.py +++ b/Documentation/sphinx/kernel_include.py @@ -171,13 +171,24 @@ class KernelInclude(Include): if 'warn-broken' in self.options: env._xref_files.add(path) =20 - if "toc" in self.options: - rawtext =3D parser.gen_toc() - else: + if "toc" not in self.options: + rawtext =3D ".. parsed-literal::\n\n" + parser.gen_output() self.apply_range(rawtext) =20 - title =3D os.path.basename(path) + include_lines =3D statemachine.string2lines(rawtext, tab_width, + convert_whitespace= =3DTrue) + + # Sphinx always blame the ".. ", so placing + # line numbers here won't make any difference + + self.state_machine.insert_input(include_lines, path) + return [] + + # TOC output is a ReST file, not a literal. So, we can add line + # numbers + + rawtext =3D parser.gen_toc() =20 include_lines =3D statemachine.string2lines(rawtext, tab_width, convert_whitespace=3DTru= e) --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CCEAE3126B7; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; cv=none; b=cy566oj0EOVUNA+hq+Pp6bQjXm4OvVR/0scoKTrCFYVjlfD06EolNpLP98hdjYLZSzJyTPaM+iMz1pWgEWXt8JMYfXM3c4OHntzdaIrbouEsVOoJfkUly3y9b731ASuY68mZ4Vt6H1lQEjxFphxJ4A+EYyL7QxIJCQsELzvnhVA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872388; c=relaxed/simple; bh=LQfriqJ7w45m1TsmRVR1XR73bImTQyUEqk93YtSCuzA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AahTBolZ4UI8WHap5/MO3ulpbY7vwMhHQYfAGbFJBG/8YpnLpVD6GA9sfZKfID4PkKCk1GmvZRQXkGJwEmflKUxxHoK1gbltev9smLjkAzBEMG6GT09T306XnGljw7B0Gex2fy4y/OsG+gaDqctbyYH1HqGqga2NBdwGmmjknWA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=us9Gpwf9; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="us9Gpwf9" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6833FC2BCFD; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=LQfriqJ7w45m1TsmRVR1XR73bImTQyUEqk93YtSCuzA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=us9Gpwf99h+qvYS/a+8T+1/mfs4B3xHFRUUQWYt2QgV1GiRQInibaRyMEGkKnlE/h FU338kCo1P6ySBNunbIfM7vpS/w50qpMq8ljjjRcfhogHMSVBf8r/3irGEnT0b3o8Z w1khWMvS7D4upMsD7P4Ls2JWNc14cgItnvE0AmeqpkAbQltjFzf8WxNV0G94XfCwCA 6wgNNybY1/yaD3qQuLpIBcwWWVJ7W3Y34L7hQ9vun9VS5OHA5nhQGCmMZ1yX2xOVQJ dVYe+xPExniLau5uJDFAEqKIsmv/kJDlD9foII7bZa4k8IwrHfZvBkERAxwkLDDpAS sG3ncmpLAHKmw== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCsF-2lMI; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , Kees Cook , linux-kernel@vger.kernel.org Subject: [PATCH v2 21/24] docs: kernel_include.py: remove Include class inheritance Date: Fri, 22 Aug 2025 16:19:33 +0200 Message-ID: X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" While the original code came from the Sphinx Include class, such class is monolithic: it has only one function that does everything, and 3 variables that are used: - required_arguments - optional_arguments - option_spec So, basically those are the only members that remain from the original class, but hey! Those are the same vars that every other Sphinx directive extension has to define! In summary, keeping inheritance here doesn't make much sense. Worse than that, kernel-include doesn't support the current set of options that the original Include class has, but it also has its own set of options. So, let's fill in the argument vars with what it does support, dropping the rest. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/kernel_include.py | 40 ++++++++++++++++++++------ 1 file changed, 32 insertions(+), 8 deletions(-) diff --git a/Documentation/sphinx/kernel_include.py b/Documentation/sphinx/= kernel_include.py index 3a1753486319..e6f734476ab3 100755 --- a/Documentation/sphinx/kernel_include.py +++ b/Documentation/sphinx/kernel_include.py @@ -62,9 +62,8 @@ import sys from docutils import io, nodes, statemachine from docutils.statemachine import ViewList from docutils.utils.error_reporting import SafeString, ErrorString -from docutils.parsers.rst import directives +from docutils.parsers.rst import Directive, directives from docutils.parsers.rst.directives.body import CodeBlock, NumberLines -from docutils.parsers.rst.directives.misc import Include =20 from sphinx.util import logging =20 @@ -81,18 +80,43 @@ RE_SIMPLE_REF =3D re.compile(r'`([^`]+)`') =20 =20 # =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D -class KernelInclude(Include): - """KernelInclude (``kernel-include``) directive""" +class KernelInclude(Directive): + """ + KernelInclude (``kernel-include``) directive =20 - # Add extra options - option_spec =3D Include.option_spec.copy() + Most of the stuff here came from Include directive defined at: + docutils/parsers/rst/directives/misc.py =20 - option_spec.update({ + Yet, overriding the class don't has any benefits: the original class + only have run() and argument list. Not all of them are implemented, + when checked against latest Sphinx version, as with time more arguments + were added. + + So, keep its own list of supported arguments + """ + + required_arguments =3D 1 + optional_arguments =3D 0 + final_argument_whitespace =3D True + option_spec =3D { + 'literal': directives.flag, + 'code': directives.unchanged, + 'encoding': directives.encoding, + 'tab-width': int, + 'start-line': int, + 'end-line': int, + 'start-after': directives.unchanged_required, + 'end-before': directives.unchanged_required, + # ignored except for 'literal' or 'code': + 'number-lines': directives.unchanged, # integer or None + 'class': directives.class_option, + + # Arguments that aren't from Sphinx Include directive 'generate-cross-refs': directives.flag, 'warn-broken': directives.flag, 'toc': directives.flag, 'exception-file': directives.unchanged, - }) + } =20 def read_rawtext(self, path, encoding): """Read and process file content with error handling""" --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8BD23126C2; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872389; cv=none; b=D6D5jEZ1rUxhaprfokIhexTbr7azR6OTvzV9zF6RV6XkOb2zoF7DonV+BNzir6pAsyRFBDa8PdMELEkK2peShwqB7ZgXpGw7/r3bFeFpaTQ6n7K2HnS7kaMTHi9M7xSPsZvAGlBRd5dNRYcDRtSgxnkKSd5Up7prNj/V3zjk7aA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872389; c=relaxed/simple; bh=LbPxF3kGKvZLO+UE6KiArHqAGzi0BSWTiNNAdqSOJS4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Xw6+zId8Lcj9b0VGX7qaptNXiApxT1ziem6y+CITVypOUtLhOMRjzUUIBprOLLT0tM2/RC3lPSSUNsxvS9dl7u5Ku1FVDTwqV8+EMP1zjZGxU+DHRRZkrDd/HhmCvKHFvZ/96Ulw6CLbTkH0fwjS3d8G1oLtry9hS3hjHGs0y0E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=TVDoFaa1; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="TVDoFaa1" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6A5A5C2BD00; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=LbPxF3kGKvZLO+UE6KiArHqAGzi0BSWTiNNAdqSOJS4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TVDoFaa1Qv0iEfJuNlXujfu/uy8r4fTD8UGniLApJSl/Y8DJ2uQY20tUG/7ZSpnVw Sz0NaaL4psw2UZ5vs8BBV7tlx0zeYBu3N/n3bPZbkLAk84WZBRsIimfrMGzTholxoh 9pMKVqr28VeNtuOyQEjXchF0/Bzh5ZVVat5jnKGPwwYOuqJTdkRrEGFpAfQLBuKjY7 y+S53LC/0JbVh/xjJRmEtLylNr6yaWy8oSMe/2LLhlUTzIvrTLywNegT9euyPGp0lN BsT1KX27vTSsb7ZUGIwuDeQqEX59lD2UvNWzONOyforSncG8lbhmViC/HgvDxjAHRk cm0td/GkcHT3w== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCsJ-2s9a; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , Kees Cook , linux-kernel@vger.kernel.org Subject: [PATCH v2 22/24] docs: kernel_include.py: document all supported parameters Date: Fri, 22 Aug 2025 16:19:34 +0200 Message-ID: X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" As we're actually a fork of Sphinx Include, update its docstring to contain the documentation for the actual implemented parameters. Let's use :param: for parameters, as defined at: https://sphinx-rtd-tutorial.readthedocs.io/en/latest/docstrings.html Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/kernel_include.py | 88 +++++++++++++++++--------- 1 file changed, 58 insertions(+), 30 deletions(-) diff --git a/Documentation/sphinx/kernel_include.py b/Documentation/sphinx/= kernel_include.py index e6f734476ab3..23566ab74866 100755 --- a/Documentation/sphinx/kernel_include.py +++ b/Documentation/sphinx/kernel_include.py @@ -2,53 +2,81 @@ # SPDX-License-Identifier: GPL-2.0 # pylint: disable=3DR0903, R0912, R0914, R0915, C0209,W0707 =20 + """ - kernel-include - ~~~~~~~~~~~~~~ +Implementation of the ``kernel-include`` reST-directive. =20 - Implementation of the ``kernel-include`` reST-directive. +:copyright: Copyright (C) 2016 Markus Heiser +:license: GPL Version 2, June 1991 see linux/COPYING for details. =20 - :copyright: Copyright (C) 2016 Markus Heiser - :license: GPL Version 2, June 1991 see linux/COPYING for details. +The ``kernel-include`` reST-directive is a replacement for the ``include`` +directive. The ``kernel-include`` directive expand environment variables in +the path name and allows to include files from arbitrary locations. =20 - The ``kernel-include`` reST-directive is a replacement for the ``inclu= de`` - directive. The ``kernel-include`` directive expand environment variabl= es in - the path name and allows to include files from arbitrary locations. +.. hint:: =20 - .. hint:: + Including files from arbitrary locations (e.g. from ``/etc``) is a + security risk for builders. This is why the ``include`` directive from + docutils *prohibit* pathnames pointing to locations *above* the filesy= stem + tree where the reST document with the include directive is placed. =20 - Including files from arbitrary locations (e.g. from ``/etc``) is a - security risk for builders. This is why the ``include`` directive fr= om - docutils *prohibit* pathnames pointing to locations *above* the file= system - tree where the reST document with the include directive is placed. +Substrings of the form $name or ${name} are replaced by the value of +environment variable name. Malformed variable names and references to +non-existing variables are left unchanged. =20 - Substrings of the form $name or ${name} are replaced by the value of - environment variable name. Malformed variable names and references to - non-existing variables are left unchanged. +**Supported Sphinx Include Options**: =20 - This extension overrides Sphinx include directory, adding some extra - arguments: +:param literal: + If present, the included file is inserted as a literal block. =20 - 1. :generate-cross-refs: +:param code: + Specify the language for syntax highlighting (e.g., 'c', 'python'). =20 - If present, instead of reading the file, it calls ParseDataStructs= () - class, which converts C data structures into cross-references to - be linked to ReST files containing a more comprehensive documentat= ion; +:param encoding: + Specify the encoding of the included file (default: 'utf-8'). =20 - 2. :exception-file: +:param tab-width: + Specify the number of spaces that a tab represents. =20 - Used together with :generate-cross-refs +:param start-line: + Line number at which to start including the file (1-based). =20 - Points to a file containing rules to ignore C data structs or to - use a different reference name, optionally using a different - reference type. +:param end-line: + Line number at which to stop including the file (inclusive). =20 - 3. :warn-broken: +:param start-after: + Include lines after the first line matching this text. =20 - Used together with :generate-cross-refs: +:param end-before: + Include lines before the first line matching this text. =20 - Detect if the auto-generated cross references doesn't exist. +:param number-lines: + Number the included lines (integer specifies start number). + Only effective with 'literal' or 'code' options. =20 +:param class: + Specify HTML class attribute for the included content. + +**Kernel-specific Extensions**: + +:param generate-cross-refs: + If present, instead of directly including the file, it calls + ParseDataStructs() to convert C data structures into cross-references + that link to comprehensive documentation in other ReST files. + +:param exception-file: + (Used with generate-cross-refs) + + Path to a file containing rules for handling special cases: + - Ignore specific C data structures + - Use alternative reference names + - Specify different reference types + +:param warn-broken: + (Used with generate-cross-refs) + + Enables warnings when auto-generated cross-references don't point to + existing documentation targets. """ =20 # =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E7E433126D0; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872389; cv=none; b=Wv4g0PdZC8QnYF9s5BUyy556FdFjLjTpoqPbKtrfAXLHQ0xB+ZO1U4cqsQC1zQ/uUPAqLEQ0MEcPRpBJyL6JnVJktYB2kdVgI5K5OwiI0pegJzEpBilDV258FBVzEXbBaP4h1tJzylvi1JaJs1ff/0W6Vk9Wh3Q2egW/nD5Fqcc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872389; c=relaxed/simple; bh=bIP/wRP3b422VCDjvrgxR9/Zg0ckPnkkNpui1ZjXpUU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DGAFGhNz1KwvxbYzZovZkrtv75lBxIoBZA/1cv9EOlNWx8MX9UPkzCDP1iSZ0ml8T5URCiSaHAGPWUsLmGupRXh2KaL+jl/m9KGDvieh6ojcsKLL/F1/YkwXiaSp26JAfEey7pNio+4PZYb/PTKoXRzC8rrZg4OTgh4AAz8v8IQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=AltfQXcs; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="AltfQXcs" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 71AB1C2BD01; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=bIP/wRP3b422VCDjvrgxR9/Zg0ckPnkkNpui1ZjXpUU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AltfQXcsWcqXeCL5zo+KzuqiIWeX1U+NQjsEqTgzJXkLe+dx+7GJznuVEH+IVGUnW W649ffBV7iJZe2d7pfYmOqrBZH4DSFsjk8t645yNDtfW1Z2EoH7/A1ZVy5Qb6xSWzg EbLl/DQNne0x7Xtya3hxLDVinkSFPgDsN3mDr2hCPgjGSzMpeM/a1QgnlmpdcuaPia uHP3IUgWYrbybESt4/vud9zEPeVDV3Rao67o8s6E1qWtpfTRlhgQGASr5IzR+1CSEW H+brM+rOGAwOdkUAFxmIhEvCOdU/ce3RRKmxUTGVSLYK7TFZ2HZ/EGtBzCsMj1mdS5 lh4tWumW4qPuw== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCsO-3068; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , =?UTF-8?q?Bj=C3=B6rn=20Roy=20Baron?= , "Mauro Carvalho Chehab" , Alex Gaynor , Alice Ryhl , Andreas Hindborg , Benjamin Gaignard , Benno Lossin , Boqun Feng , Danilo Krummrich , Erling Ljunggren , Gary Guo , Hans Verkuil , Hans de Goede , Miguel Ojeda , Ricardo Ribalda , Sean Young , Trevor Gross , Yunke Cao , linux-kernel@vger.kernel.org, linux-media@vger.kernel.org, rust-for-linux@vger.kernel.org Subject: [PATCH v2 23/24] scripts: sphinx-build-wrapper: get rid of uapi/media Makefile Date: Fri, 22 Aug 2025 16:19:35 +0200 Message-ID: <5dbb257a4b283697271c9c7b8f4713857e8191c8.1755872208.git.mchehab+huawei@kernel.org> X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Now that kernel-include directive supports parsing data structs directly, we can finally get rid of the horrible hack we added to support parsing media uAPI symbols. As a side effect, Documentation/output doesn't have anymore media auto-generated .rst files on it. Signed-off-by: Mauro Carvalho Chehab --- Documentation/Makefile | 3 +- Documentation/userspace-api/media/Makefile | 64 -- .../userspace-api/media/cec/cec-header.rst | 5 +- .../media/{ =3D> cec}/cec.h.rst.exceptions | 0 .../media/{ =3D> dvb}/ca.h.rst.exceptions | 0 .../media/{ =3D> dvb}/dmx.h.rst.exceptions | 0 .../media/{ =3D> dvb}/frontend.h.rst.exceptions | 0 .../userspace-api/media/dvb/headers.rst | 17 +- .../media/{ =3D> dvb}/net.h.rst.exceptions | 0 .../media/mediactl/media-header.rst | 5 +- .../{ =3D> mediactl}/media.h.rst.exceptions | 0 .../userspace-api/media/rc/lirc-header.rst | 4 +- .../media/{ =3D> rc}/lirc.h.rst.exceptions | 0 .../userspace-api/media/v4l/videodev.rst | 4 +- .../{ =3D> v4l}/videodev2.h.rst.exceptions | 0 scripts/sphinx-build-wrapper | 719 ++++++++++++++++++ 16 files changed, 745 insertions(+), 76 deletions(-) delete mode 100644 Documentation/userspace-api/media/Makefile rename Documentation/userspace-api/media/{ =3D> cec}/cec.h.rst.exceptions = (100%) rename Documentation/userspace-api/media/{ =3D> dvb}/ca.h.rst.exceptions (= 100%) rename Documentation/userspace-api/media/{ =3D> dvb}/dmx.h.rst.exceptions = (100%) rename Documentation/userspace-api/media/{ =3D> dvb}/frontend.h.rst.except= ions (100%) rename Documentation/userspace-api/media/{ =3D> dvb}/net.h.rst.exceptions = (100%) rename Documentation/userspace-api/media/{ =3D> mediactl}/media.h.rst.exce= ptions (100%) rename Documentation/userspace-api/media/{ =3D> rc}/lirc.h.rst.exceptions = (100%) rename Documentation/userspace-api/media/{ =3D> v4l}/videodev2.h.rst.excep= tions (100%) create mode 100755 scripts/sphinx-build-wrapper diff --git a/Documentation/Makefile b/Documentation/Makefile index 2ed334971acd..5c20c68be89a 100644 --- a/Documentation/Makefile +++ b/Documentation/Makefile @@ -87,7 +87,7 @@ loop_cmd =3D $(echo-cmd) $(cmd_$(1)) || exit; PYTHONPYCACHEPREFIX ?=3D $(abspath $(BUILDDIR)/__pycache__) =20 quiet_cmd_sphinx =3D SPHINX $@ --> file://$(abspath $(BUILDDIR)/$3/$4) - cmd_sphinx =3D $(MAKE) BUILDDIR=3D$(abspath $(BUILDDIR)) $(build)=3D= Documentation/userspace-api/media $2 && \ + cmd_sphinx =3D \ PYTHONPYCACHEPREFIX=3D"$(PYTHONPYCACHEPREFIX)" \ BUILDDIR=3D$(abspath $(BUILDDIR)) SPHINX_CONF=3D$(abspath $(src)/$5/$(SPH= INX_CONF)) \ $(PYTHON3) $(srctree)/scripts/jobserver-exec \ @@ -171,7 +171,6 @@ refcheckdocs: =20 cleandocs: $(Q)rm -rf $(BUILDDIR) - $(Q)$(MAKE) BUILDDIR=3D$(abspath $(BUILDDIR)) $(build)=3DDocumentation/us= erspace-api/media clean =20 dochelp: @echo ' Linux kernel internal documentation in different formats from Re= ST:' diff --git a/Documentation/userspace-api/media/Makefile b/Documentation/use= rspace-api/media/Makefile deleted file mode 100644 index accc734d045a..000000000000 --- a/Documentation/userspace-api/media/Makefile +++ /dev/null @@ -1,64 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0 - -# Rules to convert a .h file to inline RST documentation - -SRC_DIR=3D$(srctree)/Documentation/userspace-api/media -PARSER =3D $(srctree)/tools/docs/parse-headers.py -UAPI =3D $(srctree)/include/uapi/linux -KAPI =3D $(srctree)/include/linux - -FILES =3D ca.h.rst dmx.h.rst frontend.h.rst net.h.rst \ - videodev2.h.rst media.h.rst cec.h.rst lirc.h.rst - -TARGETS :=3D $(addprefix $(BUILDDIR)/, $(FILES)) - -gen_rst =3D \ - echo ${PARSER} $< $@ $(SRC_DIR)/$(notdir $@).exceptions; \ - ${PARSER} $< $@ $(SRC_DIR)/$(notdir $@).exceptions - -quiet_gen_rst =3D echo ' PARSE $(patsubst $(srctree)/%,%,$<)'; \ - ${PARSER} $< $@ $(SRC_DIR)/$(notdir $@).exceptions - -silent_gen_rst =3D ${gen_rst} - -$(BUILDDIR)/ca.h.rst: ${UAPI}/dvb/ca.h ${PARSER} $(SRC_DIR)/ca.h.rst.excep= tions - @$($(quiet)gen_rst) - -$(BUILDDIR)/dmx.h.rst: ${UAPI}/dvb/dmx.h ${PARSER} $(SRC_DIR)/dmx.h.rst.ex= ceptions - @$($(quiet)gen_rst) - -$(BUILDDIR)/frontend.h.rst: ${UAPI}/dvb/frontend.h ${PARSER} $(SRC_DIR)/fr= ontend.h.rst.exceptions - @$($(quiet)gen_rst) - -$(BUILDDIR)/net.h.rst: ${UAPI}/dvb/net.h ${PARSER} $(SRC_DIR)/net.h.rst.ex= ceptions - @$($(quiet)gen_rst) - -$(BUILDDIR)/videodev2.h.rst: ${UAPI}/videodev2.h ${PARSER} $(SRC_DIR)/vide= odev2.h.rst.exceptions - @$($(quiet)gen_rst) - -$(BUILDDIR)/media.h.rst: ${UAPI}/media.h ${PARSER} $(SRC_DIR)/media.h.rst.= exceptions - @$($(quiet)gen_rst) - -$(BUILDDIR)/cec.h.rst: ${UAPI}/cec.h ${PARSER} $(SRC_DIR)/cec.h.rst.except= ions - @$($(quiet)gen_rst) - -$(BUILDDIR)/lirc.h.rst: ${UAPI}/lirc.h ${PARSER} $(SRC_DIR)/lirc.h.rst.exc= eptions - @$($(quiet)gen_rst) - -# Media build rules - -.PHONY: all html texinfo epub xml latex - -all: $(IMGDOT) $(BUILDDIR) ${TARGETS} -html: all -texinfo: all -epub: all -xml: all -latex: $(IMGPDF) all -linkcheck: - -clean: - -rm -f $(DOTTGT) $(IMGTGT) ${TARGETS} 2>/dev/null - -$(BUILDDIR): - $(Q)mkdir -p $@ diff --git a/Documentation/userspace-api/media/cec/cec-header.rst b/Documen= tation/userspace-api/media/cec/cec-header.rst index d70736ac2b1d..f67003bb8740 100644 --- a/Documentation/userspace-api/media/cec/cec-header.rst +++ b/Documentation/userspace-api/media/cec/cec-header.rst @@ -6,5 +6,6 @@ CEC Header File *************** =20 -.. kernel-include:: $BUILDDIR/cec.h.rst - +.. kernel-include:: include/uapi/linux/cec.h + :generate-cross-refs: + :exception-file: cec.h.rst.exceptions diff --git a/Documentation/userspace-api/media/cec.h.rst.exceptions b/Docum= entation/userspace-api/media/cec/cec.h.rst.exceptions similarity index 100% rename from Documentation/userspace-api/media/cec.h.rst.exceptions rename to Documentation/userspace-api/media/cec/cec.h.rst.exceptions diff --git a/Documentation/userspace-api/media/ca.h.rst.exceptions b/Docume= ntation/userspace-api/media/dvb/ca.h.rst.exceptions similarity index 100% rename from Documentation/userspace-api/media/ca.h.rst.exceptions rename to Documentation/userspace-api/media/dvb/ca.h.rst.exceptions diff --git a/Documentation/userspace-api/media/dmx.h.rst.exceptions b/Docum= entation/userspace-api/media/dvb/dmx.h.rst.exceptions similarity index 100% rename from Documentation/userspace-api/media/dmx.h.rst.exceptions rename to Documentation/userspace-api/media/dvb/dmx.h.rst.exceptions diff --git a/Documentation/userspace-api/media/frontend.h.rst.exceptions b/= Documentation/userspace-api/media/dvb/frontend.h.rst.exceptions similarity index 100% rename from Documentation/userspace-api/media/frontend.h.rst.exceptions rename to Documentation/userspace-api/media/dvb/frontend.h.rst.exceptions diff --git a/Documentation/userspace-api/media/dvb/headers.rst b/Documentat= ion/userspace-api/media/dvb/headers.rst index 88c3eb33a89e..c75f64cf21d5 100644 --- a/Documentation/userspace-api/media/dvb/headers.rst +++ b/Documentation/userspace-api/media/dvb/headers.rst @@ -7,10 +7,19 @@ Digital TV uAPI header files Digital TV uAPI headers *********************** =20 -.. kernel-include:: $BUILDDIR/frontend.h.rst +.. kernel-include:: include/uapi/linux/dvb/frontend.h + :generate-cross-refs: + :exception-file: frontend.h.rst.exceptions =20 -.. kernel-include:: $BUILDDIR/dmx.h.rst +.. kernel-include:: include/uapi/linux/dvb/dmx.h + :generate-cross-refs: + :exception-file: dmx.h.rst.exceptions =20 -.. kernel-include:: $BUILDDIR/ca.h.rst +.. kernel-include:: include/uapi/linux/dvb/ca.h + :generate-cross-refs: + :exception-file: ca.h.rst.exceptions + +.. kernel-include:: include/uapi/linux/dvb/net.h + :generate-cross-refs: + :exception-file: net.h.rst.exceptions =20 -.. kernel-include:: $BUILDDIR/net.h.rst diff --git a/Documentation/userspace-api/media/net.h.rst.exceptions b/Docum= entation/userspace-api/media/dvb/net.h.rst.exceptions similarity index 100% rename from Documentation/userspace-api/media/net.h.rst.exceptions rename to Documentation/userspace-api/media/dvb/net.h.rst.exceptions diff --git a/Documentation/userspace-api/media/mediactl/media-header.rst b/= Documentation/userspace-api/media/mediactl/media-header.rst index c674271c93f5..d561d2845f3d 100644 --- a/Documentation/userspace-api/media/mediactl/media-header.rst +++ b/Documentation/userspace-api/media/mediactl/media-header.rst @@ -6,5 +6,6 @@ Media Controller Header File **************************** =20 -.. kernel-include:: $BUILDDIR/media.h.rst - +.. kernel-include:: include/uapi/linux/media.h + :generate-cross-refs: + :exception-file: media.h.rst.exceptions diff --git a/Documentation/userspace-api/media/media.h.rst.exceptions b/Doc= umentation/userspace-api/media/mediactl/media.h.rst.exceptions similarity index 100% rename from Documentation/userspace-api/media/media.h.rst.exceptions rename to Documentation/userspace-api/media/mediactl/media.h.rst.exceptions diff --git a/Documentation/userspace-api/media/rc/lirc-header.rst b/Documen= tation/userspace-api/media/rc/lirc-header.rst index 54cb40b8a065..a53328327847 100644 --- a/Documentation/userspace-api/media/rc/lirc-header.rst +++ b/Documentation/userspace-api/media/rc/lirc-header.rst @@ -6,5 +6,7 @@ LIRC Header File **************** =20 -.. kernel-include:: $BUILDDIR/lirc.h.rst +.. kernel-include:: include/uapi/linux/lirc.h + :generate-cross-refs: + :exception-file: lirc.h.rst.exceptions =20 diff --git a/Documentation/userspace-api/media/lirc.h.rst.exceptions b/Docu= mentation/userspace-api/media/rc/lirc.h.rst.exceptions similarity index 100% rename from Documentation/userspace-api/media/lirc.h.rst.exceptions rename to Documentation/userspace-api/media/rc/lirc.h.rst.exceptions diff --git a/Documentation/userspace-api/media/v4l/videodev.rst b/Documenta= tion/userspace-api/media/v4l/videodev.rst index c866fec417eb..cde485bc9a5f 100644 --- a/Documentation/userspace-api/media/v4l/videodev.rst +++ b/Documentation/userspace-api/media/v4l/videodev.rst @@ -6,4 +6,6 @@ Video For Linux Two Header File ******************************* =20 -.. kernel-include:: $BUILDDIR/videodev2.h.rst +.. kernel-include:: include/uapi/linux/videodev2.h + :generate-cross-refs: + :exception-file: videodev2.h.rst.exceptions diff --git a/Documentation/userspace-api/media/videodev2.h.rst.exceptions b= /Documentation/userspace-api/media/v4l/videodev2.h.rst.exceptions similarity index 100% rename from Documentation/userspace-api/media/videodev2.h.rst.exceptions rename to Documentation/userspace-api/media/v4l/videodev2.h.rst.exceptions diff --git a/scripts/sphinx-build-wrapper b/scripts/sphinx-build-wrapper new file mode 100755 index 000000000000..abe8c26ae137 --- /dev/null +++ b/scripts/sphinx-build-wrapper @@ -0,0 +1,719 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# Copyright (C) 2025 Mauro Carvalho Chehab +# +# pylint: disable=3DR0902, R0912, R0913, R0914, R0915, R0917, C0103 +# +# Converted from docs Makefile and parallel-wrapper.sh, both under +# GPLv2, copyrighted since 2008 by the following authors: +# +# Akira Yokosawa +# Arnd Bergmann +# Breno Leitao +# Carlos Bilbao +# Dave Young +# Donald Hunter +# Geert Uytterhoeven +# Jani Nikula +# Jan Stancek +# Jonathan Corbet +# Joshua Clayton +# Kees Cook +# Linus Torvalds +# Magnus Damm +# Masahiro Yamada +# Mauro Carvalho Chehab +# Maxim Cournoyer +# Peter Foley +# Randy Dunlap +# Rob Herring +# Shuah Khan +# Thorsten Blum +# Tomas Winkler + + +""" +Sphinx build wrapper that handles Kernel-specific business rules: + +- it gets the Kernel build environment vars; +- it determines what's the best parallelism; +- it handles SPHINXDIRS + +This tool ensures that MIN_PYTHON_VERSION is satisfied. If version is +below that, it seeks for a new Python version. If found, it re-runs using +the newer version. +""" + +import argparse +import locale +import os +import re +import shlex +import shutil +import subprocess +import sys + +from concurrent import futures +from glob import glob + +LIB_DIR =3D "lib" +SRC_DIR =3D os.path.dirname(os.path.realpath(__file__)) + +sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR)) + +from jobserver import JobserverExec # pylint: disable=3DC= 0413 + + +def parse_version(version): + """Convert a major.minor.patch version into a tuple""" + return tuple(int(x) for x in version.split(".")) + +def ver_str(version): + """Returns a version tuple as major.minor.patch""" + + return ".".join([str(x) for x in version]) + +# Minimal supported Python version needed by Sphinx and its extensions +MIN_PYTHON_VERSION =3D parse_version("3.7") + +# Default value for --venv parameter +VENV_DEFAULT =3D "sphinx_latest" + +# List of make targets and its corresponding builder and output directory +TARGETS =3D { + "cleandocs": { + "builder": "clean", + }, + "htmldocs": { + "builder": "html", + }, + "epubdocs": { + "builder": "epub", + "out_dir": "epub", + }, + "texinfodocs": { + "builder": "texinfo", + "out_dir": "texinfo", + }, + "infodocs": { + "builder": "texinfo", + "out_dir": "texinfo", + }, + "latexdocs": { + "builder": "latex", + "out_dir": "latex", + }, + "pdfdocs": { + "builder": "latex", + "out_dir": "latex", + }, + "xmldocs": { + "builder": "xml", + "out_dir": "xml", + }, + "linkcheckdocs": { + "builder": "linkcheck" + }, +} + +# Paper sizes. An empty value will pick the default +PAPER =3D ["", "a4", "letter"] + +class SphinxBuilder: + """ + Handles a sphinx-build target, adding needed arguments to build + with the Kernel. + """ + + def is_rust_enabled(self): + """Check if rust is enabled at .config""" + config_path =3D os.path.join(self.srctree, ".config") + if os.path.isfile(config_path): + with open(config_path, "r", encoding=3D"utf-8") as f: + return "CONFIG_RUST=3Dy" in f.read() + return False + + def get_path(self, path, abs_path=3DFalse): + """ + Ancillary routine to handle patches the right way, as shell does. + + It first expands "~" and "~user". Then, if patch is not absolute, + join self.srctree. Finally, if requested, convert to abspath. + """ + + path =3D os.path.expanduser(path) + if not path.startswith("/"): + path =3D os.path.join(self.srctree, path) + + if abs_path: + return os.path.abspath(path) + + return path + + def __init__(self, venv=3DNone, verbose=3DFalse, n_jobs=3DNone, intera= ctive=3DNone): + """Initialize internal variables""" + self.venv =3D venv + self.verbose =3D None + + # Normal variables passed from Kernel's makefile + self.kernelversion =3D os.environ.get("KERNELVERSION", "unknown") + self.kernelrelease =3D os.environ.get("KERNELRELEASE", "unknown") + self.pdflatex =3D os.environ.get("PDFLATEX", "xelatex") + + if not interactive: + self.latexopts =3D os.environ.get("LATEXOPTS", "-interaction= =3Dbatchmode -no-shell-escape") + else: + self.latexopts =3D os.environ.get("LATEXOPTS", "") + + if not verbose: + verbose =3D bool(os.environ.get("KBUILD_VERBOSE", "") !=3D "") + + # Handle SPHINXOPTS evironment + sphinxopts =3D shlex.split(os.environ.get("SPHINXOPTS", "")) + + # As we handle number of jobs and quiet in separate, we need to pi= ck + # it the same way as sphinx-build would pick, so let's use argparse + # do to the right argument expansion + parser =3D argparse.ArgumentParser() + parser.add_argument('-j', '--jobs', type=3Dint) + parser.add_argument('-q', '--quiet', type=3Dint) + + # Other sphinx-build arguments go as-is, so place them + # at self.sphinxopts + sphinx_args, self.sphinxopts =3D parser.parse_known_args(sphinxopt= s) + if sphinx_args.quiet =3D=3D True: + self.verbose =3D False + + if sphinx_args.jobs: + self.n_jobs =3D sphinx_args.jobs + + # Command line arguments was passed, override SPHINXOPTS + if verbose is not None: + self.verbose =3D verbose + + self.n_jobs =3D n_jobs + + # Source tree directory. This needs to be at os.environ, as + # Sphinx extensions and media uAPI makefile needs it + self.srctree =3D os.environ.get("srctree") + if not self.srctree: + self.srctree =3D "." + os.environ["srctree"] =3D self.srctree + + # Now that we can expand srctree, get other directories as well + self.sphinxbuild =3D os.environ.get("SPHINXBUILD", "sphinx-build") + self.kerneldoc =3D self.get_path(os.environ.get("KERNELDOC", + "scripts/kernel-doc.= py")) + self.obj =3D os.environ.get("obj", "Documentation") + self.builddir =3D self.get_path(os.path.join(self.obj, "output"), + abs_path=3DTrue) + + # Media uAPI needs it + os.environ["BUILDDIR"] =3D self.builddir + + # Detect if rust is enabled + self.config_rust =3D self.is_rust_enabled() + + # Get directory locations for LaTeX build toolchain + self.pdflatex_cmd =3D shutil.which(self.pdflatex) + self.latexmk_cmd =3D shutil.which("latexmk") + + self.env =3D os.environ.copy() + + # If venv parameter is specified, run Sphinx from venv + if venv: + bin_dir =3D os.path.join(venv, "bin") + if os.path.isfile(os.path.join(bin_dir, "activate")): + # "activate" virtual env + self.env["PATH"] =3D bin_dir + ":" + self.env["PATH"] + self.env["VIRTUAL_ENV"] =3D venv + if "PYTHONHOME" in self.env: + del self.env["PYTHONHOME"] + print(f"Setting venv to {venv}") + else: + sys.exit(f"Venv {venv} not found.") + + def run_sphinx(self, sphinx_build, build_args, *args, **pwargs): + """ + Executes sphinx-build using current python3 command and setting + -j parameter if possible to run the build in parallel. + """ + + with JobserverExec() as jobserver: + if jobserver.claim: + n_jobs =3D str(jobserver.claim) + else: + n_jobs =3D "auto" # Supported since Sphinx 1.7 + + cmd =3D [] + + if self.venv: + cmd.append("python") + else: + cmd.append(sys.executable) + + cmd.append(sphinx_build) + + # if present, SPHINXOPTS or command line --jobs overrides defa= ult + if self.n_jobs: + n_jobs =3D str(self.n_jobs) + + if n_jobs: + cmd +=3D [f"-j{n_jobs}"] + + if not self.verbose: + cmd.append("-q") + + cmd +=3D self.sphinxopts + + cmd +=3D build_args + + if self.verbose: + print(" ".join(cmd)) + + rc =3D subprocess.call(cmd, *args, **pwargs) + + def handle_html(self, css, output_dir): + """ + Extra steps for HTML and epub output. + + For such targets, we need to ensure that CSS will be properly + copied to the output _static directory + """ + + if not css: + return + + css =3D os.path.expanduser(css) + if not css.startswith("/"): + css =3D os.path.join(self.srctree, css) + + static_dir =3D os.path.join(output_dir, "_static") + os.makedirs(static_dir, exist_ok=3DTrue) + + try: + shutil.copy2(css, static_dir) + except (OSError, IOError) as e: + print(f"Warning: Failed to copy CSS: {e}", file=3Dsys.stderr) + + def build_pdf_file(self, latex_cmd, from_dir, path): + """Builds a single pdf file using latex_cmd""" + try: + subprocess.run(latex_cmd + [path], + cwd=3Dfrom_dir, check=3DTrue) + + return True + except subprocess.CalledProcessError: + # LaTeX PDF error code is almost useless: it returns + # error codes even when build succeeds but has warnings. + # So, we'll ignore the results + return False + + def pdf_parallel_build(self, tex_suffix, latex_cmd, tex_files, n_jobs): + """Build PDF files in parallel if possible""" + builds =3D {} + build_failed =3D False + max_len =3D 0 + has_tex =3D False + + # Process files in parallel + with futures.ThreadPoolExecutor(max_workers=3Dn_jobs) as executor: + jobs =3D {} + + for from_dir, pdf_dir, entry in tex_files: + name =3D entry.name + + if not name.endswith(tex_suffix): + continue + + name =3D name[:-len(tex_suffix)] + + max_len =3D max(max_len, len(name)) + + has_tex =3D True + + future =3D executor.submit(self.build_pdf_file, latex_cmd, + from_dir, entry.path) + jobs[future] =3D (from_dir, name, entry.path) + + for future in futures.as_completed(jobs): + from_dir, name, path =3D jobs[future] + + pdf_name =3D name + ".pdf" + pdf_from =3D os.path.join(from_dir, pdf_name) + + try: + success =3D future.result() + + if success and os.path.exists(pdf_from): + pdf_to =3D os.path.join(pdf_dir, pdf_name) + + os.rename(pdf_from, pdf_to) + builds[name] =3D os.path.relpath(pdf_to, self.buil= ddir) + else: + builds[name] =3D "FAILED" + build_failed =3D True + except Exception as e: + builds[name] =3D f"FAILED ({str(e)})" + build_failed =3D True + + # Handle case where no .tex files were found + if not has_tex: + name =3D "Sphinx LaTeX builder" + max_len =3D max(max_len, len(name)) + builds[name] =3D "FAILED (no .tex file was generated)" + build_failed =3D True + + return builds, build_failed, max_len + + def handle_pdf(self, output_dirs): + """ + Extra steps for PDF output. + + As PDF is handled via a LaTeX output, after building the .tex file, + a new build is needed to create the PDF output from the latex + directory. + """ + builds =3D {} + max_len =3D 0 + tex_suffix =3D ".tex" + + # Get all tex files that will be used for PDF build + tex_files =3D [] + for from_dir in output_dirs: + pdf_dir =3D os.path.join(from_dir, "../pdf") + os.makedirs(pdf_dir, exist_ok=3DTrue) + + if self.latexmk_cmd: + latex_cmd =3D [self.latexmk_cmd, f"-{self.pdflatex}"] + else: + latex_cmd =3D [self.pdflatex] + + latex_cmd.extend(shlex.split(self.latexopts)) + + # Get a list of tex files to process + with os.scandir(from_dir) as it: + for entry in it: + if entry.name.endswith(tex_suffix): + tex_files.append((from_dir, pdf_dir, entry)) + + # When using make, this won't be used, as the number of jobs comes + # from POSIX jobserver. So, this covers the case where build comes + # from command line. On such case, serialize by default, except if + # the user explicitly sets the number of jobs. + n_jobs =3D 1 + + # n_jobs is either an integer or "auto". Only use it if it is a nu= mber + if self.n_jobs: + try: + n_jobs =3D int(self.n_jobs) + except ValueError: + pass + + # When using make, jobserver.claim is the number of jobs that were + # used with "-j" and that aren't used by other make targets + with JobserverExec() as jobserver: + n_jobs =3D 1 + + # Handle the case when a parameter is passed via command line, + # using it as default, if jobserver doesn't claim anything + if self.n_jobs: + try: + n_jobs =3D int(self.n_jobs) + except ValueError: + pass + + if jobserver.claim: + n_jobs =3D jobserver.claim + + # Build files in parallel + builds, build_failed, max_len =3D self.pdf_parallel_build(tex_= suffix, + latex_= cmd, + tex_fi= les, + n_jobs) + + msg =3D "Summary" + msg +=3D "\n" + "=3D" * len(msg) + print() + print(msg) + + for pdf_name, pdf_file in builds.items(): + print(f"{pdf_name:<{max_len}}: {pdf_file}") + + print() + + # return an error if a PDF file is missing + + if build_failed: + sys.exit(f"PDF build failed: not all PDF files were created.") + else: + print("All PDF files were built.") + + def handle_info(self, output_dirs): + """ + Extra steps for Info output. + + For texinfo generation, an additional make is needed from the + texinfo directory. + """ + + for output_dir in output_dirs: + try: + subprocess.run(["make", "info"], cwd=3Doutput_dir, check= =3DTrue) + except subprocess.CalledProcessError as e: + sys.exit(f"Error generating info docs: {e}") + + def cleandocs(self, builder): + + shutil.rmtree(self.builddir, ignore_errors=3DTrue) + + def build(self, target, sphinxdirs=3DNone, conf=3D"conf.py", + theme=3DNone, css=3DNone, paper=3DNone): + """ + Build documentation using Sphinx. This is the core function of this + module. It prepares all arguments required by sphinx-build. + """ + + builder =3D TARGETS[target]["builder"] + out_dir =3D TARGETS[target].get("out_dir", "") + + # Cleandocs doesn't require sphinx-build + if target =3D=3D "cleandocs": + self.cleandocs(builder) + return + + # Other targets require sphinx-build + sphinxbuild =3D shutil.which(self.sphinxbuild, path=3Dself.env["PA= TH"]) + if not sphinxbuild: + sys.exit(f"Error: {self.sphinxbuild} not found in PATH.\n") + + if builder =3D=3D "latex": + if not self.pdflatex_cmd and not self.latexmk_cmd: + sys.exit("Error: pdflatex or latexmk required for PDF gene= ration") + + docs_dir =3D os.path.abspath(os.path.join(self.srctree, "Documenta= tion")) + + # Prepare base arguments for Sphinx build + kerneldoc =3D self.kerneldoc + if kerneldoc.startswith(self.srctree): + kerneldoc =3D os.path.relpath(kerneldoc, self.srctree) + + # Prepare common Sphinx options + args =3D [ + "-b", builder, + "-c", docs_dir, + ] + + if builder =3D=3D "latex": + if not paper: + paper =3D PAPER[1] + + args.extend(["-D", f"latex_elements.papersize=3D{paper}paper"]) + + if self.config_rust: + args.extend(["-t", "rustdoc"]) + + if conf: + self.env["SPHINX_CONF"] =3D self.get_path(conf, abs_path=3DTru= e) + + if not sphinxdirs: + sphinxdirs =3D os.environ.get("SPHINXDIRS", ".") + + # The sphinx-build tool has a bug: internally, it tries to set + # locale with locale.setlocale(locale.LC_ALL, ''). This causes a + # crash if language is not set. Detect and fix it. + try: + locale.setlocale(locale.LC_ALL, '') + except Exception: + self.env["LC_ALL"] =3D "C" + self.env["LANG"] =3D "C" + + # sphinxdirs can be a list or a whitespace-separated string + sphinxdirs_list =3D [] + for sphinxdir in sphinxdirs: + if isinstance(sphinxdir, list): + sphinxdirs_list +=3D sphinxdir + else: + for name in sphinxdir.split(" "): + sphinxdirs_list.append(name) + + # Build each directory + output_dirs =3D [] + for sphinxdir in sphinxdirs_list: + src_dir =3D os.path.join(docs_dir, sphinxdir) + doctree_dir =3D os.path.join(self.builddir, ".doctrees") + output_dir =3D os.path.join(self.builddir, sphinxdir, out_dir) + + # Make directory names canonical + src_dir =3D os.path.normpath(src_dir) + doctree_dir =3D os.path.normpath(doctree_dir) + output_dir =3D os.path.normpath(output_dir) + + os.makedirs(doctree_dir, exist_ok=3DTrue) + os.makedirs(output_dir, exist_ok=3DTrue) + + output_dirs.append(output_dir) + + build_args =3D args + [ + "-d", doctree_dir, + "-D", f"kerneldoc_bin=3D{kerneldoc}", + "-D", f"version=3D{self.kernelversion}", + "-D", f"release=3D{self.kernelrelease}", + "-D", f"kerneldoc_srctree=3D{self.srctree}", + src_dir, + output_dir, + ] + + # Execute sphinx-build + try: + self.run_sphinx(sphinxbuild, build_args, env=3Dself.env) + except Exception as e: + sys.exit(f"Build failed: {e}") + + # Ensure that html/epub will have needed static files + if target in ["htmldocs", "epubdocs"]: + self.handle_html(css, output_dir) + + # PDF and Info require a second build step + if target =3D=3D "pdfdocs": + self.handle_pdf(output_dirs) + elif target =3D=3D "infodocs": + self.handle_info(output_dirs) + + @staticmethod + def get_python_version(cmd): + """ + Get python version from a Python binary. As we need to detect if + are out there newer python binaries, we can't rely on sys.release = here. + """ + + result =3D subprocess.run([cmd, "--version"], check=3DTrue, + stdout=3Dsubprocess.PIPE, stderr=3Dsubproc= ess.PIPE, + universal_newlines=3DTrue) + version =3D result.stdout.strip() + + match =3D re.search(r"(\d+\.\d+\.\d+)", version) + if match: + return parse_version(match.group(1)) + + print(f"Can't parse version {version}") + return (0, 0, 0) + + @staticmethod + def find_python(): + """ + Detect if are out there any python 3.xy version newer than the + current one. + + Note: this routine is limited to up to 2 digits for python3. We + may need to update it one day, hopefully on a distant future. + """ + patterns =3D [ + "python3.[0-9]", + "python3.[0-9][0-9]", + ] + + # Seek for a python binary newer than MIN_PYTHON_VERSION + for path in os.getenv("PATH", "").split(":"): + for pattern in patterns: + for cmd in glob(os.path.join(path, pattern)): + if os.path.isfile(cmd) and os.access(cmd, os.X_OK): + version =3D SphinxBuilder.get_python_version(cmd) + if version >=3D MIN_PYTHON_VERSION: + return cmd + + return None + + @staticmethod + def check_python(): + """ + Check if the current python binary satisfies our minimal requireme= nt + for Sphinx build. If not, re-run with a newer version if found. + """ + cur_ver =3D sys.version_info[:3] + if cur_ver >=3D MIN_PYTHON_VERSION: + return + + python_ver =3D ver_str(cur_ver) + + new_python_cmd =3D SphinxBuilder.find_python() + if not new_python_cmd: + sys.exit(f"Python version {python_ver} is not supported anymor= e.") + + # Restart script using the newer version + script_path =3D os.path.abspath(sys.argv[0]) + args =3D [new_python_cmd, script_path] + sys.argv[1:] + + print(f"Python {python_ver} not supported. Changing to {new_python= _cmd}") + + try: + os.execv(new_python_cmd, args) + except OSError as e: + sys.exit(f"Failed to restart with {new_python_cmd}: {e}") + +def jobs_type(value): + """ + Handle valid values for -j. Accepts Sphinx "-jauto", plus a number + equal or bigger than one. + """ + if value is None: + return None + + if value.lower() =3D=3D 'auto': + return value.lower() + + try: + if int(value) >=3D 1: + return value + + raise argparse.ArgumentTypeError(f"Minimum jobs is 1, got {value}") + except ValueError: + raise argparse.ArgumentTypeError(f"Must be 'auto' or positive inte= ger, got {value}") + +def main(): + """ + Main function. The only mandatory argument is the target. If not + specified, the other arguments will use default values if not + specified at os.environ. + """ + parser =3D argparse.ArgumentParser(description=3D"Kernel documentation= builder") + + parser.add_argument("target", choices=3Dlist(TARGETS.keys()), + help=3D"Documentation target to build") + parser.add_argument("--sphinxdirs", nargs=3D"+", + help=3D"Specific directories to build") + parser.add_argument("--conf", default=3D"conf.py", + help=3D"Sphinx configuration file") + + parser.add_argument("--theme", help=3D"Sphinx theme to use") + + parser.add_argument("--css", help=3D"Custom CSS file for HTML/EPUB") + + parser.add_argument("--paper", choices=3DPAPER, default=3DPAPER[0], + help=3D"Paper size for LaTeX/PDF output") + + parser.add_argument("-v", "--verbose", action=3D'store_true', + help=3D"place build in verbose mode") + + parser.add_argument('-j', '--jobs', type=3Djobs_type, + help=3D"Sets number of jobs to use with sphinx-bui= ld") + + parser.add_argument('-i', '--interactive', action=3D'store_true', + help=3D"Change latex default to run in interactive= mode") + + parser.add_argument("-V", "--venv", nargs=3D'?', const=3Df'{VENV_DEFAU= LT}', + default=3DNone, + help=3Df'If used, run Sphinx from a venv dir (defa= ult dir: {VENV_DEFAULT})') + + args =3D parser.parse_args() + + SphinxBuilder.check_python() + + builder =3D SphinxBuilder(venv=3Dargs.venv, verbose=3Dargs.verbose, + n_jobs=3Dargs.jobs, interactive=3Dargs.interac= tive) + + builder.build(args.target, sphinxdirs=3Dargs.sphinxdirs, conf=3Dargs.c= onf, + theme=3Dargs.theme, css=3Dargs.css, paper=3Dargs.paper) + +if __name__ =3D=3D "__main__": + main() --=20 2.50.1 From nobody Fri Oct 3 23:08:33 2025 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E12DC3126CD; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872389; cv=none; b=Idi4tgqZ7Y3F/9hSiR/eJBUJPZb35fB/5p+B3Fj+Auf2hzCeVVrKiDUTIKDGeYIAn8YKP8FclGxqlbj9JtjLD4jl+yZj2dp/T25VPSrhD3Ggny35qLM/VJYqol1YjdRKo5Tyi5H/0uQcwvwjt3YrUVPzpjRhslIU5/HEHJrElkQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1755872389; c=relaxed/simple; bh=uYEK1a2ZG5UzKHVraa8W5O9cCQbPaE3FbRWgQy8mxrI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rCaj+iN0QfPMhZ4849L+ww/iw6pn3b02uFqvTFWWwew1OtStAEa9iqNn/+tPCmJbaB9h1PdJ66QS2HAh/sfCVTrfsTKZnlzEy//aXNtOpy6KFLIHoJ2h3SiGYWproRI2zcqDqAVU8fLVGRn1PrVw4fswEmGR3Gq8EukxCgplC5k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=qGZs/DgR; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="qGZs/DgR" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 78C2DC2BD04; Fri, 22 Aug 2025 14:19:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1755872388; bh=uYEK1a2ZG5UzKHVraa8W5O9cCQbPaE3FbRWgQy8mxrI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qGZs/DgRijv4sIpLfNua2KZwa1zOMTA7P3w+cQVGGajf4ePoCF30BlJh1vhH8eIPR G3EyOqQNqNhqQ6e721FeC0TmBJ7Ak5fVk7VM6MAP389PMiBDDU4bMjGtbhWvw/Y/bO XFARAthbCttu1p0FXd6W9aY3hktqnrJn67ZhNu9tCv20uVosxZmtHNcDyaKfdbtZff Zi+z+5Sh5UhoO/4JN1BykDUX1KyAighzDEkoDK9Zws19AcsbGhFIF4tjY4L4nUc76z T1eqOEehoua09rO56k+Z7Ipz/bBmP4jTXlFUcyYqjnruEzSw3zIXRVdBhztcVmNyQw zdkxEmsnCteDg== Received: from mchehab by mail.kernel.org with local (Exim 4.98.2) (envelope-from ) id 1upScM-0000000CCsS-36wv; Fri, 22 Aug 2025 16:19:46 +0200 From: Mauro Carvalho Chehab To: Jonathan Corbet , Linux Doc Mailing List Cc: Mauro Carvalho Chehab , "Mauro Carvalho Chehab" , linux-kernel@vger.kernel.org Subject: [PATCH v2 24/24] docs: sphinx: drop parse-headers.pl Date: Fri, 22 Aug 2025 16:19:36 +0200 Message-ID: X-Mailer: git-send-email 2.50.1 In-Reply-To: References: Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: Mauro Carvalho Chehab Content-Type: text/plain; charset="utf-8" Now that we have a replacement in place, drop the old version. Signed-off-by: Mauro Carvalho Chehab --- Documentation/sphinx/parse-headers.pl | 419 -------------------------- 1 file changed, 419 deletions(-) delete mode 100755 Documentation/sphinx/parse-headers.pl diff --git a/Documentation/sphinx/parse-headers.pl b/Documentation/sphinx/p= arse-headers.pl deleted file mode 100755 index 560685926cdb..000000000000 --- a/Documentation/sphinx/parse-headers.pl +++ /dev/null @@ -1,419 +0,0 @@ -#!/usr/bin/env perl -# SPDX-License-Identifier: GPL-2.0 -# Copyright (c) 2016 by Mauro Carvalho Chehab . - -use strict; -use Text::Tabs; -use Getopt::Long; -use Pod::Usage; - -my $debug; -my $help; -my $man; - -GetOptions( - "debug" =3D> \$debug, - 'usage|?' =3D> \$help, - 'help' =3D> \$man -) or pod2usage(2); - -pod2usage(1) if $help; -pod2usage(-exitstatus =3D> 0, -verbose =3D> 2) if $man; -pod2usage(2) if (scalar @ARGV < 2 || scalar @ARGV > 3); - -my ($file_in, $file_out, $file_exceptions) =3D @ARGV; - -my $data; -my %ioctls; -my %defines; -my %typedefs; -my %enums; -my %enum_symbols; -my %structs; - -# -# read the file and get identifiers -# - -my $is_enum =3D 0; -my $is_comment =3D 0; -open IN, $file_in or die "Can't open $file_in"; -while () { - $data .=3D $_; - - my $ln =3D $_; - if (!$is_comment) { - $ln =3D~ s,/\*.*(\*/),,g; - - $is_comment =3D 1 if ($ln =3D~ s,/\*.*,,); - } else { - if ($ln =3D~ s,^(.*\*/),,) { - $is_comment =3D 0; - } else { - next; - } - } - - if ($is_enum && $ln =3D~ m/^\s*([_\w][\w\d_]+)\s*[\,=3D]?/) { - my $s =3D $1; - my $n =3D $1; - $n =3D~ tr/A-Z/a-z/; - $n =3D~ tr/_/-/; - - $enum_symbols{$s} =3D "\\ :ref:`$s <$n>`\\ "; - - $is_enum =3D 0 if ($is_enum && m/\}/); - next; - } - $is_enum =3D 0 if ($is_enum && m/\}/); - - if ($ln =3D~ m/^\s*#\s*define\s+([_\w][\w\d_]+)\s+_IO/) { - my $s =3D $1; - my $n =3D $1; - $n =3D~ tr/A-Z/a-z/; - - $ioctls{$s} =3D "\\ :ref:`$s <$n>`\\ "; - next; - } - - if ($ln =3D~ m/^\s*#\s*define\s+([_\w][\w\d_]+)\s+/) { - my $s =3D $1; - my $n =3D $1; - $n =3D~ tr/A-Z/a-z/; - $n =3D~ tr/_/-/; - - $defines{$s} =3D "\\ :ref:`$s <$n>`\\ "; - next; - } - - if ($ln =3D~ m/^\s*typedef\s+([_\w][\w\d_]+)\s+(.*)\s+([_\w][\w\d_]+);/) { - my $s =3D $2; - my $n =3D $3; - - $typedefs{$n} =3D "\\ :c:type:`$n <$s>`\\ "; - next; - } - if ($ln =3D~ m/^\s*enum\s+([_\w][\w\d_]+)\s+\{/ - || $ln =3D~ m/^\s*enum\s+([_\w][\w\d_]+)$/ - || $ln =3D~ m/^\s*typedef\s*enum\s+([_\w][\w\d_]+)\s+\{/ - || $ln =3D~ m/^\s*typedef\s*enum\s+([_\w][\w\d_]+)$/) { - my $s =3D $1; - - $enums{$s} =3D "enum :c:type:`$s`\\ "; - - $is_enum =3D $1; - next; - } - if ($ln =3D~ m/^\s*struct\s+([_\w][\w\d_]+)\s+\{/ - || $ln =3D~ m/^\s*struct\s+([[_\w][\w\d_]+)$/ - || $ln =3D~ m/^\s*typedef\s*struct\s+([_\w][\w\d_]+)\s+\{/ - || $ln =3D~ m/^\s*typedef\s*struct\s+([[_\w][\w\d_]+)$/ - ) { - my $s =3D $1; - - $structs{$s} =3D "struct $s\\ "; - next; - } -} -close IN; - -# -# Handle multi-line typedefs -# - -my @matches =3D ($data =3D~ m/typedef\s+struct\s+\S+?\s*\{[^\}]+\}\s*(\S+)= \s*\;/g, - $data =3D~ m/typedef\s+enum\s+\S+?\s*\{[^\}]+\}\s*(\S+)\s*\;/g,); -foreach my $m (@matches) { - my $s =3D $m; - - $typedefs{$s} =3D "\\ :c:type:`$s`\\ "; - next; -} - -# -# Handle exceptions, if any -# - -my %def_reftype =3D ( - "ioctl" =3D> ":ref", - "define" =3D> ":ref", - "symbol" =3D> ":ref", - "typedef" =3D> ":c:type", - "enum" =3D> ":c:type", - "struct" =3D> ":c:type", -); - -if ($file_exceptions) { - open IN, $file_exceptions or die "Can't read $file_exceptions"; - while () { - next if (m/^\s*$/ || m/^\s*#/); - - # Parsers to ignore a symbol - - if (m/^ignore\s+ioctl\s+(\S+)/) { - delete $ioctls{$1} if (exists($ioctls{$1})); - next; - } - if (m/^ignore\s+define\s+(\S+)/) { - delete $defines{$1} if (exists($defines{$1})); - next; - } - if (m/^ignore\s+typedef\s+(\S+)/) { - delete $typedefs{$1} if (exists($typedefs{$1})); - next; - } - if (m/^ignore\s+enum\s+(\S+)/) { - delete $enums{$1} if (exists($enums{$1})); - next; - } - if (m/^ignore\s+struct\s+(\S+)/) { - delete $structs{$1} if (exists($structs{$1})); - next; - } - if (m/^ignore\s+symbol\s+(\S+)/) { - delete $enum_symbols{$1} if (exists($enum_symbols{$1})); - next; - } - - # Parsers to replace a symbol - my ($type, $old, $new, $reftype); - - if (m/^replace\s+(\S+)\s+(\S+)\s+(\S+)/) { - $type =3D $1; - $old =3D $2; - $new =3D $3; - } else { - die "Can't parse $file_exceptions: $_"; - } - - if ($new =3D~ m/^\:c\:(data|func|macro|type)\:\`(.+)\`/) { - $reftype =3D ":c:$1"; - $new =3D $2; - } elsif ($new =3D~ m/\:ref\:\`(.+)\`/) { - $reftype =3D ":ref"; - $new =3D $1; - } else { - $reftype =3D $def_reftype{$type}; - } - if (!$reftype) { - print STDERR "Warning: can't find ref type for $type"; - } - $new =3D "$reftype:`$old <$new>`"; - - if ($type eq "ioctl") { - $ioctls{$old} =3D $new if (exists($ioctls{$old})); - next; - } - if ($type eq "define") { - $defines{$old} =3D $new if (exists($defines{$old})); - next; - } - if ($type eq "symbol") { - $enum_symbols{$old} =3D $new if (exists($enum_symbols{$old})); - next; - } - if ($type eq "typedef") { - $typedefs{$old} =3D $new if (exists($typedefs{$old})); - next; - } - if ($type eq "enum") { - $enums{$old} =3D $new if (exists($enums{$old})); - next; - } - if ($type eq "struct") { - $structs{$old} =3D $new if (exists($structs{$old})); - next; - } - - die "Can't parse $file_exceptions: $_"; - } -} - -if ($debug) { - my @all_hashes =3D ( - {ioctl =3D> \%ioctls}, - {typedef =3D> \%typedefs}, - {enum =3D> \%enums}, - {struct =3D> \%structs}, - {define =3D> \%defines}, - {symbol =3D> \%enum_symbols} - ); - - foreach my $hash (@all_hashes) { - while (my ($name, $hash_ref) =3D each %$hash) { - next unless %$hash_ref; # Skip empty hashes - - print "$name:\n"; - for my $key (sort keys %$hash_ref) { - print " $key -> $hash_ref->{$key}\n"; - } - print "\n"; - } - } -} - -# -# Align block -# -$data =3D expand($data); -$data =3D " " . $data; -$data =3D~ s/\n/\n /g; -$data =3D~ s/\n\s+$/\n/g; -$data =3D~ s/\n\s+\n/\n\n/g; - -# -# Add escape codes for special characters -# -$data =3D~ s,([\_\`\*\<\>\&\\\\:\/\|\%\$\#\{\}\~\^]),\\$1,g; - -$data =3D~ s,DEPRECATED,**DEPRECATED**,g; - -# -# Add references -# - -my $start_delim =3D "[ \n\t\(\=3D\*\@]"; -my $end_delim =3D "(\\s|,|\\\\=3D|\\\\:|\\;|\\\)|\\}|\\{)"; - -foreach my $r (keys %ioctls) { - my $s =3D $ioctls{$r}; - - $r =3D~ s,([\_\`\*\<\>\&\\\\:\/]),\\\\$1,g; - - print "$r -> $s\n" if ($debug); - - $data =3D~ s/($start_delim)($r)$end_delim/$1$s$3/g; -} - -foreach my $r (keys %defines) { - my $s =3D $defines{$r}; - - $r =3D~ s,([\_\`\*\<\>\&\\\\:\/]),\\\\$1,g; - - print "$r -> $s\n" if ($debug); - - $data =3D~ s/($start_delim)($r)$end_delim/$1$s$3/g; -} - -foreach my $r (keys %enum_symbols) { - my $s =3D $enum_symbols{$r}; - - $r =3D~ s,([\_\`\*\<\>\&\\\\:\/]),\\\\$1,g; - - print "$r -> $s\n" if ($debug); - - $data =3D~ s/($start_delim)($r)$end_delim/$1$s$3/g; -} - -foreach my $r (keys %enums) { - my $s =3D $enums{$r}; - - $r =3D~ s,([\_\`\*\<\>\&\\\\:\/]),\\\\$1,g; - - print "$r -> $s\n" if ($debug); - - $data =3D~ s/enum\s+($r)$end_delim/$s$2/g; -} - -foreach my $r (keys %structs) { - my $s =3D $structs{$r}; - - $r =3D~ s,([\_\`\*\<\>\&\\\\:\/]),\\\\$1,g; - - print "$r -> $s\n" if ($debug); - - $data =3D~ s/struct\s+($r)$end_delim/$s$2/g; -} - -foreach my $r (keys %typedefs) { - my $s =3D $typedefs{$r}; - - $r =3D~ s,([\_\`\*\<\>\&\\\\:\/]),\\\\$1,g; - - print "$r -> $s\n" if ($debug); - $data =3D~ s/($start_delim)($r)$end_delim/$1$s$3/g; -} - -$data =3D~ s/\\ ([\n\s])/\1/g; - -# -# Generate output file -# - -my $title =3D $file_in; -$title =3D~ s,.*/,,; - -open OUT, "> $file_out" or die "Can't open $file_out"; -print OUT ".. -*- coding: utf-8; mode: rst -*-\n\n"; -print OUT "$title\n"; -print OUT "=3D" x length($title); -print OUT "\n\n.. parsed-literal::\n\n"; -print OUT $data; -close OUT; - -__END__ - -=3Dhead1 NAME - -parse_headers.pl - parse a C file, in order to identify functions, structs, -enums and defines and create cross-references to a Sphinx book. - -=3Dhead1 SYNOPSIS - -B [] [] - -Where can be: --debug, --help or --usage. - -=3Dhead1 OPTIONS - -=3Dover 8 - -=3Ditem B<--debug> - -Put the script in verbose mode, useful for debugging. - -=3Ditem B<--usage> - -Prints a brief help message and exits. - -=3Ditem B<--help> - -Prints a more detailed help message and exits. - -=3Dback - -=3Dhead1 DESCRIPTION - -Convert a C header or source file (C_FILE), into a ReStructured Text -included via ..parsed-literal block with cross-references for the -documentation files that describe the API. It accepts an optional -EXCEPTIONS_FILE with describes what elements will be either ignored or -be pointed to a non-default reference. - -The output is written at the (OUT_FILE). - -It is capable of identifying defines, functions, structs, typedefs, -enums and enum symbols and create cross-references for all of them. -It is also capable of distinguish #define used for specifying a Linux -ioctl. - -The EXCEPTIONS_FILE contain two rules to allow ignoring a symbol or -to replace the default references by a custom one. - -Please read Documentation/doc-guide/parse-headers.rst at the Kernel's -tree for more details. - -=3Dhead1 BUGS - -Report bugs to Mauro Carvalho Chehab - -=3Dhead1 COPYRIGHT - -Copyright (c) 2016 by Mauro Carvalho Chehab . - -License GPLv2: GNU GPL version 2 . - -This is free software: you are free to change and redistribute it. -There is NO WARRANTY, to the extent permitted by law. - -=3Dcut --=20 2.50.1