From nobody Sun Feb 8 01:33:45 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=kernel.org ARC-Seal: i=1; a=rsa-sha256; t=1769702720; cv=none; d=zohomail.com; s=zohoarc; b=egMyJLO95SmxWeHHjVndWFYxuw3lEyLelJFN7q5UacT720T66IfcPQWKwTfJtaB3+kMpusa3USo4SlxAXDW0RXnInWFHAfV9AzNwZsV6bYWayZkNHso3a70fulkvrJsbxhXwrVmroe3G2oKH0KH9jKyYkxlPiYk8J0tb0Ta3ik0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1769702720; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=r71K2esN3neKKh5Ub8fRgdkPE8x8Aa81mH29tstKSec=; b=F+xKCvIHHm1HLgHrSueExMCe7HrHVd3nCGp8fCtOYiZlrbOAotM2VTDHcXJs0ZbX3fXGt7wqQoJFfPrTMpgrgV2YciP0JMIgK+K+w3g2/7b6GZEDq9Yqlye8joVYLu7H3gSSYbf1eSOHdHfgto/vKwii6P1qb0TkuSUWpAoU4P8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1769702720047200.82150771973932; Thu, 29 Jan 2026 08:05:20 -0800 (PST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1vlUVu-0006Kx-NE; Thu, 29 Jan 2026 11:04:58 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vlUVs-0006J3-IQ for qemu-devel@nongnu.org; Thu, 29 Jan 2026 11:04:56 -0500 Received: from sea.source.kernel.org ([172.234.252.31]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1vlUVo-00055A-1V for qemu-devel@nongnu.org; Thu, 29 Jan 2026 11:04:56 -0500 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id 56B814423F; Thu, 29 Jan 2026 16:04:50 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8181BC19422; Thu, 29 Jan 2026 16:04:49 +0000 (UTC) Received: from localhost ([::1]) by mail.kernel.org with esmtp (Exim 4.99.1) (envelope-from ) id 1vlUVj-0000000FNz9-2fJs; Thu, 29 Jan 2026 17:04:47 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1769702690; bh=+ZQIZgE5598Gk3449gBfwYAGRpDT4vo7wCP23OX4MAE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YDO89oAznJl8IAJDySggfXCEI245V/KJR6NLHWywMHEMtwzdDBYjzbqa42X2KfY9+ PLxsdp/kE2yiCgbMCSjGnImKLyWXMDMo2VgE/pHkt2HufMOQ+YaFvcEPaj0LXNCRpE C5m50aXSMzmbRXHWZmHw/xniRpO5eXVV2c9jkYhVuQPFAFmtLPTZB5Sdpi4ZHiHmrk d4QyT7XnZ/HCpJ/BlLXXJ19/m/QugCbeSvXKYs8MfxJ+n560EdmS8lOFNlNKQk1u1a 8yZBPMv9XTLkBJkMht5jasa5ete1msuun4UkcnYJyuMKKvsuvZfXzA6wMO9w9ugakz xJ3lsmE7ghZaA== From: Mauro Carvalho Chehab To: John Snow Cc: Mauro Carvalho Chehab , Jonathan Corbet , qemu-devel@nongnu.org, Cleber Rosa , Peter Maydell Subject: [PATCH 1/1] scripts: Import Python kerneldoc from Linux kernel Date: Thu, 29 Jan 2026 17:04:06 +0100 Message-ID: X-Mailer: git-send-email 2.52.0 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=172.234.252.31; envelope-from=mchehab+huawei@kernel.org; helo=sea.source.kernel.org X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @kernel.org) X-ZM-MESSAGEID: 1769702721964154100 Sync QEMU kernel-doc with the latest upsteam version from linux-next, plus a set of 31 patches from myself addressing some issues with macro expansions and with changes specifically made to make easier to keep QEMU version in sync: - there is now a separate module containing all macro expansions, at xforms_lists.py; - such transforms can now be passed as a parameter when calling KernelFiles(). Such changes allow QEMU to easily override this class. While here, also override the ReST output format class. With that, all QEMU-specific changes are located on a single file (scripts/kernel-doc.py). On this patch, a note was added to scripts/kernel-doc.py to sum-up the differences between QEMU and Linux Kernel. Signed-off-by: Mauro Carvalho Chehab --- scripts/kernel-doc.py | 177 +++++++-- scripts/lib/kdoc/__init__.py | 0 scripts/lib/kdoc/enrich_formatter.py | 80 ++++ scripts/lib/kdoc/kdoc_files.py | 37 +- scripts/lib/kdoc/kdoc_item.py | 18 + scripts/lib/kdoc/kdoc_output.py | 120 ++++-- scripts/lib/kdoc/kdoc_parser.py | 432 +++++++++++---------- scripts/lib/kdoc/kdoc_re.py | 231 +++++++++--- scripts/lib/kdoc/latex_fonts.py | 184 +++++++++ scripts/lib/kdoc/parse_data_structs.py | 498 +++++++++++++++++++++++++ scripts/lib/kdoc/python_version.py | 190 ++++++++++ scripts/lib/kdoc/xforms_lists.py | 105 ++++++ 12 files changed, 1711 insertions(+), 361 deletions(-) create mode 100644 scripts/lib/kdoc/__init__.py create mode 100644 scripts/lib/kdoc/enrich_formatter.py create mode 100755 scripts/lib/kdoc/latex_fonts.py create mode 100755 scripts/lib/kdoc/parse_data_structs.py create mode 100644 scripts/lib/kdoc/python_version.py create mode 100644 scripts/lib/kdoc/xforms_lists.py diff --git a/scripts/kernel-doc.py b/scripts/kernel-doc.py index fc3d46ef519f..54bb3c38e54b 100755 --- a/scripts/kernel-doc.py +++ b/scripts/kernel-doc.py @@ -1,8 +1,29 @@ #!/usr/bin/env python3 # SPDX-License-Identifier: GPL-2.0 -# Copyright(c) 2025: Mauro Carvalho Chehab . +# Copyright(c) 2025-2026: Mauro Carvalho Chehab . # -# pylint: disable=3DC0103,R0915 +# pylint: disable=3DC0103,R0912,R0914,R0915 + +##########################################################################= ##### +# NOTE: +##########################################################################= ##### +# DON'T BLINDLY COPY THE LINUX KERNEL VERSION OF THIS FILE. +# +# This version contains QEMU-specific fork from Linux Kernel kernel-= doc. +# +# Differences against Linux Kernel upstream: +# - dropped python3 version checks; +# - the location of kernel-doc modules is different at QEMU tree; +# - the CTransforms class is overriden to add QEMU macros; +# - the RestFormat class is overriden to use some different regexes +# to match the way enum, struct, typedef, and union works on QEMU +# +# With such changes, syncing kernel-doc with a new kernel version +# should be as simple as: +# +# cp /docs/tools/lib/python/kdoc/*.py scripts/lib/kdo= c/ +##########################################################################= ##### + # # Converted from the kernel-doc script originally written in Perl # under GPLv2, copyrighted since 1998 by the following authors: @@ -79,19 +100,16 @@ # Yujie Liu =20 """ -kernel_doc -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -Print formatted kernel documentation to stdout +Print formatted kernel documentation to stdout. =20 Read C language source or header FILEs, extract embedded documentation comments, and print formatted documentation to standard output. =20 -The documentation comments are identified by the "/**" +The documentation comments are identified by the ``/**`` opening comment mark. =20 -See Documentation/doc-guide/kernel-doc.rst for the +See Linux Kernel Documentation/doc-guide/kernel-doc.rst for the documentation comment syntax. """ =20 @@ -102,13 +120,72 @@ =20 # Import Python modules =20 -LIB_DIR =3D "lib/kdoc" +# QEMU: the location of the library directory is different for QEMU +LIB_DIR =3D "lib" SRC_DIR =3D os.path.dirname(os.path.realpath(__file__)) =20 sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR)) =20 -from kdoc_files import KernelFiles # pylint: disable= =3DC0413 -from kdoc_output import RestFormat, ManFormat # pylint: disable= =3DC0413 +from kdoc.kdoc_files import KernelFiles # pylint: disable=3DC0= 415 +from kdoc.kdoc_re import KernRe # pylint: disable=3DC0= 415 +from kdoc.kdoc_output import RestFormat, ManFormat # pylint: disable=3DC0= 415 + +# QEMU: add QEMU-specific xforms +from kdoc.xforms_lists import CTransforms # pylint: disable=3DC0415 + +class QemuCTransforms(CTransforms): + def __init__(self): + self.function_xforms +=3D [ + # Add a handler for QEMU macros + (KernRe(r"QEMU_[A-Z_]+ +"), ""), + ] + +# QEMU prepend cross-references on a different way than the Linux Kernel +# So, let's redefine the RestFormat highlights + +# Those are currently identical to what Linux Kernel does: +type_constant =3D KernRe(r"\b``([^\`]+)``\b", cache=3DFalse) +type_constant2 =3D KernRe(r"\%([-_*\w]+)", cache=3DFalse) +type_func =3D KernRe(r"(\w+)\(\)", cache=3DFalse) +type_param_ref =3D KernRe(r"([\!~\*]?)\@(\w*((\.\w+)|(->\w+))*(\.\.\.)?)",= cache=3DFalse) +type_fp_param =3D KernRe(r"\@(\w+)\(\)", cache=3DFalse) +type_fp_param2 =3D KernRe(r"\@(\w+->\S+)\(\)", cache=3DFalse) + +# Those are QEMU-specific ones +type_enum =3D KernRe(r"#(enum\s*([_\w]+))", cache=3DFalse) +type_struct =3D KernRe(r"#(struct\s*([_\w]+))", cache=3DFalse) +type_typedef =3D KernRe(r"#(typedef\s*([_\w]+))", cache=3DFalse) +type_union =3D KernRe(r"#(union\s*([_\w]+))", cache=3DFalse) +type_member =3D KernRe(r"#([_\w]+)(\.|->)([_\w]+)", cache=3DFalse) +type_fallback =3D KernRe(r"#([_\w]+)", cache=3DFalse) +type_member_func =3D type_member + KernRe(r"\(\)", cache=3DFalse) + +# QEMU: override ReST highlights +class QemuRestFormat(RestFormat): + # The content here is identical to RestFormat, but it uses the local + # static vars above instead of the Kernel ones + highlights =3D [ + (type_constant, r"``\1``"), + (type_constant2, r"``\1``"), + + # Note: need to escape () to avoid func matching later + (type_member_func, r":c:type:`\1\2\3\\(\\) <\1>`"), + (type_member, r":c:type:`\1\2\3 <\1>`"), + (type_fp_param, r"**\1\\(\\)**"), + (type_fp_param2, r"**\1\\(\\)**"), + (type_func, r"\1()"), + (type_enum, r":c:type:`\1 <\2>`"), + (type_struct, r":c:type:`\1 <\2>`"), + (type_typedef, r":c:type:`\1 <\2>`"), + (type_union, r":c:type:`\1 <\2>`"), + + # in rst this can refer to any type + (type_fallback, r":c:type:`\1`"), + (type_param_ref, r"**\1\2**") + ] + + +WERROR_RETURN_CODE =3D 3 =20 DESC =3D """ Read C language source or header FILEs, extract embedded documentation com= ments, @@ -126,13 +203,13 @@ """ =20 EXPORT_DESC =3D """ -Only output documentation for the symbols that have been +Only output documentation for symbols that have been exported using EXPORT_SYMBOL() and related macros in any input FILE or -export-file FILE. """ =20 INTERNAL_DESC =3D """ -Only output documentation for the symbols that have NOT been +Only output documentation for symbols that have NOT been exported using EXPORT_SYMBOL() and related macros in any input FILE or -export-file FILE. """ @@ -155,28 +232,49 @@ """ =20 WARN_CONTENTS_BEFORE_SECTIONS_DESC =3D """ -Warns if there are contents before sections (deprecated). +Warn if there are contents before sections (deprecated). =20 This option is kept just for backward-compatibility, but it does nothing, neither here nor at the original Perl script. """ =20 +EPILOG =3D """ +The return value is: + +- 0: success or Python version is not compatible with +kernel-doc. If -Werror is not used, it will also +return 0 if there are issues at kernel-doc markups; + +- 1: an abnormal condition happened; + +- 2: argparse issued an error; + +- 3: When -Werror is used, it means that one or more unfiltered parse + warnings happened. +""" =20 class MsgFormatter(logging.Formatter): - """Helper class to format warnings on a similar way to kernel-doc.pl""" + """ + Helper class to capitalize errors and warnings, the same way + the venerable (now retired) kernel-doc.pl used to do. + """ =20 def format(self, record): record.levelname =3D record.levelname.capitalize() return logging.Formatter.format(self, record) =20 def main(): - """Main program""" + """ + Main program. + + """ =20 parser =3D argparse.ArgumentParser(formatter_class=3Dargparse.RawTextH= elpFormatter, - description=3DDESC) + description=3DDESC, epilog=3DEPILOG) =20 + # # Normal arguments - + # parser.add_argument("-v", "-verbose", "--verbose", action=3D"store_tru= e", help=3D"Verbose output, more warnings and other in= formation.") =20 @@ -191,8 +289,9 @@ def main(): action=3D"store_true", help=3D"Enable line number output (only in ReST mo= de)") =20 + # # Arguments to control the warning behavior - + # parser.add_argument("-Wreturn", "--wreturn", action=3D"store_true", help=3D"Warns about the lack of a return markup on= functions.") =20 @@ -213,8 +312,9 @@ def main(): parser.add_argument("-export-file", "--export-file", action=3D'append', help=3DEXPORT_FILE_DESC) =20 + # # Output format mutually-exclusive group - + # out_group =3D parser.add_argument_group("Output format selection (mutu= ally exclusive)") =20 out_fmt =3D out_group.add_mutually_exclusive_group() @@ -226,8 +326,9 @@ def main(): out_fmt.add_argument("-N", "-none", "--none", action=3D"store_true", help=3D"Do not output documentation, only warning= s.") =20 + # # Output selection mutually-exclusive group - + # sel_group =3D parser.add_argument_group("Output selection (mutually ex= clusive)") sel_mut =3D sel_group.add_mutually_exclusive_group() =20 @@ -240,12 +341,14 @@ def main(): sel_mut.add_argument("-s", "-function", "--symbol", action=3D'append', help=3DFUNCTION_DESC) =20 + # # Those are valid for all 3 types of filter + # parser.add_argument("-n", "-nosymbol", "--nosymbol", action=3D'append', help=3DNOSYMBOL_DESC) =20 parser.add_argument("-D", "-no-doc-sections", "--no-doc-sections", - action=3D'store_true', help=3D"Don't outputt DOC s= ections") + action=3D'store_true', help=3D"Don't output DOC se= ctions") =20 parser.add_argument("files", metavar=3D"FILE", nargs=3D"+", help=3DFILES_DESC) @@ -271,24 +374,18 @@ def main(): =20 logger.addHandler(handler) =20 - python_ver =3D sys.version_info[:2] - if python_ver < (3,6): - logger.warning("Python 3.6 or later is required by kernel-doc") - - # Return 0 here to avoid breaking compilation - sys.exit(0) - - if python_ver < (3,7): - logger.warning("Python 3.7 or later is required for correct result= s") + # + # Import kernel-doc libraries only after checking the Python version + # =20 if args.man: out_style =3D ManFormat(modulename=3Dargs.modulename) elif args.none: out_style =3D None else: - out_style =3D RestFormat() + out_style =3D QemuRestFormat() =20 - kfiles =3D KernelFiles(verbose=3Dargs.verbose, + kfiles =3D KernelFiles(verbose=3Dargs.verbose, xforms=3DQemuCTransform= s(), out_style=3Dout_style, werror=3Dargs.werror, wreturn=3Dargs.wreturn, wshort_desc=3Dargs.wshort= _desc, wcontents_before_sections=3Dargs.wcontents_before= _sections) @@ -308,18 +405,16 @@ def main(): sys.exit(0) =20 if args.werror: - print(f"{error_count} warnings as errors") - sys.exit(error_count) + print("%s warnings as errors" % error_count) # pylint: disable= =3DC0209 + sys.exit(WERROR_RETURN_CODE) =20 if args.verbose: - print(f"{error_count} errors") - - if args.none: - sys.exit(0) - - sys.exit(error_count) + print("%s errors" % error_count) # pylint: disable= =3DC0209 =20 + sys.exit(0) =20 +# # Call main method +# if __name__ =3D=3D "__main__": main() diff --git a/scripts/lib/kdoc/__init__.py b/scripts/lib/kdoc/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/scripts/lib/kdoc/enrich_formatter.py b/scripts/lib/kdoc/enrich= _formatter.py new file mode 100644 index 000000000000..d1be4e5e1962 --- /dev/null +++ b/scripts/lib/kdoc/enrich_formatter.py @@ -0,0 +1,80 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2025 by Mauro Carvalho Chehab . + +""" +Ancillary argparse HelpFormatter class that works on a similar way as +argparse.RawDescriptionHelpFormatter, e.g. description maintains line +breaks, but it also implement transformations to the help text. The +actual transformations ar given by enrich_text(), if the output is tty. + +Currently, the follow transformations are done: + + - Positional arguments are shown in upper cases; + - if output is TTY, ``var`` and positional arguments are shown prepend= ed + by an ANSI SGR code. This is usually translated to bold. On some + terminals, like, konsole, this is translated into a colored bold tex= t. +""" + +import argparse +import re +import sys + +class EnrichFormatter(argparse.HelpFormatter): + """ + Better format the output, making easier to identify the positional args + and how they're used at the __doc__ description. + """ + def __init__(self, *args, **kwargs): + """ + Initialize class and check if is TTY. + """ + super().__init__(*args, **kwargs) + self._tty =3D sys.stdout.isatty() + + def enrich_text(self, text): + r""" + Handle ReST markups (currently, only \`\`text\`\` markups). + """ + if self._tty and text: + # Replace ``text`` with ANSI SGR (bold) + return re.sub(r'\`\`(.+?)\`\`', + lambda m: f'\033[1m{m.group(1)}\033[0m', text) + return text + + def _fill_text(self, text, width, indent): + """ + Enrich descriptions with markups on it. + """ + enriched =3D self.enrich_text(text) + return "\n".join(indent + line for line in enriched.splitlines()) + + def _format_usage(self, usage, actions, groups, prefix): + """ + Enrich positional arguments at usage: line. + """ + + prog =3D self._prog + parts =3D [] + + for action in actions: + if action.option_strings: + opt =3D action.option_strings[0] + if action.nargs !=3D 0: + opt +=3D f" {action.dest.upper()}" + parts.append(f"[{opt}]") + else: + # Positional argument + parts.append(self.enrich_text(f"``{action.dest.upper()}``"= )) + + usage_text =3D f"{prefix or 'usage: '} {prog} {' '.join(parts)}\n" + return usage_text + + def _format_action_invocation(self, action): + """ + Enrich argument names. + """ + if not action.option_strings: + return self.enrich_text(f"``{action.dest.upper()}``") + + return ", ".join(action.option_strings) diff --git a/scripts/lib/kdoc/kdoc_files.py b/scripts/lib/kdoc/kdoc_files.py index 85365cc316d6..c35e033cf123 100644 --- a/scripts/lib/kdoc/kdoc_files.py +++ b/scripts/lib/kdoc/kdoc_files.py @@ -5,7 +5,8 @@ # pylint: disable=3DR0903,R0913,R0914,R0917 =20 """ -Parse lernel-doc tags on multiple kernel source files. +Classes for navigating through the files that kernel-doc needs to handle +to generate documentation. """ =20 import argparse @@ -13,8 +14,9 @@ import os import re =20 -from kdoc_parser import KernelDoc -from kdoc_output import OutputFormat +from kdoc.kdoc_parser import KernelDoc +from kdoc.xforms_lists import CTransforms +from kdoc.kdoc_output import OutputFormat =20 =20 class GlobSourceFiles: @@ -43,7 +45,7 @@ def __init__(self, srctree=3DNone, valid_extensions=3DNon= e): self.srctree =3D srctree =20 def _parse_dir(self, dirname): - """Internal function to parse files recursively""" + """Internal function to parse files recursively.""" =20 with os.scandir(dirname) as obj: for entry in obj: @@ -65,7 +67,7 @@ def _parse_dir(self, dirname): def parse_files(self, file_list, file_not_found_cb): """ Define an iterator to parse all source files from file_list, - handling directories if any + handling directories if any. """ =20 if not file_list: @@ -91,18 +93,18 @@ class KernelFiles(): =20 There are two type of parsers defined here: - self.parse_file(): parses both kernel-doc markups and - EXPORT_SYMBOL* macros; - - self.process_export_file(): parses only EXPORT_SYMBOL* macros. + ``EXPORT_SYMBOL*`` macros; + - self.process_export_file(): parses only ``EXPORT_SYMBOL*`` macro= s. """ =20 def warning(self, msg): - """Ancillary routine to output a warning and increment error count= """ + """Ancillary routine to output a warning and increment error count= .""" =20 self.config.log.warning(msg) self.errors +=3D 1 =20 def error(self, msg): - """Ancillary routine to output an error and increment error count"= "" + """Ancillary routine to output an error and increment error count.= """ =20 self.config.log.error(msg) self.errors +=3D 1 @@ -116,7 +118,7 @@ def parse_file(self, fname): if fname in self.files: return =20 - doc =3D KernelDoc(self.config, fname) + doc =3D KernelDoc(self.config, fname, self.xforms) export_table, entries =3D doc.parse_kdoc() =20 self.export_table[fname] =3D export_table @@ -128,7 +130,7 @@ def parse_file(self, fname): =20 def process_export_file(self, fname): """ - Parses EXPORT_SYMBOL* macros from a single Kernel source file. + Parses ``EXPORT_SYMBOL*`` macros from a single Kernel source file. """ =20 # Prevent parsing the same file twice if results are cached @@ -152,12 +154,12 @@ def file_not_found_cb(self, fname): =20 self.error(f"Cannot find file {fname}") =20 - def __init__(self, verbose=3DFalse, out_style=3DNone, + def __init__(self, verbose=3DFalse, out_style=3DNone, xforms=3DNone, werror=3DFalse, wreturn=3DFalse, wshort_desc=3DFalse, wcontents_before_sections=3DFalse, logger=3DNone): """ - Initialize startup variables and parse all files + Initialize startup variables and parse all files. """ =20 if not verbose: @@ -191,6 +193,11 @@ def __init__(self, verbose=3DFalse, out_style=3DNone, self.config.wshort_desc =3D wshort_desc self.config.wcontents_before_sections =3D wcontents_before_sections =20 + if xforms: + self.xforms =3D xforms + else: + self.xforms =3D CTransforms() + if not logger: self.config.log =3D logging.getLogger("kernel-doc") else: @@ -213,7 +220,7 @@ def __init__(self, verbose=3DFalse, out_style=3DNone, =20 def parse(self, file_list, export_file=3DNone): """ - Parse all files + Parse all files. """ =20 glob =3D GlobSourceFiles(srctree=3Dself.config.src_tree) @@ -242,7 +249,7 @@ def msg(self, enable_lineno=3DFalse, export=3DFalse, in= ternal=3DFalse, filenames=3DNone, export_file=3DNone): """ Interacts over the kernel-doc results and output messages, - returning kernel-doc markups on each interaction + returning kernel-doc markups on each interaction. """ =20 self.out_style.set_config(self.config) diff --git a/scripts/lib/kdoc/kdoc_item.py b/scripts/lib/kdoc/kdoc_item.py index 19805301cb2c..2b8a93f79716 100644 --- a/scripts/lib/kdoc/kdoc_item.py +++ b/scripts/lib/kdoc/kdoc_item.py @@ -4,7 +4,16 @@ # then pass into the output modules. # =20 +""" +Data class to store a kernel-doc Item. +""" + class KdocItem: + """ + A class that will, eventually, encapsulate all of the parsed data that= we + then pass into the output modules. + """ + def __init__(self, name, fname, type, start_line, **other_stuff): self.name =3D name self.fname =3D fname @@ -24,6 +33,9 @@ def __init__(self, name, fname, type, start_line, **other= _stuff): self.other_stuff =3D other_stuff =20 def get(self, key, default =3D None): + """ + Get a value from optional keys. + """ return self.other_stuff.get(key, default) =20 def __getitem__(self, key): @@ -33,10 +45,16 @@ def __getitem__(self, key): # Tracking of section and parameter information. # def set_sections(self, sections, start_lines): + """ + Set sections and start lines. + """ self.sections =3D sections self.section_start_lines =3D start_lines =20 def set_params(self, names, descs, types, starts): + """ + Set parameter list: names, descriptions, types and start lines. + """ self.parameterlist =3D names self.parameterdescs =3D descs self.parametertypes =3D types diff --git a/scripts/lib/kdoc/kdoc_output.py b/scripts/lib/kdoc/kdoc_output= .py index 25de79ea6bcd..4210b91dde5f 100644 --- a/scripts/lib/kdoc/kdoc_output.py +++ b/scripts/lib/kdoc/kdoc_output.py @@ -5,22 +5,24 @@ # pylint: disable=3DC0301,R0902,R0911,R0912,R0913,R0914,R0915,R0917 =20 """ -Implement output filters to print kernel-doc documentation. +Classes to implement output filters to print kernel-doc documentation. =20 -The implementation uses a virtual base class (OutputFormat) which +The implementation uses a virtual base class ``OutputFormat``. It contains dispatches to virtual methods, and some code to filter out output messages. =20 The actual implementation is done on one separate class per each type -of output. Currently, there are output classes for ReST and man/troff. +of output, e.g. ``RestFormat`` and ``ManFormat`` classes. + +Currently, there are output classes for ReST and man/troff. """ =20 import os import re from datetime import datetime =20 -from kdoc_parser import KernelDoc, type_param -from kdoc_re import KernRe +from kdoc.kdoc_parser import KernelDoc, type_param +from kdoc.kdoc_re import KernRe =20 =20 function_pointer =3D KernRe(r"([^\(]*\(\*)\s*\)\s*\(([^\)]*)\)", cache=3DF= alse) @@ -38,12 +40,12 @@ type_fp_param2 =3D KernRe(r"\@(\w+->\S+)\(\)", cache=3DFalse) =20 type_env =3D KernRe(r"(\$\w+)", cache=3DFalse) -type_enum =3D KernRe(r"#(enum\s*([_\w]+))", cache=3DFalse) -type_struct =3D KernRe(r"#(struct\s*([_\w]+))", cache=3DFalse) -type_typedef =3D KernRe(r"#(([A-Z][_\w]*))", cache=3DFalse) -type_union =3D KernRe(r"#(union\s*([_\w]+))", cache=3DFalse) -type_member =3D KernRe(r"#([_\w]+)(\.|->)([_\w]+)", cache=3DFalse) -type_fallback =3D KernRe(r"((?!))", cache=3DFalse) # this never matches +type_enum =3D KernRe(r"\&(enum\s*([_\w]+))", cache=3DFalse) +type_struct =3D KernRe(r"\&(struct\s*([_\w]+))", cache=3DFalse) +type_typedef =3D KernRe(r"\&(typedef\s*([_\w]+))", cache=3DFalse) +type_union =3D KernRe(r"\&(union\s*([_\w]+))", cache=3DFalse) +type_member =3D KernRe(r"\&([_\w]+)(\.|->)([_\w]+)", cache=3DFalse) +type_fallback =3D KernRe(r"\&([_\w]+)", cache=3DFalse) type_member_func =3D type_member + KernRe(r"\(\)", cache=3DFalse) =20 =20 @@ -54,16 +56,19 @@ class OutputFormat: """ =20 # output mode. - OUTPUT_ALL =3D 0 # output all symbols and doc sections - OUTPUT_INCLUDE =3D 1 # output only specified symbols - OUTPUT_EXPORTED =3D 2 # output exported symbols - OUTPUT_INTERNAL =3D 3 # output non-exported symbols + OUTPUT_ALL =3D 0 #: Output all symbols and doc sections. + OUTPUT_INCLUDE =3D 1 #: Output only specified symbols. + OUTPUT_EXPORTED =3D 2 #: Output exported symbols. + OUTPUT_INTERNAL =3D 3 #: Output non-exported symbols. =20 - # Virtual member to be overridden at the inherited classes + #: Highlights to be used in ReST format. highlights =3D [] =20 + #: Blank line character. + blankline =3D "" + def __init__(self): - """Declare internal vars and set mode to OUTPUT_ALL""" + """Declare internal vars and set mode to ``OUTPUT_ALL``.""" =20 self.out_mode =3D self.OUTPUT_ALL self.enable_lineno =3D None @@ -128,7 +133,7 @@ def out_warnings(self, args): self.config.warning(log_msg) =20 def check_doc(self, name, args): - """Check if DOC should be output""" + """Check if DOC should be output.""" =20 if self.no_doc_sections: return False @@ -177,7 +182,7 @@ def check_declaration(self, dtype, name, args): =20 def msg(self, fname, name, args): """ - Handles a single entry from kernel-doc parser + Handles a single entry from kernel-doc parser. """ =20 self.data =3D "" @@ -199,6 +204,10 @@ def msg(self, fname, name, args): self.out_enum(fname, name, args) return self.data =20 + if dtype =3D=3D "var": + self.out_var(fname, name, args) + return self.data + if dtype =3D=3D "typedef": self.out_typedef(fname, name, args) return self.data @@ -216,27 +225,31 @@ def msg(self, fname, name, args): # Virtual methods to be overridden by inherited classes # At the base class, those do nothing. def set_symbols(self, symbols): - """Get a list of all symbols from kernel_doc""" + """Get a list of all symbols from kernel_doc.""" =20 def out_doc(self, fname, name, args): - """Outputs a DOC block""" + """Outputs a DOC block.""" =20 def out_function(self, fname, name, args): - """Outputs a function""" + """Outputs a function.""" =20 def out_enum(self, fname, name, args): - """Outputs an enum""" + """Outputs an enum.""" + + def out_var(self, fname, name, args): + """Outputs a variable.""" =20 def out_typedef(self, fname, name, args): - """Outputs a typedef""" + """Outputs a typedef.""" =20 def out_struct(self, fname, name, args): - """Outputs a struct""" + """Outputs a struct.""" =20 =20 class RestFormat(OutputFormat): - """Consts and functions used by ReST output""" + """Consts and functions used by ReST output.""" =20 + #: Highlights to be used in ReST format highlights =3D [ (type_constant, r"``\1``"), (type_constant2, r"``\1``"), @@ -256,9 +269,13 @@ class RestFormat(OutputFormat): (type_fallback, r":c:type:`\1`"), (type_param_ref, r"**\1\2**") ] + blankline =3D "\n" =20 + #: Sphinx literal block regex. sphinx_literal =3D KernRe(r'^[^.].*::$', cache=3DFalse) + + #: Sphinx code block regex. sphinx_cblock =3D KernRe(r'^\.\.\ +code-block::', cache=3DFalse) =20 def __init__(self): @@ -273,7 +290,7 @@ def __init__(self): self.lineprefix =3D "" =20 def print_lineno(self, ln): - """Outputs a line number""" + """Outputs a line number.""" =20 if self.enable_lineno and ln is not None: ln +=3D 1 @@ -282,7 +299,7 @@ def print_lineno(self, ln): def output_highlight(self, args): """ Outputs a C symbol that may require being converted to ReST using - the self.highlights variable + the self.highlights variable. """ =20 input_text =3D args @@ -472,6 +489,25 @@ def out_enum(self, fname, name, args): self.lineprefix =3D oldprefix self.out_section(args) =20 + def out_var(self, fname, name, args): + oldprefix =3D self.lineprefix + ln =3D args.declaration_start_line + full_proto =3D args.other_stuff["full_proto"] + + self.lineprefix =3D " " + + self.data +=3D f"\n\n.. c:macro:: {name}\n\n{self.lineprefix}``{fu= ll_proto}``\n\n" + + self.print_lineno(ln) + self.output_highlight(args.get('purpose', '')) + self.data +=3D "\n" + + if args.other_stuff["default_val"]: + self.data +=3D f'{self.lineprefix}**Initialization**\n\n' + self.output_highlight(f'default: ``{args.other_stuff["default_= val"]}``') + + self.out_section(args) + def out_typedef(self, fname, name, args): =20 oldprefix =3D self.lineprefix @@ -544,7 +580,7 @@ def out_struct(self, fname, name, args): =20 =20 class ManFormat(OutputFormat): - """Consts and functions used by man pages output""" + """Consts and functions used by man pages output.""" =20 highlights =3D ( (type_constant, r"\1"), @@ -561,6 +597,7 @@ class ManFormat(OutputFormat): ) blankline =3D "" =20 + #: Allowed timestamp formats. date_formats =3D [ "%a %b %d %H:%M:%S %Z %Y", "%a %b %d %H:%M:%S %Y", @@ -627,7 +664,7 @@ def set_symbols(self, symbols): self.symbols =3D symbols =20 def out_tail(self, fname, name, args): - """Adds a tail for all man pages""" + """Adds a tail for all man pages.""" =20 # SEE ALSO section self.data +=3D f'.SH "SEE ALSO"' + "\n.PP\n" @@ -663,7 +700,7 @@ def msg(self, fname, name, args): def output_highlight(self, block): """ Outputs a C symbol that may require being highlighted with - self.highlights variable using troff syntax + self.highlights variable using troff syntax. """ =20 contents =3D self.highlight_block(block) @@ -694,7 +731,6 @@ def out_doc(self, fname, name, args): self.output_highlight(text) =20 def out_function(self, fname, name, args): - """output function in man""" =20 out_name =3D self.arg_name(args, name) =20 @@ -773,6 +809,26 @@ def out_enum(self, fname, name, args): self.data +=3D f'.SH "{section}"' + "\n" self.output_highlight(text) =20 + def out_var(self, fname, name, args): + out_name =3D self.arg_name(args, name) + full_proto =3D args.other_stuff["full_proto"] + + self.data +=3D f'.TH "{self.modulename}" 9 "{out_name}" "{self.man= _date}" "API Manual" LINUX' + "\n" + + self.data +=3D ".SH NAME\n" + self.data +=3D f"{name} \\- {args['purpose']}\n" + + self.data +=3D ".SH SYNOPSIS\n" + self.data +=3D f"{full_proto}\n" + + if args.other_stuff["default_val"]: + self.data +=3D f'.SH "Initialization"' + "\n" + self.output_highlight(f'default: {args.other_stuff["default_va= l"]}') + + for section, text in args.sections.items(): + self.data +=3D f'.SH "{section}"' + "\n" + self.output_highlight(text) + def out_typedef(self, fname, name, args): module =3D self.modulename purpose =3D args.get('purpose') diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser= .py index b2b790d6b837..a280fe581937 100644 --- a/scripts/lib/kdoc/kdoc_parser.py +++ b/scripts/lib/kdoc/kdoc_parser.py @@ -5,19 +5,16 @@ # pylint: disable=3DC0301,C0302,R0904,R0912,R0913,R0914,R0915,R0917,R1702 =20 """ -kdoc_parser -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D - -Read a C language source or header FILE and extract embedded -documentation comments +Classes and functions related to reading a C language source or header FILE +and extract embedded documentation comments from it. """ =20 import sys import re from pprint import pformat =20 -from kdoc_re import NestedMatch, KernRe -from kdoc_item import KdocItem +from kdoc.kdoc_re import CFunction, KernRe +from kdoc.kdoc_item import KdocItem =20 # # Regular expressions used to parse kernel-doc markups at KernelDoc class. @@ -53,7 +50,7 @@ doc_inline_start =3D KernRe(r'^\s*/\*\*\s*$', cache=3DFalse) doc_inline_sect =3D KernRe(r'\s*\*\s*(@\s*[\w][\w\.]*\s*):(.*)', cache=3DF= alse) doc_inline_end =3D KernRe(r'^\s*\*/\s*$', cache=3DFalse) -doc_inline_oneline =3D KernRe(r'^\s*/\*\*\s*(@[\w\s]+):\s*(.*)\s*\*/\s*$',= cache=3DFalse) +doc_inline_oneline =3D KernRe(r'^\s*/\*\*\s*(@\s*[\w][\w\.]*\s*):\s*(.*)\s= *\*/\s*$', cache=3DFalse) =20 export_symbol =3D KernRe(r'^\s*EXPORT_SYMBOL(_GPL)?\s*\(\s*(\w+)\s*\)\s*',= cache=3DFalse) export_symbol_ns =3D KernRe(r'^\s*EXPORT_SYMBOL_NS(_GPL)?\s*\(\s*(\w+)\s*,= \s*"\S+"\)\s*', cache=3DFalse) @@ -64,7 +61,7 @@ # Tests for the beginning of a kerneldoc block in its various forms. # doc_block =3D doc_com + KernRe(r'DOC:\s*(.*)?', cache=3DFalse) -doc_begin_data =3D KernRe(r"^\s*\*?\s*(struct|union|enum|typedef)\b\s*(\w*= )", cache =3D False) +doc_begin_data =3D KernRe(r"^\s*\*?\s*(struct|union|enum|typedef|var)\b\s*= (\w*)", cache =3D False) doc_begin_func =3D KernRe(str(doc_com) + # initial " * ' r"(?:\w+\s*\*\s*)?" + # type (not captured) r'(?:define\s+)?' + # possible "define" (not cap= tured) @@ -78,143 +75,22 @@ # struct_args_pattern =3D r'([^,)]+)' =20 -struct_xforms =3D [ - # Strip attributes - (KernRe(r"__attribute__\s*\(\([a-z0-9,_\*\s\(\)]*\)\)", flags=3Dre.I |= re.S, cache=3DFalse), ' '), - (KernRe(r'\s*__aligned\s*\([^;]*\)', re.S), ' '), - (KernRe(r'\s*__counted_by\s*\([^;]*\)', re.S), ' '), - (KernRe(r'\s*__counted_by_(le|be)\s*\([^;]*\)', re.S), ' '), - (KernRe(r'\s*__packed\s*', re.S), ' '), - (KernRe(r'\s*CRYPTO_MINALIGN_ATTR', re.S), ' '), - (KernRe(r'\s*__private', re.S), ' '), - (KernRe(r'\s*__rcu', re.S), ' '), - (KernRe(r'\s*____cacheline_aligned_in_smp', re.S), ' '), - (KernRe(r'\s*____cacheline_aligned', re.S), ' '), - (KernRe(r'\s*__cacheline_group_(begin|end)\([^\)]+\);'), ''), - # - # Unwrap struct_group macros based on this definition: - # __struct_group(TAG, NAME, ATTRS, MEMBERS...) - # which has variants like: struct_group(NAME, MEMBERS...) - # Only MEMBERS arguments require documentation. - # - # Parsing them happens on two steps: - # - # 1. drop struct group arguments that aren't at MEMBERS, - # storing them as STRUCT_GROUP(MEMBERS) - # - # 2. remove STRUCT_GROUP() ancillary macro. - # - # The original logic used to remove STRUCT_GROUP() using an - # advanced regex: - # - # \bSTRUCT_GROUP(\(((?:(?>[^)(]+)|(?1))*)\))[^;]*; - # - # with two patterns that are incompatible with - # Python re module, as it has: - # - # - a recursive pattern: (?1) - # - an atomic grouping: (?>...) - # - # I tried a simpler version: but it didn't work either: - # \bSTRUCT_GROUP\(([^\)]+)\)[^;]*; - # - # As it doesn't properly match the end parenthesis on some cases. - # - # So, a better solution was crafted: there's now a NestedMatch - # class that ensures that delimiters after a search are properly - # matched. So, the implementation to drop STRUCT_GROUP() will be - # handled in separate. - # - (KernRe(r'\bstruct_group\s*\(([^,]*,)', re.S), r'STRUCT_GROUP('), - (KernRe(r'\bstruct_group_attr\s*\(([^,]*,){2}', re.S), r'STRUCT_GROUP(= '), - (KernRe(r'\bstruct_group_tagged\s*\(([^,]*),([^,]*),', re.S), r'struct= \1 \2; STRUCT_GROUP('), - (KernRe(r'\b__struct_group\s*\(([^,]*,){3}', re.S), r'STRUCT_GROUP('), - # - # Replace macros - # - # TODO: use NestedMatch for FOO($1, $2, ...) matches - # - # it is better to also move those to the NestedMatch logic, - # to ensure that parentheses will be properly matched. - # - (KernRe(r'__ETHTOOL_DECLARE_LINK_MODE_MASK\s*\(([^\)]+)\)', re.S), - r'DECLARE_BITMAP(\1, __ETHTOOL_LINK_MODE_MASK_NBITS)'), - (KernRe(r'DECLARE_PHY_INTERFACE_MASK\s*\(([^\)]+)\)', re.S), - r'DECLARE_BITMAP(\1, PHY_INTERFACE_MODE_MAX)'), - (KernRe(r'DECLARE_BITMAP\s*\(' + struct_args_pattern + r',\s*' + struc= t_args_pattern + r'\)', - re.S), r'unsigned long \1[BITS_TO_LONGS(\2)]'), - (KernRe(r'DECLARE_HASHTABLE\s*\(' + struct_args_pattern + r',\s*' + st= ruct_args_pattern + r'\)', - re.S), r'unsigned long \1[1 << ((\2) - 1)]'), - (KernRe(r'DECLARE_KFIFO\s*\(' + struct_args_pattern + r',\s*' + struct= _args_pattern + - r',\s*' + struct_args_pattern + r'\)', re.S), r'\2 *\1'), - (KernRe(r'DECLARE_KFIFO_PTR\s*\(' + struct_args_pattern + r',\s*' + - struct_args_pattern + r'\)', re.S), r'\2 *\1'), - (KernRe(r'(?:__)?DECLARE_FLEX_ARRAY\s*\(' + struct_args_pattern + r',\= s*' + - struct_args_pattern + r'\)', re.S), r'\1 \2[]'), - (KernRe(r'DEFINE_DMA_UNMAP_ADDR\s*\(' + struct_args_pattern + r'\)', r= e.S), r'dma_addr_t \1'), - (KernRe(r'DEFINE_DMA_UNMAP_LEN\s*\(' + struct_args_pattern + r'\)', re= .S), r'__u32 \1'), -] -# -# Regexes here are guaranteed to have the end delimiter matching -# the start delimiter. Yet, right now, only one replace group -# is allowed. -# -struct_nested_prefixes =3D [ - (re.compile(r'\bSTRUCT_GROUP\('), r'\1'), -] =20 # -# Transforms for function prototypes +# Ancillary functions # -function_xforms =3D [ - (KernRe(r"^static +"), ""), - (KernRe(r"^extern +"), ""), - (KernRe(r"^asmlinkage +"), ""), - (KernRe(r"^inline +"), ""), - (KernRe(r"^__inline__ +"), ""), - (KernRe(r"^__inline +"), ""), - (KernRe(r"^__always_inline +"), ""), - (KernRe(r"^noinline +"), ""), - (KernRe(r"^__FORTIFY_INLINE +"), ""), - (KernRe(r"QEMU_[A-Z_]+ +"), ""), - (KernRe(r"__init +"), ""), - (KernRe(r"__init_or_module +"), ""), - (KernRe(r"__deprecated +"), ""), - (KernRe(r"__flatten +"), ""), - (KernRe(r"__meminit +"), ""), - (KernRe(r"__must_check +"), ""), - (KernRe(r"__weak +"), ""), - (KernRe(r"__sched +"), ""), - (KernRe(r"_noprof"), ""), - (KernRe(r"__always_unused *"), ""), - (KernRe(r"__printf\s*\(\s*\d*\s*,\s*\d*\s*\) +"), ""), - (KernRe(r"__(?:re)?alloc_size\s*\(\s*\d+\s*(?:,\s*\d+\s*)?\) +"), ""), - (KernRe(r"__diagnose_as\s*\(\s*\S+\s*(?:,\s*\d+\s*)*\) +"), ""), - (KernRe(r"DECL_BUCKET_PARAMS\s*\(\s*(\S+)\s*,\s*(\S+)\s*\)"), r"\1, \2= "), - (KernRe(r"__attribute_const__ +"), ""), - (KernRe(r"__attribute__\s*\(\((?:[\w\s]+(?:\([^)]*\))?\s*,?)+\)\)\s+")= , ""), -] =20 -# -# Apply a set of transforms to a block of text. -# -def apply_transforms(xforms, text): - for search, subst in xforms: - text =3D search.sub(subst, text) - return text - -# -# A little helper to get rid of excess white space -# multi_space =3D KernRe(r'\s\s+') def trim_whitespace(s): + """ + A little helper to get rid of excess white space. + """ return multi_space.sub(' ', s.strip()) =20 -# -# Remove struct/enum members that have been marked "private". -# def trim_private_members(text): - # + """ + Remove ``struct``/``enum`` members that have been marked "private". + """ # First look for a "public:" block that ends a private region, then # handle the "private until the end" case. # @@ -227,20 +103,21 @@ def trim_private_members(text): =20 class state: """ - State machine enums + States used by the parser's state machine. """ =20 # Parser states - NORMAL =3D 0 # normal code - NAME =3D 1 # looking for function name - DECLARATION =3D 2 # We have seen a declaration which might no= t be done - BODY =3D 3 # the body of the comment - SPECIAL_SECTION =3D 4 # doc section ending with a blank line - PROTO =3D 5 # scanning prototype - DOCBLOCK =3D 6 # documentation block - INLINE_NAME =3D 7 # gathering doc outside main block - INLINE_TEXT =3D 8 # reading the body of inline docs + NORMAL =3D 0 #: Normal code. + NAME =3D 1 #: Looking for function name. + DECLARATION =3D 2 #: We have seen a declaration which might n= ot be done. + BODY =3D 3 #: The body of the comment. + SPECIAL_SECTION =3D 4 #: Doc section ending with a blank line. + PROTO =3D 5 #: Scanning prototype. + DOCBLOCK =3D 6 #: Documentation block. + INLINE_NAME =3D 7 #: Gathering doc outside main block. + INLINE_TEXT =3D 8 #: Reading the body of inline docs. =20 + #: Names for each parser state. name =3D [ "NORMAL", "NAME", @@ -254,9 +131,12 @@ class state: ] =20 =20 -SECTION_DEFAULT =3D "Description" # default section +SECTION_DEFAULT =3D "Description" #: Default section. =20 class KernelEntry: + """ + Encapsulates a Kernel documentation entry. + """ =20 def __init__(self, config, fname, ln): self.config =3D config @@ -289,14 +169,16 @@ def __init__(self, config, fname, ln): # Management of section contents # def add_text(self, text): + """Add a new text to the entry contents list.""" self._contents.append(text) =20 def contents(self): + """Returns a string with all content texts that were added.""" return '\n'.join(self._contents) + '\n' =20 # TODO: rename to emit_message after removal of kernel-doc.pl def emit_msg(self, ln, msg, *, warning=3DTrue): - """Emit a message""" + """Emit a message.""" =20 log_msg =3D f"{self.fname}:{ln} {msg}" =20 @@ -310,10 +192,10 @@ def emit_msg(self, ln, msg, *, warning=3DTrue): self.warnings.append(log_msg) return =20 - # - # Begin a new section. - # def begin_section(self, line_no, title =3D SECTION_DEFAULT, dump =3D F= alse): + """ + Begin a new section. + """ if dump: self.dump_section(start_new =3D True) self.section =3D title @@ -367,18 +249,21 @@ class KernelDoc: documentation comments. """ =20 - # Section names - + #: Name of context section. section_context =3D "Context" + + #: Name of return section. section_return =3D "Return" =20 + #: String to write when a parameter is not described. undescribed =3D "-- undescribed --" =20 - def __init__(self, config, fname): + def __init__(self, config, fname, xforms): """Initialize internal variables""" =20 self.fname =3D fname self.config =3D config + self.xforms =3D xforms =20 # Initial state for the state machines self.state =3D state.NORMAL @@ -417,7 +302,7 @@ def emit_msg(self, ln, msg, *, warning=3DTrue): =20 def dump_section(self, start_new=3DTrue): """ - Dumps section contents to arrays/hashes intended for that purpose. + Dump section contents to arrays/hashes intended for that purpose. """ =20 if self.entry: @@ -426,9 +311,9 @@ def dump_section(self, start_new=3DTrue): # TODO: rename it to store_declaration after removal of kernel-doc.pl def output_declaration(self, dtype, name, **args): """ - Stores the entry into an entry array. + Store the entry into an entry array. =20 - The actual output and output filters will be handled elsewhere + The actual output and output filters will be handled elsewhere. """ =20 item =3D KdocItem(name, self.fname, dtype, @@ -449,24 +334,50 @@ def output_declaration(self, dtype, name, **args): =20 self.config.log.debug("Output: %s:%s =3D %s", dtype, name, pformat= (args)) =20 + def emit_unused_warnings(self): + """ + When the parser fails to produce a valid entry, it places some + warnings under `entry.warnings` that will be discarded when resett= ing + the state. + + Ensure that those warnings are not lost. + + .. note:: + + Because we are calling `config.warning()` here, those + warnings are not filtered by the `-W` parameters: they will = all + be produced even when `-Wreturn`, `-Wshort-desc`, and/or + `-Wcontents-before-sections` are used. + + Allowing those warnings to be filtered is complex, because it + would require storing them in a buffer and then filtering th= em + during the output step of the code, depending on the + selected symbols. + """ + if self.entry and self.entry not in self.entries: + for log_msg in self.entry.warnings: + self.config.warning(log_msg) + def reset_state(self, ln): """ Ancillary routine to create a new entry. It initializes all variables used by the state machine. """ =20 - # - # Flush the warnings out before we proceed further - # - if self.entry and self.entry not in self.entries: - for log_msg in self.entry.warnings: - self.config.log.warning(log_msg) + self.emit_unused_warnings() =20 self.entry =3D KernelEntry(self.config, self.fname, ln) =20 # State flags self.state =3D state.NORMAL =20 + def apply_transforms(self, xforms, text): + """Apply a set of transforms to a block of text.""" + for search, subst in xforms: + text =3D search.sub(subst, text) + + return text.strip() + def push_parameter(self, ln, decl_type, param, dtype, org_arg, declaration_name): """ @@ -664,10 +575,12 @@ def check_return_section(self, ln, declaration_name, = return_type): self.emit_msg(ln, f"No description found for return value of '{dec= laration_name}'") =20 - # - # Split apart a structure prototype; returns (struct|union, name, memb= ers) or None - # def split_struct_proto(self, proto): + """ + Split apart a structure prototype; returns (struct|union, name, + members) or ``None``. + """ + type_pattern =3D r'(struct|union)' qualifiers =3D [ "__attribute__", @@ -686,21 +599,26 @@ def split_struct_proto(self, proto): if r.search(proto): return (r.group(1), r.group(3), r.group(2)) return None - # - # Rewrite the members of a structure or union for easier formatting la= ter on. - # Among other things, this function will turn a member like: - # - # struct { inner_members; } foo; - # - # into: - # - # struct foo; inner_members; - # + def rewrite_struct_members(self, members): + """ + Process ``struct``/``union`` members from the most deeply nested + outward. + + Rewrite the members of a ``struct`` or ``union`` for easier format= ting + later on. Among other things, this function will turn a member lik= e:: + + struct { inner_members; } foo; + + into:: + + struct foo; inner_members; + """ + # - # Process struct/union members from the most deeply nested outward= . The - # trick is in the ^{ below - it prevents a match of an outer struc= t/union - # until the inner one has been munged (removing the "{" in the pro= cess). + # The trick is in the ``^{`` below - it prevents a match of an out= er + # ``struct``/``union`` until the inner one has been munged + # (removing the ``{`` in the process). # struct_members =3D KernRe(r'(struct|union)' # 0: declaration type r'([^\{\};]+)' # 1: possible name @@ -778,11 +696,12 @@ def rewrite_struct_members(self, members): tuples =3D struct_members.findall(members) return members =20 - # - # Format the struct declaration into a standard form for inclusion in = the - # resulting docs. - # def format_struct_decl(self, declaration): + """ + Format the ``struct`` declaration into a standard form for inclusi= on + in the resulting docs. + """ + # # Insert newlines, get rid of extra spaces. # @@ -816,7 +735,7 @@ def format_struct_decl(self, declaration): =20 def dump_struct(self, ln, proto): """ - Store an entry for a struct or union + Store an entry for a ``struct`` or ``union`` """ # # Do the basic parse to get the pieces of the declaration. @@ -835,11 +754,8 @@ def dump_struct(self, ln, proto): # Go through the list of members applying all of our transformatio= ns. # members =3D trim_private_members(members) - members =3D apply_transforms(struct_xforms, members) + members =3D self.apply_transforms(self.xforms.struct_xforms, membe= rs) =20 - nested =3D NestedMatch() - for search, sub in struct_nested_prefixes: - members =3D nested.sub(search, sub, members) # # Deal with embedded struct and union members, and drop enums enti= rely. # @@ -858,7 +774,7 @@ def dump_struct(self, ln, proto): =20 def dump_enum(self, ln, proto): """ - Stores an enum inside self.entries array. + Store an ``enum`` inside self.entries array. """ # # Strip preprocessor directives. Note that this depends on the @@ -928,9 +844,85 @@ def dump_enum(self, ln, proto): self.output_declaration('enum', declaration_name, purpose=3Dself.entry.declaration_purpose) =20 + def dump_var(self, ln, proto): + """ + Store variables that are part of kAPI. + """ + VAR_ATTRIBS =3D [ + "extern", + ] + OPTIONAL_VAR_ATTR =3D "^(?:" + "|".join(VAR_ATTRIBS) + ")?" + + # + # Store the full prototype before modifying it + # + full_proto =3D proto + declaration_name =3D None + + # + # Handle macro definitions + # + macro_prefixes =3D [ + KernRe(r"DEFINE_[\w_]+\s*\(([\w_]+)\)"), + ] + + for r in macro_prefixes: + match =3D r.search(proto) + if match: + declaration_name =3D match.group(1) + break + + # + # Drop comments and macros to have a pure C prototype + # + if not declaration_name: + proto =3D self.apply_transforms(self.xforms.var_xforms, proto) + + proto =3D proto.rstrip() + + # + # Variable name is at the end of the declaration + # + + default_val =3D None + + r=3D KernRe(OPTIONAL_VAR_ATTR + r"\s*[\w_\s]*\s+(?:\*+)?([\w_]+)\s= *[\d\]\[]*\s*(=3D.*)?") + if r.match(proto): + if not declaration_name: + declaration_name =3D r.group(1) + + default_val =3D r.group(2) + + self.config.log.debug("Variable proto parser: %s from '%s'", + r.groups(), proto) + + else: + r=3D KernRe(OPTIONAL_VAR_ATTR + r"(?:[\w_\s]*)?\s+(?:\*+)?(?:[= \w_]+)\s*[\d\]\[]*\s*(=3D.*)?") + + if r.match(proto): + default_val =3D r.group(1) + + if default_val: + self.config.log.debug("default: '%s'", default_val) + + if not declaration_name: + self.emit_msg(ln,f"{proto}: can't parse variable") + return + + if default_val: + default_val =3D default_val.lstrip("=3D").strip() + + self.config.log.debug("'%s' variable prototype: '%s', default: %s", + declaration_name, proto, default_val) + + self.output_declaration("var", declaration_name, + full_proto=3Dfull_proto, + default_val=3Ddefault_val, + purpose=3Dself.entry.declaration_purpose) + def dump_declaration(self, ln, prototype): """ - Stores a data declaration inside self.entries array. + Store a data declaration inside self.entries array. """ =20 if self.entry.decl_type =3D=3D "enum": @@ -939,22 +931,21 @@ def dump_declaration(self, ln, prototype): self.dump_typedef(ln, prototype) elif self.entry.decl_type in ["union", "struct"]: self.dump_struct(ln, prototype) + elif self.entry.decl_type =3D=3D "var": + self.dump_var(ln, prototype) else: # This would be a bug self.emit_message(ln, f'Unknown declaration type: {self.entry.= decl_type}') =20 def dump_function(self, ln, prototype): """ - Stores a function or function macro inside self.entries array. + Store a function or function macro inside self.entries array. """ =20 found =3D func_macro =3D False return_type =3D '' decl_type =3D 'function' - # - # Apply the initial transformations. - # - prototype =3D apply_transforms(function_xforms, prototype) + # # If we have a macro, remove the "#define" at the front. # @@ -973,6 +964,12 @@ def dump_function(self, ln, prototype): declaration_name =3D r.group(1) func_macro =3D True found =3D True + else: + # + # Apply the initial transformations. + # + prototype =3D self.apply_transforms(self.xforms.function_xform= s, + prototype) =20 # Yes, this truly is vile. We are looking for: # 1. Return type (may be nothing if we're looking at a macro) @@ -1046,7 +1043,7 @@ def dump_function(self, ln, prototype): =20 def dump_typedef(self, ln, proto): """ - Stores a typedef inside self.entries array. + Store a ``typedef`` inside self.entries array. """ # # We start by looking for function typedefs. @@ -1100,7 +1097,7 @@ def dump_typedef(self, ln, proto): @staticmethod def process_export(function_set, line): """ - process EXPORT_SYMBOL* tags + process ``EXPORT_SYMBOL*`` tags =20 This method doesn't use any variable from the class, so declare it with a staticmethod decorator. @@ -1131,7 +1128,7 @@ def process_export(function_set, line): =20 def process_normal(self, ln, line): """ - STATE_NORMAL: looking for the /** to begin everything. + STATE_NORMAL: looking for the ``/**`` to begin everything. """ =20 if not doc_start.match(line): @@ -1221,10 +1218,10 @@ def process_name(self, ln, line): else: self.emit_msg(ln, f"Cannot find identifier on line:\n{line}") =20 - # - # Helper function to determine if a new section is being started. - # def is_new_section(self, ln, line): + """ + Helper function to determine if a new section is being started. + """ if doc_sect.search(line): self.state =3D state.BODY # @@ -1256,10 +1253,10 @@ def is_new_section(self, ln, line): return True return False =20 - # - # Helper function to detect (and effect) the end of a kerneldoc commen= t. - # def is_comment_end(self, ln, line): + """ + Helper function to detect (and effect) the end of a kerneldoc comm= ent. + """ if doc_end.search(line): self.dump_section() =20 @@ -1278,7 +1275,7 @@ def is_comment_end(self, ln, line): =20 def process_decl(self, ln, line): """ - STATE_DECLARATION: We've seen the beginning of a declaration + STATE_DECLARATION: We've seen the beginning of a declaration. """ if self.is_new_section(ln, line) or self.is_comment_end(ln, line): return @@ -1307,7 +1304,7 @@ def process_decl(self, ln, line): =20 def process_special(self, ln, line): """ - STATE_SPECIAL_SECTION: a section ending with a blank line + STATE_SPECIAL_SECTION: a section ending with a blank line. """ # # If we have hit a blank line (only the " * " marker), then this @@ -1397,7 +1394,7 @@ def process_inline_text(self, ln, line): =20 def syscall_munge(self, ln, proto): # pylint: disable=3DW0613 """ - Handle syscall definitions + Handle syscall definitions. """ =20 is_void =3D False @@ -1436,7 +1433,7 @@ def syscall_munge(self, ln, proto): # pylint:= disable=3DW0613 =20 def tracepoint_munge(self, ln, proto): """ - Handle tracepoint definitions + Handle tracepoint definitions. """ =20 tracepointname =3D None @@ -1472,7 +1469,7 @@ def tracepoint_munge(self, ln, proto): return proto =20 def process_proto_function(self, ln, line): - """Ancillary routine to process a function prototype""" + """Ancillary routine to process a function prototype.""" =20 # strip C99-style comments to end of line line =3D KernRe(r"//.*$", re.S).sub('', line) @@ -1517,7 +1514,9 @@ def process_proto_function(self, ln, line): self.reset_state(ln) =20 def process_proto_type(self, ln, line): - """Ancillary routine to process a type""" + """ + Ancillary routine to process a type. + """ =20 # Strip C99-style comments and surrounding whitespace line =3D KernRe(r"//.*$", re.S).sub('', line).strip() @@ -1571,7 +1570,7 @@ def process_proto(self, ln, line): self.process_proto_type(ln, line) =20 def process_docblock(self, ln, line): - """STATE_DOCBLOCK: within a DOC: block.""" + """STATE_DOCBLOCK: within a ``DOC:`` block.""" =20 if doc_end.search(line): self.dump_section() @@ -1583,7 +1582,7 @@ def process_docblock(self, ln, line): =20 def parse_export(self): """ - Parses EXPORT_SYMBOL* macros from a single Kernel source file. + Parses ``EXPORT_SYMBOL*`` macros from a single Kernel source file. """ =20 export_table =3D set() @@ -1600,10 +1599,7 @@ def parse_export(self): =20 return export_table =20 - # - # The state/action table telling us which function to invoke in - # each state. - # + #: The state/action table telling us which function to invoke in each = state. state_actions =3D { state.NORMAL: process_normal, state.NAME: process_name, @@ -1665,6 +1661,8 @@ def parse_kdoc(self): # Hand this line to the appropriate state handler self.state_actions[self.state](self, ln, line) =20 + self.emit_unused_warnings() + except OSError: self.config.log.error(f"Error: Cannot open file {self.fname}") =20 diff --git a/scripts/lib/kdoc/kdoc_re.py b/scripts/lib/kdoc/kdoc_re.py index 2dfa1bf83d64..294051dbc050 100644 --- a/scripts/lib/kdoc/kdoc_re.py +++ b/scripts/lib/kdoc/kdoc_re.py @@ -51,6 +51,30 @@ def __str__(self): """ return self.regex.pattern =20 + def __repr__(self): + """ + Returns a displayable version of the class init. + """ + + flag_map =3D { + re.IGNORECASE: "re.I", + re.MULTILINE: "re.M", + re.DOTALL: "re.S", + re.VERBOSE: "re.X", + } + + flags =3D [] + for flag, name in flag_map.items(): + if self.regex.flags & flag: + flags.append(name) + + flags_name =3D " | ".join(flags) + + if flags_name: + return f'KernRe("{self.regex.pattern}", {flags_name})' + else: + return f'KernRe("{self.regex.pattern}")' + def __add__(self, other): """ Allows adding two regular expressions into one. @@ -61,7 +85,7 @@ def __add__(self, other): =20 def match(self, string): """ - Handles a re.match storing its results + Handles a re.match storing its results. """ =20 self.last_match =3D self.regex.match(string) @@ -69,40 +93,64 @@ def match(self, string): =20 def search(self, string): """ - Handles a re.search storing its results + Handles a re.search storing its results. """ =20 self.last_match =3D self.regex.search(string) return self.last_match =20 + def finditer(self, string): + """ + Alias to re.finditer. + """ + + return self.regex.finditer(string) + def findall(self, string): """ - Alias to re.findall + Alias to re.findall. """ =20 return self.regex.findall(string) =20 def split(self, string): """ - Alias to re.split + Alias to re.split. """ =20 return self.regex.split(string) =20 def sub(self, sub, string, count=3D0): """ - Alias to re.sub + Alias to re.sub. """ =20 return self.regex.sub(sub, string, count=3Dcount) =20 def group(self, num): """ - Returns the group results of the last match + Returns the group results of the last match. """ =20 return self.last_match.group(num) =20 + def groups(self): + """ + Returns the group results of the last match + """ + + return self.last_match.groups() + +#: Nested delimited pairs (brackets and parenthesis) +DELIMITER_PAIRS =3D { + '{': '}', + '(': ')', + '[': ']', +} + +#: compiled delimiters +RE_DELIM =3D KernRe(r'[\{\}\[\]\(\)]') + =20 class NestedMatch: """ @@ -110,7 +158,7 @@ class NestedMatch: even harder on Python with its normal re module, as there are several advanced regular expressions that are missing. =20 - This is the case of this pattern: + This is the case of this pattern:: =20 '\\bSTRUCT_GROUP(\\(((?:(?>[^)(]+)|(?1))*)\\))[^;]*;' =20 @@ -121,6 +169,7 @@ class NestedMatch: replace nested expressions. =20 The original approach was suggested by: + https://stackoverflow.com/questions/5454322/python-how-to-match-ne= sted-parentheses-with-regex =20 Although I re-implemented it to make it more generic and match 3 types @@ -128,38 +177,10 @@ class NestedMatch: will ignore the search string. """ =20 - # TODO: make NestedMatch handle multiple match groups - # - # Right now, regular expressions to match it are defined only up to - # the start delimiter, e.g.: - # - # \bSTRUCT_GROUP\( - # - # is similar to: STRUCT_GROUP\((.*)\) - # except that the content inside the match group is delimiter-aligned. - # - # The content inside parentheses is converted into a single replace - # group (e.g. r`\1'). - # - # It would be nice to change such definition to support multiple - # match groups, allowing a regex equivalent to: - # - # FOO\((.*), (.*), (.*)\) - # - # it is probably easier to define it not as a regular expression, but - # with some lexical definition like: - # - # FOO(arg1, arg2, arg3) + def __init__(self, regex): + self.regex =3D KernRe(regex) =20 - DELIMITER_PAIRS =3D { - '{': '}', - '(': ')', - '[': ']', - } - - RE_DELIM =3D re.compile(r'[\{\}\[\]\(\)]') - - def _search(self, regex, line): + def _search(self, line): """ Finds paired blocks for a regex that ends with a delimiter. =20 @@ -180,25 +201,46 @@ def _search(self, regex, line): """ =20 stack =3D [] + start =3D 0 + offset =3D 0 + pos =3D 0 =20 - for match_re in regex.finditer(line): + for match_re in self.regex.finditer(line): start =3D match_re.start() offset =3D match_re.end() + string_char =3D None + escape =3D False =20 d =3D line[offset - 1] - if d not in self.DELIMITER_PAIRS: + if d not in DELIMITER_PAIRS: continue =20 - end =3D self.DELIMITER_PAIRS[d] + end =3D DELIMITER_PAIRS[d] stack.append(end) =20 - for match in self.RE_DELIM.finditer(line[offset:]): + for match in RE_DELIM.finditer(line[offset:]): pos =3D match.start() + offset =20 d =3D line[pos] =20 - if d in self.DELIMITER_PAIRS: - end =3D self.DELIMITER_PAIRS[d] + if escape: + escape =3D False + continue + + if string_char: + if d =3D=3D '\\': + escape =3D True + elif d =3D=3D string_char: + string_char =3D None + + continue + + if d in ('"', "'"): + string_char =3D d + continue + + if d in DELIMITER_PAIRS: + end =3D DELIMITER_PAIRS[d] =20 stack.append(end) continue @@ -211,7 +253,12 @@ def _search(self, regex, line): yield start, offset, pos + 1 break =20 - def search(self, regex, line): + # When /* private */ is used, it may end the end delimiterq + if stack: + stack.pop() + yield start, offset, len(line) + 1 + + def search(self, line): """ This is similar to re.search: =20 @@ -219,19 +266,73 @@ def search(self, regex, line): returning occurrences only if all delimiters are paired. """ =20 - for t in self._search(regex, line): + for t in self._search(line): =20 yield line[t[0]:t[2]] =20 - def sub(self, regex, sub, line, count=3D0): + @staticmethod + def _split_args(all_args, delim=3D","): + """ + Helper method to split comma-separated function arguments + or struct elements, if delim is set to ";". + + It returns a list of arguments that can be used later on by + the sub() method. + """ + args =3D [all_args] + stack =3D [] + arg_start =3D 0 + string_char =3D None + escape =3D False + + for idx, d in enumerate(all_args): + if escape: + escape =3D False + continue + + if string_char: + if d =3D=3D '\\': + escape =3D True + elif d =3D=3D string_char: + string_char =3D None + + continue + + if d in ('"', "'"): + string_char =3D d + continue + + if d in DELIMITER_PAIRS: + end =3D DELIMITER_PAIRS[d] + + stack.append(end) + continue + + if stack and d =3D=3D stack[-1]: + stack.pop() + continue + + if d =3D=3D delim and not stack: + args.append(all_args[arg_start:idx].strip()) + arg_start =3D idx + 1 + + # Add the last argument (if any) + last =3D all_args[arg_start:].strip() + if last: + args.append(last) + + return args + + def sub(self, sub, line, count=3D0): """ This is similar to re.sub: =20 It matches a regex that it is followed by a delimiter, replacing occurrences only if all delimiters are paired. =20 - if r'\1' is used, it works just like re: it places there the - matched paired data with the delimiter stripped. + if r'\0' is used, it works on a similar way of using re.group(0): + it places the entire args of the matched paired data, with the + delimiter stripped. =20 If count is different than zero, it will replace at most count items. @@ -241,22 +342,22 @@ def sub(self, regex, sub, line, count=3D0): cur_pos =3D 0 n =3D 0 =20 - for start, end, pos in self._search(regex, line): + for start, end, pos in self._search(line): out +=3D line[cur_pos:start] =20 # Value, ignoring start/end delimiters value =3D line[end:pos - 1] =20 - # replaces \1 at the sub string, if \1 is used there + # replace arguments new_sub =3D sub - new_sub =3D new_sub.replace(r'\1', value) + if "\\" in sub: + args =3D self._split_args(value) + + new_sub =3D re.sub(r'\\(\d+)', + lambda m: args[int(m.group(1))], new_sub) =20 out +=3D new_sub =20 - # Drop end ';' if any - if line[pos] =3D=3D ';': - pos +=3D 1 - cur_pos =3D pos n +=3D 1 =20 @@ -268,3 +369,21 @@ def sub(self, regex, sub, line, count=3D0): out +=3D line[cur_pos:l] =20 return out + + def __repr__(self): + """ + Returns a displayable version of the class init. + """ + + return f'NestedMatch("{self.regex.regex.pattern}")' + + +class CFunction(NestedMatch): + r""" + Variant of NestedMatch. + + It overrides the init method to ensure that the regular expression will + start with a ``\b`` and end with a C function delimiter (open parenthe= sis). + """ + def __init__(self, regex): + self.regex =3D KernRe(r"\b" + regex + r"\s*\(") diff --git a/scripts/lib/kdoc/latex_fonts.py b/scripts/lib/kdoc/latex_fonts= .py new file mode 100755 index 000000000000..1d04cbda169f --- /dev/null +++ b/scripts/lib/kdoc/latex_fonts.py @@ -0,0 +1,184 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-only +# Copyright (C) Akira Yokosawa, 2024 +# +# Ported to Python by (c) Mauro Carvalho Chehab, 2025 + +""" +Detect problematic Noto CJK variable fonts +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +For ``make pdfdocs``, reports of build errors of translations.pdf started +arriving early 2024 [1]_ [2]_. It turned out that Fedora and openSUSE +tumbleweed have started deploying variable-font [3]_ format of "Noto CJK" +fonts [4]_ [5]_. For PDF, a LaTeX package named xeCJK is used for CJK +(Chinese, Japanese, Korean) pages. xeCJK requires XeLaTeX/XeTeX, which +does not (and likely never will) understand variable fonts for historical +reasons. + +The build error happens even when both of variable- and non-variable-format +fonts are found on the build system. To make matters worse, Fedora enlists +variable "Noto CJK" fonts in the requirements of langpacks-ja, -ko, -zh_CN, +-zh_TW, etc. Hence developers who have interest in CJK pages are more +likely to encounter the build errors. + +This script is invoked from the error path of "make pdfdocs" and emits +suggestions if variable-font files of "Noto CJK" fonts are in the list of +fonts accessible from XeTeX. + +.. [1] https://lore.kernel.org/r/8734tqsrt7.fsf@meer.lwn.net/ +.. [2] https://lore.kernel.org/r/1708585803.600323099@f111.i.mail.ru/ +.. [3] https://en.wikipedia.org/wiki/Variable_font +.. [4] https://fedoraproject.org/wiki/Changes/Noto_CJK_Variable_Fonts +.. [5] https://build.opensuse.org/request/show/1157217 + +Workarounds for building translations.pdf +----------------------------------------- + +* Denylist "variable font" Noto CJK fonts. + + - Create $HOME/deny-vf/fontconfig/fonts.conf from template below, with + tweaks if necessary. Remove leading "". + + - Path of fontconfig/fonts.conf can be overridden by setting an env + variable FONTS_CONF_DENY_VF. + + * Template:: + + + + + + + + + /usr/share/fonts/google-noto-*-cjk-vf-fonts + + /usr/share/fonts/truetype/Noto*CJK*-VF.otf + + + + + The denylisting is activated for "make pdfdocs". + +* For skipping CJK pages in PDF + + - Uninstall texlive-xecjk. + Denylisting is not needed in this case. + +* For printing CJK pages in PDF + + - Need non-variable "Noto CJK" fonts. + + * Fedora + + - google-noto-sans-cjk-fonts + - google-noto-serif-cjk-fonts + + * openSUSE tumbleweed + + - Non-variable "Noto CJK" fonts are not available as distro packages + as of April, 2024. Fetch a set of font files from upstream Noto + CJK Font released at: + + https://github.com/notofonts/noto-cjk/tree/main/Sans#super-otc + + and at: + + https://github.com/notofonts/noto-cjk/tree/main/Serif#super-otc + + then uncompress and deploy them. + - Remember to update fontconfig cache by running fc-cache. + +.. caution:: + Uninstalling "variable font" packages can be dangerous. + They might be depended upon by other packages important for your work. + Denylisting should be less invasive, as it is effective only while + XeLaTeX runs in "make pdfdocs". +""" + +import os +import re +import subprocess +import textwrap +import sys + +class LatexFontChecker: + """ + Detect problems with CJK variable fonts that affect PDF builds for + translations. + """ + + def __init__(self, deny_vf=3DNone): + if not deny_vf: + deny_vf =3D os.environ.get('FONTS_CONF_DENY_VF', "~/deny-vf") + + self.environ =3D os.environ.copy() + self.environ['XDG_CONFIG_HOME'] =3D os.path.expanduser(deny_vf) + + self.re_cjk =3D re.compile(r"([^:]+):\s*Noto\s+(Sans|Sans Mono|Ser= if) CJK") + + def description(self): + """ + Returns module description. + """ + return __doc__ + + def get_noto_cjk_vf_fonts(self): + """ + Get Noto CJK fonts. + """ + + cjk_fonts =3D set() + cmd =3D ["fc-list", ":", "file", "family", "variable"] + try: + result =3D subprocess.run(cmd,stdout=3Dsubprocess.PIPE, + stderr=3Dsubprocess.PIPE, + universal_newlines=3DTrue, + env=3Dself.environ, + check=3DTrue) + + except subprocess.CalledProcessError as exc: + sys.exit(f"Error running fc-list: {repr(exc)}") + + for line in result.stdout.splitlines(): + if 'variable=3DTrue' not in line: + continue + + match =3D self.re_cjk.search(line) + if match: + cjk_fonts.add(match.group(1)) + + return sorted(cjk_fonts) + + def check(self): + """ + Check for problems with CJK fonts. + """ + + fonts =3D textwrap.indent("\n".join(self.get_noto_cjk_vf_fonts()),= " ") + if not fonts: + return None + + rel_file =3D os.path.relpath(__file__, os.getcwd()) + + msg =3D "=3D" * 77 + "\n" + msg +=3D 'XeTeX is confused by "variable font" files listed below:= \n' + msg +=3D fonts + "\n" + msg +=3D textwrap.dedent(f""" + For CJK pages in PDF, they need to be hidden from XeTeX by= denylisting. + Or, CJK pages can be skipped by uninstalling texlive-xecjk. + + For more info on denylisting, other options, and variable = font, run: + + tools/docs/check-variable-fonts.py -h + """) + msg +=3D "=3D" * 77 + + return msg diff --git a/scripts/lib/kdoc/parse_data_structs.py b/scripts/lib/kdoc/pars= e_data_structs.py new file mode 100755 index 000000000000..9941cd19032e --- /dev/null +++ b/scripts/lib/kdoc/parse_data_structs.py @@ -0,0 +1,498 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# Copyright (c) 2016-2025 by Mauro Carvalho Chehab . +# pylint: disable=3DR0912,R0915 + +""" +Parse a source file or header, creating ReStructured Text cross references. + +It accepts an optional file to change the default symbol reference or to +suppress symbols from the output. + +It is capable of identifying ``define``, function, ``struct``, ``typedef``, +``enum`` and ``enum`` symbols and create cross-references for all of them. +It is also capable of distinguish #define used for specifying a Linux +ioctl. + +The optional rules file contains a set of rules like:: + + ignore ioctl VIDIOC_ENUM_FMT + replace ioctl VIDIOC_DQBUF vidioc_qbuf + replace define V4L2_EVENT_MD_FL_HAVE_FRAME_SEQ :c:type:`v4l2_event_mot= ion_det` +""" + +import os +import re +import sys + + +class ParseDataStructs: + """ + Creates an enriched version of a Kernel header file with cross-links + to each C data structure type. + + It is meant to allow having a more comprehensive documentation, where + uAPI headers will create cross-reference links to the code. + + It is capable of identifying ``define``, function, ``struct``, ``typed= ef``, + ``enum`` and ``enum`` symbols and create cross-references for all of t= hem. + It is also capable of distinguish #define used for specifying a Linux + ioctl. + + By default, it create rules for all symbols and defines, but it also + allows parsing an exception file. Such file contains a set of rules + using the syntax below: + + 1. Ignore rules:: + + ignore ` + + Removes the symbol from reference generation. + + 2. Replace rules:: + + replace + + Replaces how old_symbol with a new reference. The new_reference can= be: + + - A simple symbol name; + - A full Sphinx reference. + + 3. Namespace rules:: + + namespace + + Sets C namespace to be used during cross-reference generation. Can + be overridden by replace rules. + + On ignore and replace rules, ```` can be: + - ``ioctl``: for defines that end with ``_IO*``, e.g. ioctl defini= tions + - ``define``: for other defines + - ``symbol``: for symbols defined within enums; + - ``typedef``: for typedefs; + - ``enum``: for the name of a non-anonymous enum; + - ``struct``: for structs. + + Examples:: + + ignore define __LINUX_MEDIA_H + ignore ioctl VIDIOC_ENUM_FMT + replace ioctl VIDIOC_DQBUF vidioc_qbuf + replace define V4L2_EVENT_MD_FL_HAVE_FRAME_SEQ :c:type:`v4l2_event= _motion_det` + + namespace MC + """ + + #: Parser regex with multiple ways to capture enums. + RE_ENUMS =3D [ + re.compile(r"^\s*enum\s+([\w_]+)\s*\{"), + re.compile(r"^\s*enum\s+([\w_]+)\s*$"), + re.compile(r"^\s*typedef\s*enum\s+([\w_]+)\s*\{"), + re.compile(r"^\s*typedef\s*enum\s+([\w_]+)\s*$"), + ] + + #: Parser regex with multiple ways to capture structs. + RE_STRUCTS =3D [ + re.compile(r"^\s*struct\s+([_\w][\w\d_]+)\s*\{"), + re.compile(r"^\s*struct\s+([_\w][\w\d_]+)$"), + re.compile(r"^\s*typedef\s*struct\s+([_\w][\w\d_]+)\s*\{"), + re.compile(r"^\s*typedef\s*struct\s+([_\w][\w\d_]+)$"), + ] + + # NOTE: the original code was written a long time before Sphinx C + # domain to have multiple namespaces. To avoid to much turn at the + # existing hyperlinks, the code kept using "c:type" instead of the + # right types. To change that, we need to change the types not only + # here, but also at the uAPI media documentation. + + #: Dictionary containing C type identifiers to be transformed. + DEF_SYMBOL_TYPES =3D { + "ioctl": { + "prefix": "\\ ", + "suffix": "\\ ", + "ref_type": ":ref", + "description": "IOCTL Commands", + }, + "define": { + "prefix": "\\ ", + "suffix": "\\ ", + "ref_type": ":ref", + "description": "Macros and Definitions", + }, + # We're calling each definition inside an enum as "symbol" + "symbol": { + "prefix": "\\ ", + "suffix": "\\ ", + "ref_type": ":ref", + "description": "Enumeration values", + }, + "typedef": { + "prefix": "\\ ", + "suffix": "\\ ", + "ref_type": ":c:type", + "description": "Type Definitions", + }, + # This is the description of the enum itself + "enum": { + "prefix": "\\ ", + "suffix": "\\ ", + "ref_type": ":c:type", + "description": "Enumerations", + }, + "struct": { + "prefix": "\\ ", + "suffix": "\\ ", + "ref_type": ":c:type", + "description": "Structures", + }, + } + + def __init__(self, debug: bool =3D False): + """Initialize internal vars""" + self.debug =3D debug + self.data =3D "" + + self.symbols =3D {} + + self.namespace =3D None + self.ignore =3D [] + self.replace =3D [] + + for symbol_type in self.DEF_SYMBOL_TYPES: + self.symbols[symbol_type] =3D {} + + def read_exceptions(self, fname: str): + """ + Read an optional exceptions file, used to override defaults. + """ + + if not fname: + return + + name =3D os.path.basename(fname) + + with open(fname, "r", encoding=3D"utf-8", errors=3D"backslashrepla= ce") as f: + for ln, line in enumerate(f): + ln +=3D 1 + line =3D line.strip() + if not line or line.startswith("#"): + continue + + # ignore rules + match =3D re.match(r"^ignore\s+(\w+)\s+(\S+)", line) + + if match: + self.ignore.append((ln, match.group(1), match.group(2)= )) + continue + + # replace rules + match =3D re.match(r"^replace\s+(\S+)\s+(\S+)\s+(\S+)", li= ne) + if match: + self.replace.append((ln, match.group(1), match.group(2= ), + match.group(3))) + continue + + match =3D re.match(r"^namespace\s+(\S+)", line) + if match: + self.namespace =3D match.group(1) + continue + + sys.exit(f"{name}:{ln}: invalid line: {line}") + + def apply_exceptions(self): + """ + Process exceptions file with rules to ignore or replace references. + """ + + # Handle ignore rules + for ln, c_type, symbol in self.ignore: + if c_type not in self.DEF_SYMBOL_TYPES: + sys.exit(f"{name}:{ln}: {c_type} is invalid") + + d =3D self.symbols[c_type] + if symbol in d: + del d[symbol] + + # Handle replace rules + for ln, c_type, old, new in self.replace: + if c_type not in self.DEF_SYMBOL_TYPES: + sys.exit(f"{name}:{ln}: {c_type} is invalid") + + reftype =3D None + + # Parse reference type when the type is specified + + match =3D re.match(r"^\:c\:(\w+)\:\`(.+)\`", new) + if match: + reftype =3D f":c:{match.group(1)}" + new =3D match.group(2) + else: + match =3D re.search(r"(\:ref)\:\`(.+)\`", new) + if match: + reftype =3D match.group(1) + new =3D match.group(2) + + # If the replacement rule doesn't have a type, get default + if not reftype: + reftype =3D self.DEF_SYMBOL_TYPES[c_type].get("ref_type") + if not reftype: + reftype =3D self.DEF_SYMBOL_TYPES[c_type].get("real_ty= pe") + + new_ref =3D f"{reftype}:`{old} <{new}>`" + + # Change self.symbols to use the replacement rule + if old in self.symbols[c_type]: + (_, ln) =3D self.symbols[c_type][old] + self.symbols[c_type][old] =3D (new_ref, ln) + else: + print(f"{name}:{ln}: Warning: can't find {old} {c_type}") + + def store_type(self, ln, symbol_type: str, symbol: str, + ref_name: str =3D None, replace_underscores: bool =3D T= rue): + """ + Store a new symbol at self.symbols under symbol_type. + + By default, underscores are replaced by ``-``. + """ + defs =3D self.DEF_SYMBOL_TYPES[symbol_type] + + prefix =3D defs.get("prefix", "") + suffix =3D defs.get("suffix", "") + ref_type =3D defs.get("ref_type") + + # Determine ref_link based on symbol type + if ref_type or self.namespace: + if not ref_name: + ref_name =3D symbol.lower() + + # c-type references don't support hash + if ref_type =3D=3D ":ref" and replace_underscores: + ref_name =3D ref_name.replace("_", "-") + + # C domain references may have namespaces + if ref_type.startswith(":c:"): + if self.namespace: + ref_name =3D f"{self.namespace}.{ref_name}" + + if ref_type: + ref_link =3D f"{ref_type}:`{symbol} <{ref_name}>`" + else: + ref_link =3D f"`{symbol} <{ref_name}>`" + else: + ref_link =3D symbol + + self.symbols[symbol_type][symbol] =3D (f"{prefix}{ref_link}{suffix= }", ln) + + def store_line(self, line): + """ + Store a line at self.data, properly indented. + """ + line =3D " " + line.expandtabs() + self.data +=3D line.rstrip(" ") + + def parse_file(self, file_in: str, exceptions: str =3D None): + """ + Read a C source file and get identifiers. + """ + self.data =3D "" + is_enum =3D False + is_comment =3D False + multiline =3D "" + + self.read_exceptions(exceptions) + + with open(file_in, "r", + encoding=3D"utf-8", errors=3D"backslashreplace") as f: + for line_no, line in enumerate(f): + self.store_line(line) + line =3D line.strip("\n") + + # Handle continuation lines + if line.endswith(r"\\"): + multiline +=3D line[-1] + continue + + if multiline: + line =3D multiline + line + multiline =3D "" + + # Handle comments. They can be multilined + if not is_comment: + if re.search(r"/\*.*", line): + is_comment =3D True + else: + # Strip C99-style comments + line =3D re.sub(r"(//.*)", "", line) + + if is_comment: + if re.search(r".*\*/", line): + is_comment =3D False + else: + multiline =3D line + continue + + # At this point, line variable may be a multilined stateme= nt, + # if lines end with \ or if they have multi-line comments + # With that, it can safely remove the entire comments, + # and there's no need to use re.DOTALL for the logic below + + line =3D re.sub(r"(/\*.*\*/)", "", line) + if not line.strip(): + continue + + # It can be useful for debug purposes to print the file af= ter + # having comments stripped and multi-lines grouped. + if self.debug > 1: + print(f"line {line_no + 1}: {line}") + + # Now the fun begins: parse each type and store it. + + # We opted for a two parsing logic here due to: + # 1. it makes easier to debug issues not-parsed symbols; + # 2. we want symbol replacement at the entire content, not + # just when the symbol is detected. + + if is_enum: + match =3D re.match(r"^\s*([_\w][\w\d_]+)\s*[\,=3D]?", = line) + if match: + self.store_type(line_no, "symbol", match.group(1)) + if "}" in line: + is_enum =3D False + continue + + match =3D re.match(r"^\s*#\s*define\s+([\w_]+)\s+_IO", lin= e) + if match: + self.store_type(line_no, "ioctl", match.group(1), + replace_underscores=3DFalse) + continue + + match =3D re.match(r"^\s*#\s*define\s+([\w_]+)(\s+|$)", li= ne) + if match: + self.store_type(line_no, "define", match.group(1)) + continue + + match =3D re.match(r"^\s*typedef\s+([_\w][\w\d_]+)\s+(.*)\= s+([_\w][\w\d_]+);", + line) + if match: + name =3D match.group(2).strip() + symbol =3D match.group(3) + self.store_type(line_no, "typedef", symbol, ref_name= =3Dname) + continue + + for re_enum in self.RE_ENUMS: + match =3D re_enum.match(line) + if match: + self.store_type(line_no, "enum", match.group(1)) + is_enum =3D True + break + + for re_struct in self.RE_STRUCTS: + match =3D re_struct.match(line) + if match: + self.store_type(line_no, "struct", match.group(1)) + break + + self.apply_exceptions() + + def debug_print(self): + """ + Print debug information containing the replacement rules per symbo= l. + To make easier to check, group them per type. + """ + if not self.debug: + return + + for c_type, refs in self.symbols.items(): + if not refs: # Skip empty dictionaries + continue + + print(f"{c_type}:") + + for symbol, (ref, ln) in sorted(refs.items()): + print(f" #{ln:<5d} {symbol} -> {ref}") + + print() + + def gen_output(self): + """Write the formatted output to a file.""" + + # Avoid extra blank lines + text =3D re.sub(r"\s+$", "", self.data) + "\n" + text =3D re.sub(r"\n\s+\n", "\n\n", text) + + # Escape Sphinx special characters + text =3D re.sub(r"([\_\`\*\<\>\&\\\\:\/\|\%\$\#\{\}\~\^])", r"\\\1= ", text) + + # Source uAPI files may have special notes. Use bold font for them + text =3D re.sub(r"DEPRECATED", "**DEPRECATED**", text) + + # Delimiters to catch the entire symbol after escaped + start_delim =3D r"([ \n\t\(=3D\*\@])" + end_delim =3D r"(\s|,|\\=3D|\\:|\;|\)|\}|\{)" + + # Process all reference types + for ref_dict in self.symbols.values(): + for symbol, (replacement, _) in ref_dict.items(): + symbol =3D re.escape(re.sub(r"([\_\`\*\<\>\&\\\\:\/])", r"= \\\1", symbol)) + text =3D re.sub(fr'{start_delim}{symbol}{end_delim}', + fr'\1{replacement}\2', text) + + # Remove "\ " where not needed: before spaces and at the end of li= nes + text =3D re.sub(r"\\ ([\n ])", r"\1", text) + text =3D re.sub(r" \\ ", " ", text) + + return text + + def gen_toc(self): + """ + Create a list of symbols to be part of a TOC contents table. + """ + text =3D [] + + # Sort symbol types per description + symbol_descriptions =3D [] + for k, v in self.DEF_SYMBOL_TYPES.items(): + symbol_descriptions.append((v['description'], k)) + + symbol_descriptions.sort() + + # Process each category + for description, c_type in symbol_descriptions: + + refs =3D self.symbols[c_type] + if not refs: # Skip empty categories + continue + + text.append(f"{description}") + text.append("-" * len(description)) + text.append("") + + # Sort symbols alphabetically + for symbol, (ref, ln) in sorted(refs.items()): + text.append(f"- LINENO_{ln}: {ref}") + + text.append("") # Add empty line between categories + + return "\n".join(text) + + def write_output(self, file_in: str, file_out: str, toc: bool): + """ + Write a ReST output file. + """ + + title =3D os.path.basename(file_in) + + if toc: + text =3D self.gen_toc() + else: + text =3D self.gen_output() + + with open(file_out, "w", encoding=3D"utf-8", errors=3D"backslashre= place") as f: + f.write(".. -*- coding: utf-8; mode: rst -*-\n\n") + f.write(f"{title}\n") + f.write("=3D" * len(title) + "\n\n") + + if not toc: + f.write(".. parsed-literal::\n\n") + + f.write(text) diff --git a/scripts/lib/kdoc/python_version.py b/scripts/lib/kdoc/python_v= ersion.py new file mode 100644 index 000000000000..4ddb7ead5f56 --- /dev/null +++ b/scripts/lib/kdoc/python_version.py @@ -0,0 +1,190 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2017-2025 Mauro Carvalho Chehab + +""" +Handle Python version check logic. + +Not all Python versions are supported by scripts. Yet, on some cases, +like during documentation build, a newer version of python could be +available. + +This class allows checking if the minimal requirements are followed. + +Better than that, PythonVersion.check_python() not only checks the minimal +requirements, but it automatically switches to a the newest available +Python version if present. + +""" + +import os +import re +import subprocess +import shlex +import sys + +from glob import glob +from textwrap import indent + +class PythonVersion: + """ + Ancillary methods that checks for missing dependencies for different + types of types, like binaries, python modules, rpm deps, etc. + """ + + def __init__(self, version): + """ + =C3=8Fnitialize self.version tuple from a version string. + """ + self.version =3D self.parse_version(version) + + @staticmethod + def parse_version(version): + """ + Convert a major.minor.patch version into a tuple. + """ + return tuple(int(x) for x in version.split(".")) + + @staticmethod + def ver_str(version): + """ + Returns a version tuple as major.minor.patch. + """ + return ".".join([str(x) for x in version]) + + @staticmethod + def cmd_print(cmd, max_len=3D80): + """ + Outputs a command line, repecting maximum width. + """ + + cmd_line =3D [] + + for w in cmd: + w =3D shlex.quote(w) + + if cmd_line: + if not max_len or len(cmd_line[-1]) + len(w) < max_len: + cmd_line[-1] +=3D " " + w + continue + else: + cmd_line[-1] +=3D " \\" + cmd_line.append(w) + else: + cmd_line.append(w) + + return "\n ".join(cmd_line) + + def __str__(self): + """ + Return a version tuple as major.minor.patch from self.version. + """ + return self.ver_str(self.version) + + @staticmethod + def get_python_version(cmd): + """ + Get python version from a Python binary. As we need to detect if + are out there newer python binaries, we can't rely on sys.release = here. + """ + + kwargs =3D {} + if sys.version_info < (3, 7): + kwargs['universal_newlines'] =3D True + else: + kwargs['text'] =3D True + + result =3D subprocess.run([cmd, "--version"], + stdout =3D subprocess.PIPE, + stderr =3D subprocess.PIPE, + **kwargs, check=3DFalse) + + version =3D result.stdout.strip() + + match =3D re.search(r"(\d+\.\d+\.\d+)", version) + if match: + return PythonVersion.parse_version(match.group(1)) + + print(f"Can't parse version {version}") + return (0, 0, 0) + + @staticmethod + def find_python(min_version): + """ + Detect if are out there any python 3.xy version newer than the + current one. + + Note: this routine is limited to up to 2 digits for python3. We + may need to update it one day, hopefully on a distant future. + """ + patterns =3D [ + "python3.[0-9][0-9]", + "python3.[0-9]", + ] + + python_cmd =3D [] + + # Seek for a python binary newer than min_version + for path in os.getenv("PATH", "").split(":"): + for pattern in patterns: + for cmd in glob(os.path.join(path, pattern)): + if os.path.isfile(cmd) and os.access(cmd, os.X_OK): + version =3D PythonVersion.get_python_version(cmd) + if version >=3D min_version: + python_cmd.append((version, cmd)) + + return sorted(python_cmd, reverse=3DTrue) + + @staticmethod + def check_python(min_version, show_alternatives=3DFalse, bail_out=3DFa= lse, + success_on_error=3DFalse): + """ + Check if the current python binary satisfies our minimal requireme= nt + for Sphinx build. If not, re-run with a newer version if found. + """ + cur_ver =3D sys.version_info[:3] + if cur_ver >=3D min_version: + ver =3D PythonVersion.ver_str(cur_ver) + return + + python_ver =3D PythonVersion.ver_str(cur_ver) + + available_versions =3D PythonVersion.find_python(min_version) + if not available_versions: + print(f"ERROR: Python version {python_ver} is not supported an= ymore\n") + print(" Can't find a new version. This script may fail") + return + + script_path =3D os.path.abspath(sys.argv[0]) + + # Check possible alternatives + if available_versions: + new_python_cmd =3D available_versions[0][1] + else: + new_python_cmd =3D None + + if show_alternatives and available_versions: + print("You could run, instead:") + for _, cmd in available_versions: + args =3D [cmd, script_path] + sys.argv[1:] + + cmd_str =3D indent(PythonVersion.cmd_print(args), " ") + print(f"{cmd_str}\n") + + if bail_out: + msg =3D f"Python {python_ver} not supported. Bailing out" + if success_on_error: + print(msg, file=3Dsys.stderr) + sys.exit(0) + else: + sys.exit(msg) + + print(f"Python {python_ver} not supported. Changing to {new_python= _cmd}") + + # Restart script using the newer version + args =3D [new_python_cmd, script_path] + sys.argv[1:] + + try: + os.execv(new_python_cmd, args) + except OSError as e: + sys.exit(f"Failed to restart with {new_python_cmd}: {e}") diff --git a/scripts/lib/kdoc/xforms_lists.py b/scripts/lib/kdoc/xforms_lis= ts.py new file mode 100644 index 000000000000..6e917beceb89 --- /dev/null +++ b/scripts/lib/kdoc/xforms_lists.py @@ -0,0 +1,105 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# Copyright(c) 2026: Mauro Carvalho Chehab . + +import re + +from kdoc.kdoc_re import CFunction, KernRe + +struct_args_pattern =3D r'([^,)]+)' + +class CTransforms: + """ + Data class containing a long set of transformations to turn + structure member prefixes, and macro invocations and variables + into something we can parse and generate kdoc for. + """ + + #: Transforms for structs and unions + struct_xforms =3D [ + (CFunction("__attribute__"), ' '), + (CFunction('__aligned'), ' '), + (CFunction('__counted_by'), ' '), + (CFunction('__counted_by_(le|be)'), ' '), + (CFunction('__guarded_by'), ' '), + (CFunction('__pt_guarded_by'), ' '), + + (KernRe(r'\s*__packed\s*', re.S), ' '), + (KernRe(r'\s*CRYPTO_MINALIGN_ATTR', re.S), ' '), + (KernRe(r'\s*__private', re.S), ' '), + (KernRe(r'\s*__rcu', re.S), ' '), + (KernRe(r'\s*____cacheline_aligned_in_smp', re.S), ' '), + (KernRe(r'\s*____cacheline_aligned', re.S), ' '), + + (CFunction('__cacheline_group_(begin|end)'), ''), + + (CFunction('struct_group'), r'\2'), + (CFunction('struct_group_attr'), r'\3'), + (CFunction('struct_group_tagged'), r'struct \1 \2; \3'), + (CFunction('__struct_group'), r'\4'), + + (CFunction('__ETHTOOL_DECLARE_LINK_MODE_MASK'), r'DECLARE_BITMAP(\= 1, __ETHTOOL_LINK_MODE_MASK_NBITS)'), + (CFunction('DECLARE_PHY_INTERFACE_MASK',), r'DECLARE_BITMAP(\1, PH= Y_INTERFACE_MODE_MAX)'), + (CFunction('DECLARE_BITMAP'), r'unsigned long \1[BITS_TO_LONGS(\2)= ]'), + + (CFunction('DECLARE_HASHTABLE'), r'unsigned long \1[1 << ((\2) - 1= )]'), + (CFunction('DECLARE_KFIFO'), r'\2 *\1'), + (CFunction('DECLARE_KFIFO_PTR'), r'\2 *\1'), + (CFunction('(?:__)?DECLARE_FLEX_ARRAY'), r'\1 \2[]'), + (CFunction('DEFINE_DMA_UNMAP_ADDR'), r'dma_addr_t \1'), + (CFunction('DEFINE_DMA_UNMAP_LEN'), r'__u32 \1'), + (CFunction('VIRTIO_DECLARE_FEATURES'), r'union { u64 \1; u64 \1_ar= ray[VIRTIO_FEATURES_U64S]; }'), + ] + + #: Transforms for function prototypes + function_xforms =3D [ + (KernRe(r"^static +"), ""), + (KernRe(r"^extern +"), ""), + (KernRe(r"^asmlinkage +"), ""), + (KernRe(r"^inline +"), ""), + (KernRe(r"^__inline__ +"), ""), + (KernRe(r"^__inline +"), ""), + (KernRe(r"^__always_inline +"), ""), + (KernRe(r"^noinline +"), ""), + (KernRe(r"^__FORTIFY_INLINE +"), ""), + (KernRe(r"__init +"), ""), + (KernRe(r"__init_or_module +"), ""), + (KernRe(r"__deprecated +"), ""), + (KernRe(r"__flatten +"), ""), + (KernRe(r"__meminit +"), ""), + (KernRe(r"__must_check +"), ""), + (KernRe(r"__weak +"), ""), + (KernRe(r"__sched +"), ""), + (KernRe(r"_noprof"), ""), + (KernRe(r"__always_unused *"), ""), + (KernRe(r"__printf\s*\(\s*\d*\s*,\s*\d*\s*\) +"), ""), + (KernRe(r"__(?:re)?alloc_size\s*\(\s*\d+\s*(?:,\s*\d+\s*)?\) +"), = ""), + (KernRe(r"__diagnose_as\s*\(\s*\S+\s*(?:,\s*\d+\s*)*\) +"), ""), + (KernRe(r"DECL_BUCKET_PARAMS\s*\(\s*(\S+)\s*,\s*(\S+)\s*\)"), r"\1= , \2"), + (KernRe(r"__no_context_analysis\s*"), ""), + (KernRe(r"__attribute_const__ +"), ""), + + (CFunction("__cond_acquires"), ""), + (CFunction("__cond_releases"), ""), + (CFunction("__acquires"), ""), + (CFunction("__releases"), ""), + (CFunction("__must_hold"), ""), + (CFunction("__must_not_hold"), ""), + (CFunction("__must_hold_shared"), ""), + (CFunction("__cond_acquires_shared"), ""), + (CFunction("__acquires_shared"), ""), + (CFunction("__releases_shared"), ""), + (CFunction("__attribute__"), ""), + ] + + #: Transforms for variables + var_xforms =3D [ + (KernRe(r"__read_mostly"), ""), + (KernRe(r"__ro_after_init"), ""), + (KernRe(r'\s*__guarded_by\s*\([^\)]*\)', re.S), ""), + (KernRe(r'\s*__pt_guarded_by\s*\([^\)]*\)', re.S), ""), + (KernRe(r"LIST_HEAD\(([\w_]+)\)"), r"struct list_head \1"), + (KernRe(r"(?://.*)$"), ""), + (KernRe(r"(?:/\*.*\*/)"), ""), + (KernRe(r";$"), ""), + ] --=20 2.52.0