From nobody Sun Sep 28 17:04:58 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org ARC-Seal: i=1; a=rsa-sha256; t=1756381220; cv=none; d=zohomail.com; s=zohoarc; b=XSGrdAk+j98dV2Z8go/+v9XcI47dny6b116Uo+UShGiOgi9dM2RFGVTlAKK1jYMSOLU0J/8chi2SkyHNmgjA8SsDwA2uJn0qfKuOF9T4uirQas/MKleMHjT+w+6UlqwfenR6twNXTU2zITyNaH7H5gWV21s1AuCAwThPgZIGbP4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1756381220; h=Content-Type:Content-Transfer-Encoding:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To:Cc; bh=ML9Xq7399KsE/ai8FKHEWYvYcSHNH3BaqyKhr9/ikE4=; b=a1X98Urzhm/MDg3bOgXM47lFM+muw1EzzDV3e4RKyvuohuilJ3yrPdP7LgvciR62zAWlhp0kZ1pFN2ejFojKrOqDb/qf4AbeCwtMUSY8/dDq/LLkIyRt2rg70XrZzNG1z5mKv1G2UvKPjEiFRVA7iibfmY4I/Y5vzGeEyoW7CwU= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1756381220857953.5574964405147; Thu, 28 Aug 2025 04:40:20 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1urauG-0003Xe-HR; Thu, 28 Aug 2025 07:35:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1urauD-0003Te-3r for qemu-devel@nongnu.org; Thu, 28 Aug 2025 07:35:01 -0400 Received: from mail-wm1-x32a.google.com ([2a00:1450:4864:20::32a]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1urau7-0005nt-Ug for qemu-devel@nongnu.org; Thu, 28 Aug 2025 07:35:00 -0400 Received: by mail-wm1-x32a.google.com with SMTP id 5b1f17b1804b1-45b618b7d33so6928195e9.1 for ; Thu, 28 Aug 2025 04:34:55 -0700 (PDT) Received: from orth.archaic.org.uk (orth.archaic.org.uk. [2001:8b0:1d0::2]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-45b79799c33sm28691015e9.5.2025.08.28.04.34.51 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 Aug 2025 04:34:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; t=1756380894; x=1756985694; darn=nongnu.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=ML9Xq7399KsE/ai8FKHEWYvYcSHNH3BaqyKhr9/ikE4=; b=RC2sWo1Y29d4jSi8bE9QRRKnF0KB0hGI831dcenLciGsY09V2jxP1tHPww+Dsuyi4T FsaKntOyt8d+TtbVSInin7yL4VMngLqHAhh6TeGNAfJXW7uTsY3FEp9CJrByLP3dlS91 BSqcG/oWOjt64+TUw9hmwHLbdQ+KIs2SgiNWeEql4OHnyKS5E0aBnF+dat0mJDI7IwAf 5rYXVTscC9y9fhmQP/Xsc/tDlzsQ0iBCYI1yfEgZh0v3aizGp/HRBtkHjHua/rCsQ4d3 j7BAO5Xjnz8xS0/IFc2CQCTdbMbBgoSJ28nxh4XdeGmqkBsgZs11FdKsIbI844C2kw7I SiFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1756380894; x=1756985694; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ML9Xq7399KsE/ai8FKHEWYvYcSHNH3BaqyKhr9/ikE4=; b=U4kbg+en6ztkeVoqtKJ6ZRxhm+AWV5w5nK+Pj1zvvstH5Ax2qyykt0bHyTAEdjMq9v uKS1kBT1m+CY6vug5GikYD7YUftGWCU7qkABpFojYPPFpZZqZtP8Ey//P51keWgOgu2s ViAPYPU5O/4/pqkMmN3zY2L9ugKUhs4y5MqZJXkSRISchELLFDoMb3F5Liwf3Iud6lEN onDrpVfpYoUELpsDJQn7k7ZYPbuFdPmSFguiVGdZ9TZ+N74ieaqyDwAuWO38Mxujgxea Ezm1a0P2KoJukavSCXO0WjaXBHpewJY/77PUuNCmlk70T1gkYwDB7ItoTVLDEuMNZTkk lG2g== X-Gm-Message-State: AOJu0Ywvk9mzcOl1SqlcN58a8KxM7plb1CzidndvYaeDg4OK7fkY1MLE M1Pw8fo87WNHURa7IN0OXxV1Sb3UGLZw9jcNNkDOisjJoAmwT2F2yCg/E1Iw7EIAY9fKIUOpu2Y 6BJWv X-Gm-Gg: ASbGncsTAioDS726Pm5xZshp96rz4IjQlVhOFsXp5JUOjGaRpSwirnfp6OFvxEVRjqi fS+6fA9DRtZCQJZNxEPQZ+YO7LjE6joepAzVdh3O82iwzfqAMqVBtT2FwsKTDZfXP8tODr40INK C8MinwQxZfTSuJgp+RgOz51PtgC10E8Pq8drrwlgLM6WNTfAuSMddYT1BGyrSszh3V6xzIgcDBH TnN2HAasLmHIf/nGWlEDO1UsoTnUleIovQpkgLJva1Q37lp4bJ7jeTTIZQ9HkaCk1lBAGYiTrAt vIeNSG/AKQ3/R4KMwx5o8ouXpJhEJqW3nlAPpgSdmV8fNxx6DpdkdIxgrFDbsIDBvXNNs27uVZ+ v5dSWvyGjisYTNPHG/0bHkcge00+X X-Google-Smtp-Source: AGHT+IGgNGwVXSmowtSDgqsxtYV3QfCC8R2yKLxqNQCEAHuxmmzJbqxjx2w9ykbNmhIeyMZrUcMsOA== X-Received: by 2002:a05:600c:1f95:b0:459:dc92:b95f with SMTP id 5b1f17b1804b1-45b5cb461f6mr140554375e9.5.1756380892761; Thu, 28 Aug 2025 04:34:52 -0700 (PDT) From: Peter Maydell To: qemu-devel@nongnu.org Subject: [PULL 18/32] scripts: Import Python kerneldoc from Linux kernel Date: Thu, 28 Aug 2025 12:34:15 +0100 Message-ID: <20250828113430.3214314-19-peter.maydell@linaro.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250828113430.3214314-1-peter.maydell@linaro.org> References: <20250828113430.3214314-1-peter.maydell@linaro.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=2a00:1450:4864:20::32a; envelope-from=peter.maydell@linaro.org; helo=mail-wm1-x32a.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @linaro.org) X-ZM-MESSAGEID: 1756381222341124100 We last synced our copy of kerneldoc with Linux back in 2020. In the interim, upstream has entirely rewritten the script in Python, and the new Python version is split into a main script plus some libraries in the kernel's scripts/lib/kdoc. Import all these files. These are the versions as of kernel commit 0cc53520e68be, with no local changes. We use the same lib/kdoc/ directory as the kernel does here, so we can avoid having to edit the top-level script just to adjust a pathname, even though it is probably not the naming we would have picked if this was a purely QEMU script. The Sphinx conf.py still points at the Perl version of the script, so this Python code will not be invoked to build the docs yet. NB: checkpatch complains about many things in this commit, including the use of "GPL-2.0" rather than "GPL-2.0-only" in the SPDX tags, but since this is a third party import we can ignore this. Signed-off-by: Peter Maydell Reviewed-by: Paolo Bonzini Reviewed-by: Mauro Carvalho Chehab Message-id: 20250814171324.1614516-4-peter.maydell@linaro.org --- scripts/kernel-doc.py | 325 ++++++ scripts/lib/kdoc/kdoc_files.py | 291 ++++++ scripts/lib/kdoc/kdoc_item.py | 42 + scripts/lib/kdoc/kdoc_output.py | 749 ++++++++++++++ scripts/lib/kdoc/kdoc_parser.py | 1669 +++++++++++++++++++++++++++++++ scripts/lib/kdoc/kdoc_re.py | 270 +++++ 6 files changed, 3346 insertions(+) create mode 100755 scripts/kernel-doc.py create mode 100644 scripts/lib/kdoc/kdoc_files.py create mode 100644 scripts/lib/kdoc/kdoc_item.py create mode 100644 scripts/lib/kdoc/kdoc_output.py create mode 100644 scripts/lib/kdoc/kdoc_parser.py create mode 100644 scripts/lib/kdoc/kdoc_re.py diff --git a/scripts/kernel-doc.py b/scripts/kernel-doc.py new file mode 100755 index 00000000000..fc3d46ef519 --- /dev/null +++ b/scripts/kernel-doc.py @@ -0,0 +1,325 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# Copyright(c) 2025: Mauro Carvalho Chehab . +# +# pylint: disable=3DC0103,R0915 +# +# Converted from the kernel-doc script originally written in Perl +# under GPLv2, copyrighted since 1998 by the following authors: +# +# Aditya Srivastava +# Akira Yokosawa +# Alexander A. Klimov +# Alexander Lobakin +# Andr=C3=A9 Almeida +# Andy Shevchenko +# Anna-Maria Behnsen +# Armin Kuster +# Bart Van Assche +# Ben Hutchings +# Borislav Petkov +# Chen-Yu Tsai +# Coco Li +# Conch=C3=BAr Navid +# Daniel Santos +# Danilo Cesar Lemes de Paula +# Dan Luedtke +# Donald Hunter +# Gabriel Krisman Bertazi +# Greg Kroah-Hartman +# Harvey Harrison +# Horia Geanta +# Ilya Dryomov +# Jakub Kicinski +# Jani Nikula +# Jason Baron +# Jason Gunthorpe +# J=C3=A9r=C3=A9my Bobbio +# Johannes Berg +# Johannes Weiner +# Jonathan Cameron +# Jonathan Corbet +# Jonathan Neusch=C3=A4fer +# Kamil Rytarowski +# Kees Cook +# Laurent Pinchart +# Levin, Alexander (Sasha Levin) +# Linus Torvalds +# Lucas De Marchi +# Mark Rutland +# Markus Heiser +# Martin Waitz +# Masahiro Yamada +# Matthew Wilcox +# Mauro Carvalho Chehab +# Michal Wajdeczko +# Michael Zucchi +# Mike Rapoport +# Niklas S=C3=B6derlund +# Nishanth Menon +# Paolo Bonzini +# Pavan Kumar Linga +# Pavel Pisa +# Peter Maydell +# Pierre-Louis Bossart +# Randy Dunlap +# Richard Kennedy +# Rich Walker +# Rolf Eike Beer +# Sakari Ailus +# Silvio Fricke +# Simon Huggins +# Tim Waugh +# Tomasz Warnie=C5=82=C5=82o +# Utkarsh Tripathi +# valdis.kletnieks@vt.edu +# Vegard Nossum +# Will Deacon +# Yacine Belkadi +# Yujie Liu + +""" +kernel_doc +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Print formatted kernel documentation to stdout + +Read C language source or header FILEs, extract embedded +documentation comments, and print formatted documentation +to standard output. + +The documentation comments are identified by the "/**" +opening comment mark. + +See Documentation/doc-guide/kernel-doc.rst for the +documentation comment syntax. +""" + +import argparse +import logging +import os +import sys + +# Import Python modules + +LIB_DIR =3D "lib/kdoc" +SRC_DIR =3D os.path.dirname(os.path.realpath(__file__)) + +sys.path.insert(0, os.path.join(SRC_DIR, LIB_DIR)) + +from kdoc_files import KernelFiles # pylint: disable= =3DC0413 +from kdoc_output import RestFormat, ManFormat # pylint: disable= =3DC0413 + +DESC =3D """ +Read C language source or header FILEs, extract embedded documentation com= ments, +and print formatted documentation to standard output. + +The documentation comments are identified by the "/**" opening comment mar= k. + +See Documentation/doc-guide/kernel-doc.rst for the documentation comment s= yntax. +""" + +EXPORT_FILE_DESC =3D """ +Specify an additional FILE in which to look for EXPORT_SYMBOL information. + +May be used multiple times. +""" + +EXPORT_DESC =3D """ +Only output documentation for the symbols that have been +exported using EXPORT_SYMBOL() and related macros in any input +FILE or -export-file FILE. +""" + +INTERNAL_DESC =3D """ +Only output documentation for the symbols that have NOT been +exported using EXPORT_SYMBOL() and related macros in any input +FILE or -export-file FILE. +""" + +FUNCTION_DESC =3D """ +Only output documentation for the given function or DOC: section +title. All other functions and DOC: sections are ignored. + +May be used multiple times. +""" + +NOSYMBOL_DESC =3D """ +Exclude the specified symbol from the output documentation. + +May be used multiple times. +""" + +FILES_DESC =3D """ +Header and C source files to be parsed. +""" + +WARN_CONTENTS_BEFORE_SECTIONS_DESC =3D """ +Warns if there are contents before sections (deprecated). + +This option is kept just for backward-compatibility, but it does nothing, +neither here nor at the original Perl script. +""" + + +class MsgFormatter(logging.Formatter): + """Helper class to format warnings on a similar way to kernel-doc.pl""" + + def format(self, record): + record.levelname =3D record.levelname.capitalize() + return logging.Formatter.format(self, record) + +def main(): + """Main program""" + + parser =3D argparse.ArgumentParser(formatter_class=3Dargparse.RawTextH= elpFormatter, + description=3DDESC) + + # Normal arguments + + parser.add_argument("-v", "-verbose", "--verbose", action=3D"store_tru= e", + help=3D"Verbose output, more warnings and other in= formation.") + + parser.add_argument("-d", "-debug", "--debug", action=3D"store_true", + help=3D"Enable debug messages") + + parser.add_argument("-M", "-modulename", "--modulename", + default=3D"Kernel API", + help=3D"Allow setting a module name at the output.= ") + + parser.add_argument("-l", "-enable-lineno", "--enable_lineno", + action=3D"store_true", + help=3D"Enable line number output (only in ReST mo= de)") + + # Arguments to control the warning behavior + + parser.add_argument("-Wreturn", "--wreturn", action=3D"store_true", + help=3D"Warns about the lack of a return markup on= functions.") + + parser.add_argument("-Wshort-desc", "-Wshort-description", "--wshort-d= esc", + action=3D"store_true", + help=3D"Warns if initial short description is miss= ing") + + parser.add_argument("-Wcontents-before-sections", + "--wcontents-before-sections", action=3D"store_tru= e", + help=3DWARN_CONTENTS_BEFORE_SECTIONS_DESC) + + parser.add_argument("-Wall", "--wall", action=3D"store_true", + help=3D"Enable all types of warnings") + + parser.add_argument("-Werror", "--werror", action=3D"store_true", + help=3D"Treat warnings as errors.") + + parser.add_argument("-export-file", "--export-file", action=3D'append', + help=3DEXPORT_FILE_DESC) + + # Output format mutually-exclusive group + + out_group =3D parser.add_argument_group("Output format selection (mutu= ally exclusive)") + + out_fmt =3D out_group.add_mutually_exclusive_group() + + out_fmt.add_argument("-m", "-man", "--man", action=3D"store_true", + help=3D"Output troff manual page format.") + out_fmt.add_argument("-r", "-rst", "--rst", action=3D"store_true", + help=3D"Output reStructuredText format (default).= ") + out_fmt.add_argument("-N", "-none", "--none", action=3D"store_true", + help=3D"Do not output documentation, only warning= s.") + + # Output selection mutually-exclusive group + + sel_group =3D parser.add_argument_group("Output selection (mutually ex= clusive)") + sel_mut =3D sel_group.add_mutually_exclusive_group() + + sel_mut.add_argument("-e", "-export", "--export", action=3D'store_true= ', + help=3DEXPORT_DESC) + + sel_mut.add_argument("-i", "-internal", "--internal", action=3D'store_= true', + help=3DINTERNAL_DESC) + + sel_mut.add_argument("-s", "-function", "--symbol", action=3D'append', + help=3DFUNCTION_DESC) + + # Those are valid for all 3 types of filter + parser.add_argument("-n", "-nosymbol", "--nosymbol", action=3D'append', + help=3DNOSYMBOL_DESC) + + parser.add_argument("-D", "-no-doc-sections", "--no-doc-sections", + action=3D'store_true', help=3D"Don't outputt DOC s= ections") + + parser.add_argument("files", metavar=3D"FILE", + nargs=3D"+", help=3DFILES_DESC) + + args =3D parser.parse_args() + + if args.wall: + args.wreturn =3D True + args.wshort_desc =3D True + args.wcontents_before_sections =3D True + + logger =3D logging.getLogger() + + if not args.debug: + logger.setLevel(logging.INFO) + else: + logger.setLevel(logging.DEBUG) + + formatter =3D MsgFormatter('%(levelname)s: %(message)s') + + handler =3D logging.StreamHandler() + handler.setFormatter(formatter) + + logger.addHandler(handler) + + python_ver =3D sys.version_info[:2] + if python_ver < (3,6): + logger.warning("Python 3.6 or later is required by kernel-doc") + + # Return 0 here to avoid breaking compilation + sys.exit(0) + + if python_ver < (3,7): + logger.warning("Python 3.7 or later is required for correct result= s") + + if args.man: + out_style =3D ManFormat(modulename=3Dargs.modulename) + elif args.none: + out_style =3D None + else: + out_style =3D RestFormat() + + kfiles =3D KernelFiles(verbose=3Dargs.verbose, + out_style=3Dout_style, werror=3Dargs.werror, + wreturn=3Dargs.wreturn, wshort_desc=3Dargs.wshort= _desc, + wcontents_before_sections=3Dargs.wcontents_before= _sections) + + kfiles.parse(args.files, export_file=3Dargs.export_file) + + for t in kfiles.msg(enable_lineno=3Dargs.enable_lineno, export=3Dargs.= export, + internal=3Dargs.internal, symbol=3Dargs.symbol, + nosymbol=3Dargs.nosymbol, export_file=3Dargs.expor= t_file, + no_doc_sections=3Dargs.no_doc_sections): + msg =3D t[1] + if msg: + print(msg) + + error_count =3D kfiles.errors + if not error_count: + sys.exit(0) + + if args.werror: + print(f"{error_count} warnings as errors") + sys.exit(error_count) + + if args.verbose: + print(f"{error_count} errors") + + if args.none: + sys.exit(0) + + sys.exit(error_count) + + +# Call main method +if __name__ =3D=3D "__main__": + main() diff --git a/scripts/lib/kdoc/kdoc_files.py b/scripts/lib/kdoc/kdoc_files.py new file mode 100644 index 00000000000..9e09b45b02f --- /dev/null +++ b/scripts/lib/kdoc/kdoc_files.py @@ -0,0 +1,291 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# Copyright(c) 2025: Mauro Carvalho Chehab . +# +# pylint: disable=3DR0903,R0913,R0914,R0917 + +""" +Parse lernel-doc tags on multiple kernel source files. +""" + +import argparse +import logging +import os +import re + +from kdoc_parser import KernelDoc +from kdoc_output import OutputFormat + + +class GlobSourceFiles: + """ + Parse C source code file names and directories via an Interactor. + """ + + def __init__(self, srctree=3DNone, valid_extensions=3DNone): + """ + Initialize valid extensions with a tuple. + + If not defined, assume default C extensions (.c and .h) + + It would be possible to use python's glob function, but it is + very slow, and it is not interactive. So, it would wait to read all + directories before actually do something. + + So, let's use our own implementation. + """ + + if not valid_extensions: + self.extensions =3D (".c", ".h") + else: + self.extensions =3D valid_extensions + + self.srctree =3D srctree + + def _parse_dir(self, dirname): + """Internal function to parse files recursively""" + + with os.scandir(dirname) as obj: + for entry in obj: + name =3D os.path.join(dirname, entry.name) + + if entry.is_dir(): + yield from self._parse_dir(name) + + if not entry.is_file(): + continue + + basename =3D os.path.basename(name) + + if not basename.endswith(self.extensions): + continue + + yield name + + def parse_files(self, file_list, file_not_found_cb): + """ + Define an interator to parse all source files from file_list, + handling directories if any + """ + + if not file_list: + return + + for fname in file_list: + if self.srctree: + f =3D os.path.join(self.srctree, fname) + else: + f =3D fname + + if os.path.isdir(f): + yield from self._parse_dir(f) + elif os.path.isfile(f): + yield f + elif file_not_found_cb: + file_not_found_cb(fname) + + +class KernelFiles(): + """ + Parse kernel-doc tags on multiple kernel source files. + + There are two type of parsers defined here: + - self.parse_file(): parses both kernel-doc markups and + EXPORT_SYMBOL* macros; + - self.process_export_file(): parses only EXPORT_SYMBOL* macros. + """ + + def warning(self, msg): + """Ancillary routine to output a warning and increment error count= """ + + self.config.log.warning(msg) + self.errors +=3D 1 + + def error(self, msg): + """Ancillary routine to output an error and increment error count"= "" + + self.config.log.error(msg) + self.errors +=3D 1 + + def parse_file(self, fname): + """ + Parse a single Kernel source. + """ + + # Prevent parsing the same file twice if results are cached + if fname in self.files: + return + + doc =3D KernelDoc(self.config, fname) + export_table, entries =3D doc.parse_kdoc() + + self.export_table[fname] =3D export_table + + self.files.add(fname) + self.export_files.add(fname) # parse_kdoc() already check exp= orts + + self.results[fname] =3D entries + + def process_export_file(self, fname): + """ + Parses EXPORT_SYMBOL* macros from a single Kernel source file. + """ + + # Prevent parsing the same file twice if results are cached + if fname in self.export_files: + return + + doc =3D KernelDoc(self.config, fname) + export_table =3D doc.parse_export() + + if not export_table: + self.error(f"Error: Cannot check EXPORT_SYMBOL* on {fname}") + export_table =3D set() + + self.export_table[fname] =3D export_table + self.export_files.add(fname) + + def file_not_found_cb(self, fname): + """ + Callback to warn if a file was not found. + """ + + self.error(f"Cannot find file {fname}") + + def __init__(self, verbose=3DFalse, out_style=3DNone, + werror=3DFalse, wreturn=3DFalse, wshort_desc=3DFalse, + wcontents_before_sections=3DFalse, + logger=3DNone): + """ + Initialize startup variables and parse all files + """ + + if not verbose: + verbose =3D bool(os.environ.get("KBUILD_VERBOSE", 0)) + + if out_style is None: + out_style =3D OutputFormat() + + if not werror: + kcflags =3D os.environ.get("KCFLAGS", None) + if kcflags: + match =3D re.search(r"(\s|^)-Werror(\s|$)/", kcflags) + if match: + werror =3D True + + # reading this variable is for backwards compat just in case + # someone was calling it with the variable from outside the + # kernel's build system + kdoc_werror =3D os.environ.get("KDOC_WERROR", None) + if kdoc_werror: + werror =3D kdoc_werror + + # Some variables are global to the parser logic as a whole as they= are + # used to send control configuration to KernelDoc class. As such, + # those variables are read-only inside the KernelDoc. + self.config =3D argparse.Namespace + + self.config.verbose =3D verbose + self.config.werror =3D werror + self.config.wreturn =3D wreturn + self.config.wshort_desc =3D wshort_desc + self.config.wcontents_before_sections =3D wcontents_before_sections + + if not logger: + self.config.log =3D logging.getLogger("kernel-doc") + else: + self.config.log =3D logger + + self.config.warning =3D self.warning + + self.config.src_tree =3D os.environ.get("SRCTREE", None) + + # Initialize variables that are internal to KernelFiles + + self.out_style =3D out_style + + self.errors =3D 0 + self.results =3D {} + + self.files =3D set() + self.export_files =3D set() + self.export_table =3D {} + + def parse(self, file_list, export_file=3DNone): + """ + Parse all files + """ + + glob =3D GlobSourceFiles(srctree=3Dself.config.src_tree) + + for fname in glob.parse_files(file_list, self.file_not_found_cb): + self.parse_file(fname) + + for fname in glob.parse_files(export_file, self.file_not_found_cb): + self.process_export_file(fname) + + def out_msg(self, fname, name, arg): + """ + Return output messages from a file name using the output style + filtering. + + If output type was not handled by the syler, return None. + """ + + # NOTE: we can add rules here to filter out unwanted parts, + # although OutputFormat.msg already does that. + + return self.out_style.msg(fname, name, arg) + + def msg(self, enable_lineno=3DFalse, export=3DFalse, internal=3DFalse, + symbol=3DNone, nosymbol=3DNone, no_doc_sections=3DFalse, + filenames=3DNone, export_file=3DNone): + """ + Interacts over the kernel-doc results and output messages, + returning kernel-doc markups on each interaction + """ + + self.out_style.set_config(self.config) + + if not filenames: + filenames =3D sorted(self.results.keys()) + + glob =3D GlobSourceFiles(srctree=3Dself.config.src_tree) + + for fname in filenames: + function_table =3D set() + + if internal or export: + if not export_file: + export_file =3D [fname] + + for f in glob.parse_files(export_file, self.file_not_found= _cb): + function_table |=3D self.export_table[f] + + if symbol: + for s in symbol: + function_table.add(s) + + self.out_style.set_filter(export, internal, symbol, nosymbol, + function_table, enable_lineno, + no_doc_sections) + + msg =3D "" + if fname not in self.results: + self.config.log.warning("No kernel-doc for file %s", fname) + continue + + for arg in self.results[fname]: + m =3D self.out_msg(fname, arg.name, arg) + + if m is None: + ln =3D arg.get("ln", 0) + dtype =3D arg.get('type', "") + + self.config.log.warning("%s:%d Can't handle %s", + fname, ln, dtype) + else: + msg +=3D m + + if msg: + yield fname, msg diff --git a/scripts/lib/kdoc/kdoc_item.py b/scripts/lib/kdoc/kdoc_item.py new file mode 100644 index 00000000000..b3b22576455 --- /dev/null +++ b/scripts/lib/kdoc/kdoc_item.py @@ -0,0 +1,42 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# A class that will, eventually, encapsulate all of the parsed data that we +# then pass into the output modules. +# + +class KdocItem: + def __init__(self, name, type, start_line, **other_stuff): + self.name =3D name + self.type =3D type + self.declaration_start_line =3D start_line + self.sections =3D {} + self.sections_start_lines =3D {} + self.parameterlist =3D [] + self.parameterdesc_start_lines =3D [] + self.parameterdescs =3D {} + self.parametertypes =3D {} + # + # Just save everything else into our own dict so that the output + # side can grab it directly as before. As we move things into more + # structured data, this will, hopefully, fade away. + # + self.other_stuff =3D other_stuff + + def get(self, key, default =3D None): + return self.other_stuff.get(key, default) + + def __getitem__(self, key): + return self.get(key) + + # + # Tracking of section and parameter information. + # + def set_sections(self, sections, start_lines): + self.sections =3D sections + self.section_start_lines =3D start_lines + + def set_params(self, names, descs, types, starts): + self.parameterlist =3D names + self.parameterdescs =3D descs + self.parametertypes =3D types + self.parameterdesc_start_lines =3D starts diff --git a/scripts/lib/kdoc/kdoc_output.py b/scripts/lib/kdoc/kdoc_output= .py new file mode 100644 index 00000000000..ea8914537ba --- /dev/null +++ b/scripts/lib/kdoc/kdoc_output.py @@ -0,0 +1,749 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# Copyright(c) 2025: Mauro Carvalho Chehab . +# +# pylint: disable=3DC0301,R0902,R0911,R0912,R0913,R0914,R0915,R0917 + +""" +Implement output filters to print kernel-doc documentation. + +The implementation uses a virtual base class (OutputFormat) which +contains a dispatches to virtual methods, and some code to filter +out output messages. + +The actual implementation is done on one separate class per each type +of output. Currently, there are output classes for ReST and man/troff. +""" + +import os +import re +from datetime import datetime + +from kdoc_parser import KernelDoc, type_param +from kdoc_re import KernRe + + +function_pointer =3D KernRe(r"([^\(]*\(\*)\s*\)\s*\(([^\)]*)\)", cache=3DF= alse) + +# match expressions used to find embedded type information +type_constant =3D KernRe(r"\b``([^\`]+)``\b", cache=3DFalse) +type_constant2 =3D KernRe(r"\%([-_*\w]+)", cache=3DFalse) +type_func =3D KernRe(r"(\w+)\(\)", cache=3DFalse) +type_param_ref =3D KernRe(r"([\!~\*]?)\@(\w*((\.\w+)|(->\w+))*(\.\.\.)?)",= cache=3DFalse) + +# Special RST handling for func ptr params +type_fp_param =3D KernRe(r"\@(\w+)\(\)", cache=3DFalse) + +# Special RST handling for structs with func ptr params +type_fp_param2 =3D KernRe(r"\@(\w+->\S+)\(\)", cache=3DFalse) + +type_env =3D KernRe(r"(\$\w+)", cache=3DFalse) +type_enum =3D KernRe(r"\&(enum\s*([_\w]+))", cache=3DFalse) +type_struct =3D KernRe(r"\&(struct\s*([_\w]+))", cache=3DFalse) +type_typedef =3D KernRe(r"\&(typedef\s*([_\w]+))", cache=3DFalse) +type_union =3D KernRe(r"\&(union\s*([_\w]+))", cache=3DFalse) +type_member =3D KernRe(r"\&([_\w]+)(\.|->)([_\w]+)", cache=3DFalse) +type_fallback =3D KernRe(r"\&([_\w]+)", cache=3DFalse) +type_member_func =3D type_member + KernRe(r"\(\)", cache=3DFalse) + + +class OutputFormat: + """ + Base class for OutputFormat. If used as-is, it means that only + warnings will be displayed. + """ + + # output mode. + OUTPUT_ALL =3D 0 # output all symbols and doc sections + OUTPUT_INCLUDE =3D 1 # output only specified symbols + OUTPUT_EXPORTED =3D 2 # output exported symbols + OUTPUT_INTERNAL =3D 3 # output non-exported symbols + + # Virtual member to be overriden at the inherited classes + highlights =3D [] + + def __init__(self): + """Declare internal vars and set mode to OUTPUT_ALL""" + + self.out_mode =3D self.OUTPUT_ALL + self.enable_lineno =3D None + self.nosymbol =3D {} + self.symbol =3D None + self.function_table =3D None + self.config =3D None + self.no_doc_sections =3D False + + self.data =3D "" + + def set_config(self, config): + """ + Setup global config variables used by both parser and output. + """ + + self.config =3D config + + def set_filter(self, export, internal, symbol, nosymbol, function_tabl= e, + enable_lineno, no_doc_sections): + """ + Initialize filter variables according with the requested mode. + + Only one choice is valid between export, internal and symbol. + + The nosymbol filter can be used on all modes. + """ + + self.enable_lineno =3D enable_lineno + self.no_doc_sections =3D no_doc_sections + self.function_table =3D function_table + + if symbol: + self.out_mode =3D self.OUTPUT_INCLUDE + elif export: + self.out_mode =3D self.OUTPUT_EXPORTED + elif internal: + self.out_mode =3D self.OUTPUT_INTERNAL + else: + self.out_mode =3D self.OUTPUT_ALL + + if nosymbol: + self.nosymbol =3D set(nosymbol) + + + def highlight_block(self, block): + """ + Apply the RST highlights to a sub-block of text. + """ + + for r, sub in self.highlights: + block =3D r.sub(sub, block) + + return block + + def out_warnings(self, args): + """ + Output warnings for identifiers that will be displayed. + """ + + for log_msg in args.warnings: + self.config.warning(log_msg) + + def check_doc(self, name, args): + """Check if DOC should be output""" + + if self.no_doc_sections: + return False + + if name in self.nosymbol: + return False + + if self.out_mode =3D=3D self.OUTPUT_ALL: + self.out_warnings(args) + return True + + if self.out_mode =3D=3D self.OUTPUT_INCLUDE: + if name in self.function_table: + self.out_warnings(args) + return True + + return False + + def check_declaration(self, dtype, name, args): + """ + Checks if a declaration should be output or not based on the + filtering criteria. + """ + + if name in self.nosymbol: + return False + + if self.out_mode =3D=3D self.OUTPUT_ALL: + self.out_warnings(args) + return True + + if self.out_mode in [self.OUTPUT_INCLUDE, self.OUTPUT_EXPORTED]: + if name in self.function_table: + return True + + if self.out_mode =3D=3D self.OUTPUT_INTERNAL: + if dtype !=3D "function": + self.out_warnings(args) + return True + + if name not in self.function_table: + self.out_warnings(args) + return True + + return False + + def msg(self, fname, name, args): + """ + Handles a single entry from kernel-doc parser + """ + + self.data =3D "" + + dtype =3D args.type + + if dtype =3D=3D "doc": + self.out_doc(fname, name, args) + return self.data + + if not self.check_declaration(dtype, name, args): + return self.data + + if dtype =3D=3D "function": + self.out_function(fname, name, args) + return self.data + + if dtype =3D=3D "enum": + self.out_enum(fname, name, args) + return self.data + + if dtype =3D=3D "typedef": + self.out_typedef(fname, name, args) + return self.data + + if dtype in ["struct", "union"]: + self.out_struct(fname, name, args) + return self.data + + # Warn if some type requires an output logic + self.config.log.warning("doesn't now how to output '%s' block", + dtype) + + return None + + # Virtual methods to be overridden by inherited classes + # At the base class, those do nothing. + def out_doc(self, fname, name, args): + """Outputs a DOC block""" + + def out_function(self, fname, name, args): + """Outputs a function""" + + def out_enum(self, fname, name, args): + """Outputs an enum""" + + def out_typedef(self, fname, name, args): + """Outputs a typedef""" + + def out_struct(self, fname, name, args): + """Outputs a struct""" + + +class RestFormat(OutputFormat): + """Consts and functions used by ReST output""" + + highlights =3D [ + (type_constant, r"``\1``"), + (type_constant2, r"``\1``"), + + # Note: need to escape () to avoid func matching later + (type_member_func, r":c:type:`\1\2\3\\(\\) <\1>`"), + (type_member, r":c:type:`\1\2\3 <\1>`"), + (type_fp_param, r"**\1\\(\\)**"), + (type_fp_param2, r"**\1\\(\\)**"), + (type_func, r"\1()"), + (type_enum, r":c:type:`\1 <\2>`"), + (type_struct, r":c:type:`\1 <\2>`"), + (type_typedef, r":c:type:`\1 <\2>`"), + (type_union, r":c:type:`\1 <\2>`"), + + # in rst this can refer to any type + (type_fallback, r":c:type:`\1`"), + (type_param_ref, r"**\1\2**") + ] + blankline =3D "\n" + + sphinx_literal =3D KernRe(r'^[^.].*::$', cache=3DFalse) + sphinx_cblock =3D KernRe(r'^\.\.\ +code-block::', cache=3DFalse) + + def __init__(self): + """ + Creates class variables. + + Not really mandatory, but it is a good coding style and makes + pylint happy. + """ + + super().__init__() + self.lineprefix =3D "" + + def print_lineno(self, ln): + """Outputs a line number""" + + if self.enable_lineno and ln is not None: + ln +=3D 1 + self.data +=3D f".. LINENO {ln}\n" + + def output_highlight(self, args): + """ + Outputs a C symbol that may require being converted to ReST using + the self.highlights variable + """ + + input_text =3D args + output =3D "" + in_literal =3D False + litprefix =3D "" + block =3D "" + + for line in input_text.strip("\n").split("\n"): + + # If we're in a literal block, see if we should drop out of it. + # Otherwise, pass the line straight through unmunged. + if in_literal: + if line.strip(): # If the line is not blank + # If this is the first non-blank line in a literal blo= ck, + # figure out the proper indent. + if not litprefix: + r =3D KernRe(r'^(\s*)') + if r.match(line): + litprefix =3D '^' + r.group(1) + else: + litprefix =3D "" + + output +=3D line + "\n" + elif not KernRe(litprefix).match(line): + in_literal =3D False + else: + output +=3D line + "\n" + else: + output +=3D line + "\n" + + # Not in a literal block (or just dropped out) + if not in_literal: + block +=3D line + "\n" + if self.sphinx_literal.match(line) or self.sphinx_cblock.m= atch(line): + in_literal =3D True + litprefix =3D "" + output +=3D self.highlight_block(block) + block =3D "" + + # Handle any remaining block + if block: + output +=3D self.highlight_block(block) + + # Print the output with the line prefix + for line in output.strip("\n").split("\n"): + self.data +=3D self.lineprefix + line + "\n" + + def out_section(self, args, out_docblock=3DFalse): + """ + Outputs a block section. + + This could use some work; it's used to output the DOC: sections, a= nd + starts by putting out the name of the doc section itself, but that + tends to duplicate a header already in the template file. + """ + for section, text in args.sections.items(): + # Skip sections that are in the nosymbol_table + if section in self.nosymbol: + continue + + if out_docblock: + if not self.out_mode =3D=3D self.OUTPUT_INCLUDE: + self.data +=3D f".. _{section}:\n\n" + self.data +=3D f'{self.lineprefix}**{section}**\n\n' + else: + self.data +=3D f'{self.lineprefix}**{section}**\n\n' + + self.print_lineno(args.section_start_lines.get(section, 0)) + self.output_highlight(text) + self.data +=3D "\n" + self.data +=3D "\n" + + def out_doc(self, fname, name, args): + if not self.check_doc(name, args): + return + self.out_section(args, out_docblock=3DTrue) + + def out_function(self, fname, name, args): + + oldprefix =3D self.lineprefix + signature =3D "" + + func_macro =3D args.get('func_macro', False) + if func_macro: + signature =3D name + else: + if args.get('functiontype'): + signature =3D args['functiontype'] + " " + signature +=3D name + " (" + + ln =3D args.declaration_start_line + count =3D 0 + for parameter in args.parameterlist: + if count !=3D 0: + signature +=3D ", " + count +=3D 1 + dtype =3D args.parametertypes.get(parameter, "") + + if function_pointer.search(dtype): + signature +=3D function_pointer.group(1) + parameter + fun= ction_pointer.group(3) + else: + signature +=3D dtype + + if not func_macro: + signature +=3D ")" + + self.print_lineno(ln) + if args.get('typedef') or not args.get('functiontype'): + self.data +=3D f".. c:macro:: {name}\n\n" + + if args.get('typedef'): + self.data +=3D " **Typedef**: " + self.lineprefix =3D "" + self.output_highlight(args.get('purpose', "")) + self.data +=3D "\n\n**Syntax**\n\n" + self.data +=3D f" ``{signature}``\n\n" + else: + self.data +=3D f"``{signature}``\n\n" + else: + self.data +=3D f".. c:function:: {signature}\n\n" + + if not args.get('typedef'): + self.print_lineno(ln) + self.lineprefix =3D " " + self.output_highlight(args.get('purpose', "")) + self.data +=3D "\n" + + # Put descriptive text into a container (HTML
) to help set + # function prototypes apart + self.lineprefix =3D " " + + if args.parameterlist: + self.data +=3D ".. container:: kernelindent\n\n" + self.data +=3D f"{self.lineprefix}**Parameters**\n\n" + + for parameter in args.parameterlist: + parameter_name =3D KernRe(r'\[.*').sub('', parameter) + dtype =3D args.parametertypes.get(parameter, "") + + if dtype: + self.data +=3D f"{self.lineprefix}``{dtype}``\n" + else: + self.data +=3D f"{self.lineprefix}``{parameter}``\n" + + self.print_lineno(args.parameterdesc_start_lines.get(parameter= _name, 0)) + + self.lineprefix =3D " " + if parameter_name in args.parameterdescs and \ + args.parameterdescs[parameter_name] !=3D KernelDoc.undescri= bed: + + self.output_highlight(args.parameterdescs[parameter_name]) + self.data +=3D "\n" + else: + self.data +=3D f"{self.lineprefix}*undescribed*\n\n" + self.lineprefix =3D " " + + self.out_section(args) + self.lineprefix =3D oldprefix + + def out_enum(self, fname, name, args): + + oldprefix =3D self.lineprefix + ln =3D args.declaration_start_line + + self.data +=3D f"\n\n.. c:enum:: {name}\n\n" + + self.print_lineno(ln) + self.lineprefix =3D " " + self.output_highlight(args.get('purpose', '')) + self.data +=3D "\n" + + self.data +=3D ".. container:: kernelindent\n\n" + outer =3D self.lineprefix + " " + self.lineprefix =3D outer + " " + self.data +=3D f"{outer}**Constants**\n\n" + + for parameter in args.parameterlist: + self.data +=3D f"{outer}``{parameter}``\n" + + if args.parameterdescs.get(parameter, '') !=3D KernelDoc.undes= cribed: + self.output_highlight(args.parameterdescs[parameter]) + else: + self.data +=3D f"{self.lineprefix}*undescribed*\n\n" + self.data +=3D "\n" + + self.lineprefix =3D oldprefix + self.out_section(args) + + def out_typedef(self, fname, name, args): + + oldprefix =3D self.lineprefix + ln =3D args.declaration_start_line + + self.data +=3D f"\n\n.. c:type:: {name}\n\n" + + self.print_lineno(ln) + self.lineprefix =3D " " + + self.output_highlight(args.get('purpose', '')) + + self.data +=3D "\n" + + self.lineprefix =3D oldprefix + self.out_section(args) + + def out_struct(self, fname, name, args): + + purpose =3D args.get('purpose', "") + declaration =3D args.get('definition', "") + dtype =3D args.type + ln =3D args.declaration_start_line + + self.data +=3D f"\n\n.. c:{dtype}:: {name}\n\n" + + self.print_lineno(ln) + + oldprefix =3D self.lineprefix + self.lineprefix +=3D " " + + self.output_highlight(purpose) + self.data +=3D "\n" + + self.data +=3D ".. container:: kernelindent\n\n" + self.data +=3D f"{self.lineprefix}**Definition**::\n\n" + + self.lineprefix =3D self.lineprefix + " " + + declaration =3D declaration.replace("\t", self.lineprefix) + + self.data +=3D f"{self.lineprefix}{dtype} {name}" + ' {' + "\n" + self.data +=3D f"{declaration}{self.lineprefix}" + "};\n\n" + + self.lineprefix =3D " " + self.data +=3D f"{self.lineprefix}**Members**\n\n" + for parameter in args.parameterlist: + if not parameter or parameter.startswith("#"): + continue + + parameter_name =3D parameter.split("[", maxsplit=3D1)[0] + + if args.parameterdescs.get(parameter_name) =3D=3D KernelDoc.un= described: + continue + + self.print_lineno(args.parameterdesc_start_lines.get(parameter= _name, 0)) + + self.data +=3D f"{self.lineprefix}``{parameter}``\n" + + self.lineprefix =3D " " + self.output_highlight(args.parameterdescs[parameter_name]) + self.lineprefix =3D " " + + self.data +=3D "\n" + + self.data +=3D "\n" + + self.lineprefix =3D oldprefix + self.out_section(args) + + +class ManFormat(OutputFormat): + """Consts and functions used by man pages output""" + + highlights =3D ( + (type_constant, r"\1"), + (type_constant2, r"\1"), + (type_func, r"\\fB\1\\fP"), + (type_enum, r"\\fI\1\\fP"), + (type_struct, r"\\fI\1\\fP"), + (type_typedef, r"\\fI\1\\fP"), + (type_union, r"\\fI\1\\fP"), + (type_param, r"\\fI\1\\fP"), + (type_param_ref, r"\\fI\1\2\\fP"), + (type_member, r"\\fI\1\2\3\\fP"), + (type_fallback, r"\\fI\1\\fP") + ) + blankline =3D "" + + date_formats =3D [ + "%a %b %d %H:%M:%S %Z %Y", + "%a %b %d %H:%M:%S %Y", + "%Y-%m-%d", + "%b %d %Y", + "%B %d %Y", + "%m %d %Y", + ] + + def __init__(self, modulename): + """ + Creates class variables. + + Not really mandatory, but it is a good coding style and makes + pylint happy. + """ + + super().__init__() + self.modulename =3D modulename + + dt =3D None + tstamp =3D os.environ.get("KBUILD_BUILD_TIMESTAMP") + if tstamp: + for fmt in self.date_formats: + try: + dt =3D datetime.strptime(tstamp, fmt) + break + except ValueError: + pass + + if not dt: + dt =3D datetime.now() + + self.man_date =3D dt.strftime("%B %Y") + + def output_highlight(self, block): + """ + Outputs a C symbol that may require being highlighted with + self.highlights variable using troff syntax + """ + + contents =3D self.highlight_block(block) + + if isinstance(contents, list): + contents =3D "\n".join(contents) + + for line in contents.strip("\n").split("\n"): + line =3D KernRe(r"^\s*").sub("", line) + if not line: + continue + + if line[0] =3D=3D ".": + self.data +=3D "\\&" + line + "\n" + else: + self.data +=3D line + "\n" + + def out_doc(self, fname, name, args): + if not self.check_doc(name, args): + return + + self.data +=3D f'.TH "{self.modulename}" 9 "{self.modulename}" "{s= elf.man_date}" "API Manual" LINUX' + "\n" + + for section, text in args.sections.items(): + self.data +=3D f'.SH "{section}"' + "\n" + self.output_highlight(text) + + def out_function(self, fname, name, args): + """output function in man""" + + self.data +=3D f'.TH "{name}" 9 "{name}" "{self.man_date}" "Kernel= Hacker\'s Manual" LINUX' + "\n" + + self.data +=3D ".SH NAME\n" + self.data +=3D f"{name} \\- {args['purpose']}\n" + + self.data +=3D ".SH SYNOPSIS\n" + if args.get('functiontype', ''): + self.data +=3D f'.B "{args["functiontype"]}" {name}' + "\n" + else: + self.data +=3D f'.B "{name}' + "\n" + + count =3D 0 + parenth =3D "(" + post =3D "," + + for parameter in args.parameterlist: + if count =3D=3D len(args.parameterlist) - 1: + post =3D ");" + + dtype =3D args.parametertypes.get(parameter, "") + if function_pointer.match(dtype): + # Pointer-to-function + self.data +=3D f'".BI "{parenth}{function_pointer.group(1)= }" " ") ({function_pointer.group(2)}){post}"' + "\n" + else: + dtype =3D KernRe(r'([^\*])$').sub(r'\1 ', dtype) + + self.data +=3D f'.BI "{parenth}{dtype}" "{post}"' + "\n" + count +=3D 1 + parenth =3D "" + + if args.parameterlist: + self.data +=3D ".SH ARGUMENTS\n" + + for parameter in args.parameterlist: + parameter_name =3D re.sub(r'\[.*', '', parameter) + + self.data +=3D f'.IP "{parameter}" 12' + "\n" + self.output_highlight(args.parameterdescs.get(parameter_name, = "")) + + for section, text in args.sections.items(): + self.data +=3D f'.SH "{section.upper()}"' + "\n" + self.output_highlight(text) + + def out_enum(self, fname, name, args): + self.data +=3D f'.TH "{self.modulename}" 9 "enum {name}" "{self.ma= n_date}" "API Manual" LINUX' + "\n" + + self.data +=3D ".SH NAME\n" + self.data +=3D f"enum {name} \\- {args['purpose']}\n" + + self.data +=3D ".SH SYNOPSIS\n" + self.data +=3D f"enum {name}" + " {\n" + + count =3D 0 + for parameter in args.parameterlist: + self.data +=3D f'.br\n.BI " {parameter}"' + "\n" + if count =3D=3D len(args.parameterlist) - 1: + self.data +=3D "\n};\n" + else: + self.data +=3D ", \n.br\n" + + count +=3D 1 + + self.data +=3D ".SH Constants\n" + + for parameter in args.parameterlist: + parameter_name =3D KernRe(r'\[.*').sub('', parameter) + self.data +=3D f'.IP "{parameter}" 12' + "\n" + self.output_highlight(args.parameterdescs.get(parameter_name, = "")) + + for section, text in args.sections.items(): + self.data +=3D f'.SH "{section}"' + "\n" + self.output_highlight(text) + + def out_typedef(self, fname, name, args): + module =3D self.modulename + purpose =3D args.get('purpose') + + self.data +=3D f'.TH "{module}" 9 "{name}" "{self.man_date}" "API = Manual" LINUX' + "\n" + + self.data +=3D ".SH NAME\n" + self.data +=3D f"typedef {name} \\- {purpose}\n" + + for section, text in args.sections.items(): + self.data +=3D f'.SH "{section}"' + "\n" + self.output_highlight(text) + + def out_struct(self, fname, name, args): + module =3D self.modulename + purpose =3D args.get('purpose') + definition =3D args.get('definition') + + self.data +=3D f'.TH "{module}" 9 "{args.type} {name}" "{self.man_= date}" "API Manual" LINUX' + "\n" + + self.data +=3D ".SH NAME\n" + self.data +=3D f"{args.type} {name} \\- {purpose}\n" + + # Replace tabs with two spaces and handle newlines + declaration =3D definition.replace("\t", " ") + declaration =3D KernRe(r"\n").sub('"\n.br\n.BI "', declaration) + + self.data +=3D ".SH SYNOPSIS\n" + self.data +=3D f"{args.type} {name} " + "{" + "\n.br\n" + self.data +=3D f'.BI "{declaration}\n' + "};\n.br\n\n" + + self.data +=3D ".SH Members\n" + for parameter in args.parameterlist: + if parameter.startswith("#"): + continue + + parameter_name =3D re.sub(r"\[.*", "", parameter) + + if args.parameterdescs.get(parameter_name) =3D=3D KernelDoc.un= described: + continue + + self.data +=3D f'.IP "{parameter}" 12' + "\n" + self.output_highlight(args.parameterdescs.get(parameter_name)) + + for section, text in args.sections.items(): + self.data +=3D f'.SH "{section}"' + "\n" + self.output_highlight(text) diff --git a/scripts/lib/kdoc/kdoc_parser.py b/scripts/lib/kdoc/kdoc_parser= .py new file mode 100644 index 00000000000..fe730099eca --- /dev/null +++ b/scripts/lib/kdoc/kdoc_parser.py @@ -0,0 +1,1669 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# Copyright(c) 2025: Mauro Carvalho Chehab . +# +# pylint: disable=3DC0301,C0302,R0904,R0912,R0913,R0914,R0915,R0917,R1702 + +""" +kdoc_parser +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + +Read a C language source or header FILE and extract embedded +documentation comments +""" + +import sys +import re +from pprint import pformat + +from kdoc_re import NestedMatch, KernRe +from kdoc_item import KdocItem + +# +# Regular expressions used to parse kernel-doc markups at KernelDoc class. +# +# Let's declare them in lowercase outside any class to make easier to +# convert from the python script. +# +# As those are evaluated at the beginning, no need to cache them +# + +# Allow whitespace at end of comment start. +doc_start =3D KernRe(r'^/\*\*\s*$', cache=3DFalse) + +doc_end =3D KernRe(r'\*/', cache=3DFalse) +doc_com =3D KernRe(r'\s*\*\s*', cache=3DFalse) +doc_com_body =3D KernRe(r'\s*\* ?', cache=3DFalse) +doc_decl =3D doc_com + KernRe(r'(\w+)', cache=3DFalse) + +# @params and a strictly limited set of supported section names +# Specifically: +# Match @word: +# @...: +# @{section-name}: +# while trying to not match literal block starts like "example::" +# +known_section_names =3D 'description|context|returns?|notes?|examples?' +known_sections =3D KernRe(known_section_names, flags =3D re.I) +doc_sect =3D doc_com + \ + KernRe(r'\s*(\@[.\w]+|\@\.\.\.|' + known_section_names + r')\s*:([^:].= *)?$', + flags=3Dre.I, cache=3DFalse) + +doc_content =3D doc_com_body + KernRe(r'(.*)', cache=3DFalse) +doc_inline_start =3D KernRe(r'^\s*/\*\*\s*$', cache=3DFalse) +doc_inline_sect =3D KernRe(r'\s*\*\s*(@\s*[\w][\w\.]*\s*):(.*)', cache=3DF= alse) +doc_inline_end =3D KernRe(r'^\s*\*/\s*$', cache=3DFalse) +doc_inline_oneline =3D KernRe(r'^\s*/\*\*\s*(@[\w\s]+):\s*(.*)\s*\*/\s*$',= cache=3DFalse) +attribute =3D KernRe(r"__attribute__\s*\(\([a-z0-9,_\*\s\(\)]*\)\)", + flags=3Dre.I | re.S, cache=3DFalse) + +export_symbol =3D KernRe(r'^\s*EXPORT_SYMBOL(_GPL)?\s*\(\s*(\w+)\s*\)\s*',= cache=3DFalse) +export_symbol_ns =3D KernRe(r'^\s*EXPORT_SYMBOL_NS(_GPL)?\s*\(\s*(\w+)\s*,= \s*"\S+"\)\s*', cache=3DFalse) + +type_param =3D KernRe(r"\@(\w*((\.\w+)|(->\w+))*(\.\.\.)?)", cache=3DFalse) + +# +# Tests for the beginning of a kerneldoc block in its various forms. +# +doc_block =3D doc_com + KernRe(r'DOC:\s*(.*)?', cache=3DFalse) +doc_begin_data =3D KernRe(r"^\s*\*?\s*(struct|union|enum|typedef)\b\s*(\w*= )", cache =3D False) +doc_begin_func =3D KernRe(str(doc_com) + # initial " * ' + r"(?:\w+\s*\*\s*)?" + # type (not captured) + r'(?:define\s+)?' + # possible "define" (not cap= tured) + r'(\w+)\s*(?:\(\w*\))?\s*' + # name and optional "= (...)" + r'(?:[-:].*)?$', # description (not captured) + cache =3D False) + +# +# A little helper to get rid of excess white space +# +multi_space =3D KernRe(r'\s\s+') +def trim_whitespace(s): + return multi_space.sub(' ', s.strip()) + +class state: + """ + State machine enums + """ + + # Parser states + NORMAL =3D 0 # normal code + NAME =3D 1 # looking for function name + DECLARATION =3D 2 # We have seen a declaration which might no= t be done + BODY =3D 3 # the body of the comment + SPECIAL_SECTION =3D 4 # doc section ending with a blank line + PROTO =3D 5 # scanning prototype + DOCBLOCK =3D 6 # documentation block + INLINE_NAME =3D 7 # gathering doc outside main block + INLINE_TEXT =3D 8 # reading the body of inline docs + + name =3D [ + "NORMAL", + "NAME", + "DECLARATION", + "BODY", + "SPECIAL_SECTION", + "PROTO", + "DOCBLOCK", + "INLINE_NAME", + "INLINE_TEXT", + ] + + +SECTION_DEFAULT =3D "Description" # default section + +class KernelEntry: + + def __init__(self, config, ln): + self.config =3D config + + self._contents =3D [] + self.prototype =3D "" + + self.warnings =3D [] + + self.parameterlist =3D [] + self.parameterdescs =3D {} + self.parametertypes =3D {} + self.parameterdesc_start_lines =3D {} + + self.section_start_lines =3D {} + self.sections =3D {} + + self.anon_struct_union =3D False + + self.leading_space =3D None + + # State flags + self.brcount =3D 0 + self.declaration_start_line =3D ln + 1 + + # + # Management of section contents + # + def add_text(self, text): + self._contents.append(text) + + def contents(self): + return '\n'.join(self._contents) + '\n' + + # TODO: rename to emit_message after removal of kernel-doc.pl + def emit_msg(self, log_msg, warning=3DTrue): + """Emit a message""" + + if not warning: + self.config.log.info(log_msg) + return + + # Delegate warning output to output logic, as this way it + # will report warnings/info only for symbols that are output + + self.warnings.append(log_msg) + return + + # + # Begin a new section. + # + def begin_section(self, line_no, title =3D SECTION_DEFAULT, dump =3D F= alse): + if dump: + self.dump_section(start_new =3D True) + self.section =3D title + self.new_start_line =3D line_no + + def dump_section(self, start_new=3DTrue): + """ + Dumps section contents to arrays/hashes intended for that purpose. + """ + # + # If we have accumulated no contents in the default ("description") + # section, don't bother. + # + if self.section =3D=3D SECTION_DEFAULT and not self._contents: + return + name =3D self.section + contents =3D self.contents() + + if type_param.match(name): + name =3D type_param.group(1) + + self.parameterdescs[name] =3D contents + self.parameterdesc_start_lines[name] =3D self.new_start_line + + self.new_start_line =3D 0 + + else: + if name in self.sections and self.sections[name] !=3D "": + # Only warn on user-specified duplicate section names + if name !=3D SECTION_DEFAULT: + self.emit_msg(self.new_start_line, + f"duplicate section name '{name}'\n") + # Treat as a new paragraph - add a blank line + self.sections[name] +=3D '\n' + contents + else: + self.sections[name] =3D contents + self.section_start_lines[name] =3D self.new_start_line + self.new_start_line =3D 0 + +# self.config.log.debug("Section: %s : %s", name, pformat(vars(self= ))) + + if start_new: + self.section =3D SECTION_DEFAULT + self._contents =3D [] + + +class KernelDoc: + """ + Read a C language source or header FILE and extract embedded + documentation comments. + """ + + # Section names + + section_context =3D "Context" + section_return =3D "Return" + + undescribed =3D "-- undescribed --" + + def __init__(self, config, fname): + """Initialize internal variables""" + + self.fname =3D fname + self.config =3D config + + # Initial state for the state machines + self.state =3D state.NORMAL + + # Store entry currently being processed + self.entry =3D None + + # Place all potential outputs into an array + self.entries =3D [] + + # + # We need Python 3.7 for its "dicts remember the insertion + # order" guarantee + # + if sys.version_info.major =3D=3D 3 and sys.version_info.minor < 7: + self.emit_msg(0, + 'Python 3.7 or later is required for correct res= ults') + + def emit_msg(self, ln, msg, warning=3DTrue): + """Emit a message""" + + log_msg =3D f"{self.fname}:{ln} {msg}" + + if self.entry: + self.entry.emit_msg(log_msg, warning) + return + + if warning: + self.config.log.warning(log_msg) + else: + self.config.log.info(log_msg) + + def dump_section(self, start_new=3DTrue): + """ + Dumps section contents to arrays/hashes intended for that purpose. + """ + + if self.entry: + self.entry.dump_section(start_new) + + # TODO: rename it to store_declaration after removal of kernel-doc.pl + def output_declaration(self, dtype, name, **args): + """ + Stores the entry into an entry array. + + The actual output and output filters will be handled elsewhere + """ + + item =3D KdocItem(name, dtype, self.entry.declaration_start_line, = **args) + item.warnings =3D self.entry.warnings + + # Drop empty sections + # TODO: improve empty sections logic to emit warnings + sections =3D self.entry.sections + for section in ["Description", "Return"]: + if section in sections and not sections[section].rstrip(): + del sections[section] + item.set_sections(sections, self.entry.section_start_lines) + item.set_params(self.entry.parameterlist, self.entry.parameterdesc= s, + self.entry.parametertypes, + self.entry.parameterdesc_start_lines) + self.entries.append(item) + + self.config.log.debug("Output: %s:%s =3D %s", dtype, name, pformat= (args)) + + def reset_state(self, ln): + """ + Ancillary routine to create a new entry. It initializes all + variables used by the state machine. + """ + + self.entry =3D KernelEntry(self.config, ln) + + # State flags + self.state =3D state.NORMAL + + def push_parameter(self, ln, decl_type, param, dtype, + org_arg, declaration_name): + """ + Store parameters and their descriptions at self.entry. + """ + + if self.entry.anon_struct_union and dtype =3D=3D "" and param =3D= =3D "}": + return # Ignore the ending }; from anonymous struct/union + + self.entry.anon_struct_union =3D False + + param =3D KernRe(r'[\[\)].*').sub('', param, count=3D1) + + if dtype =3D=3D "" and param.endswith("..."): + if KernRe(r'\w\.\.\.$').search(param): + # For named variable parameters of the form `x...`, + # remove the dots + param =3D param[:-3] + else: + # Handles unnamed variable parameters + param =3D "..." + + if param not in self.entry.parameterdescs or \ + not self.entry.parameterdescs[param]: + + self.entry.parameterdescs[param] =3D "variable arguments" + + elif dtype =3D=3D "" and (not param or param =3D=3D "void"): + param =3D "void" + self.entry.parameterdescs[param] =3D "no arguments" + + elif dtype =3D=3D "" and param in ["struct", "union"]: + # Handle unnamed (anonymous) union or struct + dtype =3D param + param =3D "{unnamed_" + param + "}" + self.entry.parameterdescs[param] =3D "anonymous\n" + self.entry.anon_struct_union =3D True + + # Handle cache group enforcing variables: they do not need + # to be described in header files + elif "__cacheline_group" in param: + # Ignore __cacheline_group_begin and __cacheline_group_end + return + + # Warn if parameter has no description + # (but ignore ones starting with # as these are not parameters + # but inline preprocessor statements) + if param not in self.entry.parameterdescs and not param.startswith= ("#"): + self.entry.parameterdescs[param] =3D self.undescribed + + if "." not in param: + if decl_type =3D=3D 'function': + dname =3D f"{decl_type} parameter" + else: + dname =3D f"{decl_type} member" + + self.emit_msg(ln, + f"{dname} '{param}' not described in '{decla= ration_name}'") + + # Strip spaces from param so that it is one continuous string on + # parameterlist. This fixes a problem where check_sections() + # cannot find a parameter like "addr[6 + 2]" because it actually + # appears as "addr[6", "+", "2]" on the parameter list. + # However, it's better to maintain the param string unchanged for + # output, so just weaken the string compare in check_sections() + # to ignore "[blah" in a parameter string. + + self.entry.parameterlist.append(param) + org_arg =3D KernRe(r'\s\s+').sub(' ', org_arg) + self.entry.parametertypes[param] =3D org_arg + + + def create_parameter_list(self, ln, decl_type, args, + splitter, declaration_name): + """ + Creates a list of parameters, storing them at self.entry. + """ + + # temporarily replace all commas inside function pointer definition + arg_expr =3D KernRe(r'(\([^\),]+),') + while arg_expr.search(args): + args =3D arg_expr.sub(r"\1#", args) + + for arg in args.split(splitter): + # Strip comments + arg =3D KernRe(r'\/\*.*\*\/').sub('', arg) + + # Ignore argument attributes + arg =3D KernRe(r'\sPOS0?\s').sub(' ', arg) + + # Strip leading/trailing spaces + arg =3D arg.strip() + arg =3D KernRe(r'\s+').sub(' ', arg, count=3D1) + + if arg.startswith('#'): + # Treat preprocessor directive as a typeless variable just= to fill + # corresponding data structures "correctly". Catch it late= r in + # output_* subs. + + # Treat preprocessor directive as a typeless variable + self.push_parameter(ln, decl_type, arg, "", + "", declaration_name) + + elif KernRe(r'\(.+\)\s*\(').search(arg): + # Pointer-to-function + + arg =3D arg.replace('#', ',') + + r =3D KernRe(r'[^\(]+\(\*?\s*([\w\[\]\.]*)\s*\)') + if r.match(arg): + param =3D r.group(1) + else: + self.emit_msg(ln, f"Invalid param: {arg}") + param =3D arg + + dtype =3D KernRe(r'([^\(]+\(\*?)\s*' + re.escape(param)).s= ub(r'\1', arg) + self.push_parameter(ln, decl_type, param, dtype, + arg, declaration_name) + + elif KernRe(r'\(.+\)\s*\[').search(arg): + # Array-of-pointers + + arg =3D arg.replace('#', ',') + r =3D KernRe(r'[^\(]+\(\s*\*\s*([\w\[\]\.]*?)\s*(\s*\[\s*[= \w]+\s*\]\s*)*\)') + if r.match(arg): + param =3D r.group(1) + else: + self.emit_msg(ln, f"Invalid param: {arg}") + param =3D arg + + dtype =3D KernRe(r'([^\(]+\(\*?)\s*' + re.escape(param)).s= ub(r'\1', arg) + + self.push_parameter(ln, decl_type, param, dtype, + arg, declaration_name) + + elif arg: + arg =3D KernRe(r'\s*:\s*').sub(":", arg) + arg =3D KernRe(r'\s*\[').sub('[', arg) + + args =3D KernRe(r'\s*,\s*').split(arg) + if args[0] and '*' in args[0]: + args[0] =3D re.sub(r'(\*+)\s*', r' \1', args[0]) + + first_arg =3D [] + r =3D KernRe(r'^(.*\s+)(.*?\[.*\].*)$') + if args[0] and r.match(args[0]): + args.pop(0) + first_arg.extend(r.group(1)) + first_arg.append(r.group(2)) + else: + first_arg =3D KernRe(r'\s+').split(args.pop(0)) + + args.insert(0, first_arg.pop()) + dtype =3D ' '.join(first_arg) + + for param in args: + if KernRe(r'^(\*+)\s*(.*)').match(param): + r =3D KernRe(r'^(\*+)\s*(.*)') + if not r.match(param): + self.emit_msg(ln, f"Invalid param: {param}") + continue + + param =3D r.group(1) + + self.push_parameter(ln, decl_type, r.group(2), + f"{dtype} {r.group(1)}", + arg, declaration_name) + + elif KernRe(r'(.*?):(\w+)').search(param): + r =3D KernRe(r'(.*?):(\w+)') + if not r.match(param): + self.emit_msg(ln, f"Invalid param: {param}") + continue + + if dtype !=3D "": # Skip unnamed bit-fields + self.push_parameter(ln, decl_type, r.group(1), + f"{dtype}:{r.group(2)}", + arg, declaration_name) + else: + self.push_parameter(ln, decl_type, param, dtype, + arg, declaration_name) + + def check_sections(self, ln, decl_name, decl_type): + """ + Check for errors inside sections, emitting warnings if not found + parameters are described. + """ + for section in self.entry.sections: + if section not in self.entry.parameterlist and \ + not known_sections.search(section): + if decl_type =3D=3D 'function': + dname =3D f"{decl_type} parameter" + else: + dname =3D f"{decl_type} member" + self.emit_msg(ln, + f"Excess {dname} '{section}' description in = '{decl_name}'") + + def check_return_section(self, ln, declaration_name, return_type): + """ + If the function doesn't return void, warns about the lack of a + return description. + """ + + if not self.config.wreturn: + return + + # Ignore an empty return type (It's a macro) + # Ignore functions with a "void" return type (but not "void *") + if not return_type or KernRe(r'void\s*\w*\s*$').search(return_type= ): + return + + if not self.entry.sections.get("Return", None): + self.emit_msg(ln, + f"No description found for return value of '{dec= laration_name}'") + + def dump_struct(self, ln, proto): + """ + Store an entry for an struct or union + """ + + type_pattern =3D r'(struct|union)' + + qualifiers =3D [ + "__attribute__", + "__packed", + "__aligned", + "____cacheline_aligned_in_smp", + "____cacheline_aligned", + ] + + definition_body =3D r'\{(.*)\}\s*' + "(?:" + '|'.join(qualifiers) = + ")?" + struct_members =3D KernRe(type_pattern + r'([^\{\};]+)(\{)([^\{\}]= *)(\})([^\{\}\;]*)(\;)') + + # Extract struct/union definition + members =3D None + declaration_name =3D None + decl_type =3D None + + r =3D KernRe(type_pattern + r'\s+(\w+)\s*' + definition_body) + if r.search(proto): + decl_type =3D r.group(1) + declaration_name =3D r.group(2) + members =3D r.group(3) + else: + r =3D KernRe(r'typedef\s+' + type_pattern + r'\s*' + definitio= n_body + r'\s*(\w+)\s*;') + + if r.search(proto): + decl_type =3D r.group(1) + declaration_name =3D r.group(3) + members =3D r.group(2) + + if not members: + self.emit_msg(ln, f"{proto} error: Cannot parse struct or unio= n!") + return + + if self.entry.identifier !=3D declaration_name: + self.emit_msg(ln, + f"expecting prototype for {decl_type} {self.entr= y.identifier}. Prototype was for {decl_type} {declaration_name} instead\n") + return + + args_pattern =3D r'([^,)]+)' + + sub_prefixes =3D [ + (KernRe(r'\/\*\s*private:.*?\/\*\s*public:.*?\*\/', re.S | re.= I), ''), + (KernRe(r'\/\*\s*private:.*', re.S | re.I), ''), + + # Strip comments + (KernRe(r'\/\*.*?\*\/', re.S), ''), + + # Strip attributes + (attribute, ' '), + (KernRe(r'\s*__aligned\s*\([^;]*\)', re.S), ' '), + (KernRe(r'\s*__counted_by\s*\([^;]*\)', re.S), ' '), + (KernRe(r'\s*__counted_by_(le|be)\s*\([^;]*\)', re.S), ' '), + (KernRe(r'\s*__packed\s*', re.S), ' '), + (KernRe(r'\s*CRYPTO_MINALIGN_ATTR', re.S), ' '), + (KernRe(r'\s*____cacheline_aligned_in_smp', re.S), ' '), + (KernRe(r'\s*____cacheline_aligned', re.S), ' '), + + # Unwrap struct_group macros based on this definition: + # __struct_group(TAG, NAME, ATTRS, MEMBERS...) + # which has variants like: struct_group(NAME, MEMBERS...) + # Only MEMBERS arguments require documentation. + # + # Parsing them happens on two steps: + # + # 1. drop struct group arguments that aren't at MEMBERS, + # storing them as STRUCT_GROUP(MEMBERS) + # + # 2. remove STRUCT_GROUP() ancillary macro. + # + # The original logic used to remove STRUCT_GROUP() using an + # advanced regex: + # + # \bSTRUCT_GROUP(\(((?:(?>[^)(]+)|(?1))*)\))[^;]*; + # + # with two patterns that are incompatible with + # Python re module, as it has: + # + # - a recursive pattern: (?1) + # - an atomic grouping: (?>...) + # + # I tried a simpler version: but it didn't work either: + # \bSTRUCT_GROUP\(([^\)]+)\)[^;]*; + # + # As it doesn't properly match the end parenthesis on some cas= es. + # + # So, a better solution was crafted: there's now a NestedMatch + # class that ensures that delimiters after a search are proper= ly + # matched. So, the implementation to drop STRUCT_GROUP() will = be + # handled in separate. + + (KernRe(r'\bstruct_group\s*\(([^,]*,)', re.S), r'STRUCT_GROUP(= '), + (KernRe(r'\bstruct_group_attr\s*\(([^,]*,){2}', re.S), r'STRUC= T_GROUP('), + (KernRe(r'\bstruct_group_tagged\s*\(([^,]*),([^,]*),', re.S), = r'struct \1 \2; STRUCT_GROUP('), + (KernRe(r'\b__struct_group\s*\(([^,]*,){3}', re.S), r'STRUCT_G= ROUP('), + + # Replace macros + # + # TODO: use NestedMatch for FOO($1, $2, ...) matches + # + # it is better to also move those to the NestedMatch logic, + # to ensure that parenthesis will be properly matched. + + (KernRe(r'__ETHTOOL_DECLARE_LINK_MODE_MASK\s*\(([^\)]+)\)', re= .S), r'DECLARE_BITMAP(\1, __ETHTOOL_LINK_MODE_MASK_NBITS)'), + (KernRe(r'DECLARE_PHY_INTERFACE_MASK\s*\(([^\)]+)\)', re.S), r= 'DECLARE_BITMAP(\1, PHY_INTERFACE_MODE_MAX)'), + (KernRe(r'DECLARE_BITMAP\s*\(' + args_pattern + r',\s*' + args= _pattern + r'\)', re.S), r'unsigned long \1[BITS_TO_LONGS(\2)]'), + (KernRe(r'DECLARE_HASHTABLE\s*\(' + args_pattern + r',\s*' + a= rgs_pattern + r'\)', re.S), r'unsigned long \1[1 << ((\2) - 1)]'), + (KernRe(r'DECLARE_KFIFO\s*\(' + args_pattern + r',\s*' + args_= pattern + r',\s*' + args_pattern + r'\)', re.S), r'\2 *\1'), + (KernRe(r'DECLARE_KFIFO_PTR\s*\(' + args_pattern + r',\s*' + a= rgs_pattern + r'\)', re.S), r'\2 *\1'), + (KernRe(r'(?:__)?DECLARE_FLEX_ARRAY\s*\(' + args_pattern + r',= \s*' + args_pattern + r'\)', re.S), r'\1 \2[]'), + (KernRe(r'DEFINE_DMA_UNMAP_ADDR\s*\(' + args_pattern + r'\)', = re.S), r'dma_addr_t \1'), + (KernRe(r'DEFINE_DMA_UNMAP_LEN\s*\(' + args_pattern + r'\)', r= e.S), r'__u32 \1'), + (KernRe(r'VIRTIO_DECLARE_FEATURES\s*\(' + args_pattern + r'\)'= , re.S), r'u64 \1; u64 \1_array[VIRTIO_FEATURES_DWORDS]'), + ] + + # Regexes here are guaranteed to have the end limiter matching + # the start delimiter. Yet, right now, only one replace group + # is allowed. + + sub_nested_prefixes =3D [ + (re.compile(r'\bSTRUCT_GROUP\('), r'\1'), + ] + + for search, sub in sub_prefixes: + members =3D search.sub(sub, members) + + nested =3D NestedMatch() + + for search, sub in sub_nested_prefixes: + members =3D nested.sub(search, sub, members) + + # Keeps the original declaration as-is + declaration =3D members + + # Split nested struct/union elements + # + # This loop was simpler at the original kernel-doc perl version, as + # while ($members =3D~ m/$struct_members/) { ... } + # reads 'members' string on each interaction. + # + # Python behavior is different: it parses 'members' only once, + # creating a list of tuples from the first interaction. + # + # On other words, this won't get nested structs. + # + # So, we need to have an extra loop on Python to override such + # re limitation. + + while True: + tuples =3D struct_members.findall(members) + if not tuples: + break + + for t in tuples: + newmember =3D "" + maintype =3D t[0] + s_ids =3D t[5] + content =3D t[3] + + oldmember =3D "".join(t) + + for s_id in s_ids.split(','): + s_id =3D s_id.strip() + + newmember +=3D f"{maintype} {s_id}; " + s_id =3D KernRe(r'[:\[].*').sub('', s_id) + s_id =3D KernRe(r'^\s*\**(\S+)\s*').sub(r'\1', s_id) + + for arg in content.split(';'): + arg =3D arg.strip() + + if not arg: + continue + + r =3D KernRe(r'^([^\(]+\(\*?\s*)([\w\.]*)(\s*\).*)= ') + if r.match(arg): + # Pointer-to-function + dtype =3D r.group(1) + name =3D r.group(2) + extra =3D r.group(3) + + if not name: + continue + + if not s_id: + # Anonymous struct/union + newmember +=3D f"{dtype}{name}{extra}; " + else: + newmember +=3D f"{dtype}{s_id}.{name}{extr= a}; " + + else: + arg =3D arg.strip() + # Handle bitmaps + arg =3D KernRe(r':\s*\d+\s*').sub('', arg) + + # Handle arrays + arg =3D KernRe(r'\[.*\]').sub('', arg) + + # Handle multiple IDs + arg =3D KernRe(r'\s*,\s*').sub(',', arg) + + r =3D KernRe(r'(.*)\s+([\S+,]+)') + + if r.search(arg): + dtype =3D r.group(1) + names =3D r.group(2) + else: + newmember +=3D f"{arg}; " + continue + + for name in names.split(','): + name =3D KernRe(r'^\s*\**(\S+)\s*').sub(r'= \1', name).strip() + + if not name: + continue + + if not s_id: + # Anonymous struct/union + newmember +=3D f"{dtype} {name}; " + else: + newmember +=3D f"{dtype} {s_id}.{name}= ; " + + members =3D members.replace(oldmember, newmember) + + # Ignore other nested elements, like enums + members =3D re.sub(r'(\{[^\{\}]*\})', '', members) + + self.create_parameter_list(ln, decl_type, members, ';', + declaration_name) + self.check_sections(ln, declaration_name, decl_type) + + # Adjust declaration for better display + declaration =3D KernRe(r'([\{;])').sub(r'\1\n', declaration) + declaration =3D KernRe(r'\}\s+;').sub('};', declaration) + + # Better handle inlined enums + while True: + r =3D KernRe(r'(enum\s+\{[^\}]+),([^\n])') + if not r.search(declaration): + break + + declaration =3D r.sub(r'\1,\n\2', declaration) + + def_args =3D declaration.split('\n') + level =3D 1 + declaration =3D "" + for clause in def_args: + + clause =3D clause.strip() + clause =3D KernRe(r'\s+').sub(' ', clause, count=3D1) + + if not clause: + continue + + if '}' in clause and level > 1: + level -=3D 1 + + if not KernRe(r'^\s*#').match(clause): + declaration +=3D "\t" * level + + declaration +=3D "\t" + clause + "\n" + if "{" in clause and "}" not in clause: + level +=3D 1 + + self.output_declaration(decl_type, declaration_name, + definition=3Ddeclaration, + purpose=3Dself.entry.declaration_purpose) + + def dump_enum(self, ln, proto): + """ + Stores an enum inside self.entries array. + """ + + # Ignore members marked private + proto =3D KernRe(r'\/\*\s*private:.*?\/\*\s*public:.*?\*\/', flags= =3Dre.S).sub('', proto) + proto =3D KernRe(r'\/\*\s*private:.*}', flags=3Dre.S).sub('}', pro= to) + + # Strip comments + proto =3D KernRe(r'\/\*.*?\*\/', flags=3Dre.S).sub('', proto) + + # Strip #define macros inside enums + proto =3D KernRe(r'#\s*((define|ifdef|if)\s+|endif)[^;]*;', flags= =3Dre.S).sub('', proto) + + # + # Parse out the name and members of the enum. Typedef form first. + # + r =3D KernRe(r'typedef\s+enum\s*\{(.*)\}\s*(\w*)\s*;') + if r.search(proto): + declaration_name =3D r.group(2) + members =3D r.group(1).rstrip() + # + # Failing that, look for a straight enum + # + else: + r =3D KernRe(r'enum\s+(\w*)\s*\{(.*)\}') + if r.match(proto): + declaration_name =3D r.group(1) + members =3D r.group(2).rstrip() + # + # OK, this isn't going to work. + # + else: + self.emit_msg(ln, f"{proto}: error: Cannot parse enum!") + return + # + # Make sure we found what we were expecting. + # + if self.entry.identifier !=3D declaration_name: + if self.entry.identifier =3D=3D "": + self.emit_msg(ln, + f"{proto}: wrong kernel-doc identifier on pr= ototype") + else: + self.emit_msg(ln, + f"expecting prototype for enum {self.entry.i= dentifier}. " + f"Prototype was for enum {declaration_name} = instead") + return + + if not declaration_name: + declaration_name =3D "(anonymous)" + # + # Parse out the name of each enum member, and verify that we + # have a description for it. + # + member_set =3D set() + members =3D KernRe(r'\([^;)]*\)').sub('', members) + for arg in members.split(','): + if not arg: + continue + arg =3D KernRe(r'^\s*(\w+).*').sub(r'\1', arg) + self.entry.parameterlist.append(arg) + if arg not in self.entry.parameterdescs: + self.entry.parameterdescs[arg] =3D self.undescribed + self.emit_msg(ln, + f"Enum value '{arg}' not described in enum '= {declaration_name}'") + member_set.add(arg) + # + # Ensure that every described member actually exists in the enum. + # + for k in self.entry.parameterdescs: + if k not in member_set: + self.emit_msg(ln, + f"Excess enum value '%{k}' description in '{= declaration_name}'") + + self.output_declaration('enum', declaration_name, + purpose=3Dself.entry.declaration_purpose) + + def dump_declaration(self, ln, prototype): + """ + Stores a data declaration inside self.entries array. + """ + + if self.entry.decl_type =3D=3D "enum": + self.dump_enum(ln, prototype) + elif self.entry.decl_type =3D=3D "typedef": + self.dump_typedef(ln, prototype) + elif self.entry.decl_type in ["union", "struct"]: + self.dump_struct(ln, prototype) + else: + # This would be a bug + self.emit_message(ln, f'Unknown declaration type: {self.entry.= decl_type}') + + def dump_function(self, ln, prototype): + """ + Stores a function of function macro inside self.entries array. + """ + + func_macro =3D False + return_type =3D '' + decl_type =3D 'function' + + # Prefixes that would be removed + sub_prefixes =3D [ + (r"^static +", "", 0), + (r"^extern +", "", 0), + (r"^asmlinkage +", "", 0), + (r"^inline +", "", 0), + (r"^__inline__ +", "", 0), + (r"^__inline +", "", 0), + (r"^__always_inline +", "", 0), + (r"^noinline +", "", 0), + (r"^__FORTIFY_INLINE +", "", 0), + (r"__init +", "", 0), + (r"__init_or_module +", "", 0), + (r"__deprecated +", "", 0), + (r"__flatten +", "", 0), + (r"__meminit +", "", 0), + (r"__must_check +", "", 0), + (r"__weak +", "", 0), + (r"__sched +", "", 0), + (r"_noprof", "", 0), + (r"__printf\s*\(\s*\d*\s*,\s*\d*\s*\) +", "", 0), + (r"__(?:re)?alloc_size\s*\(\s*\d+\s*(?:,\s*\d+\s*)?\) +", "", = 0), + (r"__diagnose_as\s*\(\s*\S+\s*(?:,\s*\d+\s*)*\) +", "", 0), + (r"DECL_BUCKET_PARAMS\s*\(\s*(\S+)\s*,\s*(\S+)\s*\)", r"\1, \2= ", 0), + (r"__attribute_const__ +", "", 0), + + # It seems that Python support for re.X is broken: + # At least for me (Python 3.13), this didn't work +# (r""" +# __attribute__\s*\(\( +# (?: +# [\w\s]+ # attribute name +# (?:\([^)]*\))? # attribute arguments +# \s*,? # optional comma at the end +# )+ +# \)\)\s+ +# """, "", re.X), + + # So, remove whitespaces and comments from it + (r"__attribute__\s*\(\((?:[\w\s]+(?:\([^)]*\))?\s*,?)+\)\)\s+"= , "", 0), + ] + + for search, sub, flags in sub_prefixes: + prototype =3D KernRe(search, flags).sub(sub, prototype) + + # Macros are a special case, as they change the prototype format + new_proto =3D KernRe(r"^#\s*define\s+").sub("", prototype) + if new_proto !=3D prototype: + is_define_proto =3D True + prototype =3D new_proto + else: + is_define_proto =3D False + + # Yes, this truly is vile. We are looking for: + # 1. Return type (may be nothing if we're looking at a macro) + # 2. Function name + # 3. Function parameters. + # + # All the while we have to watch out for function pointer paramete= rs + # (which IIRC is what the two sections are for), C types (these + # regexps don't even start to express all the possibilities), and + # so on. + # + # If you mess with these regexps, it's a good idea to check that + # the following functions' documentation still comes out right: + # - parport_register_device (function pointer parameters) + # - atomic_set (macro) + # - pci_match_device, __copy_to_user (long return type) + + name =3D r'[a-zA-Z0-9_~:]+' + prototype_end1 =3D r'[^\(]*' + prototype_end2 =3D r'[^\{]*' + prototype_end =3D fr'\(({prototype_end1}|{prototype_end2})\)' + + # Besides compiling, Perl qr{[\w\s]+} works as a non-capturing gro= up. + # So, this needs to be mapped in Python with (?:...)? or (?:...)+ + + type1 =3D r'(?:[\w\s]+)?' + type2 =3D r'(?:[\w\s]+\*+)+' + + found =3D False + + if is_define_proto: + r =3D KernRe(r'^()(' + name + r')\s+') + + if r.search(prototype): + return_type =3D '' + declaration_name =3D r.group(2) + func_macro =3D True + + found =3D True + + if not found: + patterns =3D [ + rf'^()({name})\s*{prototype_end}', + rf'^({type1})\s+({name})\s*{prototype_end}', + rf'^({type2})\s*({name})\s*{prototype_end}', + ] + + for p in patterns: + r =3D KernRe(p) + + if r.match(prototype): + + return_type =3D r.group(1) + declaration_name =3D r.group(2) + args =3D r.group(3) + + self.create_parameter_list(ln, decl_type, args, ',', + declaration_name) + + found =3D True + break + if not found: + self.emit_msg(ln, + f"cannot understand function prototype: '{protot= ype}'") + return + + if self.entry.identifier !=3D declaration_name: + self.emit_msg(ln, + f"expecting prototype for {self.entry.identifier= }(). Prototype was for {declaration_name}() instead") + return + + self.check_sections(ln, declaration_name, "function") + + self.check_return_section(ln, declaration_name, return_type) + + if 'typedef' in return_type: + self.output_declaration(decl_type, declaration_name, + typedef=3DTrue, + functiontype=3Dreturn_type, + purpose=3Dself.entry.declaration_purpo= se, + func_macro=3Dfunc_macro) + else: + self.output_declaration(decl_type, declaration_name, + typedef=3DFalse, + functiontype=3Dreturn_type, + purpose=3Dself.entry.declaration_purpo= se, + func_macro=3Dfunc_macro) + + def dump_typedef(self, ln, proto): + """ + Stores a typedef inside self.entries array. + """ + + typedef_type =3D r'((?:\s+[\w\*]+\b){0,7}\s+(?:\w+\b|\*+))\s*' + typedef_ident =3D r'\*?\s*(\w\S+)\s*' + typedef_args =3D r'\s*\((.*)\);' + + typedef1 =3D KernRe(r'typedef' + typedef_type + r'\(' + typedef_id= ent + r'\)' + typedef_args) + typedef2 =3D KernRe(r'typedef' + typedef_type + typedef_ident + ty= pedef_args) + + # Strip comments + proto =3D KernRe(r'/\*.*?\*/', flags=3Dre.S).sub('', proto) + + # Parse function typedef prototypes + for r in [typedef1, typedef2]: + if not r.match(proto): + continue + + return_type =3D r.group(1).strip() + declaration_name =3D r.group(2) + args =3D r.group(3) + + if self.entry.identifier !=3D declaration_name: + self.emit_msg(ln, + f"expecting prototype for typedef {self.entr= y.identifier}. Prototype was for typedef {declaration_name} instead\n") + return + + decl_type =3D 'function' + self.create_parameter_list(ln, decl_type, args, ',', declarati= on_name) + + self.output_declaration(decl_type, declaration_name, + typedef=3DTrue, + functiontype=3Dreturn_type, + purpose=3Dself.entry.declaration_purpo= se) + return + + # Handle nested parentheses or brackets + r =3D KernRe(r'(\(*.\)\s*|\[*.\]\s*);$') + while r.search(proto): + proto =3D r.sub('', proto) + + # Parse simple typedefs + r =3D KernRe(r'typedef.*\s+(\w+)\s*;') + if r.match(proto): + declaration_name =3D r.group(1) + + if self.entry.identifier !=3D declaration_name: + self.emit_msg(ln, + f"expecting prototype for typedef {self.entr= y.identifier}. Prototype was for typedef {declaration_name} instead\n") + return + + self.output_declaration('typedef', declaration_name, + purpose=3Dself.entry.declaration_purpo= se) + return + + self.emit_msg(ln, "error: Cannot parse typedef!") + + @staticmethod + def process_export(function_set, line): + """ + process EXPORT_SYMBOL* tags + + This method doesn't use any variable from the class, so declare it + with a staticmethod decorator. + """ + + # We support documenting some exported symbols with different + # names. A horrible hack. + suffixes =3D [ '_noprof' ] + + # Note: it accepts only one EXPORT_SYMBOL* per line, as having + # multiple export lines would violate Kernel coding style. + + if export_symbol.search(line): + symbol =3D export_symbol.group(2) + elif export_symbol_ns.search(line): + symbol =3D export_symbol_ns.group(2) + else: + return False + # + # Found an export, trim out any special suffixes + # + for suffix in suffixes: + # Be backward compatible with Python < 3.9 + if symbol.endswith(suffix): + symbol =3D symbol[:-len(suffix)] + function_set.add(symbol) + return True + + def process_normal(self, ln, line): + """ + STATE_NORMAL: looking for the /** to begin everything. + """ + + if not doc_start.match(line): + return + + # start a new entry + self.reset_state(ln) + + # next line is always the function name + self.state =3D state.NAME + + def process_name(self, ln, line): + """ + STATE_NAME: Looking for the "name - description" line + """ + # + # Check for a DOC: block and handle them specially. + # + if doc_block.search(line): + + if not doc_block.group(1): + self.entry.begin_section(ln, "Introduction") + else: + self.entry.begin_section(ln, doc_block.group(1)) + + self.entry.identifier =3D self.entry.section + self.state =3D state.DOCBLOCK + # + # Otherwise we're looking for a normal kerneldoc declaration line. + # + elif doc_decl.search(line): + self.entry.identifier =3D doc_decl.group(1) + + # Test for data declaration + if doc_begin_data.search(line): + self.entry.decl_type =3D doc_begin_data.group(1) + self.entry.identifier =3D doc_begin_data.group(2) + # + # Look for a function description + # + elif doc_begin_func.search(line): + self.entry.identifier =3D doc_begin_func.group(1) + self.entry.decl_type =3D "function" + # + # We struck out. + # + else: + self.emit_msg(ln, + f"This comment starts with '/**', but isn't = a kernel-doc comment. Refer Documentation/doc-guide/kernel-doc.rst\n{line}") + self.state =3D state.NORMAL + return + # + # OK, set up for a new kerneldoc entry. + # + self.state =3D state.BODY + self.entry.identifier =3D self.entry.identifier.strip(" ") + # if there's no @param blocks need to set up default section h= ere + self.entry.begin_section(ln + 1) + # + # Find the description portion, which *should* be there but + # isn't always. + # (We should be able to capture this from the previous parsing= - someday) + # + r =3D KernRe("[-:](.*)") + if r.search(line): + self.entry.declaration_purpose =3D trim_whitespace(r.group= (1)) + self.state =3D state.DECLARATION + else: + self.entry.declaration_purpose =3D "" + + if not self.entry.declaration_purpose and self.config.wshort_d= esc: + self.emit_msg(ln, + f"missing initial short description on line:= \n{line}") + + if not self.entry.identifier and self.entry.decl_type !=3D "en= um": + self.emit_msg(ln, + f"wrong kernel-doc identifier on line:\n{lin= e}") + self.state =3D state.NORMAL + + if self.config.verbose: + self.emit_msg(ln, + f"Scanning doc for {self.entry.decl_type} {s= elf.entry.identifier}", + warning=3DFalse) + # + # Failed to find an identifier. Emit a warning + # + else: + self.emit_msg(ln, f"Cannot find identifier on line:\n{line}") + + # + # Helper function to determine if a new section is being started. + # + def is_new_section(self, ln, line): + if doc_sect.search(line): + self.state =3D state.BODY + # + # Pick out the name of our new section, tweaking it if need be. + # + newsection =3D doc_sect.group(1) + if newsection.lower() =3D=3D 'description': + newsection =3D 'Description' + elif newsection.lower() =3D=3D 'context': + newsection =3D 'Context' + self.state =3D state.SPECIAL_SECTION + elif newsection.lower() in ["@return", "@returns", + "return", "returns"]: + newsection =3D "Return" + self.state =3D state.SPECIAL_SECTION + elif newsection[0] =3D=3D '@': + self.state =3D state.SPECIAL_SECTION + # + # Initialize the contents, and get the new section going. + # + newcontents =3D doc_sect.group(2) + if not newcontents: + newcontents =3D "" + self.dump_section() + self.entry.begin_section(ln, newsection) + self.entry.leading_space =3D None + + self.entry.add_text(newcontents.lstrip()) + return True + return False + + # + # Helper function to detect (and effect) the end of a kerneldoc commen= t. + # + def is_comment_end(self, ln, line): + if doc_end.search(line): + self.dump_section() + + # Look for doc_com + + doc_end: + r =3D KernRe(r'\s*\*\s*[a-zA-Z_0-9:\.]+\*/') + if r.match(line): + self.emit_msg(ln, f"suspicious ending line: {line}") + + self.entry.prototype =3D "" + self.entry.new_start_line =3D ln + 1 + + self.state =3D state.PROTO + return True + return False + + + def process_decl(self, ln, line): + """ + STATE_DECLARATION: We've seen the beginning of a declaration + """ + if self.is_new_section(ln, line) or self.is_comment_end(ln, line): + return + # + # Look for anything with the " * " line beginning. + # + if doc_content.search(line): + cont =3D doc_content.group(1) + # + # A blank line means that we have moved out of the declaration + # part of the comment (without any "special section" parameter + # descriptions). + # + if cont =3D=3D "": + self.state =3D state.BODY + # + # Otherwise we have more of the declaration section to soak up. + # + else: + self.entry.declaration_purpose =3D \ + trim_whitespace(self.entry.declaration_purpose + ' ' += cont) + else: + # Unknown line, ignore + self.emit_msg(ln, f"bad line: {line}") + + + def process_special(self, ln, line): + """ + STATE_SPECIAL_SECTION: a section ending with a blank line + """ + # + # If we have hit a blank line (only the " * " marker), then this + # section is done. + # + if KernRe(r"\s*\*\s*$").match(line): + self.entry.begin_section(ln, dump =3D True) + self.state =3D state.BODY + return + # + # Not a blank line, look for the other ways to end the section. + # + if self.is_new_section(ln, line) or self.is_comment_end(ln, line): + return + # + # OK, we should have a continuation of the text for this section. + # + if doc_content.search(line): + cont =3D doc_content.group(1) + # + # If the lines of text after the first in a special section ha= ve + # leading white space, we need to trim it out or Sphinx will g= et + # confused. For the second line (the None case), see what we + # find there and remember it. + # + if self.entry.leading_space is None: + r =3D KernRe(r'^(\s+)') + if r.match(cont): + self.entry.leading_space =3D len(r.group(1)) + else: + self.entry.leading_space =3D 0 + # + # Otherwise, before trimming any leading chars, be *sure* + # that they are white space. We should maybe warn if this + # isn't the case. + # + for i in range(0, self.entry.leading_space): + if cont[i] !=3D " ": + self.entry.leading_space =3D i + break + # + # Add the trimmed result to the section and we're done. + # + self.entry.add_text(cont[self.entry.leading_space:]) + else: + # Unknown line, ignore + self.emit_msg(ln, f"bad line: {line}") + + def process_body(self, ln, line): + """ + STATE_BODY: the bulk of a kerneldoc comment. + """ + if self.is_new_section(ln, line) or self.is_comment_end(ln, line): + return + + if doc_content.search(line): + cont =3D doc_content.group(1) + self.entry.add_text(cont) + else: + # Unknown line, ignore + self.emit_msg(ln, f"bad line: {line}") + + def process_inline_name(self, ln, line): + """STATE_INLINE_NAME: beginning of docbook comments within a proto= type.""" + + if doc_inline_sect.search(line): + self.entry.begin_section(ln, doc_inline_sect.group(1)) + self.entry.add_text(doc_inline_sect.group(2).lstrip()) + self.state =3D state.INLINE_TEXT + elif doc_inline_end.search(line): + self.dump_section() + self.state =3D state.PROTO + elif doc_content.search(line): + self.emit_msg(ln, f"Incorrect use of kernel-doc format: {line}= ") + self.state =3D state.PROTO + # else ... ?? + + def process_inline_text(self, ln, line): + """STATE_INLINE_TEXT: docbook comments within a prototype.""" + + if doc_inline_end.search(line): + self.dump_section() + self.state =3D state.PROTO + elif doc_content.search(line): + self.entry.add_text(doc_content.group(1)) + # else ... ?? + + def syscall_munge(self, ln, proto): # pylint: disable=3DW0613 + """ + Handle syscall definitions + """ + + is_void =3D False + + # Strip newlines/CR's + proto =3D re.sub(r'[\r\n]+', ' ', proto) + + # Check if it's a SYSCALL_DEFINE0 + if 'SYSCALL_DEFINE0' in proto: + is_void =3D True + + # Replace SYSCALL_DEFINE with correct return type & function name + proto =3D KernRe(r'SYSCALL_DEFINE.*\(').sub('long sys_', proto) + + r =3D KernRe(r'long\s+(sys_.*?),') + if r.search(proto): + proto =3D KernRe(',').sub('(', proto, count=3D1) + elif is_void: + proto =3D KernRe(r'\)').sub('(void)', proto, count=3D1) + + # Now delete all of the odd-numbered commas in the proto + # so that argument types & names don't have a comma between them + count =3D 0 + length =3D len(proto) + + if is_void: + length =3D 0 # skip the loop if is_void + + for ix in range(length): + if proto[ix] =3D=3D ',': + count +=3D 1 + if count % 2 =3D=3D 1: + proto =3D proto[:ix] + ' ' + proto[ix + 1:] + + return proto + + def tracepoint_munge(self, ln, proto): + """ + Handle tracepoint definitions + """ + + tracepointname =3D None + tracepointargs =3D None + + # Match tracepoint name based on different patterns + r =3D KernRe(r'TRACE_EVENT\((.*?),') + if r.search(proto): + tracepointname =3D r.group(1) + + r =3D KernRe(r'DEFINE_SINGLE_EVENT\((.*?),') + if r.search(proto): + tracepointname =3D r.group(1) + + r =3D KernRe(r'DEFINE_EVENT\((.*?),(.*?),') + if r.search(proto): + tracepointname =3D r.group(2) + + if tracepointname: + tracepointname =3D tracepointname.lstrip() + + r =3D KernRe(r'TP_PROTO\((.*?)\)') + if r.search(proto): + tracepointargs =3D r.group(1) + + if not tracepointname or not tracepointargs: + self.emit_msg(ln, + f"Unrecognized tracepoint format:\n{proto}\n") + else: + proto =3D f"static inline void trace_{tracepointname}({tracepo= intargs})" + self.entry.identifier =3D f"trace_{self.entry.identifier}" + + return proto + + def process_proto_function(self, ln, line): + """Ancillary routine to process a function prototype""" + + # strip C99-style comments to end of line + line =3D KernRe(r"\/\/.*$", re.S).sub('', line) + # + # Soak up the line's worth of prototype text, stopping at { or ; i= f present. + # + if KernRe(r'\s*#\s*define').match(line): + self.entry.prototype =3D line + elif not line.startswith('#'): # skip other preprocessor stuff + r =3D KernRe(r'([^\{]*)') + if r.match(line): + self.entry.prototype +=3D r.group(1) + " " + # + # If we now have the whole prototype, clean it up and declare vict= ory. + # + if '{' in line or ';' in line or KernRe(r'\s*#\s*define').match(li= ne): + # strip comments and surrounding spaces + self.entry.prototype =3D KernRe(r'/\*.*\*/').sub('', self.entr= y.prototype).strip() + # + # Handle self.entry.prototypes for function pointers like: + # int (*pcs_config)(struct foo) + # by turning it into + # int pcs_config(struct foo) + # + r =3D KernRe(r'^(\S+\s+)\(\s*\*(\S+)\)') + self.entry.prototype =3D r.sub(r'\1\2', self.entry.prototype) + # + # Handle special declaration syntaxes + # + if 'SYSCALL_DEFINE' in self.entry.prototype: + self.entry.prototype =3D self.syscall_munge(ln, + self.entry.proto= type) + else: + r =3D KernRe(r'TRACE_EVENT|DEFINE_EVENT|DEFINE_SINGLE_EVEN= T') + if r.search(self.entry.prototype): + self.entry.prototype =3D self.tracepoint_munge(ln, + self.entr= y.prototype) + # + # ... and we're done + # + self.dump_function(ln, self.entry.prototype) + self.reset_state(ln) + + def process_proto_type(self, ln, line): + """Ancillary routine to process a type""" + + # Strip C99-style comments and surrounding whitespace + line =3D KernRe(r"//.*$", re.S).sub('', line).strip() + if not line: + return # nothing to see here + + # To distinguish preprocessor directive from regular declaration l= ater. + if line.startswith('#'): + line +=3D ";" + # + # Split the declaration on any of { } or ;, and accumulate pieces + # until we hit a semicolon while not inside {brackets} + # + r =3D KernRe(r'(.*?)([{};])') + for chunk in r.split(line): + if chunk: # Ignore empty matches + self.entry.prototype +=3D chunk + # + # This cries out for a match statement ... someday after w= e can + # drop Python 3.9 ... + # + if chunk =3D=3D '{': + self.entry.brcount +=3D 1 + elif chunk =3D=3D '}': + self.entry.brcount -=3D 1 + elif chunk =3D=3D ';' and self.entry.brcount <=3D 0: + self.dump_declaration(ln, self.entry.prototype) + self.reset_state(ln) + return + # + # We hit the end of the line while still in the declaration; put + # in a space to represent the newline. + # + self.entry.prototype +=3D ' ' + + def process_proto(self, ln, line): + """STATE_PROTO: reading a function/whatever prototype.""" + + if doc_inline_oneline.search(line): + self.entry.begin_section(ln, doc_inline_oneline.group(1)) + self.entry.add_text(doc_inline_oneline.group(2)) + self.dump_section() + + elif doc_inline_start.search(line): + self.state =3D state.INLINE_NAME + + elif self.entry.decl_type =3D=3D 'function': + self.process_proto_function(ln, line) + + else: + self.process_proto_type(ln, line) + + def process_docblock(self, ln, line): + """STATE_DOCBLOCK: within a DOC: block.""" + + if doc_end.search(line): + self.dump_section() + self.output_declaration("doc", self.entry.identifier) + self.reset_state(ln) + + elif doc_content.search(line): + self.entry.add_text(doc_content.group(1)) + + def parse_export(self): + """ + Parses EXPORT_SYMBOL* macros from a single Kernel source file. + """ + + export_table =3D set() + + try: + with open(self.fname, "r", encoding=3D"utf8", + errors=3D"backslashreplace") as fp: + + for line in fp: + self.process_export(export_table, line) + + except IOError: + return None + + return export_table + + # + # The state/action table telling us which function to invoke in + # each state. + # + state_actions =3D { + state.NORMAL: process_normal, + state.NAME: process_name, + state.BODY: process_body, + state.DECLARATION: process_decl, + state.SPECIAL_SECTION: process_special, + state.INLINE_NAME: process_inline_name, + state.INLINE_TEXT: process_inline_text, + state.PROTO: process_proto, + state.DOCBLOCK: process_docblock, + } + + def parse_kdoc(self): + """ + Open and process each line of a C source file. + The parsing is controlled via a state machine, and the line is pas= sed + to a different process function depending on the state. The process + function may update the state as needed. + + Besides parsing kernel-doc tags, it also parses export symbols. + """ + + prev =3D "" + prev_ln =3D None + export_table =3D set() + + try: + with open(self.fname, "r", encoding=3D"utf8", + errors=3D"backslashreplace") as fp: + for ln, line in enumerate(fp): + + line =3D line.expandtabs().strip("\n") + + # Group continuation lines on prototypes + if self.state =3D=3D state.PROTO: + if line.endswith("\\"): + prev +=3D line.rstrip("\\") + if not prev_ln: + prev_ln =3D ln + continue + + if prev: + ln =3D prev_ln + line =3D prev + line + prev =3D "" + prev_ln =3D None + + self.config.log.debug("%d %s: %s", + ln, state.name[self.state], + line) + + # This is an optimization over the original script. + # There, when export_file was used for the same file, + # it was read twice. Here, we use the already-existing + # loop to parse exported symbols as well. + # + if (self.state !=3D state.NORMAL) or \ + not self.process_export(export_table, line): + # Hand this line to the appropriate state handler + self.state_actions[self.state](self, ln, line) + + except OSError: + self.config.log.error(f"Error: Cannot open file {self.fname}") + + return export_table, self.entries diff --git a/scripts/lib/kdoc/kdoc_re.py b/scripts/lib/kdoc/kdoc_re.py new file mode 100644 index 00000000000..612223e1e72 --- /dev/null +++ b/scripts/lib/kdoc/kdoc_re.py @@ -0,0 +1,270 @@ +#!/usr/bin/env python3 +# SPDX-License-Identifier: GPL-2.0 +# Copyright(c) 2025: Mauro Carvalho Chehab . + +""" +Regular expression ancillary classes. + +Those help caching regular expressions and do matching for kernel-doc. +""" + +import re + +# Local cache for regular expressions +re_cache =3D {} + + +class KernRe: + """ + Helper class to simplify regex declaration and usage, + + It calls re.compile for a given pattern. It also allows adding + regular expressions and define sub at class init time. + + Regular expressions can be cached via an argument, helping to speedup + searches. + """ + + def _add_regex(self, string, flags): + """ + Adds a new regex or re-use it from the cache. + """ + self.regex =3D re_cache.get(string, None) + if not self.regex: + self.regex =3D re.compile(string, flags=3Dflags) + if self.cache: + re_cache[string] =3D self.regex + + def __init__(self, string, cache=3DTrue, flags=3D0): + """ + Compile a regular expression and initialize internal vars. + """ + + self.cache =3D cache + self.last_match =3D None + + self._add_regex(string, flags) + + def __str__(self): + """ + Return the regular expression pattern. + """ + return self.regex.pattern + + def __add__(self, other): + """ + Allows adding two regular expressions into one. + """ + + return KernRe(str(self) + str(other), cache=3Dself.cache or other.= cache, + flags=3Dself.regex.flags | other.regex.flags) + + def match(self, string): + """ + Handles a re.match storing its results + """ + + self.last_match =3D self.regex.match(string) + return self.last_match + + def search(self, string): + """ + Handles a re.search storing its results + """ + + self.last_match =3D self.regex.search(string) + return self.last_match + + def findall(self, string): + """ + Alias to re.findall + """ + + return self.regex.findall(string) + + def split(self, string): + """ + Alias to re.split + """ + + return self.regex.split(string) + + def sub(self, sub, string, count=3D0): + """ + Alias to re.sub + """ + + return self.regex.sub(sub, string, count=3Dcount) + + def group(self, num): + """ + Returns the group results of the last match + """ + + return self.last_match.group(num) + + +class NestedMatch: + """ + Finding nested delimiters is hard with regular expressions. It is + even harder on Python with its normal re module, as there are several + advanced regular expressions that are missing. + + This is the case of this pattern: + + '\\bSTRUCT_GROUP(\\(((?:(?>[^)(]+)|(?1))*)\\))[^;]*;' + + which is used to properly match open/close parenthesis of the + string search STRUCT_GROUP(), + + Add a class that counts pairs of delimiters, using it to match and + replace nested expressions. + + The original approach was suggested by: + https://stackoverflow.com/questions/5454322/python-how-to-match-ne= sted-parentheses-with-regex + + Although I re-implemented it to make it more generic and match 3 types + of delimiters. The logic checks if delimiters are paired. If not, it + will ignore the search string. + """ + + # TODO: make NestedMatch handle multiple match groups + # + # Right now, regular expressions to match it are defined only up to + # the start delimiter, e.g.: + # + # \bSTRUCT_GROUP\( + # + # is similar to: STRUCT_GROUP\((.*)\) + # except that the content inside the match group is delimiter's aligne= d. + # + # The content inside parenthesis are converted into a single replace + # group (e.g. r`\1'). + # + # It would be nice to change such definition to support multiple + # match groups, allowing a regex equivalent to. + # + # FOO\((.*), (.*), (.*)\) + # + # it is probably easier to define it not as a regular expression, but + # with some lexical definition like: + # + # FOO(arg1, arg2, arg3) + + DELIMITER_PAIRS =3D { + '{': '}', + '(': ')', + '[': ']', + } + + RE_DELIM =3D re.compile(r'[\{\}\[\]\(\)]') + + def _search(self, regex, line): + """ + Finds paired blocks for a regex that ends with a delimiter. + + The suggestion of using finditer to match pairs came from: + https://stackoverflow.com/questions/5454322/python-how-to-match-ne= sted-parentheses-with-regex + but I ended using a different implementation to align all three ty= pes + of delimiters and seek for an initial regular expression. + + The algorithm seeks for open/close paired delimiters and place them + into a stack, yielding a start/stop position of each match when t= he + stack is zeroed. + + The algorithm shoud work fine for properly paired lines, but will + silently ignore end delimiters that preceeds an start delimiter. + This should be OK for kernel-doc parser, as unaligned delimiters + would cause compilation errors. So, we don't need to rise exceptio= ns + to cover such issues. + """ + + stack =3D [] + + for match_re in regex.finditer(line): + start =3D match_re.start() + offset =3D match_re.end() + + d =3D line[offset - 1] + if d not in self.DELIMITER_PAIRS: + continue + + end =3D self.DELIMITER_PAIRS[d] + stack.append(end) + + for match in self.RE_DELIM.finditer(line[offset:]): + pos =3D match.start() + offset + + d =3D line[pos] + + if d in self.DELIMITER_PAIRS: + end =3D self.DELIMITER_PAIRS[d] + + stack.append(end) + continue + + # Does the end delimiter match what it is expected? + if stack and d =3D=3D stack[-1]: + stack.pop() + + if not stack: + yield start, offset, pos + 1 + break + + def search(self, regex, line): + """ + This is similar to re.search: + + It matches a regex that it is followed by a delimiter, + returning occurrences only if all delimiters are paired. + """ + + for t in self._search(regex, line): + + yield line[t[0]:t[2]] + + def sub(self, regex, sub, line, count=3D0): + """ + This is similar to re.sub: + + It matches a regex that it is followed by a delimiter, + replacing occurrences only if all delimiters are paired. + + if r'\1' is used, it works just like re: it places there the + matched paired data with the delimiter stripped. + + If count is different than zero, it will replace at most count + items. + """ + out =3D "" + + cur_pos =3D 0 + n =3D 0 + + for start, end, pos in self._search(regex, line): + out +=3D line[cur_pos:start] + + # Value, ignoring start/end delimiters + value =3D line[end:pos - 1] + + # replaces \1 at the sub string, if \1 is used there + new_sub =3D sub + new_sub =3D new_sub.replace(r'\1', value) + + out +=3D new_sub + + # Drop end ';' if any + if line[pos] =3D=3D ';': + pos +=3D 1 + + cur_pos =3D pos + n +=3D 1 + + if count and count >=3D n: + break + + # Append the remaining string + l =3D len(line) + out +=3D line[cur_pos:l] + + return out --=20 2.43.0