From nobody Thu Nov 13 15:05:36 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=none dis=none) header.from=linaro.org Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1580728567056945.8367341890394; Mon, 3 Feb 2020 03:16:07 -0800 (PST) Received: from localhost ([::1]:38170 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iyZi1-0001bm-Tj for importer@patchew.org; Mon, 03 Feb 2020 06:16:05 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:57411) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iyZfW-0005B0-Gm for qemu-devel@nongnu.org; Mon, 03 Feb 2020 06:13:32 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iyZfU-0000hX-HF for qemu-devel@nongnu.org; Mon, 03 Feb 2020 06:13:30 -0500 Received: from mail-wr1-x433.google.com ([2a00:1450:4864:20::433]:44694) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1iyZfU-0000fq-AB for qemu-devel@nongnu.org; Mon, 03 Feb 2020 06:13:28 -0500 Received: by mail-wr1-x433.google.com with SMTP id m16so17446626wrx.11 for ; Mon, 03 Feb 2020 03:13:28 -0800 (PST) Received: from orth.archaic.org.uk (orth.archaic.org.uk. [81.2.115.148]) by smtp.gmail.com with ESMTPSA id s139sm24028917wme.35.2020.02.03.03.13.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Feb 2020 03:13:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=A2PvC/kAxIK7nZ4Xa63lRXD3s09Rrr9teNP3b5zDNc8=; b=WWxdBbOchZwKIMT9ti0VvH3qpQqdUU2S00PiEG13sBT8cTtiE3ahfOynPTTZPND7EC sW+3+MszCMlhc8KNPl6lIyn+MdI3vfWkj3pgCkVqTq+tFabp3xHo7/DPS5gt58nq4SE/ NqCVpfpMqfflnm14O1Uh0wyD5V/hbTU+hps7IHrUBAz+J5tDxpRibj4/Om07k0hNE+qk sbaJTVIg5R+4klK3zjI4k5GtB3Xht8wToBXJxy/6kaOBvY0WXMaUt9rLkxNgXt9zhF+3 P/BrB4xs/E6zxH4eSD21B7PmMVYpqZZAbJKhagJCgc7etR78zA/3PuVgwXBSlpYNzmh9 fmwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=A2PvC/kAxIK7nZ4Xa63lRXD3s09Rrr9teNP3b5zDNc8=; b=XsgnUIb4A/9lo5PRvx11Hl8ogVOp94U92D6BJ26PjlyECYrvHN3j0bzxOC/lOxPbyF zTZnNGC9idFM9VLFxh+FR9qPLtb8rB0GDK7r4nCFaV6h22iK0QcJt9vpvA7MqW3QMBs5 AF8FLQ4Oe4vtoo3VqhrR+BFsagERF97EvOJUsVlqzxCUNM2DzfX10ykTbaG9XQ3LmPz+ y5RRTqV3SKWhcgNh/kwActFLgES5NPlygvImyMqi/DZnE/I9cYQQcyJ7PVF33GSAoax2 I6hyxSiWSZoiCU+02YTKb0VKdUhgIFgR9yaYRsdZZw4cROA3GWfzI2zYp50dAXv5tBlG kM/g== X-Gm-Message-State: APjAAAVcVVy1lD1CMb1InSuCNpgZ7OKharbM9EUVJD3UedT80zGtq0dX GqxxVBPqZz75pbSa0a/vKjgnqY8nxMJDLA== X-Google-Smtp-Source: APXvYqwjwdlGrw0FHOqtTMMH8dXPUEUSbo5u2K9Kws52WTscrLjJD9CEVfzj4OrQkPpBKbsVznY0lA== X-Received: by 2002:a5d:4702:: with SMTP id y2mr6942835wrq.37.1580728405846; Mon, 03 Feb 2020 03:13:25 -0800 (PST) From: Peter Maydell To: qemu-devel@nongnu.org Subject: [PULL 3/8] docs/sphinx: Add new hxtool Sphinx extension Date: Mon, 3 Feb 2020 11:13:13 +0000 Message-Id: <20200203111318.23378-4-peter.maydell@linaro.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200203111318.23378-1-peter.maydell@linaro.org> References: <20200203111318.23378-1-peter.maydell@linaro.org> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2a00:1450:4864:20::433 X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: "Qemu-devel" X-ZohoMail-DKIM: pass (identity @linaro.org) Some of our documentation includes sections which are created by assembling fragments of texinfo from a .hx source file into a .texi file, which is then included from qemu-doc.texi or qemu-img.texi. For Sphinx, rather than creating a file to include, the most natural way to handle this is to have a small custom Sphinx extension which reads the .hx file and process it. So instead of: * makefile produces foo.texi from foo.hx * qemu-doc.texi says '@include foo.texi' we have: * qemu-doc.rst says 'hxtool-doc:: foo.hx' * the Sphinx extension for hxtool has code that runs to handle that Sphinx directive which reads the .hx file and emits the appropriate documentation contents This is pretty much the same way the kerneldoc extension works right now. It also has the advantage that it should work for third-party services like readthedocs that expect to build the docs directly with sphinx rather than by invoking our makefiles. In this commit we implement the hxtool extension. Note that syntax errors in the rST fragments will be correctly reported to the user with the filename and line number within the hx file. Signed-off-by: Peter Maydell Reviewed-by: Alex Benn=C3=A9e Tested-by: Alex Benn=C3=A9e Message-id: 20200124162606.8787-4-peter.maydell@linaro.org --- docs/conf.py | 3 +- docs/sphinx/hxtool.py | 210 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 212 insertions(+), 1 deletion(-) create mode 100644 docs/sphinx/hxtool.py diff --git a/docs/conf.py b/docs/conf.py index 259c6049da7..ee7faa6b4e7 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -54,7 +54,7 @@ needs_sphinx =3D '1.3' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions =3D ['kerneldoc', 'qmp_lexer'] +extensions =3D ['kerneldoc', 'qmp_lexer', 'hxtool'] =20 # Add any paths that contain templates here, relative to this directory. templates_path =3D ['_templates'] @@ -221,3 +221,4 @@ texinfo_documents =3D [ # find everything. kerneldoc_bin =3D os.path.join(qemu_docdir, '../scripts/kernel-doc') kerneldoc_srctree =3D os.path.join(qemu_docdir, '..') +hxtool_srctree =3D os.path.join(qemu_docdir, '..') diff --git a/docs/sphinx/hxtool.py b/docs/sphinx/hxtool.py new file mode 100644 index 00000000000..5d6736f3002 --- /dev/null +++ b/docs/sphinx/hxtool.py @@ -0,0 +1,210 @@ +# coding=3Dutf-8 +# +# QEMU hxtool .hx file parsing extension +# +# Copyright (c) 2020 Linaro +# +# This work is licensed under the terms of the GNU GPLv2 or later. +# See the COPYING file in the top-level directory. +"""hxtool is a Sphinx extension that implements the hxtool-doc directive""" + +# The purpose of this extension is to read fragments of rST +# from .hx files, and insert them all into the current document. +# The rST fragments are delimited by SRST/ERST lines. +# The conf.py file must set the hxtool_srctree config value to +# the root of the QEMU source tree. +# Each hxtool-doc:: directive takes one argument which is the +# path of the .hx file to process, relative to the source tree. + +import os +import re +from enum import Enum + +from docutils import nodes +from docutils.statemachine import ViewList +from docutils.parsers.rst import directives, Directive +from sphinx.errors import ExtensionError +from sphinx.util.nodes import nested_parse_with_titles +import sphinx + +# Sphinx up to 1.6 uses AutodocReporter; 1.7 and later +# use switch_source_input. Check borrowed from kerneldoc.py. +Use_SSI =3D sphinx.__version__[:3] >=3D '1.7' +if Use_SSI: + from sphinx.util.docutils import switch_source_input +else: + from sphinx.ext.autodoc import AutodocReporter + +__version__ =3D '1.0' + +# We parse hx files with a state machine which may be in one of three +# states: reading the C code fragment, inside a texi fragment, +# or inside a rST fragment. +class HxState(Enum): + CTEXT =3D 1 + TEXI =3D 2 + RST =3D 3 + +def serror(file, lnum, errtext): + """Raise an exception giving a user-friendly syntax error message""" + raise ExtensionError('%s line %d: syntax error: %s' % (file, lnum, err= text)) + +def parse_directive(line): + """Return first word of line, if any""" + return re.split('\W', line)[0] + +def parse_defheading(file, lnum, line): + """Handle a DEFHEADING directive""" + # The input should be "DEFHEADING(some string)", though note that + # the 'some string' could be the empty string. If the string is + # empty we ignore the directive -- these are used only to add + # blank lines in the plain-text content of the --help output. + # + # Return the heading text + match =3D re.match(r'DEFHEADING\((.*)\)', line) + if match is None: + serror(file, lnum, "Invalid DEFHEADING line") + return match.group(1) + +def parse_archheading(file, lnum, line): + """Handle an ARCHHEADING directive""" + # The input should be "ARCHHEADING(some string, other arg)", + # though note that the 'some string' could be the empty string. + # As with DEFHEADING, empty string ARCHHEADINGs will be ignored. + # + # Return the heading text + match =3D re.match(r'ARCHHEADING\((.*),.*\)', line) + if match is None: + serror(file, lnum, "Invalid ARCHHEADING line") + return match.group(1) + +class HxtoolDocDirective(Directive): + """Extract rST fragments from the specified .hx file""" + required_argument =3D 1 + optional_arguments =3D 1 + option_spec =3D { + 'hxfile': directives.unchanged_required + } + has_content =3D False + + def run(self): + env =3D self.state.document.settings.env + hxfile =3D env.config.hxtool_srctree + '/' + self.arguments[0] + + # Tell sphinx of the dependency + env.note_dependency(os.path.abspath(hxfile)) + + state =3D HxState.CTEXT + # We build up lines of rST in this ViewList, which we will + # later put into a 'section' node. + rstlist =3D ViewList() + current_node =3D None + node_list =3D [] + + with open(hxfile) as f: + lines =3D (l.rstrip() for l in f) + for lnum, line in enumerate(lines, 1): + directive =3D parse_directive(line) + + if directive =3D=3D 'HXCOMM': + pass + elif directive =3D=3D 'STEXI': + if state =3D=3D HxState.RST: + serror(hxfile, lnum, 'expected ERST, found STEXI') + elif state =3D=3D HxState.TEXI: + serror(hxfile, lnum, 'expected ETEXI, found STEXI') + else: + state =3D HxState.TEXI + elif directive =3D=3D 'ETEXI': + if state =3D=3D HxState.RST: + serror(hxfile, lnum, 'expected ERST, found ETEXI') + elif state =3D=3D HxState.CTEXT: + serror(hxfile, lnum, 'expected STEXI, found ETEXI') + else: + state =3D HxState.CTEXT + elif directive =3D=3D 'SRST': + if state =3D=3D HxState.RST: + serror(hxfile, lnum, 'expected ERST, found SRST') + elif state =3D=3D HxState.TEXI: + serror(hxfile, lnum, 'expected ETEXI, found SRST') + else: + state =3D HxState.RST + elif directive =3D=3D 'ERST': + if state =3D=3D HxState.TEXI: + serror(hxfile, lnum, 'expected ETEXI, found ERST') + elif state =3D=3D HxState.CTEXT: + serror(hxfile, lnum, 'expected SRST, found ERST') + else: + state =3D HxState.CTEXT + elif directive =3D=3D 'DEFHEADING' or directive =3D=3D 'AR= CHHEADING': + if directive =3D=3D 'DEFHEADING': + heading =3D parse_defheading(hxfile, lnum, line) + else: + heading =3D parse_archheading(hxfile, lnum, line) + if heading =3D=3D "": + continue + # Put the accumulated rST into the previous node, + # and then start a fresh section with this heading. + if len(rstlist) > 0: + if current_node is None: + # We had some rST fragments before the first + # DEFHEADING. We don't have a section to put + # these in, so rather than magicing up a secti= on, + # make it a syntax error. + serror(hxfile, lnum, + 'first DEFHEADING must precede all rST = text') + self.do_parse(rstlist, current_node) + rstlist =3D ViewList() + if current_node is not None: + node_list.append(current_node) + section_id =3D 'hxtool-%d' % env.new_serialno('hxtool') + current_node =3D nodes.section(ids=3D[section_id]) + current_node +=3D nodes.title(heading, heading) + else: + # Not a directive: put in output if we are in rST frag= ment + if state =3D=3D HxState.RST: + # Sphinx counts its lines from 0 + rstlist.append(line, hxfile, lnum - 1) + + if current_node is None: + # We don't have multiple sections, so just parse the rst + # fragments into a dummy node so we can return the children. + current_node =3D nodes.section() + self.do_parse(rstlist, current_node) + return current_node.children + else: + # Put the remaining accumulated rST into the last section, and + # return all the sections. + if len(rstlist) > 0: + self.do_parse(rstlist, current_node) + node_list.append(current_node) + return node_list + + # This is from kerneldoc.py -- it works around an API change in + # Sphinx between 1.6 and 1.7. Unlike kerneldoc.py, we use + # sphinx.util.nodes.nested_parse_with_titles() rather than the + # plain self.state.nested_parse(), and so we can drop the saving + # of title_styles and section_level that kerneldoc.py does, + # because nested_parse_with_titles() does that for us. + def do_parse(self, result, node): + if Use_SSI: + with switch_source_input(self.state, result): + nested_parse_with_titles(self.state, result, node) + else: + save =3D self.state.memo.reporter + self.state.memo.reporter =3D AutodocReporter(result, self.stat= e.memo.reporter) + try: + nested_parse_with_titles(self.state, result, node) + finally: + self.state.memo.reporter =3D save + +def setup(app): + """ Register hxtool-doc directive with Sphinx""" + app.add_config_value('hxtool_srctree', None, 'env') + app.add_directive('hxtool-doc', HxtoolDocDirective) + + return dict( + version =3D __version__, + parallel_read_safe =3D True, + parallel_write_safe =3D True + ) --=20 2.20.1