[PATCH] build: Build and install the info manual.

Maxim Cournoyer posted 1 patch 3 years, 6 months ago
Test docker-quick@centos7 passed
Test docker-mingw@fedora failed
Test checkpatch failed
Test FreeBSD passed
Patches applied successfully (tree, apply log)
git fetch https://github.com/patchew-project/qemu tags/patchew/20200925024143.26492-1-maxim.cournoyer@gmail.com
configure        |  3 +++
docs/index.rst   |  2 ++
docs/meson.build | 13 +++++++++++++
meson.build      |  4 +++-
4 files changed, 21 insertions(+), 1 deletion(-)
[PATCH] build: Build and install the info manual.
Posted by Maxim Cournoyer 3 years, 6 months ago
Take advantage of the Sphinx texinfo backend to generate a QEMU info
manual.  The texinfo format allows for more structure and info readers
provide more advanced navigation capabilities compared to manpages
readers.

* configure (infodir): Add the --infodir option, which allows
configuring the directory under which the info manuals are installed.
* docs/index.rst: Include the top level documents to prevent
warnings (treated as errors by sphinx-build).
* docs/meson.build (sphinxinfo): Add new target.
---
 configure        |  3 +++
 docs/index.rst   |  2 ++
 docs/meson.build | 13 +++++++++++++
 meson.build      |  4 +++-
 4 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/configure b/configure
index e8e8e984f2..9dfd10e063 100755
--- a/configure
+++ b/configure
@@ -1069,6 +1069,8 @@ for opt do
     static="yes"
     QEMU_PKG_CONFIG_FLAGS="--static $QEMU_PKG_CONFIG_FLAGS"
   ;;
+  --infodir=*) infodir="$optarg"
+  ;;
   --mandir=*) mandir="$optarg"
   ;;
   --bindir=*) bindir="$optarg"
@@ -1786,6 +1788,7 @@ Advanced options (experts only):
   --smbd=SMBD              use specified smbd [$smbd]
   --with-git=GIT           use specified git [$git]
   --static                 enable static build [$static]
+  --infodir=PATH           install info manual in PATH
   --mandir=PATH            install man pages in PATH
   --datadir=PATH           install firmware in PATH/$qemu_suffix
   --docdir=PATH            install documentation in PATH/$qemu_suffix
diff --git a/docs/index.rst b/docs/index.rst
index 763e3d0426..4f155b51d7 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -9,6 +9,7 @@ Welcome to QEMU's documentation!
 .. toctree::
    :maxdepth: 2
    :caption: Contents:
+   :glob:
 
    system/index
    user/index
@@ -16,3 +17,4 @@ Welcome to QEMU's documentation!
    interop/index
    specs/index
    devel/index
+   *
diff --git a/docs/meson.build b/docs/meson.build
index 50f367349b..eb94e4f3cf 100644
--- a/docs/meson.build
+++ b/docs/meson.build
@@ -32,10 +32,13 @@ if build_docs
 
   sphinxdocs = []
   sphinxmans = []
+  conf_py_files = [files('conf.py')]
+
   foreach manual : manuals
     private_dir = meson.current_build_dir() / (manual + '.p')
     output_dir = meson.current_build_dir() / manual
     input_dir = meson.current_source_dir() / manual
+    conf_py_files += files(manual / 'conf.py')
 
     this_manual = custom_target(manual + ' manual',
                 build_by_default: build_docs,
@@ -70,4 +73,14 @@ if build_docs
   endforeach
   alias_target('sphinxdocs', sphinxdocs)
   alias_target('man', sphinxmans)
+
+  sphinxinfo = custom_target(
+    'QEMU info manual generated source',
+    output: ['QEMU.texi', 'sphinxinfo.stamp'],
+    input: conf_py_files,
+    depfile: 'sphinxinfo.d',
+    command: [SPHINX_ARGS, '-Ddepfile=@DEPFILE@', '-Ddepfile_stamp=@OUTPUT1@',
+             '-b', 'texinfo',
+             meson.current_source_dir(), meson.current_build_dir()])
+  alias_target('sphinxinfo', sphinxinfo)
 endif
diff --git a/meson.build b/meson.build
index f4d1ab1096..4156b585ba 100644
--- a/meson.build
+++ b/meson.build
@@ -1218,6 +1218,7 @@ if build_docs
                                               'qemu_confdir': config_host['qemu_confdir']})
 
   texi = {
+    'QEMU': sphinxinfo,
     'qemu-qmp-ref': ['docs/interop/qemu-qmp-ref.texi', qapi_doc_texi, version_texi],
   }
   if 'CONFIG_GUEST_AGENT' in config_host
@@ -1241,7 +1242,8 @@ if build_docs
                       input: input,
                       output: output,
                       install: true,
-                      install_dir: qemu_docdir / 'interop',
+                      install_dir: ext == 'info' ? get_option('infodir') \
+                                                 : qemu_docdir / 'interop',
                       command: cmd + args)
       endforeach
       alias_target(ext, t)
-- 
2.28.0


Re: [PATCH] build: Build and install the info manual.
Posted by no-reply@patchew.org 3 years, 6 months ago
Patchew URL: https://patchew.org/QEMU/20200925024143.26492-1-maxim.cournoyer@gmail.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20200925024143.26492-1-maxim.cournoyer@gmail.com
Subject: [PATCH] build: Build and install the info manual.

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 - [tag update]      patchew/20200924195058.362984-1-danielhb413@gmail.com -> patchew/20200924195058.362984-1-danielhb413@gmail.com
 - [tag update]      patchew/20200925021808.26471-1-colin.xu@intel.com -> patchew/20200925021808.26471-1-colin.xu@intel.com
 * [new tag]         patchew/20200925024143.26492-1-maxim.cournoyer@gmail.com -> patchew/20200925024143.26492-1-maxim.cournoyer@gmail.com
Switched to a new branch 'test'
76a252c build: Build and install the info manual.

=== OUTPUT BEGIN ===
ERROR: Missing Signed-off-by: line(s)

total: 1 errors, 0 warnings, 69 lines checked

Commit 76a252c26b29 (build: Build and install the info manual.) has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20200925024143.26492-1-maxim.cournoyer@gmail.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Re: [PATCH] build: Build and install the info manual.
Posted by no-reply@patchew.org 3 years, 6 months ago
Patchew URL: https://patchew.org/QEMU/20200925024143.26492-1-maxim.cournoyer@gmail.com/



Hi,

This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.

=== TEST SCRIPT BEGIN ===
#! /bin/bash
export ARCH=x86_64
make docker-image-fedora V=1 NETWORK=1
time make docker-test-mingw@fedora J=14 NETWORK=1
=== TEST SCRIPT END ===

Host machine cpu: x86_64
Target machine cpu family: x86
Target machine cpu: x86_64
../src/meson.build:10: WARNING: Module unstable-keyval has no backwards or forwards compatibility and might not exist in future releases.
Program sh found: YES
Program python3 found: YES (/usr/bin/python3)
Configuring ninjatool using configuration
---
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/tmp/qemu-test/src/slirp'
Generating nsis with a custom command
make[1]: warning: jobserver unavailable: using -j1.  Add '+' to parent make rule.
make[1]: Entering directory '/tmp/qemu-test/build'
make[2]: Entering directory '/tmp/qemu-test/src/slirp'
make[2]: Nothing to be done for 'all'.
---
Host machine cpu: i386
Target machine cpu family: x86
Target machine cpu: i386
../src/meson.build:10: WARNING: Module unstable-keyval has no backwards or forwards compatibility and might not exist in future releases.
Program sh found: YES
Program python3 found: YES (/usr/bin/python3)
Configuring ninjatool using configuration
---
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['sudo', '-n', 'docker', 'run', '--rm', '--label', 'com.qemu.instance.uuid=a0b2da4b764d490c9a93a880eb229f51', '-u', '1001', '--security-opt', 'seccomp=unconfined', '-e', 'TARGET_LIST=', '-e', 'EXTRA_CONFIGURE_OPTS=', '-e', 'V=', '-e', 'J=14', '-e', 'DEBUG=', '-e', 'SHOW_ENV=', '-e', 'CCACHE_DIR=/var/tmp/ccache', '-v', '/home/patchew/.cache/qemu-docker-ccache:/var/tmp/ccache:z', '-v', '/var/tmp/patchew-tester-tmp-gg_7lmo7/src/docker-src.2020-09-25-02.32.30.24351:/var/tmp/qemu:z,ro', 'qemu/fedora', '/var/tmp/qemu/run', 'test-mingw']' returned non-zero exit status 2.
filter=--filter=label=com.qemu.instance.uuid=a0b2da4b764d490c9a93a880eb229f51
make[1]: *** [docker-run] Error 1
make[1]: Leaving directory `/var/tmp/patchew-tester-tmp-gg_7lmo7/src'
make: *** [docker-run-test-mingw@fedora] Error 2

real    11m5.824s
user    0m19.716s


The full log is available at
http://patchew.org/logs/20200925024143.26492-1-maxim.cournoyer@gmail.com/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
Re: [PATCH] build: Build and install the info manual.
Posted by Peter Maydell 3 years, 6 months ago
On Fri, 25 Sep 2020 at 07:27, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>
> Take advantage of the Sphinx texinfo backend to generate a QEMU info
> manual.  The texinfo format allows for more structure and info readers
> provide more advanced navigation capabilities compared to manpages
> readers.

Not providing an info manual (or indeed any format other than HTML
and manpages) was a deliberate design choice. The rationale is that
checking that multiple document formats all ended up rendering
correctly is more work than people will in practice put in (as
demonstrated by various errors in the old HTML rendering, for
instance).

That said, maybe other people will disagree with me.

> * configure (infodir): Add the --infodir option, which allows
> configuring the directory under which the info manuals are installed.
> * docs/index.rst: Include the top level documents to prevent
> warnings (treated as errors by sphinx-build).

This isn't the right thing. You should be pointing sphinx-build
at each of the individual manuals (system, interop, etc) and
generating one info file for each. This is because we generate
manuals for each of these including the 'devel' manual, but
we don't want to install 'devel', because it's developer-facing
and not needed by end-users of QEMU. If you do that then you
won't find you have problems with the orphan top level documents.

(We really need to move those orphan docs into the right places
in the manual structure at some point.)

> * docs/meson.build (sphinxinfo): Add new target.

You've forgotten your Signed-off-by: line.

thanks
-- PMM

Re: [PATCH] build: Build and install the info manual.
Posted by Maxim Cournoyer 3 years, 6 months ago
Hello Peter,

Peter Maydell <peter.maydell@linaro.org> writes:

> On Fri, 25 Sep 2020 at 07:27, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>>
>> Take advantage of the Sphinx texinfo backend to generate a QEMU info
>> manual.  The texinfo format allows for more structure and info readers
>> provide more advanced navigation capabilities compared to manpages
>> readers.
>
> Not providing an info manual (or indeed any format other than HTML
> and manpages) was a deliberate design choice. The rationale is that
> checking that multiple document formats all ended up rendering
> correctly is more work than people will in practice put in (as
> demonstrated by various errors in the old HTML rendering, for
> instance).

It seems to me that any problem found in the texinfo rendering produced
by Sphinx would be a bug in Sphinx, which could simply be forwarded
upstream when encountered.  My experience with Sphinx-generated texinfo
manuals is rather good.  The CMake project allows producing an info
manual that way, for example, and the result is at least as usable as
their HTML-based doc (as an avid info user, I'd say more usable, given
most commands are indexed).

>> * configure (infodir): Add the --infodir option, which allows
>> configuring the directory under which the info manuals are installed.
>> * docs/index.rst: Include the top level documents to prevent
>> warnings (treated as errors by sphinx-build).
>
> This isn't the right thing. You should be pointing sphinx-build
> at each of the individual manuals (system, interop, etc) and
> generating one info file for each. This is because we generate
> manuals for each of these including the 'devel' manual, but
> we don't want to install 'devel', because it's developer-facing
> and not needed by end-users of QEMU.

Is this the only reason individual manuals are being generated?  It
makes sense for the manpages (which have their own macros), but
otherwise (for HTML and info) introduces a lot of complexity for not
much gain, in my opinion.  Users not wanting to know about devel
internals can simply skip that section; no harm done.

The individual manuals are all stitched back together using a top
index.html page (derived from index.html.in), anyway.  That'd be better
taken care of by Sphinx automatically upon generating the whole doc
tree, in my opinion, as it seems to be the way it was designed to work
and simplifies things all around.

I initially went the individual manual route and devised complicated
Make macros and got individual info manuals produced, only to find out I
wanted to replicate that index.html.in in the texinfo format (Texinfo is
as navigable as HTML, so having every QEMU manual under the same node as
different sections makes more sense in my opinion).  The individually
produced info manual were also all named 'QEMU Documentation' rather
than with their own more accurate names, because of directives in the
included the top-level conf.py.

> won't find you have problems with the orphan top level documents.
>
> (We really need to move those orphan docs into the right places
> in the manual structure at some point.)

We could simply put them in another section called "Others" as a
starting point.  For this patch we can also choose to ignore them rather
than include them, but if someone went to the length of writing those
docs, I'd argue it was to be read rather than forgotten :-).

>> * docs/meson.build (sphinxinfo): Add new target.
>
> You've forgotten your Signed-off-by: line.

I've added it in my local tree; will wait for your feedback on my above
comments before sending an updated patch.

Thanks for the review,

Maxim

Re: [PATCH] build: Build and install the info manual.
Posted by Peter Maydell 3 years, 6 months ago
On Sat, 26 Sep 2020 at 05:36, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
> > This isn't the right thing. You should be pointing sphinx-build
> > at each of the individual manuals (system, interop, etc) and
> > generating one info file for each. This is because we generate
> > manuals for each of these including the 'devel' manual, but
> > we don't want to install 'devel', because it's developer-facing
> > and not needed by end-users of QEMU.
>
> Is this the only reason individual manuals are being generated?  It
> makes sense for the manpages (which have their own macros), but
> otherwise (for HTML and info) introduces a lot of complexity for not
> much gain, in my opinion.  Users not wanting to know about devel
> internals can simply skip that section; no harm done.

It is the best way we found for getting Sphinx to do what we wanted.
I agree that it would be nicer to have one manual with all the user
facing parts in it, but it is apparently not possible to do that without
shipping the devel docs to users, which we didn't want to do.

thanks
-- PMM

Re: [PATCH] build: Build and install the info manual.
Posted by Maxim Cournoyer 3 years, 6 months ago
Hello Peter,

Peter Maydell <peter.maydell@linaro.org> writes:

> On Sat, 26 Sep 2020 at 05:36, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>> > This isn't the right thing. You should be pointing sphinx-build
>> > at each of the individual manuals (system, interop, etc) and
>> > generating one info file for each. This is because we generate
>> > manuals for each of these including the 'devel' manual, but
>> > we don't want to install 'devel', because it's developer-facing
>> > and not needed by end-users of QEMU.
>>
>> Is this the only reason individual manuals are being generated?  It
>> makes sense for the manpages (which have their own macros), but
>> otherwise (for HTML and info) introduces a lot of complexity for not
>> much gain, in my opinion.  Users not wanting to know about devel
>> internals can simply skip that section; no harm done.
>
> It is the best way we found for getting Sphinx to do what we wanted.
> I agree that it would be nicer to have one manual with all the user
> facing parts in it, but it is apparently not possible to do that without
> shipping the devel docs to users, which we didn't want to do.

I personally don't understand the rationale of hiding the devel section
from users, especially given the kind of users QEMU is likely to attract
(e.g, teksavvy people, perhaps themselves developers that could be
curious peeking into that section to deepen their understanding of
QEMU's architecture and internals).

But if you feel strongly about it, I'd suggest the following, which
excludes the devel section from being built when calling Sphinx from the
top level:

--8<---------------cut here---------------start------------->8---
7 files changed, 13 insertions(+), 4 deletions(-)
docs/conf.py                 | 5 ++++-
docs/cpu-hotplug.rst         | 2 ++
docs/index.rst               | 3 ---
docs/microvm.rst             | 2 ++
docs/pr-manager.rst          | 2 ++
docs/virtio-net-failover.rst | 2 ++
docs/virtio-pmem.rst         | 1 +

modified   docs/conf.py
@@ -116,7 +116,10 @@ language = None
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This patterns also effect to html_static_path and html_extra_path
-exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+exclude_patterns = ['_build',
+                    'Thumbs.db',
+                    '.DS_Store',
+                    'devel/*']

 # The name of the Pygments (syntax highlighting) style to use.
 pygments_style = 'sphinx'
modified   docs/cpu-hotplug.rst
@@ -1,3 +1,5 @@
+:orphan:
+
 ===================
 Virtual CPU hotplug
 ===================
modified   docs/index.rst
@@ -9,12 +9,9 @@ Welcome to QEMU's documentation!
 .. toctree::
    :maxdepth: 2
    :caption: Contents:
-   :glob:

    system/index
    user/index
    tools/index
    interop/index
    specs/index
-   devel/index
-   *
modified   docs/microvm.rst
@@ -1,3 +1,5 @@
+:orphan:
+
 ====================
 microvm Machine Type
 ====================
modified   docs/pr-manager.rst
@@ -1,3 +1,5 @@
+:orphan:
+
 ======================================
 Persistent reservation managers
 ======================================
modified   docs/virtio-net-failover.rst
@@ -1,3 +1,5 @@
+:orphan:
+
 ======================================
 QEMU virtio-net standby (net_failover)
 ======================================
modified   docs/virtio-pmem.rst
@@ -1,3 +1,4 @@
+:orphan:

 ========================
 QEMU virtio pmem
--8<---------------cut here---------------end--------------->8---

With this change, the top level node of "info qemu" reads as:

--8<---------------cut here---------------start------------->8---
QEMU Documentation
******************

     QEMU , Sep 26, 2020

     The QEMU Project Developers

     Copyright (C) 2020, The QEMU Project Developers

* Menu:

* QEMU System Emulation User's Guide::
* QEMU User Mode Emulation User's Guide::
* QEMU Tools Guide::
* QEMU System Emulation Management and Interoperability Guide::
* QEMU System Emulation Guest Hardware Specifications::
* Index::
--8<---------------cut here---------------end--------------->8---

I'd still prefer to include the developer's manual as the last section
of the info manual, if you don't mind, for the reason explained above.

The scheme used above to exclude the devel section could also be used to
simplify building the HTML manual (e.g., build it by calling sphinx on
the top level with a correctly defined index.rst instead of individually
for each of its constituting sections before stitching them back
together with a custom index.html).

What do you think?

Thanks again,

Maxim

Re: [PATCH] build: Build and install the info manual.
Posted by Peter Maydell 3 years, 6 months ago
On Sun, 27 Sep 2020 at 03:21, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
> Peter Maydell <peter.maydell@linaro.org> writes:
> > It is the best way we found for getting Sphinx to do what we wanted.
> > I agree that it would be nicer to have one manual with all the user
> > facing parts in it, but it is apparently not possible to do that without
> > shipping the devel docs to users, which we didn't want to do.
>
> I personally don't understand the rationale of hiding the devel section
> from users, especially given the kind of users QEMU is likely to attract
> (e.g, teksavvy people, perhaps themselves developers that could be
> curious peeking into that section to deepen their understanding of
> QEMU's architecture and internals).

Mostly I think we came to this opinion because
(a) it was how we handled developer docs before -- they tended
to be standalone files in docs/ somewhere, not part of the
old shipped-to-user Texinfo docs
(b) internals docs are much more likely to become quickly outdated:
you almost always want to be looking at the docs for current-git,
not for some older distro-installed QEMU version
(c) sure, some users might want to look at QEMU internals docs,
but they are definitely going to be the minority
(d) the developer docs are rougher in quality overall
(e) you need the source tree anyway if you're interested in
the internals, because so much is not documented, or not in
the rST manuals

That said, we are kind of working against the grain of how
Sphinx wants to be used here, which is usually not a great idea,
and it does result in some awkwardnesses (it would be nice to
have the devel docs on the qemu.org website, for instance).
I asked around on IRC and nobody seemed to be very strongly
against moving to the just-one-manual setup. So maybe we should
do that.

thanks
-- PMM

Re: [PATCH] build: Build and install the info manual.
Posted by Maxim Cournoyer 3 years, 5 months ago
Hello Peter,

Peter Maydell <peter.maydell@linaro.org> writes:

> On Sun, 27 Sep 2020 at 03:21, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

[...]

>> I personally don't understand the rationale of hiding the devel section
>> from users, especially given the kind of users QEMU is likely to attract
>> (e.g, teksavvy people, perhaps themselves developers that could be
>> curious peeking into that section to deepen their understanding of
>> QEMU's architecture and internals).
>
> Mostly I think we came to this opinion because
> (a) it was how we handled developer docs before -- they tended
> to be standalone files in docs/ somewhere, not part of the
> old shipped-to-user Texinfo docs
> (b) internals docs are much more likely to become quickly outdated:
> you almost always want to be looking at the docs for current-git,
> not for some older distro-installed QEMU version
> (c) sure, some users might want to look at QEMU internals docs,
> but they are definitely going to be the minority
> (d) the developer docs are rougher in quality overall
> (e) you need the source tree anyway if you're interested in
> the internals, because so much is not documented, or not in
> the rST manuals
>
> That said, we are kind of working against the grain of how
> Sphinx wants to be used here, which is usually not a great idea,
> and it does result in some awkwardnesses (it would be nice to
> have the devel docs on the qemu.org website, for instance).
> I asked around on IRC and nobody seemed to be very strongly
> against moving to the just-one-manual setup. So maybe we should
> do that.

OK.  Thanks for asking others if they had an opinion about it.  Given
they don't seem to feel strongly about it, I suggest:

1) Keep the developer section as the last section of the overall doc
table of content (as was done in my original patch).

2) Mark the top level couple orphaned .rst as "orphaned" as proposed in
one of my previous reply, since this is what they are.  They will get
built, but won't appear anywhere in the QEMU documentation.

If that sounds reasonable, I'll send an updated patch soon.

Another thing; there was some CI failures with the origin patches I
sent [0]; are they worth investigating or are they unrelated, "known"
failures?

Thanks,

Maxim

[0]  https://patchew.org/QEMU/20200925024143.26492-1-maxim.cournoyer@gmail.com/