When we first converted our documentation to Sphinx, we split it into
multiple manuals (system, interop, tools, etc), which are all built
separately.  The primary driver for this was wanting to be able to
avoid shipping the 'devel' manual to end-users.  However, this is
working against the grain of the way Sphinx wants to be used and
causes some annoyances:
 * Cross-references between documents become much harder or
   possibly impossible
 * There is no single index to the whole documentation
 * Within one manual there's no links or table-of-contents info
   that lets you easily navigate to the others
 * The devel manual doesn't get published on the QEMU website
   (it would be nice to able to refer to it there)

Merely hiding our developer documentation from end users seems like
it's not enough benefit for these costs.  Combine all the
documentation into a single manual (the same way that the readthedocs
site builds it) and install the whole thing.  The previous manual
divisions remain as the new top level sections in the manual.

 * The per-manual conf.py files are no longer needed
 * The man_pages[] specifications previously in each per-manual
   conf.py move to the top level conf.py
 * docs/meson.build logic is simplified as we now only need to run
   Sphinx once for the HTML and then once for the manpages5B
 * The old index.html.in that produced the top-level page with
   links to each manual is no longer needed

Unfortunately this means that we now have to build the HTML
documentation into docs/manual in the build tree rather than directly
into docs/; otherwise it is too awkward to ensure we install only the
built manual and not also the dependency info, stamp file, etc.  The
manual still ends up in the same place in the final installed
directory, but anybody who was consulting documentation from within
the build tree will have to adjust where they're looking.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
I posted this as an RFC back in November. Nobody objected, so here
it is again as a patch intended to go into master. The RFC's initial
"mark stuff as orphan" patch is no longer needed because we cleaned
up all the orphan docs for 5.2. Only other change since the RFC is
a minor rebase to deal with a new manpage whose man_pages line needs
to live in docs/conf.py now.
---
 docs/conf.py         | 40 ++++++++++++++++++++++++++-
 docs/devel/conf.py   | 15 -----------
 docs/index.html.in   | 17 ------------
 docs/interop/conf.py | 26 ------------------
 docs/meson.build     | 64 +++++++++++++++++---------------------------
 docs/specs/conf.py   | 16 -----------
 docs/system/conf.py  | 28 -------------------
 docs/tools/conf.py   | 35 ------------------------
 docs/user/conf.py    | 15 -----------
 9 files changed, 64 insertions(+), 192 deletions(-)
 delete mode 100644 docs/devel/conf.py
 delete mode 100644 docs/index.html.in
 delete mode 100644 docs/interop/conf.py
 delete mode 100644 docs/specs/conf.py
 delete mode 100644 docs/system/conf.py
 delete mode 100644 docs/tools/conf.py
 delete mode 100644 docs/user/conf.py

diff --git a/docs/conf.py b/docs/conf.py
index d40d8ff37ba..14bfeabd2a4 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -224,7 +224,45 @@ latex_documents = [
 
 # -- Options for manual page output ---------------------------------------
 # Individual manual/conf.py can override this to create man pages
-man_pages = []
+man_pages = [
+    ('interop/qemu-ga', 'qemu-ga',
+     'QEMU Guest Agent',
+     ['Michael Roth <mdroth@linux.vnet.ibm.com>'], 8),
+    ('interop/qemu-ga-ref', 'qemu-ga-ref',
+     'QEMU Guest Agent Protocol Reference',
+     [], 7),
+    ('interop/qemu-qmp-ref', 'qemu-qmp-ref',
+     'QEMU QMP Reference Manual',
+     [], 7),
+    ('system/qemu-manpage', 'qemu',
+     'QEMU User Documentation',
+     ['Fabrice Bellard'], 1),
+    ('system/qemu-block-drivers', 'qemu-block-drivers',
+     'QEMU block drivers reference',
+     ['Fabrice Bellard and the QEMU Project developers'], 7),
+    ('system/qemu-cpu-models', 'qemu-cpu-models',
+     'QEMU CPU Models',
+     ['The QEMU Project developers'], 7),
+    ('tools/qemu-img', 'qemu-img',
+     'QEMU disk image utility',
+     ['Fabrice Bellard'], 1),
+    ('tools/qemu-nbd', 'qemu-nbd',
+     'QEMU Disk Network Block Device Server',
+     ['Anthony Liguori <anthony@codemonkey.ws>'], 8),
+    ('tools/qemu-pr-helper', 'qemu-pr-helper',
+     'QEMU persistent reservation helper',
+     [], 8),
+    ('tools/qemu-trace-stap', 'qemu-trace-stap',
+     'QEMU SystemTap trace tool',
+     [], 1),
+    ('tools/virtfs-proxy-helper', 'virtfs-proxy-helper',
+     'QEMU 9p virtfs proxy filesystem helper',
+     ['M. Mohan Kumar'], 1),
+    ('tools/virtiofsd', 'virtiofsd',
+     'QEMU virtio-fs shared file system daemon',
+     ['Stefan Hajnoczi <stefanha@redhat.com>',
+      'Masayoshi Mizuma <m.mizuma@jp.fujitsu.com>'], 1),
+]
 
 # -- Options for Texinfo output -------------------------------------------
 
diff --git a/docs/devel/conf.py b/docs/devel/conf.py
deleted file mode 100644
index 7441f87e7f5..00000000000
--- a/docs/devel/conf.py
+++ /dev/null
@@ -1,15 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# QEMU documentation build configuration file for the 'devel' manual.
-#
-# This includes the top level conf file and then makes any necessary tweaks.
-import sys
-import os
-
-qemu_docdir = os.path.abspath("..")
-parent_config = os.path.join(qemu_docdir, "conf.py")
-exec(compile(open(parent_config, "rb").read(), parent_config, 'exec'))
-
-# This slightly misuses the 'description', but is the best way to get
-# the manual title to appear in the sidebar.
-html_theme_options['description'] = u'Developer''s Guide'
diff --git a/docs/index.html.in b/docs/index.html.in
deleted file mode 100644
index 33db4396ac8..00000000000
--- a/docs/index.html.in
+++ /dev/null
@@ -1,17 +0,0 @@
-<!DOCTYPE html>
-<html lang="en">
-    <head>
-        <meta charset="UTF-8">
-        <title>QEMU @VERSION@ Documentation</title>
-    </head>
-    <body>
-        <h1>QEMU @VERSION@ Documentation</h1>
-        <ul>
-            <li><a href="system/index.html">System Emulation User's Guide</a></li>
-            <li><a href="user/index.html">User Mode Emulation User's Guide</a></li>
-            <li><a href="tools/index.html">Tools Guide</a></li>
-            <li><a href="interop/index.html">System Emulation Management and Interoperability Guide</a></li>
-            <li><a href="specs/index.html">System Emulation Guest Hardware Specifications</a></li>
-        </ul>
-    </body>
-</html>
diff --git a/docs/interop/conf.py b/docs/interop/conf.py
deleted file mode 100644
index 2634ca3410a..00000000000
--- a/docs/interop/conf.py
+++ /dev/null
@@ -1,26 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# QEMU documentation build configuration file for the 'interop' manual.
-#
-# This includes the top level conf file and then makes any necessary tweaks.
-import sys
-import os
-
-qemu_docdir = os.path.abspath("..")
-parent_config = os.path.join(qemu_docdir, "conf.py")
-exec(compile(open(parent_config, "rb").read(), parent_config, 'exec'))
-
-# This slightly misuses the 'description', but is the best way to get
-# the manual title to appear in the sidebar.
-html_theme_options['description'] = u'System Emulation Management and Interoperability Guide'
-
-# One entry per manual page. List of tuples
-# (source start file, name, description, authors, manual section).
-man_pages = [
-    ('qemu-ga', 'qemu-ga', u'QEMU Guest Agent',
-     ['Michael Roth <mdroth@linux.vnet.ibm.com>'], 8),
-    ('qemu-ga-ref', 'qemu-ga-ref', 'QEMU Guest Agent Protocol Reference',
-     [], 7),
-    ('qemu-qmp-ref', 'qemu-qmp-ref', 'QEMU QMP Reference Manual',
-     [], 7),
-]
diff --git a/docs/meson.build b/docs/meson.build
index ebd85d59f98..8471fedc4c0 100644
--- a/docs/meson.build
+++ b/docs/meson.build
@@ -46,71 +46,57 @@ if build_docs
                           meson.source_root() / 'docs/sphinx/qmp_lexer.py',
                           qapi_gen_depends ]
 
-  configure_file(output: 'index.html',
-                 input: files('index.html.in'),
-                 configuration: {'VERSION': meson.project_version()},
-                 install_dir: qemu_docdir)
-  manuals = [ 'devel', 'interop', 'tools', 'specs', 'system', 'user' ]
   man_pages = {
-    'interop' : {
         'qemu-ga.8': (have_tools ? 'man8' : ''),
         'qemu-ga-ref.7': 'man7',
         'qemu-qmp-ref.7': 'man7',
-    },
-    'tools': {
         'qemu-img.1': (have_tools ? 'man1' : ''),
         'qemu-nbd.8': (have_tools ? 'man8' : ''),
         'qemu-pr-helper.8': (have_tools ? 'man8' : ''),
         'qemu-trace-stap.1': (config_host.has_key('CONFIG_TRACE_SYSTEMTAP') ? 'man1' : ''),
         'virtfs-proxy-helper.1': (have_virtfs_proxy_helper ? 'man1' : ''),
         'virtiofsd.1': (have_virtiofsd ? 'man1' : ''),
-    },
-    'system': {
         'qemu.1': 'man1',
         'qemu-block-drivers.7': 'man7',
         'qemu-cpu-models.7': 'man7'
-    },
   }
 
   sphinxdocs = []
   sphinxmans = []
-  foreach manual : manuals
-    private_dir = meson.current_build_dir() / (manual + '.p')
-    output_dir = meson.current_build_dir() / manual
-    input_dir = meson.current_source_dir() / manual
 
-    this_manual = custom_target(manual + ' manual',
+  private_dir = meson.current_build_dir() / 'manual.p'
+  output_dir = meson.current_build_dir() / 'manual'
+  input_dir = meson.current_source_dir()
+
+  this_manual = custom_target('QEMU manual',
                 build_by_default: build_docs,
-                output: [manual + '.stamp'],
-                input: [files('conf.py'), files(manual / 'conf.py')],
-                depfile: manual + '.d',
+                output: 'docs.stamp',
+                input: files('conf.py'),
+                depfile: 'docs.d',
                 depend_files: sphinx_extn_depends,
                 command: [SPHINX_ARGS, '-Ddepfile=@DEPFILE@',
                           '-Ddepfile_stamp=@OUTPUT0@',
                           '-b', 'html', '-d', private_dir,
                           input_dir, output_dir])
-    sphinxdocs += this_manual
-    if build_docs and manual != 'devel'
-      install_subdir(output_dir, install_dir: qemu_docdir)
-    endif
+  sphinxdocs += this_manual
+  install_subdir(output_dir, install_dir: qemu_docdir, strip_directory: true)
 
-    these_man_pages = []
-    install_dirs = []
-    foreach page, section : man_pages.get(manual, {})
-      these_man_pages += page
-      install_dirs += section == '' ? false : get_option('mandir') / section
-    endforeach
-    if these_man_pages.length() > 0
-      sphinxmans += custom_target(manual + ' man pages',
-                         build_by_default: build_docs,
-                         output: these_man_pages,
-                         input: this_manual,
-                         install: build_docs,
-                         install_dir: install_dirs,
-                         command: [SPHINX_ARGS, '-b', 'man', '-d', private_dir,
-                                   input_dir, meson.current_build_dir()])
-    endif
+  these_man_pages = []
+  install_dirs = []
+  foreach page, section : man_pages
+    these_man_pages += page
+    install_dirs += section == '' ? false : get_option('mandir') / section
   endforeach
+
+  sphinxmans += custom_target('QEMU man pages',
+                              build_by_default: build_docs,
+                              output: these_man_pages,
+                              input: this_manual,
+                              install: build_docs,
+                              install_dir: install_dirs,
+                              command: [SPHINX_ARGS, '-b', 'man', '-d', private_dir,
+                                        input_dir, meson.current_build_dir()])
+
   alias_target('sphinxdocs', sphinxdocs)
   alias_target('html', sphinxdocs)
   alias_target('man', sphinxmans)
diff --git a/docs/specs/conf.py b/docs/specs/conf.py
deleted file mode 100644
index 4d56f3ae13c..00000000000
--- a/docs/specs/conf.py
+++ /dev/null
@@ -1,16 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# QEMU documentation build configuration file for the 'specs' manual.
-#
-# This includes the top level conf file and then makes any necessary tweaks.
-import sys
-import os
-
-qemu_docdir = os.path.abspath("..")
-parent_config = os.path.join(qemu_docdir, "conf.py")
-exec(compile(open(parent_config, "rb").read(), parent_config, 'exec'))
-
-# This slightly misuses the 'description', but is the best way to get
-# the manual title to appear in the sidebar.
-html_theme_options['description'] = \
-    u'System Emulation Guest Hardware Specifications'
diff --git a/docs/system/conf.py b/docs/system/conf.py
deleted file mode 100644
index 6251849fefc..00000000000
--- a/docs/system/conf.py
+++ /dev/null
@@ -1,28 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# QEMU documentation build configuration file for the 'system' manual.
-#
-# This includes the top level conf file and then makes any necessary tweaks.
-import sys
-import os
-
-qemu_docdir = os.path.abspath("..")
-parent_config = os.path.join(qemu_docdir, "conf.py")
-exec(compile(open(parent_config, "rb").read(), parent_config, 'exec'))
-
-# This slightly misuses the 'description', but is the best way to get
-# the manual title to appear in the sidebar.
-html_theme_options['description'] = u'System Emulation User''s Guide'
-
-# One entry per manual page. List of tuples
-# (source start file, name, description, authors, manual section).
-man_pages = [
-    ('qemu-manpage', 'qemu', u'QEMU User Documentation',
-     ['Fabrice Bellard'], 1),
-    ('qemu-block-drivers', 'qemu-block-drivers',
-     u'QEMU block drivers reference',
-     ['Fabrice Bellard and the QEMU Project developers'], 7),
-    ('qemu-cpu-models', 'qemu-cpu-models',
-     u'QEMU CPU Models',
-     ['The QEMU Project developers'], 7)
-]
diff --git a/docs/tools/conf.py b/docs/tools/conf.py
deleted file mode 100644
index 4760d36ff2a..00000000000
--- a/docs/tools/conf.py
+++ /dev/null
@@ -1,35 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# QEMU documentation build configuration file for the 'tools' manual.
-#
-# This includes the top level conf file and then makes any necessary tweaks.
-import sys
-import os
-
-qemu_docdir = os.path.abspath("..")
-parent_config = os.path.join(qemu_docdir, "conf.py")
-exec(compile(open(parent_config, "rb").read(), parent_config, 'exec'))
-
-# This slightly misuses the 'description', but is the best way to get
-# the manual title to appear in the sidebar.
-html_theme_options['description'] = \
-    u'Tools Guide'
-
-# One entry per manual page. List of tuples
-# (source start file, name, description, authors, manual section).
-man_pages = [
-    ('qemu-img', 'qemu-img', u'QEMU disk image utility',
-     ['Fabrice Bellard'], 1),
-    ('qemu-nbd', 'qemu-nbd', u'QEMU Disk Network Block Device Server',
-     ['Anthony Liguori <anthony@codemonkey.ws>'], 8),
-    ('qemu-pr-helper', 'qemu-pr-helper', 'QEMU persistent reservation helper',
-     [], 8),
-    ('qemu-trace-stap', 'qemu-trace-stap', u'QEMU SystemTap trace tool',
-     [], 1),
-    ('virtfs-proxy-helper', 'virtfs-proxy-helper',
-     u'QEMU 9p virtfs proxy filesystem helper',
-     ['M. Mohan Kumar'], 1),
-    ('virtiofsd', 'virtiofsd', u'QEMU virtio-fs shared file system daemon',
-     ['Stefan Hajnoczi <stefanha@redhat.com>',
-      'Masayoshi Mizuma <m.mizuma@jp.fujitsu.com>'], 1),
-]
diff --git a/docs/user/conf.py b/docs/user/conf.py
deleted file mode 100644
index 4b09aedd454..00000000000
--- a/docs/user/conf.py
+++ /dev/null
@@ -1,15 +0,0 @@
-# -*- coding: utf-8 -*-
-#
-# QEMU documentation build configuration file for the 'user' manual.
-#
-# This includes the top level conf file and then makes any necessary tweaks.
-import sys
-import os
-
-qemu_docdir = os.path.abspath("..")
-parent_config = os.path.join(qemu_docdir, "conf.py")
-exec(compile(open(parent_config, "rb").read(), parent_config, 'exec'))
-
-# This slightly misuses the 'description', but is the best way to get
-# the manual title to appear in the sidebar.
-html_theme_options['description'] = u'User Mode Emulation User''s Guide'
-- 
2.20.1

On 10/12/20 21:35, Peter Maydell wrote:
> When we first converted our documentation to Sphinx, we split it into
> multiple manuals (system, interop, tools, etc), which are all built
> separately.  The primary driver for this was wanting to be able to
> avoid shipping the 'devel' manual to end-users.  However, this is
> working against the grain of the way Sphinx wants to be used and
> causes some annoyances:
>   * Cross-references between documents become much harder or
>     possibly impossible
>   * There is no single index to the whole documentation
>   * Within one manual there's no links or table-of-contents info
>     that lets you easily navigate to the others
>   * The devel manual doesn't get published on the QEMU website
>     (it would be nice to able to refer to it there)
> 
> Merely hiding our developer documentation from end users seems like
> it's not enough benefit for these costs.  Combine all the
> documentation into a single manual (the same way that the readthedocs
> site builds it) and install the whole thing.  The previous manual
> divisions remain as the new top level sections in the manual.
> 
>   * The per-manual conf.py files are no longer needed
>   * The man_pages[] specifications previously in each per-manual
>     conf.py move to the top level conf.py
>   * docs/meson.build logic is simplified as we now only need to run
>     Sphinx once for the HTML and then once for the manpages5B
>   * The old index.html.in that produced the top-level page with
>     links to each manual is no longer needed
> 
> Unfortunately this means that we now have to build the HTML
> documentation into docs/manual in the build tree rather than directly
> into docs/; otherwise it is too awkward to ensure we install only the
> built manual and not also the dependency info, stamp file, etc.  The
> manual still ends up in the same place in the final installed
> directory, but anybody who was consulting documentation from within
> the build tree will have to adjust where they're looking.
> 
> Signed-off-by: Peter Maydell <peter.maydell@linaro.org>

Sounds good!

I thought I had sent my reviewed-by in November.  If I didn't...


>     man_pages = {
> -    'interop' : {
>           'qemu-ga.8': (have_tools ? 'man8' : ''),
>           'qemu-ga-ref.7': 'man7',
>           'qemu-qmp-ref.7': 'man7',
> -    },
> -    'tools': {
>           'qemu-img.1': (have_tools ? 'man1' : ''),
>           'qemu-nbd.8': (have_tools ? 'man8' : ''),
>           'qemu-pr-helper.8': (have_tools ? 'man8' : ''),
>           'qemu-trace-stap.1': (config_host.has_key('CONFIG_TRACE_SYSTEMTAP') ? 'man1' : ''),
>           'virtfs-proxy-helper.1': (have_virtfs_proxy_helper ? 'man1' : ''),
>           'virtiofsd.1': (have_virtiofsd ? 'man1' : ''),
> -    },
> -    'system': {
>           'qemu.1': 'man1',
>           'qemu-block-drivers.7': 'man7',
>           'qemu-cpu-models.7': 'man7'
> -    },
>     }

... perhaps my only suggestion is to sort these by section and 
secondarily by name.  But no need to repost---or even to do it, in fact.

Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Paolo

On Fri, 11 Dec 2020 at 15:48, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 10/12/20 21:35, Peter Maydell wrote:
> > When we first converted our documentation to Sphinx, we split it into
> > multiple manuals (system, interop, tools, etc), which are all built
> > separately.  The primary driver for this was wanting to be able to
> > avoid shipping the 'devel' manual to end-users.  However, this is
> > working against the grain of the way Sphinx wants to be used and
> > causes some annoyances:
> >   * Cross-references between documents become much harder or
> >     possibly impossible
> >   * There is no single index to the whole documentation
> >   * Within one manual there's no links or table-of-contents info
> >     that lets you easily navigate to the others
> >   * The devel manual doesn't get published on the QEMU website
> >     (it would be nice to able to refer to it there)
> >
> > Merely hiding our developer documentation from end users seems like
> > it's not enough benefit for these costs.  Combine all the
> > documentation into a single manual (the same way that the readthedocs
> > site builds it) and install the whole thing.  The previous manual
> > divisions remain as the new top level sections in the manual.
> >
> >   * The per-manual conf.py files are no longer needed
> >   * The man_pages[] specifications previously in each per-manual
> >     conf.py move to the top level conf.py
> >   * docs/meson.build logic is simplified as we now only need to run
> >     Sphinx once for the HTML and then once for the manpages5B
> >   * The old index.html.in that produced the top-level page with
> >     links to each manual is no longer needed
> >
> > Unfortunately this means that we now have to build the HTML
> > documentation into docs/manual in the build tree rather than directly
> > into docs/; otherwise it is too awkward to ensure we install only the
> > built manual and not also the dependency info, stamp file, etc.  The
> > manual still ends up in the same place in the final installed
> > directory, but anybody who was consulting documentation from within
> > the build tree will have to adjust where they're looking.
> >
> > Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
>
> Sounds good!
>
> I thought I had sent my reviewed-by in November.  If I didn't...

I think you just said it was a good idea and that the meson
stuff didn't have any obvious errors :-)

>
>
> >     man_pages = {
> > -    'interop' : {
> >           'qemu-ga.8': (have_tools ? 'man8' : ''),
> >           'qemu-ga-ref.7': 'man7',
> >           'qemu-qmp-ref.7': 'man7',
> > -    },
> > -    'tools': {
> >           'qemu-img.1': (have_tools ? 'man1' : ''),
> >           'qemu-nbd.8': (have_tools ? 'man8' : ''),
> >           'qemu-pr-helper.8': (have_tools ? 'man8' : ''),
> >           'qemu-trace-stap.1': (config_host.has_key('CONFIG_TRACE_SYSTEMTAP') ? 'man1' : ''),
> >           'virtfs-proxy-helper.1': (have_virtfs_proxy_helper ? 'man1' : ''),
> >           'virtiofsd.1': (have_virtiofsd ? 'man1' : ''),
> > -    },
> > -    'system': {
> >           'qemu.1': 'man1',
> >           'qemu-block-drivers.7': 'man7',
> >           'qemu-cpu-models.7': 'man7'
> > -    },
> >     }
>
> ... perhaps my only suggestion is to sort these by section and
> secondarily by name.  But no need to repost---or even to do it, in fact.

I guess we could as a followup. For this patch I think it's helpful
for review that it's clear that nothing changes except the removal
of the intermediate level of data structure.

> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Thanks!

-- PMM