docs/meson.build | 12 ++++++++++++ docs/sphinx/fakedbusdoc.py | 5 +++++ docs/sphinx/qmp_lexer.py | 5 +++++ 3 files changed, 22 insertions(+)
Save a bit of build time by passing the number of jobs option to
sphinx.
To avoid warnings from sphinx, alter our plugins to inform whether
they support parallelism. The two plugins touched are quite simple and
I don't see anything that would indicate they do not support being
called in parallel, so return True for both reads and writes.
before:
$ time make -j16 man html
...
[1/2] Generating docs/QEMU manual with a custom command
[2/2] Generating docs/QEMU man pages with a custom command
real 0m49.770s
user 0m49.425s
sys 0m0.716s
after:
$ time make -j16 man html
...
[1/2] Generating docs/QEMU manual with a custom command (wrapped by meson because command contains newlines)
[2/2] Generating docs/QEMU man pages with a custom command (wrapped by meson because command contains newlines)
real 0m30.153s
user 1m5.945s
sys 0m2.440s
Signed-off-by: Fabiano Rosas <farosas@suse.de>
---
docs/meson.build | 12 ++++++++++++
docs/sphinx/fakedbusdoc.py | 5 +++++
docs/sphinx/qmp_lexer.py | 5 +++++
3 files changed, 22 insertions(+)
diff --git a/docs/meson.build b/docs/meson.build
index f220800e3e..9e4bed6fa0 100644
--- a/docs/meson.build
+++ b/docs/meson.build
@@ -10,6 +10,18 @@ if sphinx_build.found()
SPHINX_ARGS += [ '-W', '-Dkerneldoc_werror=1' ]
endif
+ sphinx_version = run_command(SPHINX_ARGS + ['--version'],
+ check: false).stdout().split()[1]
+ if sphinx_version.version_compare('>=5.1.2')
+ SPHINX_ARGS += ['-j', 'auto']
+ else
+ nproc = find_program('nproc')
+ if nproc.found()
+ jobs = run_command(nproc, check:false).stdout()
+ SPHINX_ARGS += ['-j', jobs]
+ endif
+ endif
+
# This is a bit awkward but works: create a trivial document and
# try to run it with our configuration file (which enforces a
# version requirement). This will fail if sphinx-build is too old.
diff --git a/docs/sphinx/fakedbusdoc.py b/docs/sphinx/fakedbusdoc.py
index d2c5079046..2d2e6ef640 100644
--- a/docs/sphinx/fakedbusdoc.py
+++ b/docs/sphinx/fakedbusdoc.py
@@ -23,3 +23,8 @@ def run(self):
def setup(app: Sphinx) -> Dict[str, Any]:
"""Register a fake dbus-doc directive with Sphinx"""
app.add_directive("dbus-doc", FakeDBusDocDirective)
+
+ return dict(
+ parallel_read_safe = True,
+ parallel_write_safe = True
+ )
diff --git a/docs/sphinx/qmp_lexer.py b/docs/sphinx/qmp_lexer.py
index f7e4c0e198..a59de8a079 100644
--- a/docs/sphinx/qmp_lexer.py
+++ b/docs/sphinx/qmp_lexer.py
@@ -41,3 +41,8 @@ def setup(sphinx):
sphinx.add_lexer('QMP', QMPExampleLexer)
except errors.VersionRequirementError:
sphinx.add_lexer('QMP', QMPExampleLexer())
+
+ return dict(
+ parallel_read_safe = True,
+ parallel_write_safe = True
+ )
--
2.35.3On 26/04/2023 18.03, Fabiano Rosas wrote:
> Save a bit of build time by passing the number of jobs option to
> sphinx.
>
> To avoid warnings from sphinx, alter our plugins to inform whether
> they support parallelism. The two plugins touched are quite simple and
> I don't see anything that would indicate they do not support being
> called in parallel, so return True for both reads and writes.
>
> before:
> $ time make -j16 man html
> ...
> [1/2] Generating docs/QEMU manual with a custom command
> [2/2] Generating docs/QEMU man pages with a custom command
>
> real 0m49.770s
> user 0m49.425s
> sys 0m0.716s
>
> after:
> $ time make -j16 man html
> ...
> [1/2] Generating docs/QEMU manual with a custom command (wrapped by meson because command contains newlines)
> [2/2] Generating docs/QEMU man pages with a custom command (wrapped by meson because command contains newlines)
> real 0m30.153s
> user 1m5.945s
> sys 0m2.440s
>
> Signed-off-by: Fabiano Rosas <farosas@suse.de>
> ---
> docs/meson.build | 12 ++++++++++++
> docs/sphinx/fakedbusdoc.py | 5 +++++
> docs/sphinx/qmp_lexer.py | 5 +++++
> 3 files changed, 22 insertions(+)
>
> diff --git a/docs/meson.build b/docs/meson.build
> index f220800e3e..9e4bed6fa0 100644
> --- a/docs/meson.build
> +++ b/docs/meson.build
> @@ -10,6 +10,18 @@ if sphinx_build.found()
> SPHINX_ARGS += [ '-W', '-Dkerneldoc_werror=1' ]
> endif
>
> + sphinx_version = run_command(SPHINX_ARGS + ['--version'],
> + check: false).stdout().split()[1]
> + if sphinx_version.version_compare('>=5.1.2')
> + SPHINX_ARGS += ['-j', 'auto']
> + else
> + nproc = find_program('nproc')
> + if nproc.found()
> + jobs = run_command(nproc, check:false).stdout()
> + SPHINX_ARGS += ['-j', jobs]
> + endif
> + endif
Wouldn't it be better to use the value from "make -jXX" instead of always
running with the maximum number of processors here?
Thomas
On Thu, Apr 27, 2023 at 02:24:59PM +0200, Thomas Huth wrote:
> On 26/04/2023 18.03, Fabiano Rosas wrote:
> > Save a bit of build time by passing the number of jobs option to
> > sphinx.
> >
> > To avoid warnings from sphinx, alter our plugins to inform whether
> > they support parallelism. The two plugins touched are quite simple and
> > I don't see anything that would indicate they do not support being
> > called in parallel, so return True for both reads and writes.
> >
> > before:
> > $ time make -j16 man html
> > ...
> > [1/2] Generating docs/QEMU manual with a custom command
> > [2/2] Generating docs/QEMU man pages with a custom command
> >
> > real 0m49.770s
> > user 0m49.425s
> > sys 0m0.716s
> >
> > after:
> > $ time make -j16 man html
> > ...
> > [1/2] Generating docs/QEMU manual with a custom command (wrapped by meson because command contains newlines)
> > [2/2] Generating docs/QEMU man pages with a custom command (wrapped by meson because command contains newlines)
> > real 0m30.153s
> > user 1m5.945s
> > sys 0m2.440s
> >
> > Signed-off-by: Fabiano Rosas <farosas@suse.de>
> > ---
> > docs/meson.build | 12 ++++++++++++
> > docs/sphinx/fakedbusdoc.py | 5 +++++
> > docs/sphinx/qmp_lexer.py | 5 +++++
> > 3 files changed, 22 insertions(+)
> >
> > diff --git a/docs/meson.build b/docs/meson.build
> > index f220800e3e..9e4bed6fa0 100644
> > --- a/docs/meson.build
> > +++ b/docs/meson.build
> > @@ -10,6 +10,18 @@ if sphinx_build.found()
> > SPHINX_ARGS += [ '-W', '-Dkerneldoc_werror=1' ]
> > endif
> > + sphinx_version = run_command(SPHINX_ARGS + ['--version'],
> > + check: false).stdout().split()[1]
> > + if sphinx_version.version_compare('>=5.1.2')
> > + SPHINX_ARGS += ['-j', 'auto']
> > + else
> > + nproc = find_program('nproc')
> > + if nproc.found()
> > + jobs = run_command(nproc, check:false).stdout()
> > + SPHINX_ARGS += ['-j', jobs]
> > + endif
> > + endif
>
> Wouldn't it be better to use the value from "make -jXX" instead of always
> running with the maximum number of processors here?
IIUC, this is not exposed by meson.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, Apr 26, 2023 at 01:03:46PM -0300, Fabiano Rosas wrote:
> Save a bit of build time by passing the number of jobs option to
> sphinx.
>
> To avoid warnings from sphinx, alter our plugins to inform whether
> they support parallelism. The two plugins touched are quite simple and
> I don't see anything that would indicate they do not support being
> called in parallel, so return True for both reads and writes.
>
> before:
> $ time make -j16 man html
> ...
> [1/2] Generating docs/QEMU manual with a custom command
> [2/2] Generating docs/QEMU man pages with a custom command
>
> real 0m49.770s
> user 0m49.425s
> sys 0m0.716s
>
> after:
> $ time make -j16 man html
> ...
> [1/2] Generating docs/QEMU manual with a custom command (wrapped by meson because command contains newlines)
> [2/2] Generating docs/QEMU man pages with a custom command (wrapped by meson because command contains newlines)
> real 0m30.153s
> user 1m5.945s
> sys 0m2.440s
When I test this on Fedora 38, the docs build hangs forever.
I ran sphinx directly and see it prints a traceback and then
fails to exit after this error
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/sphinx/util/parallel.py", line 105, in join
if not self._join_one():
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/sphinx/util/parallel.py", line 129, in _join_one
self._result_funcs.pop(tid)(self._args.pop(tid), result)
File "/usr/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 478, in merge
self.env.merge_info_from(docs, env, self.app)
File "/usr/lib/python3.11/site-packages/sphinx/environment/__init__.py", line 366, in merge_info_from
domain.merge_domaindata(docnames, other.domaindata[domainname])
File "/usr/lib/python3.11/site-packages/sphinx/domains/__init__.py", line 295, in merge_domaindata
raise NotImplementedError('merge_domaindata must be implemented in %s '
NotImplementedError: merge_domaindata must be implemented in <class 'dbusdomain.DBusDomain'> to be able to do parallel builds!
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/sphinx/cmd/build.py", line 281, in build_main
app.build(args.force_all, args.filenames)
File "/usr/lib/python3.11/site-packages/sphinx/application.py", line 347, in build
self.builder.build_update()
File "/usr/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 310, in build_update
self.build(to_build,
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Daniel P. Berrangé <berrange@redhat.com> writes:
> On Wed, Apr 26, 2023 at 01:03:46PM -0300, Fabiano Rosas wrote:
>> Save a bit of build time by passing the number of jobs option to
>> sphinx.
>>
>> To avoid warnings from sphinx, alter our plugins to inform whether
>> they support parallelism. The two plugins touched are quite simple and
>> I don't see anything that would indicate they do not support being
>> called in parallel, so return True for both reads and writes.
>>
>> before:
>> $ time make -j16 man html
>> ...
>> [1/2] Generating docs/QEMU manual with a custom command
>> [2/2] Generating docs/QEMU man pages with a custom command
>>
>> real 0m49.770s
>> user 0m49.425s
>> sys 0m0.716s
>>
>> after:
>> $ time make -j16 man html
>> ...
>> [1/2] Generating docs/QEMU manual with a custom command (wrapped by meson because command contains newlines)
>> [2/2] Generating docs/QEMU man pages with a custom command (wrapped by meson because command contains newlines)
>> real 0m30.153s
>> user 1m5.945s
>> sys 0m2.440s
>
> When I test this on Fedora 38, the docs build hangs forever.
>
> I ran sphinx directly and see it prints a traceback and then
> fails to exit after this error
>
> Traceback (most recent call last):
> File "/usr/lib/python3.11/site-packages/sphinx/util/parallel.py", line 105, in join
> if not self._join_one():
> ^^^^^^^^^^^^^^^^
> File "/usr/lib/python3.11/site-packages/sphinx/util/parallel.py", line 129, in _join_one
> self._result_funcs.pop(tid)(self._args.pop(tid), result)
> File "/usr/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 478, in merge
> self.env.merge_info_from(docs, env, self.app)
> File "/usr/lib/python3.11/site-packages/sphinx/environment/__init__.py", line 366, in merge_info_from
> domain.merge_domaindata(docnames, other.domaindata[domainname])
> File "/usr/lib/python3.11/site-packages/sphinx/domains/__init__.py", line 295, in merge_domaindata
> raise NotImplementedError('merge_domaindata must be implemented in %s '
> NotImplementedError: merge_domaindata must be implemented in <class 'dbusdomain.DBusDomain'> to be able to do parallel builds!
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> File "/usr/lib/python3.11/site-packages/sphinx/cmd/build.py", line 281, in build_main
> app.build(args.force_all, args.filenames)
> File "/usr/lib/python3.11/site-packages/sphinx/application.py", line 347, in build
> self.builder.build_update()
> File "/usr/lib/python3.11/site-packages/sphinx/builders/__init__.py", line 310, in build_update
> self.build(to_build,
>
Ah it seems the DBus plugin is only in effect when using sphinx > 4.
I'll try to reproduce and see what it takes to implement this
merge_domaindata. Thanks for testing.
© 2016 - 2024 Red Hat, Inc.