Makefile: Enable synchronized parallel output when possible

[Qemu-devel] [RFC PATCH] Makefile: Enable synchronized parallel output when possible

Posted by Philippe Mathieu-Daudé 5 years, 10 months ago

The '--output-sync' option is available since GNU make 4.0.

  5.4.1 Output During Parallel Execution

  When running several recipes in parallel the output from each
  recipe appears as soon as it is generated, with the result that
  messages from different recipes may be interspersed, sometimes
  even appearing on the same line. This can make reading the output
  very difficult.

  To avoid this you can use the ‘--output-sync’ (‘-O’) option. This
  option instructs make to save the output from the commands it
  invokes and print it all once the commands are completed.
  Additionally, if there are multiple recursive make invocations
  running in parallel, they will communicate so that only one of
  them is generating output at a time.

  There are four levels of granularity when synchronizing output,
  specified by giving an argument to the option (e.g., ‘-Oline’
  or ‘--output-sync=recurse’).

    none
      This is the default: all output is sent directly as it is
      generated and no synchronization is performed.

    line
      Output from each individual line of the recipe is grouped
      and printed as soon as that line is complete. If a recipe
      consists of multiple lines, they may be interspersed with
      lines from other recipes.

    target
      Output from the entire recipe for each target is grouped and
      printed once the target is complete. This is the default if
      the --output-sync or -O option is given with no argument.

    recurse
      Output from each recursive invocation of make is grouped and
      printed once the recursive invocation is complete.

  Regardless of the mode chosen, the total build time will be the
  same. The only difference is in how the output appears.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
---
Maybe 'line' is better for workstation and 'target/recurse' for
unattended builds.

doc: https://www.gnu.org/software/make/manual/html_node/Parallel-Output.html
---
 Makefile | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/Makefile b/Makefile
index e46f2b625a..e94c687afe 100644
--- a/Makefile
+++ b/Makefile
@@ -367,6 +367,10 @@ DOCS=
 endif
 
 SUBDIR_MAKEFLAGS=$(if $(V),,--no-print-directory --quiet) BUILD_DIR=$(BUILD_DIR)
+ifeq ($(shell test $(firstword $(subst ., ,$(MAKE_VERSION))) -ge 4; echo $$?),0)
+SUBDIR_MAKEFLAGS+=--output-sync=target
+endif
+
 SUBDIR_DEVICES_MAK=$(patsubst %, %/config-devices.mak, $(TARGET_LIST))
 SUBDIR_DEVICES_MAK_DEP=$(patsubst %, %-config-devices.mak.d, $(TARGET_LIST))
 
-- 
2.18.0.rc2

Re: [Qemu-devel] [RFC PATCH] Makefile: Enable synchronized parallel output when possible

Posted by Michael S. Tsirkin 5 years, 10 months ago

On Thu, Jun 21, 2018 at 11:20:35PM -0300, Philippe Mathieu-Daudé wrote:
> The '--output-sync' option is available since GNU make 4.0.
> 
>   5.4.1 Output During Parallel Execution
> 
>   When running several recipes in parallel the output from each
>   recipe appears as soon as it is generated, with the result that
>   messages from different recipes may be interspersed, sometimes
>   even appearing on the same line. This can make reading the output
>   very difficult.
> 
>   To avoid this you can use the ‘--output-sync’ (‘-O’) option. This
>   option instructs make to save the output from the commands it
>   invokes and print it all once the commands are completed.
>   Additionally, if there are multiple recursive make invocations
>   running in parallel, they will communicate so that only one of
>   them is generating output at a time.
> 
>   There are four levels of granularity when synchronizing output,
>   specified by giving an argument to the option (e.g., ‘-Oline’
>   or ‘--output-sync=recurse’).
> 
>     none
>       This is the default: all output is sent directly as it is
>       generated and no synchronization is performed.
> 
>     line
>       Output from each individual line of the recipe is grouped
>       and printed as soon as that line is complete. If a recipe
>       consists of multiple lines, they may be interspersed with
>       lines from other recipes.
> 
>     target
>       Output from the entire recipe for each target is grouped and
>       printed once the target is complete. This is the default if
>       the --output-sync or -O option is given with no argument.
> 
>     recurse
>       Output from each recursive invocation of make is grouped and
>       printed once the recursive invocation is complete.
> 
>   Regardless of the mode chosen, the total build time will be the
>   same. The only difference is in how the output appears.
> 
> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> Maybe 'line' is better for workstation and 'target/recurse' for
> unattended builds.
> 
> doc: https://www.gnu.org/software/make/manual/html_node/Parallel-Output.html
> ---
>  Makefile | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/Makefile b/Makefile
> index e46f2b625a..e94c687afe 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -367,6 +367,10 @@ DOCS=
>  endif
>  
>  SUBDIR_MAKEFLAGS=$(if $(V),,--no-print-directory --quiet) BUILD_DIR=$(BUILD_DIR)
> +ifeq ($(shell test $(firstword $(subst ., ,$(MAKE_VERSION))) -ge 4; echo $$?),0)
> +SUBDIR_MAKEFLAGS+=--output-sync=target
> +endif
> +
>  SUBDIR_DEVICES_MAK=$(patsubst %, %/config-devices.mak, $(TARGET_LIST))
>  SUBDIR_DEVICES_MAK_DEP=$(patsubst %, %-config-devices.mak.d, $(TARGET_LIST))

I'd probably just use -O. If they change the default it will probably be
to a better one.

> -- 
> 2.18.0.rc2

Re: [Qemu-devel] [RFC PATCH] Makefile: Enable synchronized parallel output when possible

Posted by Paolo Bonzini 5 years, 9 months ago

On 22/06/2018 05:22, Michael S. Tsirkin wrote:
> On Thu, Jun 21, 2018 at 11:20:35PM -0300, Philippe Mathieu-Daudé wrote:
>> The '--output-sync' option is available since GNU make 4.0.
>>
>>   5.4.1 Output During Parallel Execution
>>
>>   When running several recipes in parallel the output from each
>>   recipe appears as soon as it is generated, with the result that
>>   messages from different recipes may be interspersed, sometimes
>>   even appearing on the same line. This can make reading the output
>>   very difficult.
>>
>>   To avoid this you can use the ‘--output-sync’ (‘-O’) option. This
>>   option instructs make to save the output from the commands it
>>   invokes and print it all once the commands are completed.
>>   Additionally, if there are multiple recursive make invocations
>>   running in parallel, they will communicate so that only one of
>>   them is generating output at a time.
>>
>>   There are four levels of granularity when synchronizing output,
>>   specified by giving an argument to the option (e.g., ‘-Oline’
>>   or ‘--output-sync=recurse’).
>>
>>     none
>>       This is the default: all output is sent directly as it is
>>       generated and no synchronization is performed.
>>
>>     line
>>       Output from each individual line of the recipe is grouped
>>       and printed as soon as that line is complete. If a recipe
>>       consists of multiple lines, they may be interspersed with
>>       lines from other recipes.
>>
>>     target
>>       Output from the entire recipe for each target is grouped and
>>       printed once the target is complete. This is the default if
>>       the --output-sync or -O option is given with no argument.
>>
>>     recurse
>>       Output from each recursive invocation of make is grouped and
>>       printed once the recursive invocation is complete.
>>
>>   Regardless of the mode chosen, the total build time will be the
>>   same. The only difference is in how the output appears.
>>
>> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
>> ---
>> Maybe 'line' is better for workstation and 'target/recurse' for
>> unattended builds.

For long-running tasks, such as Docker builds or unit tests, -Otarget
might make it harder to find out what's going on?  Maybe "-Oline" is
better overall, but before you send v2, what if the user is specifying
-O him/herself?

(Now that I've been taught, I'm adding "export GNUMAKEFLAGS=-Oline" to
my bash config file. :))

Paolo


>> doc: https://www.gnu.org/software/make/manual/html_node/Parallel-Output.html
>> ---
>>  Makefile | 4 ++++
>>  1 file changed, 4 insertions(+)
>>
>> diff --git a/Makefile b/Makefile
>> index e46f2b625a..e94c687afe 100644
>> --- a/Makefile
>> +++ b/Makefile
>> @@ -367,6 +367,10 @@ DOCS=
>>  endif
>>  
>>  SUBDIR_MAKEFLAGS=$(if $(V),,--no-print-directory --quiet) BUILD_DIR=$(BUILD_DIR)
>> +ifeq ($(shell test $(firstword $(subst ., ,$(MAKE_VERSION))) -ge 4; echo $$?),0)
>> +SUBDIR_MAKEFLAGS+=--output-sync=target
>> +endif
>> +
>>  SUBDIR_DEVICES_MAK=$(patsubst %, %/config-devices.mak, $(TARGET_LIST))
>>  SUBDIR_DEVICES_MAK_DEP=$(patsubst %, %-config-devices.mak.d, $(TARGET_LIST))
> 
> I'd probably just use -O. If they change the default it will probably be
> to a better one.

Re: [Qemu-devel] [RFC PATCH] Makefile: Enable synchronized parallel output when possible

Posted by Peter Maydell 5 years, 9 months ago

On 22 June 2018 at 03:20, Philippe Mathieu-Daudé <f4bug@amsat.org> wrote:
> The '--output-sync' option is available since GNU make 4.0.

> Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
> ---
> Maybe 'line' is better for workstation and 'target/recurse' for
> unattended builds.
>
> doc: https://www.gnu.org/software/make/manual/html_node/Parallel-Output.html
> ---
>  Makefile | 4 ++++
>  1 file changed, 4 insertions(+)
>
> diff --git a/Makefile b/Makefile
> index e46f2b625a..e94c687afe 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -367,6 +367,10 @@ DOCS=
>  endif
>
>  SUBDIR_MAKEFLAGS=$(if $(V),,--no-print-directory --quiet) BUILD_DIR=$(BUILD_DIR)
> +ifeq ($(shell test $(firstword $(subst ., ,$(MAKE_VERSION))) -ge 4; echo $$?),0)

A comment here would help in figuring out what version check
you were intending to perform. Also surely we don't need to
go out to the shell just to do a comparison...

Make seems to advertise --output-sync support as part of its
.FEATURES variable, so probably the most reliable way to do
this test is to check whether .FEATURES contains "output-sync".

> +SUBDIR_MAKEFLAGS+=--output-sync=target
> +endif
> +
>  SUBDIR_DEVICES_MAK=$(patsubst %, %/config-devices.mak, $(TARGET_LIST))
>  SUBDIR_DEVICES_MAK_DEP=$(patsubst %, %-config-devices.mak.d, $(TARGET_LIST))

This only affects our sub-makes, though, right? So the user
would have to specify it on the command line anyway so that
it applies to the top level make process...

thanks
-- PMM