Add a new make macro for creating <file> from <file>.in at build
time. To be used like this:

$(foreach file,$(IN_FILES),$(eval $(call apply-build-vars,$(file))))

This can be used instead of the current approach to perform the similar
step for file.in during ./configure.

This will avoid having to run ./configure just because of modifying a
file depending on a variable set by configure.

Prepare to have multiple files as source for the replacement patterns.

Signed-off-by: Juergen Gross <jgross@suse.com>
---
V2:
- don't use pattern rule, but create explicit dependency in macro,
  don't require to rename source files (Jan Beulich, Andrew Cooper)
---
 Config.mk | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/Config.mk b/Config.mk
index e1556dfbfa..d21d67945a 100644
--- a/Config.mk
+++ b/Config.mk
@@ -159,6 +159,19 @@ define move-if-changed
 	if ! cmp -s $(1) $(2); then mv -f $(1) $(2); else rm -f $(1); fi
 endef
 
+PATH_FILES := Paths
+INC_FILES := $(foreach f, $(PATH_FILES), $(XEN_ROOT)/config/$(f).mk)
+
+include $(INC_FILES)
+
+BUILD_MAKE_VARS := $(foreach f, $(PATH_FILES), $(shell awk '$$2 == ":=" { print $$1; }' $(XEN_ROOT)/config/$(f).mk.in))
+
+# Replace @xxx@ markers in $(1).in with $(xxx) variable contents, write to $(1)
+define apply-build-vars
+ $(1): $(1).in
+	sed $$(foreach v, $$(BUILD_MAKE_VARS), -e 's#@$$(v)@#$$($$(v))#g') <$$< >$$@
+endef
+
 CFLAGS += -fno-strict-aliasing
 
 CFLAGS += -std=gnu99
-- 
2.51.0

On 14.11.2025 12:32, Juergen Gross wrote:
> Add a new make macro for creating <file> from <file>.in at build
> time. To be used like this:
> 
> $(foreach file,$(IN_FILES),$(eval $(call apply-build-vars,$(file))))
> 
> This can be used instead of the current approach to perform the similar
> step for file.in during ./configure.
> 
> This will avoid having to run ./configure just because of modifying a
> file depending on a variable set by configure.
> 
> Prepare to have multiple files as source for the replacement patterns.
> 
> Signed-off-by: Juergen Gross <jgross@suse.com>
> ---
> V2:
> - don't use pattern rule, but create explicit dependency in macro,
>   don't require to rename source files (Jan Beulich, Andrew Cooper)
> ---
>  Config.mk | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/Config.mk b/Config.mk
> index e1556dfbfa..d21d67945a 100644
> --- a/Config.mk
> +++ b/Config.mk
> @@ -159,6 +159,19 @@ define move-if-changed
>  	if ! cmp -s $(1) $(2); then mv -f $(1) $(2); else rm -f $(1); fi
>  endef
>  
> +PATH_FILES := Paths
> +INC_FILES := $(foreach f, $(PATH_FILES), $(XEN_ROOT)/config/$(f).mk)
> +
> +include $(INC_FILES)

Is any of the above part of introducing the macro? "Paths" is already a
specific case of holding patterns that want replacing. In turn ...

> +BUILD_MAKE_VARS := $(foreach f, $(PATH_FILES), $(shell awk '$$2 == ":=" { print $$1; }' $(XEN_ROOT)/config/$(f).mk.in))

... it's not quite clear to me how it can be $(PATH_FILES) here.

> +# Replace @xxx@ markers in $(1).in with $(xxx) variable contents, write to $(1)
> +define apply-build-vars
> + $(1): $(1).in

This being indented by a space looks a little unusual.

Jan

> +	sed $$(foreach v, $$(BUILD_MAKE_VARS), -e 's#@$$(v)@#$$($$(v))#g') <$$< >$$@
> +endef
> +
>  CFLAGS += -fno-strict-aliasing
>  
>  CFLAGS += -std=gnu99

On 17.11.25 13:29, Jan Beulich wrote:
> On 14.11.2025 12:32, Juergen Gross wrote:
>> Add a new make macro for creating <file> from <file>.in at build
>> time. To be used like this:
>>
>> $(foreach file,$(IN_FILES),$(eval $(call apply-build-vars,$(file))))
>>
>> This can be used instead of the current approach to perform the similar
>> step for file.in during ./configure.
>>
>> This will avoid having to run ./configure just because of modifying a
>> file depending on a variable set by configure.
>>
>> Prepare to have multiple files as source for the replacement patterns.
>>
>> Signed-off-by: Juergen Gross <jgross@suse.com>
>> ---
>> V2:
>> - don't use pattern rule, but create explicit dependency in macro,
>>    don't require to rename source files (Jan Beulich, Andrew Cooper)
>> ---
>>   Config.mk | 13 +++++++++++++
>>   1 file changed, 13 insertions(+)
>>
>> diff --git a/Config.mk b/Config.mk
>> index e1556dfbfa..d21d67945a 100644
>> --- a/Config.mk
>> +++ b/Config.mk
>> @@ -159,6 +159,19 @@ define move-if-changed
>>   	if ! cmp -s $(1) $(2); then mv -f $(1) $(2); else rm -f $(1); fi
>>   endef
>>   
>> +PATH_FILES := Paths
>> +INC_FILES := $(foreach f, $(PATH_FILES), $(XEN_ROOT)/config/$(f).mk)
>> +
>> +include $(INC_FILES)
> 
> Is any of the above part of introducing the macro? "Paths" is already a
> specific case of holding patterns that want replacing. In turn ...
> 
>> +BUILD_MAKE_VARS := $(foreach f, $(PATH_FILES), $(shell awk '$$2 == ":=" { print $$1; }' $(XEN_ROOT)/config/$(f).mk.in))
> 
> ... it's not quite clear to me how it can be $(PATH_FILES) here.

See patch 4.

PATH_FILES is specifying the .mk files containing the marker definitions.
I need the ability to have multiple such files in order to be able to let
tools/configure build its own definitions.

> 
>> +# Replace @xxx@ markers in $(1).in with $(xxx) variable contents, write to $(1)
>> +define apply-build-vars
>> + $(1): $(1).in
> 
> This being indented by a space looks a little unusual.

There are instances of that in tools/Rules.mk and stunbdom/Makefile.

OTOH it might have been me introducing those. :-)


Juergen

On 17.11.2025 13:48, Jürgen Groß wrote:
> On 17.11.25 13:29, Jan Beulich wrote:
>> On 14.11.2025 12:32, Juergen Gross wrote:
>>> --- a/Config.mk
>>> +++ b/Config.mk
>>> @@ -159,6 +159,19 @@ define move-if-changed
>>>   	if ! cmp -s $(1) $(2); then mv -f $(1) $(2); else rm -f $(1); fi
>>>   endef
>>>   
>>> +PATH_FILES := Paths
>>> +INC_FILES := $(foreach f, $(PATH_FILES), $(XEN_ROOT)/config/$(f).mk)
>>> +
>>> +include $(INC_FILES)
>>
>> Is any of the above part of introducing the macro? "Paths" is already a
>> specific case of holding patterns that want replacing. In turn ...
>>
>>> +BUILD_MAKE_VARS := $(foreach f, $(PATH_FILES), $(shell awk '$$2 == ":=" { print $$1; }' $(XEN_ROOT)/config/$(f).mk.in))
>>
>> ... it's not quite clear to me how it can be $(PATH_FILES) here.
> 
> See patch 4.
> 
> PATH_FILES is specifying the .mk files containing the marker definitions.
> I need the ability to have multiple such files in order to be able to let
> tools/configure build its own definitions.

That's a good example - why would that affect the stubdom/ part of the tree?
Imo what pattern file(s) to use wants leaving to the invokee of the macro,
not pinning down globally for everyone.

Jan

On 17.11.25 13:54, Jan Beulich wrote:
> On 17.11.2025 13:48, Jürgen Groß wrote:
>> On 17.11.25 13:29, Jan Beulich wrote:
>>> On 14.11.2025 12:32, Juergen Gross wrote:
>>>> --- a/Config.mk
>>>> +++ b/Config.mk
>>>> @@ -159,6 +159,19 @@ define move-if-changed
>>>>    	if ! cmp -s $(1) $(2); then mv -f $(1) $(2); else rm -f $(1); fi
>>>>    endef
>>>>    
>>>> +PATH_FILES := Paths
>>>> +INC_FILES := $(foreach f, $(PATH_FILES), $(XEN_ROOT)/config/$(f).mk)
>>>> +
>>>> +include $(INC_FILES)
>>>
>>> Is any of the above part of introducing the macro? "Paths" is already a
>>> specific case of holding patterns that want replacing. In turn ...
>>>
>>>> +BUILD_MAKE_VARS := $(foreach f, $(PATH_FILES), $(shell awk '$$2 == ":=" { print $$1; }' $(XEN_ROOT)/config/$(f).mk.in))
>>>
>>> ... it's not quite clear to me how it can be $(PATH_FILES) here.
>>
>> See patch 4.
>>
>> PATH_FILES is specifying the .mk files containing the marker definitions.
>> I need the ability to have multiple such files in order to be able to let
>> tools/configure build its own definitions.
> 
> That's a good example - why would that affect the stubdom/ part of the tree?
> Imo what pattern file(s) to use wants leaving to the invokee of the macro,
> not pinning down globally for everyone.

Yes, I could add the tools specific marker file in the use cases under tools.

The question is whether adding it to 6 Makefiles is really worth that
optimization, especially as only building the man files would be effected
right now (which could change in future, of course).


Juergen

On 17.11.2025 14:10, Jürgen Groß wrote:
> On 17.11.25 13:54, Jan Beulich wrote:
>> On 17.11.2025 13:48, Jürgen Groß wrote:
>>> On 17.11.25 13:29, Jan Beulich wrote:
>>>> On 14.11.2025 12:32, Juergen Gross wrote:
>>>>> --- a/Config.mk
>>>>> +++ b/Config.mk
>>>>> @@ -159,6 +159,19 @@ define move-if-changed
>>>>>    	if ! cmp -s $(1) $(2); then mv -f $(1) $(2); else rm -f $(1); fi
>>>>>    endef
>>>>>    
>>>>> +PATH_FILES := Paths
>>>>> +INC_FILES := $(foreach f, $(PATH_FILES), $(XEN_ROOT)/config/$(f).mk)
>>>>> +
>>>>> +include $(INC_FILES)
>>>>
>>>> Is any of the above part of introducing the macro? "Paths" is already a
>>>> specific case of holding patterns that want replacing. In turn ...
>>>>
>>>>> +BUILD_MAKE_VARS := $(foreach f, $(PATH_FILES), $(shell awk '$$2 == ":=" { print $$1; }' $(XEN_ROOT)/config/$(f).mk.in))
>>>>
>>>> ... it's not quite clear to me how it can be $(PATH_FILES) here.
>>>
>>> See patch 4.
>>>
>>> PATH_FILES is specifying the .mk files containing the marker definitions.
>>> I need the ability to have multiple such files in order to be able to let
>>> tools/configure build its own definitions.
>>
>> That's a good example - why would that affect the stubdom/ part of the tree?
>> Imo what pattern file(s) to use wants leaving to the invokee of the macro,
>> not pinning down globally for everyone.
> 
> Yes, I could add the tools specific marker file in the use cases under tools.
> 
> The question is whether adding it to 6 Makefiles is really worth that
> optimization, especially as only building the man files would be effected
> right now (which could change in future, of course).

Sticking to Paths.mk (which sits in the global config/ subtree anyway) might
be okay. The new (tools/ specific aiui) file you add later doesn't really
belong there, though. Therefore the macro may want to be constructed such
that it can be used both ways.

Jan

On 14/11/2025 11:32 am, Juergen Gross wrote:
> diff --git a/Config.mk b/Config.mk
> index e1556dfbfa..d21d67945a 100644
> --- a/Config.mk
> +++ b/Config.mk
> @@ -159,6 +159,19 @@ define move-if-changed
>  	if ! cmp -s $(1) $(2); then mv -f $(1) $(2); else rm -f $(1); fi
>  endef
>  
> +PATH_FILES := Paths
> +INC_FILES := $(foreach f, $(PATH_FILES), $(XEN_ROOT)/config/$(f).mk)
> +
> +include $(INC_FILES)
> +
> +BUILD_MAKE_VARS := $(foreach f, $(PATH_FILES), $(shell awk '$$2 == ":=" { print $$1; }' $(XEN_ROOT)/config/$(f).mk.in))
> +
> +# Replace @xxx@ markers in $(1).in with $(xxx) variable contents, write to $(1)
> +define apply-build-vars
> + $(1): $(1).in
> +	sed $$(foreach v, $$(BUILD_MAKE_VARS), -e 's#@$$(v)@#$$($$(v))#g') <$$< >$$@
> +endef

Shouldn't this write to a tmp file, and use move-if-changed?  Most of
the time the markers won't have changed, and we'll want to short circuit
dependent rules.

~Andrew

On 14.11.25 12:42, Andrew Cooper wrote:
> On 14/11/2025 11:32 am, Juergen Gross wrote:
>> diff --git a/Config.mk b/Config.mk
>> index e1556dfbfa..d21d67945a 100644
>> --- a/Config.mk
>> +++ b/Config.mk
>> @@ -159,6 +159,19 @@ define move-if-changed
>>   	if ! cmp -s $(1) $(2); then mv -f $(1) $(2); else rm -f $(1); fi
>>   endef
>>   
>> +PATH_FILES := Paths
>> +INC_FILES := $(foreach f, $(PATH_FILES), $(XEN_ROOT)/config/$(f).mk)
>> +
>> +include $(INC_FILES)
>> +
>> +BUILD_MAKE_VARS := $(foreach f, $(PATH_FILES), $(shell awk '$$2 == ":=" { print $$1; }' $(XEN_ROOT)/config/$(f).mk.in))
>> +
>> +# Replace @xxx@ markers in $(1).in with $(xxx) variable contents, write to $(1)
>> +define apply-build-vars
>> + $(1): $(1).in
>> +	sed $$(foreach v, $$(BUILD_MAKE_VARS), -e 's#@$$(v)@#$$($$(v))#g') <$$< >$$@
>> +endef
> 
> Shouldn't this write to a tmp file, and use move-if-changed?  Most of
> the time the markers won't have changed, and we'll want to short circuit
> dependent rules.

I can see this being an advantage when e.g. generating header files, as
those being generated again would potentially cause lots of rebuilds.

In this case I can hardly see any case where make wouldn't do the right
thing already. Either the *.in file is newer than the generated file due
to a git update or a manual edit, so make will regenerate the target (and
this is what we want), or the *.in file hasn't changed, so make won't
regenerate the file as it is newer than the *.in file already.

Or did I miss some aspect?

What IS needed is probably a dependency on $(PATH_FILES) in case someone
did a new ./configure call without a make distclean.


Juergen

On 14.11.2025 13:54, Jürgen Groß wrote:
> On 14.11.25 12:42, Andrew Cooper wrote:
>> On 14/11/2025 11:32 am, Juergen Gross wrote:
>>> diff --git a/Config.mk b/Config.mk
>>> index e1556dfbfa..d21d67945a 100644
>>> --- a/Config.mk
>>> +++ b/Config.mk
>>> @@ -159,6 +159,19 @@ define move-if-changed
>>>   	if ! cmp -s $(1) $(2); then mv -f $(1) $(2); else rm -f $(1); fi
>>>   endef
>>>   
>>> +PATH_FILES := Paths
>>> +INC_FILES := $(foreach f, $(PATH_FILES), $(XEN_ROOT)/config/$(f).mk)
>>> +
>>> +include $(INC_FILES)
>>> +
>>> +BUILD_MAKE_VARS := $(foreach f, $(PATH_FILES), $(shell awk '$$2 == ":=" { print $$1; }' $(XEN_ROOT)/config/$(f).mk.in))
>>> +
>>> +# Replace @xxx@ markers in $(1).in with $(xxx) variable contents, write to $(1)
>>> +define apply-build-vars
>>> + $(1): $(1).in
>>> +	sed $$(foreach v, $$(BUILD_MAKE_VARS), -e 's#@$$(v)@#$$($$(v))#g') <$$< >$$@
>>> +endef
>>
>> Shouldn't this write to a tmp file, and use move-if-changed?  Most of
>> the time the markers won't have changed, and we'll want to short circuit
>> dependent rules.
> 
> I can see this being an advantage when e.g. generating header files, as
> those being generated again would potentially cause lots of rebuilds.
> 
> In this case I can hardly see any case where make wouldn't do the right
> thing already. Either the *.in file is newer than the generated file due
> to a git update or a manual edit, so make will regenerate the target (and
> this is what we want), or the *.in file hasn't changed, so make won't
> regenerate the file as it is newer than the *.in file already.
> 
> Or did I miss some aspect?

Aren't some of the generated files Makefile fragments? Them being re-generated
means make re-invoking itself, which could be avoided if the contents don't
really change. (This isn't just a performance concern; this re-invocation has
been the source of, well, surprising behavior in certain cases.)

Jan

On 17.11.25 13:24, Jan Beulich wrote:
> On 14.11.2025 13:54, Jürgen Groß wrote:
>> On 14.11.25 12:42, Andrew Cooper wrote:
>>> On 14/11/2025 11:32 am, Juergen Gross wrote:
>>>> diff --git a/Config.mk b/Config.mk
>>>> index e1556dfbfa..d21d67945a 100644
>>>> --- a/Config.mk
>>>> +++ b/Config.mk
>>>> @@ -159,6 +159,19 @@ define move-if-changed
>>>>    	if ! cmp -s $(1) $(2); then mv -f $(1) $(2); else rm -f $(1); fi
>>>>    endef
>>>>    
>>>> +PATH_FILES := Paths
>>>> +INC_FILES := $(foreach f, $(PATH_FILES), $(XEN_ROOT)/config/$(f).mk)
>>>> +
>>>> +include $(INC_FILES)
>>>> +
>>>> +BUILD_MAKE_VARS := $(foreach f, $(PATH_FILES), $(shell awk '$$2 == ":=" { print $$1; }' $(XEN_ROOT)/config/$(f).mk.in))
>>>> +
>>>> +# Replace @xxx@ markers in $(1).in with $(xxx) variable contents, write to $(1)
>>>> +define apply-build-vars
>>>> + $(1): $(1).in
>>>> +	sed $$(foreach v, $$(BUILD_MAKE_VARS), -e 's#@$$(v)@#$$($$(v))#g') <$$< >$$@
>>>> +endef
>>>
>>> Shouldn't this write to a tmp file, and use move-if-changed?  Most of
>>> the time the markers won't have changed, and we'll want to short circuit
>>> dependent rules.
>>
>> I can see this being an advantage when e.g. generating header files, as
>> those being generated again would potentially cause lots of rebuilds.
>>
>> In this case I can hardly see any case where make wouldn't do the right
>> thing already. Either the *.in file is newer than the generated file due
>> to a git update or a manual edit, so make will regenerate the target (and
>> this is what we want), or the *.in file hasn't changed, so make won't
>> regenerate the file as it is newer than the *.in file already.
>>
>> Or did I miss some aspect?
> 
> Aren't some of the generated files Makefile fragments? Them being re-generated

No.

Man-pages, shell scripts and some Ocaml files (one config file and one .ml file,
which is similar to an include file I believe).

> means make re-invoking itself, which could be avoided if the contents don't
> really change. (This isn't just a performance concern; this re-invocation has
> been the source of, well, surprising behavior in certain cases.)

I still don't see a case where make would consider rebuilding the file from
its .in file without the .in file having changed, thus resulting in the built
file to change, too. Well, with one probably very rare exception: in case a
different @marker@ is used in the .in file, but without changing the resulting
file due to old and new marker resulting in the same output.

In case we really care about such cases, we should think about using
move-if-changed everywhere, as e.g. building a program with $HOSTCC could
result in an unchanged binary even with source files having changed, and the
resulting program could be used to generate other files ...


Juergen

On 17.11.2025 13:37, Jürgen Groß wrote:
> On 17.11.25 13:24, Jan Beulich wrote:
>> On 14.11.2025 13:54, Jürgen Groß wrote:
>>> On 14.11.25 12:42, Andrew Cooper wrote:
>>>> On 14/11/2025 11:32 am, Juergen Gross wrote:
>>>>> diff --git a/Config.mk b/Config.mk
>>>>> index e1556dfbfa..d21d67945a 100644
>>>>> --- a/Config.mk
>>>>> +++ b/Config.mk
>>>>> @@ -159,6 +159,19 @@ define move-if-changed
>>>>>    	if ! cmp -s $(1) $(2); then mv -f $(1) $(2); else rm -f $(1); fi
>>>>>    endef
>>>>>    
>>>>> +PATH_FILES := Paths
>>>>> +INC_FILES := $(foreach f, $(PATH_FILES), $(XEN_ROOT)/config/$(f).mk)
>>>>> +
>>>>> +include $(INC_FILES)
>>>>> +
>>>>> +BUILD_MAKE_VARS := $(foreach f, $(PATH_FILES), $(shell awk '$$2 == ":=" { print $$1; }' $(XEN_ROOT)/config/$(f).mk.in))
>>>>> +
>>>>> +# Replace @xxx@ markers in $(1).in with $(xxx) variable contents, write to $(1)
>>>>> +define apply-build-vars
>>>>> + $(1): $(1).in
>>>>> +	sed $$(foreach v, $$(BUILD_MAKE_VARS), -e 's#@$$(v)@#$$($$(v))#g') <$$< >$$@
>>>>> +endef
>>>>
>>>> Shouldn't this write to a tmp file, and use move-if-changed?  Most of
>>>> the time the markers won't have changed, and we'll want to short circuit
>>>> dependent rules.
>>>
>>> I can see this being an advantage when e.g. generating header files, as
>>> those being generated again would potentially cause lots of rebuilds.
>>>
>>> In this case I can hardly see any case where make wouldn't do the right
>>> thing already. Either the *.in file is newer than the generated file due
>>> to a git update or a manual edit, so make will regenerate the target (and
>>> this is what we want), or the *.in file hasn't changed, so make won't
>>> regenerate the file as it is newer than the *.in file already.
>>>
>>> Or did I miss some aspect?
>>
>> Aren't some of the generated files Makefile fragments? Them being re-generated
> 
> No.
> 
> Man-pages, shell scripts and some Ocaml files (one config file and one .ml file,
> which is similar to an include file I believe).
> 
>> means make re-invoking itself, which could be avoided if the contents don't
>> really change. (This isn't just a performance concern; this re-invocation has
>> been the source of, well, surprising behavior in certain cases.)
> 
> I still don't see a case where make would consider rebuilding the file from
> its .in file without the .in file having changed, thus resulting in the built
> file to change, too.

As Andrew indicated, Paths.mk might have changed, so at the very least an
explicit dependency would need adding. But as alluded to elsewhere, I'm not
quite convinced Paths.mk should be hard-coded as the sole source of patterns
in Config.mk. At the point further such file come into play, dealing with the
dependencies might get interesting / clumsy.

> Well, with one probably very rare exception: in case a
> different @marker@ is used in the .in file, but without changing the resulting
> file due to old and new marker resulting in the same output.
> 
> In case we really care about such cases, we should think about using
> move-if-changed everywhere, as e.g. building a program with $HOSTCC could
> result in an unchanged binary even with source files having changed, and the
> resulting program could be used to generate other files ...

For some of the cases this might actually be worthwhile. It all depends on
how much of a knock-on effect the re-building of a particular file has.

Jan

On 17.11.25 13:51, Jan Beulich wrote:
> On 17.11.2025 13:37, Jürgen Groß wrote:
>> On 17.11.25 13:24, Jan Beulich wrote:
>>> On 14.11.2025 13:54, Jürgen Groß wrote:
>>>> On 14.11.25 12:42, Andrew Cooper wrote:
>>>>> On 14/11/2025 11:32 am, Juergen Gross wrote:
>>>>>> diff --git a/Config.mk b/Config.mk
>>>>>> index e1556dfbfa..d21d67945a 100644
>>>>>> --- a/Config.mk
>>>>>> +++ b/Config.mk
>>>>>> @@ -159,6 +159,19 @@ define move-if-changed
>>>>>>     	if ! cmp -s $(1) $(2); then mv -f $(1) $(2); else rm -f $(1); fi
>>>>>>     endef
>>>>>>     
>>>>>> +PATH_FILES := Paths
>>>>>> +INC_FILES := $(foreach f, $(PATH_FILES), $(XEN_ROOT)/config/$(f).mk)
>>>>>> +
>>>>>> +include $(INC_FILES)
>>>>>> +
>>>>>> +BUILD_MAKE_VARS := $(foreach f, $(PATH_FILES), $(shell awk '$$2 == ":=" { print $$1; }' $(XEN_ROOT)/config/$(f).mk.in))
>>>>>> +
>>>>>> +# Replace @xxx@ markers in $(1).in with $(xxx) variable contents, write to $(1)
>>>>>> +define apply-build-vars
>>>>>> + $(1): $(1).in
>>>>>> +	sed $$(foreach v, $$(BUILD_MAKE_VARS), -e 's#@$$(v)@#$$($$(v))#g') <$$< >$$@
>>>>>> +endef
>>>>>
>>>>> Shouldn't this write to a tmp file, and use move-if-changed?  Most of
>>>>> the time the markers won't have changed, and we'll want to short circuit
>>>>> dependent rules.
>>>>
>>>> I can see this being an advantage when e.g. generating header files, as
>>>> those being generated again would potentially cause lots of rebuilds.
>>>>
>>>> In this case I can hardly see any case where make wouldn't do the right
>>>> thing already. Either the *.in file is newer than the generated file due
>>>> to a git update or a manual edit, so make will regenerate the target (and
>>>> this is what we want), or the *.in file hasn't changed, so make won't
>>>> regenerate the file as it is newer than the *.in file already.
>>>>
>>>> Or did I miss some aspect?
>>>
>>> Aren't some of the generated files Makefile fragments? Them being re-generated
>>
>> No.
>>
>> Man-pages, shell scripts and some Ocaml files (one config file and one .ml file,
>> which is similar to an include file I believe).
>>
>>> means make re-invoking itself, which could be avoided if the contents don't
>>> really change. (This isn't just a performance concern; this re-invocation has
>>> been the source of, well, surprising behavior in certain cases.)
>>
>> I still don't see a case where make would consider rebuilding the file from
>> its .in file without the .in file having changed, thus resulting in the built
>> file to change, too.
> 
> As Andrew indicated, Paths.mk might have changed, so at the very least an
> explicit dependency would need adding. But as alluded to elsewhere, I'm not

Yes, and I said that already.

> quite convinced Paths.mk should be hard-coded as the sole source of patterns
> in Config.mk. At the point further such file come into play, dealing with the
> dependencies might get interesting / clumsy.

See my answer to your next reply.

> 
>> Well, with one probably very rare exception: in case a
>> different @marker@ is used in the .in file, but without changing the resulting
>> file due to old and new marker resulting in the same output.
>>
>> In case we really care about such cases, we should think about using
>> move-if-changed everywhere, as e.g. building a program with $HOSTCC could
>> result in an unchanged binary even with source files having changed, and the
>> resulting program could be used to generate other files ...
> 
> For some of the cases this might actually be worthwhile. It all depends on
> how much of a knock-on effect the re-building of a particular file has.

As long as the effect is not WRONG (which isn't the case with my patch series),
I think we have to consider how often this would be the case.

Optimizing the build time for one case in 10.000 builds (and I think the ratio
in the case of my series is even more extreme) while making the build time even
only a tiny bit longer for all the other cases is a bad idea IMHO. And the build
time will be slower with using a tmp file and invoking the move-if-changed
macro.


Juergen

On 17/11/2025 12:24 pm, Jan Beulich wrote:
> On 14.11.2025 13:54, Jürgen Groß wrote:
>> On 14.11.25 12:42, Andrew Cooper wrote:
>>> On 14/11/2025 11:32 am, Juergen Gross wrote:
>>>> diff --git a/Config.mk b/Config.mk
>>>> index e1556dfbfa..d21d67945a 100644
>>>> --- a/Config.mk
>>>> +++ b/Config.mk
>>>> @@ -159,6 +159,19 @@ define move-if-changed
>>>>   	if ! cmp -s $(1) $(2); then mv -f $(1) $(2); else rm -f $(1); fi
>>>>   endef
>>>>   
>>>> +PATH_FILES := Paths
>>>> +INC_FILES := $(foreach f, $(PATH_FILES), $(XEN_ROOT)/config/$(f).mk)
>>>> +
>>>> +include $(INC_FILES)
>>>> +
>>>> +BUILD_MAKE_VARS := $(foreach f, $(PATH_FILES), $(shell awk '$$2 == ":=" { print $$1; }' $(XEN_ROOT)/config/$(f).mk.in))
>>>> +
>>>> +# Replace @xxx@ markers in $(1).in with $(xxx) variable contents, write to $(1)
>>>> +define apply-build-vars
>>>> + $(1): $(1).in
>>>> +	sed $$(foreach v, $$(BUILD_MAKE_VARS), -e 's#@$$(v)@#$$($$(v))#g') <$$< >$$@
>>>> +endef
>>> Shouldn't this write to a tmp file, and use move-if-changed?  Most of
>>> the time the markers won't have changed, and we'll want to short circuit
>>> dependent rules.
>> I can see this being an advantage when e.g. generating header files, as
>> those being generated again would potentially cause lots of rebuilds.
>>
>> In this case I can hardly see any case where make wouldn't do the right
>> thing already. Either the *.in file is newer than the generated file due
>> to a git update or a manual edit, so make will regenerate the target (and
>> this is what we want), or the *.in file hasn't changed, so make won't
>> regenerate the file as it is newer than the *.in file already.
>>
>> Or did I miss some aspect?
> Aren't some of the generated files Makefile fragments? Them being re-generated
> means make re-invoking itself, which could be avoided if the contents don't
> really change. (This isn't just a performance concern; this re-invocation has
> been the source of, well, surprising behavior in certain cases.)

Having thought about this some more, it needs to be FORCE and
move-if-changed, or to express a dependency on Paths.mk

Otherwise, if you ./configure --libdir=something/else and then do an
incremental build, the generated files will be wrong.

~Andrew

On 17.11.25 13:30, Andrew Cooper wrote:
> On 17/11/2025 12:24 pm, Jan Beulich wrote:
>> On 14.11.2025 13:54, Jürgen Groß wrote:
>>> On 14.11.25 12:42, Andrew Cooper wrote:
>>>> On 14/11/2025 11:32 am, Juergen Gross wrote:
>>>>> diff --git a/Config.mk b/Config.mk
>>>>> index e1556dfbfa..d21d67945a 100644
>>>>> --- a/Config.mk
>>>>> +++ b/Config.mk
>>>>> @@ -159,6 +159,19 @@ define move-if-changed
>>>>>    	if ! cmp -s $(1) $(2); then mv -f $(1) $(2); else rm -f $(1); fi
>>>>>    endef
>>>>>    
>>>>> +PATH_FILES := Paths
>>>>> +INC_FILES := $(foreach f, $(PATH_FILES), $(XEN_ROOT)/config/$(f).mk)
>>>>> +
>>>>> +include $(INC_FILES)
>>>>> +
>>>>> +BUILD_MAKE_VARS := $(foreach f, $(PATH_FILES), $(shell awk '$$2 == ":=" { print $$1; }' $(XEN_ROOT)/config/$(f).mk.in))
>>>>> +
>>>>> +# Replace @xxx@ markers in $(1).in with $(xxx) variable contents, write to $(1)
>>>>> +define apply-build-vars
>>>>> + $(1): $(1).in
>>>>> +	sed $$(foreach v, $$(BUILD_MAKE_VARS), -e 's#@$$(v)@#$$($$(v))#g') <$$< >$$@
>>>>> +endef
>>>> Shouldn't this write to a tmp file, and use move-if-changed?  Most of
>>>> the time the markers won't have changed, and we'll want to short circuit
>>>> dependent rules.
>>> I can see this being an advantage when e.g. generating header files, as
>>> those being generated again would potentially cause lots of rebuilds.
>>>
>>> In this case I can hardly see any case where make wouldn't do the right
>>> thing already. Either the *.in file is newer than the generated file due
>>> to a git update or a manual edit, so make will regenerate the target (and
>>> this is what we want), or the *.in file hasn't changed, so make won't
>>> regenerate the file as it is newer than the *.in file already.
>>>
>>> Or did I miss some aspect?
>> Aren't some of the generated files Makefile fragments? Them being re-generated
>> means make re-invoking itself, which could be avoided if the contents don't
>> really change. (This isn't just a performance concern; this re-invocation has
>> been the source of, well, surprising behavior in certain cases.)
> 
> Having thought about this some more, it needs to be FORCE and
> move-if-changed, or to express a dependency on Paths.mk

I said that already in the paragraph just after the part of my response you
cited.


Juergen