For in-tree builds relative paths are used, whereas for out-of-tree builds
in various situations absolute ones come into play. The extra paths can be
long, wasting space and e.g. serial line bandwidth. They would also get in
the way of location-independent reproducible builds. Leverage newer gcc's
(and Clang's) ability to "remap" file names. For older gcc fall back to
using the option affecting debug info only.
Signed-off-by: Jan Beulich <jbeulich@suse.com>
---
Of course we may want to consider putting this in the top-level Config.mk,
to also affect other sub-trees (presently mainly/only affecting debug
info, for which even gcc5 already supports -fdebug-prefix-remap=).
As to a Fixes: tag, I wasn't quite sure whether to "blame" the
introduction of out-of-tree builds.
--- a/xen/Makefile
+++ b/xen/Makefile
@@ -448,6 +448,8 @@ LDFLAGS-$(CONFIG_CC_IS_CLANG) += -plugin
endif
ifdef building_out_of_srctree
+ CFLAGS += $(call cc-option,$(CC),-ffile-prefix-map=$(srctree)/=, \
+ -fdebug-prefix-map=$(srctree)/=)
CFLAGS += -I$(objtree)/include
CFLAGS += -I$(objtree)/arch/$(SRCARCH)/include
endifOn 03/09/2025 4:13 pm, Jan Beulich wrote: > For in-tree builds relative paths are used, whereas for out-of-tree builds > in various situations absolute ones come into play. The extra paths can be > long, wasting space and e.g. serial line bandwidth. They would also get in > the way of location-independent reproducible builds. Leverage newer gcc's > (and Clang's) ability to "remap" file names. For older gcc fall back to > using the option affecting debug info only. > > Signed-off-by: Jan Beulich <jbeulich@suse.com> > --- > Of course we may want to consider putting this in the top-level Config.mk, > to also affect other sub-trees (presently mainly/only affecting debug > info, for which even gcc5 already supports -fdebug-prefix-remap=). > > As to a Fixes: tag, I wasn't quite sure whether to "blame" the > introduction of out-of-tree builds. > > --- a/xen/Makefile > +++ b/xen/Makefile > @@ -448,6 +448,8 @@ LDFLAGS-$(CONFIG_CC_IS_CLANG) += -plugin > endif > > ifdef building_out_of_srctree > + CFLAGS += $(call cc-option,$(CC),-ffile-prefix-map=$(srctree)/=, \ > + -fdebug-prefix-map=$(srctree)/=) > CFLAGS += -I$(objtree)/include > CFLAGS += -I$(objtree)/arch/$(SRCARCH)/include > endif We do want to be taking a change like this, but it's also definitely not limited to out-of-tree builds. I have full paths embedded even for in-tree builds. To be useful, it wants to apply to everything, not just the hypervisor, so does want to be in the top level Config.mk. https://reproducible-builds.org/docs/build-path/ has a full list of compiler versions. It looks like we need to use both options here until we can increase the minimum GCC version to 8. ~Andrew
On 03.09.2025 17:26, Andrew Cooper wrote: > On 03/09/2025 4:13 pm, Jan Beulich wrote: >> For in-tree builds relative paths are used, whereas for out-of-tree builds >> in various situations absolute ones come into play. The extra paths can be >> long, wasting space and e.g. serial line bandwidth. They would also get in >> the way of location-independent reproducible builds. Leverage newer gcc's >> (and Clang's) ability to "remap" file names. For older gcc fall back to >> using the option affecting debug info only. >> >> Signed-off-by: Jan Beulich <jbeulich@suse.com> >> --- >> Of course we may want to consider putting this in the top-level Config.mk, >> to also affect other sub-trees (presently mainly/only affecting debug >> info, for which even gcc5 already supports -fdebug-prefix-remap=). >> >> As to a Fixes: tag, I wasn't quite sure whether to "blame" the >> introduction of out-of-tree builds. >> >> --- a/xen/Makefile >> +++ b/xen/Makefile >> @@ -448,6 +448,8 @@ LDFLAGS-$(CONFIG_CC_IS_CLANG) += -plugin >> endif >> >> ifdef building_out_of_srctree >> + CFLAGS += $(call cc-option,$(CC),-ffile-prefix-map=$(srctree)/=, \ >> + -fdebug-prefix-map=$(srctree)/=) >> CFLAGS += -I$(objtree)/include >> CFLAGS += -I$(objtree)/arch/$(SRCARCH)/include >> endif > > We do want to be taking a change like this, but it's also definitely not > limited to out-of-tree builds. I have full paths embedded even for > in-tree builds. In xen-syms I see only two full paths - in debug info, supplying the base path to the tree. That's okay to stay imo. In xen.efi I see a few hundred, but they're all the same as above. As I learned earlier today, SHF_MERGE processing isn't invoked when linking ELF objects into a PE binary. > To be useful, it wants to apply to everything, not just the hypervisor, > so does want to be in the top level Config.mk. As per my first remark then. But no, I meanwhile realized that this can't go in Config.mk: For the hypervisor we want to use $(srctree), i.e. including the leaf /xen referencing the xen/ subtree. I expect that for e.g. tools/libs/ we'd want something similar - eliminate the entire path up to the base of the component's source dir. So it will need to be piecemeal. > https://reproducible-builds.org/docs/build-path/ has a full list of > compiler versions. It looks like we need to use both options here until > we can increase the minimum GCC version to 8. Not quite, -ffile-prefix-map= is documented to imply all other -f*-prefix-map=, matching my observations. Bottom line - at least for now I think the patch wants to remain as is, and further patches for other parts of the tree will need making. Jan
On 03/09/2025 5:12 pm, Jan Beulich wrote: > On 03.09.2025 17:26, Andrew Cooper wrote: >> On 03/09/2025 4:13 pm, Jan Beulich wrote: >>> For in-tree builds relative paths are used, whereas for out-of-tree builds >>> in various situations absolute ones come into play. The extra paths can be >>> long, wasting space and e.g. serial line bandwidth. They would also get in >>> the way of location-independent reproducible builds. Leverage newer gcc's >>> (and Clang's) ability to "remap" file names. For older gcc fall back to >>> using the option affecting debug info only. >>> >>> Signed-off-by: Jan Beulich <jbeulich@suse.com> >>> --- >>> Of course we may want to consider putting this in the top-level Config.mk, >>> to also affect other sub-trees (presently mainly/only affecting debug >>> info, for which even gcc5 already supports -fdebug-prefix-remap=). >>> >>> As to a Fixes: tag, I wasn't quite sure whether to "blame" the >>> introduction of out-of-tree builds. >>> >>> --- a/xen/Makefile >>> +++ b/xen/Makefile >>> @@ -448,6 +448,8 @@ LDFLAGS-$(CONFIG_CC_IS_CLANG) += -plugin >>> endif >>> >>> ifdef building_out_of_srctree >>> + CFLAGS += $(call cc-option,$(CC),-ffile-prefix-map=$(srctree)/=, \ >>> + -fdebug-prefix-map=$(srctree)/=) >>> CFLAGS += -I$(objtree)/include >>> CFLAGS += -I$(objtree)/arch/$(SRCARCH)/include >>> endif >> We do want to be taking a change like this, but it's also definitely not >> limited to out-of-tree builds. I have full paths embedded even for >> in-tree builds. > In xen-syms I see only two full paths - in debug info, supplying the base > path to the tree. That's okay to stay imo. Not for reducible builds it's not. > > In xen.efi I see a few hundred, but they're all the same as above. As I > learned earlier today, SHF_MERGE processing isn't invoked when linking > ELF objects into a PE binary. > >> To be useful, it wants to apply to everything, not just the hypervisor, >> so does want to be in the top level Config.mk. > As per my first remark then. But no, I meanwhile realized that this can't > go in Config.mk: For the hypervisor we want to use $(srctree), i.e. > including the leaf /xen referencing the xen/ subtree. I expect that for > e.g. tools/libs/ we'd want something similar - eliminate the entire path > up to the base of the component's source dir. So it will need to be > piecemeal. Relative to the root of xen.git (or the source tarball) is the only sensible option. Anything else is intentionally misleading. In fact, Marek had a more-correct form of this patch in https://lore.kernel.org/xen-devel/0370c0eb1fd9ac00acab016792132fa0b943d384.1742317309.git-series.marmarek@invisiblethingslab.com/T/#u which seems to be waiting on you to reply. Please, along with Marek and Anthony, figure out what a v2 should look like, which applies to the whole tree and does not trim the xen/ part of the path from the hypervisor. Oleksii, please track this for 4.21. Build reproducibility is not an optional exercise these days, and given multiple downstreams depending on it, it needs fixing for once and for all in upstream. ~Andrew
On 03.09.2025 18:40, Andrew Cooper wrote: > On 03/09/2025 5:12 pm, Jan Beulich wrote: >> On 03.09.2025 17:26, Andrew Cooper wrote: >>> On 03/09/2025 4:13 pm, Jan Beulich wrote: >>>> --- a/xen/Makefile >>>> +++ b/xen/Makefile >>>> @@ -448,6 +448,8 @@ LDFLAGS-$(CONFIG_CC_IS_CLANG) += -plugin >>>> endif >>>> >>>> ifdef building_out_of_srctree >>>> + CFLAGS += $(call cc-option,$(CC),-ffile-prefix-map=$(srctree)/=, \ >>>> + -fdebug-prefix-map=$(srctree)/=) >>>> CFLAGS += -I$(objtree)/include >>>> CFLAGS += -I$(objtree)/arch/$(SRCARCH)/include >>>> endif >>> We do want to be taking a change like this, but it's also definitely not >>> limited to out-of-tree builds. I have full paths embedded even for >>> in-tree builds. >> In xen-syms I see only two full paths - in debug info, supplying the base >> path to the tree. What I'm missing from your reply is clarification whether the mentioned instances are indeed the only ones you see, or whether there's more in what you have (and what I'm not seeing for whatever reason). >> That's okay to stay imo. > > Not for reducible builds it's not. Yes, I realized this later. However, I can see benefits to both: When one wants reproducible builds, no absolute path whatsoever should remain. In other (debugging) cases having a reference to the root of what everything else is relative to might be helpful. So whether to replace these remaining instances may want to be configurable (in turn making it necessary to deal with that independently for xen/ and tools/; for xen/ that would be a Kconfig option dependent upon DEBUG_INFO=y). In out-of-tree builds similar references exist to the build tree root. Once we zap both, the result is at risk of being ambiguous. I wonder whether it would be possible (supported by consumers) to replace both references by something macro-like (along the lines of $SRC/ and $BLD/). >> In xen.efi I see a few hundred, but they're all the same as above. As I >> learned earlier today, SHF_MERGE processing isn't invoked when linking >> ELF objects into a PE binary. >> >>> To be useful, it wants to apply to everything, not just the hypervisor, >>> so does want to be in the top level Config.mk. >> As per my first remark then. But no, I meanwhile realized that this can't >> go in Config.mk: For the hypervisor we want to use $(srctree), i.e. >> including the leaf /xen referencing the xen/ subtree. I expect that for >> e.g. tools/libs/ we'd want something similar - eliminate the entire path >> up to the base of the component's source dir. So it will need to be >> piecemeal. > > Relative to the root of xen.git (or the source tarball) is the only > sensible option. Anything else is intentionally misleading. I disagree. In-tree builds record things downward from xen/ only. So should out-of-tree builds do. Every individual binary (i.e. including all the tools/ ones) has no need to record anything more than is necessary to unambiguously identify the source files. In particular us bundling hypervisor, toolstack, and stubdom (and there we expand various external packages) in a single repo / tarball is an artifact, not how things normally would be arranged. > In fact, Marek had a more-correct form of this patch in > https://lore.kernel.org/xen-devel/0370c0eb1fd9ac00acab016792132fa0b943d384.1742317309.git-series.marmarek@invisiblethingslab.com/T/#u > which seems to be waiting on you to reply. I can't spot anything expecting my reply. What I can spot is a promise to submit a v2. And, having entirely forgotten that there already was an attempt, I only now realize why the options coming into play seemed somewhat familiar. Jan
© 2016 - 2025 Red Hat, Inc.