The generated golang bindings (types.gen.go and helpers.gen.go) are
left checked in so that they can be fetched from xenbits using the
golang tooling.  This means that they must be updated whenever
libxl_types.idl (or other dependencies) are updated.  However, the
golang bindings are only built optionally; we can't assume that anyone
updating libxl_types.idl will also descend into the tools/golang tree
to re-generate the bindings.

Fix this by re-generating the golang bindings from the libxl Makefile
when the IDL dependencies are updated, so that anyone who updates
libxl_types.idl will also end up updating the golang generated files
as well.

 - Make a variable for the generated files, and a target in
   xenlight/Makefile which will only re-generate the files.

 - Add a target in libxl/Makefile to call external idl generation
   targets (currently only golang).

For ease of testing, also add a specific target in libxl/Makefile just
to check and update files generated from the IDL.

This does mean that there are two potential paths for generating the
files during a parallel build; but that shouldn't be an issue, since
tools/golang/xenlight should never be built until after tools/libxl
has completed building anyway.

Signed-off-by: George Dunlap <george.dunlap@citrix.com>
---
CC: Ian Jackson <ian.jackson@citrix.com>
CC: Wei Liu <wl@xen.org>
CC: Nick Rosbrook <rosbrookn@ainfosec.com>
---
 tools/golang/xenlight/Makefile |  6 +++++-
 tools/libxl/Makefile           | 12 +++++++++++-
 2 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/tools/golang/xenlight/Makefile b/tools/golang/xenlight/Makefile
index cd0a62505f..751f916276 100644
--- a/tools/golang/xenlight/Makefile
+++ b/tools/golang/xenlight/Makefile
@@ -17,12 +17,16 @@ all: build
 .PHONY: package
 package: $(XEN_GOPATH)$(GOXL_PKG_DIR)
 
-$(XEN_GOPATH)/src/$(XEN_GOCODE_URL)/xenlight/: xenlight.go types.gen.go helpers.gen.go
+GOXL_GEN_FILES = types.gen.go helpers.gen.go
+
+$(XEN_GOPATH)/src/$(XEN_GOCODE_URL)/xenlight/: xenlight.go $(GOXL_GEN_FILES)
 	$(INSTALL_DIR) $(XEN_GOPATH)$(GOXL_PKG_DIR)
 	$(INSTALL_DATA) xenlight.go $(XEN_GOPATH)$(GOXL_PKG_DIR)
 	$(INSTALL_DATA) types.gen.go $(XEN_GOPATH)$(GOXL_PKG_DIR)
 	$(INSTALL_DATA) helpers.gen.go $(XEN_GOPATH)$(GOXL_PKG_DIR)
 
+idl-gen: $(GOXL_GEN_FILES)
+
 %.gen.go: gengotypes.py $(LIBXL_SRC_DIR)/libxl_types.idl $(LIBXL_SRC_DIR)/idl.py
 	XEN_ROOT=$(XEN_ROOT) $(PYTHON) gengotypes.py $(LIBXL_SRC_DIR)/libxl_types.idl
 
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 69fcf21577..2a06a7ebb8 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -218,7 +218,7 @@ testidl.c: libxl_types.idl gentest.py libxl.h $(AUTOINCS)
 .PHONY: all
 all: $(CLIENTS) $(TEST_PROGS) $(PKG_CONFIG) $(PKG_CONFIG_LOCAL) \
 		libxenlight.so libxenlight.a libxlutil.so libxlutil.a \
-	$(AUTOSRCS) $(AUTOINCS)
+	$(AUTOSRCS) $(AUTOINCS) idl-external
 
 $(LIBXL_OBJS) $(LIBXLU_OBJS) $(SAVE_HELPER_OBJS) \
 		$(LIBXL_TEST_OBJS) $(TEST_PROG_OBJS): \
@@ -274,6 +274,16 @@ _libxl_type%.h _libxl_type%_json.h _libxl_type%_private.h _libxl_type%.c: libxl_
 	$(call move-if-changed,__libxl_type$(stem)_json.h,_libxl_type$(stem)_json.h)
 	$(call move-if-changed,__libxl_type$(stem).c,_libxl_type$(stem).c)
 
+.PHONY: idl-external
+idl-external:
+	$(MAKE) -C $(XEN_ROOT)/tools/golang/xenlight idl-gen
+
+LIBXL_IDLGEN_FILES = _libxl_types.h _libxl_types_json.h _libxl_types_private.h _libxl_types.c \
+	_libxl_types_internal.h _libxl_types_internal_json.h _libxl_types_internal_private.h _libxl_types_internal.c
+
+
+idl-gen: $(LIBXL_GEN_FILES) idl-external
+
 libxenlight.so: libxenlight.so.$(MAJOR)
 	$(SYMLINK_SHLIB) $< $@
 
-- 
2.25.1

George Dunlap writes ("[PATCH 3/5] libxl: Generate golang bindings in libxl Makefile"):
> +.PHONY: idl-external
> +idl-external:
> +	$(MAKE) -C $(XEN_ROOT)/tools/golang/xenlight idl-gen

Unfortunately this kind of thing is forbidden.  At least, without a
rigorous proof that this isn't a concurrency hazard.

The problem is that with parallel make, the concurrency correctness
principles are as follows:
(1) different targets use nonoverlapping temporary and output files
    (makefile authors' responsibiliy)
(2) one invocation of make won't make the same target twice at the
    same time (fundamental principle of operation for make)
(3) the same makefile (or different makefiles with overlapping
    targets) may not be entered multiple times in parallel
    (build system authors' responsibility; preclucdes most use of
    make -C to sibling directories rather than to children)

A correctness proof to make an exception would involve demonstrating
that the tools/golang directories never touch this file when invoked
as part of a recursive build.  NB, consider the clean targets too.

Alternatively, move the generated golang files to tools/libxl maybe,
and perhaps leave symlinks behind.

Or convert the whole (of tools/, maybe) to nonrecursive make using eg
subdirmk :-).  https://diziet.dreamwidth.org/5763.html

Sorry,
Ian.

> On May 26, 2020, at 2:53 PM, Ian Jackson <ian.jackson@citrix.com> wrote:
> 
> George Dunlap writes ("[PATCH 3/5] libxl: Generate golang bindings in libxl Makefile"):
>> +.PHONY: idl-external
>> +idl-external:
>> +	$(MAKE) -C $(XEN_ROOT)/tools/golang/xenlight idl-gen
> 
> Unfortunately this kind of thing is forbidden.  At least, without a
> rigorous proof that this isn't a concurrency hazard.
> 
> The problem is that with parallel make, the concurrency correctness
> principles are as follows:
> (1) different targets use nonoverlapping temporary and output files
>    (makefile authors' responsibiliy)
> (2) one invocation of make won't make the same target twice at the
>    same time (fundamental principle of operation for make)
> (3) the same makefile (or different makefiles with overlapping
>    targets) may not be entered multiple times in parallel
>    (build system authors' responsibility; preclucdes most use of
>    make -C to sibling directories rather than to children)
> 
> A correctness proof to make an exception would involve demonstrating
> that the tools/golang directories never touch this file when invoked
> as part of a recursive build.  NB, consider the clean targets too.

tools/golang/xenlight/Makefile:*.gen.go target will be triggered by xenlight/Makefile:idl-gen and xenlight/Makefile:build.

xenlight/Makefile:build is called from tools/golang/Makfile:subdirs-all, which is called from tools/Makefile:subdirs-all.

xenlight/Makefile:idl-gen is called from tools/libxl/Makefile:idl-external, which is called from tools/libxl/Makefile:all, which is called from tools/Makefile:subdirs-all.

tools/Makefile:subdirs-all is implemented as a non-parallel for loop executing over SUBDIRS-y; tools/golang comes after tools/libxl in that list, and so tools/golang:all will never be called until after tools/libxl:all has completed.  This invariant — that tools/golang/Makefile:all must not be called until tools/libxl/Makefile:all has completed must be kept regardless of this patch, since xenlight/Makefile:build depends on other output from tools/libxl/Makefile:all.

So as long as nothing else calls tools/libxl:all or tools/libxl:idl-external, this should be safe.  We could add a comments near xenlight/Makefile:idl-gen saying it must only be called from libxl/Makefile:idl-external; and to libxl/Makefile:idl-external saying it must not be called recursively from another Makefile.

> Alternatively, move the generated golang files to tools/libxl maybe,
> and perhaps leave symlinks behind.

Would that result in the files being accessible to the golang build tools at https://xenbits.xenproject.org/git-http/xen.git/tools/golang/xenlight ?  If not, it defeats the purpose of having the files checked into the tree.

> Or convert the whole (of tools/, maybe) to nonrecursive make using eg
> subdirmk :-).  https://diziet.dreamwidth.org/5763.html

This isn’t really a practical suggestion: I don’t have time to refactor the entire libxl Makefile tree, and certainly don’t have time by Friday.

 -George

George Dunlap writes ("Re: [PATCH 3/5] libxl: Generate golang bindings in libxl Makefile"):
> tools/Makefile:subdirs-all is implemented as a non-parallel for loop executing over SUBDIRS-y; tools/golang comes after tools/libxl in that list, and so tools/golang:all will never be called until after tools/libxl:all has completed.  This invariant — that tools/golang/Makefile:all must not be called until tools/libxl/Makefile:all has completed must be kept regardless of this patch, since xenlight/Makefile:build depends on other output from tools/libxl/Makefile:all.

I had not spotted this aspect of the situation.  But the toplevel
Makefile is parallel.  I think this means that make -C works between
different directories in tools/.

Provided no-one says `make all install' (which is a thing that people
expect to work but which is already badly broken).

> So as long as nothing else calls tools/libxl:all or tools/libxl:idl-external, this should be safe.  We could add a comments near xenlight/Makefile:idl-gen saying it must only be called from libxl/Makefile:idl-external; and to libxl/Makefile:idl-external saying it must not be called recursively from another Makefile.

So, err, I'm sold on the original patch, I think.

Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>

I'll answer your other comments anyway:

> > Alternatively, move the generated golang files to tools/libxl maybe,
> > and perhaps leave symlinks behind.
> 
> Would that result in the files being accessible to the golang build tools at https://xenbits.xenproject.org/git-http/xen.git/tools/golang/xenlight ?  If not, it defeats the purpose of having the files checked into the tree.

Yes.  git can convey symlinks.  (I'm assuming that the golang build
tools fetch the git objects and do git checkout, rather than
trying to download individual raw files from gitweb...)

> > Or convert the whole (of tools/, maybe) to nonrecursive make using eg
> > subdirmk :-).  https://diziet.dreamwidth.org/5763.html
> 
> This isn’t really a practical suggestion: I don’t have time to refactor the entire libxl Makefile tree, and certainly don’t have time by Friday.

Yes, it wasn't a serious suggestion.  Sorry that that apparently
wasn't clear.

Regards,
Ian.

> The generated golang bindings (types.gen.go and helpers.gen.go) are
> left checked in so that they can be fetched from xenbits using the
> golang tooling.  This means that they must be updated whenever
> libxl_types.idl (or other dependencies) are updated.  However, the
> golang bindings are only built optionally; we can't assume that anyone
> updating libxl_types.idl will also descend into the tools/golang tree
> to re-generate the bindings.
>
> Fix this by re-generating the golang bindings from the libxl Makefile
> when the IDL dependencies are updated, so that anyone who updates
> libxl_types.idl will also end up updating the golang generated files
> as well.
>
>  - Make a variable for the generated files, and a target in
>    xenlight/Makefile which will only re-generate the files.
>
>  - Add a target in libxl/Makefile to call external idl generation
>    targets (currently only golang).
>
> For ease of testing, also add a specific target in libxl/Makefile just
> to check and update files generated from the IDL.
>
> This does mean that there are two potential paths for generating the
> files during a parallel build; but that shouldn't be an issue, since
> tools/golang/xenlight should never be built until after tools/libxl
> has completed building anyway.
>
> Signed-off-by: George Dunlap <george.dunlap@citrix.com>

For the golang side:

Reviewed-by: Nick Rosbrook <rosbrookn@ainfosec.com>