[XEN PATCH for-4.17 v5 00/17] Toolstack build system improvement, toward non-recursive makefiles

Anthony PERARD posted 17 patches 1 year, 6 months ago
Failed in applying to current master (apply log)
There is a newer version of this series
tools/configure.ac                     |  2 +
tools/console/client/Makefile          |  1 -
tools/console/daemon/Makefile          |  1 -
tools/debugger/gdbsx/Makefile          | 20 +++----
tools/debugger/gdbsx/gx/Makefile       | 15 +++---
tools/debugger/gdbsx/xg/Makefile       | 25 +++------
tools/debugger/kdd/Makefile            |  1 -
tools/flask/utils/Makefile             |  1 -
tools/fuzz/cpu-policy/Makefile         |  2 +-
tools/golang/xenlight/Makefile         |  8 +--
tools/hotplug/common/Makefile          | 10 +---
tools/include/Makefile                 | 28 +++++-----
tools/libs/call/Makefile               |  1 +
tools/libs/ctrl/Makefile               |  3 --
tools/libs/devicemodel/Makefile        |  1 +
tools/libs/evtchn/Makefile             |  1 +
tools/libs/foreignmemory/Makefile      |  1 +
tools/libs/gnttab/Makefile             |  1 +
tools/libs/guest/Makefile              |  3 --
tools/libs/hypfs/Makefile              |  1 +
tools/libs/light/Makefile              | 72 +++++++++++++++-----------
tools/libs/stat/Makefile               |  2 +-
tools/libs/store/Makefile              |  1 +
tools/libs/toolcore/Makefile           |  1 +
tools/libs/toollog/Makefile            |  1 +
tools/libs/util/Makefile               |  3 --
tools/libs/vchan/Makefile              |  3 --
tools/misc/Makefile                    |  1 -
tools/ocaml/libs/eventchn/Makefile     |  2 +-
tools/ocaml/libs/xc/Makefile           |  2 +-
tools/ocaml/libs/xentoollog/Makefile   |  2 +-
tools/ocaml/libs/xl/Makefile           |  2 +-
tools/tests/cpu-policy/Makefile        |  2 +-
tools/tests/depriv/Makefile            |  2 +-
tools/tests/resource/Makefile          |  1 -
tools/tests/tsx/Makefile               |  1 -
tools/tests/xenstore/Makefile          |  1 -
tools/xcutils/Makefile                 |  2 -
tools/xenmon/Makefile                  |  1 -
tools/xenpaging/Makefile               |  1 -
tools/xenpmd/Makefile                  |  1 -
tools/xentop/Makefile                  |  2 +-
tools/xentrace/Makefile                | 29 ++++-------
tools/xl/Makefile                      |  2 +-
tools/Rules.mk                         | 20 +++++++
tools/debugger/gdbsx/Rules.mk          |  2 +-
tools/firmware/Rules.mk                |  2 -
tools/libfsimage/common.mk             |  2 +-
tools/libs/libs.mk                     | 21 +++++---
tools/libs/light/libxl_x86_acpi.c      |  2 +-
tools/ocaml/common.make                |  2 +-
.gitignore                             |  6 ---
config/Tools.mk.in                     |  1 +
scripts/git-checkout.sh                |  4 +-
tools/configure                        | 29 ++++++++++-
tools/golang/xenlight/gengotypes.py    | 10 +++-
tools/hotplug/common/hotplugpath.sh.in | 16 ++++++
tools/libs/light/gentypes.py           |  9 ++--
tools/xenstore/Makefile.common         |  1 -
59 files changed, 218 insertions(+), 172 deletions(-)
create mode 100644 tools/hotplug/common/hotplugpath.sh.in
[XEN PATCH for-4.17 v5 00/17] Toolstack build system improvement, toward non-recursive makefiles
Posted by Anthony PERARD 1 year, 6 months ago
Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.toolstack-build-system-v5

Changes in v5:
- rebased on staging
- added "tools: Rework linking options for ocaml binding libraries"

Changes in v4:
- several new patches
- some changes to other patches listed in their changelogs

Changes in v3:
- rebased
- several new patches, starting with 13/25 "tools/libs/util: cleanup Makefile"
- introducing macros to deal with linking with in-tree xen libraries
- Add -Werror to CFLAGS for all builds in tools/

Changes in v2:
- one new patch
- other changes described in patch notes

Hi everyone,

I've been looking at reworking the build system we have for the "tools/", and
transforming it to something that suit it better. There are a lot of
dependencies between different sub-directories so it would be nice if GNU make
could actually handle them. This is possible with "non-recursive makefiles".

With non-recursive makefiles, make will have to load/include all the makefiles
and thus will have complete overview of all the dependencies. This will allow
make to build the necessary targets in other directory, and we won't need to
build sub-directories one by one.

To help with this transformation, I've chosen to go with a recent project
called "subdirmk". It help to deal with the fact that all makefiles will share
the same namespace, it is hooked into autoconf, we can easily run `make` from
any subdirectory. Together "autoconf" and "subdirmk" will also help to get
closer to be able to do out-of-tree build of the tools, but I'm mainly looking
to have non-recursive makefile.

Link to the project:
    https://www.chiark.greenend.org.uk/ucgi/~ian/git/subdirmk.git/

But before getting to the main course, I've got quite a few cleanup and some
changes to the makefiles. I start the patch series with patches that remove old
left over stuff, then start reworking makefiles. They are some common changes like
removing the "build" targets in many places as "all" would be the more common
way to spell it and "all" is the default target anyway. They are other changes
related to the conversion to "subdirmk", I start to use the variable $(TARGETS)
in several makefiles, this variable will have a special meaning in subdirmk
which will build those target by default.

As for the conversion to non-recursive makefile, with subdirmk, I have this WIP
branch, it contains some changes that I'm trying out, some notes, and the
conversion, one Makefile per commit. Cleanup are still needed, some makefile
not converted yet, but it's otherwise mostly done.

    https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.toolstack-build-system-v1-wip-extra

With that branch, you could tried something like:
    ./configure; cd tools/xl; make
and `xl` should be built as well as all the xen library needed.
Also, things like `make clean` or rebuild should be faster in the all tools/
directory.

Cheers,

Anthony PERARD (17):
  tools/debugger/gdbsx: Fix and cleanup makefiles
  tools/xentrace: rework Makefile
  tools: Introduce $(xenlibs-ldflags, ) macro
  tools: Add -Werror by default to all tools/
  tools: Remove -Werror everywhere else
  tools/hotplug: Generate "hotplugpath.sh" with configure
  libs/light/gentypes.py: allow to generate headers in subdirectory
  git-checkout.sh: handle running git-checkout from a different
    directory
  libs: Avoid exposing -Wl,--version-script to other built library
  libs: Fix auto-generation of version-script for unstable libs
  tools/include: Rework Makefile
  libs/light: Rework acpi table build targets
  libs/light: Rework generation of include/_libxl_*.h
  libs/light: Rework targets prerequisites
  libs/light: Makefile cleanup
  tools/golang/xenlight: Rework gengotypes.py and generation of *.gen.go
  tools: Rework linking options for ocaml binding libraries

 tools/configure.ac                     |  2 +
 tools/console/client/Makefile          |  1 -
 tools/console/daemon/Makefile          |  1 -
 tools/debugger/gdbsx/Makefile          | 20 +++----
 tools/debugger/gdbsx/gx/Makefile       | 15 +++---
 tools/debugger/gdbsx/xg/Makefile       | 25 +++------
 tools/debugger/kdd/Makefile            |  1 -
 tools/flask/utils/Makefile             |  1 -
 tools/fuzz/cpu-policy/Makefile         |  2 +-
 tools/golang/xenlight/Makefile         |  8 +--
 tools/hotplug/common/Makefile          | 10 +---
 tools/include/Makefile                 | 28 +++++-----
 tools/libs/call/Makefile               |  1 +
 tools/libs/ctrl/Makefile               |  3 --
 tools/libs/devicemodel/Makefile        |  1 +
 tools/libs/evtchn/Makefile             |  1 +
 tools/libs/foreignmemory/Makefile      |  1 +
 tools/libs/gnttab/Makefile             |  1 +
 tools/libs/guest/Makefile              |  3 --
 tools/libs/hypfs/Makefile              |  1 +
 tools/libs/light/Makefile              | 72 +++++++++++++++-----------
 tools/libs/stat/Makefile               |  2 +-
 tools/libs/store/Makefile              |  1 +
 tools/libs/toolcore/Makefile           |  1 +
 tools/libs/toollog/Makefile            |  1 +
 tools/libs/util/Makefile               |  3 --
 tools/libs/vchan/Makefile              |  3 --
 tools/misc/Makefile                    |  1 -
 tools/ocaml/libs/eventchn/Makefile     |  2 +-
 tools/ocaml/libs/xc/Makefile           |  2 +-
 tools/ocaml/libs/xentoollog/Makefile   |  2 +-
 tools/ocaml/libs/xl/Makefile           |  2 +-
 tools/tests/cpu-policy/Makefile        |  2 +-
 tools/tests/depriv/Makefile            |  2 +-
 tools/tests/resource/Makefile          |  1 -
 tools/tests/tsx/Makefile               |  1 -
 tools/tests/xenstore/Makefile          |  1 -
 tools/xcutils/Makefile                 |  2 -
 tools/xenmon/Makefile                  |  1 -
 tools/xenpaging/Makefile               |  1 -
 tools/xenpmd/Makefile                  |  1 -
 tools/xentop/Makefile                  |  2 +-
 tools/xentrace/Makefile                | 29 ++++-------
 tools/xl/Makefile                      |  2 +-
 tools/Rules.mk                         | 20 +++++++
 tools/debugger/gdbsx/Rules.mk          |  2 +-
 tools/firmware/Rules.mk                |  2 -
 tools/libfsimage/common.mk             |  2 +-
 tools/libs/libs.mk                     | 21 +++++---
 tools/libs/light/libxl_x86_acpi.c      |  2 +-
 tools/ocaml/common.make                |  2 +-
 .gitignore                             |  6 ---
 config/Tools.mk.in                     |  1 +
 scripts/git-checkout.sh                |  4 +-
 tools/configure                        | 29 ++++++++++-
 tools/golang/xenlight/gengotypes.py    | 10 +++-
 tools/hotplug/common/hotplugpath.sh.in | 16 ++++++
 tools/libs/light/gentypes.py           |  9 ++--
 tools/xenstore/Makefile.common         |  1 -
 59 files changed, 218 insertions(+), 172 deletions(-)
 create mode 100644 tools/hotplug/common/hotplugpath.sh.in

-- 
Anthony PERARD
Re: [XEN PATCH for-4.17 v5 00/17] Toolstack build system improvement, toward non-recursive makefiles
Posted by Andrew Cooper 1 year, 6 months ago
On 13/10/2022 14:04, Anthony PERARD wrote:
> Patch series available in this git branch:
> https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.toolstack-build-system-v5

I've done a full rebuild of this (with the 2 noted fixes) with Xapi, and
it does resolve the Ocaml library linking issues.  Thanks.

So T-by.  If an allowance for 4.17 appears, I can see about getting some
of the more obviously-trivial areas reviewed and committed.

~Andrew
RE: [XEN PATCH for-4.17 v5 00/17] Toolstack build system improvement, toward non-recursive makefiles
Posted by Henry Wang 1 year, 6 months ago
Hi Andrew,

> -----Original Message-----
> From: Andrew Cooper <Andrew.Cooper3@citrix.com>
> Subject: Re: [XEN PATCH for-4.17 v5 00/17] Toolstack build system
> improvement, toward non-recursive makefiles
> 
> On 13/10/2022 14:04, Anthony PERARD wrote:
> > Patch series available in this git branch:
> > https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git
> br.toolstack-build-system-v5
> 
> I've done a full rebuild of this (with the 2 noted fixes) with Xapi, and
> it does resolve the Ocaml library linking issues.  Thanks.
> 
> So T-by.  If an allowance for 4.17 appears, I can see about getting some
> of the more obviously-trivial areas reviewed and committed.

Release-acked-by: Henry Wang <Henry.Wang@arm.com>

Since this series is quite large and just out of curiosity, do we have a
pipeline or CI jobs to test all the modules that affected by the makefile
touched by this series? Would be good to have some test results to
boost our confidence of committing the series.

Kind regards,
Henry

> 
> ~Andrew
Re: [XEN PATCH for-4.17 v5 00/17] Toolstack build system improvement, toward non-recursive makefiles
Posted by Anthony PERARD 1 year, 6 months ago
On Fri, Oct 14, 2022 at 06:26:09AM +0000, Henry Wang wrote:
> > -----Original Message-----
> > From: Andrew Cooper <Andrew.Cooper3@citrix.com>
> > Subject: Re: [XEN PATCH for-4.17 v5 00/17] Toolstack build system
> > improvement, toward non-recursive makefiles
> > 
> > On 13/10/2022 14:04, Anthony PERARD wrote:
> > > Patch series available in this git branch:
> > > https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git
> > br.toolstack-build-system-v5
> > 
> > I've done a full rebuild of this (with the 2 noted fixes) with Xapi, and
> > it does resolve the Ocaml library linking issues.  Thanks.
> > 
> > So T-by.  If an allowance for 4.17 appears, I can see about getting some
> > of the more obviously-trivial areas reviewed and committed.
> 
> Release-acked-by: Henry Wang <Henry.Wang@arm.com>

Thanks.

> Since this series is quite large and just out of curiosity, do we have a
> pipeline or CI jobs to test all the modules that affected by the makefile
> touched by this series? Would be good to have some test results to
> boost our confidence of committing the series.

I did push my branch to Gitlab, and that the pipeline:
    https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/665987083
    (private to xen-project members)
Only failures are I think the same one as on staging, test failure on
arm, but no build issues.

I'm not sure that everything is tested, but most of it is.

Cheers,

-- 
Anthony PERARD
Re: [XEN PATCH for-4.17 v5 00/17] Toolstack build system improvement, toward non-recursive makefiles
Posted by Andrew Cooper 1 year, 6 months ago
On 14/10/2022 11:14, Anthony PERARD wrote:
> On Fri, Oct 14, 2022 at 06:26:09AM +0000, Henry Wang wrote:
>>> -----Original Message-----
>>> From: Andrew Cooper <Andrew.Cooper3@citrix.com>
>>> Subject: Re: [XEN PATCH for-4.17 v5 00/17] Toolstack build system
>>> improvement, toward non-recursive makefiles
>>>
>>> On 13/10/2022 14:04, Anthony PERARD wrote:
>>>> Patch series available in this git branch:
>>>> https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git
>>> br.toolstack-build-system-v5
>>>
>>> I've done a full rebuild of this (with the 2 noted fixes) with Xapi, and
>>> it does resolve the Ocaml library linking issues.  Thanks.
>>>
>>> So T-by.  If an allowance for 4.17 appears, I can see about getting some
>>> of the more obviously-trivial areas reviewed and committed.
>> Release-acked-by: Henry Wang <Henry.Wang@arm.com>
> Thanks.
>
>> Since this series is quite large and just out of curiosity, do we have a
>> pipeline or CI jobs to test all the modules that affected by the makefile
>> touched by this series? Would be good to have some test results to
>> boost our confidence of committing the series.
> I did push my branch to Gitlab, and that the pipeline:
>     https://gitlab.com/xen-project/people/anthonyper/xen/-/pipelines/665987083
>     (private to xen-project members)
> Only failures are I think the same one as on staging, test failure on
> arm, but no build issues.
>
> I'm not sure that everything is tested, but most of it is.

The XenServer specfile lists all build artefacts specifically, because
we've had too many bugs where things have silently disappeared.

My testing confirms that nothing has appeared or disappeared as a result
of these changes, and the functional looks pretty good.  (except for the
latest shadow assertion found, which was a regression in XSA-410.  Fix
is in review elsewhere.)

~Andrew
RE: [XEN PATCH for-4.17 v5 00/17] Toolstack build system improvement, toward non-recursive makefiles
Posted by Henry Wang 1 year, 6 months ago
Hi Andrew,

> -----Original Message-----
> From: Andrew Cooper <Andrew.Cooper3@citrix.com>
> Subject: Re: [XEN PATCH for-4.17 v5 00/17] Toolstack build system
> improvement, toward non-recursive makefiles
> The XenServer specfile lists all build artefacts specifically, because
> we've had too many bugs where things have silently disappeared.
> 
> My testing confirms that nothing has appeared or disappeared as a result
> of these changes, and the functional looks pretty good.  (except for the
> latest shadow assertion found, which was a regression in XSA-410.  Fix
> is in review elsewhere.)

This is good to know. Thanks for the confirmation. No problem from
my side then.

Kind regards,
Henry

> 
> ~Andrew
RE: [XEN PATCH for-4.17 v5 00/17] Toolstack build system improvement, toward non-recursive makefiles
Posted by Henry Wang 1 year, 6 months ago
Hi Anthony,

> -----Original Message-----
> From: Anthony PERARD <anthony.perard@citrix.com>
> Subject: Re: [XEN PATCH for-4.17 v5 00/17] Toolstack build system
> improvement, toward non-recursive makefiles
> 
> On Fri, Oct 14, 2022 at 06:26:09AM +0000, Henry Wang wrote:
> > > -----Original Message-----
> > > From: Andrew Cooper <Andrew.Cooper3@citrix.com>
> > > Subject: Re: [XEN PATCH for-4.17 v5 00/17] Toolstack build system
> > > improvement, toward non-recursive makefiles
> > >
> > > On 13/10/2022 14:04, Anthony PERARD wrote:
> > > > Patch series available in this git branch:
> > > > https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git
> > > br.toolstack-build-system-v5
> > >
> > > I've done a full rebuild of this (with the 2 noted fixes) with Xapi, and
> > > it does resolve the Ocaml library linking issues.� Thanks.
> > >
> > > So T-by.� If an allowance for 4.17 appears, I can see about getting some
> > > of the more obviously-trivial areas reviewed and committed.
> >
> > Release-acked-by: Henry Wang <Henry.Wang@arm.com>
> 
> Thanks.

No problem, my pleasure.

> 
> > Since this series is quite large and just out of curiosity, do we have a
> > pipeline or CI jobs to test all the modules that affected by the makefile
> > touched by this series? Would be good to have some test results to
> > boost our confidence of committing the series.
> 
> I did push my branch to Gitlab, and that the pipeline:
>     https://gitlab.com/xen-project/people/anthonyper/xen/-
> /pipelines/665987083
>     (private to xen-project members)
> Only failures are I think the same one as on staging, test failure on
> arm, but no build issues.

Sorry for the arm failure, I will try to respin the patch ASAP and hopefully
the CI will be unblocked soon.

Kind regards,
Henry

> 
> I'm not sure that everything is tested, but most of it is.
> 
> Cheers,
> 
> --
> Anthony PERARD
Re: [XEN PATCH for-4.17 v5 00/17] Toolstack build system improvement, toward non-recursive makefiles
Posted by Christian Lindig 1 year, 6 months ago

On 13 Oct 2022, at 14:04, Anthony PERARD <anthony.perard@citrix.com<mailto:anthony.perard@citrix.com>> wrote:

Patch series available in this git branch:
https://xenbits.xen.org/git-http/people/aperard/xen-unstable.git br.toolstack-build-system-v5

Changes in v5:
- rebased on staging
- added "tools: Rework linking options for ocaml binding libraries"

Changes in v4:
- several new patches
- some changes to other patches listed in their changelogs

Changes in v3:
- rebased
- several new patches, starting with 13/25 "tools/libs/util: cleanup Makefile"
- introducing macros to deal with linking with in-tree xen libraries
- Add -Werror to CFLAGS for all builds in tools/

Changes in v2:
- one new patch
- other changes described in patch notes

Acked-by: Christian Lindig <christian.lindig@citrix.com<mailto:christian.lindig@citrix.com>>