When building the tarball to pass into the docker/vm test image,
the code relies on the git submodules being checked out in the
main checkout.

ie if the developer has not run 'git submodule update --init dtc'
most (all?) of the docker tests will fail due to the missing dtc
package in the test images. Patchew manually checks out the dtc
submodule in the main git checkout, but this is a bad idea. The
docker tests should never mess around with the developer's main
GIT checkout.

When running tests we want to have a predictable set of submodules
included in the source that's tested. The build environment is
completely independant of the developers host OS, so the submodules
the developer has checked out should not be considered relevant for
the tests.

This changes the archive-source.sh script so that it clones the
current git checkout into a temporary directory, checks out a
fixed set of submodules, builds the tarball and finally removes
the temporary git clone.

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
---
 scripts/archive-source.sh | 23 ++++++++++++++++++++---
 1 file changed, 20 insertions(+), 3 deletions(-)

diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
index c4e7d98f4d..0d2046a80e 100755
--- a/scripts/archive-source.sh
+++ b/scripts/archive-source.sh
@@ -18,9 +18,15 @@ if test $# -lt 1; then
     error "Usage: $0 <output tarball>"
 fi
 
-tar_file="$1"
-list_file="$1.list"
-submodules=$(git submodule foreach --recursive --quiet 'echo $name')
+tar_file=`realpath "$1"`
+list_file="${tar_file}.list"
+vroot_dir="${tar_file}.vroot"
+
+# We want a predictable list of submodules for builds, that is
+# independant of what the developer currently has initialized
+# in their checkout, because the build environment is completely
+# different to the host OS.
+submodules="dtc"
 
 if test $? -ne 0; then
     error "git submodule command failed"
@@ -28,6 +34,14 @@ fi
 
 trap "status=$?; rm -f \"$list_file\"; exit \$status" 0 1 2 3 15
 
+git clone --shared . "$vroot_dir"
+here=`pwd`
+cd "$vroot_dir"
+
+for sm in $submodules; do
+    git submodule update --init $sm
+done
+
 if test -n "$submodules"; then
     {
         git ls-files || error "git ls-files failed"
@@ -48,4 +62,7 @@ fi
 
 tar -cf "$tar_file" -T "$list_file" || error "failed to create tar file"
 
+cd "$here"
+rm -rf "$vroot_dir"
+
 exit 0
-- 
2.13.5

On Thu, 09/28 09:44, Daniel P. Berrange wrote:
> When building the tarball to pass into the docker/vm test image,
> the code relies on the git submodules being checked out in the
> main checkout.
> 
> ie if the developer has not run 'git submodule update --init dtc'
> most (all?) of the docker tests will fail due to the missing dtc
> package in the test images.

Not all, fail only in envs/configs where libfdt devel doesn't exist. But I'm not
sure about this patch (*).

> Patchew manually checks out the dtc
> submodule in the main git checkout, but this is a bad idea. The
> docker tests should never mess around with the developer's main
> GIT checkout.

Or in general, scripts should not mess around with developer's GIT checkouts,
which is the reason ./configure doesn't do it?

How about we clone/checkout dtc into a temporary dir from ./configure
automatically instead?

> 
> When running tests we want to have a predictable set of submodules
> included in the source that's tested. The build environment is
> completely independant of the developers host OS, so the submodules
> the developer has checked out should not be considered relevant for
> the tests.
> 
> This changes the archive-source.sh script so that it clones the
> current git checkout into a temporary directory, checks out a
> fixed set of submodules, builds the tarball and finally removes
> the temporary git clone.
> 
> Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
> ---
>  scripts/archive-source.sh | 23 ++++++++++++++++++++---
>  1 file changed, 20 insertions(+), 3 deletions(-)
> 
> diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
> index c4e7d98f4d..0d2046a80e 100755
> --- a/scripts/archive-source.sh
> +++ b/scripts/archive-source.sh
> @@ -18,9 +18,15 @@ if test $# -lt 1; then
>      error "Usage: $0 <output tarball>"
>  fi
>  
> -tar_file="$1"
> -list_file="$1.list"
> -submodules=$(git submodule foreach --recursive --quiet 'echo $name')
> +tar_file=`realpath "$1"`
> +list_file="${tar_file}.list"
> +vroot_dir="${tar_file}.vroot"
> +
> +# We want a predictable list of submodules for builds, that is
> +# independant of what the developer currently has initialized
> +# in their checkout, because the build environment is completely
> +# different to the host OS.
> +submodules="dtc"
>  
>  if test $? -ne 0; then
>      error "git submodule command failed"
> @@ -28,6 +34,14 @@ fi
>  
>  trap "status=$?; rm -f \"$list_file\"; exit \$status" 0 1 2 3 15
>  
> +git clone --shared . "$vroot_dir"

(*) This loses the feature that uncommitted changes are also reflected in the
tarball. Personally I rarely use it but since day one of docker tests we've
deliberately supported that.

Cc'ing Alex who originally implemented that with "git diff-index" and "git
archive".

Fam

On Thu, Sep 28, 2017 at 05:44:53PM +0800, Fam Zheng wrote:
> On Thu, 09/28 09:44, Daniel P. Berrange wrote:
> > When building the tarball to pass into the docker/vm test image,
> > the code relies on the git submodules being checked out in the
> > main checkout.
> > 
> > ie if the developer has not run 'git submodule update --init dtc'
> > most (all?) of the docker tests will fail due to the missing dtc
> > package in the test images.
> 
> Not all, fail only in envs/configs where libfdt devel doesn't exist.

I didn't test all of them but of the ~5 I did test all failed, so I
just assumed none contain libftd-devel, since you have patchew
manually check it out.

> > Patchew manually checks out the dtc
> > submodule in the main git checkout, but this is a bad idea. The
> > docker tests should never mess around with the developer's main
> > GIT checkout.
> 
> Or in general, scripts should not mess around with developer's GIT checkouts,
> which is the reason ./configure doesn't do it?
> 
> How about we clone/checkout dtc into a temporary dir from ./configure
> automatically instead?
> 
> > 
> > When running tests we want to have a predictable set of submodules
> > included in the source that's tested. The build environment is
> > completely independant of the developers host OS, so the submodules
> > the developer has checked out should not be considered relevant for
> > the tests.
> > 
> > This changes the archive-source.sh script so that it clones the
> > current git checkout into a temporary directory, checks out a
> > fixed set of submodules, builds the tarball and finally removes
> > the temporary git clone.
> > 
> > Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
> > ---
> >  scripts/archive-source.sh | 23 ++++++++++++++++++++---
> >  1 file changed, 20 insertions(+), 3 deletions(-)
> > 
> > diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
> > index c4e7d98f4d..0d2046a80e 100755
> > --- a/scripts/archive-source.sh
> > +++ b/scripts/archive-source.sh
> > @@ -18,9 +18,15 @@ if test $# -lt 1; then
> >      error "Usage: $0 <output tarball>"
> >  fi
> >  
> > -tar_file="$1"
> > -list_file="$1.list"
> > -submodules=$(git submodule foreach --recursive --quiet 'echo $name')
> > +tar_file=`realpath "$1"`
> > +list_file="${tar_file}.list"
> > +vroot_dir="${tar_file}.vroot"
> > +
> > +# We want a predictable list of submodules for builds, that is
> > +# independant of what the developer currently has initialized
> > +# in their checkout, because the build environment is completely
> > +# different to the host OS.
> > +submodules="dtc"
> >  
> >  if test $? -ne 0; then
> >      error "git submodule command failed"
> > @@ -28,6 +34,14 @@ fi
> >  
> >  trap "status=$?; rm -f \"$list_file\"; exit \$status" 0 1 2 3 15
> >  
> > +git clone --shared . "$vroot_dir"
> 
> (*) This loses the feature that uncommitted changes are also reflected in the
> tarball. Personally I rarely use it but since day one of docker tests we've
> deliberately supported that.

Oh, I missed that we supported that.  We can still cope with that by
using the 'git stash create' trick the docker script used before. e.g.

   stash=`git stash create`
   git clone --shared . $vroot_dir
   cd $vroot_dir
   git checkout $stash

the cloned repo contains the stash object, because we use --shared to
directly share the .git dir contents.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Thu, 09/28 10:52, Daniel P. Berrange wrote:
> On Thu, Sep 28, 2017 at 05:44:53PM +0800, Fam Zheng wrote:
> > On Thu, 09/28 09:44, Daniel P. Berrange wrote:
> > > When building the tarball to pass into the docker/vm test image,
> > > the code relies on the git submodules being checked out in the
> > > main checkout.
> > > 
> > > ie if the developer has not run 'git submodule update --init dtc'
> > > most (all?) of the docker tests will fail due to the missing dtc
> > > package in the test images.
> > 
> > Not all, fail only in envs/configs where libfdt devel doesn't exist.
> 
> I didn't test all of them but of the ~5 I did test all failed, so I
> just assumed none contain libftd-devel, since you have patchew
> manually check it out.

Patchew does it just because mingw needs it, this is not required for native
build:

$ git grep libfdt tests/docker
tests/docker/dockerfiles/centos6.docker:    libfdt-devel \
tests/docker/dockerfiles/centos7.docker:    libfdt-devel \
tests/docker/dockerfiles/fedora.docker:    glib2-devel pixman-devel zlib-devel SDL-devel libfdt-devel \
tests/docker/dockerfiles/min-glib.docker:RUN yum install -y libfdt-devel ccache
tests/docker/dockerfiles/ubuntu.docker:    libspice-protocol-dev libnss3-dev libfdt-dev \

Fam

On Thu, Sep 28, 2017 at 2:44 AM, Fam Zheng <famz@redhat.com> wrote:
> On Thu, 09/28 09:44, Daniel P. Berrange wrote:
>> When building the tarball to pass into the docker/vm test image,
>> the code relies on the git submodules being checked out in the
>> main checkout.
>>
>> ie if the developer has not run 'git submodule update --init dtc'
>> most (all?) of the docker tests will fail due to the missing dtc
>> package in the test images.
>
> Not all, fail only in envs/configs where libfdt devel doesn't exist. But I'm not
> sure about this patch (*).
>
>> Patchew manually checks out the dtc
>> submodule in the main git checkout, but this is a bad idea. The
>> docker tests should never mess around with the developer's main
>> GIT checkout.
>
> Or in general, scripts should not mess around with developer's GIT checkouts,
> which is the reason ./configure doesn't do it?
>
> How about we clone/checkout dtc into a temporary dir from ./configure
> automatically instead?

I like this! Submodules cause lots of issues for us. Other users
building QEMU don't expect to have to clone submodules and we get
complaints back to the QEMU. Especially DTC which is always required.

Thanks,
Alistair