1. Patchew
  2. QEMU
  3. [Qemu-devel] [PATCH v7 0/6] Initial support for keycodemapdb GIT submodule
  4. View patch

[Qemu-devel] [PATCH v7 2/6] docker: don't rely on submodules existing in the main checkout

Daniel P. Berrange posted 6 patches 8 years, 4 months ago
There is a newer version of this series
[Qemu-devel] [PATCH v7 2/6] docker: don't rely on submodules existing in the main checkout
Posted by Daniel P. Berrange 8 years, 4 months ago 
When building the tarball to pass into the docker/vm test image,
the code relies on the git submodules being checked out in the
main checkout.

ie if the developer has not run 'git submodule update --init dtc'
many of the docker tests will fail due to the libfdt package not
being present in the test images. Patchew manually checks out the
dtc submodule in the main git checkout, but this is a bad idea.

When running tests we want to have a predictable set of submodules
included in the source that's tested. The build environment is
completely independant of the developers host OS, so the submodules
the developer has checked out should not be considered relevant for
the tests.

This changes the archive-source.sh script so that it clones the
current git checkout into a temporary directory, checks out a
fixed set of submodules, builds the tarball and finally removes
the temporary git clone.

Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
---
 scripts/archive-source.sh | 30 +++++++++++++++++++++++++++---
 1 file changed, 27 insertions(+), 3 deletions(-)

diff --git a/scripts/archive-source.sh b/scripts/archive-source.sh
index c4e7d98f4d..0c06af0d00 100755
--- a/scripts/archive-source.sh
+++ b/scripts/archive-source.sh
@@ -18,9 +18,15 @@ if test $# -lt 1; then
     error "Usage: $0 <output tarball>"
 fi
 
-tar_file="$1"
-list_file="$1.list"
-submodules=$(git submodule foreach --recursive --quiet 'echo $name')
+tar_file=`realpath "$1"`
+list_file="${tar_file}.list"
+vroot_dir="${tar_file}.vroot"
+
+# We want a predictable list of submodules for builds, that is
+# independant of what the developer currently has initialized
+# in their checkout, because the build environment is completely
+# different to the host OS.
+submodules="dtc"
 
 if test $? -ne 0; then
     error "git submodule command failed"
@@ -28,6 +34,21 @@ fi
 
 trap "status=$?; rm -f \"$list_file\"; exit \$status" 0 1 2 3 15
 
+if git diff-index --quiet HEAD -- &>/dev/null
+then
+    HEAD=HEAD
+else
+    HEAD=`git stash create`
+fi
+git clone --shared . "$vroot_dir"
+here=`pwd`
+cd "$vroot_dir"
+git checkout $HEAD
+
+for sm in $submodules; do
+    git submodule update --init $sm
+done
+
 if test -n "$submodules"; then
     {
         git ls-files || error "git ls-files failed"
@@ -48,4 +69,7 @@ fi
 
 tar -cf "$tar_file" -T "$list_file" || error "failed to create tar file"
 
+cd "$here"
+rm -rf "$vroot_dir"
+
 exit 0
-- 
2.13.5
Re: [Qemu-devel] [PATCH v7 2/6] docker: don't rely on submodules existing in the main checkout
Posted by Eric Blake 8 years, 4 months ago 
On 09/28/2017 07:06 AM, Daniel P. Berrange wrote:
> When building the tarball to pass into the docker/vm test image,
> the code relies on the git submodules being checked out in the
> main checkout.
> 
> ie if the developer has not run 'git submodule update --init dtc'
> many of the docker tests will fail due to the libfdt package not

'dtc' vs. 'libfdt' - is one of the acronyms wrong?

> being present in the test images. Patchew manually checks out the
> dtc submodule in the main git checkout, but this is a bad idea.
> 
> When running tests we want to have a predictable set of submodules
> included in the source that's tested. The build environment is
> completely independant of the developers host OS, so the submodules

s/independant/independent/

> the developer has checked out should not be considered relevant for
> the tests.
> 
> This changes the archive-source.sh script so that it clones the
> current git checkout into a temporary directory, checks out a
> fixed set of submodules, builds the tarball and finally removes
> the temporary git clone.
> 
> Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
> ---
>  scripts/archive-source.sh | 30 +++++++++++++++++++++++++++---
>  1 file changed, 27 insertions(+), 3 deletions(-)
> 

> +++ b/scripts/archive-source.sh

> +# We want a predictable list of submodules for builds, that is
> +# independant of what the developer currently has initialized

and again

> +# in their checkout, because the build environment is completely
> +# different to the host OS.
> +submodules="dtc"
>  
>  if test $? -ne 0; then
>      error "git submodule command failed"
> @@ -28,6 +34,21 @@ fi
>  
>  trap "status=$?; rm -f \"$list_file\"; exit \$status" 0 1 2 3 15
>  
> +if git diff-index --quiet HEAD -- &>/dev/null
> +then
> +    HEAD=HEAD
> +else
> +    HEAD=`git stash create`
> +fi
> +git clone --shared . "$vroot_dir"

set -e is not in effect; do you need error checking here?

> +here=`pwd`

$PWD is cheaper than spawning a subprocess for `pwd`.

> +cd "$vroot_dir"

Definitely need error checking here.

> +git checkout $HEAD
> +
> +for sm in $submodules; do
> +    git submodule update --init $sm
> +done
> +
>  if test -n "$submodules"; then
>      {
>          git ls-files || error "git ls-files failed"
> @@ -48,4 +69,7 @@ fi
>  
>  tar -cf "$tar_file" -T "$list_file" || error "failed to create tar file"
>  
> +cd "$here"

and again here

> +rm -rf "$vroot_dir"
> +
>  exit 0
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org
Re: [Qemu-devel] [PATCH v7 2/6] docker: don't rely on submodules existing in the main checkout
Posted by Daniel P. Berrange 8 years, 4 months ago 
On Thu, Sep 28, 2017 at 01:03:41PM -0500, Eric Blake wrote:
> On 09/28/2017 07:06 AM, Daniel P. Berrange wrote:
> > When building the tarball to pass into the docker/vm test image,
> > the code relies on the git submodules being checked out in the
> > main checkout.
> > 
> > ie if the developer has not run 'git submodule update --init dtc'
> > many of the docker tests will fail due to the libfdt package not
> 
> 'dtc' vs. 'libfdt' - is one of the acronyms wrong?

For reasons I don't understand we called our submodule 'dtc'
but it builds a libfdt library. So at least my wording here
reflects that mismatch.

> 
> > being present in the test images. Patchew manually checks out the
> > dtc submodule in the main git checkout, but this is a bad idea.
> > 
> > When running tests we want to have a predictable set of submodules
> > included in the source that's tested. The build environment is
> > completely independant of the developers host OS, so the submodules
> 
> s/independant/independent/
> 
> > the developer has checked out should not be considered relevant for
> > the tests.
> > 
> > This changes the archive-source.sh script so that it clones the
> > current git checkout into a temporary directory, checks out a
> > fixed set of submodules, builds the tarball and finally removes
> > the temporary git clone.
> > 
> > Signed-off-by: Daniel P. Berrange <berrange@redhat.com>
> > ---
> >  scripts/archive-source.sh | 30 +++++++++++++++++++++++++++---
> >  1 file changed, 27 insertions(+), 3 deletions(-)
> > 
> 
> > +++ b/scripts/archive-source.sh
> 
> > +# We want a predictable list of submodules for builds, that is
> > +# independant of what the developer currently has initialized
> 
> and again
> 
> > +# in their checkout, because the build environment is completely
> > +# different to the host OS.
> > +submodules="dtc"
> >  
> >  if test $? -ne 0; then
> >      error "git submodule command failed"
> > @@ -28,6 +34,21 @@ fi
> >  
> >  trap "status=$?; rm -f \"$list_file\"; exit \$status" 0 1 2 3 15
> >  
> > +if git diff-index --quiet HEAD -- &>/dev/null
> > +then
> > +    HEAD=HEAD
> > +else
> > +    HEAD=`git stash create`
> > +fi
> > +git clone --shared . "$vroot_dir"
> 
> set -e is not in effect; do you need error checking here?

Yes we shouldl

> 
> > +here=`pwd`
> 
> $PWD is cheaper than spawning a subprocess for `pwd`.

I always forget about $PWD :-)

> > +cd "$vroot_dir"
> 
> Definitely need error checking here.

Yep

> 
> > +git checkout $HEAD
> > +
> > +for sm in $submodules; do
> > +    git submodule update --init $sm

And here too

> > +done
> > +
> >  if test -n "$submodules"; then
> >      {
> >          git ls-files || error "git ls-files failed"
> > @@ -48,4 +69,7 @@ fi
> >  
> >  tar -cf "$tar_file" -T "$list_file" || error "failed to create tar file"
> >  
> > +cd "$here"
> 
> and again here
> 
> > +rm -rf "$vroot_dir"

I can move this into the earlier 'trap' cleanup and remove
this need to cd && rm.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Re: [Qemu-devel] [PATCH v7 2/6] docker: don't rely on submodules existing in the main checkout
Posted by Peter Maydell 8 years, 4 months ago 
On 29 September 2017 at 10:14, Daniel P. Berrange <berrange@redhat.com> wrote:
> On Thu, Sep 28, 2017 at 01:03:41PM -0500, Eric Blake wrote:
>> On 09/28/2017 07:06 AM, Daniel P. Berrange wrote:
>> > When building the tarball to pass into the docker/vm test image,
>> > the code relies on the git submodules being checked out in the
>> > main checkout.
>> >
>> > ie if the developer has not run 'git submodule update --init dtc'
>> > many of the docker tests will fail due to the libfdt package not
>>
>> 'dtc' vs. 'libfdt' - is one of the acronyms wrong?
>
> For reasons I don't understand we called our submodule 'dtc'
> but it builds a libfdt library. So at least my wording here
> reflects that mismatch.

'dtc' is the name of the upstream project, whose sources can
build various things; the specific part we want happens to be
libfdt.

thanks
-- PMM

Patchew 2.1-devel-b67e4c2

© 2016 - 2026 Red Hat, Inc.