Tests: introduce custom jobs

[PATCH 1/8] Acceptance Jobs: preserve the cache for pip on GitLab CI

Posted by Cleber Rosa 4 years, 7 months ago

The acceptance jobs (via `make check-venv`) will setup a virtual
environment, and after that install packages defined in
tests/requirements.txt via pip.

Let's enable pip's default cache directory, so that we can save
a bit on time/bandwidth.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 .gitlab-ci.yml | 1 +
 1 file changed, 1 insertion(+)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 52d65d6c04..9cc4676912 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -53,6 +53,7 @@ include:
     key: "${CI_JOB_NAME}-cache"
     paths:
       - ${CI_PROJECT_DIR}/avocado-cache
+      - ~/.cache/pip
     policy: pull-push
   artifacts:
     name: "$CI_JOB_NAME-$CI_COMMIT_REF_SLUG"
-- 
2.25.4

Re: [PATCH 1/8] Acceptance Jobs: preserve the cache for pip on GitLab CI

Posted by Thomas Huth 4 years, 7 months ago

On 15/04/2021 23.51, Cleber Rosa wrote:
> The acceptance jobs (via `make check-venv`) will setup a virtual
> environment, and after that install packages defined in
> tests/requirements.txt via pip.
> 
> Let's enable pip's default cache directory, so that we can save
> a bit on time/bandwidth.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>   .gitlab-ci.yml | 1 +
>   1 file changed, 1 insertion(+)
> 
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index 52d65d6c04..9cc4676912 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -53,6 +53,7 @@ include:
>       key: "${CI_JOB_NAME}-cache"
>       paths:
>         - ${CI_PROJECT_DIR}/avocado-cache
> +      - ~/.cache/pip

Did you check whether this works? AFAIK the cache directories have to be 
part of the project directory, see:

  https://docs.gitlab.com/ee/ci/yaml/README.html#cache

We already tried to cache ~/avocado/data/cache in the past, but it did not 
work and we had to move the cache manually to the current working directory 
(see commit 5896c539547).

  Thomas

Re: [PATCH 1/8] Acceptance Jobs: preserve the cache for pip on GitLab CI

Posted by Cleber Rosa 4 years, 7 months ago

On Fri, Apr 16, 2021 at 05:56:10AM +0200, Thomas Huth wrote:
> On 15/04/2021 23.51, Cleber Rosa wrote:
> > The acceptance jobs (via `make check-venv`) will setup a virtual
> > environment, and after that install packages defined in
> > tests/requirements.txt via pip.
> > 
> > Let's enable pip's default cache directory, so that we can save
> > a bit on time/bandwidth.
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > ---
> >   .gitlab-ci.yml | 1 +
> >   1 file changed, 1 insertion(+)
> > 
> > diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> > index 52d65d6c04..9cc4676912 100644
> > --- a/.gitlab-ci.yml
> > +++ b/.gitlab-ci.yml
> > @@ -53,6 +53,7 @@ include:
> >       key: "${CI_JOB_NAME}-cache"
> >       paths:
> >         - ${CI_PROJECT_DIR}/avocado-cache
> > +      - ~/.cache/pip
> 
> Did you check whether this works? AFAIK the cache directories have to be
> part of the project directory, see:
> 
>  https://docs.gitlab.com/ee/ci/yaml/README.html#cache
> 
> We already tried to cache ~/avocado/data/cache in the past, but it did not
> work and we had to move the cache manually to the current working directory
> (see commit 5896c539547).
> 
>  Thomas

You're absolutely right, it won't work like that.  My bad.

I was trying to avoid having to set variables or configurations, but
something like the following will be needed:

    before_script:
        - export PIP_CACHE_DIR=${CI_PROJECT_DIR}/pip-cache
        - mkdir -p ${CI_PROJECT_DIR}/pip-cache
    cache:
        paths:
            - ${CI_PROJECT_DIR}/pip-cache

Resulting in:

    Using cached avocado_framework-87.0-py3-none-any.whl (399 kB)

And the likes of:

    https://gitlab.com/cleber.gnu/democi/-/jobs/1186910932#L166

I'll change that on v2.

Thanks!
- Cleber.