[v1] Clean up the functional download cache after some months

[PATCH 1/2] tests/functional: Set current time stamp of assets when using them

Posted by Thomas Huth 1 month ago

From: Thomas Huth <thuth@redhat.com>

We are going to remove obsolete assets from the cache, so keep
the time stamps of the assets that we use up-to-date to have a way
to detect stale assets later.

Signed-off-by: Thomas Huth <thuth@redhat.com>
---
 tests/functional/qemu_test/asset.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_test/asset.py
index 2971a989d1e..251953ed99f 100644
--- a/tests/functional/qemu_test/asset.py
+++ b/tests/functional/qemu_test/asset.py
@@ -10,6 +10,7 @@
 import os
 import stat
 import sys
+import time
 import unittest
 import urllib.request
 from time import sleep
@@ -113,6 +114,11 @@ def _wait_for_other_download(self, tmp_cache_file):
         self.log.debug("Time out while waiting for %s!", tmp_cache_file)
         raise
 
+    def _save_time_stamp(self):
+        with open(self.cache_file.with_suffix(".stamp"), 'w',
+                  encoding='utf-8') as fh:
+            fh.write(f"{int(time.time())}")
+
     def fetch(self):
         if not self.cache_dir.exists():
             self.cache_dir.mkdir(parents=True, exist_ok=True)
@@ -120,6 +126,7 @@ def fetch(self):
         if self.valid():
             self.log.debug("Using cached asset %s for %s",
                            self.cache_file, self.url)
+            self._save_time_stamp()
             return str(self.cache_file)
 
         if not self.fetchable():
@@ -208,6 +215,7 @@ def fetch(self):
             tmp_cache_file.unlink()
             raise AssetError(self, "Hash does not match %s" % self.hash)
         tmp_cache_file.replace(self.cache_file)
+        self._save_time_stamp()
         # Remove write perms to stop tests accidentally modifying them
         os.chmod(self.cache_file, stat.S_IRUSR | stat.S_IRGRP)
 
-- 
2.51.0

Re: [PATCH 1/2] tests/functional: Set current time stamp of assets when using them

Posted by Daniel P. Berrangé 1 month ago

On Fri, Oct 10, 2025 at 11:32:42AM +0200, Thomas Huth wrote:
> From: Thomas Huth <thuth@redhat.com>
> 
> We are going to remove obsolete assets from the cache, so keep
> the time stamps of the assets that we use up-to-date to have a way
> to detect stale assets later.
> 
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
>  tests/functional/qemu_test/asset.py | 8 ++++++++
>  1 file changed, 8 insertions(+)
> 
> diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_test/asset.py
> index 2971a989d1e..251953ed99f 100644
> --- a/tests/functional/qemu_test/asset.py
> +++ b/tests/functional/qemu_test/asset.py
> @@ -10,6 +10,7 @@
>  import os
>  import stat
>  import sys
> +import time
>  import unittest
>  import urllib.request
>  from time import sleep
> @@ -113,6 +114,11 @@ def _wait_for_other_download(self, tmp_cache_file):
>          self.log.debug("Time out while waiting for %s!", tmp_cache_file)
>          raise
>  
> +    def _save_time_stamp(self):
> +        with open(self.cache_file.with_suffix(".stamp"), 'w',
> +                  encoding='utf-8') as fh:
> +            fh.write(f"{int(time.time())}")

Rather than creating a parallel timestamp file, it feels like we could
just call  'os.utime(self.cache_file)' which will set atime + mtime
to the current timestamp, which we can check later with os.stat().

> +
>      def fetch(self):
>          if not self.cache_dir.exists():
>              self.cache_dir.mkdir(parents=True, exist_ok=True)
> @@ -120,6 +126,7 @@ def fetch(self):
>          if self.valid():
>              self.log.debug("Using cached asset %s for %s",
>                             self.cache_file, self.url)
> +            self._save_time_stamp()
>              return str(self.cache_file)
>  
>          if not self.fetchable():
> @@ -208,6 +215,7 @@ def fetch(self):
>              tmp_cache_file.unlink()
>              raise AssetError(self, "Hash does not match %s" % self.hash)
>          tmp_cache_file.replace(self.cache_file)
> +        self._save_time_stamp()
>          # Remove write perms to stop tests accidentally modifying them
>          os.chmod(self.cache_file, stat.S_IRUSR | stat.S_IRGRP)
>  
> -- 
> 2.51.0
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 1/2] tests/functional: Set current time stamp of assets when using them

Posted by Thomas Huth 1 month ago

On 10/10/2025 11.39, Daniel P. Berrangé wrote:
> On Fri, Oct 10, 2025 at 11:32:42AM +0200, Thomas Huth wrote:
>> From: Thomas Huth <thuth@redhat.com>
>>
>> We are going to remove obsolete assets from the cache, so keep
>> the time stamps of the assets that we use up-to-date to have a way
>> to detect stale assets later.
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>   tests/functional/qemu_test/asset.py | 8 ++++++++
>>   1 file changed, 8 insertions(+)
>>
>> diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_test/asset.py
>> index 2971a989d1e..251953ed99f 100644
>> --- a/tests/functional/qemu_test/asset.py
>> +++ b/tests/functional/qemu_test/asset.py
>> @@ -10,6 +10,7 @@
>>   import os
>>   import stat
>>   import sys
>> +import time
>>   import unittest
>>   import urllib.request
>>   from time import sleep
>> @@ -113,6 +114,11 @@ def _wait_for_other_download(self, tmp_cache_file):
>>           self.log.debug("Time out while waiting for %s!", tmp_cache_file)
>>           raise
>>   
>> +    def _save_time_stamp(self):
>> +        with open(self.cache_file.with_suffix(".stamp"), 'w',
>> +                  encoding='utf-8') as fh:
>> +            fh.write(f"{int(time.time())}")
> 
> Rather than creating a parallel timestamp file, it feels like we could
> just call  'os.utime(self.cache_file)' which will set atime + mtime
> to the current timestamp, which we can check later with os.stat().

That was my first idea, too (sorry, I should maybe have mentioned it in the 
patch description), but it does not work: In the gitlab CI runners, the 
files get re-initialized from the runners cache each time the functional 
test job is started, so the atime and mtime is always close to the the 
current date there --> we would never expire files in the gitlab CI that way.

  Thomas

Re: [PATCH 1/2] tests/functional: Set current time stamp of assets when using them

Posted by Daniel P. Berrangé 1 month ago

On Fri, Oct 10, 2025 at 11:46:44AM +0200, Thomas Huth wrote:
> On 10/10/2025 11.39, Daniel P. Berrangé wrote:
> > On Fri, Oct 10, 2025 at 11:32:42AM +0200, Thomas Huth wrote:
> > > From: Thomas Huth <thuth@redhat.com>
> > > 
> > > We are going to remove obsolete assets from the cache, so keep
> > > the time stamps of the assets that we use up-to-date to have a way
> > > to detect stale assets later.
> > > 
> > > Signed-off-by: Thomas Huth <thuth@redhat.com>
> > > ---
> > >   tests/functional/qemu_test/asset.py | 8 ++++++++
> > >   1 file changed, 8 insertions(+)
> > > 
> > > diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_test/asset.py
> > > index 2971a989d1e..251953ed99f 100644
> > > --- a/tests/functional/qemu_test/asset.py
> > > +++ b/tests/functional/qemu_test/asset.py
> > > @@ -10,6 +10,7 @@
> > >   import os
> > >   import stat
> > >   import sys
> > > +import time
> > >   import unittest
> > >   import urllib.request
> > >   from time import sleep
> > > @@ -113,6 +114,11 @@ def _wait_for_other_download(self, tmp_cache_file):
> > >           self.log.debug("Time out while waiting for %s!", tmp_cache_file)
> > >           raise
> > > +    def _save_time_stamp(self):
> > > +        with open(self.cache_file.with_suffix(".stamp"), 'w',
> > > +                  encoding='utf-8') as fh:
> > > +            fh.write(f"{int(time.time())}")
> > 
> > Rather than creating a parallel timestamp file, it feels like we could
> > just call  'os.utime(self.cache_file)' which will set atime + mtime
> > to the current timestamp, which we can check later with os.stat().
> 
> That was my first idea, too (sorry, I should maybe have mentioned it in the
> patch description), but it does not work: In the gitlab CI runners, the
> files get re-initialized from the runners cache each time the functional
> test job is started, so the atime and mtime is always close to the the
> current date there --> we would never expire files in the gitlab CI that
> way.

Oh well, I had assumed gitlab would preserve timestamps when taring/untaring
files for the cache :-(


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|