[PULL 05/10] tests: Evict stale files in the functional download cache after a while

Thomas Huth posted 10 patches 4 weeks, 1 day ago
Maintainers: Yonggang Luo <luoyonggang@gmail.com>, "Alex Bennée" <alex.bennee@linaro.org>, "Philippe Mathieu-Daudé" <philmd@linaro.org>, Thomas Huth <thuth@redhat.com>, John Snow <jsnow@redhat.com>, Cleber Rosa <crosa@redhat.com>, "Daniel P. Berrangé" <berrange@redhat.com>, Richard Henderson <richard.henderson@linaro.org>, David Hildenbrand <david@redhat.com>, Ilya Leoshkevich <iii@linux.ibm.com>, Paolo Bonzini <pbonzini@redhat.com>, Radoslaw Biernacki <rad@semihalf.com>, Peter Maydell <peter.maydell@linaro.org>, Leif Lindholm <leif.lindholm@oss.qualcomm.com>
[PULL 05/10] tests: Evict stale files in the functional download cache after a while
Posted by Thomas Huth 4 weeks, 1 day ago
From: Thomas Huth <thuth@redhat.com>

The download cache of the functional tests is currently only growing.
But sometimes tests get removed or changed to use different assets,
thus we should clean up the stale old assets after a while when they
are not in use anymore. So add a script that looks at the time stamps
of the assets and removes them if they haven't been touched for more
than half of a year. Since there might also be some assets around that
have been added to the cache before we added the time stamp files,
assume a default time stamp that is close to the creation date of this
patch, so that we don't delete these files too early (so we still have
all assets around in case we have to bisect an issue in the recent past
of QEMU).

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-ID: <20251014083424.103202-3-thuth@redhat.com>
---
 MAINTAINERS                       |  1 +
 scripts/clean_functional_cache.py | 45 +++++++++++++++++++++++++++++++
 tests/Makefile.include            |  1 +
 3 files changed, 47 insertions(+)
 create mode 100755 scripts/clean_functional_cache.py

diff --git a/MAINTAINERS b/MAINTAINERS
index 0c766961f39..667acd933c7 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4397,6 +4397,7 @@ M: Thomas Huth <thuth@redhat.com>
 R: Philippe Mathieu-Daudé <philmd@linaro.org>
 R: Daniel P. Berrange <berrange@redhat.com>
 F: docs/devel/testing/functional.rst
+F: scripts/clean_functional_cache.py
 F: tests/functional/qemu_test/
 
 Windows Hosted Continuous Integration
diff --git a/scripts/clean_functional_cache.py b/scripts/clean_functional_cache.py
new file mode 100755
index 00000000000..c3370ffbb87
--- /dev/null
+++ b/scripts/clean_functional_cache.py
@@ -0,0 +1,45 @@
+#!/usr/bin/env python3
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+"""Delete stale assets from the download cache of the functional tests"""
+
+import os
+import stat
+import sys
+import time
+from pathlib import Path
+
+
+cache_dir_env = os.getenv('QEMU_TEST_CACHE_DIR')
+if cache_dir_env:
+    cache_dir = Path(cache_dir_env, "download")
+else:
+    cache_dir = Path(Path("~").expanduser(), ".cache", "qemu", "download")
+
+if not cache_dir.exists():
+    print(f"Cache dir {cache_dir} does not exist!", file=sys.stderr)
+    sys.exit(1)
+
+os.chdir(cache_dir)
+
+for file in cache_dir.iterdir():
+    # Only consider the files that use a sha256 as filename:
+    if len(file.name) != 64:
+        continue
+
+    try:
+        timestamp = int(file.with_suffix(".stamp").read_text())
+    except FileNotFoundError:
+        # Assume it's an old file that was already in the cache before we
+        # added the code for evicting stale assets. Use the release date
+        # of QEMU v10.1 as a default timestamp.
+        timestamp = time.mktime((2025, 8, 26, 0, 0, 0, 0, 0, 0))
+
+    age = time.time() - timestamp
+
+    # Delete files older than half of a year (183 days * 24h * 60m * 60s)
+    if age > 15811200:
+        print(f"Removing {cache_dir}/{file.name}.")
+        file.chmod(stat.S_IWRITE)
+        file.unlink()
diff --git a/tests/Makefile.include b/tests/Makefile.include
index e47ef4d45c9..d4dfbf3716d 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -111,6 +111,7 @@ $(FUNCTIONAL_TARGETS): check-venv
 .PHONY: check-functional
 check-functional: check-venv
 	@$(NINJA) precache-functional
+	@$(PYTHON) $(SRC_PATH)/scripts/clean_functional_cache.py
 	@QEMU_TEST_NO_DOWNLOAD=1 $(MAKE) SPEED=thorough check-func check-func-quick
 
 .PHONY: check-func check-func-quick
-- 
2.51.0