From nobody Fri Nov 14 18:16:50 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1760088804; cv=none; d=zohomail.com; s=zohoarc; b=Hy4LuYvSM3XBEV9/d790baND32LEGN4me1MLobQLdKiPjTXMi7PRoB2GjG69sKWanPc6inPoDqCoDagkAWDwIfTEBq5WO4Kxh/0iM9kwqAogxAzatFLJPB3H9WY0jw0Q1FugCxY8JjxGvhnN9AMByMvFii4JM204huFcC6qQUBo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1760088804; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=Mc7w1dD4+slOqf5GdCnR7SEfCdhq4eyMsgtA3XCLZ1E=; b=DNF4Q6el7n1FJGdGaa/8IP6NfdltcSbPo4eG6xwfU0pljyl8XdoC/3XEUyKFI9TVV5Ijw+4wksqpFyl8lXhgOEkRnUb3dY6uFyyF0bx05v1qAeZ9IHILbNoLmvHuWopVaoGr2lIahITzp/D1Jogt/Ffg62PcasMGxQXsFiYD9Ng= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1760088804510870.9127134204134; Fri, 10 Oct 2025 02:33:24 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1v79Ui-0000Q9-Dz; Fri, 10 Oct 2025 05:33:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1v79Ug-0000Px-Lq for qemu-devel@nongnu.org; Fri, 10 Oct 2025 05:32:58 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1v79Ue-0008Rt-JO for qemu-devel@nongnu.org; Fri, 10 Oct 2025 05:32:58 -0400 Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-325-qLMQyyWsMTmw44XS_bpuHg-1; Fri, 10 Oct 2025 05:32:52 -0400 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 248D61800452; Fri, 10 Oct 2025 09:32:51 +0000 (UTC) Received: from thuth-p1g4.redhat.com (unknown [10.44.34.98]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 1C29D180035E; Fri, 10 Oct 2025 09:32:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760088775; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Mc7w1dD4+slOqf5GdCnR7SEfCdhq4eyMsgtA3XCLZ1E=; b=SpFWNwE3SmloyNY90z3ztfZhp/GBXrjrIqw8N7zdsDiL79StGWeVa3zEBa1Yt8VpaEYRFO 8hzyVeIeYrqln/TRnKAVKZE0QrcpseKSjz+nUNLCPxE0z8A52DyOi6tOCTFt65wLJ9dFJr hJQGHt+RnXb0VLcmRH27cnxw36PWZ4E= X-MC-Unique: qLMQyyWsMTmw44XS_bpuHg-1 X-Mimecast-MFC-AGG-ID: qLMQyyWsMTmw44XS_bpuHg_1760088771 From: Thomas Huth To: qemu-devel@nongnu.org, =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Cc: John Snow , =?UTF-8?q?Alex=20Benn=C3=A9e?= , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= Subject: [PATCH 1/2] tests/functional: Set current time stamp of assets when using them Date: Fri, 10 Oct 2025 11:32:42 +0200 Message-ID: <20251010093244.807544-2-thuth@redhat.com> In-Reply-To: <20251010093244.807544-1-thuth@redhat.com> References: <20251010093244.807544-1-thuth@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=thuth@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -24 X-Spam_score: -2.5 X-Spam_bar: -- X-Spam_report: (-2.5 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.438, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1760088807646158500 Content-Type: text/plain; charset="utf-8" From: Thomas Huth We are going to remove obsolete assets from the cache, so keep the time stamps of the assets that we use up-to-date to have a way to detect stale assets later. Signed-off-by: Thomas Huth --- tests/functional/qemu_test/asset.py | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/tests/functional/qemu_test/asset.py b/tests/functional/qemu_te= st/asset.py index 2971a989d1e..251953ed99f 100644 --- a/tests/functional/qemu_test/asset.py +++ b/tests/functional/qemu_test/asset.py @@ -10,6 +10,7 @@ import os import stat import sys +import time import unittest import urllib.request from time import sleep @@ -113,6 +114,11 @@ def _wait_for_other_download(self, tmp_cache_file): self.log.debug("Time out while waiting for %s!", tmp_cache_file) raise =20 + def _save_time_stamp(self): + with open(self.cache_file.with_suffix(".stamp"), 'w', + encoding=3D'utf-8') as fh: + fh.write(f"{int(time.time())}") + def fetch(self): if not self.cache_dir.exists(): self.cache_dir.mkdir(parents=3DTrue, exist_ok=3DTrue) @@ -120,6 +126,7 @@ def fetch(self): if self.valid(): self.log.debug("Using cached asset %s for %s", self.cache_file, self.url) + self._save_time_stamp() return str(self.cache_file) =20 if not self.fetchable(): @@ -208,6 +215,7 @@ def fetch(self): tmp_cache_file.unlink() raise AssetError(self, "Hash does not match %s" % self.hash) tmp_cache_file.replace(self.cache_file) + self._save_time_stamp() # Remove write perms to stop tests accidentally modifying them os.chmod(self.cache_file, stat.S_IRUSR | stat.S_IRGRP) =20 --=20 2.51.0 From nobody Fri Nov 14 18:16:50 2025 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1760088812; cv=none; d=zohomail.com; s=zohoarc; b=PZ8uaeFJYSHHOJCN++/7fYzUzk44rJGOafOoMaeFy127JJwQtC4TU/uFnaeOdC4vYNxw5AGX3XgyRBJ/Ik/UJP17Mc7yJLZXV09qMFg9RPSZ30WTp4pFFXqkSkmE9qyjbk84cVqtz0ZcFTgUvRwbogwniDkPEupe+sX6+VkSMCc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1760088812; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=N+UuhDVO9ypudlBt0lPyEmielLIGAP9FzQZjqJdYmu4=; b=NvrUCVuDCzozDa1hLtj84M+pJlTYRwh5ZBcvuZe3HJi5jCfFKk/VQtlgjfsq+fB2AF/hTMZdugFFCWXvvUOmhdiypnULohiEIOE+qKM9gZPXDv84MKASuZLPENkuZyJwSJ4LGh3X2tNwxU82vyCF8kl3zDOx83nx2TMCbAlh/IM= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1760088812775629.4040090288919; Fri, 10 Oct 2025 02:33:32 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1v79Ul-0000W9-Oc; Fri, 10 Oct 2025 05:33:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1v79Uj-0000Tu-Cn for qemu-devel@nongnu.org; Fri, 10 Oct 2025 05:33:01 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1v79Uf-0008SD-PF for qemu-devel@nongnu.org; Fri, 10 Oct 2025 05:33:00 -0400 Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-64-PoY7MS4QNlO7ZdFSsHBiEA-1; Fri, 10 Oct 2025 05:32:55 -0400 Received: from mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.111]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id C83A51800366; Fri, 10 Oct 2025 09:32:53 +0000 (UTC) Received: from thuth-p1g4.redhat.com (unknown [10.44.34.98]) by mx-prod-int-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id A960D180035E; Fri, 10 Oct 2025 09:32:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1760088776; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=N+UuhDVO9ypudlBt0lPyEmielLIGAP9FzQZjqJdYmu4=; b=e+o/R/Qt/KkpERpnaKhcUB/kRnOgPqUg2V95AIvwlzhSyWFLL6EzcqsglZM56FYJi5mmtU sn5fV25xY4EFioirP2OjFZC+Ma/oWXYKpxUiCOy3jIStU9u0efyUu9Wh4DFnHrzVjgnY3a R5WWEHxlHZSMdTm7ytojR7X2K5c2jYo= X-MC-Unique: PoY7MS4QNlO7ZdFSsHBiEA-1 X-Mimecast-MFC-AGG-ID: PoY7MS4QNlO7ZdFSsHBiEA_1760088774 From: Thomas Huth To: qemu-devel@nongnu.org, =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= Cc: John Snow , =?UTF-8?q?Alex=20Benn=C3=A9e?= , =?UTF-8?q?Philippe=20Mathieu-Daud=C3=A9?= Subject: [PATCH 2/2] tests: Evict stale files in the functional download cache after a while Date: Fri, 10 Oct 2025 11:32:43 +0200 Message-ID: <20251010093244.807544-3-thuth@redhat.com> In-Reply-To: <20251010093244.807544-1-thuth@redhat.com> References: <20251010093244.807544-1-thuth@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.111 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=thuth@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -24 X-Spam_score: -2.5 X-Spam_bar: -- X-Spam_report: (-2.5 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.438, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1760088819763158500 From: Thomas Huth The download cache of the functional tests is currently only growing. But sometimes tests get removed or changed to use different assets, thus we should clean up the stale old assets after a while when they are not in use anymore. So add a script that looks at the time stamps of the assets and removes them if they haven't been touched for more than half of a year. Since there might also be some assets around that have been added to the cache before we added the time stamp files, assume a default time stamp that is close to the creation date of this patch, so that we don't delete these files too early. Signed-off-by: Thomas Huth --- MAINTAINERS | 1 + scripts/clean_functional_cache.py | 47 +++++++++++++++++++++++++++++++ tests/Makefile.include | 1 + 3 files changed, 49 insertions(+) create mode 100755 scripts/clean_functional_cache.py diff --git a/MAINTAINERS b/MAINTAINERS index 84cfd85e1fa..4c468d45337 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -4398,6 +4398,7 @@ M: Thomas Huth R: Philippe Mathieu-Daud=C3=A9 R: Daniel P. Berrange F: docs/devel/testing/functional.rst +F: scripts/clean_functional_cache.py F: tests/functional/qemu_test/ =20 Windows Hosted Continuous Integration diff --git a/scripts/clean_functional_cache.py b/scripts/clean_functional_c= ache.py new file mode 100755 index 00000000000..e5c4d1acaf3 --- /dev/null +++ b/scripts/clean_functional_cache.py @@ -0,0 +1,47 @@ +#!/usr/bin/env python3 +# +# SPDX-License-Identifier: GPL-2.0-or-later +# +"""Delete stale assets from the download cache of the functional tests""" + +import os +import stat +import sys +import time +from pathlib import Path + + +cache_dir_env =3D os.getenv('QEMU_TEST_CACHE_DIR') +if cache_dir_env: + cache_dir =3D Path(cache_dir_env, "download") +else: + cache_dir =3D Path(Path("~").expanduser(), ".cache", "qemu", "download= ") + +if not os.path.exists(cache_dir): + print(f"Cache dir {cache_dir} does not exist!", file=3Dsys.stderr) + sys.exit(1) + +os.chdir(cache_dir) + +for file in os.listdir(cache_dir): + filename =3D os.fsdecode(file) + # Only consider the files that use a sha256 as filename: + if len(filename) !=3D 64: + continue + + try: + with open(filename + ".stamp", "r", encoding=3D'utf-8') as fh: + timestamp =3D int(fh.read()) + except FileNotFoundError: + # Assume it's an old file that was already in the cache before we + # added the code for evicting stale assets. Use the release date + # of QEMU v10.1 as a default timestamp. + timestamp =3D time.mktime((2025, 8, 26, 0, 0, 0, 0, 0, 0)) + + age =3D time.time() - timestamp + + # Delete files older than half of a year (183 days * 24h * 60m * 60s) + if age > 15811200: + print(f"Removing {cache_dir}/{filename}.") + os.chmod(filename, stat.S_IWRITE) + os.remove(filename) diff --git a/tests/Makefile.include b/tests/Makefile.include index e47ef4d45c9..d4dfbf3716d 100644 --- a/tests/Makefile.include +++ b/tests/Makefile.include @@ -111,6 +111,7 @@ $(FUNCTIONAL_TARGETS): check-venv .PHONY: check-functional check-functional: check-venv @$(NINJA) precache-functional + @$(PYTHON) $(SRC_PATH)/scripts/clean_functional_cache.py @QEMU_TEST_NO_DOWNLOAD=3D1 $(MAKE) SPEED=3Dthorough check-func check-func= -quick =20 .PHONY: check-func check-func-quick --=20 2.51.0