From nobody Sun Feb 8 23:36:55 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) client-ip=63.128.21.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1616054981; cv=none; d=zohomail.com; s=zohoarc; b=d44yYgkOZ+MD5EiFhwxO6dHiH5Bl1csGLLYpXYYmURy5uxpKzhKcKjNcfZd9nMLWARuX6Q2cZ0Eg2EQS9CpHiP/ufiW8ROwXPyCTkPBIUvb2fg5datx+stipaICgYAXUT2e21EAhZIGHdFiST4pRveLN+ulz+ep0rzHdPHPjmvc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1616054981; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=c8oIH2dNbEM9/Cqou9HU7XSS8j6M0b0jkJZA6O4sYuM=; b=DXi348c9MHcKvHrkS3mTO/c2TeoI0+zlrRxDp0LfcrJ56fxYcdziQwZ9PxZXYAOdOXuN6OcglBrlT85ku990DDR6YQJpDEyGVZWu+KiL0AsHMCfAFn6KUvSRGXBA+d+EmMONB/HAwqfuNbdx5CT4GViXhwW5Y/zR7HHbPmTtsJ8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) by mx.zohomail.com with SMTPS id 1616054981114711.4102630937356; Thu, 18 Mar 2021 01:09:41 -0700 (PDT) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-576-CipzZ61hO02aLhiR7-7mOg-1; Thu, 18 Mar 2021 04:09:38 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B065B84B9A2; Thu, 18 Mar 2021 08:09:30 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8B49A1378D; Thu, 18 Mar 2021 08:09:30 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 4B842180B450; Thu, 18 Mar 2021 08:09:30 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 12I89RJV030393 for ; Thu, 18 Mar 2021 04:09:27 -0400 Received: by smtp.corp.redhat.com (Postfix) id 67E091F041; Thu, 18 Mar 2021 08:09:27 +0000 (UTC) Received: from nautilus.redhat.com (unknown [10.40.192.180]) by smtp.corp.redhat.com (Postfix) with ESMTP id B6D7319C45; Thu, 18 Mar 2021 08:09:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1616054980; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=c8oIH2dNbEM9/Cqou9HU7XSS8j6M0b0jkJZA6O4sYuM=; b=DmcuD9FGG0IrT5R+yXnO6DK5VMaDegaHPidpSuBru5YtXWJ4mgSJWv4Dgwpv7uOIG3mbyp ygyQtItZDZlliggPaJoswTGHnfzJESSdJv2kRIlTMjCIiLW+nNp6O9VGpf8IS6DQoFxpux PQYVirDjkKZ76/aJDz/V5zqQ+67KslU= X-MC-Unique: CipzZ61hO02aLhiR7-7mOg-1 From: Erik Skultety To: libvir-list@redhat.com Subject: [libvirt PATCH v3 5/6] ci: util: Add a registry checker for stale images Date: Thu, 18 Mar 2021 09:09:16 +0100 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-loop: libvir-list@redhat.com Cc: eskultet@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" This function checks whether there are any stale Docker images in the registry that can be purged. Since we're pulling available container images from our GitLab registry with the 'list-images' action, it could happen that we'd list old (already unsupported) images and make them available for the user to consume and run a build in them. Naturally, the build will most likely fail leaving the user confused. Signed-off-by: Erik Skultety --- ci/helper | 45 ++++++++++++++++++++++++++++++++++++++++++++- ci/util.py | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 96 insertions(+), 1 deletion(-) diff --git a/ci/helper b/ci/helper index 31cf72fbdf..b5255db835 100755 --- a/ci/helper +++ b/ci/helper @@ -130,7 +130,7 @@ class Parser: refreshparser =3D subparsers.add_parser( "refresh", help=3D"refresh data generated with lcitool", - parents=3D[lcitoolparser], + parents=3D[lcitoolparser, gitlabparser], formatter_class=3Dargparse.ArgumentDefaultsHelpFormatter, ) refreshparser.add_argument( @@ -139,6 +139,13 @@ class Parser: default=3DFalse, help=3D"refresh data silently" ) + refreshparser.add_argument( + "--check-stale", + action=3D"store", + choices=3D["yes", "no"], + default=3D"yes", + help=3D"check for existence of stale images on the GitLab inst= ance" + ) refreshparser.set_defaults(func=3DApplication.action_refresh) =20 def parse(self): @@ -287,10 +294,46 @@ class Application: print("Available cross-compiler container images:\n") print(spacing + ("\n" + spacing).join(cross)) =20 + def check_stale_images(self): + if self.args.check_stale !=3D "yes" or self.args.quiet: + return + + namespace =3D self.args.namespace + gitlab_uri =3D self.args.gitlab_uri + registry_uri =3D util.get_registry_uri(namespace, gitlab_uri) + lcitool_hosts =3D self.lcitool_get_hosts() + + stale_images =3D util.get_registry_stale_images(registry_uri, + lcitool_hosts) + if stale_images: + spacing =3D "\n" + 4 * " " + stale_fmt =3D [f"{k} (ID: {v})" for k, v in stale_images.items= ()] + stale_details =3D spacing + spacing.join(stale_fmt) + stale_ids =3D ' '.join([str(id) for id in stale_images.values(= )]) + registry_uri =3D util.get_registry_uri(namespace, gitlab_uri) + + print(f""" +The following images are stale and can be purged from the registry: +{stale_details} + +You can remove the above images over the API with the following code snipp= et: + + $ for image_id in {stale_ids}; do \\ + curl --request DELETE --header "PRIVATE-TOKEN: " \\ + {registry_uri}/$image_id \\ + done + +You can generate a personal access token here: + {gitlab_uri}/-/profile/personal_access_tokens""") + def action_refresh(self): + # refresh Dockerfiles and vars files self.refresh_containers() self.refresh_cirrus() =20 + # check for stale images + self.check_stale_images() + def run(self): self.args.func(self) =20 diff --git a/ci/util.py b/ci/util.py index f9f8320276..d69c246872 100644 --- a/ci/util.py +++ b/ci/util.py @@ -38,3 +38,55 @@ def get_registry_images(uri: str) -> List[Dict]: =20 # read the HTTP response and load the JSON part of it return json.loads(r.read().decode()) + + +def get_image_distro(image_name: str) -> str: + """ + Extract the name of the distro in the GitLab image registry name, e.g. + ci-debian-9-cross-mipsel --> debian-9 + + :param image_name: name of the GitLab registry image + :return: distro name as a string + """ + name_prefix =3D "ci-" + name_suffix =3D "-cross-" + + distro =3D image_name[len(name_prefix):] + + index =3D distro.find(name_suffix) + if index > 0: + distro =3D distro[:index] + + return distro + + +def get_registry_stale_images(registry_uri: str, + supported_distros: List[str]) -> Dict[str, i= nt]: + """ + Check the GitLab image registry for images that we no longer support a= nd + which should be deleted. + + :param uri: URI pointing to a GitLab instance's image registry + :param supported_distros: list of hosts supported by lcitool + :return: dictionary formatted as: {: } + """ + + images =3D get_registry_images(registry_uri) + + # extract distro names from the list of registry images + registry_distros =3D [get_image_distro(i["name"]) for i in images] + + # - compare the distros powering the images in GitLab registry with + # the list of host available from lcitool + # - @unsupported is a set containing the distro names which we no long= er + # support; we need to map these back to registry image names + unsupported =3D set(registry_distros) - set(supported_distros) + if unsupported: + stale_images =3D {} + for distro in unsupported: + for img in images: + # gitlab images are named like "ci--?" + if distro in img["name"]: + stale_images[img["name"]] =3D img["id"] + + return stale_images --=20 2.29.2