From nobody Fri Dec 19 20:40:03 2025 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) client-ip=63.128.21.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1616083937; cv=none; d=zohomail.com; s=zohoarc; b=iSE17XLhnXGPFS9f+VJjjralz7SeOgulEVb/J8oLkxWqUA1iBmZ0viyMdlEYyyH/3SR0oO8IT3oqKTmTKtfPA9AF1e6t7dc5Il+rH3A1nwS6qbeyJvmSbwu1L+VV0YCVQ/ECwVsbn40L+L5Ec5aL50X4B3u7Kv+OyQ+XJDJx3ck= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1616083937; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=L1j2yTxLBvG/HASoJPR3sGyATiwme4Td/Z1Rx7hC64E=; b=Xz3TdQDqsZVLzU9eYKnBq3faqut7n5bpT/LBUTmjau/3mDM71jb82er1UKT3TmjCmLAoTJuwCqKTKrCKXh1ATeF0lckkzen+4byeLtxPL1zYxn2Y0TNJQw7S6THwPZZN9cWgHnCNFSaQpwbjGhslui5ga/Wewm1yRQgH3HGU1AA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) by mx.zohomail.com with SMTPS id 1616083937425172.6548269730107; Thu, 18 Mar 2021 09:12:17 -0700 (PDT) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-472-1tpnNhBqO6-Wbj0IBb4k8w-1; Thu, 18 Mar 2021 12:12:13 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 30890107B7C5; Thu, 18 Mar 2021 16:12:07 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 095C260CEC; Thu, 18 Mar 2021 16:12:07 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id BAD6E4A705; Thu, 18 Mar 2021 16:12:06 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 12IGBtOD024219 for ; Thu, 18 Mar 2021 12:11:55 -0400 Received: by smtp.corp.redhat.com (Postfix) id E69A917DC3; Thu, 18 Mar 2021 16:11:55 +0000 (UTC) Received: from nautilus.redhat.com (unknown [10.40.192.180]) by smtp.corp.redhat.com (Postfix) with ESMTP id 41EFA60CD7; Thu, 18 Mar 2021 16:11:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1616083936; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=L1j2yTxLBvG/HASoJPR3sGyATiwme4Td/Z1Rx7hC64E=; b=Bk9D406PISMiDBNwm6hYV3iHXnRvcFSAzP10Ba0LKAOj7Kk6hs9RQE1IvO1ym3qjWkge71 adbdeSKasRPXEgQfthT6alXjUfCK/mTpUMrtG/fK3oqFu24MUEdxJOJ8eN2UUkvhw/e+yj MuzMDyXLs60jkf6V5qZS24A/BUZjeRo= X-MC-Unique: 1tpnNhBqO6-Wbj0IBb4k8w-1 From: Erik Skultety To: libvir-list@redhat.com Subject: [libvirt PATCH v4 5/6] ci: util: Add a registry checker for stale images Date: Thu, 18 Mar 2021 17:11:45 +0100 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-loop: libvir-list@redhat.com Cc: eskultet@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" This function checks whether there are any stale Docker images in the registry that can be purged. Since we're pulling available container images from our GitLab registry with the 'list-images' action, it could happen that we'd list old (already unsupported) images and make them available for the user to consume and run a build in them. Naturally, the build will most likely fail leaving the user confused. Signed-off-by: Erik Skultety --- ci/helper | 46 +++++++++++++++++++++++++++++++++++++++++++++- ci/util.py | 41 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 86 insertions(+), 1 deletion(-) diff --git a/ci/helper b/ci/helper index 31cf72fbdf..0743f95e13 100755 --- a/ci/helper +++ b/ci/helper @@ -10,6 +10,7 @@ import pty import shutil import subprocess import sys +import textwrap =20 import util =20 @@ -130,7 +131,7 @@ class Parser: refreshparser =3D subparsers.add_parser( "refresh", help=3D"refresh data generated with lcitool", - parents=3D[lcitoolparser], + parents=3D[lcitoolparser, gitlabparser], formatter_class=3Dargparse.ArgumentDefaultsHelpFormatter, ) refreshparser.add_argument( @@ -139,6 +140,13 @@ class Parser: default=3DFalse, help=3D"refresh data silently" ) + refreshparser.add_argument( + "--check-stale", + action=3D"store", + choices=3D["yes", "no"], + default=3D"yes", + help=3D"check for existence of stale images on the GitLab inst= ance" + ) refreshparser.set_defaults(func=3DApplication.action_refresh) =20 def parse(self): @@ -287,10 +295,46 @@ class Application: print("Available cross-compiler container images:\n") print(spacing + ("\n" + spacing).join(cross)) =20 + def check_stale_images(self): + namespace =3D self.args.namespace + gitlab_uri =3D self.args.gitlab_uri + registry_uri =3D util.get_registry_uri(namespace, gitlab_uri) + lcitool_hosts =3D self.lcitool_get_hosts() + + stale_images =3D util.get_registry_stale_images(registry_uri, + lcitool_hosts) + if stale_images: + spacing =3D "\n" + 4 * " " + stale_fmt =3D [f"{k} (ID: {v})" for k, v in stale_images.items= ()] + stale_details =3D spacing.join(stale_fmt) + stale_ids =3D ' '.join([str(id) for id in stale_images.values(= )]) + registry_uri =3D util.get_registry_uri(namespace, gitlab_uri) + + msg =3D textwrap.dedent(f""" + The following images are stale and can be purged from the = registry: + + STALE_DETAILS + + You can delete the images listed above using this shell sn= ippet: + + $ for image_id in {stale_ids}; do + curl --request DELETE --header "PRIVATE-TOKEN: " \\ + {registry_uri}/$image_id; + done + + You can generate a personal access token here: + + {gitlab_uri}/-/profile/personal_access_tokens + """) + print(msg.replace("STALE_DETAILS", stale_details)) + def action_refresh(self): self.refresh_containers() self.refresh_cirrus() =20 + if self.args.check_stale =3D=3D "yes" and not self.args.quiet: + self.check_stale_images() + def run(self): self.args.func(self) =20 diff --git a/ci/util.py b/ci/util.py index f9f8320276..90d58454be 100644 --- a/ci/util.py +++ b/ci/util.py @@ -38,3 +38,44 @@ def get_registry_images(uri: str) -> List[Dict]: =20 # read the HTTP response and load the JSON part of it return json.loads(r.read().decode()) + + +def get_image_distro(image_name: str) -> str: + """ + Extract the name of the distro in the GitLab image registry name, e.g. + ci-debian-9-cross-mipsel --> debian-9 + + :param image_name: name of the GitLab registry image + :return: distro name as a string + """ + name_prefix =3D "ci-" + name_suffix =3D "-cross-" + + distro =3D image_name[len(name_prefix):] + + index =3D distro.find(name_suffix) + if index > 0: + distro =3D distro[:index] + + return distro + + +def get_registry_stale_images(registry_uri: str, + supported_distros: List[str]) -> Dict[str, i= nt]: + """ + Check the GitLab image registry for images that we no longer support a= nd + which should be deleted. + + :param uri: URI pointing to a GitLab instance's image registry + :param supported_distros: list of hosts supported by lcitool + :return: dictionary formatted as: {: } + """ + + images =3D get_registry_images(registry_uri) + + stale_images =3D {} + for img in images: + if get_image_distro(img["name"]) not in supported_distros: + stale_images[img["name"]] =3D img["id"] + + return stale_images --=20 2.30.2