From nobody Tue Feb 10 22:15:18 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 216.205.24.124 as permitted sender) client-ip=216.205.24.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 216.205.24.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1631264229; cv=none; d=zohomail.com; s=zohoarc; b=GbL5h620DdM1afQlALze4VFiqOm3oF/i1P38HXmr6FGznCzoJQWoEFIqzHfyRBH3TUgvqC/SELhDV1Kz+ULMSiA27XTvkz9/1nyb8K6sr2pI6nKaXUIckWDtBlcknekyiO+kyl0dIQT9v6Y0uHIZ4yeGiBfa89Lx2fmgQ5BujAg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1631264229; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=hSKihpr4w7/P99X1TNCEHEkA1gU7K37R+yUH6XP1ysk=; b=K5/OWlIidgpu+tvsIBw2kPXwT9sdT59gsbmyIH8A7QOSOe/iXXWNom8k3s9E/MBNGRiKEEiMIgVuVQcS878S0jfbJCaurcxQjgS58HEM/7z3oW5GRbFJC9Sn78BIr1sMnjjMpromroLPd1xBePXmM09vY/HdSvLFzJ2xDOAxGdQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 216.205.24.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mx.zohomail.com with SMTPS id 1631264229844890.6189748428263; Fri, 10 Sep 2021 01:57:09 -0700 (PDT) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-333-bnJdKJapPGSVkW2LDgjVHw-1; Fri, 10 Sep 2021 04:57:07 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1FFC6108468C; Fri, 10 Sep 2021 08:57:01 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 025B577F2D; Fri, 10 Sep 2021 08:57:01 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id C39151806D00; Fri, 10 Sep 2021 08:57:00 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 18A8udRW026327 for ; Fri, 10 Sep 2021 04:56:39 -0400 Received: by smtp.corp.redhat.com (Postfix) id 66F3019736; Fri, 10 Sep 2021 08:56:39 +0000 (UTC) Received: from localhost.localdomain.com (unknown [10.39.194.181]) by smtp.corp.redhat.com (Postfix) with ESMTP id BE54719C79; Fri, 10 Sep 2021 08:56:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1631264228; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=hSKihpr4w7/P99X1TNCEHEkA1gU7K37R+yUH6XP1ysk=; b=G5A/V/HJIQoHbjyZds8cWip227mpeSLM2GWZbKChcyGj/XNWYKU7t4qvZkjcI3917r9UNw OCUhzLqkJ6SSJnLS7OrdPDEwU4QjA4qyVJ8WtXFlw8A3PZNG96+XIHNE6kcdhb19eIZGSA hnvdeQDO9HnRKsSjH//hzLrxI6n7YqY= X-MC-Unique: bnJdKJapPGSVkW2LDgjVHw-1 From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: libvir-list@redhat.com Subject: [libvirt PATCH 7/7] ci: remove obsolete logic for refreshing containers/cirrus Date: Fri, 10 Sep 2021 09:56:26 +0100 Message-Id: <20210910085626.943674-8-berrange@redhat.com> In-Reply-To: <20210910085626.943674-1-berrange@redhat.com> References: <20210910085626.943674-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 X-loop: libvir-list@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1631264231989100001 We now use lcitool's manifest feature to generate files. The logic for checking for stale containers in the registry, however, is still relevant so that is propagated to a standalone command. Signed-off-by: Daniel P. Berrang=C3=A9 --- ci/helper | 121 ++++------------------------------------------------- ci/util.py | 32 +++++++------- 2 files changed, 23 insertions(+), 130 deletions(-) diff --git a/ci/helper b/ci/helper index 441258f511..8b8d0f68cb 100755 --- a/ci/helper +++ b/ci/helper @@ -7,7 +7,6 @@ import argparse import os import pathlib import pty -import shutil import subprocess import sys import textwrap @@ -60,15 +59,6 @@ class Parser: help=3D"additional arguments passed to ninja", ) =20 - # Options that are common to all actions that use lcitool - lcitoolparser =3D argparse.ArgumentParser(add_help=3DFalse) - lcitoolparser.add_argument( - "--lcitool", - metavar=3D"PATH", - default=3D"lcitool", - help=3D"path to lcitool binary", - ) - # Options that are common to actions communicating with a GitLab # instance gitlabparser =3D argparse.ArgumentParser(add_help=3DFalse) @@ -127,27 +117,14 @@ class Parser: ) listimagesparser.set_defaults(func=3DApplication._action_list_imag= es) =20 - # refresh action - refreshparser =3D subparsers.add_parser( - "refresh", - help=3D"refresh data generated with lcitool", - parents=3D[lcitoolparser, gitlabparser], + # check_stale action + check_staleparser =3D subparsers.add_parser( + "check-stale", + help=3D"check for existence of stale images on the GitLab inst= ance", + parents=3D[gitlabparser], formatter_class=3Dargparse.ArgumentDefaultsHelpFormatter, ) - refreshparser.add_argument( - "--quiet", - action=3D"store_true", - default=3DFalse, - help=3D"refresh data silently" - ) - refreshparser.add_argument( - "--check-stale", - action=3D"store", - choices=3D["yes", "no"], - default=3D"yes", - help=3D"check for existence of stale images on the GitLab inst= ance" - ) - refreshparser.set_defaults(func=3DApplication._action_refresh) + check_staleparser.set_defaults(func=3DApplication._action_check_st= ale) =20 def parse(self): return self._parser.parse_args() @@ -158,10 +135,6 @@ class Application: self._basedir =3D pathlib.Path(__file__).resolve().parent self._args =3D Parser().parse() =20 - if self._args.action =3D=3D "refresh": - if not shutil.which(self._args.lcitool): - sys.exit("error: 'lcitool' not installed") - def _make_run(self, target): args =3D [ "-C", @@ -194,84 +167,12 @@ class Application: output =3D self._lcitool_run(["targets"]) return output.splitlines() =20 - def _generate_dockerfile(self, target, cross=3DNone): - args =3D ["dockerfile", target, "libvirt"] - outdir =3D self._basedir.joinpath("containers") - outfile =3D f"{target}.Dockerfile" - - if cross: - args.extend(["--cross", cross]) - outfile =3D f"{target}-cross-{cross}.Dockerfile" - - outpath =3D outdir.joinpath(outfile) - if not self._args.quiet: - print(outpath) - - output =3D self._lcitool_run(args) - with open(outpath, "w") as f: - f.write(output) - - def _generate_vars(self, target): - args =3D ["variables", target, "libvirt"] - outdir =3D self._basedir.joinpath("cirrus") - outfile =3D f"{target}.vars" - - outpath =3D outdir.joinpath(outfile) - if not self._args.quiet: - print(outpath) - - output =3D self._lcitool_run(args) - with open(outpath, "w") as f: - f.write(output) - - def _refresh_containers(self): - debian_cross =3D [ - "aarch64", - "armv6l", - "armv7l", - "i686", - "mips", - "mips64el", - "mipsel", - "ppc64le", - "s390x", - ] - fedora_cross =3D [ - "mingw32", - "mingw64", - ] - - for target in self._lcitool_get_targets(): - if target.startswith("freebsd-") or target.startswith("macos-"= ): - continue - - self._generate_dockerfile(target) - - if target =3D=3D "fedora-rawhide": - for cross in fedora_cross: - self._generate_dockerfile(target, cross) - - if target.startswith("debian-"): - for cross in debian_cross: - if target =3D=3D "debian-sid" and cross =3D=3D "mips": - continue - self._generate_dockerfile(target, cross) - - def _refresh_cirrus(self): - for target in self._lcitool_get_targets(): - if not (target.startswith("freebsd-") or target.startswith("ma= cos-")): - continue - - self._generate_vars(target) - def _check_stale_images(self): namespace =3D self._args.namespace gitlab_uri =3D self._args.gitlab_uri registry_uri =3D util.get_registry_uri(namespace, gitlab_uri) - lcitool_targets =3D self._lcitool_get_targets() =20 - stale_images =3D util.get_registry_stale_images(registry_uri, - lcitool_targets) + stale_images =3D util.get_registry_stale_images(registry_uri, self= ._basedir) if stale_images: spacing =3D "\n" + 4 * " " stale_fmt =3D [f"{k} (ID: {v})" for k, v in stale_images.items= ()] @@ -328,12 +229,8 @@ class Application: print("Available cross-compiler container images:\n") print(spacing + ("\n" + spacing).join(cross)) =20 - def _action_refresh(self): - self._refresh_containers() - self._refresh_cirrus() - - if self._args.check_stale =3D=3D "yes" and not self._args.quiet: - self._check_stale_images() + def _action_check_stale(self): + self._check_stale_images() =20 def run(self): self._args.func(self) diff --git a/ci/util.py b/ci/util.py index 90d58454be..d8be6631eb 100644 --- a/ci/util.py +++ b/ci/util.py @@ -1,4 +1,5 @@ import json +import pathlib import urllib.request import urllib.parse =20 @@ -40,42 +41,37 @@ def get_registry_images(uri: str) -> List[Dict]: return json.loads(r.read().decode()) =20 =20 -def get_image_distro(image_name: str) -> str: +def get_dockerfiles(base_dir) -> List: """ - Extract the name of the distro in the GitLab image registry name, e.g. - ci-debian-9-cross-mipsel --> debian-9 + List all container dockerfiles in the local directory. =20 - :param image_name: name of the GitLab registry image - :return: distro name as a string + :return: list of dockerfile names """ - name_prefix =3D "ci-" - name_suffix =3D "-cross-" =20 - distro =3D image_name[len(name_prefix):] + dkrs =3D [] + d =3D pathlib.Path(base_dir, "containers") + for f in d.iterdir(): + if f.suffix =3D=3D ".Dockerfile": + dkrs.append(f.stem) + return dkrs =20 - index =3D distro.find(name_suffix) - if index > 0: - distro =3D distro[:index] =20 - return distro - - -def get_registry_stale_images(registry_uri: str, - supported_distros: List[str]) -> Dict[str, i= nt]: +def get_registry_stale_images(registry_uri: str, base_dir: str) -> Dict[st= r, int]: """ Check the GitLab image registry for images that we no longer support a= nd which should be deleted. =20 :param uri: URI pointing to a GitLab instance's image registry - :param supported_distros: list of hosts supported by lcitool + :param base_dir: local repository base directory :return: dictionary formatted as: {: } """ =20 + dockerfiles =3D get_dockerfiles(base_dir) images =3D get_registry_images(registry_uri) =20 stale_images =3D {} for img in images: - if get_image_distro(img["name"]) not in supported_distros: + if img["name"][3:] not in dockerfiles: stale_images[img["name"]] =3D img["id"] =20 return stale_images --=20 2.31.1