From nobody Sun May 5 17:12:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) client-ip=63.128.21.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1612977897; cv=none; d=zohomail.com; s=zohoarc; b=nE+wdDg7PDjF8+E7sRYZNgFbQCetkCaZsn/qnaQyR4TEWV+Sea7cUYIYhbx/5f4+7/lFyMVCVDYGtwj3RaJhTRcpH+bNAhtndXN9P9UKKwU++Zf+cxAGGMRH+Bdsh0meyCNIV7c7tmM68/Sk2xM7sKqLVFMfmfq7Nlb+8rZ9Xmw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612977897; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=D+o+0Klhlg/u/LOuWHa72LU1/QTdxIK2siwZbTpvbzw=; b=P6lsLv+dbV79XaviZ8auSTFx2IlOTQYLuzr4Xl/XDrqC5QDqnQduRoma8Kb1bgkbiQOTWQUhF64oRlp72aYDcGuRD3XsI0AwzF2CbC9DZxopkNcbr2ezbJEx6mXsnjywDVeO0Ba/7vURDdEdiV9YeMsuh6RG5HK2mKWeK0tWhAQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) by mx.zohomail.com with SMTPS id 161297789696172.52315414079442; Wed, 10 Feb 2021 09:24:56 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-554-A89bosVbOqm7mHTbfYwZ9Q-1; Wed, 10 Feb 2021 12:24:52 -0500 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7A6511936B61; Wed, 10 Feb 2021 17:24:47 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 40C4719C59; Wed, 10 Feb 2021 17:24:47 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 46CAC4E58E; Wed, 10 Feb 2021 17:24:44 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 11AHOhma019429 for ; Wed, 10 Feb 2021 12:24:43 -0500 Received: by smtp.corp.redhat.com (Postfix) id 2C1D75D9E8; Wed, 10 Feb 2021 17:24:43 +0000 (UTC) Received: from nautilus.redhat.com (unknown [10.40.192.74]) by smtp.corp.redhat.com (Postfix) with ESMTP id 749FE5D9DC; Wed, 10 Feb 2021 17:24:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1612977896; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=D+o+0Klhlg/u/LOuWHa72LU1/QTdxIK2siwZbTpvbzw=; b=MazNo5e8UpDAbI/73OOCE51F2g2kRQDyOavP/r8WUrKSi+UL6FXsBuUH34/KeLaWrbmiMY vNO/aVho8IsPFqGUDku28GTlu1k7b7cSRDuWiULHwbtrwc1BPGl9NLzWS62YyU2V+5F61I hURe20+o//kbXiIAqEEcADoljRE27Io= X-MC-Unique: A89bosVbOqm7mHTbfYwZ9Q-1 From: Erik Skultety To: libvir-list@redhat.com Subject: [libvirt PATCH 1/4] ci: Makefile: Specify a help target to replace ci-help Date: Wed, 10 Feb 2021 18:24:33 +0100 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-loop: libvir-list@redhat.com Cc: eskultet@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" It's quite pointless to have a 'ci-help' target in the Makefile when one needs to actually open the Makefile to go through the list of targets to know what functionality the Makefile actually provides. It's much more intuitive to run "make help" in that case. Therefore, add a 'help' target and replace the old 'ci-help' target with it. Signed-off-by: Erik Skultety Reviewed-by: Andrea Bolognani --- ci/Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/ci/Makefile b/ci/Makefile index 7938e14c15..1f71701047 100644 --- a/ci/Makefile +++ b/ci/Makefile @@ -226,7 +226,7 @@ ci-list-images: @sh list-images.sh "$(CI_IMAGE_PREFIX)" | grep cross @echo =20 -ci-help: +help: @echo "Build libvirt inside containers used for CI" @echo @echo "Available targets:" @@ -235,7 +235,7 @@ ci-help: @echo " ci-check@\$$IMAGE - run a 'ninja test'" @echo " ci-shell@\$$IMAGE - run an interactive shell" @echo " ci-list-images - list available images" - @echo " ci-help - show this help message" + @echo " help - show this help message" @echo @echo "Available make variables:" @echo --=20 2.29.2 From nobody Sun May 5 17:12:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 216.205.24.124 as permitted sender) client-ip=216.205.24.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 216.205.24.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1612977913; cv=none; d=zohomail.com; s=zohoarc; b=ZJX7Xr11XnpdpUbIf8Zyxi1lzs8BfnB8bSP0uexSa9Cvyy4teNa3SIoDqzb+b4gVWAU7M7P+rCqofuoL/OS3ry+NdhOS/rr+hL/I3geniMsxoRFvc5DgQp74PBbE2lN67BD2dnTsy1Kn/jv5kEEBx82v1ICjBfswOPVbIJAEqI4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612977913; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=y5eCCXzzI5LnUY+bUSoukTFnNyC6mI7ma3QuBS5ehVc=; b=Te81vFkjQZVnHDlGtrP60IF8liO49inKqbxRA+XusGqaUsx+6E9e8QXT0r4l9soau7lXgRziptfQ7fhqqYzPL03aVyPzXaXHXd0NroU6iThD+y251bR+dWEmdOhNqP9Xy3YvLIFBKfQ+mIP3tXoLZ7RPWsnSpuZmqd8zUQnulqA= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 216.205.24.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by mx.zohomail.com with SMTPS id 1612977913333808.0917428623126; Wed, 10 Feb 2021 09:25:13 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-357-4mlC-pzONMKg-e4TZEAuUA-1; Wed, 10 Feb 2021 12:25:09 -0500 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id ADFE2835E28; Wed, 10 Feb 2021 17:25:03 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 832485D9DC; Wed, 10 Feb 2021 17:25:03 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 38DD04EEF6; Wed, 10 Feb 2021 17:25:03 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 11AHOiOX019439 for ; Wed, 10 Feb 2021 12:24:44 -0500 Received: by smtp.corp.redhat.com (Postfix) id 33F755D9E8; Wed, 10 Feb 2021 17:24:44 +0000 (UTC) Received: from nautilus.redhat.com (unknown [10.40.192.74]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7EBA45D9E3; Wed, 10 Feb 2021 17:24:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1612977912; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=y5eCCXzzI5LnUY+bUSoukTFnNyC6mI7ma3QuBS5ehVc=; b=hVgimlyzI9xR2JD+g7sYyXkklObD8X/p7ar75hyDxHpF4jq0Au7HV2f6/D/Du5OOP2TBSt FJRcq6fyf+vxYdm1SHmHW+w7fqwYD116WbcukwqNBSrTMrBa+W8d94FVOREPZnbeV2H/v6 NEAi6adtKOmkmXChrvRVXCRx5ZzMhjs= X-MC-Unique: 4mlC-pzONMKg-e4TZEAuUA-1 From: Erik Skultety To: libvir-list@redhat.com Subject: [libvirt PATCH 2/4] ci: Rewrite list-images from Bash to Python Date: Wed, 10 Feb 2021 18:24:34 +0100 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-loop: libvir-list@redhat.com Cc: eskultet@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" Convert the script to some human readable form. Speed-wise, both are comparable. Signed-off-by: Erik Skultety --- ci/Makefile | 8 +------- ci/list-images.py | 29 +++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+), 7 deletions(-) create mode 100644 ci/list-images.py diff --git a/ci/Makefile b/ci/Makefile index 1f71701047..92032e8ca0 100644 --- a/ci/Makefile +++ b/ci/Makefile @@ -217,13 +217,7 @@ ci-test@%: =20 ci-list-images: @echo - @echo "Available x86 container images:" - @echo - @sh list-images.sh "$(CI_IMAGE_PREFIX)" | grep -v cross - @echo - @echo "Available cross-compiler container images:" - @echo - @sh list-images.sh "$(CI_IMAGE_PREFIX)" | grep cross + @python3 list-images.py @echo =20 help: diff --git a/ci/list-images.py b/ci/list-images.py new file mode 100644 index 0000000000..6862ac347f --- /dev/null +++ b/ci/list-images.py @@ -0,0 +1,29 @@ +#!/usr/bin/env python3 + +import json +import urllib.request as urllib + +PROJECT_ID =3D 192693 +DOMAIN =3D "https://gitlab.com" +API =3D "api/v4" + +uri =3D f"{DOMAIN}/{API}/projects/{PROJECT_ID}/registry/repositories" +r =3D urllib.urlopen(uri + "?per_page=3D100") + +# read the HTTP response and load the JSON part of it +data =3D json.loads(r.read().decode()) + +# skip the "ci-" prefix each of our container images' name has +names =3D [i["name"][3:] for i in data] +names.sort() + +names_native =3D [name for name in names if "cross" not in name] +names_cross =3D [name for name in names if "cross" in name] + +print("Available x86 container images:\n") +print("\t" + "\n\t".join(names_native)) + +if names_cross: + print() + print("Available cross-compiler container images:\n") + print("\t" + "\n\t".join(names_cross)) --=20 2.29.2 From nobody Sun May 5 17:12:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) client-ip=63.128.21.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1612977898; cv=none; d=zohomail.com; s=zohoarc; b=V0IvPHv2Zog5DimsJSk0aYQM+s9TJLLzYuw31P4lmytbqDEhCbLPE0wSb/2V+hdsrfGEC/Kc5PYhy5SoffdaSAr537RgEtIY3Gw0Tp1HdUbHojJgYtWludl+owAy2hAvTACstAmSv/UuZpZ6OGBQS2gW2rTiQdWeHUtzq2XWK+4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612977898; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=7u97paZIKAqBfYFHaqqL4wmfT6iUw+mbkZJpNwhGrH0=; b=aIYD6BVMsWeAw+oF/Nul9Ys9GurIvjqZBYzJguSJucP+6wYEaoX3dcPz35pQ5M58ir+hqgtGFYXRtZrKZM5QpMKW6N6mvVgCUKyHa3mOdCFPw6cXucqxPb6c10sitJ8N3XSgxWBRlQJh+mqvQerOOwsJglKmZxzlTL/CfOrUbKc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) by mx.zohomail.com with SMTPS id 1612977897934639.0702398302514; Wed, 10 Feb 2021 09:24:57 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-364-BmXTMLyGN_avBenMFULVGg-1; Wed, 10 Feb 2021 12:24:54 -0500 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id B0487100CCC6; Wed, 10 Feb 2021 17:24:48 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.20]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 8AC901876A; Wed, 10 Feb 2021 17:24:48 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 2CAD918095CD; Wed, 10 Feb 2021 17:24:48 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 11AHOjJZ019444 for ; Wed, 10 Feb 2021 12:24:45 -0500 Received: by smtp.corp.redhat.com (Postfix) id 3BA025D9E8; Wed, 10 Feb 2021 17:24:45 +0000 (UTC) Received: from nautilus.redhat.com (unknown [10.40.192.74]) by smtp.corp.redhat.com (Postfix) with ESMTP id 864BB5D9DC; Wed, 10 Feb 2021 17:24:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1612977896; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=7u97paZIKAqBfYFHaqqL4wmfT6iUw+mbkZJpNwhGrH0=; b=OCn2E7IZvGUlKZyh+HCGIk76pejxOtbpZ6HK3dKh0z/cAnHy/sSvn/QNtsNw/QHYTvoYml 50lo1qknmBPi0oFUh1NRc1ud8n0dSBbzhMPbnE70OPGlZnSCavIfhNgsJKCP/MbH4wl7dG OM57I4kKTIvH/oRs5CDTQNmViCDThPU= X-MC-Unique: BmXTMLyGN_avBenMFULVGg-1 From: Erik Skultety To: libvir-list@redhat.com Subject: [libvirt PATCH 3/4] ci: list-images: Split some generic logic to a util module Date: Wed, 10 Feb 2021 18:24:35 +0100 Message-Id: In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-loop: libvir-list@redhat.com Cc: eskultet@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" Parts of the code can be reused by another registry checker script introduced in the next patch. Signed-off-by: Erik Skultety --- ci/containers/util.py | 25 +++++++++++++++++++++++++ ci/list-images.py | 43 +++++++++++++++++++++++-------------------- 2 files changed, 48 insertions(+), 20 deletions(-) create mode 100644 ci/containers/util.py diff --git a/ci/containers/util.py b/ci/containers/util.py new file mode 100644 index 0000000000..1ce326bba2 --- /dev/null +++ b/ci/containers/util.py @@ -0,0 +1,25 @@ +import json +import urllib.request as urllib + + +PROJECT_ID =3D 192693 + + +def get_registry_uri(project_id, + base_uri=3D"https://gitlab.com", + api_version=3D4): + uri =3D f"{base_uri}/api/v{str(api_version)}/projects/{project_id}/reg= istry/repositories" + return uri + + +def list_images(uri): + """ + Returns all container images as currently available for the given GitL= ab + project. + + ret: list of container image names + """ + r =3D urllib.urlopen(uri) + + # read the HTTP response and load the JSON part of it + return json.loads(r.read().decode()) diff --git a/ci/list-images.py b/ci/list-images.py index 6862ac347f..1a1d87c10b 100644 --- a/ci/list-images.py +++ b/ci/list-images.py @@ -1,29 +1,32 @@ #!/usr/bin/env python3 =20 -import json -import urllib.request as urllib +import containers.util as util =20 -PROJECT_ID =3D 192693 -DOMAIN =3D "https://gitlab.com" -API =3D "api/v4" =20 -uri =3D f"{DOMAIN}/{API}/projects/{PROJECT_ID}/registry/repositories" -r =3D urllib.urlopen(uri + "?per_page=3D100") +def list_image_names(uri): + images =3D util.list_images(uri) =20 -# read the HTTP response and load the JSON part of it -data =3D json.loads(r.read().decode()) + # skip the "ci-" prefix each of our container images' name has + names =3D [i["name"][3:] for i in images] + names.sort() =20 -# skip the "ci-" prefix each of our container images' name has -names =3D [i["name"][3:] for i in data] -names.sort() + return names =20 -names_native =3D [name for name in names if "cross" not in name] -names_cross =3D [name for name in names if "cross" in name] =20 -print("Available x86 container images:\n") -print("\t" + "\n\t".join(names_native)) +def main(): + names =3D list_image_names(util.get_registry_uri(util.PROJECT_ID) + "?= per_page=3D100") =20 -if names_cross: - print() - print("Available cross-compiler container images:\n") - print("\t" + "\n\t".join(names_cross)) + names_native =3D [name for name in names if "cross" not in name] + names_cross =3D [name for name in names if "cross" in name] + + print("Available x86 container images:\n") + print("\t" + "\n\t".join(names_native)) + + if names_cross: + print() + print("Available cross-compiler container images:\n") + print("\t" + "\n\t".join(names_cross)) + + +if __name__ =3D=3D "__main__": + main() --=20 2.29.2 From nobody Sun May 5 17:12:20 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) client-ip=63.128.21.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1612977917; cv=none; d=zohomail.com; s=zohoarc; b=KTr0UK2wMfH/0taJbCfPuwREWCQgnwcItctZ3MmBcxDfWM+UG4p8+7fl0FIiGcA6YkEMlE0gvmjq/1qHxI+HR1krZoNQ58lysIGRdjxmBbC6Wglpy9yZWaId3Jlidnm1uRRTVycuhMXBBVHSDpdtGZkQFc/l6JoAuVgMJRqNVLg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1612977917; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=fhsnn3ympFRVUvn/QBS9R2O/T2cGyOBk/WJx5EVngA8=; b=aL969BPwGXeHX7Y0wFSoGj/OYHWYnFqMlA6FEthrr1br1kdzD/yma867xcdCfqoiQwWcN69tT35Z27EOQunn3O6xDrP5hcam0KOvMIarSM9fU9na6F8OBf4koxXZwQ2ASUva1Pvn80tPcJqfJzKLyX0lmwUW3C3lFOsq6FB/nKI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 63.128.21.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) by mx.zohomail.com with SMTPS id 1612977917312578.8883342651766; Wed, 10 Feb 2021 09:25:17 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-102-iSGxobtoMl2dtNzFJDybkg-1; Wed, 10 Feb 2021 12:25:13 -0500 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 6A3C3C7401; Wed, 10 Feb 2021 17:25:07 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3E94B4C; Wed, 10 Feb 2021 17:25:07 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 00BABC841; Wed, 10 Feb 2021 17:25:06 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 11AHOkq9019456 for ; Wed, 10 Feb 2021 12:24:46 -0500 Received: by smtp.corp.redhat.com (Postfix) id 422505D9E3; Wed, 10 Feb 2021 17:24:46 +0000 (UTC) Received: from nautilus.redhat.com (unknown [10.40.192.74]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8E0F05D9DC; Wed, 10 Feb 2021 17:24:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1612977916; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=fhsnn3ympFRVUvn/QBS9R2O/T2cGyOBk/WJx5EVngA8=; b=DFlQLs8u0slbTBzsdFYrArbHmenzW+ElKYUaa7JnLsre7l36u5PtJkMt06WnfZbEMwD/Du uClGhr60xQiR1BAR6qad4GLAzEv2TKIXANMie93d+hj8d345/J1x68MyqJWvXhnKMxdtHI YMeARmazsW6h6wh1SGkcqDI7G2G6Hf4= X-MC-Unique: iSGxobtoMl2dtNzFJDybkg-1 From: Erik Skultety To: libvir-list@redhat.com Subject: [libvirt PATCH 4/4] ci: Introduce a new checker script 'check-registry.py' Date: Wed, 10 Feb 2021 18:24:36 +0100 Message-Id: <765ef45d8b45eb6adb54beed9e92273d651b7f9c.1612977855.git.eskultet@redhat.com> In-Reply-To: References: MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-loop: libvir-list@redhat.com Cc: eskultet@redhat.com X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com X-Scanned-By: MIMEDefang 2.84 on 10.5.11.23 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" When removing the dependency on sudo in our CI Makefile, I realized that the list of images that we pull contains some old images that we no longer support in lcitool, but can be still pulled from the registry. This is a simple checker which runs in context of the refresh script which informs the developer/maintainer that there are some old redundant images that should be purged from the registry, so that people don't try to run local builds in those (which will most likely fail anyway). Signed-off-by: Erik Skultety --- ci/containers/check-registry.py | 96 +++++++++++++++++++++++++++++++++ ci/containers/refresh | 5 ++ 2 files changed, 101 insertions(+) create mode 100644 ci/containers/check-registry.py diff --git a/ci/containers/check-registry.py b/ci/containers/check-registry= .py new file mode 100644 index 0000000000..f0159da54d --- /dev/null +++ b/ci/containers/check-registry.py @@ -0,0 +1,96 @@ +#!/usr/bin/env python3 + +import argparse +import subprocess +import shutil +import sys + +import util + + +def get_image_distro(image_name: str): + name_prefix =3D "ci-" + name_suffix =3D "-cross-" + + distro =3D image_name[len(name_prefix):] + + index =3D distro.find(name_suffix) + if index > 0: + distro =3D distro[:index] + + return distro + + +def get_undesirables(registry_distros_d, hosts_l): + """ Returns a dictionary of 'id':'name' pairs of images that can be pu= rged""" + + undesirables_d =3D {} + for distro in registry_distros_d: + if distro not in hosts_l: + for image in registry_distros_d[distro]: + undesirables_d[str(image["id"])] =3D image["path"] + + return undesirables_d if undesirables_d else None + + +def get_lcitool_hosts(lcitool_path): + """ Returns a list of supported hosts by lcitool + @param lcitool_path: absolute path to local copy of lcitool, if om= itted it is + assumed lcitool is installed in the current environment""" + + if not lcitool_path: + lcitool_path =3D "lcitool" + if not shutil.which(lcitool_path): + sys.exit("error: 'lcitool' not installed") + + lcitool_out =3D subprocess.check_output([lcitool_path, "hosts"]) + return lcitool_out.decode("utf-8").splitlines() + + +def main(): + parser =3D argparse.ArgumentParser() + parser.add_argument("project_id", + help=3D"GitLab project ID") + parser.add_argument("--lcitool_path", + dest=3D"lcitool", + metavar=3D"PATH", + help=3D"absolute path to lcitool, CWD is used if o= mitted") + + args =3D parser.parse_args() + + uri =3D util.get_registry_uri(util.PROJECT_ID) + images_json =3D util.list_images(uri + "?per_page=3D100") + + registry_distros_d =3D {} + for image in images_json: + distro_name =3D get_image_distro(image["name"]) + + try: + registry_distros_d[distro_name].append(image) + except KeyError: + registry_distros_d[distro_name] =3D [image] + + hosts_l =3D get_lcitool_hosts(args.lcitool) + + # print the list of images that we can safely purge from the registry + undesirables_d =3D get_undesirables(registry_distros_d, hosts_l) + if undesirables_d: + undesirable_image_names =3D "\t" + "\n\t".join(undesirables_d.valu= es()) + undesirable_image_ids =3D " ".join(undesirables_d.keys()) + + sys.exit(f""" +The following images can be purged from the registry: + +{undesirable_image_names} + +You can remove the above images over the API with the following code snipp= et: + +\t$ for image_id in {undesirable_image_ids} \\ +\t;do \\ +\t\tcurl --request DELETE --header "PRIVATE-TOKEN: " \\ +\t\t{util.get_registry_uri(util.PROJECT_ID)}/$image_id \\ +\t;done""") + + +if __name__ =3D=3D "__main__": + main() diff --git a/ci/containers/refresh b/ci/containers/refresh index a39d0b0996..9dfaf75300 100755 --- a/ci/containers/refresh +++ b/ci/containers/refresh @@ -40,3 +40,8 @@ do =20 $LCITOOL dockerfile $host libvirt > ci-$host.Dockerfile done + +# Check whether there are some old images in the container registry that c= ould +# be purged - mainly because the list-images script would list them as +# available +/usr/bin/env python3 check-registry.py --lcitool_path "$LCITOOL" 192693 --=20 2.29.2