From nobody Sat May 18 23:23:57 2024 Delivered-To: importer@patchew.org Received-SPF: none (zohomail.com: 205.139.111.44 is neither permitted nor denied by domain of listman.redhat.com) client-ip=205.139.111.44; envelope-from=libvir-list-bounces@listman.redhat.com; helo=us-smtp-delivery-44.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=none (zohomail.com: 205.139.111.44 is neither permitted nor denied by domain of listman.redhat.com) smtp.mailfrom=libvir-list-bounces@listman.redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com Return-Path: Received: from us-smtp-delivery-44.mimecast.com (us-smtp-delivery-44.mimecast.com [205.139.111.44]) by mx.zohomail.com with SMTPS id 1613553911892768.3557814080918; Wed, 17 Feb 2021 01:25:11 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-30-7GIMiJm_NzS0fYVzSwPBMw-1; Wed, 17 Feb 2021 04:25:06 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1AD8619357B1; Wed, 17 Feb 2021 09:25:01 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id C170260C5F; Wed, 17 Feb 2021 09:25:00 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 8A83C4E58E; Wed, 17 Feb 2021 09:24:59 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 11H9OvVt024072 for ; Wed, 17 Feb 2021 04:24:57 -0500 Received: by smtp.corp.redhat.com (Postfix) id 3D9AD10016FA; Wed, 17 Feb 2021 09:24:57 +0000 (UTC) Received: from harajuku.usersys.redhat.com (unknown [10.40.192.150]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 1C2C310013D7 for ; Wed, 17 Feb 2021 09:24:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1613553910; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=BmCP93uJuVOtUYzrTT2QhOtrEJH0ee/Mzj8YQwjAxPk=; b=avDSkHZfndLf6IVqGeE6Ggw3cdGcSiPv4B5F2zLHSZK2Tnrq4DWf9VuJTCfLnAZqDUIrdQ czy+INcL1BNmVKQcsVNWi/N3LUmYUUYZAgBO5V0hiDfRU8sdP1FMaOlHbX8J49R0PfBaT7 NgasxINfjz+DcldDtE7sJLjgiXJTEdU= X-MC-Unique: 7GIMiJm_NzS0fYVzSwPBMw-1 From: Andrea Bolognani To: libvir-list@redhat.com Subject: [libvirt PATCH v2 1/1] ci: Add helper script Date: Wed, 17 Feb 2021 10:24:51 +0100 Message-Id: <20210217092451.215696-2-abologna@redhat.com> In-Reply-To: <20210217092451.215696-1-abologna@redhat.com> References: <20210217092451.215696-1-abologna@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.84 on 10.5.11.22 X-loop: libvir-list@redhat.com X-BeenThere: libvir-list@listman.redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@listman.redhat.com Errors-To: libvir-list-bounces@listman.redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@listman.redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: listman.redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) Content-Type: text/plain; charset="utf-8" This is intended to be the perform a number of CI-related operations that currently are implemented in various different scripts written in various different programming languages; in this first iteration it does two things: * implement the functionality of the existing "refresh" scripts, which it supersedes; * provide a nicer front-end for a subset of the functionality exposed by the ci/Makefile scaffolding, such as running basic builds. Over time, the plan is to rewrite all CI-related functionality in Python and move it into this script. Advantages: * it provides a more standard, more convenient command line interface; * it refreshes all lcitool-generated files in one go; * it can be called from the top-level source directory; * it automatically finds lcitool if it's somewhere in the user's $PATH; * it produces some output to keep the user updated on the progress of the current operation; * it's written in a real programming language, which will hopefully help maintainability. Signed-off-by: Andrea Bolognani --- ci/cirrus/refresh | 22 ----- ci/containers/refresh | 41 --------- ci/helper | 187 ++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 187 insertions(+), 63 deletions(-) delete mode 100755 ci/cirrus/refresh delete mode 100755 ci/containers/refresh create mode 100755 ci/helper diff --git a/ci/cirrus/refresh b/ci/cirrus/refresh deleted file mode 100755 index 63ca794134..0000000000 --- a/ci/cirrus/refresh +++ /dev/null @@ -1,22 +0,0 @@ -#!/bin/sh - -if test -z "$1" -then - echo "syntax: $0 PATH-TO-LCITOOL" - exit 1 -fi - -LCITOOL=3D$1 - -if ! test -x "$LCITOOL" -then - echo "$LCITOOL is not executable" - exit 1 -fi - -HOSTS=3D$($LCITOOL hosts | grep -E 'freebsd|macos') - -for host in $HOSTS -do - $LCITOOL variables "$host" libvirt >"$host.vars" -done diff --git a/ci/containers/refresh b/ci/containers/refresh deleted file mode 100755 index f38d3634b5..0000000000 --- a/ci/containers/refresh +++ /dev/null @@ -1,41 +0,0 @@ -#!/bin/sh - -if test -z "$1" -then - echo "syntax: $0 PATH-TO-LCITOOL" - exit 1 -fi - -LCITOOL=3D$1 - -if ! test -x "$LCITOOL" -then - echo "$LCITOOL is not executable" - exit 1 -fi - -HOSTS=3D$($LCITOOL hosts | grep -Ev 'freebsd|macos') - -for host in $HOSTS -do - case "$host" in - fedora-rawhide) - for cross in mingw32 mingw64 - do - $LCITOOL dockerfile $host libvirt --cross $cross > ci-$host-cr= oss-$cross.Dockerfile - done - ;; - debian-*) - for cross in aarch64 armv6l armv7l i686 mips mips64el mipsel ppc64= le s390x - do - if test "$host-cross-$cross" =3D "debian-sid-cross-mips" - then - continue - fi - $LCITOOL dockerfile $host libvirt --cross $cross > ci-$host-cr= oss-$cross.Dockerfile - done - ;; - esac - - $LCITOOL dockerfile $host libvirt > ci-$host.Dockerfile -done diff --git a/ci/helper b/ci/helper new file mode 100755 index 0000000000..dec24ac741 --- /dev/null +++ b/ci/helper @@ -0,0 +1,187 @@ +#!/usr/bin/env python3 +# +# Copyright (C) 2021 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# . + +import argparse +import os +import pty +import shutil +import subprocess +import sys + + +class Parser: + def __init__(self): + self.parser =3D argparse.ArgumentParser() + subparsers =3D self.parser.add_subparsers( + dest=3D"action", + metavar=3D"ACTION", + ) + subparsers.required =3D True + + buildparser =3D subparsers.add_parser( + "build", + help=3D"run a build in a container", + ) + self.add_target_arg(buildparser) + self.add_engine_arg(buildparser) + self.add_login_arg(buildparser) + + shellparser =3D subparsers.add_parser( + "shell", + help=3D"start a shell in a container", + ) + self.add_target_arg(shellparser) + self.add_engine_arg(shellparser) + self.add_login_arg(shellparser) + + refreshparser =3D subparsers.add_parser( + "refresh", + help=3D"refresh data generated with lcitool", + ) + self.add_lcitool_arg(refreshparser) + + def add_target_arg(self, parser): + parser.add_argument( + "target", + help=3D"build on target OS", + ) + + def add_engine_arg(self, parser): + parser.add_argument( + "--engine", + choices=3D["auto", "podman", "docker"], + default=3D"auto", + help=3D"container engine to use", + ) + + def add_login_arg(self, parser): + parser.add_argument( + "--login", + default=3Dos.getlogin(), + help=3D"login to use inside the container", + ) + + def add_lcitool_arg(self, parser): + parser.add_argument( + "--lcitool", + metavar=3D"PATH", + default=3D"lcitool", + help=3D"path to lcitool binary", + ) + + def parse(self): + return self.parser.parse_args() + + +class Application: + def __init__(self): + self.basedir =3D os.path.dirname(os.path.realpath(__file__)) + + args =3D Parser().parse() + self.action =3D args.action + + if args.action =3D=3D "refresh": + self.lcitool =3D args.lcitool + if not shutil.which(self.lcitool): + sys.exit("error: 'lcitool' not installed") + + elif args.action in ["build", "shell"]: + self.target =3D args.target + self.engine =3D args.engine + self.login =3D args.login + + def make_run(self, target): + args =3D [ + "-C", self.basedir, target, + f"CI_ENGINE=3D{self.engine}", + f"CI_USER_LOGIN=3D{self.login}", + ] + + if pty.spawn(["make"] + args) !=3D 0: + raise Exception("make failed") + + def lcitool_run(self, args): + output =3D subprocess.check_output([self.lcitool] + args) + return output.decode("utf-8") + + def lcitool_get_hosts(self): + output =3D self.lcitool_run(["hosts"]) + return output.splitlines() + + def generate_dockerfile(self, host, cross=3DNone): + args =3D ["dockerfile", host, "libvirt"] + outfile =3D f"{self.basedir}/containers/ci-{host}.Dockerfile" + + if cross: + args.extend(["--cross", cross]) + outfile =3D f"{self.basedir}/containers/ci-{host}-cross-{cross= }.Dockerfile" + + output =3D self.lcitool_run(args) + with open(outfile, "w") as f: + f.write(output) + + def generate_vars(self, host): + output =3D self.lcitool_run(["variables", host, "libvirt"]) + with open(f"{self.basedir}/cirrus/{host}.vars", "w") as f: + f.write(output) + + def refresh_containers(self): + for host in self.lcitool_get_hosts(): + if "freebsd" in host or "macos" in host: + continue + + if host =3D=3D "fedora-rawhide": + for cross in ["mingw32", "mingw64"]: + print(f"containers/{host} ({cross})") + self.generate_dockerfile(host, cross) + + if "debian-" in host: + for cross in ["aarch64", "armv6l", "armv7l", "i686", "mips= ", "mips64el", "mipsel", "ppc64le", "s390x"]: + if host =3D=3D "debian-sid" and cross =3D=3D "mips": + continue + print(f"containers/{host} ({cross})") + self.generate_dockerfile(host, cross) + + print(f"containers/{host}") + self.generate_dockerfile(host) + + def refresh_cirrus(self): + for host in self.lcitool_get_hosts(): + if "freebsd" not in host and "macos" not in host: + continue + + print(f"cirrus/{host}") + self.generate_vars(host) + + def action_refresh(self): + self.refresh_containers() + self.refresh_cirrus() + + def action_build(self): + self.make_run(f"ci-build@{self.target}") + + def action_shell(self): + self.make_run(f"ci-shell@{self.target}") + + def run(self): + method =3D "action_{}".format(self.action.replace("-", "_")) + getattr(self, method).__call__() + + +if __name__ =3D=3D "__main__": + Application().run() --=20 2.26.2