From nobody Mon Dec 30 17:20:55 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) client-ip=170.10.129.124; envelope-from=patchew-devel-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1645689199; cv=none; d=zohomail.com; s=zohoarc; b=PIUqLCDiv6bv3vo55m56Nnh/NpxXJWNHkkPvK3tpxiBDz/u036wuRUNcoo+zTBNM4on9fUCg2AFhDS17MebfEBqTVOkiVsSNAhpY3+aLpXg+T2epDXQG2QFTGjsJQZZwV4BdrAHCpHQiN/SE4IFiK17DAOLVq/XHfkUd4gh+86U= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1645689199; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=7FYw0MJ07hwqYa3KJNO5bExqmhUcQFvAQ8XI0H6uH6E=; b=buTcLm32M5s+zYtxKlLK/0HUi7UTKBP0X4DBz8FCDESuOZEhBtSnTH7O7cw7w42jEStIQPOhvlumHu6EAhb/SpHZqBhmBgmqkZxRAgarjnrv4W7zGDjEiYiPwsQtMAk/1jPBvMKyOopG8s0K75GOlpgLegFJ5BqbZhRh90KXQ/s= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mx.zohomail.com with SMTPS id 1645689199862317.0134272793065; Wed, 23 Feb 2022 23:53:19 -0800 (PST) Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-625-qwFf6EV-NK6e_CytngVB_g-1; Thu, 24 Feb 2022 02:53:10 -0500 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id D86501091DA1; Thu, 24 Feb 2022 07:53:08 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D0C82519D0; Thu, 24 Feb 2022 07:53:08 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 8633E4A701; Thu, 24 Feb 2022 07:53:08 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id 21O7r7kR008815 for ; Thu, 24 Feb 2022 02:53:07 -0500 Received: by smtp.corp.redhat.com (Postfix) id 4DBBA519D0; Thu, 24 Feb 2022 07:53:07 +0000 (UTC) Received: from avogadro.lan (unknown [10.39.193.11]) by smtp.corp.redhat.com (Postfix) with ESMTP id 546F0519CF; Thu, 24 Feb 2022 07:52:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1645689198; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=7FYw0MJ07hwqYa3KJNO5bExqmhUcQFvAQ8XI0H6uH6E=; b=JrTaMhgLvZAvrCf9wjvv6OWdrB/yYX2dPJaIFZ6oJhEvgVrI0kTMdsdrx6HMhu9TPv42Xm PQC6n7X5gGgvN1QBWt7cnv15x5famVDGt6L/JLyTZCUhJj2CJ6vpcTGSYrP6XdSds8ihhE BKKi7rnyMy0F8VFw8yhlXpLdwBKfDMk= X-MC-Unique: qwFf6EV-NK6e_CytngVB_g-1 From: Paolo Bonzini To: patchew-devel@redhat.com Date: Thu, 24 Feb 2022 08:52:58 +0100 Message-Id: <20220224075258.316670-1-pbonzini@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-loop: patchew-devel@redhat.com Cc: dcaratti@redhat.com Subject: [Patchew-devel] [PATCH] scripts: separate applier and importer X-BeenThere: patchew-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Patchew development and discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: patchew-devel-bounces@redhat.com Errors-To: patchew-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=patchew-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1645689200843100003 Content-Type: text/plain; charset="utf-8" With two different kinds of importer now in existence, separate the git application and the email import tasks. Signed-off-by: Paolo Bonzini --- scripts/deploy | 10 +++- scripts/dockerfiles/applier.docker | 11 +++++ scripts/dockerfiles/importer.docker | 8 +--- scripts/patchew-applier | 37 ++++++++++++++ scripts/patchew-importer | 16 ------- scripts/playbooks/deploy-appliers.yml | 48 +++++++++++++++++++ scripts/playbooks/deploy-importers.yml | 13 ----- scripts/playbooks/templates/applier-config.j2 | 4 ++ .../playbooks/templates/importer-config.j2 | 1 - 9 files changed, 110 insertions(+), 38 deletions(-) create mode 100644 scripts/dockerfiles/applier.docker create mode 100755 scripts/patchew-applier create mode 100644 scripts/playbooks/deploy-appliers.yml create mode 100644 scripts/playbooks/templates/applier-config.j2 diff --git a/scripts/deploy b/scripts/deploy index 1953e9b..e05984d 100755 --- a/scripts/deploy +++ b/scripts/deploy @@ -23,6 +23,8 @@ def parse_args(): help=3D"Tester host address") parser.add_argument("--importer", "-i", nargs=3D"?", help=3D"Importer host address") + parser.add_argument("--applier", "-a", nargs=3D"?", + help=3D"Importer host address") parser.add_argument("--debug", "-D", action=3D"store_true", help=3D"Enable debug output") return parser.parse_known_args() @@ -36,6 +38,9 @@ def generate_inventory_file(args): [db] %s =20 +[appliers] +%s + [importers] %s =20 @@ -43,6 +48,7 @@ def generate_inventory_file(args): %s""" \ % (args.web_server or "", args.db_server or "", + args.applier or "", args.importer or "", "\n".join(args.testers or []))) f.flush() @@ -60,10 +66,12 @@ def main(): playbooks.append("deploy-db.yml") if args.testers: playbooks.append("deploy-testers.yml") + if args.applier: + playbooks.append("deploy-appliers.yml") if args.importer: playbooks.append("deploy-importers.yml") if not playbooks: - print("At least one host (web server, importer or tester) must be = specified") + print("At least one host (server/db/applier/importer/tester) must = be specified") print("Run: '%s -h' for more information" % sys.argv[0]) return 1 v =3D ["-vvv"] if args.debug else [] diff --git a/scripts/dockerfiles/applier.docker b/scripts/dockerfiles/appli= er.docker new file mode 100644 index 0000000..dded23a --- /dev/null +++ b/scripts/dockerfiles/applier.docker @@ -0,0 +1,11 @@ +FROM fedora:latest +RUN dnf install -y python git wget +RUN git config --global user.email "applier@patchew.org" +RUN git config --global user.name "Patchew Applier" +RUN mkdir -p -m 0700 ~/.ssh +RUN echo IdentityFile=3D/data/patchew/identity > ~/.ssh/config +RUN echo StrictHostKeyChecking no >> ~/.ssh/config +RUN echo UserKnownHostsFile=3D/dev/null >> ~/.ssh/config +ENV LC_ALL en_US.UTF-8 +COPY . /opt/patchew/ +CMD /opt/patchew/scripts/patchew-applier diff --git a/scripts/dockerfiles/importer.docker b/scripts/dockerfiles/impo= rter.docker index 0021d93..b08b023 100644 --- a/scripts/dockerfiles/importer.docker +++ b/scripts/dockerfiles/importer.docker @@ -1,11 +1,5 @@ FROM fedora:latest -RUN dnf install -y python offlineimap findutils git wget -RUN git config --global user.email "importer@patchew.org" -RUN git config --global user.name "Patchew Importer" -RUN mkdir -p -m 0700 ~/.ssh -RUN echo IdentityFile=3D/data/patchew/identity > ~/.ssh/config -RUN echo StrictHostKeyChecking no >> ~/.ssh/config -RUN echo UserKnownHostsFile=3D/dev/null >> ~/.ssh/config +RUN dnf install -y python offlineimap findutils wget ENV LC_ALL en_US.UTF-8 COPY . /opt/patchew/ CMD /opt/patchew/scripts/patchew-importer diff --git a/scripts/patchew-applier b/scripts/patchew-applier new file mode 100755 index 0000000..b06b7d2 --- /dev/null +++ b/scripts/patchew-applier @@ -0,0 +1,37 @@ +#!/bin/bash +# +# Copyright 2016 Red Hat, Inc. +# +# Authors: +# Fam Zheng +# +# This work is licensed under the MIT License. Please see the LICENSE fil= e or +# http://opensource.org/licenses/MIT. + +set -e + +. /data/patchew/config + +BASEDIR=3D"/data/patchew" +PATCHEW_CLI=3D"/opt/patchew/patchew-cli -s $PATCHEW_SERVER" + +while :; do + $PATCHEW_CLI login "$PATCHEW_USER" "$PATCHEW_PASS" + while :; do + wd=3D$(mktemp -d /var/tmp/patchew-applier.XXXXX) + applier_args=3D'--applier-mode ' + if test "$PATCHEW_TARGET_REPO" !=3D ""; then + applier_args=3D"$applier_args --applier-target $PATCHEW_TARGET_REPO" + fi + if (cd $wd; $PATCHEW_CLI apply $applier_args); then + rm -rf $wd + elif [ $? -eq 3 ]; then + rm -rf $wd + break + else + rm -rf $wd + fi + done + $PATCHEW_CLI project update + sleep 60 +done diff --git a/scripts/patchew-importer b/scripts/patchew-importer index 24e0b96..d7754f1 100755 --- a/scripts/patchew-importer +++ b/scripts/patchew-importer @@ -107,22 +107,6 @@ offlineimap_import() =20 while :; do $PATCHEW_CLI login "$PATCHEW_USER" "$PATCHEW_PASS" - while :; do - wd=3D$(mktemp -d /var/tmp/patchew-applier.XXXXX) - applier_args=3D'--applier-mode ' - if test "$PATCHEW_TARGET_REPO" !=3D ""; then - applier_args=3D"$applier_args --applier-target $PATCHEW_TARGET_REPO" - fi - if (cd $wd; $PATCHEW_CLI apply $applier_args); then - rm -rf $wd - elif [ $? -eq 3 ]; then - rm -rf $wd - break - else - rm -rf $wd - fi - done - $PATCHEW_CLI project update offlineimap_import sleep 60 done diff --git a/scripts/playbooks/deploy-appliers.yml b/scripts/playbooks/depl= oy-appliers.yml new file mode 100644 index 0000000..18edac7 --- /dev/null +++ b/scripts/playbooks/deploy-appliers.yml @@ -0,0 +1,48 @@ +- hosts: appliers + vars_prompt: + - name: instance_name + prompt: "The instance name" + default: patchew-applier + private: no + - name: "patchew_server" + prompt: "The address of patchew server" + default: "https://patchew.org" + private: no + - name: "applier_user" + prompt: "Username for the applier to login to the server" + private: no + default: "applier" + - name: "applier_pass" + prompt: "Password for the applier to login to the server" + private: yes + - name: "applier_identity" + prompt: "Path to file containing private key" + private: no + - name: "applier_repo" + prompt: "Target repository allowed by the private key (e.g. git@gith= ub.com:youruser)" + default: "" + private: no + vars: + base_dir: "/data/{{ instance_name }}" + src_dir: "{{ base_dir }}/src" + data_dir: "{{ base_dir }}/data" + config_file: "{{ data_dir }}/config" + identity_file: "{{ data_dir }}/identity" + tasks: + - name: Create data dir + file: + path: "{{ data_dir }}" + state: directory + - name: Store SSH key + copy: + src: "{{ applier_identity }}" + dest: "{{ identity_file }}" + validate: test -f %s + mode: 0400 + - name: Create config + template: + src: "templates/applier-config.j2" + dest: "{{ config_file }}" + - import_tasks: tasks/docker-deploy.yml + vars: + instance_role: applier diff --git a/scripts/playbooks/deploy-importers.yml b/scripts/playbooks/dep= loy-importers.yml index 2e45db8..4eb7cbd 100644 --- a/scripts/playbooks/deploy-importers.yml +++ b/scripts/playbooks/deploy-importers.yml @@ -15,13 +15,6 @@ - name: "importer_pass" prompt: "Password for the importer to login to the server" private: yes - - name: "importer_identity" - prompt: "Path to file containing private key" - private: no - - name: "importer_repo" - prompt: "Target repository allowed by the private key (e.g. git@gith= ub.com:youruser)" - default: "" - private: no - name: "imap_server" prompt: "IMAP server address to download patches" private: no @@ -58,12 +51,6 @@ file: path: "{{ data_dir }}" state: directory - - name: Store SSH key - copy: - src: "{{ importer_identity }}" - dest: "{{ identity_file }}" - validate: test -f %s - mode: 0400 - name: Create config template: src: "templates/importer-config.j2" diff --git a/scripts/playbooks/templates/applier-config.j2 b/scripts/playbo= oks/templates/applier-config.j2 new file mode 100644 index 0000000..6a5687e --- /dev/null +++ b/scripts/playbooks/templates/applier-config.j2 @@ -0,0 +1,4 @@ +PATCHEW_SERVER=3D{{ patchew_server }} +PATCHEW_USER=3D{{ applier_user }} +PATCHEW_PASS=3D{{ applier_pass }} +PATCHEW_TARGET_REPO=3D{{ applier_repo }} diff --git a/scripts/playbooks/templates/importer-config.j2 b/scripts/playb= ooks/templates/importer-config.j2 index 39f1fd9..b6bc844 100644 --- a/scripts/playbooks/templates/importer-config.j2 +++ b/scripts/playbooks/templates/importer-config.j2 @@ -1,7 +1,6 @@ PATCHEW_SERVER=3D{{ patchew_server }} PATCHEW_USER=3D{{ importer_user }} PATCHEW_PASS=3D{{ importer_pass }} -PATCHEW_TARGET_REPO=3D{{ importer_repo }} =20 IMAP_SERVER=3D{{ imap_server }} IMAP_USER=3D{{ imap_user }} --=20 2.34.1 _______________________________________________ Patchew-devel mailing list Patchew-devel@redhat.com https://listman.redhat.com/mailman/listinfo/patchew-devel