From nobody Thu Mar 28 15:32:56 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) client-ip=170.10.133.124; envelope-from=patchew-devel-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1649258924; cv=none; d=zohomail.com; s=zohoarc; b=DxhPPrFW4+rE7t0QjJdpODLK+D61pStNPxu5J3Z0vw9PiO7j7O+fWPbCuJCXKyAgbSQdyycLEndemvVia0iPWmj7OfKJ5F7SfCuzC6DOK54zTE1kiTSu2q7Ed1fKnUkqdvUGxgkl5p3L778X6rZQgjG1KzhORXvzisLo8VY6KWc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1649258924; h=Content-Type:Content-Transfer-Encoding:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=tEJtnScLOL/iX6874Rm7zLUGrIwEK3qVPrNV11fIIvs=; b=X01RICytgxwDFRIVrW3vUmEFK9jiTHfEPmTCXPXcrRSDTAESMK4PlIP54hXDyJli/o849sHsV88kGxdrQg8sAq0E0CbWxOpk6Um1uYQWyM00PJAHGAiFW7yJeEO2JkwWcJhkQRR3pQnBiuqSZkFKaCeCkAsAFl7CwI2RF7wuoN8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mx.zohomail.com with SMTPS id 1649258924543318.85920367883284; Wed, 6 Apr 2022 08:28:44 -0700 (PDT) Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-472-EM0jsnYeM6G81EsW2FlVfw-1; Wed, 06 Apr 2022 11:28:40 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 26004185A79C; Wed, 6 Apr 2022 15:28:40 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1C12B40470ED; Wed, 6 Apr 2022 15:28:40 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id D730B1949763; Wed, 6 Apr 2022 15:28:39 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 023F71949763 for ; Wed, 6 Apr 2022 15:28:38 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id D18F5C1D38B; Wed, 6 Apr 2022 15:28:38 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast03.extmail.prod.ext.rdu2.redhat.com [10.11.55.19]) by smtp.corp.redhat.com (Postfix) with ESMTPS id CD985C1D3AD for ; Wed, 6 Apr 2022 15:28:38 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AE567811E84 for ; Wed, 6 Apr 2022 15:28:38 +0000 (UTC) Received: from mail-ej1-f54.google.com (mail-ej1-f54.google.com [209.85.218.54]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-199-k8-HzA-YPV6JlPbDzN876w-1; Wed, 06 Apr 2022 11:28:37 -0400 Received: by mail-ej1-f54.google.com with SMTP id k23so5060955ejd.3 for ; Wed, 06 Apr 2022 08:28:36 -0700 (PDT) Received: from avogadro.lan ([2001:b07:6468:f312:c8dd:75d4:99ab:290a]) by smtp.gmail.com with ESMTPSA id x4-20020a170906b08400b006e493cb583esm6682490ejy.47.2022.04.06.08.28.35 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 06 Apr 2022 08:28:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649258923; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=tEJtnScLOL/iX6874Rm7zLUGrIwEK3qVPrNV11fIIvs=; b=Y+0sUfj1f17eOy3EAKKhfSEN6U1Yp0qLTcgXphbvrMdH0gvtmZOjB9wBfiFulKUpu2s5Fy 6+M2M+YCLIqkl8OJx5GbvHvDXMOzB+TJa/Xw4uGNo24skkiqyYgRN7MhTHsUgqxlDNS1TH xWEdWa2lkY4HakbIqNCsNR9sIcXW1do= X-MC-Unique: EM0jsnYeM6G81EsW2FlVfw-1 X-Original-To: patchew-devel@listman.corp.redhat.com X-MC-Unique: k8-HzA-YPV6JlPbDzN876w-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:from:to:subject:date:message-id :mime-version:content-transfer-encoding; bh=oxqqRRUAUTsVdWpp1af26QYPKZuKDv9yL404v15hHi0=; b=7OB0cknahrfX2rlgWFZjxnzc7ZdG4z8zzio3x9TT5qsn7oSeg2sWfBPRlNtsJE7e6Q TOdWmuCPljd/i229D89S2qFZprxOyYo3TlPmhphgHvPoB0mNIJp0CuzYjvNXz478cjy5 m8RdkBEksSzhsAcfw4m9r/3Ib4AyU3CFTyL0yKf+QpfiQumjw/FCIbVqq3q7z1RfZNml 3uBR8R26WEs/SReqDXL+jpHrhKji9BcSPjhUaKwiSBLFK6DuQD3Pm+tr5SHwyXs5LD9X 6o6Wd497/6Ln30zegahdAHFQRY9CQogLBXvOEAXIS8qvfkbUDwuBT7rwesdUTdlR9hKN F7Kw== X-Gm-Message-State: AOAM533W7XpaJEfH4Dml8WsfeD7ii/szq4Ndm3h0SPqgXdPKaJ/Rdczc CNur2FaIRLvXby6nOhsj/NYJkyLHzRug3g== X-Google-Smtp-Source: ABdhPJyNLBKtXdrsyKR846UGfV21Dpjemq821L98OvAw0j4+jhb1OJmurj0bK5nCjKQ0MjGcS4ONOg== X-Received: by 2002:a17:906:c151:b0:6e8:1e6:da3 with SMTP id dp17-20020a170906c15100b006e801e60da3mr8781717ejc.732.1649258915875; Wed, 06 Apr 2022 08:28:35 -0700 (PDT) From: Paolo Bonzini To: patchew-devel@redhat.com Date: Wed, 6 Apr 2022 17:28:34 +0200 Message-Id: <20220406152834.25422-1-pbonzini@redhat.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.85 on 10.11.54.8 Subject: [Patchew-devel] [RFC PATCH] switch from docker to podman X-BeenThere: patchew-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Patchew development and discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: patchew-devel-bounces@redhat.com Sender: "Patchew-devel" X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=patchew-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1649258925928100003 Content-Type: text/plain; charset="utf-8" This is not really complete yet, and should be split into multiple patches, but I have already tried running the playbook on a spare server except for the systemd start step; it seems to work. I would like to move the patchew.org services to a new server because next.patchew.org is already running low on disk space, and I would like to experiment there with removing the blob storage for messages. I'd like to avoid that running out of disk space during the conversion messes up the other containers. While sooner or later I would like to move to Kubernetes, this intermediate step also lets us move the hosts to a more recent version of CentOS than CentOS 7. Paolo --- patchew-server.service | 13 ----- patchew-tester.service | 15 ------ scripts/playbooks/deploy-appliers.yml | 8 ++- scripts/playbooks/deploy-db.yml | 8 ++- scripts/playbooks/deploy-importers-lore.yml | 10 ++-- scripts/playbooks/deploy-importers.yml | 8 ++- scripts/playbooks/deploy-servers.yml | 12 +++-- .../{docker-deploy.yml =3D> podman-deploy.yml} | 51 +++++++++---------- .../{docker.service.j2 =3D> podman.service.j2} | 12 ++--- 9 files changed, 63 insertions(+), 74 deletions(-) delete mode 100644 patchew-server.service delete mode 100644 patchew-tester.service rename scripts/playbooks/tasks/{docker-deploy.yml =3D> podman-deploy.yml} = (48%) rename scripts/playbooks/templates/{docker.service.j2 =3D> podman.service.= j2} (38%) diff --git a/patchew-server.service b/patchew-server.service deleted file mode 100644 index d8b17ea..0000000 --- a/patchew-server.service +++ /dev/null @@ -1,13 +0,0 @@ -[Unit] -Description=3DPatchew server docker instance -Requires=3Ddocker.service -After=3Ddocker.service - -[Service] -Restart=3Dalways -ExecStartPre=3D-/usr/bin/docker stop patchew-server ; -/usr/bin/docker rm = patchew-server -ExecStart=3D /usr/bin/docker run --privileged --name patchew-server -v /da= ta:/data:rw -p 8081:80 patchew:server -ExecStop=3D/usr/bin/docker stop -t 10 patchew-server - -[Install] -WantedBy=3Ddefault.target diff --git a/patchew-tester.service b/patchew-tester.service deleted file mode 100644 index 118c76a..0000000 --- a/patchew-tester.service +++ /dev/null @@ -1,15 +0,0 @@ -[Unit] -Description=3DPatchew Tester -After=3Dnetwork.target - -[Service] -ExecStart=3D/usr/bin/patchew-tester -Restart=3Dalways -RestartSec=3D30 -User=3Droot -StandardOutput=3Dsyslog+journal -StandardError=3Dinherit - -[Install] -WantedBy=3Dmulti-user.target - diff --git a/scripts/playbooks/deploy-appliers.yml b/scripts/playbooks/depl= oy-appliers.yml index 18edac7..863c167 100644 --- a/scripts/playbooks/deploy-appliers.yml +++ b/scripts/playbooks/deploy-appliers.yml @@ -4,6 +4,10 @@ prompt: "The instance name" default: patchew-applier private: no + - name: container_dir + prompt: "The base directory for the container" + default: /data + private: no - name: "patchew_server" prompt: "The address of patchew server" default: "https://patchew.org" @@ -23,7 +27,7 @@ default: "" private: no vars: - base_dir: "/data/{{ instance_name }}" + base_dir: "{{ container_dir }}/{{ instance_name }}" src_dir: "{{ base_dir }}/src" data_dir: "{{ base_dir }}/data" config_file: "{{ data_dir }}/config" @@ -43,6 +47,6 @@ template: src: "templates/applier-config.j2" dest: "{{ config_file }}" - - import_tasks: tasks/docker-deploy.yml + - import_tasks: tasks/podman-deploy.yml vars: instance_role: applier diff --git a/scripts/playbooks/deploy-db.yml b/scripts/playbooks/deploy-db.= yml index 2b69995..1db810f 100644 --- a/scripts/playbooks/deploy-db.yml +++ b/scripts/playbooks/deploy-db.yml @@ -4,8 +4,12 @@ prompt: "Instance name of the patchew postgres db" default: patchew-server-db private: no + - name: container_dir + prompt: "The base directory for the container" + default: /data + private: no vars: - base_dir: "/data/{{ instance_name }}" + base_dir: "{{ container_dir }}/{{ instance_name }}" src_dir: "{{ base_dir }}/src" data_dir: "{{ base_dir }}/data" tasks: @@ -13,6 +17,6 @@ file: path: "{{ data_dir }}" state: directory - - import_tasks: tasks/docker-deploy.yml + - import_tasks: tasks/podman-deploy.yml vars: instance_role: db diff --git a/scripts/playbooks/deploy-importers-lore.yml b/scripts/playbook= s/deploy-importers-lore.yml index 9f80d93..ab76ba6 100644 --- a/scripts/playbooks/deploy-importers-lore.yml +++ b/scripts/playbooks/deploy-importers-lore.yml @@ -4,6 +4,10 @@ prompt: "The instance name" default: patchew-importer-lore private: no + - name: container_dir + prompt: "The base directory for the container" + default: /data + private: no - name: "patchew_server" prompt: "The address of patchew server" default: "https://patchew.org" @@ -20,11 +24,11 @@ default: "https://lore.kernel.org/lkml/" private: no vars: - base_dir: "/data/{{ instance_name }}" + base_dir: "{{ container_dir }}/{{ instance_name }}" src_dir: "{{ base_dir }}/src" data_dir: "{{ base_dir }}/data" config_file: "{{ data_dir }}/config" - docker_run_args: "--init" + podman_run_args: "--init" tasks: - name: Create data dir file: @@ -34,6 +38,6 @@ template: src: "templates/importer-lore-config.j2" dest: "{{ config_file }}" - - import_tasks: tasks/docker-deploy.yml + - import_tasks: tasks/podman-deploy.yml vars: instance_role: importer-lore diff --git a/scripts/playbooks/deploy-importers.yml b/scripts/playbooks/dep= loy-importers.yml index 8d362ee..3711dbd 100644 --- a/scripts/playbooks/deploy-importers.yml +++ b/scripts/playbooks/deploy-importers.yml @@ -4,6 +4,10 @@ prompt: "The instance name" default: patchew-importer private: no + - name: container_dir + prompt: "The base directory for the container" + default: /data + private: no - name: "patchew_server" prompt: "The address of patchew server" default: "https://patchew.org" @@ -41,7 +45,7 @@ default: "y" private: no vars: - base_dir: "/data/{{ instance_name }}" + base_dir: "{{ container_dir }}/{{ instance_name }}" src_dir: "{{ base_dir }}/src" data_dir: "{{ base_dir }}/data" config_file: "{{ data_dir }}/config" @@ -54,6 +58,6 @@ template: src: "templates/importer-config.j2" dest: "{{ config_file }}" - - import_tasks: tasks/docker-deploy.yml + - import_tasks: tasks/podman-deploy.yml vars: instance_role: importer diff --git a/scripts/playbooks/deploy-servers.yml b/scripts/playbooks/deplo= y-servers.yml index 18acb36..4a90c75 100644 --- a/scripts/playbooks/deploy-servers.yml +++ b/scripts/playbooks/deploy-servers.yml @@ -4,6 +4,10 @@ prompt: "Instance name of the webserver" default: patchew-server private: no + - name: container_dir + prompt: "The base directory for the container" + default: /data + private: no - name: superuser_name prompt: "Admin account to create (optional)" default: "" @@ -13,22 +17,22 @@ default: "adminpass" private: yes vars: - base_dir: "/data/{{ instance_name }}" + base_dir: "{{ container_dir }}/{{ instance_name }}" src_dir: "{{ base_dir }}/src" data_dir: "{{ base_dir }}/data" - docker_run_args: "--link {{ instance_name }}-db:patchew-db" + podman_run_args: "--link {{ instance_name }}-db:patchew-db" tasks: - name: Create data dir file: path: "{{ data_dir }}" state: directory - - import_tasks: tasks/docker-deploy.yml + - import_tasks: tasks/podman-deploy.yml vars: instance_role: server - name: Create superuser when: superuser_name !=3D "" shell: | - docker exec -i {{ instance_name }} bash -c " + podman exec -i {{ instance_name }} bash -c " cd /opt/patchew && . venv/bin/activate && ./manage.py migrate && diff --git a/scripts/playbooks/tasks/docker-deploy.yml b/scripts/playbooks/= tasks/podman-deploy.yml similarity index 48% rename from scripts/playbooks/tasks/docker-deploy.yml rename to scripts/playbooks/tasks/podman-deploy.yml index 2cd2f32..45328ba 100644 --- a/scripts/playbooks/tasks/docker-deploy.yml +++ b/scripts/playbooks/tasks/podman-deploy.yml @@ -4,27 +4,17 @@ name: rsync - name: Install pip package: - name: python-pip -- name: Install docker + name: python3-pip +- name: Install podman package: - name: docker -- name: Start docker daemon - service: - name: docker - state: started -- name: Install docker python library for ansible - pip: - name: docker-py -- name: Install SELinux python bindings (for systemd setup) - package: - name: libselinux-python + name: podman - name: Stop systemd service for "{{ instance_name }}" service: name: "{{ instance_name }}" state: stopped ignore_errors: yes -- name: Stop docker instance - docker_container: +- name: Stop podman instance + containers.podman.podman_container: name: "{{ instance_name }}" state: absent - name: Create patchew data folder @@ -43,24 +33,33 @@ - "--exclude=3D*.pyc" - "--exclude=3D*.swp" - "--exclude=3D/venv" -- name: Delete old docker image stash - docker_image: - force_absent: yes +- name: Check for existing backup image + containers.podman.podman_image_info: name: "patchew:{{ instance_name }}-prev" - state: absent -- name: Stash docker image - shell: docker tag "patchew:{{ instance_name }}" "patchew:{{ instance_nam= e }}-prev" || true -- name: Rebuild docker image - # docker_image module doesn't work on Fedora - shell: docker build -t "patchew:{{ instance_name }}" -f "{{ src_dir }}/s= cripts/dockerfiles/{{ instance_role }}.docker" "{{ src_dir }}" + register: prev_image +- name: Delete old podman image stash + shell: "podman untag 'patchew:{{ instance_name }}-prev'" + when: prev_image.images +- name: Check for existing image + containers.podman.podman_image_info: + name: "patchew:{{ instance_name }}" + register: current_image +- name: Stash podman image + containers.podman.podman_tag: + image: "patchew:{{ instance_name }}" + target_names: "patchew:{{ instance_name }}-prev" + when: current_image.images +- name: Rebuild podman image + # docker_image module wants a file named Dockerfile + shell: "podman build -t 'patchew:{{ instance_name }}' -f '{{ src_dir }}/= scripts/dockerfiles/{{ instance_role }}.docker' '{{ src_dir }}'" - name: Install systemd service template: - src: "templates/docker.service.j2" + src: "templates/podman.service.j2" dest: "/etc/systemd/system/{{ instance_name }}.service" - name: Systemd daemon reload systemd: daemon_reload: yes -- name: Start docker instance +- name: Start podman instance service: name: "{{ instance_name }}" state: restarted diff --git a/scripts/playbooks/templates/docker.service.j2 b/scripts/playbo= oks/templates/podman.service.j2 similarity index 38% rename from scripts/playbooks/templates/docker.service.j2 rename to scripts/playbooks/templates/podman.service.j2 index 85dd9d2..e1edfb9 100644 --- a/scripts/playbooks/templates/docker.service.j2 +++ b/scripts/playbooks/templates/podman.service.j2 @@ -1,18 +1,16 @@ [Unit] -Description=3DPatchew docker instance control service for {{ instance_name= }} -Requires=3Ddocker.service -After=3Ddocker.service +Description=3DPatchew podman instance control service for {{ instance_name= }} StartLimitIntervalSec=3D0 =20 [Service] Restart=3Dalways -ExecStartPre=3D-/usr/bin/docker stop {{ instance_name }} ; -/usr/bin/docke= r rm {{ instance_name }} -ExecStart=3D/usr/bin/docker run --privileged --name {{ instance_name }} \ +ExecStartPre=3D-podman stop {{ instance_name }} ; -podman rm {{ instance_n= ame }} +ExecStart=3Dpodman run --privileged --name {{ instance_name }} \ -v {{ data_dir }}:/data/patchew:rw \ -e PATCHEW_DATA_DIR=3D/data/patchew \ - {{ docker_run_args | default() }} \ + {{ podman_run_args | default() }} \ patchew:{{ instance_name }} -ExecStop=3D/usr/bin/docker stop -t 10 {{ instance_name }} +ExecStop=3Dpodman stop -t 10 {{ instance_name }} RestartSec=3D60 =20 [Install] --=20 2.35.1 _______________________________________________ Patchew-devel mailing list Patchew-devel@redhat.com https://listman.redhat.com/mailman/listinfo/patchew-devel