From nobody Fri Apr 19 15:40:09 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) client-ip=170.10.133.124; envelope-from=patchew-devel-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1649321253; cv=none; d=zohomail.com; s=zohoarc; b=QAQe982EGvILtW1iDnwPnvg9knBPbv0tlsvNK4g3KTJct1BHpWde8kKunlE8HDrWvfuTPMXZ05i88SEVGkj543rJZCSjcXIrvfnhU0rvHD1s6HwQRMjguOIuGoBZ/pNVH3aNKlA6sVE49JxqhEZEyd+HgJQBAx4X7eKhH6HRmU8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1649321253; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=8cO47q28X3eoOEORNihs8AU1/U00DwuP4OXp5zQrnGc=; b=fQQ3XfSJ8jOr089b0TLXEwuGybsT0kKM1knLs1q0PxllBWpw06U+5S8CIqyqI31vohx1UBOp4fzXOIg7CbkxMsRfyHK4XYqze69T90FDdGwEZpV7CGkHwuOPH1VRaVVsHEcvemFRwH+IBxueqxDgPSeECyXN3nk7DFSOWdDf8e4= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mx.zohomail.com with SMTPS id 1649321253265248.5417458443053; Thu, 7 Apr 2022 01:47:33 -0700 (PDT) Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-102-VvzJa2sxPLm87dZ1BNbdfA-1; Thu, 07 Apr 2022 04:47:29 -0400 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6FE912999B54; Thu, 7 Apr 2022 08:47:28 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3F38E432721; Thu, 7 Apr 2022 08:47:25 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id BA5C0194034C; Thu, 7 Apr 2022 08:47:23 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id CA5D9194034F for ; Thu, 7 Apr 2022 08:47:22 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id 8202E4066C9F; Thu, 7 Apr 2022 08:47:22 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast06.extmail.prod.ext.rdu2.redhat.com [10.11.55.22]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 7D0FA407DEC5 for ; Thu, 7 Apr 2022 08:47:22 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-1.mimecast.com [207.211.31.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 0F17F18A6595 for ; Thu, 7 Apr 2022 08:47:22 +0000 (UTC) Received: from mail-ej1-f54.google.com (mail-ej1-f54.google.com [209.85.218.54]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-621-MRU2J8iQNXG1tXmHfZyX5Q-1; Thu, 07 Apr 2022 04:47:09 -0400 Received: by mail-ej1-f54.google.com with SMTP id dr20so9310335ejc.6 for ; Thu, 07 Apr 2022 01:47:08 -0700 (PDT) Received: from avogadro.redhat.com ([2001:b07:6468:f312:c8dd:75d4:99ab:290a]) by smtp.gmail.com with ESMTPSA id r22-20020a17090638d600b006d584aaa9c9sm7453703ejd.133.2022.04.07.01.47.07 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Apr 2022 01:47:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649321252; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=8cO47q28X3eoOEORNihs8AU1/U00DwuP4OXp5zQrnGc=; b=W9xYPP4QaHVdjEOS2BqtESsoCFvYmLhJL9VzgYI764ar9ba5b7HEETS5IEzYmijeAWYj5w Q+SsUBwAxOQFn76Bs6Uz/1auXTjgvC8HvKf8hn5xT2BW2YMN7xhdfwQuqS+kwCu5uJbPGP Sj7v+oXovJEoD/By772JIjR+lXyQ670= X-MC-Unique: VvzJa2sxPLm87dZ1BNbdfA-1 X-Original-To: patchew-devel@listman.corp.redhat.com X-MC-Unique: MRU2J8iQNXG1tXmHfZyX5Q-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:from:to:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=cOdPXC/jS1v3YAz06fczPWslHZzTDOnzs8Dl9ZiV15A=; b=QhOUlI6Fu8ri2vofG2mUNPLbwJD03K1tStYiSZPYXAu7eySjvBOgxRD9GH8bnj0iaS 6trGPq30K7lSaXeD0/AVLh/FeRRSasJvhNKCUCj5a8JrE0GRrmqPe/vT4un96SUlJW1I cvzwqzYznkqwa5w+kIFuskNQCXmHClf/YuRv1Daqaykbvg9kWc661xu5Fq8JmLvvIbX0 dKka6MYRT6Cwud/Sa9xhpklPwgAd894NcHHXKvTuHF1dlUqpYTAEU8qffcQ1tpi4e7cx Gc2q4Lo6hrbRn5lfed2pQeH25iRo+x17UmiIxq5ITADRfUQJPvY1v8M9sYxhZYuvxFnr 0NxA== X-Gm-Message-State: AOAM532AHJGoC4CWZhADHXPWLMsGZP2psmTFkyZ5RB3O1Ppji8d5cjTv mRTgVwt2pJ4uXrP5+DqRRaX49xRiMpGHRg== X-Google-Smtp-Source: ABdhPJyPwKaYqJoBjZFxD6PNgIcx8deikuAtB00l1fijd7RP17nCwg7p2Mfkas53Fdwiu+YVFdEVug== X-Received: by 2002:a17:907:7294:b0:6e8:3962:80e6 with SMTP id dt20-20020a170907729400b006e8396280e6mr1697636ejc.297.1649321227906; Thu, 07 Apr 2022 01:47:07 -0700 (PDT) From: Paolo Bonzini To: patchew-devel@redhat.com Date: Thu, 7 Apr 2022 10:47:00 +0200 Message-Id: <20220407084704.78533-2-pbonzini@redhat.com> In-Reply-To: <20220407084704.78533-1-pbonzini@redhat.com> References: <20220407084704.78533-1-pbonzini@redhat.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.84 on 10.11.54.1 Subject: [Patchew-devel] [PATCH 1/5] models: start moving away from placing mboxes in the filesystem X-BeenThere: patchew-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Patchew development and discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: patchew-devel-bounces@redhat.com Sender: "Patchew-devel" X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=patchew-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1649321254614100008 Content-Type: text/plain; charset="utf-8" Teach the Message class to retrieve the mbox from the database instead of going to the filesystem. An enclosed scripts can be run within "./manage.py shell" to migrate all messages from the filesystem into the database while the server is running. --- api/blobs.py | 8 -------- api/migrations/0061_message_mbox_bytes.py | 18 ++++++++++++++++ api/models.py | 24 ++++++++++++++-------- scripts/blob_migrate.py | 25 +++++++++++++++++++++++ 4 files changed, 58 insertions(+), 17 deletions(-) create mode 100644 api/migrations/0061_message_mbox_bytes.py create mode 100644 scripts/blob_migrate.py diff --git a/api/blobs.py b/api/blobs.py index ce87a35..502076e 100644 --- a/api/blobs.py +++ b/api/blobs.py @@ -17,14 +17,6 @@ from django.conf import settings import lzma =20 =20 -def save_blob(data, name=3DNone): - if not name: - name =3D str(uuid.uuid4()) - fn =3D os.path.join(settings.DATA_DIR, "blob", name + ".xz") - lzma.open(fn, "w").write(data.encode("utf-8")) - return name - - def load_blob(name): fn =3D os.path.join(settings.DATA_DIR, "blob", name + ".xz") return lzma.open(fn, "r").read().decode("utf-8") diff --git a/api/migrations/0061_message_mbox_bytes.py b/api/migrations/006= 1_message_mbox_bytes.py new file mode 100644 index 0000000..18c4dd5 --- /dev/null +++ b/api/migrations/0061_message_mbox_bytes.py @@ -0,0 +1,18 @@ +# Generated by Django 3.1.14 on 2022-04-06 07:01 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies =3D [ + ('api', '0060_auto_20210106_1120'), + ] + + operations =3D [ + migrations.AddField( + model_name=3D'message', + name=3D'mbox_bytes', + field=3Dmodels.BinaryField(null=3DTrue), + ), + ] diff --git a/api/models.py b/api/models.py index 19b3203..ce38913 100644 --- a/api/models.py +++ b/api/models.py @@ -22,7 +22,7 @@ import lzma from mbox import MboxMessage, decode_payload from patchew.tags import lines_iter from event import emit_event, declare_event -from .blobs import save_blob, load_blob +from .blobs import delete_blob, load_blob import mod =20 =20 @@ -634,6 +634,7 @@ class Message(models.Model): is_obsolete =3D models.BooleanField(default=3DFalse) is_tested =3D models.BooleanField(default=3DFalse) is_reviewed =3D models.BooleanField(default=3DFalse) + mbox_bytes =3D models.BinaryField(null=3DTrue) =20 # is series head if not Null topic =3D models.ForeignKey( @@ -653,19 +654,24 @@ class Message(models.Model): maintainers =3D jsonfield.JSONField(blank=3DTrue, default=3D[]) properties =3D jsonfield.JSONField(default=3D{}) =20 - def save_mbox(self, mbox_blob): - save_blob(mbox_blob, self.message_id) + def save_mbox(self, mbox): + mbox_bytes =3D mbox.encode("utf-8") + if self.mbox_bytes is None: + delete_blob(self.message_id) + self.mbox_bytes =3D mbox_bytes =20 def get_mbox_obj(self): - self.get_mbox() + if not hasattr(self, "_mbox_obj"): + self._mbox_obj =3D MboxMessage(self.mbox) return self._mbox_obj =20 def get_mbox(self): - if hasattr(self, "mbox_blob"): - return self.mbox_blob - self.mbox_blob =3D load_blob(self.message_id) - self._mbox_obj =3D MboxMessage(self.mbox_blob) - return self.mbox_blob + if not hasattr(self, "_mbox_decoded"): + if self.mbox_bytes: + self._mbox_decoded =3D str(self.mbox_bytes, "utf-8") + else: + self._mbox_decoded =3D load_blob(self.message_id) + return self._mbox_decoded =20 mbox =3D property(get_mbox) =20 diff --git a/scripts/blob_migrate.py b/scripts/blob_migrate.py new file mode 100644 index 0000000..ee7ff73 --- /dev/null +++ b/scripts/blob_migrate.py @@ -0,0 +1,25 @@ +#! /usr/bin/env python3 + +from api.models import Message, Project +from django.db import transaction + +def doit(n=3D1000): + done =3D 0 + for p in Project.objects.all(): + start =3D Message.objects.filter(project=3Dp, mbox_bytes=3DNone).o= rder_by("-date").first() + while start: + first_date =3D start.date + print(done, p, first_date) + with transaction.atomic(): + previously =3D done + q =3D Message.objects.filter(project=3Dp, date__lte=3Dfirs= t_date, mbox_bytes=3DNone).order_by("-date")[:n] + for msg in q: + try: + msg.save_mbox(msg.mbox) + msg.save() + done +=3D 1 + except Exception as e: + print(msg, type(e)) + start =3D msg + if done =3D=3D previously and start.date =3D=3D first_date: + start =3D None --=20 2.35.1 _______________________________________________ Patchew-devel mailing list Patchew-devel@redhat.com https://listman.redhat.com/mailman/listinfo/patchew-devel From nobody Fri Apr 19 15:40:09 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) client-ip=170.10.129.124; envelope-from=patchew-devel-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1649321268; cv=none; d=zohomail.com; s=zohoarc; b=egJcu+/YjUkdhXT6FWa4ehWKJfh/9lfJA2I4JGHSo0S0A8MZPGbpOcVBqFBtf2J8Cop7KudqwXHiC5qSdeMcZby+zBYFIEgU9kunZ7GnU9ANdMlyrVABAVNo0w8+f870puySJu+tbgNL+f3hitF0BNPLT6y0WZ+Hkq68+bwuqvg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1649321268; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=QqpcamrVFZbuvRvhuKyg1HODrG9ZsoEtUw8epTbmfsU=; b=m+D3c+D9bqlJ2BwBhSOooAEFnrp3DTJ0LErYt6RWaH/ku0iJ/qh2yu55zaWoUBvoSMg8LuTzw1S01cS/Mhf+JpEr215RXUmZgGsxXeHTKcjysa+5aQbodZZmG4BE2UOBSXEiSgby2CrnIeqFRBYftjBOL1+3sbob73JJoCmqNfg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mx.zohomail.com with SMTPS id 1649321268821400.8760925587934; Thu, 7 Apr 2022 01:47:48 -0700 (PDT) Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-101-oJd51b6AOyGcKIdPhyMcdw-1; Thu, 07 Apr 2022 04:47:46 -0400 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3A0BE2999B49; Thu, 7 Apr 2022 08:47:46 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 30A5440D1B98; Thu, 7 Apr 2022 08:47:46 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id DE713194034B; Thu, 7 Apr 2022 08:47:45 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 75D311949763 for ; Thu, 7 Apr 2022 08:47:44 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id 5CAD72024CC2; Thu, 7 Apr 2022 08:47:44 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast10.extmail.prod.ext.rdu2.redhat.com [10.11.55.26]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 57AFD2024CC6 for ; Thu, 7 Apr 2022 08:47:23 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5363C1C0784E for ; Thu, 7 Apr 2022 08:47:22 +0000 (UTC) Received: from mail-ej1-f54.google.com (mail-ej1-f54.google.com [209.85.218.54]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-383-LvcMSspONe6bNT3Q104PiA-2; Thu, 07 Apr 2022 04:47:10 -0400 Received: by mail-ej1-f54.google.com with SMTP id bq8so9324414ejb.10 for ; Thu, 07 Apr 2022 01:47:10 -0700 (PDT) Received: from avogadro.redhat.com ([2001:b07:6468:f312:c8dd:75d4:99ab:290a]) by smtp.gmail.com with ESMTPSA id r22-20020a17090638d600b006d584aaa9c9sm7453703ejd.133.2022.04.07.01.47.09 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Apr 2022 01:47:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649321267; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=QqpcamrVFZbuvRvhuKyg1HODrG9ZsoEtUw8epTbmfsU=; b=Q2mNasqz5Q7mUjfEF6QcspLfrB32PL7Mzi2LN1hbC7tyCPGujpEhtxh2fsYkKrw0o6rHHd KKX3Ajtd3h8xzFUnDwDNq1fWuTJapKHTAcdeLlNLXqjHUiysGBLIjOzYqQ1YsxVcBULSke DKWQXGKPOTkr5JM8YMw62e3XlQ60jc0= X-MC-Unique: oJd51b6AOyGcKIdPhyMcdw-1 X-Original-To: patchew-devel@listman.corp.redhat.com X-MC-Unique: LvcMSspONe6bNT3Q104PiA-2 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:from:to:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=mi3LZ+K8whZrpYQLxGH6/vWEwDHgw3PXqFpgEjBfHh4=; b=dzlQ8kyXCZUPNig7qU/PyYyKp329FM9SOCGWFjqdGbklzi+JT0mewNl3rDADHal+yq nrbIoaTYz8UTijUteiGcQOWGa1G/WBnhNjQAw+paJHuNjxzvat8yVnDuUjeVYYE0U/mJ iLXHlP1W4AK223hqWOqF+VbuB9cBmKmP9iO9XdomaJmpKbJagXvKBbw1A/AeR8l1ESa9 ocGKlbXo8OXctIBNMfXqER/vuadyHGp1IkM6pCKSVKDPJF4qXbC/Mv5M3YCSJkJPU15f +pOuZ44MCcJb11yEbDR/HzobxZ+2qfUQXw2lVDDbKW2+v9ql7uRTug9fyK24X/4jmJ5S ONGw== X-Gm-Message-State: AOAM530lhUfHaWhzpKO3ctRPRhkHz2G14Nf2b2TPPRGnTIWHGArFVQAv 6/ZqfriKP8ZO7/0Hs6JIkql1gqhWPsfX7w== X-Google-Smtp-Source: ABdhPJyxXqKfCqR3mdqers0C8vBA+v1GijhK0xZ1Sp1CRB3eWrfrDE8LFicVuFaU5yzUj3JmbIkk2A== X-Received: by 2002:a17:907:3e12:b0:6da:f8d9:efeb with SMTP id hp18-20020a1709073e1200b006daf8d9efebmr12263131ejc.634.1649321229917; Thu, 07 Apr 2022 01:47:09 -0700 (PDT) From: Paolo Bonzini To: patchew-devel@redhat.com Date: Thu, 7 Apr 2022 10:47:01 +0200 Message-Id: <20220407084704.78533-3-pbonzini@redhat.com> In-Reply-To: <20220407084704.78533-1-pbonzini@redhat.com> References: <20220407084704.78533-1-pbonzini@redhat.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 Subject: [Patchew-devel] [PATCH 2/5] models: change ad hoc deblobbing script to migration X-BeenThere: patchew-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Patchew development and discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: patchew-devel-bounces@redhat.com Sender: "Patchew-devel" X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=patchew-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1649321270695100001 Content-Type: text/plain; charset="utf-8" The deblobbing script was written to be invoked while the server is running. Change it to a migration for the sake of running it in the developer setup. Signed-off-by: Paolo Bonzini --- .../migrations/0062_deblob_messages.py | 25 +++++++++++++++---- 1 file changed, 20 insertions(+), 5 deletions(-) rename scripts/blob_migrate.py =3D> api/migrations/0062_deblob_messages.py= (51%) diff --git a/scripts/blob_migrate.py b/api/migrations/0062_deblob_messages.= py similarity index 51% rename from scripts/blob_migrate.py rename to api/migrations/0062_deblob_messages.py index ee7ff73..22ee12f 100644 --- a/scripts/blob_migrate.py +++ b/api/migrations/0062_deblob_messages.py @@ -1,9 +1,13 @@ #! /usr/bin/env python3 +from __future__ import unicode_literals =20 -from api.models import Message, Project -from django.db import transaction +from django.db import migrations, transaction =20 -def doit(n=3D1000): +from api import blobs + +def deblob_messages(apps, schema_editor): + Project =3D apps.get_model("api", "Project") + Message =3D apps.get_model("api", "Message") done =3D 0 for p in Project.objects.all(): start =3D Message.objects.filter(project=3Dp, mbox_bytes=3DNone).o= rder_by("-date").first() @@ -12,14 +16,25 @@ def doit(n=3D1000): print(done, p, first_date) with transaction.atomic(): previously =3D done - q =3D Message.objects.filter(project=3Dp, date__lte=3Dfirs= t_date, mbox_bytes=3DNone).order_by("-date")[:n] + q =3D Message.objects.filter(project=3Dp, date__lte=3Dfirs= t_date, mbox_bytes=3DNone).order_by("-date")[:1000] for msg in q: try: - msg.save_mbox(msg.mbox) + mbox_decoded =3D blobs.load_blob(msg.message_id) + msg.mbox_bytes =3D mbox_decoded.encode("utf-8") msg.save() + blobs.delete_blob(msg.message_id) done +=3D 1 except Exception as e: print(msg, type(e)) start =3D msg if done =3D=3D previously and start.date =3D=3D first_date: start =3D None + +class Migration(migrations.Migration): + + dependencies =3D [("api", "0061_message_mbox_bytes")] + + operations =3D [ + migrations.RunPython(deblob_messages, reverse_code=3Dmigrations.Ru= nPython.noop) + ] + --=20 2.35.1 _______________________________________________ Patchew-devel mailing list Patchew-devel@redhat.com https://listman.redhat.com/mailman/listinfo/patchew-devel From nobody Fri Apr 19 15:40:09 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) client-ip=170.10.133.124; envelope-from=patchew-devel-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1649321264; cv=none; d=zohomail.com; s=zohoarc; b=POewgKopQlzGDLz6MD4Gl4sKE3wy6aVNPA9fLCxhj5eCy7OsYPup6zQ3/7b+ST+pZZUMy2FWOpaLdP+StTf5TxkaL0OVfPrKk/6vFj4CBKSmVnDE3Xn+Ol+y1ijgEdHc7XabbrCRUzXrwuLtRhbpUtj8O4WCrmM8rtmughJchBw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1649321264; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=G0HbkJC/oZiwTRkr5VqHwfPV/AbL9hEQoHLsZ9YrSRs=; b=c3CeD07wiY0l9qYwsmOQP6l4iTiBAnqjSrrpFGPFMz1YYTSsRvc84VwhmnHbqlCiqC9iBYHFNBI+jdW53/9sxsKbKjJN9wDgblmw9Ngq3Wdf4xE97rNSDUHSx0u6dh8LtgkdfaCDL+/WdrTH9AwlYAMHUHIqvfwJcB2ffoj1Bog= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mx.zohomail.com with SMTPS id 1649321264369974.019597530732; Thu, 7 Apr 2022 01:47:44 -0700 (PDT) Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-340-xqfjDgxNN02J8H09OdshHQ-1; Thu, 07 Apr 2022 04:47:40 -0400 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 24293899EC4; Thu, 7 Apr 2022 08:47:40 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 18FBC432721; Thu, 7 Apr 2022 08:47:40 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id D1294194034B; Thu, 7 Apr 2022 08:47:39 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id CABDA194034F for ; Thu, 7 Apr 2022 08:47:38 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id 533672024CC2; Thu, 7 Apr 2022 08:47:38 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast05.extmail.prod.ext.rdu2.redhat.com [10.11.55.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4EFAF2024CC5 for ; Thu, 7 Apr 2022 08:47:23 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-2.mimecast.com [207.211.31.81]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8DE31833959 for ; Thu, 7 Apr 2022 08:47:23 +0000 (UTC) Received: from mail-ej1-f48.google.com (mail-ej1-f48.google.com [209.85.218.48]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-225-5FP8KCTOMKadvbdjzMxrLA-2; Thu, 07 Apr 2022 04:47:11 -0400 Received: by mail-ej1-f48.google.com with SMTP id n6so9280711ejc.13 for ; Thu, 07 Apr 2022 01:47:11 -0700 (PDT) Received: from avogadro.redhat.com ([2001:b07:6468:f312:c8dd:75d4:99ab:290a]) by smtp.gmail.com with ESMTPSA id r22-20020a17090638d600b006d584aaa9c9sm7453703ejd.133.2022.04.07.01.47.09 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Apr 2022 01:47:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649321263; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=G0HbkJC/oZiwTRkr5VqHwfPV/AbL9hEQoHLsZ9YrSRs=; b=FvAdXdmO8Om4GX2AZGwPF8JC5H84FvvZsG45xpjBSwI1bwebt9ht5bRiiVTdL7JDNDanWR iaxn104hs37zGtPfIB+6C1yfEGyWa8YEZ1BnMegapc2Viaqdj/PH6jFw/Lt5H/Pgadv452 U3hmtRqPd7hCQPqHnTcRY7CcEVVLPFY= X-MC-Unique: xqfjDgxNN02J8H09OdshHQ-1 X-Original-To: patchew-devel@listman.corp.redhat.com X-MC-Unique: 5FP8KCTOMKadvbdjzMxrLA-2 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:from:to:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=pZtpfkqxr1icpdW7ThUniXs3vRzDZf2nkx7eisOdTpc=; b=qYmlO2OGJ5NJJ8UyeRlrdjvm4HkDJcdjwE62Q9XJ95I9/FSUnQEea6EYlQPYH2L6xF VsXav69LVZY7tWBZkHqk1gIAxLXu44S5NdABeYyKircnmfJkFQJul/aXd8sdFc0BSo7j s6vKNVtsVSBSWg50aQ1gSST7g2bGjv6YVZHWkqV7dyDDnDH4lS7Ot6JuftXM7cnaF+OQ uLoMMMO3Epj8dxGBqSPzQEhEK+87+/QxtdlXrkloQZ+/Byvl1NlfWoQEtES4vKPrjP7N 2HdkOCVQBJAxffSqQvcEooE4OQILSbiyk55l7facHUKvbBstODYTTP0KSWL+sg4hk/gy UnnA== X-Gm-Message-State: AOAM533xa/TdtpTSwl+LNmBqZoeCg3VDuP0dgMY2F172ny0crDU+EOfE jYfzBQ7d7pRCVlMxXKnIBM8e1rejYkRx3Q== X-Google-Smtp-Source: ABdhPJxxYFDCXSq4OFCSUOD6bXFjGPsaYI2qqEJ2GhgGtLWfo7EEtnm2R7IWq2Ig/oBJzynjeMgbAQ== X-Received: by 2002:a17:906:d10c:b0:6cd:4aa2:cd62 with SMTP id b12-20020a170906d10c00b006cd4aa2cd62mr12210587ejz.229.1649321231092; Thu, 07 Apr 2022 01:47:11 -0700 (PDT) From: Paolo Bonzini To: patchew-devel@redhat.com Date: Thu, 7 Apr 2022 10:47:02 +0200 Message-Id: <20220407084704.78533-4-pbonzini@redhat.com> In-Reply-To: <20220407084704.78533-1-pbonzini@redhat.com> References: <20220407084704.78533-1-pbonzini@redhat.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 Subject: [Patchew-devel] [PATCH 3/5] models: confine blob handling to migrations X-BeenThere: patchew-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Patchew development and discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: patchew-devel-bounces@redhat.com Sender: "Patchew-devel" X-Scanned-By: MIMEDefang 2.85 on 10.11.54.9 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=patchew-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1649321264613100003 Content-Type: text/plain; charset="utf-8" Blobs are not used anymore by api.models except for migrating historical databases. Move the api.blobs code into the api.migrations package. Signed-off-by: Paolo Bonzini --- api/blobs.py | 30 ------------------------ api/migrations/0030_deblob_properties.py | 4 ++-- api/migrations/0062_deblob_messages.py | 6 ++--- api/migrations/__init__.py | 24 +++++++++++++++---- api/models.py | 16 +++---------- 5 files changed, 28 insertions(+), 52 deletions(-) delete mode 100644 api/blobs.py diff --git a/api/blobs.py b/api/blobs.py deleted file mode 100644 index 502076e..0000000 --- a/api/blobs.py +++ /dev/null @@ -1,30 +0,0 @@ -#!/usr/bin/env python3 -# -# Copyright 2016, 2018 Red Hat, Inc. -# -# Authors: -# Fam Zheng -# Paolo Bonzini -# -# This work is licensed under the MIT License. Please see the LICENSE fil= e or -# http://opensource.org/licenses/MIT. - - -import os -import uuid - -from django.conf import settings -import lzma - - -def load_blob(name): - fn =3D os.path.join(settings.DATA_DIR, "blob", name + ".xz") - return lzma.open(fn, "r").read().decode("utf-8") - - -def delete_blob(name): - fn =3D os.path.join(settings.DATA_DIR, "blob", name + ".xz") - try: - os.remove(fn) - except FileNotFoundError: - pass diff --git a/api/migrations/0030_deblob_properties.py b/api/migrations/0030= _deblob_properties.py index 5fca390..826bf68 100644 --- a/api/migrations/0030_deblob_properties.py +++ b/api/migrations/0030_deblob_properties.py @@ -3,7 +3,7 @@ from __future__ import unicode_literals =20 from django.db import migrations =20 -from api import blobs +from . import load_blob =20 =20 def deblob_properties(apps, schema_editor): @@ -12,7 +12,7 @@ def deblob_properties(apps, schema_editor): for obj in objects: obj.blob =3D False if obj.value is not None: - obj.value =3D blobs.load_blob(obj.value) + obj.value =3D load_blob(obj.value) obj.save() =20 # We can't import the models directly as they may be a newer diff --git a/api/migrations/0062_deblob_messages.py b/api/migrations/0062_d= eblob_messages.py index 22ee12f..47894fa 100644 --- a/api/migrations/0062_deblob_messages.py +++ b/api/migrations/0062_deblob_messages.py @@ -3,7 +3,7 @@ from __future__ import unicode_literals =20 from django.db import migrations, transaction =20 -from api import blobs +from . import load_blob, delete_blob =20 def deblob_messages(apps, schema_editor): Project =3D apps.get_model("api", "Project") @@ -19,10 +19,10 @@ def deblob_messages(apps, schema_editor): q =3D Message.objects.filter(project=3Dp, date__lte=3Dfirs= t_date, mbox_bytes=3DNone).order_by("-date")[:1000] for msg in q: try: - mbox_decoded =3D blobs.load_blob(msg.message_id) + mbox_decoded =3D load_blob(msg.message_id) msg.mbox_bytes =3D mbox_decoded.encode("utf-8") msg.save() - blobs.delete_blob(msg.message_id) + delete_blob(msg.message_id) done +=3D 1 except Exception as e: print(msg, type(e)) diff --git a/api/migrations/__init__.py b/api/migrations/__init__.py index 02f4c63..35b5483 100644 --- a/api/migrations/__init__.py +++ b/api/migrations/__init__.py @@ -1,22 +1,38 @@ #!/usr/bin/env python3 # -# Copyright 2018 Red Hat, Inc. +# Copyright 2016, 2018 Red Hat, Inc. # # Authors: +# Fam Zheng # Paolo Bonzini # # This work is licensed under the MIT License. Please see the LICENSE fil= e or # http://opensource.org/licenses/MIT. =20 +from django.conf import settings from django.db import migrations =20 import json -from api import blobs +import lzma +import os +import uuid =20 =20 +def load_blob(name): + fn =3D os.path.join(settings.DATA_DIR, "blob", name + ".xz") + return lzma.open(fn, "r").read().decode("utf-8") + + +def delete_blob(name): + fn =3D os.path.join(settings.DATA_DIR, "blob", name + ".xz") + try: + os.remove(fn) + except FileNotFoundError: + pass + def load_blob_json_safe(name): try: - return json.loads(blobs.load_blob(name)) + return json.loads(load_blob(name)) except Exception as e: return "Failed to load blob %s: %s" % (name, e) =20 @@ -44,7 +60,7 @@ def get_property(model, name, **kwargs): def delete_property_blob(model, name, **kwargs): mp =3D get_property_raw(model, name, **kwargs) if hasattr(mp, "blob") and mp.blob: - blobs.delete_blob(mp.value) + delete_blob(mp.value) =20 =20 def set_property(model, name, value, **kwargs): diff --git a/api/models.py b/api/models.py index ce38913..7bf8021 100644 --- a/api/models.py +++ b/api/models.py @@ -22,7 +22,6 @@ import lzma from mbox import MboxMessage, decode_payload from patchew.tags import lines_iter from event import emit_event, declare_event -from .blobs import delete_blob, load_blob import mod =20 =20 @@ -493,7 +492,7 @@ class MessageManager(models.Manager): msg.is_patch =3D m.is_patch() msg.patch_num =3D m.get_num()[0] msg.project =3D project - msg.save_mbox(mbox) + msg.mbox_bytes =3D mbox.encode("utf-8") msg.save() emit_event("MessageAdded", message=3Dmsg) self.update_series(msg) @@ -533,7 +532,7 @@ class MessageManager(models.Manager): msg.project =3D p if self.filter(message_id=3Dmsgid, project__name=3Dp.name).fir= st(): raise self.DuplicateMessageError(msgid) - msg.save_mbox(mbox) + msg.mbox_bytes =3D mbox.encode("utf-8") msg.save() emit_event("MessageAdded", message=3Dmsg) self.update_series(msg) @@ -654,12 +653,6 @@ class Message(models.Model): maintainers =3D jsonfield.JSONField(blank=3DTrue, default=3D[]) properties =3D jsonfield.JSONField(default=3D{}) =20 - def save_mbox(self, mbox): - mbox_bytes =3D mbox.encode("utf-8") - if self.mbox_bytes is None: - delete_blob(self.message_id) - self.mbox_bytes =3D mbox_bytes - def get_mbox_obj(self): if not hasattr(self, "_mbox_obj"): self._mbox_obj =3D MboxMessage(self.mbox) @@ -667,10 +660,7 @@ class Message(models.Model): =20 def get_mbox(self): if not hasattr(self, "_mbox_decoded"): - if self.mbox_bytes: - self._mbox_decoded =3D str(self.mbox_bytes, "utf-8") - else: - self._mbox_decoded =3D load_blob(self.message_id) + self._mbox_decoded =3D str(self.mbox_bytes, "utf-8") return self._mbox_decoded =20 mbox =3D property(get_mbox) --=20 2.35.1 _______________________________________________ Patchew-devel mailing list Patchew-devel@redhat.com https://listman.redhat.com/mailman/listinfo/patchew-devel From nobody Fri Apr 19 15:40:09 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) client-ip=170.10.129.124; envelope-from=patchew-devel-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1649321254; cv=none; d=zohomail.com; s=zohoarc; b=Y1BhgqW4zLAt4wA3CeC65UDQkKh7LL1bEigTl5xuUV+Tl3Dl1UsBC4aC0YKu3aKUlPv7WYtYa1xCvc25YHPJ1LhrbRDaNyVtekDqaSo+C34/fquWa5qMbqNJ0/tAxWtpKNdYlQoHS0ehI5YdGN3H+ON2i9MhQXhANvX4CqlJTxk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1649321254; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=WGaquhM3xpEops6CEeu+plfJrGCQavOtJLtR9ei/SiY=; b=iEfKakk5PDjvn2wBJ3zzJXrW95k37GcxiOxE5N0m0EgNv/AscgRgvwEws2puC8ZwZscyPxPPz/o58x8nmwKKnS2SRKnALEuFGtJbD9I0Hr415rru1Q/x8tIO2c08d1se0xmFRg1XRNpkRXcR633qoquEfw/qCKsHVL/G9jCZXR0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mx.zohomail.com with SMTPS id 1649321254080504.7842212231751; Thu, 7 Apr 2022 01:47:34 -0700 (PDT) Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-329-hVI3vtiwNVSSzVRgUCyzCg-1; Thu, 07 Apr 2022 04:47:30 -0400 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.rdu2.redhat.com [10.11.54.1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 058B43801FF5; Thu, 7 Apr 2022 08:47:29 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7714840CFD11; Thu, 7 Apr 2022 08:47:28 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id C4BB3194034B; Thu, 7 Apr 2022 08:47:26 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 54E241949763 for ; Thu, 7 Apr 2022 08:47:25 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id EFF94C28102; Thu, 7 Apr 2022 08:47:24 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast05.extmail.prod.ext.rdu2.redhat.com [10.11.55.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id EBD55C27DB3 for ; Thu, 7 Apr 2022 08:47:24 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-1.mimecast.com [205.139.110.61]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9A28E833969 for ; Thu, 7 Apr 2022 08:47:23 +0000 (UTC) Received: from mail-ej1-f48.google.com (mail-ej1-f48.google.com [209.85.218.48]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-644-74hKGEKdNgCaEVTt-Qiwww-1; Thu, 07 Apr 2022 04:47:13 -0400 Received: by mail-ej1-f48.google.com with SMTP id a6so9470761ejk.0 for ; Thu, 07 Apr 2022 01:47:13 -0700 (PDT) Received: from avogadro.redhat.com ([2001:b07:6468:f312:c8dd:75d4:99ab:290a]) by smtp.gmail.com with ESMTPSA id r22-20020a17090638d600b006d584aaa9c9sm7453703ejd.133.2022.04.07.01.47.11 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Apr 2022 01:47:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649321253; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=WGaquhM3xpEops6CEeu+plfJrGCQavOtJLtR9ei/SiY=; b=biizm1gUuoZUv2+P8DoCMW/c5y1Yl9Alze/qp7rSrr/K7iu6Vk8hrVPgN8EbtJnXTgEhXX C/94A2kGvgcfdgK1XmOf94essndQ5LXgsNziti0p0izIgTv6XEq8iUK4+ze1QRIty8RnZ9 mJvUoMh8UryuNaM0gSUNapv3vhAjOcY= X-MC-Unique: hVI3vtiwNVSSzVRgUCyzCg-1 X-Original-To: patchew-devel@listman.corp.redhat.com X-MC-Unique: 74hKGEKdNgCaEVTt-Qiwww-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:from:to:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=as1AhGprgxV1wSiza/WY07qWiTnB1lqzugpT3BVc8VU=; b=ZElAQ2YA/0+uylQJEMRXlBs7xZ2W5dL9a3i1UGRzZ89HteE8hgNGrEnzodPiZ/oE2b NQmTgWt7uSHzSAemXMLKYSe5n5QgXKVCrikDfTleBFi6yoQJ6CmmXOOVZnHhtYiGMbyC RAtUFrDk0r8R1ACddLndaCDVW4/pwrR7v+OuyYzHJVa+jIK6SdsjCpWkOWbP9xmBeVvx 7GmK45zUFazk0VGrsClS9fztvn5x3aLuLBXq3ktP+pxW3x9Hfq0aCfbDuOH1QjLRhP/a QKu9pVe0cs3Co5dut0Gw/HMvagkswXb9qGCTHt4VS5FQZVnNV9vBTR6nthJ/acBsh+OX cu5A== X-Gm-Message-State: AOAM530CS0GABGfZ+VWjRpfLeKjkQIXzYZlzTSNR8+k9K3kGqcHee7F5 OQn/aZ/C2J/cOPr2854h/BfgW+glbcu94A== X-Google-Smtp-Source: ABdhPJw5HRWbLRF6sIanoSKXwu7k0pTjftxP7KdrLpnq90Az36coJ0ApNmH1vD+Gy6UgTn3bfJe72w== X-Received: by 2002:a17:907:3e95:b0:6e8:358:6b7c with SMTP id hs21-20020a1709073e9500b006e803586b7cmr12352721ejc.236.1649321231636; Thu, 07 Apr 2022 01:47:11 -0700 (PDT) From: Paolo Bonzini To: patchew-devel@redhat.com Date: Thu, 7 Apr 2022 10:47:03 +0200 Message-Id: <20220407084704.78533-5-pbonzini@redhat.com> In-Reply-To: <20220407084704.78533-1-pbonzini@redhat.com> References: <20220407084704.78533-1-pbonzini@redhat.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.85 on 10.11.54.8 Subject: [Patchew-devel] [PATCH 4/5] models: remove messages for which the blobs were lost X-BeenThere: patchew-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Patchew development and discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: patchew-devel-bounces@redhat.com Sender: "Patchew-devel" X-Scanned-By: MIMEDefang 2.84 on 10.11.54.1 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=patchew-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1649321254607100007 Content-Type: text/plain; charset="utf-8" Messages that do not have an associated blob (typically because their message-id included a slash) have never worked. Remove them from the database. This migration can also be performed directly on the database while the server is running: update api_topic set latest_id =3D null where latest_id in (select id from = api_message where mbox_bytes is null); delete from api_messageresult where message_id in (select id from api_messa= ge where mbox_bytes is null); delete from api_message where mbox_bytes is null; delete from api_topic where not exists (select * from api_message where api= _message.topic_id =3D api_topic.id); Signed-off-by: Paolo Bonzini --- api/migrations/0063_remove_broken_messages.py | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 api/migrations/0063_remove_broken_messages.py diff --git a/api/migrations/0063_remove_broken_messages.py b/api/migrations= /0063_remove_broken_messages.py new file mode 100644 index 0000000..d160e01 --- /dev/null +++ b/api/migrations/0063_remove_broken_messages.py @@ -0,0 +1,19 @@ +#! /usr/bin/env python3 +from __future__ import unicode_literals + +from django.db import migrations, transaction + +def remove_null_mbox_messages(apps, schema_editor): + Message =3D apps.get_model("api", "Message") + Topic =3D apps.get_model("api", "Topic") + Topic.objects.filter(latest__mbox_bytes=3DNone).update(latest=3DNone) + Message.objects.filter(mbox_bytes=3DNone).delete() + +class Migration(migrations.Migration): + + dependencies =3D [("api", "0062_deblob_messages")] + + operations =3D [ + migrations.RunPython(remove_null_mbox_messages, reverse_code=3Dmig= rations.RunPython.noop) + ] + --=20 2.35.1 _______________________________________________ Patchew-devel mailing list Patchew-devel@redhat.com https://listman.redhat.com/mailman/listinfo/patchew-devel From nobody Fri Apr 19 15:40:09 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) client-ip=170.10.133.124; envelope-from=patchew-devel-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1649321250; cv=none; d=zohomail.com; s=zohoarc; b=HZtW9wxnP1RinLkd60Szl7zItWUNl0LR9Ji21kzyl4+oAhl1qqiCiM94oP9wvUNw15v92RT376qikEKDe5fhkRiBEOouIADKaYVohN0e5Jwww3ZhfFqaB5aqI1N9qihgquto7CZtluEdj9M6PG67rxi9yzOAZmk3MDnk0q+d/XU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1649321250; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=5HiYKnXv7SkoAcWhEAtBFnXgPFRsOozsIFxHnAXIYcI=; b=Y3mds9RhKe6QNvpZVoFSMIR7yrqRCd+YdRwp1RS1XtQm+qmYI7mUtc5Tfz8v3+lpvPO1UBjAlKqr6DeHEgT8dKx2XiL7+KOu8y69WNgtHR16U7kG37dSEMD+Uj4dtEXEsaJqN/0wC165q+3ItnwHwPY/15nSLf9hjYHxj2ji4Qs= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mx.zohomail.com with SMTPS id 1649321250839202.61498268107948; Thu, 7 Apr 2022 01:47:30 -0700 (PDT) Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-444-AWl6_eXzMMiohNnL7Wy2uA-1; Thu, 07 Apr 2022 04:47:26 -0400 Received: from smtp.corp.redhat.com (int-mx08.intmail.prod.int.rdu2.redhat.com [10.11.54.8]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1E6B3296A621; Thu, 7 Apr 2022 08:47:21 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1F0DCC28109; Thu, 7 Apr 2022 08:47:20 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 865CA194034C; Thu, 7 Apr 2022 08:47:19 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.rdu2.redhat.com [10.11.54.2]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 6CEB11949763 for ; Thu, 7 Apr 2022 08:47:18 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id BF18B40470E1; Thu, 7 Apr 2022 08:47:17 +0000 (UTC) Received: from mimecast-mx02.redhat.com (mimecast07.extmail.prod.ext.rdu2.redhat.com [10.11.55.23]) by smtp.corp.redhat.com (Postfix) with ESMTPS id BB58840EC00C for ; Thu, 7 Apr 2022 08:47:16 +0000 (UTC) Received: from us-smtp-1.mimecast.com (us-smtp-delivery-1.mimecast.com [207.211.31.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3230B3C1484F for ; Thu, 7 Apr 2022 08:47:15 +0000 (UTC) Received: from mail-ej1-f41.google.com (mail-ej1-f41.google.com [209.85.218.41]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-563-wQvsv2SGMgytc5Mjd6n0vA-1; Thu, 07 Apr 2022 04:47:13 -0400 Received: by mail-ej1-f41.google.com with SMTP id n6so9282580ejc.13 for ; Thu, 07 Apr 2022 01:47:13 -0700 (PDT) Received: from avogadro.redhat.com ([2001:b07:6468:f312:c8dd:75d4:99ab:290a]) by smtp.gmail.com with ESMTPSA id r22-20020a17090638d600b006d584aaa9c9sm7453703ejd.133.2022.04.07.01.47.11 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 Apr 2022 01:47:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1649321249; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=5HiYKnXv7SkoAcWhEAtBFnXgPFRsOozsIFxHnAXIYcI=; b=D9v2AVbwdzMCH2Nozvu4eQoEQv3SKxAqwkUOX7VKcbCxJV8cjwHiZmuMaKUjr70zwCDBan FWR7Zk+kjVA/UpOX3eySM+mOrJ15kNk0hAZiSdOY2766igno3pYXeej5ZtSFaxm2k7/GdE QJYOyt5QzK/CuVjgLHBpRTPO4lLQKw0= X-MC-Unique: AWl6_eXzMMiohNnL7Wy2uA-1 X-Original-To: patchew-devel@listman.corp.redhat.com X-MC-Unique: wQvsv2SGMgytc5Mjd6n0vA-1 X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:sender:from:to:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=8G7yjNru5rwU1TNdJlCqTWOS+wps3BI+Bsa70uegjVs=; b=cc4L19GMXp70IrEvKnH0hysY7rsb4P0F4/PeBqraMKNgjBupyNj2etSAlroQgOWSiG MGDVWYZnC7EGuSz5fvnTVNfFPa5K5BBNyTF2lGI8kxaomLm57sVY9FjUqk4oEiXz5stU gjQbBSLiQ77xjg7Shd7d+XAdnKKyoql05+Izqz8qTZ6O1Mx8j4syNHJV+EqYCEV3Wx5c 4vpEyXUOuBbXCeaSzNiH8mX+TQNB6lenZc706kkXOlPBjw3xWelpzSFvPOITaA/XZaNX f8wexqVePCqqS7ignbauNYTbc2vVMdpcC5/MjIufoeNSF+iaGg9WxhVnKPP9jlcrvcpe 4FIA== X-Gm-Message-State: AOAM5338Ngj4Xbai3QfffMTzqeo3yVsTPFp7he1qD2AkE/K5FvBGCsvX wlS2y9aPAz+tik8s0TW2p6XaMb4F3pGy6w== X-Google-Smtp-Source: ABdhPJz3IgQ3oHM0z8xUDCEb/HBHY09/xlKzprIKGwzZwIo+eSsx4MU5WfrLylB/2vhQQd7Dfgr9Yg== X-Received: by 2002:a17:906:6a1a:b0:6e1:87a:151f with SMTP id qw26-20020a1709066a1a00b006e1087a151fmr12023930ejc.715.1649321232179; Thu, 07 Apr 2022 01:47:12 -0700 (PDT) From: Paolo Bonzini To: patchew-devel@redhat.com Date: Thu, 7 Apr 2022 10:47:04 +0200 Message-Id: <20220407084704.78533-6-pbonzini@redhat.com> In-Reply-To: <20220407084704.78533-1-pbonzini@redhat.com> References: <20220407084704.78533-1-pbonzini@redhat.com> MIME-Version: 1.0 X-Mimecast-Impersonation-Protect: Policy=CLT - Impersonation Protection Definition; Similar Internal Domain=false; Similar Monitored External Domain=false; Custom External Domain=false; Mimecast External Domain=false; Newly Observed Domain=false; Internal User Name=false; Custom Display Name List=false; Reply-to Address Mismatch=false; Targeted Threat Dictionary=false; Mimecast Threat Dictionary=false; Custom Threat Dictionary=false X-Scanned-By: MIMEDefang 2.84 on 10.11.54.2 Subject: [Patchew-devel] [PATCH 5/5] models: make mbox_bytes non-null X-BeenThere: patchew-devel@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Patchew development and discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: patchew-devel-bounces@redhat.com Sender: "Patchew-devel" X-Scanned-By: MIMEDefang 2.85 on 10.11.54.8 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=patchew-devel-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1649321252566100001 Content-Type: text/plain; charset="utf-8" The mbox_bytes field now cannot be NULL, adjust it in the database. --- api/migrations/0064_auto_20220407_0742.py | 18 ++++++++++++++++++ api/models.py | 2 +- 2 files changed, 19 insertions(+), 1 deletion(-) create mode 100644 api/migrations/0064_auto_20220407_0742.py diff --git a/api/migrations/0064_auto_20220407_0742.py b/api/migrations/006= 4_auto_20220407_0742.py new file mode 100644 index 0000000..343da71 --- /dev/null +++ b/api/migrations/0064_auto_20220407_0742.py @@ -0,0 +1,18 @@ +# Generated by Django 3.1.14 on 2022-04-07 07:42 + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies =3D [ + ('api', '0063_remove_broken_messages'), + ] + + operations =3D [ + migrations.AlterField( + model_name=3D'message', + name=3D'mbox_bytes', + field=3Dmodels.BinaryField(), + ), + ] diff --git a/api/models.py b/api/models.py index 7bf8021..7bca8ee 100644 --- a/api/models.py +++ b/api/models.py @@ -633,7 +633,7 @@ class Message(models.Model): is_obsolete =3D models.BooleanField(default=3DFalse) is_tested =3D models.BooleanField(default=3DFalse) is_reviewed =3D models.BooleanField(default=3DFalse) - mbox_bytes =3D models.BinaryField(null=3DTrue) + mbox_bytes =3D models.BinaryField() =20 # is series head if not Null topic =3D models.ForeignKey( --=20 2.35.1 _______________________________________________ Patchew-devel mailing list Patchew-devel@redhat.com https://listman.redhat.com/mailman/listinfo/patchew-devel