From nobody Thu Mar 28 23:36:41 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) client-ip=209.132.183.28; envelope-from=patchew-devel-bounces@redhat.com; helo=mx1.redhat.com; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1556795901; cv=none; d=zoho.com; s=zohoarc; b=OCIBATPHiKHvfL4SvJm96PBNvBpEm0Kx9iBYHa+RjNN9VNxNk1NKAwAiidwQ4pUFO3Uc+z2b9T//Y8Ru+vbVGW3XcsPbVfaWTFlIexj/7y6ddE4a8M1QcqGz9gLdtDUFVCuWOvG0Wn69itRS8h9hbiDX2Uk2nJVyDynjMTJmOOI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1556795901; h=Content-Type:Content-Transfer-Encoding:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To:ARC-Authentication-Results; bh=jMXM9vidNQEbpb/KUXb8JRAqd7PNI+tiAy1XMFmtkqs=; b=j9XNPRucBZOVKvPkSSQjrsOn2J8lrQjlEEBdyTR9Q9CpAyOSrpNQOHM8/UyC++r+ZRf0pC+b6OqDb9U8cyD1rdigBpLDfTo9558nVF0C23TnAg+MNJgfWmmh9Cus/lfspfn3ub2/gqhi7Aii2lprHmlKa2xw2Ly7PIe5CSYEwp0= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=fail; spf=pass (zoho.com: domain of redhat.com designates 209.132.183.28 as permitted sender) smtp.mailfrom=patchew-devel-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) header.from= Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mx.zohomail.com with SMTPS id 1556795901554218.89762170114807; Thu, 2 May 2019 04:18:21 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5224C3087930; Thu, 2 May 2019 11:18:20 +0000 (UTC) Received: from colo-mx.corp.redhat.com (colo-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.21]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 464DB17D56; Thu, 2 May 2019 11:18:20 +0000 (UTC) Received: from lists01.pubmisc.prod.ext.phx2.redhat.com (lists01.pubmisc.prod.ext.phx2.redhat.com [10.5.19.33]) by colo-mx.corp.redhat.com (Postfix) with ESMTP id 3CA373FB11; Thu, 2 May 2019 11:18:20 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) by lists01.pubmisc.prod.ext.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id x42BIJJf009030 for ; Thu, 2 May 2019 07:18:19 -0400 Received: by smtp.corp.redhat.com (Postfix) id C52157EE9D; Thu, 2 May 2019 11:18:19 +0000 (UTC) Received: from mx1.redhat.com (ext-mx04.extmail.prod.ext.phx2.redhat.com [10.5.110.28]) by smtp.corp.redhat.com (Postfix) with ESMTPS id BDD627EE8B for ; Thu, 2 May 2019 11:18:19 +0000 (UTC) Received: from mail-oi1-f174.google.com (mail-oi1-f174.google.com [209.85.167.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 9FF9F821CC for ; Thu, 2 May 2019 11:18:18 +0000 (UTC) Received: by mail-oi1-f174.google.com with SMTP id w130so940550oie.6 for ; Thu, 02 May 2019 04:18:18 -0700 (PDT) Received: from localhost.localdomain ([198.59.53.9]) by smtp.gmail.com with ESMTPSA id o1sm19095697otj.11.2019.05.02.04.18.16 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Thu, 02 May 2019 04:18:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:to:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=SM95v7fMcOve5m3YS9e6WYnudru36cE8f2PoaJ0Ffkg=; b=aZx1Lkni2VWRDbZL7Uzc2VFSVqMb5yHdBa5ZpGjPo0DHFKq97A1oUWaViGIHyT/tdU EqrIrBFmzMPUWCndZCmLmbv/9MYVt2xiqzzmedee5WINVO4Sp44hZ7j8MbdoOcxTfavW F4cQSXbrnzVIOfpqpmtm1FMr7ozcGxv1zxq+t1EY69qoU8x+OJ13iIiSe63VFRM947DO l2O3BK0+Cry9H2wYL/DLWkzPzdjho4v1LFPhNFCzM/zh2AjMn5VubyLOzhBnT1w/iyV1 BdZXmQ0BcmR3So+PJ/u75VHERmn/aKZxIJWMdxNDJIjZ6WzfNJZP3f/LyF1qV/EQ0n+p 6Isw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:to:subject:date:message-id :in-reply-to:references:mime-version:content-transfer-encoding; bh=SM95v7fMcOve5m3YS9e6WYnudru36cE8f2PoaJ0Ffkg=; b=pO1sThdEeit+jicc+mFS7YFYZFDTLU8jO1hwJsnc1ooGtdtDrmEcaOQwM51+RsDXRw pCS/rtiRtdkuwcUbUKvu8dgZQ6DL+pnCj/vuoqm1zpyzXJ/dM75VNPjkefQI0rJtPcSc iraHMPhv9M0JeX8C2cGQmQWpZFtXTgZvaFv0A71GtyG0kcoDd3ide0v8nOcMWT6wuCI6 xeqohZdfzJCbZegnR7gO/VxUrr5Wy/JArx1Hn77qfGWRa7egMcC4C9DNBxh+QAnm4CGQ sBrlSAO+/m5aQ1QW+Ho0VD6Mgd+wMU6YyZTQPgQ7Xz9DhPuA4Dvlb65aV/fYRn5B+xIl RBZg== X-Gm-Message-State: APjAAAV3VgYj/NdbVqSSpcJ97uG4UHWPry5Nxw6DeRoQLdU0PleK5DK4 9afBDKvxtF+1e3mI+GwrxU0LGdyFRNU= X-Google-Smtp-Source: APXvYqxlXr7tj542bZQTQYaO+Q2LRlsE8NSMkCT6XbwKFREOwaI4N9+uGeFmUK4gFwgJ6A7jrS/JTw== X-Received: by 2002:aca:d6c7:: with SMTP id n190mr1935669oig.118.1556795897603; Thu, 02 May 2019 04:18:17 -0700 (PDT) From: Paolo Bonzini To: patchew-devel@redhat.com Date: Thu, 2 May 2019 05:18:01 -0600 Message-Id: <20190502111804.15843-15-pbonzini@redhat.com> In-Reply-To: <20190502111804.15843-1-pbonzini@redhat.com> References: <20190502111804.15843-1-pbonzini@redhat.com> MIME-Version: 1.0 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Thu, 02 May 2019 11:18:18 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.28]); Thu, 02 May 2019 11:18:18 +0000 (UTC) for IP:'209.85.167.174' DOMAIN:'mail-oi1-f174.google.com' HELO:'mail-oi1-f174.google.com' FROM:'paolo.bonzini@gmail.com' RCPT:'' X-RedHat-Spam-Score: 0.126 (DKIM_SIGNED, DKIM_VALID, FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS, RCVD_IN_DNSWL_NONE, SPF_PASS) 209.85.167.174 mail-oi1-f174.google.com 209.85.167.174 mail-oi1-f174.google.com X-RedHat-Possible-Forgery: Paolo Bonzini X-Scanned-By: MIMEDefang 2.78 on 10.5.110.28 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 X-loop: patchew-devel@redhat.com Subject: [Patchew-devel] [PATCH 14/17] models: introduce flags into Messages X-BeenThere: patchew-devel@redhat.com X-Mailman-Version: 2.1.12 Precedence: junk List-Id: Patchew development and discussion list List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Sender: patchew-devel-bounces@redhat.com Errors-To: patchew-devel-bounces@redhat.com X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.45]); Thu, 02 May 2019 11:18:20 +0000 (UTC) X-ZohoMail-DKIM: fail (Header signature does not verify) Content-Type: text/plain; charset="utf-8" Flags are used as a replacement for Boolean properties that are used by search queries. They can be searched efficiently using a trigram index. --- api/migrations/0048_message_flags.py | 20 +++++++++ api/migrations/0049_populate_message_flags.py | 42 +++++++++++++++++++ .../0050_message_flags_postgres_fts.py | 22 ++++++++++ api/models.py | 13 ++++++ api/search.py | 13 +++--- mods/tags.py | 5 ++- mods/testing.py | 22 +++++----- tests/test_testing.py | 9 ++-- 8 files changed, 124 insertions(+), 22 deletions(-) create mode 100644 api/migrations/0048_message_flags.py create mode 100644 api/migrations/0049_populate_message_flags.py create mode 100644 api/migrations/0050_message_flags_postgres_fts.py diff --git a/api/migrations/0048_message_flags.py b/api/migrations/0048_mes= sage_flags.py new file mode 100644 index 0000000..00d05c4 --- /dev/null +++ b/api/migrations/0048_message_flags.py @@ -0,0 +1,20 @@ +# -*- coding: utf-8 -*- +# Generated by Django 1.11.20 on 2019-04-18 14:30 +from __future__ import unicode_literals + +from django.db import migrations, models + + +class Migration(migrations.Migration): + + dependencies =3D [ + ('api', '0047_populate_project_config'), + ] + + operations =3D [ + migrations.AddField( + model_name=3D'message', + name=3D'flags', + field=3Dmodels.CharField(blank=3DTrue, default=3D'', max_lengt= h=3D64), + ), + ] diff --git a/api/migrations/0049_populate_message_flags.py b/api/migrations= /0049_populate_message_flags.py new file mode 100644 index 0000000..e8a77be --- /dev/null +++ b/api/migrations/0049_populate_message_flags.py @@ -0,0 +1,42 @@ +# -*- coding: utf-8 -*- +from __future__ import unicode_literals + +from django.conf import settings +from django.db import migrations + +from api.search import FLAG_TESTED, FLAG_REVIEWED, FLAG_OBSOLETE + +def property_to_flags(apps, schema_editor): + MessageProperty =3D apps.get_model('api', 'MessageProperty') + for p in MessageProperty.objects.filter(name=3D'obsoleted-by'): + p.message.flags +=3D FLAG_OBSOLETE + p.message.save() + for p in MessageProperty.objects.filter(name=3D'reviewed'): + p.message.flags +=3D FLAG_REVIEWED + p.message.save() + for p in MessageProperty.objects.filter(name=3D'testing.done'): + p.message.flags +=3D FLAG_TESTED + p.message.save() + MessageProperty.objects.filter(name=3D'reviewed').delete() + MessageProperty.objects.filter(name=3D'testing.done').delete() + +def flags_to_property(apps, schema_editor): + Message =3D apps.get_model('api', 'Message') + for m in Message.objects.exclude(flags=3D''): + if '[reviewed]' in m.flags: + new_prop =3D MessageProperty(message=3Dp.message, name=3D'revi= ewed', value=3DTrue) + new_prop.save() + if FLAG_TESTED in m.flags: + new_prop =3D MessageProperty(message=3Dp.message, name=3D'test= ing.done', value=3DTrue) + new_prop.save() + +class Migration(migrations.Migration): + + dependencies =3D [ + ('api', '0048_message_flags'), + ] + + operations =3D [ + migrations.RunPython(property_to_flags, + reverse_code=3Dflags_to_property), + ] diff --git a/api/migrations/0050_message_flags_postgres_fts.py b/api/migrat= ions/0050_message_flags_postgres_fts.py new file mode 100644 index 0000000..e6137d0 --- /dev/null +++ b/api/migrations/0050_message_flags_postgres_fts.py @@ -0,0 +1,22 @@ +# -*- coding: utf-8 -*- +# Generated by Django 1.11.16 on 2018-11-07 15:28 +from __future__ import unicode_literals + +from django.db import migrations, models +import django.db.models.deletion +from django.contrib.postgres.operations import TrigramExtension + +from api.migrations import PostgresOnlyMigration + + +class Migration(PostgresOnlyMigration): + + dependencies =3D [ + ('api', '0049_populate_message_flags'), + ] + + operations =3D [ + TrigramExtension(), + migrations.RunSQL("create index api_message_flags_gin on api_messa= ge using gin(flags gin_trgm_ops);", + "drop index api_message_flags_gin"), + ] diff --git a/api/models.py b/api/models.py index 68e90e1..67d1ed5 100644 --- a/api/models.py +++ b/api/models.py @@ -519,6 +519,7 @@ class Message(models.Model): objects =3D MessageManager() =20 maintainers =3D jsonfield.JSONField(blank=3DTrue, default=3D[]) + flags =3D models.CharField(max_length=3D64, blank=3DTrue, default=3D"") =20 def save_mbox(self, mbox_blob): save_blob(mbox_blob, self.message_id) @@ -607,6 +608,18 @@ class Message(models.Model): self.refresh_num_patches() return self.num_patches =20 + def add_flag(self, flag): + assert flag[0] =3D=3D '[' and flag[-1] =3D=3D ']' + if not flag in self.flags: + self.flags +=3D flag + self.save() + + def remove_flag(self, flag): + assert flag[0] =3D=3D '[' and flag[-1] =3D=3D ']' + if flag in self.flags: + self.flags =3D self.flags.replace(flag, '') + self.save() + def get_property(self, prop, default=3DNone): return self.get_properties().get(prop, default) =20 diff --git a/api/search.py b/api/search.py index baf6893..1c5aaaf 100644 --- a/api/search.py +++ b/api/search.py @@ -18,6 +18,10 @@ from django.contrib.postgres.search import SearchQuery, = SearchVector, SearchVect from django.db.models import Lookup from django.db.models.fields import Field =20 +# These are used by migrations. Do not change them! +FLAG_REVIEWED =3D '[reviewed]' +FLAG_OBSOLETE =3D '[obsolete]' +FLAG_TESTED =3D '[tested]' =20 @Field.register_lookup class NotEqual(Lookup): @@ -254,16 +258,13 @@ Search text keyword in the email message. Example: self._add_to_keywords('PULL') return Q(subject__contains=3D'[PULL') | Q(subject__contains=3D= '[GIT PULL') elif cond =3D=3D "reviewed": - return self._make_filter_subquery(MessageProperty, Q(name=3D"r= eviewed", value=3DTrue)) + return Q(flags__contains=3DFLAG_REVIEWED) elif cond in ("obsoleted", "old"): - return self._make_filter_subquery( - MessageProperty, - Q(name=3D"obsoleted-by", value__isnull=3DFalse) & ~Q(name= =3D"obsoleted-by", value__iexact=3D'') - ) + return Q(flags__contains=3DFLAG_OBSOLETE) elif cond =3D=3D "applied": return self._make_filter_subquery(MessageResult, Q(name=3D"git= ", status=3DResult.SUCCESS)) elif cond =3D=3D "tested": - return self._make_filter_subquery(MessageProperty, Q(name=3D"t= esting.done", value=3DTrue)) + return Q(flags__contains=3DFLAG_TESTED) elif cond =3D=3D "merged": return Q(is_merged=3DTrue) return None diff --git a/mods/tags.py b/mods/tags.py index 02c493e..6b5bcc9 100644 --- a/mods/tags.py +++ b/mods/tags.py @@ -13,6 +13,7 @@ from mbox import addr_db_to_rest, parse_address from event import register_handler, emit_event, declare_event from api.models import Message from api.rest import PluginMethodField +from api.search import FLAG_OBSOLETE, FLAG_REVIEWED import rest_framework =20 REV_BY_PREFIX =3D "Reviewed-by:" @@ -73,8 +74,10 @@ series cover letter, patch mail body and their replies. return m1.version > m2.version and m1.date >=3D m2.date for m in series.get_alternative_revisions(): if newer_than(m, series): + series.add_flag(FLAG_OBSOLETE) series.set_property("obsoleted-by", m.message_id) elif newer_than(series, m): + m.add_flag(FLAG_OBSOLETE) m.set_property("obsoleted-by", series.message_id) =20 updated =3D self.update_tags(series) @@ -100,7 +103,7 @@ series cover letter, patch mail body and their replies. series_reviewers =3D _find_reviewers(series) reviewers =3D reviewers.union(series_reviewers) if num_reviewed =3D=3D series.get_num()[1] or series_reviewers: - series.set_property("reviewed", True) + series.add_flag(FLAG_REVIEWED) series.set_property("reviewers", list(reviewers)) if updated: emit_event("TagsUpdate", series=3Dseries) diff --git a/mods/testing.py b/mods/testing.py index 2c45001..969ce84 100644 --- a/mods/testing.py +++ b/mods/testing.py @@ -22,7 +22,7 @@ from api.views import APILoginRequiredView from api.models import (Message, MessageProperty, MessageResult, Project, ProjectResult, Result) from api.rest import PluginMethodField, reverse_detail -from api.search import SearchEngine +from api.search import SearchEngine, FLAG_TESTED from event import emit_event, declare_event, register_handler from patchew.logviewer import LogView from schema import * @@ -138,9 +138,9 @@ class TestingModule(PatchewModule): _instance.tester_check_in(po, result.data['tester']) if not self.get_testing_results(obj, status__in=3D(Result.PENDING, Result.RUNNING)).= exists(): - obj.set_property("testing.done", True) obj.set_property("testing.tested-head", result.data["head"= ]) if isinstance(obj, Message): + obj.add_flag(FLAG_TESTED) obj.set_property("testing.tested-base", self.get_msg_base_tags(obj)) if isinstance(obj, Project): @@ -188,10 +188,11 @@ class TestingModule(PatchewModule): for tn in all_tests: if not tn in done_tests: obj.create_result(name=3D'testing.' + tn, status=3DRes= ult.PENDING).save() - if len(done_tests) < len(all_tests): - obj.set_property("testing.done", None) - return - obj.set_property("testing.done", True) + if isinstance(obj, Message): + if len(all_tests) and len(done_tests) =3D=3D len(all_tests): + obj.add_flag(FLAG_TESTED) + else: + obj.remove_flag(FLAG_TESTED) =20 def project_recalc_pending_tests(self, project): self.recalc_pending_tests(project) @@ -204,10 +205,9 @@ class TestingModule(PatchewModule): self.recalc_pending_tests(obj) =20 def clear_and_start_testing(self, obj, test=3D""): - for k in list(obj.get_properties().keys()): - if k =3D=3D "testing.done" or \ - k =3D=3D "testing.tested-head": - obj.set_property(k, None) + obj.set_property("testing.tested-head", None) + if isinstance(obj, Message): + obj.remove_flag(FLAG_TESTED) if test: r =3D self.get_testing_result(obj, test) if r: @@ -342,7 +342,7 @@ class TestingModule(PatchewModule): "type": "danger", "char": "T", }) - elif message.get_property("testing.done"): + elif FLAG_TESTED in message.flags: message.status_tags.append({ "title": "Testing passed", "url": reverse("series_detail", diff --git a/tests/test_testing.py b/tests/test_testing.py index 2a01b7a..efdda71 100755 --- a/tests/test_testing.py +++ b/tests/test_testing.py @@ -12,6 +12,7 @@ import abc import subprocess =20 from api.models import Message, Result +from api.search import FLAG_TESTED =20 from .patchewtest import PatchewTestCase, main =20 @@ -69,7 +70,6 @@ class TestingTestCase(PatchewTestCase, metaclass=3Dabc.AB= CMeta): if 'status' not in kwargs: kwargs['status'] =3D Result.SUCCESS self.modify_test_result(obj, **kwargs) - obj.set_property("testing.done", True) =20 def do_testing_report(self, **report): self.api_login() @@ -180,6 +180,7 @@ class MessageTestingTest(TestingTestCase): =20 def do_testing_done(self, **kwargs): self._do_testing_done(self.msg, **kwargs) + self.msg.add_flag(FLAG_TESTED) =20 def do_testing_report(self, **report): r =3D super(MessageTestingTest, self).do_testing_report(**report) @@ -364,7 +365,7 @@ class TestingResetTest(PatchewTestCase): "testing.a": Result.SUCCESS, "testing.b": Result.SUCCESS, "testing.c": Result.FAILURE}) - self.assertTrue(msg.get_property("testing.done")) + self.assertIn(FLAG_TESTED, msg.flags) =20 self.api_login() self.client.post('/login/', {'username': self.user, 'password': se= lf.password}) @@ -375,7 +376,7 @@ class TestingResetTest(PatchewTestCase): "testing.a": Result.PENDING, "testing.b": Result.SUCCESS, "testing.c": Result.FAILURE}) - self.assertFalse(msg.get_property("testing.done")) + self.assertNotIn(FLAG_TESTED, msg.flags) =20 self.client.get('/testing-reset/%s/?type=3Dmessage&test=3Db' % msg= .message_id) self.client.get('/testing-reset/%s/?type=3Dmessage&test=3Dc' % msg= .message_id) @@ -383,7 +384,7 @@ class TestingResetTest(PatchewTestCase): "testing.a": Result.PENDING, "testing.b": Result.PENDING, "testing.c": Result.PENDING}) - self.assertFalse(msg.get_property("testing.done")) + self.assertNotIn(FLAG_TESTED, msg.flags) =20 =20 class TestingDisableTest(PatchewTestCase): --=20 2.21.0 _______________________________________________ Patchew-devel mailing list Patchew-devel@redhat.com https://www.redhat.com/mailman/listinfo/patchew-devel