From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596195527; cv=none; d=zohomail.com; s=zohoarc; b=JqOkJAToPWrhaAAElZc06gcFPStMippbJ9xLZbAhuDNAvbZq5kOGU5Pa62JwPROiNR5tw/g/N9iuo28B0EtTCdlveTPIsswDTlRTfGiHExAQmk3pePi/D32CPIDYfL4nEA/hoLaPPsGBkiy8qjjA5ONvQwXH5yPE41AJX2GCqIE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596195527; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=uMJYXoeyXzeCykAMfXmxAWU0AYdUih/vpypy1Ymmmi8=; b=jTyqshq7Jt2KXzudJc/b9QwgW9ZZULo0L7ZcSY4R/K76pDZXp0kIyuvdEhxRsdaLmr5bxs8aIQdly58ooWNwzCQw10lsKrmRPwZ1Xyw5t/16iKT5b2utho1ukJbuKfKefalXrkM2zqbkjU6N0nEg2aJ8WjCWhQc2AoO/kAEu+nQ= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596195527399535.7589017688828; Fri, 31 Jul 2020 04:38:47 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TMu-0005lT-Qe; Fri, 31 Jul 2020 11:38:32 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TMt-0005kZ-08 for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 11:38:31 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 59721e70-d322-11ea-8e26-bc764e2007e4; Fri, 31 Jul 2020 11:38:27 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMo-0001W4-Im; Fri, 31 Jul 2020 12:38:26 +0100 X-Inumbo-ID: 59721e70-d322-11ea-8e26-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 01/41] Add cperl-indent-level to .dir-locals.el Date: Fri, 31 Jul 2020 12:37:40 +0100 Message-Id: <20200731113820.5765-2-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" My personal config on my laptop has this set to 2 and that makes editing osstest, which uses 4, quite annoying. Signed-off-by: Ian Jackson --- New in v2. --- .dir-locals.el | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.dir-locals.el b/.dir-locals.el index d87916f7..ad4fa3dc 100644 --- a/.dir-locals.el +++ b/.dir-locals.el @@ -1 +1,2 @@ -((nil . ((indent-tabs-mode . t)))) +((nil . ((indent-tabs-mode . t))) + (cperl-mode . ((cperl-indent-level . 4)))) --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596195537; cv=none; d=zohomail.com; s=zohoarc; b=VEJvtTcQIpFJfAYHDqjj0Z9rr6E8one5m6BtsHZaTbm/m9iCZllwOcioocJmQIlxnQRaRHPQGlQlv51NDEgvhFhzvf/Jg0+ypi/yKsqFoQ2X4KmPshxZZp+4lO/PcZMCLJtJwEVtp6+uVy1SEtzeik/xXN1/kPCukWOCuu7iGgk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596195537; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=u2Mh6z5VsH46QJxH8u/m8MGw03fRdRiMtz/Dpw4uVwY=; b=lAfXroS5kv5EOmfgHl5YMk0Pl/xAYNcBPajib/P7OFuKEsO0F8kUXdrGOS/Vrd5gQX0976XViG4VyQUbg5WglpY9n9XB5fjmFjJm37ZtmzcUEgB1s1svGEwEnobmuHpQycxDiupkYU8oyHRAjmyzG+A+8AYty+LPDdGjtcV3Jco= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596195536988650.6135309285354; Fri, 31 Jul 2020 04:38:56 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TMz-0005mh-3A; Fri, 31 Jul 2020 11:38:37 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TMy-0005kZ-0D for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 11:38:36 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 59a52144-d322-11ea-8e26-bc764e2007e4; Fri, 31 Jul 2020 11:38:27 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMp-0001W4-15; Fri, 31 Jul 2020 12:38:27 +0100 X-Inumbo-ID: 59a52144-d322-11ea-8e26-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 02/41] SQL: Use "LIKE" rather than "like", etc. Date: Fri, 31 Jul 2020 12:37:41 +0100 Message-Id: <20200731113820.5765-3-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" This is more like the rest of the style. It will also make it easier to find instances of the mistaken LIKE syntax. I found these with "git grep" and manually edited them. I have checked the before-and-after result of find * -type f | xargs perl -i~ -pe 's/\bLIKE\b/like/g' and it has only the few expected changes to ANDs and ORs. No functional change. Signed-off-by: Ian Jackson --- New in v2. --- cr-ensure-disk-space | 4 ++-- cs-adjust-flight | 2 +- mg-force-push | 2 +- mg-report-host-usage-collect | 10 +++++----- ms-planner | 2 +- sg-report-flight | 2 +- sg-report-host-history | 4 ++-- 7 files changed, 13 insertions(+), 13 deletions(-) diff --git a/cr-ensure-disk-space b/cr-ensure-disk-space index 3e0288f9..11d801b0 100755 --- a/cr-ensure-disk-space +++ b/cr-ensure-disk-space @@ -99,8 +99,8 @@ my $chkq=3D db_prepare("SELECT * FROM flights WHERE fligh= t=3D?"); my $refq=3D db_prepare(<=3D ? END =20 diff --git a/cs-adjust-flight b/cs-adjust-flight index 98d40891..d04a2fd7 100755 --- a/cs-adjust-flight +++ b/cs-adjust-flight @@ -526,7 +526,7 @@ sub change__repro_buildjobs { } } my $testq =3D db_prepare(<{job}' - AND name like 'built_revision_%' + AND name LIKE 'built_revision_%' ORDER BY name END # We report in jobtext revisions in non-main-revision jobs, too. diff --git a/sg-report-host-history b/sg-report-host-history index 54738e68..c22a1704 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -37,7 +37,7 @@ our @blessings; =20 open DEBUG, ">/dev/null"; =20 -my $namecond=3D "(name =3D 'host' or name like '%_host')"; +my $namecond=3D "(name =3D 'host' OR name LIKE '%_host')"; csreadconfig(); =20 while (@ARGV && $ARGV[0] =3D~ m/^-/) { @@ -456,7 +456,7 @@ foreach my $host (@ARGV) { SELECT DISTINCT val FROM runvars WHERE flight=3D? - AND (name =3D 'host' or name like '%_host') + AND (name =3D 'host' OR name LIKE '%_host') END $hostsinflightq->execute($flight); while (my $row =3D $hostsinflightq->fetchrow_hashref()) { --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596195537; cv=none; d=zohomail.com; s=zohoarc; b=avuzO5ByVlT1/qhrH4yy7NJps0qj+8nUsE1NCGV54mTI1k3YjvBO4Uh0rPjsH5p6GrWyso0bB+AtG4W3ZoACsmiYJj1RGE8lb57SblydZHSIg3pV85nxwEyqWneCUEgiCW2XCiD/VeSamE37FASY5a0VKDo+mo41ngFX8loGHKU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596195537; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=dXwlQ2GZqH5neYidxJ2NGQ3JgLliKejnsXMRVpNAvII=; b=NgvtCrKR5pXuX3azYrnW8ScL9aBvoUAJaaL9c2thNUYYkYgSMLlclCBwl4gSiLkFlTHM7HqxhliAWe99xk9KngjGrOuWaNAP1HciXyJHk+idEwItJzfSLlxYNSqIOPLtDRCHCu/J3D4Z/gCzfCi/jpLHGrhH75fL3Zt1uNGJW5g= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596195537210961.513595354984; Fri, 31 Jul 2020 04:38:57 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TN4-0005oL-At; Fri, 31 Jul 2020 11:38:42 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TN3-0005kZ-0H for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 11:38:41 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 59d8fa33-d322-11ea-8e26-bc764e2007e4; Fri, 31 Jul 2020 11:38:28 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMp-0001W4-Cw; Fri, 31 Jul 2020 12:38:27 +0100 X-Inumbo-ID: 59d8fa33-d322-11ea-8e26-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 03/41] SQL: Fix incorrect LIKE pattern syntax (literals) Date: Fri, 31 Jul 2020 12:37:42 +0100 Message-Id: <20200731113820.5765-4-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" LIKE takes a weird SQLish glob pattern, where % is like a glob * and (relevantly, here) _ is like a glob ?. Every _ in one of these LIKE patterns needs to be escaped with \. Do that for all the literal LIKE patterns. This fixes bugs. Generally, bugs where the wrong rows might be returned (except that the data probably doesn't have any such rows). Signed-off-by: Ian Jackson --- New in v2. --- mg-force-push | 2 +- mg-report-host-usage-collect | 2 +- sg-report-flight | 2 +- sg-report-host-history | 6 +++--- ts-logs-capture | 2 +- 5 files changed, 7 insertions(+), 7 deletions(-) diff --git a/mg-force-push b/mg-force-push index 001e0c47..3a701a11 100755 --- a/mg-force-push +++ b/mg-force-push @@ -54,7 +54,7 @@ END FROM rv url JOIN rv built ON url.job =3D built.job - AND url.name LIKE 'tree_%' + AND url.name LIKE 'tree\_%' AND built.name =3D 'built_revision_' || substring(url.name, 6) WHERE url.val =3D ? END diff --git a/mg-report-host-usage-collect b/mg-report-host-usage-collect index 3fab490a..1944c8d7 100755 --- a/mg-report-host-usage-collect +++ b/mg-report-host-usage-collect @@ -166,7 +166,7 @@ END SELECT val, synth FROM runvars WHERE flight=3D? AND job=3D? - AND (name LIKE '%_host' OR name=3D'host') + AND (name LIKE '%\_host' OR name=3D'host') END =20 my $finishq =3D db_prepare(<{job}' - AND name LIKE 'built_revision_%' + AND name LIKE 'built\_revision\_%' ORDER BY name END # We report in jobtext revisions in non-main-revision jobs, too. diff --git a/sg-report-host-history b/sg-report-host-history index c22a1704..7505b18b 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -37,7 +37,7 @@ our @blessings; =20 open DEBUG, ">/dev/null"; =20 -my $namecond=3D "(name =3D 'host' OR name LIKE '%_host')"; +my $namecond=3D "(name =3D 'host' OR name LIKE '%\_host')"; csreadconfig(); =20 while (@ARGV && $ARGV[0] =3D~ m/^-/) { @@ -256,7 +256,7 @@ END FROM runvars WHERE flight=3D? AND job=3D? AND ( - name LIKE (? || '_power_%') + name LIKE (? || '\_power\_%') ) END =20 @@ -456,7 +456,7 @@ foreach my $host (@ARGV) { SELECT DISTINCT val FROM runvars WHERE flight=3D? - AND (name =3D 'host' OR name LIKE '%_host') + AND (name =3D 'host' OR name LIKE '%\_host') END $hostsinflightq->execute($flight); while (my $row =3D $hostsinflightq->fetchrow_hashref()) { diff --git a/ts-logs-capture b/ts-logs-capture index d75a2fda..62c281b8 100755 --- a/ts-logs-capture +++ b/ts-logs-capture @@ -44,7 +44,7 @@ our (@allguests, @guests); sub find_guests () { my $sth=3D $dbh_tests->prepare(<execute($flight, $job); --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596195545; cv=none; d=zohomail.com; s=zohoarc; b=Y9dGmiYIBlp6oMiLKucrC0Alk7QKAOSrIq8/2gjf2pk+6n/RJLz043qI8A3Xaw2Qdnq/fefdIUZo6NTOWBVKRr7cpbDQUyBgvoRvLlY/hPqH0TJ7Nh2guo9er/DS7V48eVxQ/SR4sOzd3esrbi5mx+OggceUKx0FdG2DI85s560= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596195545; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=9ekyn9oSNS6zA9z5AHKbi919FsAnFJPGtvyuHVb/etY=; b=bcnwq7eSgGa/pVpGGYo6PpwPegUdOG5YhYqqsWo7qlJqiT6J3SYNBcLV6+ye3qcSq5PxbOgaXJT1Zz3yR85tctfvhTpGezCL6bxBD5M7QfGYkdwWpipbDK9xDYY7+k+iYyr5f3bQeGkcMfuVna5JzGPuD2K8pmxNIM6PatyDLTs= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 159619554550660.52974763317616; Fri, 31 Jul 2020 04:39:05 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TN9-0005q6-LI; Fri, 31 Jul 2020 11:38:47 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TN8-0005kZ-0K for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 11:38:46 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 5a418c50-d322-11ea-8e26-bc764e2007e4; Fri, 31 Jul 2020 11:38:28 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMp-0001W4-T6; Fri, 31 Jul 2020 12:38:28 +0100 X-Inumbo-ID: 5a418c50-d322-11ea-8e26-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 04/41] SQL: Fix incorrect LIKE pattern syntax (program variables) Date: Fri, 31 Jul 2020 12:37:43 +0100 Message-Id: <20200731113820.5765-5-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" In two places the pattern for LIKE is constructed programmatically. In this case, too, we need to escape % and _. We pass the actual pattern (or pattern fragment) via ?, so we do not need to worry about '. Signed-off-by: Ian Jackson --- New in v2. --- Osstest.pm | 8 +++++++- Osstest/JobDB/Executive.pm | 2 +- sg-report-host-history | 3 ++- 3 files changed, 10 insertions(+), 3 deletions(-) diff --git a/Osstest.pm b/Osstest.pm index 63dddd95..b2b6b741 100644 --- a/Osstest.pm +++ b/Osstest.pm @@ -39,7 +39,7 @@ BEGIN { main_revision_job_cond other_revision_job_suffix $dbh_tests db_retry db_retry_retry db_retry_abort db_readonly_report - db_begin_work db_prepare + db_begin_work db_prepare db_quote_like_pattern get_harness_rev blessing_must_not_modify_host ensuredir get_filecontents_core_quiet system_checked nonempty visible_undef show_abs_time @@ -358,6 +358,12 @@ sub postfork () { $mjobdb->jobdb_postfork(); } =20 +sub db_quote_like_pattern ($) { + local ($_) =3D @_; + s{[_%\\]}{\\$&}g; + $_; +} + #---------- script entrypoints ---------- =20 sub csreadconfig () { diff --git a/Osstest/JobDB/Executive.pm b/Osstest/JobDB/Executive.pm index be5588fc..39deb8a2 100644 --- a/Osstest/JobDB/Executive.pm +++ b/Osstest/JobDB/Executive.pm @@ -143,7 +143,7 @@ sub _check_testdbs ($) { AND live AND username LIKE (? || '@%') END - $sth->execute($c{Username}); + $sth->execute(db_quote_like_pattern($c{Username})); my $allok =3D 1; while (my $row =3D $sth->fetchrow_hashref()) { next if $row->{dbname} =3D~ m/^$re$/o; diff --git a/sg-report-host-history b/sg-report-host-history index 7505b18b..9730ae7a 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -380,7 +380,8 @@ END $runvarq_hits++; } else { $runvarq_misses++; - $jrunvarq->execute($jr->{flight}, $jr->{job}, $ident); + $jrunvarq->execute($jr->{flight}, $jr->{job}, + db_quote_like_pattern($ident)); my %runvars; while (my ($n, $v) =3D $jrunvarq->fetchrow_array()) { $runvars{$n} =3D $v; --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596195545; cv=none; d=zohomail.com; s=zohoarc; b=bkUvWOfrE69FNXFvJki/a2vaNlote8dLk3ok83WMb/dfsJHGwtpuvGyb2XkXdC4q0PnDUHp54MYsj4DtVz3EhwFNinveZ1zPcdfshgTgDwafuBGuDarjXI1Q70HwlCFlnm9uLuT7G/p3IlB/aURxVlbc9MMMPRyDAd+jdIHFr+s= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596195545; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=yUogw8n2uoWN2tFVS7WO86mkCSwPrPD0fjp3SHYzbkU=; b=HxeOKU/BW3v1tRGlMM4rUxUvOdcpamcGPu2LJf744qC6ut8U+xNg0cnT2uap2EgNddGA3cBJ9kb/msqxuo0tOJZwLzPbcKHx7ODUeTbiojT7TVDVpfIjNKkD7C7XYiWCTQ710qQSAhomgjYkDF6/9wvmBzzOUDBAAubMJhbiWy4= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596195545602306.6600910417958; Fri, 31 Jul 2020 04:39:05 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TNF-0005sq-1o; Fri, 31 Jul 2020 11:38:53 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TND-0005kZ-0M for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 11:38:51 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 59d8fa34-d322-11ea-8e26-bc764e2007e4; Fri, 31 Jul 2020 11:38:29 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMq-0001W4-Bf; Fri, 31 Jul 2020 12:38:28 +0100 X-Inumbo-ID: 59d8fa34-d322-11ea-8e26-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 05/41] sg-report-flight: Add a comment re same-flight search narrowing Date: Fri, 31 Jul 2020 12:37:44 +0100 Message-Id: <20200731113820.5765-6-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" In afe851ca1771e5da6395b596afa69e509dbbc278 sg-report-flight: When justifying, disregard out-of-flight build jobs we narrowed sg-report-flight's search algorith. An extensive justification is in the commit message. I think much of this information belongs in-tree, so c&p it (with slight edits) here. No code change. Signed-off-by: Ian Jackson --- sg-report-flight | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/sg-report-flight b/sg-report-flight index 831917a9..fc439495 100755 --- a/sg-report-flight +++ b/sg-report-flight @@ -242,9 +242,27 @@ END # jobs. We start with all jobs in $tflight, and for each job # we also process any other jobs it refers to in *buildjob runvars. # + # The real thing we want to check that the build jobs *in the + # same flight as the justifying job* used the right revisions. + # Build jobs from other flights were either (i) build jobs for + # components not being targed for testing by this branch, but + # which were necessary for the justifying job and for which we + # decided to reuse another build job (in which case we don't + # really care what versions they used, even if underlying it + # all there might be a different version of a tree we are + # actually interested in (ii) the kind of continuous update + # thing seen with freebsdbuildjob. + # + # (This is rather different to cs-bisection-step, which is + # less focused on changes in a particular set of trees.) + # + # So we limit the scope of our recursive descent into build + # jobs, to jobs in the same flight. + # # We don't actually use a recursive algorithm because that # would involve recursive use of the same sql query object; # hence the @binfos_todo queue. + my @binfos_todo; my $binfos_queue =3D sub { my ($inflight,$q,$why) =3D @_; --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596195552; cv=none; d=zohomail.com; s=zohoarc; b=YWBhev6+W1Q5ofAwjxc2PV//YpqcKn/EIdDbmKCdRr7QM6FN6gMubvX2b5mh/9Y35ZCAVGbvh/VZxqXQ1Nq5LACREUkB8UqogP6QSWP+sWR8YzWlIOp5B70mC2Thrrp3niLUhb/m4gD7P5auWN3/vUqAQuqLXfTDHf6dkxCFpk0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596195552; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=dYmKfIyf3zyfApj1Uh9geH4juXs/t1b0GiLT/RYECgU=; b=Mb91JbnGmonENlf+hVgVHVE0qGWX+NW4AsPnW5zSwolrfS6xweH51eS0p2hqVRU+5eflMjSKpBwdlvSn595L/babuGo+UantTPh2qDtOouW0krzvHqIQc8R6qqfqcqxu58glKZR6VXtpISfct9xjC27u2IDH0MGtyC1cxg5ljCk= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596195552242695.2184655003588; Fri, 31 Jul 2020 04:39:12 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TNJ-0005vS-Bl; Fri, 31 Jul 2020 11:38:57 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TNI-0005kZ-0W for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 11:38:56 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 5aa0cf12-d322-11ea-8e26-bc764e2007e4; Fri, 31 Jul 2020 11:38:29 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMq-0001W4-Ki; Fri, 31 Jul 2020 12:38:28 +0100 X-Inumbo-ID: 5aa0cf12-d322-11ea-8e26-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 06/41] sg-report-flight: Sort failures by job name as last resort Date: Fri, 31 Jul 2020 12:37:45 +0100 Message-Id: <20200731113820.5765-7-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" This removes some nondeterminism from the output. Signed-off-by: Ian Jackson --- sg-report-flight | 1 + 1 file changed, 1 insertion(+) diff --git a/sg-report-flight b/sg-report-flight index fc439495..7f2790ce 100755 --- a/sg-report-flight +++ b/sg-report-flight @@ -813,6 +813,7 @@ END # they finished in the same second, we pick the lower-numbered # step, which is the earlier one (if they are sequential at # all). + or $a->{Job} cmp $b->{Job} } @failures; =20 --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596195555; cv=none; d=zohomail.com; s=zohoarc; b=BOmM+DyGRqP27KqhaHAp9PjIcE18wAO/Zhl/vRA0yF2yIqhMJRbXYoyG8JG+EcqSpPwWsXsl28QUtCCdGSrF8+gw9ETLIWrm1pDeSdcSKrqaFjJNZS96ticw7F5efxH8VfBx7mKmybESnrR7ip5mXO886fLstG7wKm7goctYYcA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596195555; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=PumJWVVPG2XU+alftmxUw1f/KPWxOUnoHWyRdmI6kok=; b=TvrDzCkv0ASB4ATmiBhnW1asJ1z4A2TJQo/V2mHSCNRFPY5aWJfwpT0Nmt2G4ooln7Fig42ktW+Elp+x3tJn4TmL2tAyYXIB+2FhekuevtKRPYS9mkyX2eCJ2/1Y+muUeeBt12o509FF3CKj83jz2jowofEzO6YJjLN11JiVats= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 15961955555241007.5905768519067; Fri, 31 Jul 2020 04:39:15 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TNO-0005yx-Mi; Fri, 31 Jul 2020 11:39:02 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TNN-0005kZ-0h for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 11:39:01 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 5aa0cf14-d322-11ea-8e26-bc764e2007e4; Fri, 31 Jul 2020 11:38:29 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMr-0001W4-0L; Fri, 31 Jul 2020 12:38:29 +0100 X-Inumbo-ID: 5aa0cf14-d322-11ea-8e26-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 07/41] schema: Provide indices for sg-report-flight Date: Fri, 31 Jul 2020 12:37:46 +0100 Message-Id: <20200731113820.5765-8-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson , George Dunlap Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" These indexes allow very fast lookup of "relevant" flights eg when trying to justify failures. In my ad-hoc test case, these indices (along with the subsequent changes to sg-report-flight and Executive.pm, reduce the runtime of sg-report-flight from 2-3ks (unacceptably long!) to as little as 5-7s seconds - a speedup of about 500x. (Getting the database snapshot may take a while first, but deploying this code should help with that too by reducing long-running transactions. Quoted perf timings are from snapshot acquisition.) Without these new indexes there may be a performance change from the query changes. I haven't benchmarked this so I am setting the schema updates to be Preparatory/Needed (ie, "Schema first" as schema/README.updates has it), to say that the index should be created before the new code is deployed. Testing: I have tested this series by creating experimental indices "trial_..." in the actual production instance. (Transactional DDL was very helpful with this.) I have verified with \d that schema update instructions in this commit generate indexes which are equivalent to the trial indices. Deployment: AFter these schema updates are applied, the trial indices are redundant duplicates and should be deleted. CC: George Dunlap Signed-off-by: Ian Jackson Reviewed-by: George Dunlap --- v2: Use proper \ escaping for underscores in LIKE --- schema/runvars-built-index.sql | 7 +++++++ schema/runvars-revision-index.sql | 7 +++++++ schema/steps-job-index.sql | 7 +++++++ 3 files changed, 21 insertions(+) create mode 100644 schema/runvars-built-index.sql create mode 100644 schema/runvars-revision-index.sql create mode 100644 schema/steps-job-index.sql diff --git a/schema/runvars-built-index.sql b/schema/runvars-built-index.sql new file mode 100644 index 00000000..7108e0af --- /dev/null +++ b/schema/runvars-built-index.sql @@ -0,0 +1,7 @@ +-- ##OSSTEST## 007 Preparatory +-- +-- This index helps sg-report-flight find relevant flights. + +CREATE INDEX runvars_built_revision_idx + ON runvars (val) + WHERE name LIKE 'built\_revision\_%'; diff --git a/schema/runvars-revision-index.sql b/schema/runvars-revision-in= dex.sql new file mode 100644 index 00000000..8871b528 --- /dev/null +++ b/schema/runvars-revision-index.sql @@ -0,0 +1,7 @@ +-- ##OSSTEST## 008 Preparatory +-- +-- This index helps Executive::report__find_test find relevant flights. + +CREATE INDEX runvars_revision_idx + ON runvars (val) + WHERE name LIKE 'revision\_%'; diff --git a/schema/steps-job-index.sql b/schema/steps-job-index.sql new file mode 100644 index 00000000..07dc5a30 --- /dev/null +++ b/schema/steps-job-index.sql @@ -0,0 +1,7 @@ +-- ##OSSTEST## 006 Preparatory +-- +-- This index helps sg-report-flight find if a test ever passed. + +CREATE INDEX steps_job_testid_status_idx + ON steps (job, testid, status); + --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596195565; cv=none; d=zohomail.com; s=zohoarc; b=WLE1phVqc8lqau1bipoJkvvPhiS9W64ROoIm9KQcCHtKlX9m4s5JPNaPNcR7NKKJJyeHK7V77ESHXm0jvaonw4iXWK3WghES/2BYZm8Tlr9uscHfdycdI7Eu59nRlk5WRlYJG97Kyh5JwrUx6mHzbkNPztXRJugElETK4cyqInk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596195565; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=iHvhAqM4Jv0kdGA5PBSwxHYhV+D8BAJFjt13XbeG78w=; b=g8k/eIXee0mu24rh/Nsegp/nVFn2yu6uKSbY62scDijabyn+UazwMpPazsetpWu5pmevBHh2lhyUuIYiwc7WNyX6TELj/75VhfDwNLXrJQTb3kcUlMDnfZPaJ+WDuasNbluHutMPTRztol3IYfzbrTgBnM7ai1/VLxvB6uHkC3U= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596195565356548.0908354993468; Fri, 31 Jul 2020 04:39:25 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TNU-00062m-0s; Fri, 31 Jul 2020 11:39:08 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TNS-0005kZ-0g for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 11:39:06 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 5aa0cf15-d322-11ea-8e26-bc764e2007e4; Fri, 31 Jul 2020 11:38:30 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMr-0001W4-BA; Fri, 31 Jul 2020 12:38:29 +0100 X-Inumbo-ID: 5aa0cf15-d322-11ea-8e26-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 08/41] sg-report-flight: Ask the db for flights of interest Date: Fri, 31 Jul 2020 12:37:47 +0100 Message-Id: <20200731113820.5765-9-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson , George Dunlap Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" Specifically, we narrow the initial query to flights which have at least some job with the built_revision_foo we are looking for. This condition is strictly broader than that implemented inside the flight search loop, so there is no functional change. Perf: runtime of my test case now ~300s-500s. Example query before (from the Perl DBI trace): SELECT * FROM ( SELECT flight, blessing FROM flights WHERE (branch=3D'xen-unstable') AND EXISTS (SELECT 1 FROM jobs WHERE jobs.flight =3D flights.flight AND jobs.job =3D ?) AND ( (TRUE AND flight <=3D 151903) AND (blessing=3D'real') ) ORDER BY flight DESC LIMIT 1000 ) AS sub ORDER BY blessing ASC, flight DESC With these bind variables: "test-armhf-armhf-libvirt" After: SELECT * FROM ( SELECT DISTINCT flight, blessing FROM flights JOIN runvars r1 USING (flight) WHERE (branch=3D'xen-unstable') AND ( (TRUE AND flight <=3D 151903) AND (blessing=3D'real') ) AND EXISTS (SELECT 1 FROM jobs WHERE jobs.flight =3D flights.flight AND jobs.job =3D ?) AND r1.name LIKE 'built\_revision\_%' AND r1.name =3D ? AND r1.val=3D ? ORDER BY flight DESC LIMIT 1000 ) AS sub ORDER BY blessing ASC, flight DESC With these bind variables: "test-armhf-armhf-libvirt" 'built_revision_xen' '165f3afbfc3db70fcfdccad07085cde0a03c858b' Diff to the query: SELECT * FROM ( - SELECT flight, blessing FROM flights + SELECT DISTINCT flight, blessing + FROM flights + JOIN runvars r1 USING (flight) + WHERE (branch=3D'xen-unstable') + AND ( (TRUE AND flight <=3D 151903) AND (blessing=3D'real') ) AND EXISTS (SELECT 1 FROM jobs WHERE jobs.flight =3D flights.flight AND jobs.job =3D ?) - AND ( (TRUE AND flight <=3D 151903) AND (blessing=3D'real') ) + AND r1.name LIKE 'built\_revision\_%' + AND r1.name =3D ? + AND r1.val=3D ? + ORDER BY flight DESC LIMIT 1000 ) AS sub CC: George Dunlap Signed-off-by: Ian Jackson Reviewed-by: George Dunlap --- v2: Use proper \ escaping for underscores in LIKE --- schema/runvars-built-index.sql | 2 +- sg-report-flight | 64 ++++++++++++++++++++++++++++++++-- 2 files changed, 62 insertions(+), 4 deletions(-) diff --git a/schema/runvars-built-index.sql b/schema/runvars-built-index.sql index 7108e0af..128e69e9 100644 --- a/schema/runvars-built-index.sql +++ b/schema/runvars-built-index.sql @@ -1,4 +1,4 @@ --- ##OSSTEST## 007 Preparatory +-- ##OSSTEST## 007 Needed -- -- This index helps sg-report-flight find relevant flights. =20 diff --git a/sg-report-flight b/sg-report-flight index 7f2790ce..10127582 100755 --- a/sg-report-flight +++ b/sg-report-flight @@ -185,19 +185,77 @@ END if (defined $job) { push @flightsq_params, $job; $flightsq_jobcond =3D < (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596195565795866.217322798193; Fri, 31 Jul 2020 04:39:25 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TNX-00065Q-AR; Fri, 31 Jul 2020 11:39:11 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TNX-0005kZ-0j for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 11:39:11 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 5aa0cf16-d322-11ea-8e26-bc764e2007e4; Fri, 31 Jul 2020 11:38:30 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMr-0001W4-KU; Fri, 31 Jul 2020 12:38:29 +0100 X-Inumbo-ID: 5aa0cf16-d322-11ea-8e26-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 09/41] sg-report-flight: Use WITH to use best index use for $flightsq Date: Fri, 31 Jul 2020 12:37:48 +0100 Message-Id: <20200731113820.5765-10-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson , George Dunlap Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" While we're here, convert this EXISTS subquery to a JOIN. Perf: runtime of my test case now ~200-300s. Example query before (from the Perl DBI trace): SELECT * FROM ( SELECT DISTINCT flight, blessing FROM flights JOIN runvars r1 USING (flight) WHERE (branch=3D'xen-unstable') AND ( (TRUE AND flight <=3D 151903) AND (blessing=3D'real') ) AND EXISTS (SELECT 1 FROM jobs WHERE jobs.flight =3D flights.flight AND jobs.job =3D ?) AND r1.name LIKE 'built_revision_%' AND r1.name =3D ? AND r1.val=3D ? ORDER BY flight DESC LIMIT 1000 ) AS sub ORDER BY blessing ASC, flight DESC With bind variables: "test-armhf-armhf-libvirt" 'built_revision_xen' '165f3afbfc3db70fcfdccad07085cde0a03c858b' After: WITH sub AS ( SELECT DISTINCT flight, blessing FROM flights JOIN runvars r1 USING (flight) WHERE (branch=3D'xen-unstable') AND ( (TRUE AND flight <=3D 151903) AND (blessing=3D'real') ) AND r1.name LIKE 'built_revision_%' AND r1.name =3D ? AND r1.val=3D ? ORDER BY flight DESC LIMIT 1000 ) SELECT * FROM sub JOIN jobs USING (flight) WHERE (1=3D1) AND jobs.job =3D ? ORDER BY blessing ASC, flight DESC With bind variables: 'built_revision_xen' '165f3afbfc3db70fcfdccad07085cde0a03c858b' "test-armhf-armhf-libvirt" Diff to the query: - SELECT * FROM ( + WITH sub AS ( SELECT DISTINCT flight, blessing FROM flights JOIN runvars r1 USING (flight) WHERE (branch=3D'xen-unstable') AND ( (TRUE AND flight <=3D 151903) AND (blessing=3D'real') ) - AND EXISTS (SELECT 1 - FROM jobs - WHERE jobs.flight =3D flights.flight - AND jobs.job =3D ?) - AND r1.name LIKE 'built_revision_%' AND r1.name =3D ? AND r1.val=3D ? ORDER BY flight DESC LIMIT 1000 - ) AS sub + ) + SELECT * + FROM sub + JOIN jobs USING (flight) + + WHERE (1=3D1) + AND jobs.job =3D ? + ORDER BY blessing ASC, flight DESC Reviewed-by: George Dunlap Signed-off-by: Ian Jackson --- sg-report-flight | 39 ++++++++++++++++++++++++--------------- 1 file changed, 24 insertions(+), 15 deletions(-) diff --git a/sg-report-flight b/sg-report-flight index 10127582..d06be292 100755 --- a/sg-report-flight +++ b/sg-report-flight @@ -180,18 +180,6 @@ END return undef; } =20 - my @flightsq_params; - my $flightsq_jobcond=3D'(1=3D1)'; - if (defined $job) { - push @flightsq_params, $job; - $flightsq_jobcond =3D < (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596197586121865.2847382877674; Fri, 31 Jul 2020 05:13:06 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tu9-00049E-IC; Fri, 31 Jul 2020 12:12:53 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tu8-000489-8T for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:12:52 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 27b4b37a-d327-11ea-8e2c-bc764e2007e4; Fri, 31 Jul 2020 12:12:51 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMr-0001W4-UZ; Fri, 31 Jul 2020 12:38:30 +0100 X-Inumbo-ID: 27b4b37a-d327-11ea-8e2c-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 10/41] sg-report-flight: Use WITH clause to use index for $anypassq Date: Fri, 31 Jul 2020 12:37:49 +0100 Message-Id: <20200731113820.5765-11-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson , George Dunlap Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" Perf: runtime of my test case now ~11s Example query before (from the Perl DBI trace): SELECT * FROM flights JOIN steps USING (flight) WHERE (branch=3D'xen-unstable') AND job=3D? and testid=3D? and status=3D'pass' AND ( (TRUE AND flight <=3D 151903) AND (blessing=3D'real') ) LIMIT 1 After: WITH s AS ( SELECT * FROM steps WHERE job=3D? and testid=3D? and status=3D'pass' ) SELECT * FROM flights JOIN s USING (flight) WHERE (branch=3D'xen-unstable') AND ( (TRUE AND flight <=3D 151903) AND (blessing=3D'real') ) LIMIT 1 In both cases with bind vars: "test-amd64-i386-xl-pvshim" "guest-start" Diff to the query: - SELECT * FROM flights JOIN steps USING (flight) + WITH s AS + ( + SELECT * FROM steps + WHERE job=3D? and testid=3D? and status=3D'pass' + ) + SELECT * FROM flights JOIN s USING (flight) WHERE (branch=3D'xen-unstable') - AND job=3D? and testid=3D? and status=3D'pass' AND ( (TRUE AND flight <=3D 151903) AND (blessing=3D'real') ) LIMIT 1 Signed-off-by: Ian Jackson Reviewed-by: George Dunlap --- schema/steps-job-index.sql | 2 +- sg-report-flight | 14 ++++++++++++-- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/schema/steps-job-index.sql b/schema/steps-job-index.sql index 07dc5a30..2c33af72 100644 --- a/schema/steps-job-index.sql +++ b/schema/steps-job-index.sql @@ -1,4 +1,4 @@ --- ##OSSTEST## 006 Preparatory +-- ##OSSTEST## 006 Needed -- -- This index helps sg-report-flight find if a test ever passed. =20 diff --git a/sg-report-flight b/sg-report-flight index d06be292..d218b24e 100755 --- a/sg-report-flight +++ b/sg-report-flight @@ -849,10 +849,20 @@ sub justifyfailures ($;$) { =20 my @failures=3D values %{ $fi->{Failures} }; =20 + # In psql 9.6 this WITH clause makes postgresql do the steps query + # first. This is good because if this test never passed we can + # determine that really quickly using the new index, without + # having to scan the flights table. (If the test passed we will + # probably not have to look at many flights to find one, so in + # that case this is not much worse.) my $anypassq=3D < (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196944616649.722991783201; Fri, 31 Jul 2020 05:02:24 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tjk-00022R-Nm; Fri, 31 Jul 2020 12:02:08 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tjk-000224-5W for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:02:08 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id a796ee7b-d325-11ea-8e2b-bc764e2007e4; Fri, 31 Jul 2020 12:02:07 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMs-0001W4-9R; Fri, 31 Jul 2020 12:38:30 +0100 X-Inumbo-ID: a796ee7b-d325-11ea-8e2b-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 11/41] sg-report-flight: Use the job row from the intitial query Date: Fri, 31 Jul 2020 12:37:50 +0100 Message-Id: <20200731113820.5765-12-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" $jcheckq is redundant: we looked this up right at the start. This is not expected to speed things up very much, but it makes things somewhat cleaner and clearer. Signed-off-by: Ian Jackson --- sg-report-flight | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/sg-report-flight b/sg-report-flight index d218b24e..cb6b8174 100755 --- a/sg-report-flight +++ b/sg-report-flight @@ -160,10 +160,6 @@ sub findaflight ($$$$$) { return undef; } =20 - my $jcheckq=3D db_prepare(<fetchrow_array) { + while (my ($tflight, $tjstatus) =3D $flightsq->fetchrow_array) { # Recurse from the starting flight looking for relevant build # jobs. We start with all jobs in $tflight, and for each job # we also process any other jobs it refers to in *buildjob runvars. @@ -407,8 +403,7 @@ END $checkq->execute($tflight, $job, $testid); ($chkst) =3D $checkq->fetchrow_array(); if (!defined $chkst) { - $jcheckq->execute($tflight, $job); - my ($jchkst) =3D $jcheckq->fetchrow_array(); + my $jchkst =3D $tflight->{status}; $chkst =3D $jchkst if $jchkst eq 'starved'; } } --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196578; cv=none; d=zohomail.com; s=zohoarc; b=lFwoavaXS34xyyBzsCvvW/KyoPzWFsElSbAqEWtT1Pwpl6kdRC5xhlo9TWecNEfXAS5T76c7GbfR1IeakE/cgUIXRIWTZ0R5wBuZiDyb3ydkXT0ciL77rQbqRiEgAq3FQXI0fhbgpA8Rn0AXxLhnpO+Bn7tAiFPaCWnFcTi6Wq4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196578; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=qXbo1KtKsCgoydhkickDgUU2t1V1ZpRMSHXgt8jWBkE=; b=lZ7knfIhPhzjhHpQxEBPRIz9ZA63D1hu/GV5hThiahecpCBfv5s5MZLhEUqAWsr3t79eUefWH2ACLguG6/FO3+2T/5hlykrxnd4Gfk/3mkl5rpPEHe2f97rDi7eFgsEzWzocAKBkjLMlZ1D1w4CVdK7eGxntClIX1trawR+k0ds= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196578365123.26564842685605; Fri, 31 Jul 2020 04:56:18 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tdp-0008SJ-9I; Fri, 31 Jul 2020 11:56:01 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tdn-0008RD-RE for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 11:55:59 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id cc2c9506-d324-11ea-8e29-bc764e2007e4; Fri, 31 Jul 2020 11:55:59 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMs-0001W4-It; Fri, 31 Jul 2020 12:38:30 +0100 X-Inumbo-ID: cc2c9506-d324-11ea-8e29-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 12/41] Executive: Use index for report__find_test Date: Fri, 31 Jul 2020 12:37:51 +0100 Message-Id: <20200731113820.5765-13-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson , George Dunlap Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" After we refactor this query then we can enable the index use. (Both of these things together in this commit because I haven't perf tested the version with just the refactoring.) (We have provided an index that can answer this question really quickly if a version is specified. But the query planner couldn't see that because it works without seeing the bind variables, so doesn't know that the value of name is going to be suitable for this index.) * Convert the two EXISTS subqueries into JOIN/AND with a DISTINCT clause naming the fields on flights, so as to replicate the previous result rows. Then do $selection field last. The subquery is a convenient way to let this do the previous thing for all the values of $selection (including, notably, *). * Add the additional AND clause for r.name, which has no logical effect given the actual values of name, enabling the query planner to use this index. Perf: In my test case the sg-report-flight runtime is now ~8s. I am reasonably confident that this will not make other use cases of this code worse. Perf: runtime of my test case now ~11s Example query before (from the Perl DBI trace): SELECT * FROM flights f WHERE EXISTS ( SELECT 1 FROM runvars r WHERE name=3D? AND val=3D? AND r.flight=3Df.flight AND ( (CASE WHEN (r.job) LIKE 'build-%-prev' THEN 'xprev' WHEN ((r.job) LIKE 'build-%-freebsd' AND 'x' =3D 'freebsdbuildjob') THEN 'DISCARD' ELSE '' END) =3D '') ) AND ( (TRUE AND flight <=3D 151903) AND (blessing=3D'real') ) AND (branch=3D?) ORDER BY flight DESC LIMIT 1 After: SELECT * FROM ( SELECT DISTINCT flight, started, blessing, branch, intended FROM flights f JOIN runvars r USING (flight) WHERE name=3D? AND name LIKE 'revision\_%' AND val=3D? AND r.flight=3Df.flight AND ( (CASE WHEN (r.job) LIKE 'build-%-prev' THEN 'xprev' WHEN ((r.job) LIKE 'build-%-freebsd' AND 'x' =3D 'freebsdbuildjob') THEN 'DISCARD' ELSE '' END) =3D '') AND ( (TRUE AND flight <=3D 151903) AND (blessing=3D'real') ) AND (branch=3D?) ) AS sub WHERE TRUE ORDER BY flight DESC LIMIT 1 In both cases with bind vars: 'revision_xen' '165f3afbfc3db70fcfdccad07085cde0a03c858b' "xen-unstable" Diff to the example query: @@ -1,10 +1,10 @@ SELECT * + FROM ( SELECT DISTINCT + flight, started, blessing, branch, intended FROM flights f - WHERE - EXISTS ( - SELECT 1 - FROM runvars r + JOIN runvars r USING (flight) WHERE name=3D? + AND name LIKE 'revision\_%' AND val=3D? AND r.flight=3Df.flight AND ( (CASE @@ -14,8 +14,8 @@ ELSE '' END) =3D '') - ) AND ( (TRUE AND flight <=3D 151903) AND (blessing=3D'real') ) AND (branch=3D?) +) AS sub WHERE TRUE ORDER BY flight DESC LIMIT 1 Reviewed-by: George Dunlap Signed-off-by: Ian Jackson --- v2: Use proper \ escaping for underscores in LIKE --- Osstest/Executive.pm | 20 ++++++++------------ schema/runvars-revision-index.sql | 2 +- 2 files changed, 9 insertions(+), 13 deletions(-) diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm index c3dc1261..9208d8af 100644 --- a/Osstest/Executive.pm +++ b/Osstest/Executive.pm @@ -415,37 +415,32 @@ sub report__find_test ($$$$$$$) { =20 my $querytext =3D < (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196893274366.66409545956435; Fri, 31 Jul 2020 05:01:33 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tis-0001TI-4l; Fri, 31 Jul 2020 12:01:14 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tiq-0001R9-36 for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:01:12 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 8663f32e-d325-11ea-8e2b-bc764e2007e4; Fri, 31 Jul 2020 12:01:11 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMs-0001W4-Th; Fri, 31 Jul 2020 12:38:31 +0100 X-Inumbo-ID: 8663f32e-d325-11ea-8e2b-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 13/41] duration_estimator: Ignore truncated jobs unless we know the step Date: Fri, 31 Jul 2020 12:37:52 +0100 Message-Id: <20200731113820.5765-14-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" If we are looking for a particular step then we will ignore jobs without that step, so any job which was truncated before it will be ignored. Otherwise we are looking for the whole job duration and a truncated job is not a good representative. This is a bugfix (to duration estimation), not a performance improvement like the preceding and subsequent changes. Signed-off-by: Ian Jackson --- Osstest/Executive.pm | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm index 9208d8af..f528edd0 100644 --- a/Osstest/Executive.pm +++ b/Osstest/Executive.pm @@ -1142,6 +1142,10 @@ sub duration_estimator ($$;$$) { # estimated (and only jobs which contained that step will be # considered). =20 + my $or_status_truncated =3D ''; + if ($will_uptoincl_testid) { + $or_status_truncated =3D "OR j.status=3D'truncated'!"; + } my $recentflights_q=3D $dbh_tests->prepare(<=3D ? ORDER BY f.started DESC --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196893; cv=none; d=zohomail.com; s=zohoarc; b=W6OKDIe79FdmKPUaYk8q8+bP2JSUS4aUuSBQRQlJwTop6jg+YNSlRIDXIaeIUoKF26sxgRxVqnBzjHc8d/JQ1KYu3lU7kTdqonEkJOE9NbLM6o8YrwKIQh+85d0vzLud2UePGvoneifjarpPASjENc6j7YKXo22EgWK0/7hahe4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196893; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=1c9NCmVnpk0aW7md/lTfBD2Ag32/ZdjJZ6PaMFt7f/k=; b=C0OLEx353OaX6HiYx7maKW2aWKXmTCqU/H64mU0GQC8ve8xnmFNq1WEWEAByBJoPQFfjFR8b88MkuMfFe2TD1gXvlPq+hkjXmWIQn9K0yI2cM8KVrVvWF/Bq7237l11Vr1Ax6LSVHu/gFGlHj41u83/gXibiTxnwG8O43NJKtEE= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196893198981.7621434776862; Fri, 31 Jul 2020 05:01:33 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tiu-0001Uc-ED; Fri, 31 Jul 2020 12:01:16 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tit-0001R9-Mw for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:01:15 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 88960e84-d325-11ea-8e2b-bc764e2007e4; Fri, 31 Jul 2020 12:01:15 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMt-0001W4-GK; Fri, 31 Jul 2020 12:38:31 +0100 X-Inumbo-ID: 88960e84-d325-11ea-8e2b-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 14/41] duration_estimator: Introduce some _qtxt variables Date: Fri, 31 Jul 2020 12:37:53 +0100 Message-Id: <20200731113820.5765-15-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" No functional change. Signed-off-by: Ian Jackson --- Osstest/Executive.pm | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm index f528edd0..4cb22cc9 100644 --- a/Osstest/Executive.pm +++ b/Osstest/Executive.pm @@ -1146,7 +1146,7 @@ sub duration_estimator ($$;$$) { if ($will_uptoincl_testid) { $or_status_truncated =3D "OR j.status=3D'truncated'!"; } - my $recentflights_q=3D $dbh_tests->prepare(<prepare(<prepare($recentflights_qtxt); + my $duration_anyref_q=3D $dbh_tests->prepare($duration_anyref_qtxt); my $duration_duration_q =3D $dbh_tests->prepare($duration_duration_qtx= t); =20 return sub { --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196869; cv=none; d=zohomail.com; s=zohoarc; b=TiFtG8X1oZtebf5kDZKyBe8jBRo7t+5hH11vfYoGLboL+1EUs47LVMt1Q+61h+Rv43xQCUKdqniGnv+8Ve1l8Q/O182oCFQbaZI1udRLxV5Mu2FMcMyqlVV4wS6MPEzxZOkDM4mf9pD2nPn5btDN9LwY7HJ4eYpXh7OVlCY/jbI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196869; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=LM9b8CL8w2KX8uAK3bpT5+aanqjOhWQh26wlgmI+h24=; b=CM8NA1czBj5CAOM9PKYaIQKEP9tOlOAtPb/vu674qVAONJ5i0qKNZJlQ1YZcHfnEfDAoOqeyXpDfiAfek4/a7A8uOI2121ZQ1y0UdvZmbnHpRbGABE3NRuO4LddCWn4hezqSvVzcHillP28+5ZE/m5rlZbnPGNk9z+JdbNofTYA= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196869778218.41671694244815; Fri, 31 Jul 2020 05:01:09 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TiV-0001M3-99; Fri, 31 Jul 2020 12:00:51 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TiU-0001Jk-KF for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:00:50 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 78ce6d3e-d325-11ea-8e2b-bc764e2007e4; Fri, 31 Jul 2020 12:00:48 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMt-0001W4-O5; Fri, 31 Jul 2020 12:38:31 +0100 X-Inumbo-ID: 78ce6d3e-d325-11ea-8e2b-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 15/41] duration_estimator: Explicitly provide null in general host q Date: Fri, 31 Jul 2020 12:37:54 +0100 Message-Id: <20200731113820.5765-16-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" Our spec. says we return nulls for started and status if we don't find a job matching the host spec. The way this works right now is that we look up the nonexistent entries in $refs->[0]. This is not really brilliant and is going to be troublesome as we continue to refactor. Provide these values explicitly. No functional change. Signed-off-by: Ian Jackson --- Osstest/Executive.pm | 2 ++ 1 file changed, 2 insertions(+) diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm index 4cb22cc9..d45d6557 100644 --- a/Osstest/Executive.pm +++ b/Osstest/Executive.pm @@ -1169,6 +1169,8 @@ END =20 my $duration_anyref_qtxt=3D < (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196889804395.84097417918963; Fri, 31 Jul 2020 05:01:29 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tin-0001Rg-S7; Fri, 31 Jul 2020 12:01:09 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tim-0001R9-5A for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:01:08 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 83774e7d-d325-11ea-8e2b-bc764e2007e4; Fri, 31 Jul 2020 12:01:07 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMu-0001W4-4C; Fri, 31 Jul 2020 12:38:32 +0100 X-Inumbo-ID: 83774e7d-d325-11ea-8e2b-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 16/41] duration_estimator: Return job column in first query Date: Fri, 31 Jul 2020 12:37:55 +0100 Message-Id: <20200731113820.5765-17-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson , George Dunlap Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" Right now this is pointless since the Perl code doesn't need it. But this row is going to be part of a WITH clause soon. No functional change. Diffs to two example queries (from the Perl DBI trace): SELECT f.flight AS flight, + j.job AS job, f.started AS started, j.status AS status FROM flights f JOIN jobs j USING (flight) JOIN runvars r ON f.flight=3Dr.flight AND r.name=3D? WHERE j.job=3Dr.job AND f.blessing=3D? AND f.branch=3D? AND j.job=3D? AND r.val=3D? AND (j.status=3D'pass' OR j.status=3D'fail' OR j.status=3D'truncated'!) AND f.started IS NOT NULL AND f.started >=3D ? ORDER BY f.started DESC SELECT f.flight AS flight, + s.job AS job, NULL as started, NULL as status, max(s.finished) AS max_finished FROM steps s JOIN flights f ON s.flight=3Df.flight WHERE s.job=3D? AND f.blessing=3D? AND f.branch=3D? AND s.finished IS NOT NULL AND f.started IS NOT NULL AND f.started >=3D ? - GROUP BY f.flight + GROUP BY f.flight, s.job ORDER BY max_finished DESC CC: George Dunlap Signed-off-by: Ian Jackson --- Osstest/Executive.pm | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm index d45d6557..359120c0 100644 --- a/Osstest/Executive.pm +++ b/Osstest/Executive.pm @@ -1148,6 +1148,7 @@ sub duration_estimator ($$;$$) { } my $recentflights_qtxt=3D <=3D ? - GROUP BY f.flight + GROUP BY f.flight, s.job ORDER BY max_finished DESC END # s J J J # fix perl-mode --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196920; cv=none; d=zohomail.com; s=zohoarc; b=Nre5tzjh/iidsUYFvcXzLt6L25bBhXFnt510mR+0AfWWpXH3T899Z+cOxKHTvPfVM+h0pDPiyoJHBuSDmv+OLyKUkzCN8cOofBa+vBWPCkSNL/GAwORywMkEhCaJLnPT8QUjzFzgxI69h23W0a4+9Ztl3i2jv7y3z6iWyGWe5Uc= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196920; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=TWfn7V+Evs3mkX2qR4ua47XqgN+LoMIUyRaWKkjw46M=; b=mcdeii9p7E2wcofCSOiHZJv4WPrVolrPZuT+c8AsDEPygCH+ZmoWWZfCJlCgcwBS3v/Rw38iFVR9d29X9kd/slFQg0Ho+Lm6NIk1jM7J7hX3lzR/NaWPsymypIDqu9UMfe/WBtEc2SR0RJgYD81uDiYoPXGc58V7xc0ZvsYU5UY= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196920526507.15755228743683; Fri, 31 Jul 2020 05:02:00 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TjN-0001oE-DQ; Fri, 31 Jul 2020 12:01:45 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TjL-0001lW-Kc for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:01:43 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 9903baab-d325-11ea-8e2b-bc764e2007e4; Fri, 31 Jul 2020 12:01:43 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMu-0001W4-DT; Fri, 31 Jul 2020 12:38:32 +0100 X-Inumbo-ID: 9903baab-d325-11ea-8e2b-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 17/41] duration_estimator: Move $uptincl_testid to separate @x_params Date: Fri, 31 Jul 2020 12:37:56 +0100 Message-Id: <20200731113820.5765-18-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" This is going to be useful soon. No functional change. Signed-off-by: Ian Jackson --- Osstest/Executive.pm | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm index 359120c0..fb975dac 100644 --- a/Osstest/Executive.pm +++ b/Osstest/Executive.pm @@ -1223,6 +1223,9 @@ END_ALWAYS return sub { my ($job, $hostidname, $onhost, $uptoincl_testid) =3D @_; =20 + my @x_params; + push @x_params, $uptoincl_testid if $will_uptoincl_testid; + my $dbg=3D $debug ? sub { $debug->("DUR $branch $blessing $job $hostidname $onhost @_"); } : sub { }; @@ -1257,7 +1260,7 @@ END_ALWAYS my $duration_max=3D 0; foreach my $ref (@$refs) { my @d_d_args =3D ($ref->{flight}, $job); - push @d_d_args, $uptoincl_testid if $will_uptoincl_testid; + push @d_d_args, @x_params; $duration_duration_q->execute(@d_d_args); my ($duration) =3D $duration_duration_q->fetchrow_array(); $duration_duration_q->finish(); --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596197585; cv=none; d=zohomail.com; s=zohoarc; b=E22SWBC5JVRbg+pmYfKElU54bSP4qEU6QFQdEJbYk0BekT7+IBhbu8dPvvUKOQu3uyIYvOAIUzMDKC5Xj3H9KiZHRAKDEmZVWwep5iqn5Guq6NYAE09/Aw2Znky5jBmQxxFQ42Qif9fq1TphhFwPkqdbTauyZCANTQonACBOTCk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596197585; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=gz8iLvjXjYgOL1IAL0xayLO1YRaBB5aeoWBaBjpybsE=; b=YmM/JBWxNJEGWScndaO6zmlkeeGkkADosRyO5CYqWgiusaq3CxX91FLT1+Qezk+f6Krnx65c9PD2TQuFPyR7OU6HmFjLP67OhqX9jFSvoKAl9DrZVIrxlvK30ECdISpFE4nxrjDvwdmc9Y+c+GIaJt3Hkmk/pnKK+DnyDuq1BXc= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596197585485366.48930840552975; Fri, 31 Jul 2020 05:13:05 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Ttz-00047x-25; Fri, 31 Jul 2020 12:12:43 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tty-00047s-0v for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:12:42 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 2140508a-d327-11ea-8e2c-bc764e2007e4; Fri, 31 Jul 2020 12:12:40 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMv-0001W4-2L; Fri, 31 Jul 2020 12:38:33 +0100 X-Inumbo-ID: 2140508a-d327-11ea-8e2c-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 18/41] duration_estimator: Move duration query loop into database Date: Fri, 31 Jul 2020 12:37:57 +0100 Message-Id: <20200731113820.5765-19-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson , George Dunlap Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" Stuff the two queries together: we use the firsty query as a WITH clause. This is significantly faster, perhaps because the query optimiser does a better job but probably just because it saves on round trips. No functional change. Perf: subjectively this seemed to help when the cache was cold. Now I have a warm cache and it doesn't seem to make much difference. Perf: runtime of my test case now ~5-7s. Example queries before (from the debugging output): Query A part I: SELECT f.flight AS flight, j.job AS job, f.started AS started, j.status AS status FROM flights f JOIN jobs j USING (flight) JOIN runvars r ON f.flight=3Dr.flight AND r.name=3D? WHERE j.job=3Dr.job AND f.blessing=3D? AND f.branch=3D? AND j.job=3D? AND r.val=3D? AND (j.status=3D'pass' OR j.status=3D'fail' OR j.status=3D'truncated'!) AND f.started IS NOT NULL AND f.started >=3D ? ORDER BY f.started DESC With bind variables: "test-amd64-i386-xl-pvshim" "guest-start" Query B part I: SELECT f.flight AS flight, s.job AS job, NULL as started, NULL as status, max(s.finished) AS max_finished FROM steps s JOIN flights f ON s.flight=3Df.flight WHERE s.job=3D? AND f.blessing=3D? AND f.branch=3D? AND s.finished IS NOT NULL AND f.started IS NOT NULL AND f.started >=3D ? GROUP BY f.flight, s.job ORDER BY max_finished DESC With bind variables: "test-armhf-armhf-libvirt" 'real' "xen-unstable" 1594144469 Query common part II: WITH tsteps AS ( SELECT * FROM steps WHERE flight=3D? AND job=3D? ) , tsteps2 AS ( SELECT * FROM tsteps WHERE finished <=3D (SELECT finished FROM tsteps WHERE tsteps.testid =3D ?) ) SELECT ( SELECT max(finished)-min(started) FROM tsteps2 ) - ( SELECT sum(finished-started) FROM tsteps2 WHERE step =3D 'ts-hosts-allocate' ) AS duration With bind variables from previous query, eg: 152045 "test-armhf-armhf-libvirt" "guest-start.2" After: Query A (combined): WITH f AS ( SELECT f.flight AS flight, j.job AS job, f.started AS started, j.status AS status FROM flights f JOIN jobs j USING (flight) JOIN runvars r ON f.flight=3Dr.flight AND r.name=3D? WHERE j.job=3Dr.job AND f.blessing=3D? AND f.branch=3D? AND j.job=3D? AND r.val=3D? AND (j.status=3D'pass' OR j.status=3D'fail' OR j.status=3D'truncated'!) AND f.started IS NOT NULL AND f.started >=3D ? ORDER BY f.started DESC ) SELECT flight, max_finished, job, started, status, ( WITH tsteps AS ( SELECT * FROM steps WHERE flight=3Df.flight AND job=3Df.job ) , tsteps2 AS ( SELECT * FROM tsteps WHERE finished <=3D (SELECT finished FROM tsteps WHERE tsteps.testid =3D ?) ) SELECT ( SELECT max(finished)-min(started) FROM tsteps2 ) - ( SELECT sum(finished-started) FROM tsteps2 WHERE step =3D 'ts-hosts-allocate' ) AS duration ) FROM f Query B (combined): WITH f AS ( SELECT f.flight AS flight, s.job AS job, NULL as started, NULL as status, max(s.finished) AS max_finished FROM steps s JOIN flights f ON s.flight=3Df.flight WHERE s.job=3D? AND f.blessing=3D? AND f.branch=3D? AND s.finished IS NOT NULL AND f.started IS NOT NULL AND f.started >=3D ? GROUP BY f.flight, s.job ORDER BY max_finished DESC ) SELECT flight, max_finished, job, started, status, ( WITH tsteps AS ( SELECT * FROM steps WHERE flight=3Df.flight AND job=3Df.job ) , tsteps2 AS ( SELECT * FROM tsteps WHERE finished <=3D (SELECT finished FROM tsteps WHERE tsteps.testid =3D ?) ) SELECT ( SELECT max(finished)-min(started) FROM tsteps2 ) - ( SELECT sum(finished-started) FROM tsteps2 WHERE step =3D 'ts-hosts-allocate' ) AS duration ) FROM f Diff for query A: @@ -1,3 +1,4 @@ + WITH f AS ( SELECT f.flight AS flight, j.job AS job, f.started AS started, @@ -18,11 +19,14 @@ AND f.started >=3D ? ORDER BY f.started DESC + ) + SELECT flight, max_finished, job, started, status, + ( WITH tsteps AS ( SELECT * FROM steps - WHERE flight=3D? AND job=3D? + WHERE flight=3Df.flight AND job=3Df.job ) , tsteps2 AS ( @@ -42,3 +46,5 @@ WHERE step =3D 'ts-hosts-allocate' ) AS duration + + ) FROM f Diff for query B: @@ -1,3 +1,4 @@ + WITH f AS ( SELECT f.flight AS flight, s.job AS job, NULL as started, @@ -12,11 +13,14 @@ GROUP BY f.flight, s.job ORDER BY max_finished DESC + ) + SELECT flight, max_finished, job, started, status, + ( WITH tsteps AS ( SELECT * FROM steps - WHERE flight=3D? AND job=3D? + WHERE flight=3Df.flight AND job=3Df.job ) , tsteps2 AS ( @@ -36,3 +40,5 @@ WHERE step =3D 'ts-hosts-allocate' ) AS duration + + ) FROM f Reviewed-by: George Dunlap Signed-off-by: Ian Jackson --- Osstest/Executive.pm | 31 ++++++++++++++++++++----------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm index fb975dac..684cafc3 100644 --- a/Osstest/Executive.pm +++ b/Osstest/Executive.pm @@ -1192,7 +1192,7 @@ END ( SELECT * FROM steps - WHERE flight=3D? AND job=3D? + WHERE flight=3Df.flight AND job=3Df.job ) END_ALWAYS , tsteps2 AS @@ -1216,9 +1216,20 @@ END_UPTOINCL AS duration END_ALWAYS =09 - my $recentflights_q=3D $dbh_tests->prepare($recentflights_qtxt); - my $duration_anyref_q=3D $dbh_tests->prepare($duration_anyref_qtxt); - my $duration_duration_q =3D $dbh_tests->prepare($duration_duration_qtx= t); + my $prepare_combi =3D sub { + db_prepare(<($recentflights_qtxt); + my $duration_anyref_q=3D $prepare_combi->($duration_anyref_qtxt); =20 return sub { my ($job, $hostidname, $onhost, $uptoincl_testid) =3D @_; @@ -1239,14 +1250,16 @@ END_ALWAYS $branch, $job, $onhost, - $limit); + $limit, + @x_params); $refs=3D $recentflights_q->fetchall_arrayref({}); $recentflights_q->finish(); $dbg->("SAME-HOST GOT ".scalar(@$refs)); } =20 if (!@$refs) { - $duration_anyref_q->execute($job, $blessing, $branch, $limit); + $duration_anyref_q->execute($job, $blessing, $branch, $limit, + @x_params); $refs=3D $duration_anyref_q->fetchall_arrayref({}); $duration_anyref_q->finish(); $dbg->("ANY-HOST GOT ".scalar(@$refs)); @@ -1259,11 +1272,7 @@ END_ALWAYS =20 my $duration_max=3D 0; foreach my $ref (@$refs) { - my @d_d_args =3D ($ref->{flight}, $job); - push @d_d_args, @x_params; - $duration_duration_q->execute(@d_d_args); - my ($duration) =3D $duration_duration_q->fetchrow_array(); - $duration_duration_q->finish(); + my ($duration) =3D $ref->{duration}; if ($duration) { $dbg->("REF $ref->{flight} DURATION $duration ". ($ref->{status} // '')); --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196571; cv=none; d=zohomail.com; s=zohoarc; b=CRSh6cSNEuHLDTrJ/HyqDhzW8c7Ps5SUWswTmn4UGwWya46TjSYNQnWRfFbDjr/YPAeD4LopT9A/qoDS7R+n6af+a8HsPwPZT0Nr9YRry/W3DBZIuJ//JNvyvyp7I0vUXGokoehkZinTAwUc1eQLsUNJv8yt75VX5KVI04y5L/0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196571; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=kQ86HCobpUMXu+EdXz+cL+eQQEtzPE4+EwN4yHhKHtA=; b=a+tZWWAnOm8FC1N6IIZi9hPPlBkswDYss9skLURSrIq7JiMDEHDGaFlD07s2Q0vvMmyIGAkwSXKHTRykU5eorD6XnDue9B626epmuMbkK9lEaM/F4eyD7QX37AoMQjMGX1tf6gobTjdtL81R2IOIL6P3o4rMNyLCMaV2qX5v4sI= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196571009503.64989947943866; Fri, 31 Jul 2020 04:56:11 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tda-0008Po-F6; Fri, 31 Jul 2020 11:55:46 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TdY-0008Pj-VK for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 11:55:44 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id c25a5cad-d324-11ea-8e29-bc764e2007e4; Fri, 31 Jul 2020 11:55:43 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMv-0001W4-F1; Fri, 31 Jul 2020 12:38:33 +0100 X-Inumbo-ID: c25a5cad-d324-11ea-8e29-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 19/41] Executive: Drop redundant AND clause Date: Fri, 31 Jul 2020 12:37:58 +0100 Message-Id: <20200731113820.5765-20-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson , George Dunlap Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" In "Executive: Use index for report__find_test" we changed an EXISTS subquery into a JOIN. Now, the condition r.flight=3Df.flight is redundant because this is the join column (from USING). No functional change. CC: George Dunlap Signed-off-by: Ian Jackson Reviewed-by: George Dunlap --- Osstest/Executive.pm | 1 - 1 file changed, 1 deletion(-) diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm index 684cafc3..2f81e89d 100644 --- a/Osstest/Executive.pm +++ b/Osstest/Executive.pm @@ -433,7 +433,6 @@ END WHERE name=3D? AND name LIKE 'revision\_%' AND val=3D? - AND r.flight=3Df.flight AND ${\ main_revision_job_cond('r.job') } END push @params, "revision_$tree", $revision; --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196571; cv=none; d=zohomail.com; s=zohoarc; b=KzhUyNTVNJAgH0ercnog3lkOdUOo7ZJhQDvN+l1e2geJRT/JhgoW3MFfOvB6XhPF7PFFXC5VKe+An+Dlygl1B6upJ4Bx/Oc3sHDMJntY4qFY/GwZ11VMUHUWuYhn5q81hpfjismAplv8KAfcJ4k+4VgsEOZ71nC2QJiLKrwR8xU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196571; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=j10mCkeTm3By/T2lEAXQWuHomVNWsJNgc74CvuTsuTk=; b=nWo1BuEKJIzrqDaEUgIW/rcN0ORzSVtPnrFbpEafPHDwAOlRH2X1art3W7bLJN0jRS8wl1PQ4pgEqvwXF+7AEpjtuVaIJrkQLxvDUB6coKfwlUxR1pO6zSnWlrWE+X99zH3keP1w6sJjvKXq7b00NW05/iYMHyp1sOSjMuwzWsY= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196571584778.2499875801648; Fri, 31 Jul 2020 04:56:11 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tdg-0008QM-NO; Fri, 31 Jul 2020 11:55:52 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tdf-0008Q8-CS for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 11:55:51 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id c723ccb4-d324-11ea-8e29-bc764e2007e4; Fri, 31 Jul 2020 11:55:50 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMv-0001W4-Qw; Fri, 31 Jul 2020 12:38:34 +0100 X-Inumbo-ID: c723ccb4-d324-11ea-8e29-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 20/41] schema: Add index for quick lookup by host Date: Fri, 31 Jul 2020 12:37:59 +0100 Message-Id: <20200731113820.5765-21-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" Signed-off-by: Ian Jackson --- v2: Use proper \ escaping for underscores in LIKE --- schema/runvars-host-index.sql | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 schema/runvars-host-index.sql diff --git a/schema/runvars-host-index.sql b/schema/runvars-host-index.sql new file mode 100644 index 00000000..222a0a30 --- /dev/null +++ b/schema/runvars-host-index.sql @@ -0,0 +1,8 @@ +-- ##OSSTEST## 009 Preparatory +-- +-- This index helps sg-report-host-history find relevant flights. + +CREATE INDEX runvars_host_idx + ON runvars (val, flight) + WHERE name =3D'host' + OR name LIKE '%\_host'; --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196588; cv=none; d=zohomail.com; s=zohoarc; b=V5744s7ib5kwa0kxqoMFU2KLTn0idBNECiNrwpmTWt1e1RGUxCpjIqYUu/wYsHerXk8PMlbS2/RJn2T9bKpGhsMHP3dVsQKDJCZ/rztEBgcawr5KHNgEmmMH9Xv9/JqkM25b32s92uaPKrrL4KrrnlkwyOVkvlfrd3tcXM4AmiA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196588; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=iY+M1XZO2pSKL1zFU6kXsa95FO4XLmksOQwoHa9X8x4=; b=O8S/sU1GsvjHJXOcJs2XeV/zLdk32kYVaTNroe77tokN9YqV0s6OadFsTnctEPpA4qlg0bZRY9QZD1OJWotjFl1z2QF62ySwT4kd6GBtU26xe6f8rEMrrooEqUenX/MmUpqkFnBYxuRL2WtRN/GIJDjVu+At9HQ/mCUKjUAxAT8= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196588410569.0727686374328; Fri, 31 Jul 2020 04:56:28 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tdy-00005B-S1; Fri, 31 Jul 2020 11:56:10 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tdx-0008RD-Td for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 11:56:09 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id d21043bf-d324-11ea-8e29-bc764e2007e4; Fri, 31 Jul 2020 11:56:09 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMx-0001W4-0B; Fri, 31 Jul 2020 12:38:35 +0100 X-Inumbo-ID: d21043bf-d324-11ea-8e29-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 21/41] sg-report-host-history: Find flight limit by flight start date Date: Fri, 31 Jul 2020 12:38:00 +0100 Message-Id: <20200731113820.5765-22-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" By default we look for anything in (roughly) the last year. This query is in fact quite fast because the flights table is small. There is still the per-host limit of $limit (2000) recent runs. Signed-off-by: Ian Jackson --- sg-report-host-history | 56 ++++++++++++++++++++---------------------- 1 file changed, 27 insertions(+), 29 deletions(-) diff --git a/sg-report-host-history b/sg-report-host-history index 9730ae7a..a159df3e 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -29,6 +29,7 @@ use POSIX; use Osstest::Executive qw(:DEFAULT :colours); =20 our $limit=3D 2000; +our $timelimit=3D 86400 * (366 + 14); our $flightlimit; our $htmlout =3D "."; our $read_existing=3D1; @@ -45,6 +46,8 @@ while (@ARGV && $ARGV[0] =3D~ m/^-/) { last if m/^--?$/; if (m/^--(limit)\=3D([1-9]\d*)$/) { $$1=3D $2; + } elsif (m/^--time-limit\=3D([1-9]\d*)$/) { + $timelimit=3D $1; } elsif (m/^--flight-limit\=3D([1-9]\d*)$/) { $flightlimit=3D $1; } elsif (restrictflight_arg($_)) { @@ -108,38 +111,33 @@ sub read_existing_logs ($) { } =20 sub computeflightsrange () { - if (!$flightlimit) { - my $flagscond =3D - '('.join(' OR ', map { "f.hostflag =3D 'blessed-$_'" } @blessings).')= '; - my $nhostsq =3D db_prepare(<execute(); - my ($nhosts) =3D $nhostsq->fetchrow_array(); - print DEBUG "COUNTED $nhosts hosts\n"; - $flightlimit =3D $nhosts * $limit * 2; - } - - my $minflightsq =3D db_prepare(<execute(); + ($minflight,) =3D $minflightsq->fetchrow_array(); + } else { + my $minflightsq =3D db_prepare(<=3D ? + ORDER BY flight ASC + LIMIT 1 END - $minflightsq->execute(); - ($minflight,) =3D $minflightsq->fetchrow_array(); + my $now =3D time // die $!; + $minflightsq->execute($now - $timelimit); + ($minflight,) =3D $minflightsq->fetchrow_array(); + } $minflight //=3D 0; =20 $flightcond =3D "(flight > $minflight)"; --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596197594; cv=none; d=zohomail.com; s=zohoarc; b=PV9MsdOrQSRVpYrG2JxqpgRMmYXAvtMc1Knb/Iu+swncTxYU3zg7DvlH8xa7YOfvBmQNfie8D3xt9Ld1VsMFM5j4ub2DrZ2CKhlA9GT6uaoT8LE27d9B7po2ONtKt87hCASwKZhMMdMO2ba1k4xbWKxvopoDOY4CM5J7nWjxjv8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596197594; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=SxSP0Qt/VEjV+4btCejYHXrsn7sNCMuU8O0fvP+UaT0=; b=CKtzZi93KYU6IV3XQYqxwxfh/o+tcmQjK8yYSuLDuE7WB/5usq5k/hwhZ7y/sd1hYGj2pzB47uXPd1c9uyqp0ZKYehTVDy0eQgOfdYxIB77byeKhB4NypsOnGwFNGhLGTws+45qDJ4RMpgfp0rEbZ5FRrL62IEN1sH4zAxgIkFk= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596197594690895.7069034840945; Fri, 31 Jul 2020 05:13:14 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TuF-0004C7-QY; Fri, 31 Jul 2020 12:12:59 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TuD-0004Ac-Pm for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:12:57 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 2b067d9c-d327-11ea-8e2c-bc764e2007e4; Fri, 31 Jul 2020 12:12:57 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMx-0001W4-Jl; Fri, 31 Jul 2020 12:38:35 +0100 X-Inumbo-ID: 2b067d9c-d327-11ea-8e2c-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 22/41] sg-report-host-history: Drop per-job debug etc. Date: Fri, 31 Jul 2020 12:38:01 +0100 Message-Id: <20200731113820.5765-23-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" This printing has a significant effect on the performance of this program, at least after we optimise various other things. Signed-off-by: Ian Jackson --- sg-report-host-history | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/sg-report-host-history b/sg-report-host-history index a159df3e..a34458e0 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -102,9 +102,9 @@ sub read_existing_logs ($) { my $k =3D $1; s{\%([0-9a-f]{2})}{ chr hex $1 }ge; $ch->{$k} =3D $_; - print DEBUG "GOTCACHE $hostname $k\n"; +# print DEBUG "GOTCACHE $hostname $k\n"; } - print DEBUG "GOTCACHE $hostname \@ $jr->{flight} $jr->{job} $jr->{status}= ,$jr->{name}\n"; +# print DEBUG "GOTCACHE $hostname \@ $jr->{flight} $jr->{job} $jr->{status= },$jr->{name}\n"; $tcache->{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}} =3D $jr; } close H; @@ -272,7 +272,7 @@ END my @rows; my $cachehits =3D 0; foreach my $jr (@$inrows) { - print DEBUG "JOB $jr->{flight}.$jr->{job} "; + #print DEBUG "JOB $jr->{flight}.$jr->{job} "; =20 my $cacherow =3D $tcache->{$jr->{flight},$jr->{job},$jr->{status},$jr->{name}}; @@ -283,11 +283,11 @@ END =20 my $endedrow =3D jobquery($endedq, $jr, 'e'); if (!$endedrow) { - print DEBUG "no-finished\n"; + #print DEBUG "no-finished\n"; next; } - print DEBUG join " ", map { $endedrow->{$_} } sort keys %$endedrow; - print DEBUG ".\n"; + #print DEBUG join " ", map { $endedrow->{$_} } sort keys %$endedrow; + #print DEBUG ".\n"; =20 push @rows, { %$jr, %$endedrow }; } @@ -329,7 +329,7 @@ END next; } =20 - print DEBUG "JR $jr->{flight}.$jr->{job}\n"; + #print DEBUG "JR $jr->{flight}.$jr->{job}\n"; my $ir =3D jobquery($infoq, $jr, 'i'); my $ar =3D jobquery($allocdq, $jr, 'a'); my $ident =3D $jr->{name}; --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596197606; cv=none; d=zohomail.com; s=zohoarc; b=bikFZ2bzHreA6ySZ1SVovhuumIrC710BZLoZT4kUcGagJBMTyzVpbMaamXTbVmmzUYa0l9Zb4pzt6F5S5M+JIOttKtmIwOCvdRzQSXZwgxEivb60lnkygmYXvN12uC+h+CBDC1kyX4mOKRD77Ob1tVXfz8ypgvGmuKvwyV2Tsz8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596197606; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=2sfIde6fSGNOMXW88IZKAAfz7MCcaakLaRCZYpdK2Hk=; b=Ru0VlyAPK1pG7rwd7V2goznecNBM/Uvjm2lGkTyBNM1mYx5hCWbQXLXShesN7YWJRQ7hJ1r9S+QLljLIMiJpOnRHVZ5EX8t7x8lg0r/vmjESXbRUbUxRamhYABmopabu9n9gVW+thx7ZOHYUVb48BYN/K57DB3PRTJ41bQQgHrY= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596197606101871.3571992723683; Fri, 31 Jul 2020 05:13:26 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TuO-0004GX-E4; Fri, 31 Jul 2020 12:13:08 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TuM-0004Ac-VB for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:13:06 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 2defe6f6-d327-11ea-8e2c-bc764e2007e4; Fri, 31 Jul 2020 12:13:02 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMy-0001W4-0G; Fri, 31 Jul 2020 12:38:36 +0100 X-Inumbo-ID: 2defe6f6-d327-11ea-8e2c-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 23/41] Executive: Export opendb_tests Date: Fri, 31 Jul 2020 12:38:02 +0100 Message-Id: <20200731113820.5765-24-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" sg-report-host-history is going to want this in a moment Signed-off-by: Ian Jackson --- Osstest/Executive.pm | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm index 2f81e89d..8e4c5b9a 100644 --- a/Osstest/Executive.pm +++ b/Osstest/Executive.pm @@ -49,7 +49,7 @@ BEGIN { task_spec_desc findtask findtask_spec @all_lock_tabl= es restrictflight_arg restrictflight_cond report_run_getinfo report_altcolour - report_altchangecolour + report_altchangecolour opendb_tests report_blessingscond report_find_push_age_info tcpconnect_queuedaemon plan_search manual_allocation_base_jobinfo --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196919; cv=none; d=zohomail.com; s=zohoarc; b=nmfu/l/sdxk55vOg6ZOPAT/RNktFstmVjlVaOJ7q2Rwf3a8ixt1/0/jWEzRxJhnAHobFWn6sTqWgCa/0ISCNK5OCQjdzGj5bmzuEZDFm1LKEyjhlsSST7cNN4C5wmV0RxIPsuK8PxuC2D6YpWJpjeBy+K0hlY5ihCOkZb7oG2ws= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196919; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=mPq/wa2aoEpnRZk3KsMlklKKT3z8yPFDNHBZEvcQt60=; b=WLE1mExoLk26qY0BorxC0bv55Ccg2POUWqWSWyyvTMKXj3HdrgGXHm1QblSLMj+j/xfhwFxPkai2+i6fGwvPIqnmEdmrbUrjDH2NZ/HopZYzjEBX2S04klODzRoeNTsEJE+wH4frBQJlvb2+0xPU17gG3xQlJCpIBXfBGbYVDyc= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196919204194.31792450725413; Fri, 31 Jul 2020 05:01:59 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TjL-0001mI-3V; Fri, 31 Jul 2020 12:01:43 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TjJ-0001lW-Sp for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:01:41 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 982be3e6-d325-11ea-8e2b-bc764e2007e4; Fri, 31 Jul 2020 12:01:41 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMy-0001W4-Vh; Fri, 31 Jul 2020 12:38:37 +0100 X-Inumbo-ID: 982be3e6-d325-11ea-8e2b-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 24/41] sg-report-host-history: Add a debug print after sorting jobs Date: Fri, 31 Jul 2020 12:38:03 +0100 Message-Id: <20200731113820.5765-25-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" This helps rule this sort out as a source of slowness. Signed-off-by: Ian Jackson --- sg-report-host-history | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sg-report-host-history b/sg-report-host-history index a34458e0..1c2d19ae 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -318,6 +318,8 @@ END =20 @rows =3D sort { $b->{finished} <=3D> $a->{finished} } @rows; =20 + print DEBUG "SORTED\n"; + my $alternate =3D 0; my $wrote =3D 0; my $runvarq_hits =3D 0; --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196907; cv=none; d=zohomail.com; s=zohoarc; b=Kao4E97EXUlHn23D7qX2E9LLLEq6bXMv7sAeW0gvmI2eiRGY3qSHbv5iIg2aeTSD+Xb7Ci178/PLPo+if6dZ+PmKdUNdKCGz5xVYBqVJShvcqs58OMDoQ2eerh+oKPqDd1Isnhemy+iu+r1RJCIZBabjWYdKMwb++5cny0c26r4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196907; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=ri4jaXBUHrFflgSrLbWZI8QG4Lwg+r81yaLZb3cP1fA=; b=Fm8LnjpEAbKoLywSV132sX/6SpSYhaYP0HDYNU5pAGscnyPP88bAUVBIJh20SsbNazfHnlKLnvTubnSW2S2KbLqU4KOury7mxMTuanVyZxfciS4RKLYh90tV+UA+2JMQPQ5GD/pvmCgziEFbL20lskDaaoiRAvIpSMbVmBqKoPo= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196907647834.3134580067485; Fri, 31 Jul 2020 05:01:47 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tj7-0001cQ-Ag; Fri, 31 Jul 2020 12:01:29 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tj6-0001R9-3c for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:01:28 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 8fd671a2-d325-11ea-8e2b-bc764e2007e4; Fri, 31 Jul 2020 12:01:27 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMz-0001W4-DE; Fri, 31 Jul 2020 12:38:37 +0100 X-Inumbo-ID: 8fd671a2-d325-11ea-8e2b-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 25/41] sg-report-host-history: Do the main query per host Date: Fri, 31 Jul 2020 12:38:04 +0100 Message-Id: <20200731113820.5765-26-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" In f6001d628c3b3fd42b10cd15351981a04bc02572 we combined these queries into one: sg-report-host-history: Aggregate runvars query for all hosts Now that we have an index, there is a faster way for the db to do this query: via that index. But it doesn't like to do that if be aggregate the queries. Experimentally, doing this query separately once per host is significantly faster. Also, later, it will allow us to parallelise this work. So, we undo that. (Not by reverting, though.) Signed-off-by: Ian Jackson --- v2: Use proper \ escaping for underscores in LIKE --- schema/runvars-host-index.sql | 2 +- sg-report-host-history | 27 +++++++++------------------ 2 files changed, 10 insertions(+), 19 deletions(-) diff --git a/schema/runvars-host-index.sql b/schema/runvars-host-index.sql index 222a0a30..6a3ef377 100644 --- a/schema/runvars-host-index.sql +++ b/schema/runvars-host-index.sql @@ -1,4 +1,4 @@ --- ##OSSTEST## 009 Preparatory +-- ##OSSTEST## 009 Needed -- -- This index helps sg-report-host-history find relevant flights. =20 diff --git a/sg-report-host-history b/sg-report-host-history index 1c2d19ae..15866ab6 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -165,34 +165,25 @@ sub jobquery ($$$) { our %hosts; =20 sub mainquery () { - our $valcond =3D join " OR ", map { "val =3D ?" } keys %hosts; - our @params =3D keys %hosts; - our $runvarq //=3D db_prepare(< ? ORDER BY flight DESC - LIMIT ($limit * 3 + 100) * ? + LIMIT $limit * 2 END + foreach my $host (sort keys %hosts) { + print DEBUG "MAINQUERY $host...\n"; + $runvarq->execute($host, $minflight); =20 - push @params, $minflight; - push @params, scalar keys %hosts; - - print DEBUG "MAINQUERY...\n"; - $runvarq->execute(@params); - - print DEBUG "FIRST PASS\n"; - while (my $jr=3D $runvarq->fetchrow_hashref()) { - print DEBUG " $jr->{flight}.$jr->{job} "; - push @{ $hosts{$jr->{val}} }, $jr; + $hosts{$host} =3D $runvarq->fetchall_arrayref({}); + print DEBUG "MAINQUERY $host got ".(scalar @{ $hosts{$host} })."\n"; } - print DEBUG "\n"; } =20 sub reporthost ($) { --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196955; cv=none; d=zohomail.com; s=zohoarc; b=Ez3IPUkUBYqCRTh99Yw9d4oWp0C2Vk5h40w+lwWxSGum62IUWXSQNMH1E25AYNaexvMjGTmca7bTKiFRdn7/dc4hcybke8PrT+tkyA6pbTmtfmSgOFJyUKTuR4QE48U4y1WVPSiy+vbycHtgAuMsnXxgF8oa7CkMzfwM1x8zDyE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196955; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=4uU87FyXQPTumbQXj5S1kXLCrb1RGaJDo3mvIikplxw=; b=PqAelrV3HbRQ0rK2bjbg/F5fbJb/iz3HnDqV+AIiLrlPv5cQZXpaiFFhsa0bMgqrj64V2rCGBi4FOPSOGM/kmxoC941XZz9IVFiA21XlhFKFI2S4Monau7mHGUaZ02vCjptdr43PJ1RxIepjj2c8cDeQy000S6mVFalFhr4lQB8= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 159619695569182.44323563990281; Fri, 31 Jul 2020 05:02:35 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tjv-00029K-9j; Fri, 31 Jul 2020 12:02:19 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tju-00026u-4q for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:02:18 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id ad15deb1-d325-11ea-8e2b-bc764e2007e4; Fri, 31 Jul 2020 12:02:16 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TMz-0001W4-QA; Fri, 31 Jul 2020 12:38:37 +0100 X-Inumbo-ID: ad15deb1-d325-11ea-8e2b-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 26/41] sg-report-host-history: Rerganisation: Make mainquery per-host Date: Fri, 31 Jul 2020 12:38:05 +0100 Message-Id: <20200731113820.5765-27-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" This moves the loop over hosts into the main program. We are working our way to a new code structure. No functional change. Signed-off-by: Ian Jackson --- sg-report-host-history | 19 +++++++++++-------- 1 file changed, 11 insertions(+), 8 deletions(-) diff --git a/sg-report-host-history b/sg-report-host-history index 15866ab6..34216aa2 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -164,7 +164,9 @@ sub jobquery ($$$) { =20 our %hosts; =20 -sub mainquery () { +sub mainquery ($) { + my ($host) =3D @_; + our $runvarq //=3D db_prepare(<execute($host, $minflight); =20 - $hosts{$host} =3D $runvarq->fetchall_arrayref({}); - print DEBUG "MAINQUERY $host got ".(scalar @{ $hosts{$host} })."\n"; - } + print DEBUG "MAINQUERY $host...\n"; + $runvarq->execute($host, $minflight); + + $hosts{$host} =3D $runvarq->fetchall_arrayref({}); + print DEBUG "MAINQUERY $host got ".(scalar @{ $hosts{$host} })."\n"; } =20 sub reporthost ($) { @@ -474,7 +475,9 @@ db_retry($dbh_tests, [], sub { }); =20 db_retry($dbh_tests, [], sub { - mainquery(); + foreach my $host (sort keys %hosts) { + mainquery($host); + } }); =20 foreach my $host (sort keys %hosts) { --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196579; cv=none; d=zohomail.com; s=zohoarc; b=hxeuNhbMLPxUXFG0huHztZW6VC5nJFcewJyiwuDLBY/B1phJBsizej1Ze3iCrmnHW/6AFYsDWwqXVgqJKC4iF8pWuRHXAU8pTJEDfr5/+tTFVWhjskf346f2bFCPNZtYx4zZQ2u7kLzx+DrWJvqAMx24uHk4kpHUNmotLArcUu0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196579; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=I0lwOGoN7QHgx4BX7TtErz6/7M/yh+u3WM5DQNp/VzA=; b=XcInb6ygwRqWTyeaVu/V60AcBRpKutmBIlmabLeQooA9Q1+qoAg/gNblpfQFshOora8nGc48TtDBTpjqVUrc3oIp5MbdpLn3YaThwR2ulYSlYP9Cy4KzjheJSxb+J3D3ZKxbsUt/6NKOGEMUAwnJzRmArcanCr0q+tV38IcZxQM= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 159619657909850.31463220852686; Fri, 31 Jul 2020 04:56:19 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tdu-0008Uj-IZ; Fri, 31 Jul 2020 11:56:06 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tds-0008RD-Pc for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 11:56:04 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id ce5a97ba-d324-11ea-8e29-bc764e2007e4; Fri, 31 Jul 2020 11:56:02 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TN0-0001W4-4W; Fri, 31 Jul 2020 12:38:38 +0100 X-Inumbo-ID: ce5a97ba-d324-11ea-8e29-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 27/41] sg-report-host-history: Rerganisation: Read old logs later Date: Fri, 31 Jul 2020 12:38:06 +0100 Message-Id: <20200731113820.5765-28-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" Perhaps at one point something read from these logs influenced the db query for thye flights range, but that is no longer the case and it doesn't seem likely to need to come back. We want to move the per-host stuff together. No functional change. Signed-off-by: Ian Jackson --- sg-report-host-history | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/sg-report-host-history b/sg-report-host-history index 34216aa2..3f4670e5 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -466,14 +466,14 @@ END =20 exit 0 unless %hosts; =20 -foreach (keys %hosts) { - read_existing_logs($_); -} - db_retry($dbh_tests, [], sub { computeflightsrange(); }); =20 +foreach (keys %hosts) { + read_existing_logs($_); +} + db_retry($dbh_tests, [], sub { foreach my $host (sort keys %hosts) { mainquery($host); --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596197601; cv=none; d=zohomail.com; s=zohoarc; b=FgHyyBXNIKVB+VxGiUMKvM931ZsTFJ0TzPagi6/gLQ90vf8M02Ewp1LbbsaWVxmOraikm8+ugUhpzjdYtJEHKTPaJ8hv43wTfWPXTj/mEgnsKq4vKLrOpJQTxSQnxehIay9v/+HxD7Le2ao6kzI92ZCn1FKVThNDq2i8F+YeftQ= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596197601; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=ojN+tPlTuV++z+KhcNURIwdVh/0xcdatys0vqvUaIgw=; b=ZaTPP284tXUKnZJ0BsAxoMhHaDrZ8t8oSnVLAf40Tjp5LaH2pVAStyMl93kMuIwP7GDesVAGxEK5u0yu0K18hff1K/rgyjDPnRxgcsFjHYT2g+6J44SvScp9FOoq20sMw9Yj0KY5ircZtIjiXOUz0ybBWpZG/Uvq4wN+9nxRvmM= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596197601337643.7540966482362; Fri, 31 Jul 2020 05:13:21 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TuJ-0004EJ-4s; Fri, 31 Jul 2020 12:13:03 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TuH-0004Ac-V0 for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:13:01 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 2c6d87ca-d327-11ea-8e2c-bc764e2007e4; Fri, 31 Jul 2020 12:12:59 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TN0-0001W4-HO; Fri, 31 Jul 2020 12:38:38 +0100 X-Inumbo-ID: 2c6d87ca-d327-11ea-8e2c-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 28/41] sg-report-host-history: Rerganisation: Change loops Date: Fri, 31 Jul 2020 12:38:07 +0100 Message-Id: <20200731113820.5765-29-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" Move the per-host code all into the same per-host loop. One effect is to transpose the db_retry and host loops for mainquery. No functional change. Signed-off-by: Ian Jackson --- sg-report-host-history | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/sg-report-host-history b/sg-report-host-history index 3f4670e5..2ca0e235 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -470,18 +470,10 @@ db_retry($dbh_tests, [], sub { computeflightsrange(); }); =20 -foreach (keys %hosts) { - read_existing_logs($_); -} - -db_retry($dbh_tests, [], sub { - foreach my $host (sort keys %hosts) { - mainquery($host); - } -}); - foreach my $host (sort keys %hosts) { + read_existing_logs($host); db_retry($dbh_tests, [], sub { + mainquery($host); reporthost $host; }); } --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596197613; cv=none; d=zohomail.com; s=zohoarc; b=HtUltecjU8b6e6xyyZj3li9gLCRJrrZYgKTqMMZ/bvciwyiPFZsRhQmkXMf/HeHJckQdPJyZGe8ZlludVCyP2Na66jd4fUOkNVPayl647M5iga1s15PftU+GfgbBszuyl+83ZEaB837KlL/8aCIs4ybfOoKPZx+QPMP1W/vNmdM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596197613; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=esMdtOYlvKBH+HboBjv9HCMu9nBpR7yPVo/UxMBfkto=; b=WJqeMBZcV/v9wbALtxObf0UoqflAS9Gw8D4oxYNK3UhLWa6VQeS17buCRMArCTsWa/mG3V8a3isHfqG97J5L5ojACTWTbyldoW80NRI2Lc7kYrdKvp1e0Q68iGbUpJfULjygEpZGx6D0ar5RZDED7aLuaxnM8QDp+V0e8/9X6/4= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596197613252549.2920200824866; Fri, 31 Jul 2020 05:13:33 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TuY-0004MN-5g; Fri, 31 Jul 2020 12:13:18 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TuW-0004Ac-VG for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:13:16 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 33015f95-d327-11ea-8e2c-bc764e2007e4; Fri, 31 Jul 2020 12:13:10 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TN0-0001W4-Th; Fri, 31 Jul 2020 12:38:39 +0100 X-Inumbo-ID: 33015f95-d327-11ea-8e2c-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 29/41] sg-report-host-history: Drop a redundznt AND clause Date: Fri, 31 Jul 2020 12:38:08 +0100 Message-Id: <20200731113820.5765-30-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" This condition is the same as $flightcond. (This has no effect on the db performance since the query planner figures it out, but it is confusing.) Signed-off-by: Ian Jackson --- sg-report-host-history | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/sg-report-host-history b/sg-report-host-history index 2ca0e235..f4352fc3 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -175,13 +175,12 @@ sub mainquery ($) { AND val =3D ? AND $flightcond AND $restrictflight_cond - AND flight > ? ORDER BY flight DESC LIMIT $limit * 2 END =20 print DEBUG "MAINQUERY $host...\n"; - $runvarq->execute($host, $minflight); + $runvarq->execute($host); =20 $hosts{$host} =3D $runvarq->fetchall_arrayref({}); print DEBUG "MAINQUERY $host got ".(scalar @{ $hosts{$host} })."\n"; --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196572; cv=none; d=zohomail.com; s=zohoarc; b=jm0hkiqShS9suuPUlP2P1C2EyKcQrHjiER4bzejZPnPuKn9eU64+dcwai+70/9nDl3vdvj8lva7rJ6TkAHOQDigNSqju6jvXAqpS80GsULj6RKCLHufItEUujjZ6Mrx1LKr/60YHGtaVr2MeVjuz0CRoxsGUxiP9D5tZXpayXU8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196572; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=BnWhPoCAJ9Bpc8rx0pGnp+T7GMpVrNVjUZQwqmH3wz8=; b=OGFGgrusgoB+6QHvhYvI0z7PovMyJMdboFbXfYVer9ZSKF3jo/C3jynyvoCj35PGtHaxDDtGszkB7ulOYKxqzV3nWOjGjHvBzMyp7QuDzZEj1Wv8L1nOGmD7Zfy+bIm4+f0P90jPKr4f8vkKcvuU4Wbyf2UKSLWY19TYz5Sl2s8= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196572536716.3374235646306; Fri, 31 Jul 2020 04:56:12 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tdn-0008RS-0O; Fri, 31 Jul 2020 11:55:59 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tdl-0008RD-O1 for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 11:55:57 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id caff2b44-d324-11ea-8e29-bc764e2007e4; Fri, 31 Jul 2020 11:55:57 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TN1-0001W4-JE; Fri, 31 Jul 2020 12:38:39 +0100 X-Inumbo-ID: caff2b44-d324-11ea-8e29-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 30/41] sg-report-host-history: Fork Date: Fri, 31 Jul 2020 12:38:09 +0100 Message-Id: <20200731113820.5765-31-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" Run each host's report in a separate child. This is considerably faster. Signed-off-by: Ian Jackson --- sg-report-host-history | 47 +++++++++++++++++++++++++++++++++++------- 1 file changed, 40 insertions(+), 7 deletions(-) diff --git a/sg-report-host-history b/sg-report-host-history index f4352fc3..dc694ebe 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -34,6 +34,7 @@ our $flightlimit; our $htmlout =3D "."; our $read_existing=3D1; our $doinstall=3D1; +our $maxjobs=3D10; our @blessings; =20 open DEBUG, ">/dev/null"; @@ -44,7 +45,7 @@ csreadconfig(); while (@ARGV && $ARGV[0] =3D~ m/^-/) { $_=3D shift @ARGV; last if m/^--?$/; - if (m/^--(limit)\=3D([1-9]\d*)$/) { + if (m/^--(limit|maxjobs)\=3D([1-9]\d*)$/) { $$1=3D $2; } elsif (m/^--time-limit\=3D([1-9]\d*)$/) { $timelimit=3D $1; @@ -469,12 +470,44 @@ db_retry($dbh_tests, [], sub { computeflightsrange(); }); =20 +undef $dbh_tests; + +our %children; +our $worst =3D 0; + +sub wait_for_max_children ($) { + my ($lim) =3D @_; + while (keys(%children) > $lim) { + $!=3D0; $?=3D0; my $got =3D wait; + die "$! $got $?" unless exists $children{$got}; + my $host =3D $children{$got}; + delete $children{$got}; + $worst =3D $? if $? > $worst; + if ($?) { + print STDERR "sg-report-flight[: [$got] failed for $host: $?\n"; + } else { + print DEBUG "REAPED [$got] $host\n"; + } + } +} + foreach my $host (sort keys %hosts) { - read_existing_logs($host); - db_retry($dbh_tests, [], sub { - mainquery($host); - reporthost $host; - }); + wait_for_max_children($maxjobs); + + my $pid =3D fork // die $!; + if (!$pid) { + opendb_tests(); + read_existing_logs($host); + db_retry($dbh_tests, [], sub { + mainquery($host); + reporthost $host; + }); + print DEBUG "JQ CACHE ".($jqtotal-$jqcachemisses)." / $jqtotal\n"; + exit(0); + } + print DEBUG "SPAWNED [$pid] $host\n"; + $children{$pid} =3D $host; } =20 -print DEBUG "JQ CACHE ".($jqtotal-$jqcachemisses)." / $jqtotal\n"; +wait_for_max_children(0); +exit $worst; --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196863; cv=none; d=zohomail.com; s=zohoarc; b=K/xO3pSAKLur5xZ+dFqxgGanuTooEFCdqRa3f8U4OPa61WHgofCqHKkxslfNXvjdkE+kklLaJV3AQPQy2IjlRnV7v6LR+IEzF+G0ieYZ0m147ERS7XHeYzspTDOU4FC4lRW1U74hXNOMTiw18vvu7XlufnYNNHRpXwQF8fatIZ8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196863; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=EvrU87VUmhqPQcLcoQxj1pEzDXnicDLvMnlvyon9imw=; b=WCFpFzmTUykt6OCeN+FiMtpt4vbsnCr8xFbfMPhTuGe1GorqDRlFsQbGffFTSJkvVqkWXSHLP8Z5Mlu3zHhjsLnaP4pgckEp6pq5Iap10DDoL2EL1WjALbNhRTnRkGt0iQqeyiWCQs5VIT4MSbE0/tPe9xu4dR5jqLCtzJne2HY= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196863827320.4276708451257; Fri, 31 Jul 2020 05:01:03 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TiL-0001Jt-P9; Fri, 31 Jul 2020 12:00:41 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TiK-0001Jk-L8 for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:00:40 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 737bffed-d325-11ea-8e2b-bc764e2007e4; Fri, 31 Jul 2020 12:00:40 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TN1-0001W4-UH; Fri, 31 Jul 2020 12:38:40 +0100 X-Inumbo-ID: 737bffed-d325-11ea-8e2b-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 31/41] schema: Add index to help cs-bisection-step Date: Fri, 31 Jul 2020 12:38:10 +0100 Message-Id: <20200731113820.5765-32-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" cs-bisection step basis search involves looking for recent flights that weren't broken. A flight is broken if it has broken steps. Make an index for this to save it scanning the steps table. Signed-off-by: Ian Jackson --- schema/steps-broken-index.sql | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 schema/steps-broken-index.sql diff --git a/schema/steps-broken-index.sql b/schema/steps-broken-index.sql new file mode 100644 index 00000000..770747cc --- /dev/null +++ b/schema/steps-broken-index.sql @@ -0,0 +1,7 @@ +-- ##OSSTEST## 010 Harmless +-- +-- This index helps cs-bisection-flight check if flighss are broken. + +CREATE INDEX steps_broken_idx + ON steps (flight) + WHERE status=3D'broken'; --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196963; cv=none; d=zohomail.com; s=zohoarc; b=LDwfSf8vNYWA5SHIy8yRP5J75Hy1X5Fp8thRXJdGdEILT+cXuTtQw7yqa9JqcfM29vrSCyT7869lXSwNU8FOoErXUr0y8I2NB+YOhMLTxWUDXHYpnQ7fYmJ/vTosuWVs5azzgCueaAtXLL3NERrRJj1gbJXPIYWga/sINVB89AY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196963; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=W1mgnQbXDqH08M0+fLZK8yStUyw83xouXKkSgeL0pDg=; b=hVoiBq+IefnZbvL86xrpFZstC67vppWvpUeQQ13bo7TVTA/XFfEHf9uBzqkCC5pJbUrMXSnHnaQB7Kd2tradlTWrNOgPK64KnbxTan8L1LFAXqvnLNNS5c6HMItk3JZ99RTscj5M4nbHy3hMzX/Nr8bt5hL7sd64K3gbazQ11vM= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196963550490.43909521614; Fri, 31 Jul 2020 05:02:43 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tk4-0002HY-Si; Fri, 31 Jul 2020 12:02:28 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tk4-00026u-5E for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:02:28 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id b261fa71-d325-11ea-8e2b-bc764e2007e4; Fri, 31 Jul 2020 12:02:26 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TN2-0001W4-AN; Fri, 31 Jul 2020 12:38:40 +0100 X-Inumbo-ID: b261fa71-d325-11ea-8e2b-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 32/41] adhoc-revtuple-generator: Fix an undef warning in a debug print Date: Fri, 31 Jul 2020 12:38:11 +0100 Message-Id: <20200731113820.5765-33-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" $parents might be undef here. Signed-off-by: Ian Jackson --- New in v2. --- adhoc-revtuple-generator | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/adhoc-revtuple-generator b/adhoc-revtuple-generator index c8d6f4ad..ec33305a 100755 --- a/adhoc-revtuple-generator +++ b/adhoc-revtuple-generator @@ -463,7 +463,7 @@ sub coalesce { $out->{$node}{Date}=3D $explode_date; my $parents=3D $graphs[$explode_i]{ $node[$explode_i] }{Parents}; print DEBUG "#$explode_i $explode_isearliest". - " $explode_date x".scalar(@$parents)."\n"; + " $explode_date x".($parents ? scalar(@$parents) : "-")."\n"; =20 foreach my $subparent (@$parents) { $node[$explode_i]=3D $subparent; --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196952; cv=none; d=zohomail.com; s=zohoarc; b=elVWVhsDt6eQAdrYeffnyiKPVdN7EVsFzwTLqk4RBSv5ERfL0CorPL2yAfHAxHMcHLmcZl6KF/D0H1RAm+w2rlMT/bua4Sf5pLLqIOlHtz/DrRZ3t1B7XWVl2Iz8aHZK+KBkftLPRfNYZCrihG2+eeXPHGXlNznWoLrySvENX68= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196952; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=pjeDVzUtdR4kb5dabVHEyISibR6LFVgNAcfFS243cTw=; b=gKDYkhrNQAjPZlvWCNvZkyuJ0ZdwZDRzVBgazG6yq6NbVI+4rFqajJ7X7Fwhhr/b/eG7k9aU7P77bNcmxAJeD2IO3EQ9LkTAcLfq8yHKQTIqg+ttKyMYYL0Y2mJPcOEGROMmaiRjJBR9Suzy20nbqs2R5jhaH5ncL/YJJM/6qSA= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196952791448.990419202603; Fri, 31 Jul 2020 05:02:32 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tjt-00027b-0S; Fri, 31 Jul 2020 12:02:17 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tjr-00026u-HR for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:02:15 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id ac328bec-d325-11ea-8e2b-bc764e2007e4; Fri, 31 Jul 2020 12:02:14 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TN2-0001W4-LZ; Fri, 31 Jul 2020 12:38:40 +0100 X-Inumbo-ID: ac328bec-d325-11ea-8e2b-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 33/41] cs-bisection-step: Generalise qtxt_common_rev_ok Date: Fri, 31 Jul 2020 12:38:12 +0100 Message-Id: <20200731113820.5765-34-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" * Make it into a subref which takes a $table argument. * Change the two references into function calls using the @{...} syntax * Move the definition earlier in the file No change to the generated query. Signed-off-by: Ian Jackson --- v2: New patch. --- cs-bisection-step | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/cs-bisection-step b/cs-bisection-step index 9a0fee39..5d4e179e 100755 --- a/cs-bisection-step +++ b/cs-bisection-step @@ -182,6 +182,14 @@ END sub flight_rmap ($$) { my ($flight, $need_urls) =3D @_; =20 + my $qtxt_common_rev_ok =3D sub { + my ($table) =3D @_; + [<do(<('rev') } =20 ORDER by rev.name; =20 @@ -262,7 +266,7 @@ $qtxt_common_tables CROSS JOIN tmp_build_info AS url =20 WHERE -$qtxt_common_rev_condition +@{ $qtxt_common_rev_ok->('rev') } AND url.name LIKE E'tree\\_%' AND url.use =3D rev.use AND url.job =3D rev.job --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596197608; cv=none; d=zohomail.com; s=zohoarc; b=Esw8IxhqQDBGsfRjnLHqV5elweVQLFpbbnFrYJ72+QN1wd7rlEsSYiEOSYCQeLcepiIXQiN13/BWIiBeorBGRJd8lNYbkJmJhYQPmABpfJpd0idP+YSCG7aRF/rn1M047M+nRn00I3uvpFnDr1gXMPQALbgEXZfxVZIoYUhsvU0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596197608; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=I+yHXsjgQsVLW9kv3beIh4i0TMmxbVMJW8YeEJOogz0=; b=ZIWJ9n43ZIqk9ZH3Oin7+rWcwui+p51V/aGTcivDv08BDQ/+uTSFGe3lTJKUpYYJ13d7LFA5b9XlYiHWlajnVcViMH73xt4Q1kDOwm6hHw5jb05BXdGC9N10BB7sG/Ek4ZkmU0zgLPNC3OUSyhWMdiV/h9h9B9YUuIE8dbBwh/c= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596197608358408.85934234884576; Fri, 31 Jul 2020 05:13:28 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TuT-0004Jk-Re; Fri, 31 Jul 2020 12:13:13 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TuR-0004Ac-VB for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:13:11 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 3007e43e-d327-11ea-8e2c-bc764e2007e4; Fri, 31 Jul 2020 12:13:05 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TN3-0001W4-1J; Fri, 31 Jul 2020 12:38:41 +0100 X-Inumbo-ID: 3007e43e-d327-11ea-8e2c-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 34/41] cs-bisection-step: Move an AND Date: Fri, 31 Jul 2020 12:38:13 +0100 Message-Id: <20200731113820.5765-35-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" This obviously-fine change makes the next commit easier to review. Signed-off-by: Ian Jackson --- v2: New patch. --- cs-bisection-step | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/cs-bisection-step b/cs-bisection-step index 5d4e179e..f11726aa 100755 --- a/cs-bisection-step +++ b/cs-bisection-step @@ -266,8 +266,8 @@ $qtxt_common_tables CROSS JOIN tmp_build_info AS url =20 WHERE -@{ $qtxt_common_rev_ok->('rev') } - AND url.name LIKE E'tree\\_%' +@{ $qtxt_common_rev_ok->('rev') } AND + url.name LIKE E'tree\\_%' AND url.use =3D rev.use AND url.job =3D rev.job AND (rev.name =3D 'built_revision_' || substr(url.name,6) OR --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196910; cv=none; d=zohomail.com; s=zohoarc; b=fytp5dkF9ILM6bM7hlrsBr6qL6LbE45E8eNVr8Bfl7PQYReiCSYVZmunYTlbIGXX6U4lTijANrJ4n0N7LmZpoXKpx1WQwSL4fFxNd7nDU8KctXZOnB27FSEkIvJdy5R5JwslGkff3u/6rIqrU+MpNvisb8lfeGZHQeV96G4WUBk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196910; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=InYHhxbyA4nCYttLwfs8hYWNW7tlc4rqzmzsjeaO3og=; b=IKYnJtOyvBNEtEBhuaEPFOIyAfZ1u3QpEJYbxXjl2u0jdmafZM8epyoRITTr6wjfQh3QUMSz61eEa+JMxKSwMyFLFbSfB36XkJtfQh4kunAefJW68gdmAHmBmvKCx+aYFEXJ7hMII2R5DKADmQopbMat43aqYIN+UAFwBVTE6xU= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196910822990.9421892547322; Fri, 31 Jul 2020 05:01:50 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TjB-0001ft-Pk; Fri, 31 Jul 2020 12:01:33 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TjB-0001R9-3l for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:01:33 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 9260ce9a-d325-11ea-8e2b-bc764e2007e4; Fri, 31 Jul 2020 12:01:31 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TN3-0001W4-Dg; Fri, 31 Jul 2020 12:38:41 +0100 X-Inumbo-ID: 9260ce9a-d325-11ea-8e2b-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 35/41] cs-bisection-step: Break out qtxt_common_ok Date: Fri, 31 Jul 2020 12:38:14 +0100 Message-Id: <20200731113820.5765-36-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" Make this bit of query into a subref which takes a $table argument. No change to the generated query. Signed-off-by: Ian Jackson --- v2: New patch. --- cs-bisection-step | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/cs-bisection-step b/cs-bisection-step index f11726aa..ba0c6424 100755 --- a/cs-bisection-step +++ b/cs-bisection-step @@ -190,6 +190,13 @@ sub flight_rmap ($$) { END }; =20 + my $qtxt_common_tree_ok =3D sub { + my ($table) =3D @_; + [<do(<('rev') } AND - url.name LIKE E'tree\\_%' +@{ $qtxt_common_tree_ok->('url') } AND url.use =3D rev.use AND url.job =3D rev.job AND (rev.name =3D 'built_revision_' || substr(url.name,6) OR --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596197586; cv=none; d=zohomail.com; s=zohoarc; b=AeAAavlM6rR29w8D2S6XZa9KoLuCTnO1EaEiIEhGpmFGu+YbE9t3JDdoGBltzFjmxzftsGhIXWEyoaA6CowDz+ftt73imJs3nba2KEAkk0wd5uoGV0sLt31PLw0uOCo7uNP/OhJC1yDPkLcIIQP1jkNbQDOLheshMnNnm2alOFo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596197586; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=D+eMwV1eDgPJj5AZWejW+3mIh7F/vQ/RohOV1VcmYzY=; b=cj/NM1jU/oiI8DsnyrA9G7AbgsHtonO97QKUiNjy6S7e83MFnPJRTwtae1ZyKpHB8DePZtuqsZhvOrq3Kx0x0nMcJEHkUImOjMxbGw68FsiCzL2bEXWY45EEG4p6dwQ9TdBc2HwiHB3t1IpOsuLIfnn6kXxajZ03lhAOt4rVkZk= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596197586559311.1160582742798; Fri, 31 Jul 2020 05:13:06 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tu4-00048N-9w; Fri, 31 Jul 2020 12:12:48 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tu3-000489-Db for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:12:47 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 24dc3ab0-d327-11ea-8e2c-bc764e2007e4; Fri, 31 Jul 2020 12:12:46 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TN3-0001W4-Pd; Fri, 31 Jul 2020 12:38:41 +0100 X-Inumbo-ID: 24dc3ab0-d327-11ea-8e2c-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 36/41] cs-bisection-step: Use db_prepare a few times instead of ->do Date: Fri, 31 Jul 2020 12:38:15 +0100 Message-Id: <20200731113820.5765-37-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" With $dbh_tests->do(...), we can only get a debug trace of the queries by using DBI_TRACE which produces voluminous output. Using our own db_prepare invokes our own debugging. No functional change. Signed-off-by: Ian Jackson --- v2: New patch. --- cs-bisection-step | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/cs-bisection-step b/cs-bisection-step index ba0c6424..1c165b78 100755 --- a/cs-bisection-step +++ b/cs-bisection-step @@ -197,7 +197,7 @@ END END }; =20 - $dbh_tests->do(<execute(); CREATE TEMP TABLE tmp_build_info ( use varchar NOT NULL, name varchar NOT NULL, @@ -206,7 +206,7 @@ END ) END =20 - $dbh_tests->do(<execute($job, $flight); =20 INSERT INTO tmp_build_info SELECT t.name AS use, @@ -230,7 +230,7 @@ END END) END =20 - $dbh_tests->do(<execute($job, $flight); =20 INSERT INTO tmp_build_info SELECT '' AS use, --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196863; cv=none; d=zohomail.com; s=zohoarc; b=T8Gre/NwEgSDlKbUT/YyBv/8Ar2MVOPpBA1ISGMxjcwUnlEF9UJrOa/WGOg2Y6JQQvXEx4n4wyqAlJrz8Odmz/GSLunPJBYwFjVrdnQlJfFLhZggtF/4IWwBmhdGurrb6ujhNwWEJmrtMcQCfS3SJtsljK0Ddex4txZ0DE6q30w= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196863; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=1dQ//Sw7HHt4gHGTSJ1nQQt0MUpFB7vysgRSCik9i/g=; b=TzUfM5Z1LrAqtl90ieOm1EtFhwhG3YS7M3HJ9d8PwfttnlUpooxIJfXrkUZmbCjqKy6wA835lnuIJOwdjZ3n1zsbwPLMsKs5dowb5HYgj8juM+pp6OgkpMIbLCemOP6gyxv6jli662JJhn6ovZTqvMZBALq/HYbk/a3itHO/XEU= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196863444792.7415399916684; Fri, 31 Jul 2020 05:01:03 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TiR-0001KY-0p; Fri, 31 Jul 2020 12:00:47 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TiP-0001Jk-Jv for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:00:45 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 76268c7e-d325-11ea-8e2b-bc764e2007e4; Fri, 31 Jul 2020 12:00:44 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TN4-0001W4-8W; Fri, 31 Jul 2020 12:38:42 +0100 X-Inumbo-ID: 76268c7e-d325-11ea-8e2b-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 37/41] cs-bisection-step: temporary table: Insert only rows we care about Date: Fri, 31 Jul 2020 12:38:16 +0100 Message-Id: <20200731113820.5765-38-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" Every use of this table has a WHERE or ON which invokes at least one of these conditions. So put only those rows into the table. This provides a significant speedup (which I haven't properly measured). No overall functional change. Signed-off-by: Ian Jackson --- v2: New patch. --- cs-bisection-step | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/cs-bisection-step b/cs-bisection-step index 1c165b78..718c87b0 100755 --- a/cs-bisection-step +++ b/cs-bisection-step @@ -219,7 +219,9 @@ END =20 WHERE t.job =3D ? AND t.flight =3D ? - AND t.name LIKE '%buildjob' + AND t.name LIKE '%buildjob' AND +(@{ $qtxt_common_rev_ok->('b') } OR + @{ $qtxt_common_tree_ok->('b') }) AND b.flight =3D (CASE WHEN t.val NOT LIKE '%.%' THEN t.flight ELSE cast(split_part(t.val, '.', 1) AS i= nt) @@ -239,7 +241,9 @@ END job AS job FROM runvars WHERE job =3D ? - AND flight =3D ? + AND flight =3D ? AND +(@{ $qtxt_common_rev_ok->('runvars') } OR + @{ $qtxt_common_tree_ok->('runvars') }) END =20 my $qtxt_common_results =3D < (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196900684293.6268083331437; Fri, 31 Jul 2020 05:01:40 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tix-0001Wj-NT; Fri, 31 Jul 2020 12:01:19 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tiw-0001R9-6M for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:01:18 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 8a0dea02-d325-11ea-8e2b-bc764e2007e4; Fri, 31 Jul 2020 12:01:17 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TN4-0001W4-Jj; Fri, 31 Jul 2020 12:38:43 +0100 X-Inumbo-ID: 8a0dea02-d325-11ea-8e2b-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 38/41] SQL: Change LIKE E'...\\_...' to LIKE '...\_...' Date: Fri, 31 Jul 2020 12:38:17 +0100 Message-Id: <20200731113820.5765-39-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" E'...' means to interpret \-escapes. But we don't want them: without E, we can avoid some toothpick-doubling. No functional change. Signed-off-by: Ian Jackson --- v2: New patch. --- cs-bisection-step | 8 ++++---- sg-report-job-history | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/cs-bisection-step b/cs-bisection-step index 718c87b0..a82cbfb8 100755 --- a/cs-bisection-step +++ b/cs-bisection-step @@ -185,15 +185,15 @@ sub flight_rmap ($$) { my $qtxt_common_rev_ok =3D sub { my ($table) =3D @_; [<{Rtuple}; diff --git a/sg-report-job-history b/sg-report-job-history index d5f91ff1..22a28627 100755 --- a/sg-report-job-history +++ b/sg-report-job-history @@ -92,7 +92,7 @@ if (defined($flight)) { our $revisionsq=3D db_prepare(< (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196902012266.30422814798726; Fri, 31 Jul 2020 05:01:42 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tj2-0001Z5-1Q; Fri, 31 Jul 2020 12:01:24 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tj1-0001R9-3N for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:01:23 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 8c6e59e4-d325-11ea-8e2b-bc764e2007e4; Fri, 31 Jul 2020 12:01:21 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TN5-0001W4-Ch; Fri, 31 Jul 2020 12:38:43 +0100 X-Inumbo-ID: 8c6e59e4-d325-11ea-8e2b-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 39/41] cs-bisection-step: Add a debug print when we run dot(1) Date: Fri, 31 Jul 2020 12:38:18 +0100 Message-Id: <20200731113820.5765-40-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" Amongst other things this was useful for perf investigation. Signed-off-by: Ian Jackson --- v2: New patch. --- cs-bisection-step | 1 + 1 file changed, 1 insertion(+) diff --git a/cs-bisection-step b/cs-bisection-step index a82cbfb8..027032a1 100755 --- a/cs-bisection-step +++ b/cs-bisection-step @@ -1114,6 +1114,7 @@ END =20 if (eval { foreach my $fmt (qw(ps png svg)) { + print DEBUG "RUNNING dot -T$fmt\n"; system_checked("dot", "-T$fmt", "-o$graphfile.$fmt", "$graphfile.dot"); } --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196961; cv=none; d=zohomail.com; s=zohoarc; b=S41YIyuNNXNG6doMdu2fUmoLEGIcy7Ihlsb56+Hz+I4Veh55KIwqqZgKe0GE3yYDNEqxV6hsaSpFi5wh7druMW3mWelM9T5PG/bx+UfqpsO3KbrsUVxg7DEOXc5+t7uUCzBT30L8hsaR7YVzx43NRt/81kbP6JQfQhLovVQNiIo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196961; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=WoFUjXBWBDlvECsedWsXS7r5AuUWvl6nXsXWgFuYp4I=; b=VNeWdh5tQzZvdhV5vubQhACgEJ6Op3eYY1iRo1wVbgncsv6MCcg/mSOCo2b+Zt5QvPXpVdADau+6LfQlwZw5L1jDqyviK6urG7XbPi3nFy8AjcxhySnDADTJkC4SE1UG8S3+sszO3CVhTY2AxRJpT6mQnFA2CKqG7oN7ezREkhE= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196961322353.6645214056422; Fri, 31 Jul 2020 05:02:41 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tk0-0002Cz-IP; Fri, 31 Jul 2020 12:02:24 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tjz-00026u-58 for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:02:23 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id af900828-d325-11ea-8e2b-bc764e2007e4; Fri, 31 Jul 2020 12:02:20 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TN6-0001W4-2a; Fri, 31 Jul 2020 12:38:44 +0100 X-Inumbo-ID: af900828-d325-11ea-8e2b-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 40/41] cs-bisection-step: Lay out the revision tuple graph once Date: Fri, 31 Jul 2020 12:38:19 +0100 Message-Id: <20200731113820.5765-41-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" The graph layout algorithm is not very fast, particularly if the revision graph is big. In my test case this saves about 10s. Signed-off-by: Ian Jackson --- v2: New patch. --- cs-bisection-step | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/cs-bisection-step b/cs-bisection-step index 027032a1..8544bac0 100755 --- a/cs-bisection-step +++ b/cs-bisection-step @@ -1113,10 +1113,15 @@ END or die "$!"; =20 if (eval { + print DEBUG "RUNNING dot -Txdot\n"; + system_checked("dot", "-Txdot", "-o$graphfile.xdot", + "$graphfile.dot"); foreach my $fmt (qw(ps png svg)) { - print DEBUG "RUNNING dot -T$fmt\n"; - system_checked("dot", "-T$fmt", "-o$graphfile.$fmt", - "$graphfile.dot"); + # neato rather than dot, because neato just uses positions + # etc. in the input whereas dot does (re)calculation work. + print DEBUG "RUNNING neato -n2 -T$fmt\n"; + system_checked("neato", "-n2", "-T$fmt", "-o$graphfile.$fmt", + "$graphfile.xdot"); } open SVGI, "$graphfile.svg" or die "$graphfile.svg $!"; open SVGO, ">", "$graphfile.svg.new" or die "$graphfile.svg.new $!"; --=20 2.20.1 From nobody Wed Apr 24 05:37:02 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196875; cv=none; d=zohomail.com; s=zohoarc; b=n78zPi7CuIagcVr4jBI/APIYGSyRSAv0GTLl6YQMlOIkloaqDpGwNZDhXkYG5h1bQFhfY2LUZuTeyWHkqAL4ByzNwMGMqwFpo1tkxfG9XAM6mifsFviq9RvPATcY5jrGO4FPtPGd+0Ob/nRqrYz7zh+ZeEimK5jE7eSlI6oFWQk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196875; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=nzq7uIEWnzIVzmBtSHVhZQXrFC5aTI84uYDmA+BKSCE=; b=HbVfJ0hMbcYMo7PQrQlTlgceOQLgSB4CmGY6EWcFHjMvEaCHEbsfw3liDT4UqcQbbNy1PGsGsltTv0rYq9jxbbFmKSSbYmE+zs4ndApgzWBEhItCDExmVUWfk6koFGLB5Ramy9tI0scvyKw4FPTJCXWLzGU+TqA/lGb71X5eXyw= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196875035481.95077287546246; Fri, 31 Jul 2020 05:01:15 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tia-0001Nh-I8; Fri, 31 Jul 2020 12:00:56 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1TiZ-0001Jk-KC for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 12:00:55 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 7a17f7c8-d325-11ea-8e2b-bc764e2007e4; Fri, 31 Jul 2020 12:00:50 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TN6-0001W4-Jg; Fri, 31 Jul 2020 12:38:44 +0100 X-Inumbo-ID: 7a17f7c8-d325-11ea-8e2b-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 41/41] duration_estimator: Clarify recentflights query a bit Date: Fri, 31 Jul 2020 12:38:20 +0100 Message-Id: <20200731113820.5765-42-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson , George Dunlap Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" The condition on r.job is more naturally thought of as a join condition than a where condition. (This is an inner join, so the semantics are identical.) Also, for clarity, swap the flight and job conditions round, so that the ON clause is a series of r.thing =3D otherthing. No functional change. Signed-off-by: Ian Jackson CC: George Dunlap Reviewed-by: George Dunlap --- v2: New patch. --- Osstest/Executive.pm | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/Osstest/Executive.pm b/Osstest/Executive.pm index 8e4c5b9a..a69c624f 100644 --- a/Osstest/Executive.pm +++ b/Osstest/Executive.pm @@ -1153,10 +1153,10 @@ sub duration_estimator ($$;$$) { FROM flights f JOIN jobs j USING (flight) JOIN runvars r - ON f.flight=3Dr.flight + ON r.flight=3Df.flight + AND r.job=3Dj.job=3D AND r.name=3D? - WHERE j.job=3Dr.job - AND f.blessing=3D? + WHERE f.blessing=3D? AND f.branch=3D? AND j.job=3D? AND r.val=3D? --=20 2.20.1