From nobody Mon Feb 9 16:18:27 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail(p=none dis=none) header.from=eu.citrix.com ARC-Seal: i=1; a=rsa-sha256; t=1596196572; cv=none; d=zohomail.com; s=zohoarc; b=jm0hkiqShS9suuPUlP2P1C2EyKcQrHjiER4bzejZPnPuKn9eU64+dcwai+70/9nDl3vdvj8lva7rJ6TkAHOQDigNSqju6jvXAqpS80GsULj6RKCLHufItEUujjZ6Mrx1LKr/60YHGtaVr2MeVjuz0CRoxsGUxiP9D5tZXpayXU8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596196572; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:To; bh=BnWhPoCAJ9Bpc8rx0pGnp+T7GMpVrNVjUZQwqmH3wz8=; b=OGFGgrusgoB+6QHvhYvI0z7PovMyJMdboFbXfYVer9ZSKF3jo/C3jynyvoCj35PGtHaxDDtGszkB7ulOYKxqzV3nWOjGjHvBzMyp7QuDzZEj1Wv8L1nOGmD7Zfy+bIm4+f0P90jPKr4f8vkKcvuU4Wbyf2UKSLWY19TYz5Sl2s8= ARC-Authentication-Results: i=1; mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org; dmarc=fail header.from= (p=none dis=none) header.from= Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1596196572536716.3374235646306; Fri, 31 Jul 2020 04:56:12 -0700 (PDT) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tdn-0008RS-0O; Fri, 31 Jul 2020 11:55:59 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1k1Tdl-0008RD-O1 for xen-devel@lists.xenproject.org; Fri, 31 Jul 2020 11:55:57 +0000 Received: from chiark.greenend.org.uk (unknown [2001:ba8:1e3::]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id caff2b44-d324-11ea-8e29-bc764e2007e4; Fri, 31 Jul 2020 11:55:57 +0000 (UTC) Received: from [172.18.45.5] (helo=zealot.relativity.greenend.org.uk) by chiark.greenend.org.uk (Debian Exim 4.84_2 #1) with esmtp (return-path ijackson@chiark.greenend.org.uk) id 1k1TN1-0001W4-JE; Fri, 31 Jul 2020 12:38:39 +0100 X-Inumbo-ID: caff2b44-d324-11ea-8e29-bc764e2007e4 From: Ian Jackson To: xen-devel@lists.xenproject.org Subject: [OSSTEST PATCH v2 30/41] sg-report-host-history: Fork Date: Fri, 31 Jul 2020 12:38:09 +0100 Message-Id: <20200731113820.5765-31-ian.jackson@eu.citrix.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200731113820.5765-1-ian.jackson@eu.citrix.com> References: <20200731113820.5765-1-ian.jackson@eu.citrix.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Ian Jackson Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" Content-Type: text/plain; charset="utf-8" Run each host's report in a separate child. This is considerably faster. Signed-off-by: Ian Jackson --- sg-report-host-history | 47 +++++++++++++++++++++++++++++++++++------- 1 file changed, 40 insertions(+), 7 deletions(-) diff --git a/sg-report-host-history b/sg-report-host-history index f4352fc3..dc694ebe 100755 --- a/sg-report-host-history +++ b/sg-report-host-history @@ -34,6 +34,7 @@ our $flightlimit; our $htmlout =3D "."; our $read_existing=3D1; our $doinstall=3D1; +our $maxjobs=3D10; our @blessings; =20 open DEBUG, ">/dev/null"; @@ -44,7 +45,7 @@ csreadconfig(); while (@ARGV && $ARGV[0] =3D~ m/^-/) { $_=3D shift @ARGV; last if m/^--?$/; - if (m/^--(limit)\=3D([1-9]\d*)$/) { + if (m/^--(limit|maxjobs)\=3D([1-9]\d*)$/) { $$1=3D $2; } elsif (m/^--time-limit\=3D([1-9]\d*)$/) { $timelimit=3D $1; @@ -469,12 +470,44 @@ db_retry($dbh_tests, [], sub { computeflightsrange(); }); =20 +undef $dbh_tests; + +our %children; +our $worst =3D 0; + +sub wait_for_max_children ($) { + my ($lim) =3D @_; + while (keys(%children) > $lim) { + $!=3D0; $?=3D0; my $got =3D wait; + die "$! $got $?" unless exists $children{$got}; + my $host =3D $children{$got}; + delete $children{$got}; + $worst =3D $? if $? > $worst; + if ($?) { + print STDERR "sg-report-flight[: [$got] failed for $host: $?\n"; + } else { + print DEBUG "REAPED [$got] $host\n"; + } + } +} + foreach my $host (sort keys %hosts) { - read_existing_logs($host); - db_retry($dbh_tests, [], sub { - mainquery($host); - reporthost $host; - }); + wait_for_max_children($maxjobs); + + my $pid =3D fork // die $!; + if (!$pid) { + opendb_tests(); + read_existing_logs($host); + db_retry($dbh_tests, [], sub { + mainquery($host); + reporthost $host; + }); + print DEBUG "JQ CACHE ".($jqtotal-$jqcachemisses)." / $jqtotal\n"; + exit(0); + } + print DEBUG "SPAWNED [$pid] $host\n"; + $children{$pid} =3D $host; } =20 -print DEBUG "JQ CACHE ".($jqtotal-$jqcachemisses)." / $jqtotal\n"; +wait_for_max_children(0); +exit $worst; --=20 2.20.1