From nobody Tue Apr 30 12:15:19 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+63641+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+63641+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1596420224; cv=none; d=zohomail.com; s=zohoarc; b=RKvKtrtvfdADIGtDCylfJ7MBDVs3DzvbSf54wlQBH8J2RwQa3Qn8AWBfrCMF45ndAcGl2zxYn5ChZK8WyA16Zsd8wp53FvEwVofBV7UOWYBG5185z9g4/b6gP9e5tQjsg7xtvx0Rz8x1LohPklULourpbmLdUOXwPQKATBukTK8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596420224; h=Content-Transfer-Encoding:Cc:Date:From:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Sender:Subject:To; bh=eEHOEKryN9AFjriPBH+guEMiO177bgmZsvZ5w68ILtc=; b=C0G/CZixUaSTZpWWjzCAln9rpepvzt+qm1VK7Hwcp+x74yBn8XImjdu9w1dAaXAjaIli0pHg2lGsXlYhaBc3aKncurGHj8Txn9W3GRyHUZwm6D1sq3beWiCbptWjT9IeA5KyaJBf/6tMRm1QZtam3V6CV4xxk96DJNYddoSFZxQ= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+63641+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1596420224899944.6953805160755; Sun, 2 Aug 2020 19:03:44 -0700 (PDT) Return-Path: X-Received: by 127.0.0.2 with SMTP id LugRYY1788612xQghHkdzxch; Sun, 02 Aug 2020 19:03:44 -0700 X-Received: from mga06.intel.com (mga06.intel.com [134.134.136.31]) by mx.groups.io with SMTP id smtpd.web11.2816.1596420223572388601 for ; Sun, 02 Aug 2020 19:03:43 -0700 IronPort-SDR: 76GwbL73+51deQDD3F40f0Ng46GAn4lJwJh9r+/FyKs5j7ECjPfcitbVDkDaYP7n0eqOaczaBS HUGMMHiZvAcg== X-IronPort-AV: E=McAfee;i="6000,8403,9701"; a="213567613" X-IronPort-AV: E=Sophos;i="5.75,428,1589266800"; d="scan'208";a="213567613" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Aug 2020 19:03:42 -0700 IronPort-SDR: cuTLmzBRY0Yfw1dR7RYyIRc8GfiJ+I9/ju+bmOykAP5lJw70edDbntomYsD1rUsMqeFSNJmO5x 4Gu80uo9gPhA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.75,428,1589266800"; d="scan'208";a="491657468" X-Received: from shwdepsi1121.ccr.corp.intel.com ([10.239.158.83]) by fmsmga006.fm.intel.com with ESMTP; 02 Aug 2020 19:03:40 -0700 From: "Bob Feng" To: devel@edk2.groups.io Cc: "Feng, Bob C" , Liming Gao , Yuwei Chen , Lucy Yan Subject: [edk2-devel] [Patch] BaseTools: Improve the method of checking queue empty Date: Mon, 3 Aug 2020 10:03:38 +0800 Message-Id: <20200803020338.24712-1-bob.c.feng@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,bob.c.feng@intel.com X-Gm-Message-State: Do2QOK0E5egLpwR9ZxrQgQH6x1787277AA= Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1596420224; bh=l1McjPUcY7dV7MNRyfVw3d0llRgRsEL4IkCeGR42QHg=; h=Cc:Date:From:Reply-To:Subject:To; b=nhCFzBt5Rdfrwifs+XVb6O5l11FRXGYzFot3rQU47mxJm+zpycjY2oQuex6rGFYUEKl 1CRL2AcNIh1/6YCI2ASXUG8dkasNlhDKNH6TeqSdrOSqAwHiGLoaxT9T/PY7K12MpwWb0 Atima9Ay6LcCSorCx/tfXJhrLQ0jTTGkYmM= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" From: "Feng, Bob C" Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3D2807 The Queue.empty() method is not reliable in the multiple process runtime environment. This patch uses a new method to check if all modules are processed and workers need to be stopped. That is to add a None item at the bottom of the queue. Worker check if it gets that None item to know if all the module is processed. Signed-off-by: Bob Feng Cc: Liming Gao Cc: Yuwei Chen Cc: Lucy Yan Reviewed-by: Liming Gao --- .../Source/Python/AutoGen/AutoGenWorker.py | 26 ++++++++++++++----- BaseTools/Source/Python/build/build.py | 3 ++- 2 files changed, 22 insertions(+), 7 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py b/BaseTools/S= ource/Python/AutoGen/AutoGenWorker.py index 563d91b421..017f676399 100755 --- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py +++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py @@ -22,10 +22,11 @@ except: from Queue import Empty import traceback import sys from AutoGen.DataPipe import MemoryDataPipe import logging +import time =20 def clearQ(q): try: while True: q.get_nowait() @@ -109,11 +110,15 @@ class AutoGenManager(threading.Thread): badnews =3D self.feedback_q.get() if badnews is None: break if badnews =3D=3D "Done": fin_num +=3D 1 + elif badnews =3D=3D "QueueEmpty": + EdkLogger.debug(EdkLogger.DEBUG_9, "Worker %s: %s" % (= os.getpid(), badnews)) + self.TerminateWorkers() else: + EdkLogger.debug(EdkLogger.DEBUG_9, "Worker %s: %s" % (= os.getpid(), badnews)) self.Status =3D False self.TerminateWorkers() if fin_num =3D=3D len(self.autogen_workers): self.clearQueue() for w in self.autogen_workers: @@ -225,16 +230,25 @@ class AutoGenWorkerInProcess(mp.Process): FfsCmd =3D {} GlobalData.FfsCmd =3D FfsCmd PlatformMetaFile =3D self.GetPlatformMetaFile(self.data_pipe.G= et("P_Info").get("ActivePlatform"), self.data_pipe.Get("P_Info").= get("WorkspaceDir")) while True: - if self.module_queue.empty(): - break if self.error_event.is_set(): break module_count +=3D 1 - module_file,module_root,module_path,module_basename,module= _originalpath,module_arch,IsLib =3D self.module_queue.get_nowait() + try: + module_file,module_root,module_path,module_basename,mo= dule_originalpath,module_arch,IsLib =3D self.module_queue.get_nowait() + except Empty: + EdkLogger.debug(EdkLogger.DEBUG_9, "Worker %s: %s" % (= os.getpid(), "Fake Empty.")) + time.sleep(0.01) + continue + if module_file is None: + EdkLogger.debug(EdkLogger.DEBUG_9, "Worker %s: %s" % (= os.getpid(), "Worker get the last item in the queue.")) + self.feedback_q.put("QueueEmpty") + time.sleep(0.01) + continue + modulefullpath =3D os.path.join(module_root,module_file) taskname =3D " : ".join((modulefullpath,module_arch)) module_metafile =3D PathClass(module_file,module_root) if module_path: module_metafile.Path =3D module_path @@ -278,15 +292,15 @@ class AutoGenWorkerInProcess(mp.Process): self.cache_q.put((Ma.MetaFile.Path, Ma.Arch, "Make= Cache", True)) continue else: self.cache_q.put((Ma.MetaFile.Path, Ma.Arch, "Make= Cache", False)) =20 - except Empty: - pass - except: + except Exception as e: + EdkLogger.debug(EdkLogger.DEBUG_9, "Worker %s: %s" % (os.getpi= d(), str(e))) self.feedback_q.put(taskname) finally: + EdkLogger.debug(EdkLogger.DEBUG_9, "Worker %s: %s" % (os.getpi= d(), "Done")) self.feedback_q.put("Done") self.cache_q.put("CacheDone") =20 def printStatus(self): print("Processs ID: %d Run %d modules in AutoGen " % (os.getpid(),= len(AutoGen.Cache()))) diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Pyth= on/build/build.py index 1ab1e60a64..59ceacfed0 100755 --- a/BaseTools/Source/Python/build/build.py +++ b/BaseTools/Source/Python/build/build.py @@ -1215,11 +1215,11 @@ class Build(): if Target not in ['clean', 'cleanlib', 'cleanall', 'run', 'fds']: # for target which must generate AutoGen code and makefile mqueue =3D mp.Queue() for m in AutoGenObject.GetAllModuleInfo: mqueue.put(m) - + mqueue.put((None,None,None,None,None,None,None)) AutoGenObject.DataPipe.DataContainer =3D {"CommandTarget": sel= f.Target} AutoGenObject.DataPipe.DataContainer =3D {"Workspace_timestamp= ": AutoGenObject.Workspace._SrcTimeStamp} AutoGenObject.CreateLibModuelDirs() AutoGenObject.DataPipe.DataContainer =3D {"LibraryBuildDirecto= ryList":AutoGenObject.LibraryBuildDirectoryList} AutoGenObject.DataPipe.DataContainer =3D {"ModuleBuildDirector= yList":AutoGenObject.ModuleBuildDirectoryList} @@ -2172,10 +2172,11 @@ class Build(): ToolChain, Arch, self.PlatformFile,Pa.Da= taPipe) self.AllModules.add(Ma) data_pipe_file =3D os.path.join(Pa.BuildDir, "GlobalVar_%s_%s.= bin" % (str(Pa.Guid),Pa.Arch)) Pa.DataPipe.dump(data_pipe_file) =20 + mqueue.put((None,None,None,None,None,None,None)) autogen_rt, errorcode =3D self.StartAutoGen(mqueue, Pa.DataPip= e, self.SkipAutoGen, PcdMaList, cqueue) =20 if not autogen_rt: self.AutoGenMgr.TerminateWorkers() self.AutoGenMgr.join(1) --=20 2.20.1.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#63641): https://edk2.groups.io/g/devel/message/63641 Mute This Topic: https://groups.io/mt/75958155/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-