From nobody Mon Feb 9 03:13:09 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+44935+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+44935+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1565075732; cv=none; d=zoho.com; s=zohoarc; b=a9H8iW1YSZ8CCXbNsX1NxtU3xBvD3c1zYB40fStQkCfYHbJcPJRb/9w5Yhyj6ApW7fHcWHmEFCq9KNf/Ma0F+p1wK3YXl9p5J7hm1AxNj5xS8joK8zpo3Wedz9q2NNoJB2GgQGjnSjsRogrhvUeBgC9ydjPeoAuW+q0dpZ9dM4I= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1565075732; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To:ARC-Authentication-Results; bh=yEVYGswXHYPkRsHha+hs3Y0MNcFy4qTlD8jxZOMSVYg=; b=CzWKrCH8hyTHnUCHiyhaPkG6CY1usmNTFAs0Z5W7VNf4uO+GhQLHKF6MaFKOR2GxjLhsdZZUPRoqLWJZR0c0boFZeIRWvtrkfDkPQRUD3J2Xp/MoPta9gJPhJ8ppqLNymPK5r++sPoZe2H6+D+K5j4u3PmJ7zkXRKAz1iI8jOqk= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+44935+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1565075732494805.3625604020673; Tue, 6 Aug 2019 00:15:32 -0700 (PDT) Return-Path: X-Received: from mga05.intel.com (mga05.intel.com []) by groups.io with SMTP; Tue, 06 Aug 2019 00:15:31 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga004.jf.intel.com ([10.7.209.38]) by fmsmga105.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 06 Aug 2019 00:15:31 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,352,1559545200"; d="scan'208";a="325551500" X-Received: from shwdepsi1121.ccr.corp.intel.com ([10.239.158.47]) by orsmga004.jf.intel.com with ESMTP; 06 Aug 2019 00:15:30 -0700 From: "Bob Feng" To: devel@edk2.groups.io Cc: Liming Gao , Bob Feng Subject: [edk2-devel] [Patch 10/10 V7] BaseTools: Enable block queue log agent. Date: Tue, 6 Aug 2019 15:15:11 +0800 Message-Id: <20190806071511.11836-11-bob.c.feng@intel.com> In-Reply-To: <20190806071511.11836-1-bob.c.feng@intel.com> References: <20190806071511.11836-1-bob.c.feng@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,bob.c.feng@intel.com Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1565075732; bh=B+JA94PoWkJqCMQJMtWYCrvvR3Q+i5hx2crRUUo4acI=; h=Cc:Date:From:Reply-To:Subject:To; b=CAXXC17/ZuFEHvLqgPfOCXUz9y5Worir4ORNGkZvAzyNer/+xw/Aqk0icujY5XnHaeX y1myykZg3n+VnTjCWGI+Dk9Gooc84pTpvDzqJPJkhcukuoSbjyOKPtKDXTSyg1ouCRKat sm63+uDr0qScl+yi7saCSDsF1+UGg6pWFdE= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3D1875 To support Ctrl+S and Ctrl+Q, we enable block queue for log. Cc: Liming Gao Signed-off-by: Bob Feng --- .../Source/Python/AutoGen/AutoGenWorker.py | 6 ++- BaseTools/Source/Python/Common/EdkLogger.py | 10 +++-- BaseTools/Source/Python/build/build.py | 39 ++++++++++--------- 3 files changed, 31 insertions(+), 24 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py b/BaseTools/S= ource/Python/AutoGen/AutoGenWorker.py index 0a3c1d8e0ebd..e583828741bd 100644 --- a/BaseTools/Source/Python/AutoGen/AutoGenWorker.py +++ b/BaseTools/Source/Python/AutoGen/AutoGenWorker.py @@ -21,17 +21,18 @@ try: except: from Queue import Empty import traceback import sys from AutoGen.DataPipe import MemoryDataPipe +import logging + def clearQ(q): try: while True: q.get_nowait() except Empty: pass -import logging =20 class LogAgent(threading.Thread): def __init__(self,log_q,log_level,log_file=3DNone): super(LogAgent,self).__init__() self.log_q =3D log_q @@ -121,13 +122,14 @@ class AutoGenManager(threading.Thread): except Exception: return =20 def clearQueue(self): taskq =3D self.autogen_workers[0].module_queue + logq =3D self.autogen_workers[0].log_q clearQ(taskq) clearQ(self.feedback_q) - + clearQ(logq) def TerminateWorkers(self): self.error_event.set() def kill(self): self.feedback_q.put(None) class AutoGenWorkerInProcess(mp.Process): diff --git a/BaseTools/Source/Python/Common/EdkLogger.py b/BaseTools/Source= /Python/Common/EdkLogger.py index 15fd1458a95a..06da4a9d0a1d 100644 --- a/BaseTools/Source/Python/Common/EdkLogger.py +++ b/BaseTools/Source/Python/Common/EdkLogger.py @@ -93,11 +93,13 @@ except: """ try: self.enqueue(self.prepare(record)) except Exception: self.handleError(record) - +class BlockQueueHandler(QueueHandler): + def enqueue(self, record): + self.queue.put(record,True) ## Log level constants DEBUG_0 =3D 1 DEBUG_1 =3D 2 DEBUG_2 =3D 3 DEBUG_3 =3D 4 @@ -290,23 +292,23 @@ def LogClientInitialize(log_q): # Since we use different format to log different levels of message int= o different # place (stdout or stderr), we have to use different "Logger" objects = to do this. # # For DEBUG level (All DEBUG_0~9 are applicable) _DebugLogger.setLevel(INFO) - _DebugChannel =3D QueueHandler(log_q) + _DebugChannel =3D BlockQueueHandler(log_q) _DebugChannel.setFormatter(_DebugFormatter) _DebugLogger.addHandler(_DebugChannel) =20 # For VERBOSE, INFO, WARN level _InfoLogger.setLevel(INFO) - _InfoChannel =3D QueueHandler(log_q) + _InfoChannel =3D BlockQueueHandler(log_q) _InfoChannel.setFormatter(_InfoFormatter) _InfoLogger.addHandler(_InfoChannel) =20 # For ERROR level _ErrorLogger.setLevel(INFO) - _ErrorCh =3D QueueHandler(log_q) + _ErrorCh =3D BlockQueueHandler(log_q) _ErrorCh.setFormatter(_ErrorFormatter) _ErrorLogger.addHandler(_ErrorCh) =20 ## Set log level # diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Pyth= on/build/build.py index 6b99ed086ba2..e709fe6a1d4d 100644 --- a/BaseTools/Source/Python/build/build.py +++ b/BaseTools/Source/Python/build/build.py @@ -707,11 +707,11 @@ class Build(): self.Fdf =3D BuildOptions.FdfFile self.FdList =3D BuildOptions.RomImage self.FvList =3D BuildOptions.FvImage self.CapList =3D BuildOptions.CapName self.SilentMode =3D BuildOptions.SilentMode - self.ThreadNumber =3D BuildOptions.ThreadNumber + self.ThreadNumber =3D 1 self.SkipAutoGen =3D BuildOptions.SkipAutoGen self.Reparse =3D BuildOptions.Reparse self.SkuId =3D BuildOptions.SkuId if self.SkuId: GlobalData.gSKUID_CMD =3D self.SkuId @@ -881,23 +881,10 @@ class Build(): ToolChainFamily.append(TAB_COMPILER_MSFT) else: ToolChainFamily.append(ToolDefinition[TAB_TOD_DEFINES_FAMI= LY][Tool]) self.ToolChainFamily =3D ToolChainFamily =20 - if self.ThreadNumber is None: - self.ThreadNumber =3D self.TargetTxt.TargetTxtDictionary[TAB_T= AT_DEFINES_MAX_CONCURRENT_THREAD_NUMBER] - if self.ThreadNumber =3D=3D '': - self.ThreadNumber =3D 0 - else: - self.ThreadNumber =3D int(self.ThreadNumber, 0) - - if self.ThreadNumber =3D=3D 0: - try: - self.ThreadNumber =3D multiprocessing.cpu_count() - except (ImportError, NotImplementedError): - self.ThreadNumber =3D 1 - if not self.PlatformFile: PlatformFile =3D self.TargetTxt.TargetTxtDictionary[TAB_TAT_DE= FINES_ACTIVE_PLATFORM] if not PlatformFile: # Try to find one in current directory WorkingDirectory =3D os.getcwd() @@ -911,11 +898,11 @@ class Build(): else: EdkLogger.error("build", RESOURCE_NOT_AVAILABLE, ExtraData=3D"No active platform specif= ied in target.txt or command line! Nothing can be built.\n") =20 self.PlatformFile =3D PathClass(NormFile(PlatformFile, self.Wo= rkspaceDir), self.WorkspaceDir) - + self.ThreadNumber =3D ThreadNum() ## Initialize build configuration # # This method will parse DSC file and merge the configurations from # command line and target.txt, then get the final build configuratio= ns. # @@ -2055,16 +2042,17 @@ class Build(): for m in Pa.GetAllModuleInfo: mqueue.put(m) data_pipe_file =3D os.path.join(Pa.BuildDir, "GlobalVa= r_%s_%s.bin" % (str(Pa.Guid),Pa.Arch)) Pa.DataPipe.dump(data_pipe_file) autogen_rt, errorcode =3D self.StartAutoGen(mqueue, Pa= .DataPipe, self.SkipAutoGen, PcdMaList,self.share_data) - self.Progress.Stop("done!") - self.AutoGenTime +=3D int(round((time.time() - AutoGen= Start))) + if not autogen_rt: self.AutoGenMgr.TerminateWorkers() self.AutoGenMgr.join(0.1) raise FatalError(errorcode) + self.AutoGenTime +=3D int(round((time.time() - AutoGenStar= t))) + self.Progress.Stop("done!") for Arch in Wa.ArchList: MakeStart =3D time.time() for Ma in BuildModules: # Generate build task for the module if not Ma.IsBinaryModule: @@ -2296,27 +2284,42 @@ def LogBuildTime(Time): else: TimeDurStr =3D time.strftime("%H:%M:%S", TimeDur) return TimeDurStr else: return None +def ThreadNum(): + ThreadNumber =3D BuildOption.ThreadNumber + if ThreadNumber is None: + ThreadNumber =3D TargetTxt.TargetTxtDictionary[TAB_TAT_DEFINES_MAX= _CONCURRENT_THREAD_NUMBER] + if ThreadNumber =3D=3D '': + ThreadNumber =3D 0 + else: + ThreadNumber =3D int(ThreadNumber, 0) =20 + if ThreadNumber =3D=3D 0: + try: + ThreadNumber =3D multiprocessing.cpu_count() + except (ImportError, NotImplementedError): + ThreadNumber =3D 1 + return ThreadNumber ## Tool entrance method # # This method mainly dispatch specific methods per the command line option= s. # If no error found, return zero value so the caller of this tool can know # if it's executed successfully or not. # # @retval 0 Tool was successful # @retval 1 Tool failed # +LogQMaxSize =3D ThreadNum() * 10 def Main(): StartTime =3D time.time() =20 # # Create a log Queue # - LogQ =3D mp.Queue() + LogQ =3D mp.Queue(LogQMaxSize) # Initialize log system EdkLogger.LogClientInitialize(LogQ) GlobalData.gCommand =3D sys.argv[1:] # # Parse the options and args --=20 2.20.1.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#44935): https://edk2.groups.io/g/devel/message/44935 Mute This Topic: https://groups.io/mt/32740907/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-