From nobody Tue Nov 26 01:36:12 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+63271+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+63271+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1595776871; cv=none; d=zohomail.com; s=zohoarc; b=Vnz0WkbHa5rlCGYKo7vPnJyW8OH1CGOYmrgn/3is1vRDja7DtSVPsm62ICW5gjolM6JlY4zkrY2W1Jg2T6nDELj8E0WbjkJTKvuyR43ekEHraYDrWoWivCpkjLKJHtTT+hd+c33oB7KFoKbHRlPU/1eeIR2Yt8GhPupVMaH7BFA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1595776871; h=Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:Message-ID:Reply-To:References:Sender:Subject:To; bh=tY4IZRbnetyD9UuyI0+YSF3pQ+XMEbTnF4mF0BwwPM8=; b=UR0g9LMuMnQsSn+jTNPW53Osu8Ua9frVTocMRcQHxZqepOfzlhQj6a2zD+NzXk58uakTpfg24BKQWOasECwSjoqD2M/ChmD9BrgZVx2cAasD7RF6ST9tYfK0KxdEJAhGH2AYq8lcx5RQ1xo6dCbJojU62i4jRFhWwkzBTwN4zX8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+63271+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1595776871694519.2324508082455; Sun, 26 Jul 2020 08:21:11 -0700 (PDT) Return-Path: X-Received: by 127.0.0.2 with SMTP id JXqCYY1788612x1Gvtm69tb4; Sun, 26 Jul 2020 08:21:11 -0700 X-Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by mx.groups.io with SMTP id smtpd.web11.30647.1595776870639915608 for ; Sun, 26 Jul 2020 08:21:10 -0700 IronPort-SDR: kAOQK18wn5jt7sQQOCKSQ8sV+I1zQSy/aWXqOAwNmt9IEYbQX8mw+DCQFGpS/iErEcCcZmBkaE F6NP0h4iLyvA== X-IronPort-AV: E=McAfee;i="6000,8403,9694"; a="138414958" X-IronPort-AV: E=Sophos;i="5.75,399,1589266800"; d="scan'208";a="138414958" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Jul 2020 08:21:10 -0700 IronPort-SDR: B/K7L4HLqmsMv934EgBJAUQDLTo6xwm93EvsS8TCCKYpqMudYN0LhIywqE7SHQGGbGWMsKYT1B NFipETQwl1nA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.75,399,1589266800"; d="scan'208";a="273045246" X-Received: from shenglei-dev.ccr.corp.intel.com ([10.239.154.36]) by fmsmga008.fm.intel.com with ESMTP; 26 Jul 2020 08:21:06 -0700 From: "Zhang, Shenglei" To: devel@edk2.groups.io Cc: Sean Brogan , Bret Barkelew , Michael D Kinney , Liming Gao Subject: [edk2-devel] [PATCH v8 02/16] .pytool/Plugin: Add a plugin EccCheck Date: Sun, 26 Jul 2020 23:20:46 +0800 Message-Id: <20200726152046.95664-3-shenglei.zhang@intel.com> In-Reply-To: <20200726152046.95664-1-shenglei.zhang@intel.com> References: <20200726152046.95664-1-shenglei.zhang@intel.com> Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,shenglei.zhang@intel.com X-Gm-Message-State: YuyttbXEMAy5k5doXhF7KBGbx1787277AA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1595776871; bh=IolN1eONpT4Karq1QdYAdtP1JsryavUY18ANUlB7ShM=; h=Cc:Date:From:Reply-To:Subject:To; b=bqrPxIxjrTUdQZtjINdNGcdN0m7sQJY/5AhZ9PuEwgVIBLWmW53TxQZ+b4bWCu9jlsJ sA3svzGA/JVxILqw6/T/Xu2vP/FdCjUUYW7y+/TytRv5MPSrDXJ5MihDqibiMw9A+iJsr XpopScsJ+ZpIWA7D/LJ/kdUvCAqXkid5Eas= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" REF: https://bugzilla.tianocore.org/show_bug.cgi?id=3D2606 EccCheck is a plugin to report Ecc issues for code in pull request , which will be run on open ci. But note not each kind of issue could be reported out. It can only handle the issues, whose line number in CSV report accurately map with their code in source code files. And Ecc issues about comments can also be handled. Cc: Sean Brogan Cc: Bret Barkelew Cc: Michael D Kinney Cc: Liming Gao Signed-off-by: Shenglei Zhang --- .pytool/Plugin/EccCheck/EccCheck.py | 291 ++++++++++++++++++ .pytool/Plugin/EccCheck/EccCheck_plug_in.yaml | 11 + .pytool/Plugin/EccCheck/Readme.md | 15 + 3 files changed, 317 insertions(+) create mode 100644 .pytool/Plugin/EccCheck/EccCheck.py create mode 100644 .pytool/Plugin/EccCheck/EccCheck_plug_in.yaml create mode 100644 .pytool/Plugin/EccCheck/Readme.md diff --git a/.pytool/Plugin/EccCheck/EccCheck.py b/.pytool/Plugin/EccCheck/= EccCheck.py new file mode 100644 index 000000000000..f23b85bc5a2a --- /dev/null +++ b/.pytool/Plugin/EccCheck/EccCheck.py @@ -0,0 +1,291 @@ +# @file EccCheck.py +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +import os +import re +import csv +import xml.dom.minidom +from typing import List, Dict, Tuple +import logging +from io import StringIO +from edk2toolext.environment import shell_environment +from edk2toolext.environment.plugintypes.ci_build_plugin import ICiBuildPl= ugin +from edk2toolext.environment.var_dict import VarDict +from edk2toollib.utility_functions import RunCmd + + +class EccCheck(ICiBuildPlugin): + """ + A CiBuildPlugin that finds the Ecc issues of newly added code in pull = request. + + Configuration options: + "EccCheck": { + "ExceptionList": [], + "IgnoreFiles": [] + }, + """ + + ReModifyFile =3D re.compile(r'[B-Q,S-Z]+[\d]*\t(.*)') + FindModifyFile =3D re.compile(r'\+\+\+ b\/(.*)') + LineScopePattern =3D (r'@@ -\d*\,*\d* \+\d*\,*\d* @@.*') + LineNumRange =3D re.compile(r'@@ -\d*\,*\d* \+(\d*)\,*(\d*) @@.*') + + def GetTestName(self, packagename: str, environment: VarDict) -> tuple: + """ Provide the testcase name and classname for use in reporting + testclassname: a descriptive string for the testcase can inclu= de whitespace + classname: should be patterned .. + + Args: + packagename: string containing name of package to build + environment: The VarDict for the test to run in + Returns: + a tuple containing the testcase name and the classname + (testcasename, classname) + """ + return ("Check for efi coding style for " + packagename, packagena= me + ".EccCheck") + + ## + # External function of plugin. This function is used to perform the t= ask of the ci_build_plugin Plugin + # + # - package is the edk2 path to package. This means workspace/packa= gepath relative. + # - edk2path object configured with workspace and packages path + # - PkgConfig Object (dict) for the pkg + # - EnvConfig Object + # - Plugin Manager Instance + # - Plugin Helper Obj Instance + # - Junit Logger + # - output_stream the StringIO output stream from this plugin via lo= gging + def RunBuildPlugin(self, packagename, Edk2pathObj, pkgconfig, environm= ent, PLM, PLMHelper, tc, output_stream=3DNone): + edk2_path =3D Edk2pathObj.WorkspacePath + python_path =3D os.path.join(edk2_path, "BaseTools", "Source", "Py= thon") + env =3D shell_environment.GetEnvironment() + env.set_shell_var('PYTHONPATH', python_path) + env.set_shell_var('WORKSPACE', edk2_path) + self.ECC_PASS =3D True + self.ApplyConfig(pkgconfig, edk2_path, packagename) + modify_dir_list =3D self.GetModifyDir(packagename) + patch =3D self.GetDiff(packagename) + ecc_diff_range =3D self.GetDiffRange(patch, packagename, edk2_path) + self.GenerateEccReport(modify_dir_list, ecc_diff_range, edk2_path) + ecc_log =3D os.path.join(edk2_path, "Ecc.log") + if self.ECC_PASS: + tc.SetSuccess() + self.RemoveFile(ecc_log) + return 0 + else: + with open(ecc_log, encoding=3D'utf8') as output: + ecc_output =3D output.readlines() + for line in ecc_output: + logging.error(line) + self.RemoveFile(ecc_log) + tc.SetFailed("EccCheck failed for {0}".format(packagename), "E= cc detected issues") + return 1 + + def GetDiff(self, pkg: str) -> List[str]: + return_buffer =3D StringIO() + params =3D "diff --unified=3D0 origin/master HEAD" + RunCmd("git", params, outstream=3Dreturn_buffer) + p =3D return_buffer.getvalue().strip() + patch =3D p.split("\n") + return_buffer.close() + + return patch + + def RemoveFile(self, file: str) -> None: + if os.path.exists(file): + os.remove(file) + return + + def GetModifyDir(self, pkg: str) -> List[str]: + return_buffer =3D StringIO() + params =3D "diff --name-status" + ' HEAD' + ' origin/master' + RunCmd("git", params, outstream=3Dreturn_buffer) + p1 =3D return_buffer.getvalue().strip() + dir_list =3D p1.split("\n") + return_buffer.close() + modify_dir_list =3D [] + for modify_dir in dir_list: + file_path =3D self.ReModifyFile.findall(modify_dir) + if file_path: + file_dir =3D os.path.dirname(file_path[0]) + else: + continue + if pkg in file_dir and file_dir !=3D pkg: + modify_dir_list.append('%s' % file_dir) + else: + continue + + modify_dir_list =3D list(set(modify_dir_list)) + return modify_dir_list + + def GetDiffRange(self, patch_diff: List[str], pkg: str, workingdir: st= r) -> Dict[str, List[Tuple[int, int]]]: + IsDelete =3D True + StartCheck =3D False + range_directory: Dict[str, List[Tuple[int, int]]] =3D {} + for line in patch_diff: + modify_file =3D self.FindModifyFile.findall(line) + if modify_file and pkg in modify_file[0] and not StartCheck an= d os.path.isfile(modify_file[0]): + modify_file_comment_dic =3D self.GetCommentRange(modify_fi= le[0], workingdir) + IsDelete =3D False + StartCheck =3D True + modify_file_dic =3D modify_file[0] + modify_file_dic =3D modify_file_dic.replace("/", os.sep) + range_directory[modify_file_dic] =3D [] + elif line.startswith('--- '): + StartCheck =3D False + elif re.match(self.LineScopePattern, line, re.I) and not IsDel= ete and StartCheck: + start_line =3D self.LineNumRange.search(line).group(1) + line_range =3D self.LineNumRange.search(line).group(2) + if not line_range: + line_range =3D '1' + range_directory[modify_file_dic].append((int(start_line), = int(start_line) + int(line_range) - 1)) + for i in modify_file_comment_dic: + if int(i[0]) <=3D int(start_line) <=3D int(i[1]): + range_directory[modify_file_dic].append(i) + return range_directory + + def GetCommentRange(self, modify_file: str, workingdir: str) -> List[T= uple[int, int]]: + modify_file_path =3D os.path.join(workingdir, modify_file) + with open(modify_file_path) as f: + line_no =3D 1 + comment_range: List[Tuple[int, int]] =3D [] + Start =3D False + for line in f: + if line.startswith('/**'): + start_no =3D line_no + Start =3D True + if line.startswith('**/') and Start: + end_no =3D line_no + Start =3D False + comment_range.append((int(start_no), int(end_no))) + line_no +=3D 1 + + if comment_range and comment_range[0][0] =3D=3D 1: + del comment_range[0] + return comment_range + + def GenerateEccReport(self, modify_dir_list: List[str], ecc_diff_range= : Dict[str, List[Tuple[int, int]]], + edk2_path: str) -> None: + ecc_need =3D False + ecc_run =3D True + config =3D os.path.join(edk2_path, "BaseTools", "Source", "Python"= , "Ecc", "config.ini") + exception =3D os.path.join(edk2_path, "BaseTools", "Source", "Pyth= on", "Ecc", "exception.xml") + report =3D os.path.join(edk2_path, "Ecc.csv") + for modify_dir in modify_dir_list: + target =3D os.path.join(edk2_path, modify_dir) + logging.info('Run ECC tool for the commit in %s' % modify_dir) + ecc_need =3D True + ecc_params =3D "-c {0} -e {1} -t {2} -r {3}".format(config, ex= ception, target, report) + return_code =3D RunCmd("Ecc", ecc_params, workingdir=3Dedk2_pa= th) + if return_code !=3D 0: + ecc_run =3D False + break + if not ecc_run: + logging.error('Fail to run ECC tool') + self.ParseEccReport(ecc_diff_range, edk2_path) + + if not ecc_need: + logging.info("Doesn't need run ECC check") + + revert_params =3D "checkout -- {}".format(exception) + RunCmd("git", revert_params) + return + + def ParseEccReport(self, ecc_diff_range: Dict[str, List[Tuple[int, int= ]]], edk2_path: str) -> None: + ecc_log =3D os.path.join(edk2_path, "Ecc.log") + ecc_csv =3D "Ecc.csv" + file =3D os.listdir(edk2_path) + row_lines =3D [] + ignore_error_code =3D self.GetIgnoreErrorCode() + if ecc_csv in file: + with open(ecc_csv) as csv_file: + reader =3D csv.reader(csv_file) + for row in reader: + for modify_file in ecc_diff_range: + if modify_file in row[3]: + for i in ecc_diff_range[modify_file]: + line_no =3D int(row[4]) + if i[0] <=3D line_no <=3D i[1] and row[1] = not in ignore_error_code: + row[0] =3D '\nEFI coding style error' + row[1] =3D 'Error code: ' + row[1] + row[3] =3D 'file: ' + row[3] + row[4] =3D 'Line number: ' + row[4] + row_line =3D '\n *'.join(row) + row_lines.append(row_line) + break + break + if row_lines: + self.ECC_PASS =3D False + + with open(ecc_log, 'a') as log: + all_line =3D '\n'.join(row_lines) + all_line =3D all_line + '\n' + log.writelines(all_line) + return + + def ApplyConfig(self, pkgconfig: Dict[str, List[str]], edk2_path: str,= pkg: str) -> None: + if "IgnoreFiles" in pkgconfig: + for a in pkgconfig["IgnoreFiles"]: + a =3D os.path.join(edk2_path, pkg, a) + a =3D a.replace(os.sep, "/") + try: + logging.info("Ignoring File {0}".format(a)) + self.RemoveFile(a) + except Exception: + logging.info("EccCheck.IgnoreInf -> {0} not found in f= ilesystem. Invalid ignore file".format(a)) + if "ExceptionList" in pkgconfig: + exception_list =3D pkgconfig["ExceptionList"] + exception_xml =3D os.path.join(edk2_path, "BaseTools", "Source= ", "Python", "Ecc", "exception.xml") + try: + logging.info("Appending exceptions") + self.AppendException(exception_list, exception_xml) + except Exception as e: + logging.error("Fail to apply exceptions") + raise e + return + + def AppendException(self, exception_list: List[str], exception_xml: st= r) -> None: + error_code_list =3D exception_list[::2] + keyword_list =3D exception_list[1::2] + dom_tree =3D xml.dom.minidom.parse(exception_xml) + root_node =3D dom_tree.documentElement + for error_code, keyword in zip(error_code_list, keyword_list): + customer_node =3D dom_tree.createElement("Exception") + keyword_node =3D dom_tree.createElement("KeyWord") + keyword_node_text_value =3D dom_tree.createTextNode(keyword) + keyword_node.appendChild(keyword_node_text_value) + customer_node.appendChild(keyword_node) + error_code_node =3D dom_tree.createElement("ErrorID") + error_code_text_value =3D dom_tree.createTextNode(error_code) + error_code_node.appendChild(error_code_text_value) + customer_node.appendChild(error_code_node) + root_node.appendChild(customer_node) + with open(exception_xml, 'w') as f: + dom_tree.writexml(f, indent=3D'', addindent=3D'', newl=3D'\n',= encoding=3D'UTF-8') + return + + def GetIgnoreErrorCode(self) -> set: + ignore_error_code =3D { + "10000", + "10001", + "10002", + "10003", + "10004", + "10005", + "10006", + "10007", + "10008", + "10009", + "10010", + "10011", + "10012", + "10013", + "10015", + "10016", + "10017", + "10022", + } + return ignore_error_code diff --git a/.pytool/Plugin/EccCheck/EccCheck_plug_in.yaml b/.pytool/Plugin= /EccCheck/EccCheck_plug_in.yaml new file mode 100644 index 000000000000..0d121ecb1861 --- /dev/null +++ b/.pytool/Plugin/EccCheck/EccCheck_plug_in.yaml @@ -0,0 +1,11 @@ +## @file +# CiBuildPlugin used to check Ecc issues +# +# Copyright (c) 2020, Intel Corporation. All rights reserved.
+# SPDX-License-Identifier: BSD-2-Clause-Patent +## +{ + "scope": "cibuild", + "name": "EccCheck Test", + "module": "EccCheck" +} diff --git a/.pytool/Plugin/EccCheck/Readme.md b/.pytool/Plugin/EccCheck/Re= adme.md new file mode 100644 index 000000000000..85c439a5edf1 --- /dev/null +++ b/.pytool/Plugin/EccCheck/Readme.md @@ -0,0 +1,15 @@ +# EFI Coding style Check Plugin + +This CiBuildPlugin finds the Ecc issues of newly added code in pull reques= t. + +## Configuration + +The plugin can be configured to ignore certain files and issues. + +"EccCheck": { + "ExceptionList": [], + "IgnoreFiles": [] + }, + """ + +OPTIONAL List of file to ignore. --=20 2.18.0.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#63271): https://edk2.groups.io/g/devel/message/63271 Mute This Topic: https://groups.io/mt/75804107/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-