From nobody Mon Feb 9 14:03:00 2026 Delivered-To: importer@patchew.org Received-SPF: pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+49603+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+49603+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1572378930; cv=none; d=zoho.com; s=zohoarc; b=DOZ/mEAdYOct9b/uXhYbNHTQZ04t++tcGcn8PzrXb3Rc35rVqXOhW4gpba/VZdpeAN0+ud9A9b2NVuUxEbYsGI92IPc5wcJaAmkGD3VrqjUzrkhkB4kiXAI/CftWt+3U89Q3Dk9e1wIrxk3/tc3HtVapKaa4kMxKiFb9VJVcxIs= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zoho.com; s=zohoarc; t=1572378930; h=Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:References:Sender:Subject:To; bh=vDJjfB6CZK9Dg3zLP7AYOSRQ6pSODHifV5VD3UX0JTU=; b=DYt2hxnw2vXLNdwcs0RX7ToHRN6xTch10/okkr+s5UxG9TeRPt1eZeZy7AqY+ojRy3Hsb1hMriysMRLTMDZ+AEZ3ZDkR+61EfLTnJHtYB5GfdQvM1AOLzzYtBT7nIfqdxfdNxhG0FxnNiMXmzapTsSTtfPIEkd8cmAl3/dJliIg= ARC-Authentication-Results: i=1; mx.zoho.com; dkim=pass; spf=pass (zoho.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+49603+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1572378930942605.610786269889; Tue, 29 Oct 2019 12:55:30 -0700 (PDT) Return-Path: X-Received: by 127.0.0.2 with SMTP id BLaYYY1788612x3Nsytz9nWA; Tue, 29 Oct 2019 12:55:29 -0700 X-Received: from mga12.intel.com (mga12.intel.com []) by mx.groups.io with SMTP id smtpd.web12.707.1572378927372028712 for ; Tue, 29 Oct 2019 12:55:29 -0700 X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 29 Oct 2019 12:55:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.68,245,1569308400"; d="scan'208";a="211170261" X-Received: from mdkinney-mobl2.amr.corp.intel.com ([10.251.0.39]) by fmsmga001.fm.intel.com with ESMTP; 29 Oct 2019 12:55:28 -0700 From: "Michael D Kinney" To: devel@edk2.groups.io Cc: Sean Brogan , Bret Barkelew , Liming Gao Subject: [edk2-devel] [Patch v3 19/22] .pytool: Add CISettings.py and Readme.md Date: Tue, 29 Oct 2019 12:55:14 -0700 Message-Id: <20191029195517.20028-20-michael.d.kinney@intel.com> In-Reply-To: <20191029195517.20028-1-michael.d.kinney@intel.com> References: <20191029195517.20028-1-michael.d.kinney@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,michael.d.kinney@intel.com X-Gm-Message-State: haxCvxPOIiH2u01jM0S2Crc7x1787277AA= Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1572378929; bh=1JTw95wf9nxTrB3XTTBG7nKrfn3NFoI1I2pkLV2qtFQ=; h=Cc:Date:From:Reply-To:Subject:To; b=p2WvgMClXUP0DdvChXaC58v+J+yIl2cv0dtqDRQ0wtfEcyQSLcsIVhbasAzbUHzHroP Z/3oglyURk6iSz0jq/6iXTS9O+eYG+ks+C7EVxP34lgfovJgyhyeryKbFkknBzd95zGu0 +nMTVqjein1zCAUVYwjpjJ/DjX9IVUlcmc0= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" From: Sean Brogan https://bugzilla.tianocore.org/show_bug.cgi?id=3D2315 Add main python script for EDK II Continuous Integration (CI) builds along with a Readme.md that provides a summary of the packages, platforms, and checks performs during a CI build. Cc: Sean Brogan Cc: Bret Barkelew Cc: Liming Gao Signed-off-by: Michael D Kinney --- .pytool/CISettings.py | 173 ++++++++++++++++++++++++++++++++++++++++++ .pytool/Readme.md | 128 +++++++++++++++++++++++++++++++ 2 files changed, 301 insertions(+) create mode 100644 .pytool/CISettings.py create mode 100644 .pytool/Readme.md diff --git a/.pytool/CISettings.py b/.pytool/CISettings.py new file mode 100644 index 0000000000..a78e8b974c --- /dev/null +++ b/.pytool/CISettings.py @@ -0,0 +1,173 @@ +# @file +# +# Copyright (c) 2018, Microsoft Corporation +# SPDX-License-Identifier: BSD-2-Clause-Patent +## +import os +import logging +from edk2toolext.environment import shell_environment +from edk2toolext.invocables.edk2_ci_build import CiBuildSettingsManager +from edk2toolext.invocables.edk2_setup import SetupSettingsManager, Requir= edSubmodule +from edk2toolext.invocables.edk2_update import UpdateSettingsManager +from edk2toolext.invocables.edk2_pr_eval import PrEvalSettingsManager +from edk2toollib.utility_functions import GetHostInfo + + +class Settings(CiBuildSettingsManager, UpdateSettingsManager, SetupSetting= sManager, PrEvalSettingsManager): + + def __init__(self): + self.ActualPackages =3D [] + self.ActualTargets =3D [] + self.ActualArchitectures =3D [] + self.ActualToolChainTag =3D "" + + # ####################################################################= ################### # + # Extra CmdLine configuration = # + # ####################################################################= ################### # + + def AddCommandLineOptions(self, parserObj): + pass + + def RetrieveCommandLineOptions(self, args): + pass + + # ####################################################################= ################### # + # Default Support for this Ci Build = # + # ####################################################################= ################### # + + def GetPackagesSupported(self): + ''' return iterable of edk2 packages supported by this build. + These should be edk2 workspace relative paths ''' + + return ("MdePkg", + "MdeModulePkg", + "NetworkPkg", + "PcAtChipsetPkg", + "SecurityPkg", + "UefiCpuPkg", + "FmpDevicePkg", + "ShellPkg", + "FatPkg", + "CryptoPkg" + ) + + def GetArchitecturesSupported(self): + ''' return iterable of edk2 architectures supported by this build = ''' + return ("IA32", + "X64", + "ARM", + "AARCH64") + + def GetTargetsSupported(self): + ''' return iterable of edk2 target tags supported by this build ''' + return ("DEBUG", "RELEASE", "NO-TARGET", "NOOPT") + + # ####################################################################= ################### # + # Verify and Save requested Ci Build Config = # + # ####################################################################= ################### # + + def SetPackages(self, list_of_requested_packages): + ''' Confirm the requested package list is valid and configure Sett= ingsManager + to build the requested packages. + + Raise UnsupportedException if a requested_package is not supported + ''' + unsupported =3D set(list_of_requested_packages) - \ + set(self.GetPackagesSupported()) + if(len(unsupported) > 0): + logging.critical( + "Unsupported Package Requested: " + " ".join(unsupported)) + raise Exception("Unsupported Package Requested: " + + " ".join(unsupported)) + self.ActualPackages =3D list_of_requested_packages + + def SetArchitectures(self, list_of_requested_architectures): + ''' Confirm the requests architecture list is valid and configure = SettingsManager + to run only the requested architectures. + + Raise Exception if a list_of_requested_architectures is not suppor= ted + ''' + unsupported =3D set(list_of_requested_architectures) - \ + set(self.GetArchitecturesSupported()) + if(len(unsupported) > 0): + logging.critical( + "Unsupported Architecture Requested: " + " ".join(unsuppor= ted)) + raise Exception( + "Unsupported Architecture Requested: " + " ".join(unsuppor= ted)) + self.ActualArchitectures =3D list_of_requested_architectures + + def SetTargets(self, list_of_requested_target): + ''' Confirm the request target list is valid and configure Setting= sManager + to run only the requested targets. + + Raise UnsupportedException if a requested_target is not supported + ''' + unsupported =3D set(list_of_requested_target) - \ + set(self.GetTargetsSupported()) + if(len(unsupported) > 0): + logging.critical( + "Unsupported Targets Requested: " + " ".join(unsupported)) + raise Exception("Unsupported Targets Requested: " + + " ".join(unsupported)) + self.ActualTargets =3D list_of_requested_target + + # ####################################################################= ################### # + # Actual Configuration for Ci Build = # + # ####################################################################= ################### # + + def GetActiveScopes(self): + ''' return tuple containing scopes that should be active for this = process ''' + scopes =3D ("cibuild","edk2-build") + + self.ActualToolChainTag =3D shell_environment.GetBuildVars().GetVa= lue("TOOL_CHAIN_TAG", "") + + if GetHostInfo().os.upper() =3D=3D "LINUX" and self.ActualToolChai= nTag.upper().startswith("GCC"): + if "AARCH64" in self.ActualArchitectures: + scopes +=3D ("gcc_aarch64_linux",) + if "ARM" in self.ActualArchitectures: + scopes +=3D ("gcc_arm_linux",) + + return scopes + + def GetRequiredSubmodules(self): + ''' return iterable containing RequiredSubmodule objects. + If no RequiredSubmodules return an empty iterable + ''' + rs=3D[] + rs.append(RequiredSubmodule( + "ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3", False)) + rs.append(RequiredSubmodule( + "CryptoPkg/Library/OpensslLib/openssl", False)) + return rs + + def GetName(self): + return "Edk2" + + def GetDependencies(self): + return [] + + def GetPackagesPath(self): + return () + + def GetWorkspaceRoot(self): + ''' get WorkspacePath ''' + return os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + + def FilterPackagesToTest(self, changedFilesList: list, potentialPackag= esList: list) -> list: + ''' Filter potential packages to test based on changed files. ''' + build_these_packages=3D[] + possible_packages=3DpotentialPackagesList.copy() + for f in changedFilesList: + nodes=3Df.split("/") # split each part of path for comparison= later + + # python file change in .pytool folder causes building all + if f.endswith(".py") and ".pytool" in nodes: + build_these_packages =3D possible_packages + break + + # BaseTools files that might change the build + if "BaseTools" in nodes: + if os.path.splitext(f) not in [".txt", ".md"]: + build_these_packages =3D possible_packages + break + return build_these_packages diff --git a/.pytool/Readme.md b/.pytool/Readme.md new file mode 100644 index 0000000000..5c1b107213 --- /dev/null +++ b/.pytool/Readme.md @@ -0,0 +1,128 @@ +# Edk2 Continuous Integration + +## Basic Status + +| Package | Windows VS2019 | Ubuntu GCC5 | Known Issues | +| :---- | :----- | :---- | :--- | +|ArmPkg | | | +|ArmPlatformPkg ||| +|ArmVirtPkg ||| +|CryptoPkg | :heavy_check_mark: | :heavy_check_mark: | New changes for NU= LL basecryptlib and tlslib, add IntrinsicLib.inf to dsc| +|DynamicTablesPkg ||| +|EmbeddedPkg ||| +|EmulatorPkg ||| +|FatPkg|:heavy_check_mark:|:heavy_check_mark:|| +|FmpDevicePkg|:heavy_check_mark:|:heavy_check_mark:|| +|IntelFsp2Pkg||| +|IntelFsp2WrapperPkg||| +| MdeModulePkg | :heavy_check_mark: |:heavy_check_mark: |DxeIpl dependency= on ArmPkg, Missing Visual Studio AARCH64/ARM support for EBC | +|MdePkg | :heavy_check_mark: |:heavy_check_mark: | Update DSC to add UefiF= ileHandleLib, Update BaseIoLibIntrinsic to support MSFT ARM/AARCH64| +|NetworkPkg | :heavy_check_mark: |:heavy_check_mark:| Libraries missing fr= om components section for package dsc| +|OvmfPkg ||| +|PcAtChipsetPkg | :heavy_check_mark: |:heavy_check_mark:| +|SecurityPkg|:heavy_check_mark:|:heavy_check_mark:| Incorrect libraryclass= in dec| +|ShellPkg | :heavy_check_mark: |:heavy_check_mark:|| +|SignedCapsulePkg||| +|SourceLevelDebugPkg||| +|StandaloneMmPkg||| +|UefiCpuPkg | :heavy_check_mark: |:heavy_check_mark: || +|UefiPayloadPkg||| + +For more detailed status look at the test results of the latest CI run on = the repo readme. + +## Background + +While a number of CI solutions exist, this proposal will focus on the usag= e of Azure Dev Ops and Build Pipelines. For demonstration, a sample [TianoC= ore repo](https://github.com/tianocore/edk2-staging.git) (branch master) an= d [Dev Ops Pipeline](https://dev.azure.com/tianocore/edk2-ci-play/_build?de= finitionId=3D14) have been set up. + +Furthermore, this proposal will leverage the TianoCore python tools PIP mo= dules: [library](https://pypi.org/project/edk2-pytool-library/) and [extens= ions](https://pypi.org/project/edk2-pytool-extensions/) (with repos located= [here](https://github.com/tianocore/edk2-pytool-library) and [here](https:= //github.com/tianocore/edk2-pytool-extensions)). + +The primary execution flows can be found in the `ci/AzurePipelines/Windows= -VS2019.yml` and `ci/AzurePipelines/Ubuntu-GCC5.yml` files. These YAML file= s are consumed by the Azure Dev Ops Build Pipeline and dictate what server = resources should be used, how they should be configured, and what processes= should be run on them. An overview of this schema can be found [here](http= s://docs.microsoft.com/en-us/azure/devops/pipelines/yaml-schema?view=3Dazur= e-devops&tabs=3Dschema). + +Inspection of these files reveals the EDKII Tools commands that make up th= e primary processes for the CI build: 'stuart_setup', 'stuart_update', and = 'stuart_ci_build'. These commands come from the EDKII Tools PIP modules and= are configured as described below. More documentation on the stuart tools = can be found [here](https://github.com/tianocore/edk2-pytool-extensions/blo= b/master/docs/using.md) and [here](https://github.com/tianocore/edk2-pytool= -extensions/blob/master/docs/features/feature_invocables.md). + +## Configuration + +Configuration of the CI process consists of (in order of precedence): + +* command-line arguments passed in via the Pipeline YAML +* a per-package configuration file (e.g. `.ci.yaml`) that is= detected by the CI system in EDKII Tools. +* a global configuration Python module (e.g. `CISetting.py`) passed in via= the command-line + +The global configuration file is described in [this readme](https://github= .com/tianocore/edk2-pytool-extensions/blob/master/docs/usability/using_sett= ings_manager.md) from the EDKII Tools documentation. This configuration is = written as a Python module so that decisions can be made dynamically based = on command line parameters and codebase state. + +The per-package configuration file can override most settings in the globa= l configuration file, but is not dynamic. This file can be used to skip or = customize tests that may be incompatible with a specific package. By defaul= t, the global configuration will try to run all tests on all packages. + +## Current PyTool Test Capabilities + +All CI tests are instances of EDKII Tools plugins. Documentation on the pl= ugin system can be found [here](https://github.com/tianocore/edk2-pytool-ex= tensions/blob/master/docs/usability/using_plugin_manager.md) and [here](htt= ps://github.com/tianocore/edk2-pytool-extensions/blob/master/docs/features/= feature_plugin_manager.md). Upon invocation, each plugin will be passed the= path to the current package under test and a dictionary containing its tar= geted configuration, as assembled from the command line, per-package config= uration, and global configuration. + +Note: CI plugins are considered unique from build plugins and helper plugi= ns, even though some CI plugins may execute steps of a build. + +In the example, these plugins live alongside the code under test (in the `= ci/Plugin` directory), but may be moved to the 'edk2-test' repo if that loc= ation makes more sense for the community. + +### Module Inclusion Test - DscCompleteCheck + +This test scans all available modules (via INF files) and compares them to= the package-level DSC file for the package each module is contained within= . The test considers it an error if any module does not appear in the `Comp= onents` section of at least one package-level DSC (indicating that it would= not be built if the package were built). + +### Code Compilation Test - CompilerPlugin + +Once the Module Inclusion Test has verified that all modules would be buil= t if all package-level DSCs were built, the Code Compilation Test simply ru= ns through and builds every package-level DSC on every toolchain and for ev= ery architecture that is supported. Any module that fails to build is consi= dered an error. + +### GUID Uniqueness Test - GuidCheck + +This test works on the collection of all packages rather than an individua= l package. It looks at all FILE_GUIDs and GUIDs declared in DEC files and e= nsures that they are unique for the codebase. This prevents, for example, a= ccidental duplication of GUIDs when using an existing INF as a template for= a new module. + +### Cross-Package Dependency Test - DependencyCheck + +This test compares the list of all packages used in INFs files for a given= package against a list of "allowed dependencies" in plugin configuration f= or that package. Any module that depends on a disallowed package will cause= a test failure. + +### Library Declaration Test - LibraryClassCheck + +This test scans at all library header files found in the `Library` folders= in all of the package's declared include directories and ensures that all = files have a matching LibraryClass declaration in the DEC file for the pack= age. Any missing declarations will cause a failure. + +### Invalid Character Test - CharEncodingCheck + +This test scans all files in a package to make sure that there are no inva= lid Unicode characters that may cause build errors in some character sets/l= ocalizations. + +### Spell Checking - cspell + +This test runs a spell checker on all files within the package. This is d= one using the NodeJs cspell tool. For details check `ci/Plugin/SpellCheck`= . For this plugin to run during ci you must install nodejs and cspell and = have both available to the command line when running your CI. + +Install + +* Install nodejs from https://nodejs.org/en/ +* Install cspell + 1. Open cmd prompt with access to node and npm + 2. Run `npm install -g cspell` + + More cspell info: https://github.com/streetsidesoftware/cspell + +## Current Azure Pipeline Tests + +When adding a test it can be added as either a *PyTool* test or just added= to the CI build process. This should be a deliberate choice. Any change = added as a pipeline test is not as easily run on a private/local workspace.= But there are times where this is still the preferred method. + +## PyTool Scopes + +Scopes are how the PyTool ext_dep, path_env, and plugins are activated. M= eaning that if an invocable process has a scope active then those ext_dep a= nd path_env will be active. To allow easy integration of PyTools capabiliti= es there are a few standard scopes. + +| Scope | Invocable | Description | +| :---- | :----- | :---- | +| global | edk2_invocable++ - should be base_abstract_invocable | R= unning an invocables | +| global-win | edk2_invocable++ | Running on Microsoft Windows | +| global-nix | edk2_invocable++ | Running on Linux based OS | +| edk2-build | | This indicates that an invocable is building EDK2 based = UEFI code | +| cibuild | set in .pytool/CISettings.py | Suggested target for edk2 conti= nuous integration builds. Tools used for CiBuilds can use this scope. Exa= mple: asl compiler | + +## Future investments + +* PatchCheck tests as plugins +* MacOS/xcode support +* Clang/LLVM support +* Visual Studio AARCH64 and ARM support +* BaseTools C tools CI/PR and binary release process +* BaseTools Python tools CI/PR process +* Host based unit testing +* Extensible private/closed source platform reporting +* Platform builds, validation +* UEFI SCTs +* Other automation --=20 2.21.0.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#49603): https://edk2.groups.io/g/devel/message/49603 Mute This Topic: https://groups.io/mt/39614217/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-