From nobody Sun May 5 06:00:36 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) client-ip=66.175.222.12; envelope-from=bounce+27952+63805+1787277+3901457@groups.io; helo=web01.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+63805+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=intel.com ARC-Seal: i=1; a=rsa-sha256; t=1596769023; cv=none; d=zohomail.com; s=zohoarc; b=aK9NHwr5emgS6lgOzZjT4hLiqwF7tI4RhBjw1EipqfGSNwjfic0Qsg2gYSdrP7ZtvySypNUPA9c+vCxNRUUkK+BgWZPaqUqemwpVQrb5/oyH58IOYQYktFtco9JZw5ynIk4897cV+EseYUFs1Y4NsXbpW6E4yX+/dx0frjIMtTk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1596769023; h=Content-Transfer-Encoding:Cc:Date:From:List-Id:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Sender:Subject:To; bh=sxyxFa4Sy/3U4Nf4aRMCA+XrCsTy7ouS0hIyNwZHlY8=; b=gAzKXpSiyx34YyvwEIjoYozavnJKmHxme+gM3v2cCkLPNbkd+PJEsJ4FoI1nqRlU+t0hIbLz1svLpVJ/oyWhnWwUYYVjmcDZsTVGGfODmYTFpvVEcIDMLamLGeN41fONVyDcXF6bOqmM/8YtsuTBtvXDUc7tR3brUt7uM6w8TqI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.12 as permitted sender) smtp.mailfrom=bounce+27952+63805+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) header.from= Received: from web01.groups.io (web01.groups.io [66.175.222.12]) by mx.zohomail.com with SMTPS id 1596769023048776.5981943725262; Thu, 6 Aug 2020 19:57:03 -0700 (PDT) Return-Path: X-Received: by 127.0.0.2 with SMTP id l8SrYY1788612xOxG2LIhmIz; Thu, 06 Aug 2020 19:57:02 -0700 X-Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mx.groups.io with SMTP id smtpd.web11.3213.1596769021787750636 for ; Thu, 06 Aug 2020 19:57:01 -0700 IronPort-SDR: 5c1dNOoZg37gq3+nK7ZPW+TAIgVAjuf4QT7BuhfE3NU4yAxM70Qluk6O6tl41egGGdBZ9h87/Z F13E+th+Ac2g== X-IronPort-AV: E=McAfee;i="6000,8403,9705"; a="154109058" X-IronPort-AV: E=Sophos;i="5.75,443,1589266800"; d="scan'208";a="154109058" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False X-Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 06 Aug 2020 19:57:00 -0700 IronPort-SDR: PlKqR1pM8/K6Eyp2i+n18BCfD5KEeUhheLHIhMfwvUonHKF9SHN6qU32qU4BBRUaoDD3GdOPJH 63Kc60k5oVMw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.75,443,1589266800"; d="scan'208";a="293520386" X-Received: from mdkinney-mobl2.amr.corp.intel.com ([10.254.82.243]) by orsmga006.jf.intel.com with ESMTP; 06 Aug 2020 19:57:00 -0700 From: "Michael D Kinney" To: devel@edk2.groups.io Cc: Bret Barkelew , Sean Brogan , Bret Barkelew Subject: [edk2-devel] [Patch] UnitTestFrameworkPkg/Readme.md: Update documentation for latest features Date: Thu, 6 Aug 2020 19:56:56 -0700 Message-Id: <20200807025656.38956-1-michael.d.kinney@intel.com> MIME-Version: 1.0 Precedence: Bulk List-Unsubscribe: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,michael.d.kinney@intel.com X-Gm-Message-State: ZgAbj53KPzE0kEwY7vexN0EJx1787277AA= Content-Transfer-Encoding: quoted-printable DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=groups.io; q=dns/txt; s=20140610; t=1596769022; bh=VKwKJI6Bgqang8C9a8tJAZSgYzfeQOE2qkeA3uOyaDk=; h=Cc:Date:From:Reply-To:Subject:To; b=g3O4VJMYRCdL2OXqrGHgtH8DnZyp+OHcXnnM5ZioiAk3kPJwqZ4SqDM9++yd4LhdC2h YD2thoxskS+VU47V84SDW4oMGfO/73B0qpUGuuOTtMhAyPTIzimwtQHa7XDqHEV7WEmxf mqcjhjkRC7vTWXUX4MU/P3PJTCeg2p7rsjo= X-ZohoMail-DKIM: pass (identity @groups.io) Content-Type: text/plain; charset="utf-8" From: Bret Barkelew * Add additional documentation about running tests locally * Add a note about XML formatting * Update readme with BaseLib and UNIT_TESTING_DEBUG Cc: Michael D Kinney Cc: Sean Brogan Signed-off-by: Bret Barkelew Reviewed-by: Michael D Kinney Reviewed-by: Sean Brogan --- UnitTestFrameworkPkg/ReadMe.md | 165 +++++++++++++++++++++++++++++++++ 1 file changed, 165 insertions(+) diff --git a/UnitTestFrameworkPkg/ReadMe.md b/UnitTestFrameworkPkg/ReadMe.md index 64386941cb..70656a422c 100644 --- a/UnitTestFrameworkPkg/ReadMe.md +++ b/UnitTestFrameworkPkg/ReadMe.md @@ -229,6 +229,8 @@ https://api.cmocka.org/ =20 ## Development =20 +### Iterating on a Single Test + When using the EDK2 Pytools for CI testing, the host-based unit tests will= be built and run on any build that includes the `NOOPT` build target. =20 @@ -239,6 +241,169 @@ the following command will build only the SafeIntLib = host-based test from the Md stuart_ci_build -c .pytool/CISettings.py TOOL_CHAIN_TAG=3DVS2017 -p MdePkg= -t NOOPT BUILDMODULE=3DMdePkg/Test/UnitTest/Library/BaseSafeIntLib/TestBas= eSafeIntLib.inf ``` =20 +### Hooking BaseLib + +Most unit test mocking can be performed by the functions provided in the U= nitTestFramework libraries, but since +BaseLib is consumed by the Framework itself, it requires different techniq= ues to substitute parts of the +functionality. + +To solve some of this, the UnitTestFramework consumes a special implementa= tion of BaseLib for host-based tests. +This implementation contains a [hook table](https://github.com/tianocore/e= dk2/blob/e188ecc8b4aed8fdd26b731d43883861f5e5e7b4/MdePkg/Test/UnitTest/Incl= ude/Library/UnitTestHostBaseLib.h#L507) +that can be used to substitute test functionality for any of the BaseLib f= unctions. By default, this implementation +will use the underlying BaseLib implementation, so the unit test writer on= ly has to supply minimal code to test a +particular case. + +### Debugging the Framework Itself + +While most of the tests that are produced by the UnitTestFramework are eas= y to step through in a debugger, the Framework +itself consumes code (mostly Cmocka) that sets its own build flags. These = flags cause parts of the Framework to not +export symbols and captures exceptions, and as such are harder to debug. W= e have provided a Stuart parameter to force +symbolic debugging to be enabled. + +You can run a build by adding the `BLD_*_UNIT_TESTING_DEBUG=3DTRUE` parame= ter to enable this build option. + +```bash +stuart_ci_build -c .pytool/CISettings.py TOOL_CHAIN_TAG=3DVS2019 -p MdePkg= -t NOOPT BLD_*_UNIT_TESTING_DEBUG=3DTRUE +``` + +## Building and Running Host-Based Tests + +The EDK2 CI infrastructure provides a convenient way to run all host-based= tests -- in the the entire tree or just +selected packages -- and aggregate all the the reports, including highligh= ting any failures. This functionality is +provided through the Stuart build system (published by EDK2-PyTools) and t= he `NOOPT` build target. + +### Building Locally + +First, to make sure you're working with the latest PyTools, run the follow= ing command: + +```bash +# Would recommend to run this in a Python venv, but that's out of scope fo= r this doc. +python -m pip install --upgrade -r ./pip-requirements.txt +``` + +After that, the following commands will set up the build and run the host-= based tests. + +```bash +# Setup repo for building +# stuart_setup -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=3D +stuart_setup -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=3DVS2019 + +# Update all binary dependencies +# stuart_update -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=3D +stuart_update -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=3DVS2019 + +# Build and run the tests +# stuart_ci_build -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=3D -t NOOPT [-p ] +stuart_ci_build -c ./.pytool/CISettings.py TOOL_CHAIN_TAG=3DVS2019 -t NOOP= T -p MdePkg +``` + +### Evaluating the Results + +In your immediate output, any build failures will be highlighted. You can = see these below as "WARNING" and "ERROR" messages. + +```text +(edk_env) PS C:\_uefi\edk2> stuart_ci_build -c .\.pytool\CISettings.py TOO= L_CHAIN_TAG=3DVS2019 -t NOOPT -p MdePkg + +SECTION - Init SDE +SECTION - Loading Plugins +SECTION - Start Invocable Tool +SECTION - Getting Environment +SECTION - Loading plugins +SECTION - Building MdePkg Package +PROGRESS - --Running MdePkg: Host Unit Test Compiler Plugin NOOPT -- +WARNING - Allowing Override for key TARGET_ARCH +PROGRESS - Start time: 2020-07-27 17:18:08.521672 +PROGRESS - Setting up the Environment +PROGRESS - Running Pre Build +PROGRESS - Running Build NOOPT +PROGRESS - Running Post Build +SECTION - Run Host based Unit Tests +SUBSECTION - Testing for architecture: X64 +WARNING - TestBaseSafeIntLibHost.exe Test Failed +WARNING - Test SafeInt8ToUint8 - UT_ASSERT_EQUAL(0x5b:5b, Result:5c) +c:\_uefi\edk2\MdePkg\Test\UnitTest\Library\BaseSafeIntLib\TestBaseSafeIntL= ib.c:35: error: Failure! +ERROR - Plugin Failed: Host-Based Unit Test Runner returned 1 +CRITICAL - Post Build failed +PROGRESS - End time: 2020-07-27 17:18:19.792313 Total time Elapsed: 0:00:= 11 +ERROR - --->Test Failed: Host Unit Test Compiler Plugin NOOPT returned 1 +ERROR - Overall Build Status: Error +PROGRESS - There were 1 failures out of 1 attempts +SECTION - Summary +ERROR - Error + +(edk_env) PS C:\_uefi\edk2> +``` + +If a test fails, you can run it manually to get more details... + +```text +(edk_env) PS C:\_uefi\edk2> .\Build\MdePkg\HostTest\NOOPT_VS2019\X64\TestB= aseSafeIntLibHost.exe + +Int Safe Lib Unit Test Application v0.1 +--------------------------------------------------------- +------------ RUNNING ALL TEST SUITES -------------- +--------------------------------------------------------- +--------------------------------------------------------- +RUNNING TEST SUITE: Int Safe Conversions Test Suite +--------------------------------------------------------- +[=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D] Running 71 test(s). +[ RUN ] Test SafeInt8ToUint8 +[ ERROR ] --- UT_ASSERT_EQUAL(0x5b:5b, Result:5c) +[ LINE ] --- c:\_uefi\edk2\MdePkg\Test\UnitTest\Library\BaseSafeIntLib= \TestBaseSafeIntLib.c:35: error: Failure! +[ FAILED ] Test SafeInt8ToUint8 +[ RUN ] Test SafeInt8ToUint16 +[ OK ] Test SafeInt8ToUint16 +[ RUN ] Test SafeInt8ToUint32 +[ OK ] Test SafeInt8ToUint32 +[ RUN ] Test SafeInt8ToUintn +[ OK ] Test SafeInt8ToUintn +... +``` + +You can also, if you are so inclined, read the output from the exact insta= nce of the test that was run during +`stuart_ci_build`. The ouput file can be found on a path that looks like: + +`Build//HostTest//...result.= xml` + +A sample of this output looks like: + +```xml + + + + + + + + + + + + + +``` + +### XML Reporting Mode + +Since these applications are built using the CMocka framework, they can al= so use the following env variables to output +in a structured XML rather than text: + +```text +CMOCKA_MESSAGE_OUTPUT=3Dxml +CMOCKA_XML_FILE=3D +``` + +This mode is used by the test running plugin to aggregate the results for = CI test status reporting in the web view. + +### Important Note + +This works on both Windows and Linux, but is currently limited to x64 arch= itectures. Working on getting others, but we +also welcome contributions. + ## Known Limitations =20 ### PEI, DXE, SMM --=20 2.21.0.windows.1 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#63805): https://edk2.groups.io/g/devel/message/63805 Mute This Topic: https://groups.io/mt/76042308/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-