From nobody Fri May 17 10:44:22 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) client-ip=66.175.222.108; envelope-from=bounce+27952+109695+1787277+3901457@groups.io; helo=mail02.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+109695+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=linux.microsoft.com ARC-Seal: i=1; a=rsa-sha256; t=1697591110; cv=none; d=zohomail.com; s=zohoarc; b=KZXftyGbekTeTnZXUS4OXwU0y/FqTZX71lmjahUldcTxridrtZXDDwSPtoNJCSHos62PwTwmzLJIarCQzy15ses/p6VD7tMWFyUaDxAXDvswNg+QNfzcjASKQm9vIWAuwr8A3F8n7Dm/YlqH81ZBSTcqChx+QO2Ry9XNUAoVjbA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1697591110; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:References:Sender:Subject:Subject:To:To:Message-Id; bh=IFN1RW5jUOkVZrOc85Ngff80UKzZ+FIqSClP/+qsEis=; b=ZxkiMGb1qWQ9Fu+Rymi7NDdXNX/3MxoEggfvAbpFKJGbGSPxkBrZq6BF7xoCVetQOU6vkvLBKW6He/cVTYlL5VEgkLnHBotQmv841aF9J/LUXeNkxoB35/5S9d0cLt8PILrDBIU6gbB9VVcU2agNVPHbpEPr6qe23aby7bYDIVc= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+109695+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by mx.zohomail.com with SMTPS id 1697591110150543.0727710139884; Tue, 17 Oct 2023 18:05:10 -0700 (PDT) Return-Path: DKIM-Signature: a=rsa-sha256; bh=WBJK9m/jpSOlmYq3g5MEtIlvk18obaEpRHgxW9PJ4Xo=; c=relaxed/simple; d=groups.io; h=DKIM-Filter:From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References:MIME-Version:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Transfer-Encoding; s=20140610; t=1697591109; v=1; b=peQOZy4kFKgwktKaEG7I+kCuiwI4sKKki09EHtz9whFZ6UF5SuWjHMnZdunUcSOHD74rJobY zGoPbAWJsE5YN28B+7B+IFGdhmuCBrDIDN2c0W7I7iQO/N3KV6T/GsdOlsrmw0eCTlI7n8T9+Gx SIzz1ppnPQB2S2Z8/E+VczM0= X-Received: by 127.0.0.2 with SMTP id qdIGYY1788612xnSroo7ZW0v; Tue, 17 Oct 2023 18:05:09 -0700 X-Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by mx.groups.io with SMTP id smtpd.web11.271876.1697591109227960077 for ; Tue, 17 Oct 2023 18:05:09 -0700 X-Received: from localhost.localdomain (unknown [47.201.241.95]) by linux.microsoft.com (Postfix) with ESMTPSA id 4D73920B74C2; Tue, 17 Oct 2023 18:05:08 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 4D73920B74C2 From: "Michael Kubacki" To: devel@edk2.groups.io Cc: Sean Brogan , Michael D Kinney Subject: [edk2-devel] [PATCH v3 1/7] Remove existing CodeQL infrastructure Date: Tue, 17 Oct 2023 21:04:39 -0400 Message-ID: <20231018010445.528-2-mikuback@linux.microsoft.com> In-Reply-To: <20231018010445.528-1-mikuback@linux.microsoft.com> References: <20231018010445.528-1-mikuback@linux.microsoft.com> MIME-Version: 1.0 Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,mikuback@linux.microsoft.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: 12RY8B1QDDDNYLFnYZ195K5Dx1787277AA= Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @groups.io) X-ZM-MESSAGEID: 1697591110384100005 Content-Type: text/plain; charset="utf-8" From: Michael Kubacki CodeQL currently runs via the codeql-analysis.yml GitHub workflow which uses the `github/codeql-action/init@v2` action (pre-build) and the `github/codeql-action/analyze@v2` action (post-build) to setup the CodeQL environment and extract results. This infrastructure is removed in preparation for a new design that will directly run the CodeQL CLI as part of the build. This will allow CodeQL to be run locally as part of the normal build process with results that match 1:1 with CI builds. Cc: Sean Brogan Cc: Michael D Kinney Signed-off-by: Michael Kubacki --- .github/codeql/codeql-config.yml | 29 ----- .github/codeql/edk2.qls | 24 ---- .github/workflows/codeql-analysis.yml | 118 -------------------- 3 files changed, 171 deletions(-) diff --git a/.github/codeql/codeql-config.yml b/.github/codeql/codeql-confi= g.yml deleted file mode 100644 index a51db141ebe3..000000000000 --- a/.github/codeql/codeql-config.yml +++ /dev/null @@ -1,29 +0,0 @@ -## @file -# CodeQL configuration file for edk2. -# -# Copyright (c) Microsoft Corporation. -# SPDX-License-Identifier: BSD-2-Clause-Patent -## - -name: "CodeQL config" - -# The following line disables the default queries. This is used because we= want to enable on query at a time by -# explicitly specifying each query in a "queries" array as they are enable= d. -# -# See the following for more information about adding custom queries: -# https://docs.github.com/en/code-security/code-scanning/automatically-sca= nning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#us= ing-a-custom-configuration-file - -#disable-default-queries: true - -queries: - - name: EDK2 CodeQL Query List - uses: ./.github/codeql/edk2.qls - -# We must specify a query for CodeQL to run. Until the first query is enab= led, enable the security query suite but -# exclude all problem levels from impacting the results. After the first q= uery is enabled, this filter can be relaxed -# to find the level of problems desired from the query. -query-filters: -- exclude: - problem.severity: - - warning - - recommendation diff --git a/.github/codeql/edk2.qls b/.github/codeql/edk2.qls deleted file mode 100644 index 9bea9ba01f24..000000000000 --- a/.github/codeql/edk2.qls +++ /dev/null @@ -1,24 +0,0 @@ ---- -- description: EDK2 (C++) queries - -# Bring in all queries from the official cpp-queries suite so individual q= ueries can be explicitly enabled. - -- queries: '.' - from: codeql/cpp-queries - -# Enable individual queries below. - -- include: - id: cpp/conditionallyuninitializedvariable -- include: - id: cpp/infinite-loop-with-unsatisfiable-exit-condition -- include: - id: cpp/overflow-buffer -- include: - id: cpp/overrunning-write -- include: - id: cpp/overrunning-write-with-float -- include: - id: cpp/pointer-overflow-check -- include: - id: cpp/very-likely-overrunning-write diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/code= ql-analysis.yml deleted file mode 100644 index 992b3b6f654e..000000000000 --- a/.github/workflows/codeql-analysis.yml +++ /dev/null @@ -1,118 +0,0 @@ -# @file -# GitHub Workflow for CodeQL Analysis -# -# Copyright (c) Microsoft Corporation. -# -# SPDX-License-Identifier: BSD-2-Clause-Patent -## - -name: "CodeQL" - -on: - push: - branches: - - master - pull_request: - branches: - - master - paths-ignore: - - '**/*.bat' - - '**/*.md' - - '**/*.py' - - '**/*.rst' - - '**/*.sh' - - '**/*.txt' - - schedule: - # https://crontab.guru/#20_23_*_*_4 - - cron: '20 23 * * 4' - -jobs: - analyze: - name: Analyze - runs-on: windows-2019 - permissions: - actions: read - contents: read - security-events: write - - strategy: - fail-fast: false - matrix: - include: - - Package: "ArmPkg" - ArchList: "IA32,X64" - - Package: "CryptoPkg" - ArchList: "IA32" - - Package: "CryptoPkg" - ArchList: "X64" - - Package: "DynamicTablesPkg" - ArchList: "IA32,X64" - - Package: "FatPkg" - ArchList: "IA32,X64" - - Package: "FmpDevicePkg" - ArchList: "IA32,X64" - - Package: "IntelFsp2Pkg" - ArchList: "IA32,X64" - - Package: "IntelFsp2WrapperPkg" - ArchList: "IA32,X64" - - Package: "MdeModulePkg" - ArchList: "IA32" - - Package: "MdeModulePkg" - ArchList: "X64" - - Package: "MdePkg" - ArchList: "IA32,X64" - - Package: "PcAtChipsetPkg" - ArchList: "IA32,X64" - - Package: "PrmPkg" - ArchList: "IA32,X64" - - Package: "SecurityPkg" - ArchList: "IA32,X64" - - Package: "ShellPkg" - ArchList: "IA32,X64" - - Package: "SourceLevelDebugPkg" - ArchList: "IA32,X64" - - Package: "StandaloneMmPkg" - ArchList: "IA32,X64" - - Package: "UefiCpuPkg" - ArchList: "IA32,X64" - - Package: "UnitTestFrameworkPkg" - ArchList: "IA32,X64" - steps: - - name: Checkout repository - uses: actions/checkout@v3 - - - name: Install Python - uses: actions/setup-python@v4 - with: - python-version: '3.11' - cache: 'pip' - cache-dependency-path: 'pip-requirements.txt' - - # Initializes the CodeQL tools for scanning. - - name: Initialize CodeQL - uses: github/codeql-action/init@v2 - with: - languages: 'cpp' - # CodeQL supports [ 'cpp', 'csharp', 'go', 'java', 'javascript', '= python', 'ruby' ] - # Learn more about CodeQL language support at https://codeql.githu= b.com/docs/codeql-overview/supported-languages-and-frameworks/ - config-file: ./.github/codeql/codeql-config.yml - # Note: Add new queries to codeql-config.yml file as they are enab= led. - - - name: Install/Upgrade pip Modules - run: pip install -r pip-requirements.txt --upgrade - - - name: Setup - run: stuart_setup -c .pytool/CISettings.py -t DEBUG -a ${{ matrix.Ar= chList }} TOOL_CHAIN_TAG=3DVS2019 - - - name: Update - run: stuart_update -c .pytool/CISettings.py -t DEBUG -a ${{ matrix.A= rchList }} TOOL_CHAIN_TAG=3DVS2019 - - - name: Build Tools From Source - run: python BaseTools/Edk2ToolsBuild.py -t VS2019 - - - name: CI Build - run: stuart_ci_build -c .pytool/CISettings.py -p ${{ matrix.Package = }} -t DEBUG -a ${{ matrix.ArchList }} TOOL_CHAIN_TAG=3DVS2019 - - - name: Perform CodeQL Analysis - uses: github/codeql-action/analyze@v2 --=20 2.42.0.windows.2 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#109695): https://edk2.groups.io/g/devel/message/109695 Mute This Topic: https://groups.io/mt/102031056/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Fri May 17 10:44:22 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) client-ip=66.175.222.108; envelope-from=bounce+27952+109696+1787277+3901457@groups.io; helo=mail02.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+109696+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=linux.microsoft.com ARC-Seal: i=1; a=rsa-sha256; t=1697591113; cv=none; d=zohomail.com; s=zohoarc; b=GW+63xUfdY5kHX5S5ZOx1k05/egrVvN6Syr4rdAiIfoAsrHmwAIZ/uXAEoFdqTPwIeYghRrr5L1wrN+uEqIa2VQd0cNIo1Al+dMSd0GkL8A25g0mzi8Q9Gyw7475MQ/fvIU9v9upRhH7c3BVdTRPTynkbu2s1ZFAMnCf3Gt8Z1U= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1697591113; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:References:Sender:Subject:Subject:To:To:Message-Id; bh=KGtQhi6QptV0PSBc3VlQ8nG4Zlm76Ucbq9FlcWah/T0=; b=D+dmqv2DhNrdI2uY2/XFFSz/OrvXoc1P+Xclka9ZemR+3mfuBFb/Pw8iK4NppYkv7KR1CpQNDTa/E8NqeTNl0+ecANKmnFgldxmvBfDV2OkkO8xG4LPlPJjY6Lgnf3Db6Kklx7kz6SIv/Wlm+5cCIIqhfr6CpyLK3A4c2Sv/MBg= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+109696+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by mx.zohomail.com with SMTPS id 1697591113694759.2594780243875; Tue, 17 Oct 2023 18:05:13 -0700 (PDT) Return-Path: DKIM-Signature: a=rsa-sha256; bh=D96Gk/xzxPOlCF8hCJPHFHNrH76hCxj7OMUETJY4tcs=; c=relaxed/simple; d=groups.io; h=DKIM-Filter:From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References:MIME-Version:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Transfer-Encoding; s=20140610; t=1697591113; v=1; b=JL1bJ/LdVx1qsTsu+mzUC/jglt1Djdc/TP9ENk0TZF4dIhTSrb08Umzi3PwG38F4aFwrAySX IKPO4xbyZnz7LpCBd3b6ImV+RCsKiQVkp5W2GZYif0kpmaspNfHMIMJO31H7iYur9FSJsoWVmIN P1Tr5tYwWYy/Gj6Ow5UPQjCE= X-Received: by 127.0.0.2 with SMTP id uRusYY1788612xSzBeDm2f5D; Tue, 17 Oct 2023 18:05:13 -0700 X-Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by mx.groups.io with SMTP id smtpd.web11.271878.1697591112596321978 for ; Tue, 17 Oct 2023 18:05:12 -0700 X-Received: from localhost.localdomain (unknown [47.201.241.95]) by linux.microsoft.com (Postfix) with ESMTPSA id 3B1A420B74C0; Tue, 17 Oct 2023 18:05:11 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 3B1A420B74C0 From: "Michael Kubacki" To: devel@edk2.groups.io Cc: Bob Feng , Liming Gao , Michael D Kinney , Rebecca Cran , Sean Brogan , Yuwei Chen Subject: [edk2-devel] [PATCH v3 2/7] BaseTools/Plugin/CodeQL: Add CodeQL build plugin Date: Tue, 17 Oct 2023 21:04:40 -0400 Message-ID: <20231018010445.528-3-mikuback@linux.microsoft.com> In-Reply-To: <20231018010445.528-1-mikuback@linux.microsoft.com> References: <20231018010445.528-1-mikuback@linux.microsoft.com> MIME-Version: 1.0 Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,mikuback@linux.microsoft.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: ujiFiqkuL8qhkHd3I3jDbaQNx1787277AA= Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @groups.io) X-ZM-MESSAGEID: 1697591114633100002 Content-Type: text/plain; charset="utf-8" From: Michael Kubacki Adds a CodeQL plugin that supports CodeQL in the build system. 1. CodeQlBuildPlugin - Generates a CodeQL database for a given build. 2. CodeQlAnalyzePlugin - Analyzes a CodeQL database and interprets results. 3. External dependencies - Assist with downloading the CodeQL CLI and making it available to the CodeQL plugins. 4. CodeQlQueries.qls - A C/C++ CodeQL query set run against the code. 5. Readme.md - A comprehensive readme file to help: - Platform integrators understand how to configure the plugin - Developers understand how to modify the plugin - Users understand how to use the plugin Read Readme.md for additional details. Cc: Bob Feng Cc: Liming Gao Cc: Michael D Kinney Cc: Rebecca Cran Cc: Sean Brogan Cc: Yuwei Chen Signed-off-by: Michael Kubacki Reviewed-by: Yuwei Chen --- BaseTools/Plugin/CodeQL/CodeQlAnalyzePlugin.py | 222 +++++++++++ BaseTools/Plugin/CodeQL/CodeQlAnalyze_plug_in.yaml | 13 + BaseTools/Plugin/CodeQL/CodeQlBuildPlugin.py | 172 +++++++++ BaseTools/Plugin/CodeQL/CodeQlBuild_plug_in.yaml | 13 + BaseTools/Plugin/CodeQL/CodeQlQueries.qls | 75 ++++ BaseTools/Plugin/CodeQL/Readme.md | 388 +++++++++++++= +++++++ BaseTools/Plugin/CodeQL/analyze/__init__.py | 0 BaseTools/Plugin/CodeQL/analyze/analyze_filter.py | 176 +++++++++ BaseTools/Plugin/CodeQL/analyze/globber.py | 132 +++++++ BaseTools/Plugin/CodeQL/codeqlcli_ext_dep.yaml | 26 ++ BaseTools/Plugin/CodeQL/codeqlcli_linux_ext_dep.yaml | 24 ++ BaseTools/Plugin/CodeQL/codeqlcli_windows_ext_dep.yaml | 24 ++ BaseTools/Plugin/CodeQL/common/__init__.py | 0 BaseTools/Plugin/CodeQL/common/codeql_plugin.py | 74 ++++ 14 files changed, 1339 insertions(+) diff --git a/BaseTools/Plugin/CodeQL/CodeQlAnalyzePlugin.py b/BaseTools/Plu= gin/CodeQL/CodeQlAnalyzePlugin.py new file mode 100644 index 000000000000..199b0ad478ed --- /dev/null +++ b/BaseTools/Plugin/CodeQL/CodeQlAnalyzePlugin.py @@ -0,0 +1,222 @@ +# @file CodeQAnalyzePlugin.py +# +# A build plugin that analyzes a CodeQL database. +# +# Copyright (c) Microsoft Corporation. All rights reserved. +# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +import json +import logging +import os +import yaml + +from analyze import analyze_filter +from common import codeql_plugin + +from edk2toolext import edk2_logging +from edk2toolext.environment.plugintypes.uefi_build_plugin import \ + IUefiBuildPlugin +from edk2toolext.environment.uefi_build import UefiBuilder +from edk2toollib.uefi.edk2.path_utilities import Edk2Path +from edk2toollib.utility_functions import RunCmd +from pathlib import Path + + +class CodeQlAnalyzePlugin(IUefiBuildPlugin): + + def do_post_build(self, builder: UefiBuilder) -> int: + """CodeQL analysis post-build functionality. + + Args: + builder (UefiBuilder): A UEFI builder object for this build. + + Returns: + int: The number of CodeQL errors found. Zero indicates that + AuditOnly mode is enabled or no failures were found. + """ + + pp =3D builder.pp.split(os.pathsep) + edk2_path =3D Edk2Path(builder.ws, pp) + + self.builder =3D builder + self.package =3D edk2_path.GetContainingPackage( + builder.mws.join(builder.ws, + builder.env.GetValue( + "ACTIVE_PLATFORM"))) + self.package_path =3D Path( + edk2_path.GetAbsolutePathOnThisSystemFromEdk2RelativePath( + self.package)) + self.target =3D builder.env.GetValue("TARGET") + + self.codeql_db_path =3D codeql_plugin.get_codeql_db_path( + builder.ws, self.package, self.target, + new_path=3DFalse) + + self.codeql_path =3D codeql_plugin.get_codeql_cli_path() + if not self.codeql_path: + logging.critical("CodeQL build enabled but CodeQL CLI applicat= ion " + "not found.") + return -1 + + codeql_sarif_dir_path =3D self.codeql_db_path[ + :self.codeql_db_path.rindex('-')] + codeql_sarif_dir_path =3D codeql_sarif_dir_path.replace( + "-db-", "-analysis-") + self.codeql_sarif_path =3D os.path.join( + codeql_sarif_dir_path, + (os.path.basename( + self.codeql_db_path) + + ".sarif")) + + edk2_logging.log_progress(f"Analyzing {self.package} ({self.target= }) " + f"CodeQL database at:\n" + f" {self.codeql_db_path}") + edk2_logging.log_progress(f"Results will be written to:\n" + f" {self.codeql_sarif_path}") + + # Packages are allowed to specify package-specific query specifiers + # in the package CI YAML file that override the global query speci= fier. + audit_only =3D False + query_specifiers =3D None + package_config_file =3D Path(os.path.join( + self.package_path, self.package + ".ci.yam= l")) + plugin_data =3D None + if package_config_file.is_file(): + with open(package_config_file, 'r') as cf: + package_config_file_data =3D yaml.safe_load(cf) + if "CodeQlAnalyze" in package_config_file_data: + plugin_data =3D package_config_file_data["CodeQlAnalyz= e"] + if "AuditOnly" in plugin_data: + audit_only =3D plugin_data["AuditOnly"] + if "QuerySpecifiers" in plugin_data: + logging.debug(f"Loading CodeQL query specifiers in= " + f"{str(package_config_file)}") + query_specifiers =3D plugin_data["QuerySpecifiers"] + + global_audit_only =3D builder.env.GetValue("STUART_CODEQL_AUDIT_ON= LY") + if global_audit_only: + if global_audit_only.strip().lower() =3D=3D "true": + audit_only =3D True + + if audit_only: + logging.info(f"CodeQL Analyze plugin is in audit only mode for= " + f"{self.package} ({self.target}).") + + # Builds can override the query specifiers defined in this plugin + # by setting the value in the STUART_CODEQL_QUERY_SPECIFIERS + # environment variable. + if not query_specifiers: + query_specifiers =3D builder.env.GetValue( + "STUART_CODEQL_QUERY_SPECIFIERS") + + # Use this plugins query set file as the default fallback if it is + # not overridden. It is possible the file is not present if modifi= ed + # locally. In that case, skip the plugin. + plugin_query_set =3D Path(Path(__file__).parent, "CodeQlQueries.ql= s") + + if not query_specifiers and plugin_query_set.is_file(): + query_specifiers =3D str(plugin_query_set.resolve()) + + if not query_specifiers: + logging.warning("Skipping CodeQL analysis since no CodeQL quer= y " + "specifiers were provided.") + return 0 + + codeql_params =3D (f'database analyze {self.codeql_db_path} ' + f'{query_specifiers} --format=3Dsarifv2.1.0 ' + f'--output=3D{self.codeql_sarif_path} --download ' + f'--threads=3D0') + + # CodeQL requires the sarif file parent directory to exist already. + Path(self.codeql_sarif_path).parent.mkdir(exist_ok=3DTrue, parents= =3DTrue) + + cmd_ret =3D RunCmd(self.codeql_path, codeql_params) + if cmd_ret !=3D 0: + logging.critical(f"CodeQL CLI analysis failed with return code= " + f"{cmd_ret}.") + + if not os.path.isfile(self.codeql_sarif_path): + logging.critical(f"The sarif file {self.codeql_sarif_path} was= " + f"not created. Analysis cannot continue.") + return -1 + + filter_pattern_data =3D [] + global_filter_file_value =3D builder.env.GetValue( + "STUART_CODEQL_FILTER_FILES") + if global_filter_file_value: + global_filter_files =3D global_filter_file_value.strip().split= (',') + global_filter_files =3D [Path(f) for f in global_filter_files] + + for global_filter_file in global_filter_files: + if global_filter_file.is_file(): + with open(global_filter_file, 'r') as ff: + global_filter_file_data =3D yaml.safe_load(ff) + if "Filters" in global_filter_file_data: + current_pattern_data =3D \ + global_filter_file_data["Filters"] + if type(current_pattern_data) is not list: + logging.critical( + f"CodeQL pattern data must be a list o= f " + f"strings. Data in " + f"{str(global_filter_file.resolve())} = is " + f"invalid. CodeQL analysis is incomple= te.") + return -1 + filter_pattern_data +=3D current_pattern_data + else: + logging.critical( + f"CodeQL global filter file " + f"{str(global_filter_file.resolve())} is " + f"malformed. Missing Filters section. Code= QL " + f"analysis is incomplete.") + return -1 + else: + logging.critical( + f"CodeQL global filter file " + f"{str(global_filter_file.resolve())} was not foun= d. " + f"CodeQL analysis is incomplete.") + return -1 + + if plugin_data and "Filters" in plugin_data: + if type(plugin_data["Filters"]) is not list: + logging.critical( + "CodeQL pattern data must be a list of strings. " + "CodeQL analysis is incomplete.") + return -1 + filter_pattern_data.extend(plugin_data["Filters"]) + + if filter_pattern_data: + logging.info("Applying CodeQL SARIF result filters.") + analyze_filter.filter_sarif( + self.codeql_sarif_path, + self.codeql_sarif_path, + filter_pattern_data, + split_lines=3DFalse) + + with open(self.codeql_sarif_path, 'r') as sf: + sarif_file_data =3D json.load(sf) + + try: + # Perform minimal JSON parsing to find the number of errors. + total_errors =3D 0 + for run in sarif_file_data['runs']: + total_errors +=3D len(run['results']) + except KeyError: + logging.critical("Sarif file does not contain expected data. " + "Analysis cannot continue.") + return -1 + + if total_errors > 0: + if audit_only: + # Show a warning message so CodeQL analysis is not forgott= en. + # If the repo owners truly do not want to fix CodeQL issue= s, + # analysis should be disabled entirely. + logging.warning(f"{self.package} ({self.target}) CodeQL " + f"analysis ignored {total_errors} errors d= ue " + f"to audit mode being enabled.") + return 0 + else: + logging.error(f"{self.package} ({self.target}) CodeQL " + f"analysis failed with {total_errors} errors= .") + + return total_errors diff --git a/BaseTools/Plugin/CodeQL/CodeQlAnalyze_plug_in.yaml b/BaseTools= /Plugin/CodeQL/CodeQlAnalyze_plug_in.yaml new file mode 100644 index 000000000000..ec01e55c53aa --- /dev/null +++ b/BaseTools/Plugin/CodeQL/CodeQlAnalyze_plug_in.yaml @@ -0,0 +1,13 @@ +## @file CodeQlAnalyze_plug_in.py +# +# Build plugin used to analyze CodeQL results. +# +# Copyright (c) Microsoft Corporation. All rights reserved. +# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +{ + "scope": "codeql-analyze", + "name": "CodeQL Analyze Plugin", + "module": "CodeQlAnalyzePlugin" +} diff --git a/BaseTools/Plugin/CodeQL/CodeQlBuildPlugin.py b/BaseTools/Plugi= n/CodeQL/CodeQlBuildPlugin.py new file mode 100644 index 000000000000..2fbf554f8fa4 --- /dev/null +++ b/BaseTools/Plugin/CodeQL/CodeQlBuildPlugin.py @@ -0,0 +1,172 @@ +# @file CodeQlBuildPlugin.py +# +# A build plugin that produces CodeQL results for the present build. +# +# Copyright (c) Microsoft Corporation. All rights reserved. +# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +import glob +import logging +import os +import stat +from common import codeql_plugin +from pathlib import Path + +from edk2toolext import edk2_logging +from edk2toolext.environment.plugintypes.uefi_build_plugin import \ + IUefiBuildPlugin +from edk2toolext.environment.uefi_build import UefiBuilder +from edk2toollib.uefi.edk2.path_utilities import Edk2Path +from edk2toollib.utility_functions import GetHostInfo, RemoveTree + + +class CodeQlBuildPlugin(IUefiBuildPlugin): + + def do_pre_build(self, builder: UefiBuilder) -> int: + """CodeQL pre-build functionality. + + Args: + builder (UefiBuilder): A UEFI builder object for this build. + + Returns: + int: The plugin return code. Zero indicates the plugin ran + successfully. A non-zero value indicates an unexpected error + occurred during plugin execution. + """ + + if not builder.SkipBuild: + pp =3D builder.pp.split(os.pathsep) + edk2_path =3D Edk2Path(builder.ws, pp) + + self.builder =3D builder + self.package =3D edk2_path.GetContainingPackage( + builder.mws.join(builder.ws, + builder.env.GetValue( + "ACTIVE_PLATFORM"))) + self.target =3D builder.env.GetValue("TARGET") + + self.build_output_dir =3D builder.env.GetValue("BUILD_OUTPUT_B= ASE") + + self.codeql_db_path =3D codeql_plugin.get_codeql_db_path( + builder.ws, self.package, self.target) + + edk2_logging.log_progress(f"{self.package} will be built for C= odeQL") + edk2_logging.log_progress(f" CodeQL database will be written = to " + f"{self.codeql_db_path}") + + self.codeql_path =3D codeql_plugin.get_codeql_cli_path() + if not self.codeql_path: + logging.critical("CodeQL build enabled but CodeQL CLI appl= ication " + "not found.") + return -1 + + # CodeQL can only generate a database on clean build + # + # Note: builder.CleanTree() cannot be used here as some platfo= rms + # have build steps that run before this plugin that store + # files in the build output directory. + # + # CodeQL does not care about with those files or many ot= hers such + # as the FV directory, build logs, etc. so instead focus= on + # removing only the directories with compilation/linker = output + # for the architectures being built (that need clean run= s for + # CodeQL to work). + targets =3D self.builder.env.GetValue("TARGET_ARCH").split(" ") + for target in targets: + directory_to_delete =3D Path(self.build_output_dir, target) + + if directory_to_delete.is_dir(): + logging.debug(f"Removing {str(directory_to_delete)} to= have a " + f"clean build for CodeQL.") + RemoveTree(str(directory_to_delete)) + + # CodeQL CLI does not handle spaces passed in CLI commands well + # (perhaps at all) as discussed here: + # 1. https://github.com/github/codeql-cli-binaries/issues/73 + # 2. https://github.com/github/codeql/issues/4910 + # + # Since it's unclear how quotes are handled and may change in = the + # future, this code is going to use the workaround to place the + # command in an executable file that is instead passed to Code= QL. + self.codeql_cmd_path =3D Path(builder.mws.join( + builder.ws, self.build_output_dir, + "codeql_build_command")) + + build_params =3D self._get_build_params() + + codeql_build_cmd =3D "" + if GetHostInfo().os =3D=3D "Windows": + self.codeql_cmd_path =3D self.codeql_cmd_path.parent / ( + self.codeql_cmd_path.name + '.bat') + elif GetHostInfo().os =3D=3D "Linux": + self.codeql_cmd_path =3D self.codeql_cmd_path.parent / ( + self.codeql_cmd_path.name + '.sh') + codeql_build_cmd +=3D f"#!/bin/bash{os.linesep * 2}" + codeql_build_cmd +=3D "build " + build_params + + self.codeql_cmd_path.parent.mkdir(exist_ok=3DTrue, parents=3DT= rue) + self.codeql_cmd_path.write_text(encoding=3D'utf8', data=3Dcode= ql_build_cmd) + + if GetHostInfo().os =3D=3D "Linux": + os.chmod(self.codeql_cmd_path, + os.stat(self.codeql_cmd_path).st_mode | stat.S_IEX= EC) + for f in glob.glob(os.path.join( + os.path.dirname(self.codeql_path), '**/*'), recursive= =3DTrue): + os.chmod(f, os.stat(f).st_mode | stat.S_IEXEC) + + codeql_params =3D (f'database create {self.codeql_db_path} ' + f'--language=3Dcpp ' + f'--source-root=3D{builder.ws} ' + f'--command=3D{self.codeql_cmd_path}') + + # Set environment variables so the CodeQL build command is pic= ked up + # as the active build command. + # + # Note: Requires recent changes in edk2-pytool-extensions (0.2= 0.0) + # to support reading these variables. + builder.env.SetValue( + "EDK_BUILD_CMD", self.codeql_path, "Set in CodeQL Build Pl= ugin") + builder.env.SetValue( + "EDK_BUILD_PARAMS", codeql_params, "Set in CodeQL Build Pl= ugin") + + return 0 + + def _get_build_params(self) -> str: + """Returns the build command parameters for this build. + + Based on the well-defined `build` command-line parameters. + + Returns: + str: A string representing the parameters for the build comman= d. + """ + build_params =3D f"-p {self.builder.env.GetValue('ACTIVE_PLATFORM'= )}" + build_params +=3D f" -b {self.target}" + build_params +=3D f" -t {self.builder.env.GetValue('TOOL_CHAIN_TAG= ')}" + + max_threads =3D self.builder.env.GetValue('MAX_CONCURRENT_THREAD_N= UMBER') + if max_threads is not None: + build_params +=3D f" -n {max_threads}" + + rt =3D self.builder.env.GetValue("TARGET_ARCH").split(" ") + for t in rt: + build_params +=3D " -a " + t + + if (self.builder.env.GetValue("BUILDREPORTING") =3D=3D "TRUE"): + build_params +=3D (" -y " + + self.builder.env.GetValue("BUILDREPORT_FILE")) + rt =3D self.builder.env.GetValue("BUILDREPORT_TYPES").split(" = ") + for t in rt: + build_params +=3D " -Y " + t + + # add special processing to handle building a single module + mod =3D self.builder.env.GetValue("BUILDMODULE") + if (mod is not None and len(mod.strip()) > 0): + build_params +=3D " -m " + mod + edk2_logging.log_progress("Single Module Build: " + mod) + + build_vars =3D self.builder.env.GetAllBuildKeyValues(self.target) + for key, value in build_vars.items(): + build_params +=3D " -D " + key + "=3D" + value + + return build_params diff --git a/BaseTools/Plugin/CodeQL/CodeQlBuild_plug_in.yaml b/BaseTools/P= lugin/CodeQL/CodeQlBuild_plug_in.yaml new file mode 100644 index 000000000000..13baa58d0cdf --- /dev/null +++ b/BaseTools/Plugin/CodeQL/CodeQlBuild_plug_in.yaml @@ -0,0 +1,13 @@ +## @file CodeQlBuild_plug_in.py +# +# Build plugin used to produce a CodeQL database from a build. +# +# Copyright (c) Microsoft Corporation. All rights reserved. +# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +{ + "scope": "codeql-build", + "name": "CodeQL Build Plugin", + "module": "CodeQlBuildPlugin" +} diff --git a/BaseTools/Plugin/CodeQL/CodeQlQueries.qls b/BaseTools/Plugin/C= odeQL/CodeQlQueries.qls new file mode 100644 index 000000000000..3f97bcd583d5 --- /dev/null +++ b/BaseTools/Plugin/CodeQL/CodeQlQueries.qls @@ -0,0 +1,75 @@ +--- +- description: C++ queries + +- queries: '.' + from: codeql/cpp-queries + +##########################################################################= ################ +# Queries +##########################################################################= ################ + +## Enable When Time is Available to Fix Issues +# Hundreds of issues. Most appear valid. Type: Recommendation. +#- include: +# id: cpp/missing-null-test + +## Errors +- include: + id: cpp/overrunning-write +- include: + id: cpp/overrunning-write-with-float +- include: + id: cpp/pointer-overflow-check +- include: + id: cpp/very-likely-overrunning-write + +## Warnings +- include: + id: cpp/conditionallyuninitializedvariable +- include: + id: cpp/infinite-loop-with-unsatisfiable-exit-condition +- include: + id: cpp/overflow-buffer + +# Note: Some queries above are not active by default with the below filter. +# Update the filter and run the queries again to get all results. +- include: + tags: + - "security" + - "correctness" + severity: + - "error" + - "warning" + - "recommendation" + +# Specifically hide the results of these. +# +# The following rules have been evaluated and explicitly not included for = the following reasons: +# - `cpp/allocation-too-small` - Appears to be hardcoded for C standard = library functions `malloc`, `calloc`, +# `realloc`, so it consumes time without much value with custom alloca= tion functions in the codebase. +# - `cpp/commented-out-code` - Triggers often. Needs further review. +# - `cpp/duplicate-include-guard` - The EntryPoint.h files includ= es a common include guard value +# `__MODULE_ENTRY_POINT_H__`. This was the only occurrence found. So n= ot very useful. +# - `cpp/invalid-pointer-deref` - Very limited results with what appear = to be false positives. +# - `cpp/use-of-goto` - Goto is valid and allowed in the codebase. +# - `cpp/useless-expression` - Triggers too often on cases where a NULL = lib implementation is provided for a function. +# Because the implementation simply returns, the check considers it us= eless. +# - `cpp/weak-crypto/*` - Crypto algorithms are tracked outside CodeQL. +- exclude: + id: cpp/allocation-too-small +- exclude: + id: cpp/commented-out-code +- exclude: + id: cpp/duplicate-include-guard +- exclude: + id: cpp/invalid-pointer-deref +- exclude: + id: cpp/use-of-goto +- exclude: + id: cpp/useless-expression +- exclude: + id: cpp/weak-crypto/banned-hash-algorithms +- exclude: + id: cpp/weak-crypto/capi/banned-modes +- exclude: + id: cpp/weak-crypto/openssl/banned-hash-algorithms diff --git a/BaseTools/Plugin/CodeQL/Readme.md b/BaseTools/Plugin/CodeQL/Re= adme.md new file mode 100644 index 000000000000..18587e2b258d --- /dev/null +++ b/BaseTools/Plugin/CodeQL/Readme.md @@ -0,0 +1,388 @@ +# CodeQL Plugin + +The set of CodeQL plugins provided include two main plugins that seamlessl= y integrate into a Stuart build environment: + +1. `CodeQlBuildPlugin` - Used to produce a CodeQL database from a build. +2. `CodeQlAnalyzePlugin` - Used to analyze a CodeQL database. + +While CodeQL can be run in a CI environment with other approaches. This pl= ugin offers the following advantages: + +1. Provides exactly the same results locally as on a CI server. +2. Integrates very well into VS Code. +3. Very simple to use - just use normal Stuart update and build commands. +4. Very simple to understand - minimally wraps the official CodeQL CLI. +5. Very simple to integrate - works like any other Stuart build plugin. + - Integration is usually just a few lines of code. +6. Portable - not tied to Azure DevOps specific, GitHub specific, or other= host infrastructure. +7. Versioned - the query and filters are versioned in source control so ea= sy to find and track. + +It is very important to read the Integration Instructions in this file and= determine how to best integrate the +CodeQL plugin into your environment. + +Due to the total size of dependencies required to run CodeQL and the flexi= bility needed by a platform to determine what +CodeQL queries to run and how to interpret results, a number of configurat= ion options are provided to allow a high +degree of flexibility during platform integration. + +This document is focused on those setting up the CodeQL plugin in their en= vironment. Once setup, end users simply need +to use their normal build commands and process and CodeQL will be integrat= ed with it. The most relevant section for +such users is [Local Development Tips](#local-development-tips). + +## Table of Contents + +1. [Database and Analysis Result Locations](#database-and-analysis-result-= locations) +2. [Global Configuration](#global-configuration) +3. [Package-Specific Configuration](#package-specific-configuration) +4. [Filter Patterns](#filter-patterns) +5. [Integration Instructions](#integration-instructions) + - [Integration Step 1 - Choose Scopes](#integration-step-1---choose-sco= pes) + - [Scopes Available](#scopes-available) + - [Integration Step 2 - Choose CodeQL Queries](#integration-step-2---ch= oose-codeql-queries) + - [Integration Step 3 - Determine Global Configuration Values](#integra= tion-step-3---determine-global-configuration-values) + - [Integration Step 4 - Determine Package-Specific Configuration Values= ](#integration-step-4---determine-package-specific-configuration-values) + - [Integration Step 5 - Testing](#integration-step-5---testing) + - [Integration Step 6 - Define Inclusion and Exclusion Filter Patterns]= (#integration-step-6---define-inclusion-and-exclusion-filter-patterns) +6. [High-Level Operation](#high-level-operation) + - [CodeQlBuildPlugin](#codeqlbuildplugin) + - [CodeQlAnalyzePlugin](#codeqlanalyzeplugin) +7. [Local Development Tips](#local-development-tips) +8. [Resolution Guidelines](#resolution-guidelines) + +## Database and Analysis Result Locations + +The CodeQL database is written to a directory unique to the package and ta= rget being built: + + `Build/codeql-db---` + +For example: `Build/codeql-db-mdemodulepkg-debug-0` + +The plugin does not delete or overwrite existing databases, the instance v= alue is simply increased. This is +because databases are large, take a long time to generate, and are importa= nt for reproducing analysis results. The user +is responsible for deleting database directories when they are no longer n= eeded. + +Similarly, analysis results are written to a directory unique to the packa= ge and target. For analysis, results are +stored in individual files so those files are stored in a single directory. + +For example, all analysis results for the above package and target will be= stored in: + `codeql-analysis-mdemodulepkg-debug` + +CodeQL results are stored in [SARIF](https://sarifweb.azurewebsites.net/) = (Static Analysis Results Interchange Format) +([CodeQL SARIF documentation](https://codeql.github.com/docs/codeql-cli/sa= rif-output/)) files. Each SARIF file +corresponding to a database will be stored in a file with an instance matc= hing the database instance. + +For example, the analysis result file for the above database would be stor= ed in this file: + `codeql-analysis-mdemodulepkg-debug/codeql-db-mdemodulepkg-debug-0.sarif` + +Result files are overwritten. This is because result files are quick to ge= nerate and need to represent the latest +results for the last analysis operation performed. The user is responsible= for backing up SARIF result files if they +need to saved. + +## Global Configuration + +Global configuration values are specified with build environment variables. + +These values are all optional. They provide a convenient mechanism for a b= uild script to set the value for all packages +built by the script. + +- `STUART_CODEQL_AUDIT_ONLY` - If `true` (case insensitive), `CodeQlAnalyz= ePlugin` will be in audit-only mode. In this + mode all CodeQL failures are ignored. +- `STUART_CODEQL_PATH` - The path to the CodeQL CLI application to use. +- `STUART_CODEQL_QUERY_SPECIFIERS` - The CodeQL CLI query specifiers to us= e. See [Running codeql database analyze](https://codeql.github.com/docs/cod= eql-cli/analyzing-databases-with-the-codeql-cli/#running-codeql-database-an= alyze) + for possible options. +- `STUART_CODEQL_FILTER_FILES` - The path to "filter" files that contains = filter patterns as described in + [Filter Patterns](#filter-patterns). + - More than one file may be specified by separating each absolute file p= ath with a comma. + - This might be useful to reference a global filter file from an upstr= eam repo and also include a global filter + file for the local repo. + - Filters are concatenated in the order of files in the variable. Patt= erns in later files can override patterns + in earlier files. + - The file only needs to contain a list of filter pattern strings under = a `"Filters"` key. For example: + + ```yaml + { + "Filters": [ + "", + "" + ] + } + ... + ``` + + Comments are allowed in the filter files and begin with `#` (like a no= rmal YAML file). + +## Package-Specific Configuration + +Package-specific configuration values reuse existing package-level configu= ration approaches to simplify adjusting +CodeQL plugin behavior per package. + +These values are all optional. They provide a convenient mechanism for a p= ackage owner to adjust settings specific to +the package. + +``` yaml + "CodeQlAnalyze": { + "AuditOnly": False, # Don't fail the build if there are erro= rs. Just log them. + "QuerySpecifiers": "" # Query specifiers to pass to CodeQL CLI. + "Filters": "" # Inclusion/exclusion filters + } +``` + +> _NOTE:_ If a global filter set is provided via `STUART_CODEQL_FILTER_FIL= ES` and a package has a package-specific +> list, then the package-specific filter list (in a package CI YAML file) = is appended onto the global filter list and +> may be used to override settings in the global list. + +The format used to specify items in `"Filters"` is specified in [Filter Pa= tterns](#filter-patterns). + +## Filter Patterns + +As you inspect results, you may want to include or exclude certain sets of= results. For example, exclude some files by +file path entirely or adjust the CodeQL rule applied to a certain file. Th= is plugin reuses logic from a popular +GitHub Action called [`filter-sarif`](https://github.com/advanced-security= /filter-sarif) to allow filtering as part of +the plugin analysis process. + +If any results are excluded using filters, the results are removed from th= e SARIF file. This allows the exclude results +seen locally to exactly match the results on the CI server. + +Read the ["Patterns"](https://github.com/advanced-security/filter-sarif#pa= tterns) section there for more details. The +patterns section is also copied below with some updates to make the inform= ation more relevant for an edk2 codebase +for convenience. + +Each pattern line is of the form: + +```plaintext +[+/-][:] +``` + +For example: + +```yaml +-**/*Test*.c:** # exclusion pattern: remove all alerts from al= l test files +-**/*Test*.c # ditto, short form of the line above ++**/*.c:cpp/infiniteloop # inclusion pattern: This line has precedence = over the first two + # and thus "allow lists" alerts of type "cpp/i= nfiniteloop" +**/*.c:cpp/infiniteloop # ditto, the "+" in inclusion patterns is opti= onal +** # allow all alerts in all files (reverses all = previous lines) +``` + +- The path separator character in patterns is always `/`, independent of t= he platform the code is running on and + independent of the paths in the SARIF file. +- `*` matches any character, except a path separator +- `**` matches any character and is only allowed between path separators, = e.g. `/**/file.txt`, `**/file.txt` or `**`. + NOT allowed: `**.txt`, `/etc**` +- The rule pattern is optional. If omitted, it will apply to alerts of all= types. +- Subsequent lines override earlier ones. By default all alerts are includ= ed. +- If you need to use the literals `+`, `-`, `\` or `:` in your pattern, yo= u can escape them with `\`, e.g. + `\-this/is/an/inclusion/file/pattern\:with-a-semicolon:and/a/rule/patter= n/with/a/\\/backslash`. For `+` and `-`, this + is only necessary if they appear at the beginning of the pattern line. + +## Integration Instructions + +First, note that most CodeQL CLI operations will take a long time the firs= t time they are run. This is due to: + +1. Downloads - Downloading the CodeQL CLI binary (during `stuart_update`) = and downloading CodeQL queries during + CodeQL plugin execution +2. Cache not established - CodeQL CLI caches data as it performs analysis.= The first time analysis is performed will + take more time than in the future. + +Second, these are build plugins. This means a build needs to take place fo= r the plugins to run. This typically happens +in the following two scenarios: + +1. `stuart_build` - A single package is built and the build process is sta= rted by the stuart tools. +2. `stuart_ci_build` - A number of packages may be built and the build pro= cess is started by the `CompilerPlugin`. + +In any case, each time a package is built, the CodeQL plugins will be run = if their scopes are active. + +### Integration Step 1 - Choose Scopes + +Decide which scopes need to be enabled in your platform, see [Scopes Avail= able](#scopes-available). + +Consider using a build profile to enable CodeQL so developers and pipeline= s can use the profile when they are +interested in CodeQL results but in other cases they can easily work witho= ut CodeQL in the way. + +Furthermore, build-script specific command-line parameters might be useful= to control CodeQL scopes and other +behavior. + +#### Scopes Available + +This CodeQL plugin leverages scopes to control major pieces of functionali= ty. Any combination of scopes can be +returned from the `GetActiveScopes()` function in the platform settings ma= nager to add and remove functionality. + +Plugin scopes: + +- `codeql-analyze` - Activate `CodeQlAnalyzePlugin` to perform post-build = analysis of the last generated database for + the package and target specified. +- `codeql-build` - Activate `CodeQlBuildPlugin` to hook the firmware build= in pre-build such that the build will + generate a CodeQL database during build. + +In most cases, to perform a full CodeQL run, `codeql-build` should be enab= led so a new CodeQL database is generated +during build and `codeql-analyze` should be be enabled so analysis of that= database is performed after the build is +completed. + +External dependency scopes: + +- `codeql-ext-dep` - Downloads the cross-platform CodeQL CLI as an externa= l dependency. +- `codeql-linux-ext-dep` - Downloads the Linux CodeQL CLI as an external d= ependency. +- `codeql-windows-ext-dep` - Downloads the Windows CodeQL CLI as an extern= al dependency. + +Note, that the CodeQL CLI is large in size. Sizes as of the [v2.11.2 relea= se](https://github.com/github/codeql-cli-binaries/releases/tag/v2.11.2). + +| Cross-platform | Linux | Windows | +|:--------------:|:------:|:-------:| +| 934 MB | 415 MB | 290 MB | + +Therefore, the following is recommended: + +1. **Ideal** - Create container images for build agents and install the Co= deQL CLI for the container OS into the + container. +2. Leverage host-OS detection (e.g. [`GetHostInfo()`](https://github.com/t= ianocore/edk2-pytool-library/blob/42ad6561af73ba34564f1577f64f7dbaf1d0a5a2/= edk2toollib/utility_functions.py#L112)) +to set the scope for the appropriate operating system. This will download = the much smaller OS-specific application. + +> _NOTE:_ You should never have more than one CodeQL external dependency s= cope enabled at a time. + +### Integration Step 2 - Choose CodeQL Queries + +Determine which queries need to be run against packages in your repo. In m= ost cases, the same set of queries will be +run against all packages. It is also possible to customize the queries run= at the package level. + +The default set of Project Mu CodeQL queries is specified in the `MuCodeQl= Queries.qls` file in this plugin. + +> _NOTE:_ The queries in `MuCodeQlQueries.qls` may change at any time. If = you do not want these changes to impact +> your platform, do not relay on option (3). + +The plugin decides what queries to run based on the following, in order of= preference: + +1. Package CI YAML file query specifier +2. Build environment variable query specifier +3. Plugin default query set file + +For details on how to set (1) and (2), see the Package CI Configuration an= d Environment Variable sections respectively. + +> _NOTE:_ The value specified is directly passed as a `query specifier` to= CodeQL CLI. Therefore, the arguments +> allowed by the `` argument of CodeQL CLI are allowed h= ere. See +> [Running codeql database analyze](https://codeql.github.com/docs/codeql-= cli/analyzing-databases-with-the-codeql-cli/#running-codeql-database-analyz= e). + +A likely scenario is that a platform needs to run local/closed source quer= ies in addition to the open-source queries. +There's various ways to handle that: + +1. Create a query specifier that includes all the queries needed, both pub= lic and private and use that query specifier, + either globally or at package-level. + + For example, at the global level - `STUART_CODEQL_QUERY_SPECIFIERS` =3D= _"Absolute_path_to_AllMyQueries.qls"_ + +2. Specify a query specifier that includes the closed sources queries and = reuse the public query list provided by + this plugin. + + For example, at the global level - `STUART_CODEQL_QUERY_SPECIFIERS` =3D= _"Absolute_path_to_MuCodeQlQueries.qls + Absolute_path_to_ClosedSourceQueries.qls"_ + +Refer to the CodeQL documentation noted above on query specifiers to devis= e other options. + +### Integration Step 3 - Determine Global Configuration Values + +Review the Environment Variable section to determine which, if any, global= values need to be set in your build script. + +### Integration Step 4 - Determine Package-Specific Configuration Values + +Review the Package CI Configuration section to determine which, if any, gl= obal values need to be set in your +package's CI YAML file. + +### Integration Step 5 - Testing + +Verify a `stuart_update` and `stuart_build` (or `stuart_ci_build`) command= work. + +### Integration Step 6 - Define Inclusion and Exclusion Filter Patterns + +After reviewing the test results from Step 5, determine if you need to app= ly any filters as described in +[Filter Patterns](#filter-patterns). + +## High-Level Operation + +This section summarizes the complete CodeQL plugin flow. This is to help d= evelopers understand basic theory of +operation behind the plugin and can be skipped by anyone not interested in= those details. + +### CodeQlBuildPlugin + +1. Register a pre-build hook +2. Determine the package and target being built +3. Determine the best CodeQL CLI path to use + - First choice, the `STUART_CODEQL_PATH` environment variable + - Note: This is set by the CodeQL CLI external dependency if that is = used + - Second choice, `codeql` as found on the system path +4. Determine the directory name for the CodeQL database + - Format: `Build/codeql-db---` +5. Clean the build directory of the active platform and target + - CodeQL database generation only works on clean builds +6. Ensure the "build" step is not skipped as a build is needed to generate= a CodeQL database +7. Build a CodeQL file that wraps around the edk2 build + - Written to the package build directory + - Example: `Build/MdeModulePkg/VS2022/codeql_build_command.bat` +8. Set the variables necessary for stuart to call CodeQL CLI during the bu= ild phase + - Sets `EDK_BUILD_CMD` and `EDK_BUILD_PARAMS` + +### CodeQlAnalyzePlugin + +1. Register a post-build hook +2. Determine the package and target being built +3. Determine the best CodeQL CLI path to use + - First choice, the `STUART_CODEQL_PATH` environment variable + - Note: This is set by the CodeQL CLI external dependency if that is = used + - Second choice, `codeql` as found on the system path +4. Determine the directory name for the most recent CodeQL database + - Format: `Build/codeql-db---` +5. Determine plugin audit status for the given package and target + - Check if `AuditOnly` is enabled either globally or for the package +6. Determine the CodeQL query specifiers to use for the given package and = target + - First choice, the package CI YAML file value + - Second choice, the `STUART_CODEQL_QUERY_SPECIFIERS` + - Third choice, use `CodeQlQueries.qls` (in the plugin directory) +7. Run CodeQL CLI to perform database analysis +8. Parse the analysis SARIF file to determine the number of CodeQL failures +9. Return the number of failures (or zero if `AuditOnly` is enabled) + +## Local Development Tips + +This section contains helpful tips to expedite common scenarios when worki= ng with CodeQL locally. + +1. Pre-build, Build, and Post-Build + + Generating a database requires the pre-build and build steps. Analyzing= a database requires the post-build step. + + Therefore, if you are making tweaks that don't affect the build, such a= s modifying the CodeQL queries used or level + of severity reported, you can save time by skipping pre-build and post-= build (e.g. `--skipprebuild` and + `--skipbuild`). + +2. Scopes + + Similar to (1), add/remove `codeql-build` and `codeql-analyze` from the= active scopes to save time depending on what + you are trying to do. + + If you are focusing on coding, remove the code CodeQL scopes if they ar= e active. If you are ready to check your + changes against CodeQL, simply add the scopes back. It is recommended t= o use build profiles to do this more + conveniently. + + If you already have CodeQL CLI enabled, you can remove the `codeql-ext-= dep` scope locally. The build will use the + `codeql` command on your path. + +3. CodeQL Output is in the CI Build Log + + To see exactly which queries CodeQL ran or why it might be taking longe= r than expected, look in the CI build log + (i.e. `Build/CI_BUILDLOG.txt`) where the CodeQL CLI application output = is written. + + Search for the text you see in the progress output (e.g. "Analyzing _Md= eModulePkg_ (_DEBUG_) CodeQL database at") + to jump to the section of the log just before the CodeQL CLI is invoked. + +4. Use a SARIF Viewer to Read Results + +The [SARIF Viewer extension for VS Code](https://marketplace.visualstudio.= com/items?itemName=3DMS-SarifVSCode.sarif-viewer) +can open the .sarif file generated by this plugin and allow you to click l= inks directly to the problem area in source +files. + +## Resolution Guidelines + +This section captures brief guidelines to keep in mind while resolving Cod= eQL issues. + +1. Look at surrounding code. Changes should always take into account the c= ontext of nearby code. The new logic may + need to account conditions not immediately obvious based on the issue a= lone. It is easy to focus only on the line + of code highlighted by CodeQL and miss the code's role in the big pictu= re. +2. A CodeQL alert may be benign but the code can be refactored to prevent = the alert. Often refactoring the code makes + the code intention clearer and avoids an unnecessary exception. +3. Consider adding unit tests while making CodeQL fixes especially for com= monly used code and code with a high volume + of CodeQL alerts. diff --git a/BaseTools/Plugin/CodeQL/analyze/__init__.py b/BaseTools/Plugin= /CodeQL/analyze/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/BaseTools/Plugin/CodeQL/analyze/analyze_filter.py b/BaseTools/= Plugin/CodeQL/analyze/analyze_filter.py new file mode 100644 index 000000000000..9a544e3192c6 --- /dev/null +++ b/BaseTools/Plugin/CodeQL/analyze/analyze_filter.py @@ -0,0 +1,176 @@ +# @file analyze_filter.py +# +# Filters results in a SARIF file. +# +# Based on code in: +# https://github.com/advanced-security/filter-sarif +# +# Specifically: +# https://github.com/advanced-security/filter-sarif/blob/main/filter_sar= if.py +# +# That code is licensed under: +# Apache License +# Version 2.0, January 2004 +# http://www.apache.org/licenses/ +# +# View the full and complete license as provided by that repository here: +# https://github.com/advanced-security/filter-sarif/blob/main/LICENSE +# +# This file has been altered from its original form. +# +# It primarily contains modifications made to integrate with the CodeQL pl= ugin. +# +# Copyright (c) Microsoft Corporation. All rights reserved. +# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +import json +import logging +import re +from os import PathLike +from typing import Iterable, List, Tuple + +from analyze.globber import match + + +def _match_path_and_rule( + path: str, rule: str, patterns: Iterable[str]) -> bool: + """Returns whether a given path matches a given rule. + + Args: + path (str): A file path string. + rule (str): A rule file path string. + patterns (Iterable[str]): An iterable of pattern strings. + + Returns: + bool: True if the path matches a rule. Otherwise, False. + """ + result =3D True + for s, fp, rp in patterns: + if match(rp, rule) and match(fp, path): + result =3D s + return result + + +def _parse_pattern(line: str) -> Tuple[str]: + """Parses a given pattern line. + + Args: + line (str): The line string that contains the rule. + + Returns: + Tuple[str]: The parsed sign, file pattern, and rule pattern from t= he + line. + """ + sep_char =3D ':' + esc_char =3D '\\' + file_pattern =3D '' + rule_pattern =3D '' + seen_separator =3D False + sign =3D True + + # inclusion or exclusion pattern? + u_line =3D line + if line: + if line[0] =3D=3D '-': + sign =3D False + u_line =3D line[1:] + elif line[0] =3D=3D '+': + u_line =3D line[1:] + + i =3D 0 + while i < len(u_line): + c =3D u_line[i] + i =3D i + 1 + if c =3D=3D sep_char: + if seen_separator: + raise Exception( + 'Invalid pattern: "' + line + '" Contains more than on= e ' + 'separator!') + seen_separator =3D True + continue + elif c =3D=3D esc_char: + next_c =3D u_line[i] if (i < len(u_line)) else None + if next_c in ['+' , '-', esc_char, sep_char]: + i =3D i + 1 + c =3D next_c + if seen_separator: + rule_pattern =3D rule_pattern + c + else: + file_pattern =3D file_pattern + c + + if not rule_pattern: + rule_pattern =3D '**' + + return sign, file_pattern, rule_pattern + + +def filter_sarif(input_sarif: PathLike, + output_sarif: PathLike, + patterns: List[str], + split_lines: bool) -> None: + """Filters a SARIF file with a given set of filter patterns. + + Args: + input_sarif (PathLike): Input SARIF file path. + output_sarif (PathLike): Output SARIF file path. + patterns (PathLike): List of filter pattern strings. + split_lines (PathLike): Whether to split lines in individual patte= rns. + """ + if split_lines: + tmp =3D [] + for p in patterns: + tmp =3D tmp + re.split('\r?\n', p) + patterns =3D tmp + + patterns =3D [_parse_pattern(p) for p in patterns if p] + + logging.debug('Given patterns:') + for s, fp, rp in patterns: + logging.debug( + 'files: {file_pattern} rules: {rule_pattern} ({sign})'.form= at( + file_pattern=3Dfp, + rule_pattern=3Drp, + sign=3D'positive' if s else 'negative')) + + with open(input_sarif, 'r') as f: + s =3D json.load(f) + + for run in s.get('runs', []): + if run.get('results', []): + new_results =3D [] + for r in run['results']: + if r.get('locations', []): + new_locations =3D [] + for l in r['locations']: + # TODO: The uri field is optional. We might have to + # fetch the actual uri from "artifacts" via + # "index" + # (see https://github.com/microsoft/sarif-tutorial= s/blob/main/docs/2-Basics.md#-linking-results-to-artifacts) + uri =3D l.get( + 'physicalLocation', {}).get( + 'artifactLocation', {}).get( + 'uri', None) + + # TODO: The ruleId field is optional and potential= ly + # ambiguous. We might have to fetch the actu= al + # ruleId from the rule metadata via the rule= Index + # field. + # (see https://github.com/microsoft/sarif-tutorial= s/blob/main/docs/2-Basics.md#rule-metadata) + ruleId =3D r['ruleId'] + + if (uri is None or + _match_path_and_rule(uri, ruleId, patterns)): + new_locations.append(l) + r['locations'] =3D new_locations + if new_locations: + new_results.append(r) + else: + # locations array doesn't exist or is empty, so we can= 't + # match on anything. Therefore, we include the result = in + # the output. + new_results.append(r) + run['results'] =3D new_results + + with open(output_sarif, 'w') as f: + json.dump(s, f, indent=3D2) diff --git a/BaseTools/Plugin/CodeQL/analyze/globber.py b/BaseTools/Plugin/= CodeQL/analyze/globber.py new file mode 100644 index 000000000000..25548fc9c754 --- /dev/null +++ b/BaseTools/Plugin/CodeQL/analyze/globber.py @@ -0,0 +1,132 @@ +# @file globber.py +# +# Provides global functionality for use by the CodeQL plugin. +# +# Copyright 2019 Jaakko Kangasharju +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# This file has been altered from its original form. +# +# Based on code in: +# https://github.com/advanced-security/filter-sarif +# +# Specifically: +# https://github.com/advanced-security/filter-sarif/blob/main/filter_sar= if.py +# +# That code is licensed under: +# Apache License +# Version 2.0, January 2004 +# http://www.apache.org/licenses/ +# +# This file has been altered from its original form. Primarily modificatio= ns +# made to integrate with the CodeQL plugin. +# +# Copyright (c) Microsoft Corporation. All rights reserved. +# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +import re + +_double_star_after_invalid_regex =3D re.compile(r'[^/\\]\*\*') +_double_star_first_before_invalid_regex =3D re.compile('^\\*\\*[^/]') +_double_star_middle_before_invalid_regex =3D re.compile(r'[^\\]\*\*[^/]') + + +def _match_component(pattern_component, file_name_component): + if len(pattern_component) =3D=3D 0 and len(file_name_component) =3D=3D= 0: + return True + elif len(pattern_component) =3D=3D 0: + return False + elif len(file_name_component) =3D=3D 0: + return pattern_component =3D=3D '*' + elif pattern_component[0] =3D=3D '*': + return (_match_component(pattern_component, file_name_component[1:= ]) or + _match_component(pattern_component[1:], file_name_componen= t)) + elif pattern_component[0] =3D=3D '?': + return _match_component(pattern_component[1:], file_name_component= [1:]) + elif pattern_component[0] =3D=3D '\\': + return (len(pattern_component) >=3D 2 and + pattern_component[1] =3D=3D file_name_component[0] and + _match_component( + pattern_component[2:], file_name_component[1:])) + elif pattern_component[0] !=3D file_name_component[0]: + return False + else: + return _match_component(pattern_component[1:], file_name_component= [1:]) + + +def _match_components(pattern_components, file_name_components): + if len(pattern_components) =3D=3D 0 and len(file_name_components) =3D= =3D 0: + return True + if len(pattern_components) =3D=3D 0: + return False + if len(file_name_components) =3D=3D 0: + return len(pattern_components) =3D=3D 1 and pattern_components[0] = =3D=3D '**' + if pattern_components[0] =3D=3D '**': + return (_match_components(pattern_components, file_name_components= [1:]) + or _match_components( + pattern_components[1:], file_name_components)) + else: + return ( + _match_component( + pattern_components[0], file_name_components[0]) and + _match_components( + pattern_components[1:], file_name_components[1:])) + + +def match(pattern: str, file_name: str): + """Match a glob pattern against a file name. + + Glob pattern matching is for file names, which do not need to exist as + files on the file system. + + A file name is a sequence of directory names, possibly followed by the= name + of a file, with the components separated by a path separator. A glob + pattern is similar, except it may contain special characters: A '?' ma= tches + any character in a name. A '*' matches any sequence of characters (pos= sibly + empty) in a name. Both of these match only within a single component, = i.e., + they will not match a path separator. A component in a pattern may als= o be + a literal '**', which matches zero or more components in the complete = file + name. A backslash '\\' in a pattern acts as an escape character, and + indicates that the following character is to be matched literally, eve= n if + it is a special character. + + Args: + pattern (str): The pattern to match. The path separator in pattern= s is + always '/'. + file_name (str): The file name to match against. The path separato= r in + file names is the platform separator + + Returns: + bool: True if the pattern matches, False otherwise. + """ + if (_double_star_after_invalid_regex.search(pattern) is not None or + _double_star_first_before_invalid_regex.search( + pattern) is not None or + _double_star_middle_before_invalid_regex.search(pattern) is not No= ne): + raise ValueError( + '** in {} not alone between path separators'.format(pattern)) + + pattern =3D pattern.rstrip('/') + file_name =3D file_name.rstrip('/') + + while '**/**' in pattern: + pattern =3D pattern.replace('**/**', '**') + + pattern_components =3D pattern.split('/') + + # We split on '\' as well as '/' to support unix and windows-style pat= hs + file_name_components =3D re.split(r'[\\/]', file_name) + + return _match_components(pattern_components, file_name_components) diff --git a/BaseTools/Plugin/CodeQL/codeqlcli_ext_dep.yaml b/BaseTools/Plu= gin/CodeQL/codeqlcli_ext_dep.yaml new file mode 100644 index 000000000000..37c7c9f595ca --- /dev/null +++ b/BaseTools/Plugin/CodeQL/codeqlcli_ext_dep.yaml @@ -0,0 +1,26 @@ +## @file codeqlcli_ext_dep.yaml +# +# Downloads the CodeQL Command-Line Interface (CLI) application that suppo= rt Linux, Windows, and Mac OS X. +# +# This download is very large but conveniently provides support for all op= erating systems. Use it if you +# need CodeQL CLI support without concern for the host operating system. +# +# In an environment where a platform might build in different operating sy= stems, it is recommended to set +# the scope for the appropriate CodeQL external dependency based on the ho= st operating system being used. +# +# Copyright (c) Microsoft Corporation. All rights reserved. +# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +{ + "scope": "codeql-ext-dep", + "type": "web", + "name": "codeql_cli", + "source": "https://github.com/github/codeql-cli-binaries/releases/downlo= ad/v2.12.4/codeql.zip", + "version": "2.12.4", + "sha256": "f682f1155d627ad97f10b1bcad97f682011986717bd3823e9cf831ed83ac9= 6e7", + "compression_type": "zip", + "internal_path": "/codeql/", + "flags": ["set_shell_var", ], + "var_name": "STUART_CODEQL_PATH" +} diff --git a/BaseTools/Plugin/CodeQL/codeqlcli_linux_ext_dep.yaml b/BaseToo= ls/Plugin/CodeQL/codeqlcli_linux_ext_dep.yaml new file mode 100644 index 000000000000..a6ca5d0f34cc --- /dev/null +++ b/BaseTools/Plugin/CodeQL/codeqlcli_linux_ext_dep.yaml @@ -0,0 +1,24 @@ +## @file codeqlcli_linux_ext_dep.yaml +# +# Downloads the Linux CodeQL Command-Line Interface (CLI) application. +# +# This download only supports Linux. In an environment where a platform mi= ght build in different operating +# systems, it is recommended to set the scope for the appropriate CodeQL e= xternal dependency based on the +# host operating system being used. +# +# Copyright (c) Microsoft Corporation. All rights reserved. +# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +{ + "scope": "codeql-linux-ext-dep", + "type": "web", + "name": "codeql_linux_cli", + "source": "https://github.com/github/codeql-cli-binaries/releases/downlo= ad/v2.14.5/codeql-linux64.zip", + "version": "2.14.5", + "sha256": "72aa5d748ff9ab57cfd86045560683bdc4897e0fe6d9f9a2786d9394674ae= 733", + "compression_type": "zip", + "internal_path": "/codeql/", + "flags": ["set_shell_var", ], + "var_name": "STUART_CODEQL_PATH" +} diff --git a/BaseTools/Plugin/CodeQL/codeqlcli_windows_ext_dep.yaml b/BaseT= ools/Plugin/CodeQL/codeqlcli_windows_ext_dep.yaml new file mode 100644 index 000000000000..e706a7cabf9f --- /dev/null +++ b/BaseTools/Plugin/CodeQL/codeqlcli_windows_ext_dep.yaml @@ -0,0 +1,24 @@ +## @file codeqlcli_windows_ext_dep.yaml +# +# Downloads the Windows CodeQL Command-Line Interface (CLI) application. +# +# This download only supports Windows. In an environment where a platform = might build in different operating +# systems, it is recommended to set the scope for the appropriate CodeQL e= xternal dependency based on the +# host operating system being used. +# +# Copyright (c) Microsoft Corporation. All rights reserved. +# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +{ + "scope": "codeql-windows-ext-dep", + "type": "web", + "name": "codeql_windows_cli", + "source": "https://github.com/github/codeql-cli-binaries/releases/downlo= ad/v2.14.5/codeql-win64.zip", + "version": "2.14.5", + "sha256": "861fcb38365cc311efee0c3a28c77494e93c69a969885b72e53173ad473f6= 1aa", + "compression_type": "zip", + "internal_path": "/codeql/", + "flags": ["set_shell_var", ], + "var_name": "STUART_CODEQL_PATH" +} diff --git a/BaseTools/Plugin/CodeQL/common/__init__.py b/BaseTools/Plugin/= CodeQL/common/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/BaseTools/Plugin/CodeQL/common/codeql_plugin.py b/BaseTools/Pl= ugin/CodeQL/common/codeql_plugin.py new file mode 100644 index 000000000000..c827cc30ae8b --- /dev/null +++ b/BaseTools/Plugin/CodeQL/common/codeql_plugin.py @@ -0,0 +1,74 @@ +# @file codeql_plugin.py +# +# Common logic shared across the CodeQL plugin. +# +# Copyright (c) Microsoft Corporation. All rights reserved. +# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +import os +import shutil +from os import PathLike + +from edk2toollib.utility_functions import GetHostInfo + + +def get_codeql_db_path(workspace: PathLike, package: str, target: str, + new_path: bool =3D True) -> str: + """Return the CodeQL database path for this build. + + Args: + workspace (PathLike): The workspace path. + package (str): The package name (e.g. "MdeModulePkg") + target (str): The target (e.g. "DEBUG") + new_path (bool, optional): Whether to create a new database path or + return an existing path. Defaults to Tr= ue. + + Returns: + str: The absolute path to the CodeQL database directory. + """ + codeql_db_dir_name =3D "codeql-db-" + package + "-" + target + codeql_db_dir_name =3D codeql_db_dir_name.lower() + codeql_db_path =3D os.path.join("Build", codeql_db_dir_name) + codeql_db_path =3D os.path.join(workspace, codeql_db_path) + + i =3D 0 + while os.path.isdir(f"{codeql_db_path + '-%s' % i}"): + i +=3D 1 + + if not new_path: + if i =3D=3D 0: + return None + else: + i -=3D 1 + + return codeql_db_path + f"-{i}" + + +def get_codeql_cli_path() -> str: + """Return the current CodeQL CLI path. + + Returns: + str: The absolute path to the CodeQL CLI application to use for + this build. + """ + # The CodeQL executable path can be passed via the + # STUART_CODEQL_PATH environment variable (to override with a + # custom value for this run) or read from the system path. + codeql_path =3D None + + if "STUART_CODEQL_PATH" in os.environ: + codeql_path =3D os.environ["STUART_CODEQL_PATH"] + + if GetHostInfo().os =3D=3D "Windows": + codeql_path =3D os.path.join(codeql_path, "codeql.exe") + else: + codeql_path =3D os.path.join(codeql_path, "codeql") + + if not os.path.isfile(codeql_path): + codeql_path =3D None + + if not codeql_path: + codeql_path =3D shutil.which("codeql") + + return codeql_path --=20 2.42.0.windows.2 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#109696): https://edk2.groups.io/g/devel/message/109696 Mute This Topic: https://groups.io/mt/102031057/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Fri May 17 10:44:22 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) client-ip=66.175.222.108; envelope-from=bounce+27952+109697+1787277+3901457@groups.io; helo=mail02.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+109697+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=linux.microsoft.com ARC-Seal: i=1; a=rsa-sha256; t=1697591117; cv=none; d=zohomail.com; s=zohoarc; b=myS2veIZvQjobNecF0gc5VoFG/fbxSsb3LjXDiCAGDqOGA3tZIJhAqHnfHWGw8TffClskHhZK+QUmhJQLCAnTt5PxipoZWYzGMX6f5QU6Rk7BkoECf/aNiC+mgsy3EZr5E2uy+GUpW5Ee2hz0B/9NN5gGXysg8VCSnDE/2qfgnk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1697591117; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:References:Sender:Subject:Subject:To:To:Message-Id; bh=CtOCU67EtcGH0FFS7ULZCiwNZSlowwyAxwwlct8k/Zg=; b=COpjMqNo9C6IRK3/jm9+5MbLaKUMvCfLKk2zZb41cFFBpIOxK/8sX+GJmzUg1DBE5v4xpzE3jp2uOjLBbVOGtYe//azqlcdcLE2Z8O8vKDkcOdxDgQ2tMEu9TmBAiqmEXnEUc2cm/FYr0SSYZ29R//OVsEmYgEbMKZqtAtEq5c8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+109697+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by mx.zohomail.com with SMTPS id 1697591117676408.1788927983597; Tue, 17 Oct 2023 18:05:17 -0700 (PDT) Return-Path: DKIM-Signature: a=rsa-sha256; bh=W0MNKVed1l70/0xqv8bHUbU0/8vOPBX+YeM+kFwk10Y=; c=relaxed/simple; d=groups.io; h=DKIM-Filter:From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References:MIME-Version:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Transfer-Encoding; s=20140610; t=1697591117; v=1; b=V68YxD8HTHm/nl0muB5PJAqlwjdkdVk6c+KuMkImZ/G/rXUMgAMDJRtDZeyH0PjgW0fSa257 4e+1EjNWD4t5nHCJx4KlkXASZcCvXIdxUvAWkKvlfG5IhbcqIooa0XuxlZO/RlEAcwgmP+aYILO XGDnF+aK8LHKb4/6Qz5zOdiM= X-Received: by 127.0.0.2 with SMTP id w9ATYY1788612xXxzfyYPjwT; Tue, 17 Oct 2023 18:05:17 -0700 X-Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by mx.groups.io with SMTP id smtpd.web11.271880.1697591116845273407 for ; Tue, 17 Oct 2023 18:05:16 -0700 X-Received: from localhost.localdomain (unknown [47.201.241.95]) by linux.microsoft.com (Postfix) with ESMTPSA id 8DAF620B74C1; Tue, 17 Oct 2023 18:05:15 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 8DAF620B74C1 From: "Michael Kubacki" To: devel@edk2.groups.io Cc: Bob Feng , Liming Gao , Michael D Kinney , Rebecca Cran , Sean Brogan , Yuwei Chen Subject: [edk2-devel] [PATCH v3 3/7] BaseTools/Plugin/CodeQL: Add integration helpers Date: Tue, 17 Oct 2023 21:04:41 -0400 Message-ID: <20231018010445.528-4-mikuback@linux.microsoft.com> In-Reply-To: <20231018010445.528-1-mikuback@linux.microsoft.com> References: <20231018010445.528-1-mikuback@linux.microsoft.com> MIME-Version: 1.0 Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,mikuback@linux.microsoft.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: TMUwAQbtckdWbXHr3daRvSCTx1787277AA= Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @groups.io) X-ZM-MESSAGEID: 1697591118522100003 Content-Type: text/plain; charset="utf-8" From: Michael Kubacki Adds a Python module to the CodeQL plugin directory that exports functions commonly needed for Stuart-based platforms to easily enable CodeQL in their platform build. This functionality has already moved to edk2-pytool-extensions https://github.com/tianocore/edk2-pytool-extensions in the `edk2toolext/codeql.py` file but edk2 is too far behind to use that. Additional integration changes are needed in edk2 and the series to add those has not made it past review. In the meantime, the functions are available locally in this commit and this commit can be reverted after edk2-pytool-extensions 0.24.1 or greater is used in edk2. Cc: Bob Feng Cc: Liming Gao Cc: Michael D Kinney Cc: Rebecca Cran Cc: Sean Brogan Cc: Yuwei Chen Signed-off-by: Michael Kubacki --- BaseTools/Plugin/CodeQL/integration/__init__.py | 0 BaseTools/Plugin/CodeQL/integration/stuart_codeql.py | 79 ++++++++++++++++= ++++ 2 files changed, 79 insertions(+) diff --git a/BaseTools/Plugin/CodeQL/integration/__init__.py b/BaseTools/Pl= ugin/CodeQL/integration/__init__.py new file mode 100644 index 000000000000..e69de29bb2d1 diff --git a/BaseTools/Plugin/CodeQL/integration/stuart_codeql.py b/BaseToo= ls/Plugin/CodeQL/integration/stuart_codeql.py new file mode 100644 index 000000000000..a3941d13157f --- /dev/null +++ b/BaseTools/Plugin/CodeQL/integration/stuart_codeql.py @@ -0,0 +1,79 @@ +# @file stuart_codeql.py +# +# Exports functions commonly needed for Stuart-based platforms to easily +# enable CodeQL in their platform build. +# +# Copyright (c) Microsoft Corporation. All rights reserved. +# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +from edk2toolext.environment.uefi_build import UefiBuilder +from edk2toollib.utility_functions import GetHostInfo +from argparse import ArgumentParser, Namespace +from typing import Tuple + + +def add_command_line_option(parser: ArgumentParser) -> None: + """Adds the CodeQL command to the platform command line options. + + Args: + parser (ArgumentParser): The argument parser used in this build. + + """ + parser.add_argument( + '--codeql', + dest=3D'codeql', + action=3D'store_true', + default=3DFalse, + help=3D"Optional - Produces CodeQL results from the build. See " + "BaseTools/Plugin/CodeQL/Readme.md for more info.") + + +def get_scopes(codeql_enabled: bool) -> Tuple[str]: + """Returns the active CodeQL scopes for this build. + + Args: + codeql_enabled (bool): Whether CodeQL is enabled. + + Returns: + Tuple[str]: A tuple of strings containing scopes that enable the + CodeQL plugin. + """ + active_scopes =3D () + + if codeql_enabled: + if GetHostInfo().os =3D=3D "Linux": + active_scopes +=3D ("codeql-linux-ext-dep",) + else: + active_scopes +=3D ("codeql-windows-ext-dep",) + active_scopes +=3D ("codeql-build", "codeql-analyze") + + return active_scopes + + +def is_codeql_enabled_on_command_line(args: Namespace) -> bool: + """Returns whether CodeQL was enabled on the command line. + + Args: + args (Namespace): Object holding a string representation of command + line arguments. + + Returns: + bool: True if CodeQL is enabled on the command line. Otherwise, fa= lse. + """ + return args.codeql + + +def set_audit_only_mode(uefi_builder: UefiBuilder) -> None: + """Configures the CodeQL plugin to run in audit only mode. + + Args: + uefi_builder (UefiBuilder): The UefiBuilder object for this platfo= rm + build. + + """ + + uefi_builder.env.SetValue( + "STUART_CODEQL_AUDIT_ONLY", + "true", + "Platform Defined") --=20 2.42.0.windows.2 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#109697): https://edk2.groups.io/g/devel/message/109697 Mute This Topic: https://groups.io/mt/102031058/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Fri May 17 10:44:22 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) client-ip=66.175.222.108; envelope-from=bounce+27952+109698+1787277+3901457@groups.io; helo=mail02.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+109698+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=linux.microsoft.com ARC-Seal: i=1; a=rsa-sha256; t=1697591121; cv=none; d=zohomail.com; s=zohoarc; b=DxvbdH+zBl+zjDh1ogYc9tz0PMOQzKW3EK0Tdzyu6bH1SdrWCtL8x4hpXSHi8Ysxfq1dL6P3AZ1oplrvh3IpiMeTIuBKUjuxKvIxS8/HyjusYuWGAvvRchlv4SywG4y961MQnvkQShEL1lx5mnCV91tcFw9gN/Ge3teeIeGcrNo= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1697591121; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:References:Sender:Subject:Subject:To:To:Message-Id; bh=NXaZYZ1e8xFIJa2xLX+hIvwkCKZgerMrnwpuTlujdQI=; b=CysSd6EduTKi8gz4/kUrmXv2+Cg24bWPMKPyjXJQG0lAfN9eRi4g80mpi6a+9Pov8aQClc+WoWZ8/0UY3YKdKZUQMWMMNU7Yr9DfARk5VzpJYFMmZhdnhJO1nXYS/7WAI3sSs/zaMBBJdcEHofROlyAIZgptCIDRmZsU0oEVkq8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+109698+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by mx.zohomail.com with SMTPS id 1697591121045259.4333362513887; Tue, 17 Oct 2023 18:05:21 -0700 (PDT) Return-Path: DKIM-Signature: a=rsa-sha256; bh=ozfmRod53fxBDN5Vn+sxx2uTIO72m0h856W9K0+a4x0=; c=relaxed/simple; d=groups.io; h=DKIM-Filter:From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References:MIME-Version:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Transfer-Encoding; s=20140610; t=1697591120; v=1; b=FRJjQwIjqXZgVaa0vnYbVWH3Cmzf2EGc2AETn8hvVGCIS4oEeL3dgX1O4l08T2j7QLyLldm/ ZM2qbG2A6vAmOEsencZ4g06/0TZZ2mvGqeghpyLoyUV9BqI8jWrTWRHAnnaAFclVs6DpWcBnMld pizDqLYfQzlNEBO2oBz8CgbU= X-Received: by 127.0.0.2 with SMTP id 08zcYY1788612xoOFG0aOwiQ; Tue, 17 Oct 2023 18:05:20 -0700 X-Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by mx.groups.io with SMTP id smtpd.web11.271881.1697591120206279469 for ; Tue, 17 Oct 2023 18:05:20 -0700 X-Received: from localhost.localdomain (unknown [47.201.241.95]) by linux.microsoft.com (Postfix) with ESMTPSA id 3E57520B74C0; Tue, 17 Oct 2023 18:05:19 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 3E57520B74C0 From: "Michael Kubacki" To: devel@edk2.groups.io Cc: Sean Brogan , Michael D Kinney , Liming Gao Subject: [edk2-devel] [PATCH v3 4/7] .pytool/CISettings.py: Integrate CodeQL Date: Tue, 17 Oct 2023 21:04:42 -0400 Message-ID: <20231018010445.528-5-mikuback@linux.microsoft.com> In-Reply-To: <20231018010445.528-1-mikuback@linux.microsoft.com> References: <20231018010445.528-1-mikuback@linux.microsoft.com> MIME-Version: 1.0 Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,mikuback@linux.microsoft.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: ecNHGnyKA9y05lKlvS9bzaXtx1787277AA= Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @groups.io) X-ZM-MESSAGEID: 1697591122496100003 Content-Type: text/plain; charset="utf-8" From: Michael Kubacki Adds the `--codeql` parameter to `stuart_update` and `stuart_ci_build`. - `stuart_update --codeql` - Downloads the CodeQL CLI locally. The command will pull the appropriate binary for the host OS. - `stuart_ci_build --codeql` - Runs CodeQL during the build resulting in a CodeQL database and SARIF result file in the `Build` directory. Cc: Sean Brogan Cc: Michael D Kinney Cc: Liming Gao Signed-off-by: Michael Kubacki --- .pytool/CISettings.py | 30 ++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/.pytool/CISettings.py b/.pytool/CISettings.py index c5803a877c36..b8b8080439c1 100644 --- a/.pytool/CISettings.py +++ b/.pytool/CISettings.py @@ -7,12 +7,27 @@ ## import os import logging +import sys from edk2toolext.environment import shell_environment from edk2toolext.invocables.edk2_ci_build import CiBuildSettingsManager from edk2toolext.invocables.edk2_setup import SetupSettingsManager, Requir= edSubmodule from edk2toolext.invocables.edk2_update import UpdateSettingsManager from edk2toolext.invocables.edk2_pr_eval import PrEvalSettingsManager from edk2toollib.utility_functions import GetHostInfo +from pathlib import Path + + +try: + # Temporarily needed until edk2 can update to the latest edk2-pytools + # that has the CodeQL helpers. + # + # May not be present until submodules are populated. + # + root =3D Path(__file__).parent.parent.resolve() + sys.path.append(str(root/'BaseTools'/'Plugin'/'CodeQL'/'integration')) + import stuart_codeql as codeql_helpers +except ImportError: + pass =20 =20 class Settings(CiBuildSettingsManager, UpdateSettingsManager, SetupSetting= sManager, PrEvalSettingsManager): @@ -34,6 +49,11 @@ class Settings(CiBuildSettingsManager, UpdateSettingsMan= ager, SetupSettingsManag group.add_argument("-force_piptools", "--fpt", dest=3D"force_pipto= ols", action=3D"store_true", default=3DFalse, help=3D"Force the system to u= se pip tools") group.add_argument("-no_piptools", "--npt", dest=3D"no_piptools", = action=3D"store_true", default=3DFalse, help=3D"Force the system to not use= pip tools") =20 + try: + codeql_helpers.add_command_line_option(parserObj) + except NameError: + pass + def RetrieveCommandLineOptions(self, args): super().RetrieveCommandLineOptions(args) if args.force_piptools: @@ -41,6 +61,11 @@ class Settings(CiBuildSettingsManager, UpdateSettingsMan= ager, SetupSettingsManag if args.no_piptools: self.UseBuiltInBaseTools =3D False =20 + try: + self.codeql =3D codeql_helpers.is_codeql_enabled_on_command_li= ne(args) + except NameError: + pass + # ####################################################################= ################### # # Default Support for this Ci Build = # # ####################################################################= ################### # @@ -169,6 +194,11 @@ class Settings(CiBuildSettingsManager, UpdateSettingsM= anager, SetupSettingsManag else: logging.warning("Falling back to using in-tree BaseTools") =20 + try: + scopes +=3D codeql_helpers.get_scopes(self.codeql) + except NameError: + pass + self.ActualScopes =3D scopes return self.ActualScopes =20 --=20 2.42.0.windows.2 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#109698): https://edk2.groups.io/g/devel/message/109698 Mute This Topic: https://groups.io/mt/102031059/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Fri May 17 10:44:22 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) client-ip=66.175.222.108; envelope-from=bounce+27952+109699+1787277+3901457@groups.io; helo=mail02.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+109699+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=linux.microsoft.com ARC-Seal: i=1; a=rsa-sha256; t=1697591123; cv=none; d=zohomail.com; s=zohoarc; b=C/0wvIbqSlX4aABpUzfSa15B5QxMmvHbtAq42zXo5W77LPSy11QF+MErHR/9mRB9XffrKZOHvc/tHkD/oCb6Zxp7Kz1qnpFmg8gufB4kOcEVeTjqQ/eAWLmdhd1LzDufpqbA0b6/Vo9JHJAbyxRpGC2UKlpv3KGjEmpGLvNfqoM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1697591123; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:References:Sender:Subject:Subject:To:To:Message-Id; bh=r48W3AuHfpQY2lJVcxZBgGdQyv+0BufqR9KNQVzSe+g=; b=e6jbXwhXkkOIxCeN81ifbCAIC86NSQ6523JXu0N1CdnKJZ7d84wm5iDqh4/3Rh+B7B6uMERpFXz15TvI6V8si2yVMqakaQi0FUKX4aqe3IgGnWTgJNWQvEx0FJFMAaQSryp3+FT/THkJhbfgr1FFK1V5E3LADYplQgqm864Q4RE= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+109699+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by mx.zohomail.com with SMTPS id 1697591123941670.190559411626; Tue, 17 Oct 2023 18:05:23 -0700 (PDT) Return-Path: DKIM-Signature: a=rsa-sha256; bh=iQiS+ACX1zIBJSJZ0LPfyRcBTNH6oG7f2FWdqUO8r4k=; c=relaxed/simple; d=groups.io; h=DKIM-Filter:From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References:MIME-Version:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Transfer-Encoding; s=20140610; t=1697591123; v=1; b=v2xznRXbyPzJcLlz9rdfUeW3LzN7CqWBDTN+NSP7tGGgcSWT7SSZK6yBfPgpD5sVUgrSH/zQ 9J602hEKSt1VfwEjxYZ+v6WNZoX7bTnH6hhwdivTpQ3k2ld83GROl9s2eEbWY+dRt6IbLhU7loI qEYXIo0ALgs+e86hVJT8pGFs= X-Received: by 127.0.0.2 with SMTP id 4I7HYY1788612xsXgGuCJmZp; Tue, 17 Oct 2023 18:05:23 -0700 X-Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by mx.groups.io with SMTP id smtpd.web10.272336.1697591123012868554 for ; Tue, 17 Oct 2023 18:05:23 -0700 X-Received: from localhost.localdomain (unknown [47.201.241.95]) by linux.microsoft.com (Postfix) with ESMTPSA id 3B01E20B74C2; Tue, 17 Oct 2023 18:05:22 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 3B01E20B74C2 From: "Michael Kubacki" To: devel@edk2.groups.io Cc: Sean Brogan , Michael D Kinney Subject: [edk2-devel] [PATCH v3 5/7] .github/workflows/codeql.yml: Add CodeQL workflow Date: Tue, 17 Oct 2023 21:04:43 -0400 Message-ID: <20231018010445.528-6-mikuback@linux.microsoft.com> In-Reply-To: <20231018010445.528-1-mikuback@linux.microsoft.com> References: <20231018010445.528-1-mikuback@linux.microsoft.com> MIME-Version: 1.0 Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,mikuback@linux.microsoft.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: 6suZ5QTt7ywkEH3BVbF05oAnx1787277AA= Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @groups.io) X-ZM-MESSAGEID: 1697591124480100006 Content-Type: text/plain; charset="utf-8" From: Michael Kubacki Adds a workflow to run CodeQL against all packages built in .pytool/CISettings.py. The following is done: 1. Determine which packages to build against. Those that support are managed by .pytool/CISettings.py will be selected. For each package: 2. Determine how to interact with the package. Such as whether `stuart_ci_setup` or `stuart_setup` should be used. 3. Perform supported Stuart steps for setup and update. 4. Discover the CodeQL plugin directory in the repo. 5. Attempt to load the CodeQL CLI specific to the host OS from a GitHub cache. 6. Perform the build. 7. Clean up some files after build to improve robustness. 8. Upload the CodeQL results (generated SARIF file) to GitHub Code Scanning. The results will be associated with the trigger of the workflow. After each step that can upload logs such as the setup, update, and build steps the logs are uploaded as an artifact to the workflow run. This allows easy debugging in case there's an error in the step. The SARIF file is also uploaded to the workflow run so it can be downloaded and analyzed. Cc: Sean Brogan Cc: Michael D Kinney Signed-off-by: Michael Kubacki --- .github/workflows/codeql.yml | 338 ++++++++++++++++++++ 1 file changed, 338 insertions(+) diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml new file mode 100644 index 000000000000..72ece9dcb446 --- /dev/null +++ b/.github/workflows/codeql.yml @@ -0,0 +1,338 @@ +# This workflow runs CodeQL against the repository. +# +# Results are uploaded to GitHub Code Scanning. +# +# Due to a known issue with the CodeQL extractor when building the edk2 +# codebase on Linux systems, only Windows agents are used for build with +# the VS toolchain. +# +# Copyright (c) Microsoft Corporation. +# SPDX-License-Identifier: BSD-2-Clause-Patent + +name: "CodeQL" + +on: + push: + branches: + - master + pull_request: + branches: + - master + paths-ignore: + - '!**.c' + - '!**.h' + +jobs: + analyze: + name: Analyze + runs-on: windows-2019 + permissions: + actions: read + contents: read + security-events: write + + strategy: + fail-fast: false + matrix: + include: + - Package: "ArmPkg" + ArchList: "IA32,X64" + - Package: "CryptoPkg" + ArchList: "IA32" + - Package: "CryptoPkg" + ArchList: "X64" + - Package: "DynamicTablesPkg" + ArchList: "IA32,X64" + - Package: "FatPkg" + ArchList: "IA32,X64" + - Package: "FmpDevicePkg" + ArchList: "IA32,X64" + - Package: "IntelFsp2Pkg" + ArchList: "IA32,X64" + - Package: "IntelFsp2WrapperPkg" + ArchList: "IA32,X64" + - Package: "MdeModulePkg" + ArchList: "IA32" + - Package: "MdeModulePkg" + ArchList: "X64" + - Package: "MdePkg" + ArchList: "IA32,X64" + - Package: "PcAtChipsetPkg" + ArchList: "IA32,X64" + - Package: "PrmPkg" + ArchList: "IA32,X64" + - Package: "SecurityPkg" + ArchList: "IA32,X64" + - Package: "ShellPkg" + ArchList: "IA32,X64" + - Package: "SourceLevelDebugPkg" + ArchList: "IA32,X64" + - Package: "StandaloneMmPkg" + ArchList: "IA32,X64" + - Package: "UefiCpuPkg" + ArchList: "IA32,X64" + - Package: "UnitTestFrameworkPkg" + ArchList: "IA32,X64" + + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Install Python + uses: actions/setup-python@v4 + with: + python-version: '3.11' + cache: 'pip' + cache-dependency-path: 'pip-requirements.txt' + + - name: Use Git Long Paths on Windows + if: runner.os =3D=3D 'Windows' + shell: pwsh + run: | + git config --system core.longpaths true + + - name: Install/Upgrade pip Modules + run: pip install -r pip-requirements.txt --upgrade requests + + - name: Determine CI Settings File Supported Operations + id: get_ci_file_operations + shell: python + run: | + import importlib + import os + import sys + from pathlib import Path + from edk2toolext.invocables.edk2_ci_setup import CiSetupSettingsMa= nager + from edk2toolext.invocables.edk2_setup import SetupSettingsManager + + # Find the repo CI Settings file + ci_settings_file =3D list(Path(os.environ['GITHUB_WORKSPACE']).rgl= ob('.pytool/CISettings.py')) + + # Note: At this point, submodules have not been pulled, only one C= I Settings file should exist + if len(ci_settings_file) !=3D 1 or not ci_settings_file[0].is_file= (): + print("::error title=3DWorkspace Error!::Failed to find CI Set= tings file!") + sys.exit(1) + + ci_settings_file =3D ci_settings_file[0] + + # Try Finding the Settings class in the file + module_name =3D 'ci_settings' + + spec =3D importlib.util.spec_from_file_location(module_name, ci_se= ttings_file) + module =3D importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + + try: + settings =3D getattr(module, 'Settings') + except AttributeError: + print("::error title=3DWorkspace Error!::Failed to find Settin= gs class in CI Settings file!") + sys.exit(1) + + # Determine Which Operations Are Supported by the Settings Class + ci_setup_supported =3D issubclass(settings, CiSetupSettingsManager) + setup_supported =3D issubclass(settings, SetupSettingsManager) + + with open(os.environ['GITHUB_OUTPUT'], 'a') as fh: + print(f'ci_setup_supported=3D{str(ci_setup_supported).lower()}= ', file=3Dfh) + print(f'setup_supported=3D{str(setup_supported).lower()}', fil= e=3Dfh) + + - name: Setup + if: steps.get_ci_file_operations.outputs.setup_supported =3D=3D 'tru= e' + run: stuart_setup -c .pytool/CISettings.py -t DEBUG -a ${{ matrix.Ar= chList }} TOOL_CHAIN_TAG=3DVS2019 + + - name: Upload Setup Log As An Artifact + uses: actions/upload-artifact@v3 + if: (success() || failure()) && steps.get_ci_file_operations.outputs= .setup_supported =3D=3D 'true' + with: + name: ${{ matrix.Package }}-Logs + path: | + **/SETUPLOG.txt + retention-days: 7 + if-no-files-found: ignore + + - name: CI Setup + if: steps.get_ci_file_operations.outputs.ci_setup_supported =3D=3D '= true' + run: stuart_ci_setup -c .pytool/CISettings.py -t DEBUG -a ${{ matrix= .ArchList }} TOOL_CHAIN_TAG=3DVS2019 + + - name: Upload CI Setup Log As An Artifact + uses: actions/upload-artifact@v3 + if: (success() || failure()) && steps.get_ci_file_operations.outputs= .ci_setup_supported =3D=3D 'true' + with: + name: ${{ matrix.Package }}-Logs + path: | + **/CISETUP.txt + retention-days: 7 + if-no-files-found: ignore + + - name: Update + run: stuart_update -c .pytool/CISettings.py -t DEBUG -a ${{ matrix.A= rchList }} TOOL_CHAIN_TAG=3DVS2019 + + - name: Upload Update Log As An Artifact + uses: actions/upload-artifact@v3 + if: success() || failure() + with: + name: ${{ matrix.Package }}-Logs + path: | + **/UPDATE_LOG.txt + retention-days: 7 + if-no-files-found: ignore + + - name: Build Tools From Source + run: python BaseTools/Edk2ToolsBuild.py -t VS2019 + + - name: Find CodeQL Plugin Directory + id: find_dir + shell: python + run: | + import os + import sys + from pathlib import Path + + # Find the plugin directory that contains the CodeQL plugin + plugin_dir =3D list(Path(os.environ['GITHUB_WORKSPACE']).rglob('Ba= seTools/Plugin/CodeQL')) + + # This should only be found once + if len(plugin_dir) =3D=3D 1: + plugin_dir =3D str(plugin_dir[0]) + + with open(os.environ['GITHUB_OUTPUT'], 'a') as fh: + print(f'codeql_plugin_dir=3D{plugin_dir}', file=3Dfh) + else: + print("::error title=3DWorkspace Error!::Failed to find CodeQL= plugin directory!") + sys.exit(1) + + - name: Get CodeQL CLI Cache Data + id: cache_key_gen + env: + CODEQL_PLUGIN_DIR: ${{ steps.find_dir.outputs.codeql_plugin_dir }} + shell: python + run: | + import os + import yaml + + codeql_cli_ext_dep_name =3D 'codeqlcli_windows_ext_dep' + codeql_plugin_file =3D os.path.join(os.environ['CODEQL_PLUGIN_DIR'= ], codeql_cli_ext_dep_name + '.yaml') + + with open (codeql_plugin_file) as pf: + codeql_cli_ext_dep =3D yaml.safe_load(pf) + + cache_key_name =3D codeql_cli_ext_dep['name'] + cache_key_version =3D codeql_cli_ext_dep['version'] + cache_key =3D f'{cache_key_name}-{cache_key_version}' + + codeql_plugin_cli_ext_dep_dir =3D os.path.join(os.environ['COD= EQL_PLUGIN_DIR'], codeql_cli_ext_dep['name'].strip() + '_extdep') + + with open(os.environ['GITHUB_OUTPUT'], 'a') as fh: + print(f'codeql_cli_cache_key=3D{cache_key}', file=3Dfh) + print(f'codeql_cli_ext_dep_dir=3D{codeql_plugin_cli_ext_de= p_dir}', file=3Dfh) + + - name: Attempt to Load CodeQL CLI From Cache + id: codeqlcli_cache + uses: actions/cache@v3 + with: + path: ${{ steps.cache_key_gen.outputs.codeql_cli_ext_dep_dir }} + key: ${{ steps.cache_key_gen.outputs.codeql_cli_cache_key }} + + - name: Download CodeQL CLI + if: steps.codeqlcli_cache.outputs.cache-hit !=3D 'true' + run: stuart_update -c .pytool/CISettings.py -t DEBUG -a ${{ matrix.A= rchList }} TOOL_CHAIN_TAG=3DVS2019 --codeql + + - name: Remove CI Plugins Irrelevant to CodeQL + shell: python + env: + CODEQL_PLUGIN_DIR: ${{ steps.find_dir.outputs.codeql_plugin_dir }} + run: | + import os + import shutil + from pathlib import Path + + # Only these two plugins are needed for CodeQL + plugins_to_keep =3D ['CompilerPlugin'] + + plugin_dir =3D Path('.pytool/Plugin').absolute() + if plugin_dir.is_dir(): + for dir in plugin_dir.iterdir(): + if str(dir.stem) not in plugins_to_keep: + shutil.rmtree(str(dir.absolute()), ignore_errors=3DTru= e) + + - name: CI Build + env: + STUART_CODEQL_PATH: ${{ steps.cache_key_gen.outputs.codeql_cli_ext= _dep_dir }} + run: stuart_ci_build -c .pytool/CISettings.py -t DEBUG -p ${{ matrix= .Package }} -a ${{ matrix.ArchList }} TOOL_CHAIN_TAG=3DVS2019 --codeql + + - name: Build Cleanup + id: build_cleanup + shell: python + run: | + import os + import shutil + from pathlib import Path + + dirs_to_delete =3D ['ia32', 'x64', 'arm', 'aarch64'] + + def delete_dirs(path: Path): + if path.exists() and path.is_dir(): + if path.name.lower() in dirs_to_delete: + print(f'Removed {str(path)}') + shutil.rmtree(path) + return + + for child_dir in path.iterdir(): + delete_dirs(child_dir) + + build_path =3D Path(os.environ['GITHUB_WORKSPACE'], 'Build') + delete_dirs(build_path) + + - name: Upload Build Logs As An Artifact + uses: actions/upload-artifact@v3 + if: success() || failure() + with: + name: ${{ matrix.Package }}-Logs + path: | + **/BUILD_REPORT.TXT + **/OVERRIDELOG.TXT + **/BUILDLOG_*.md + **/BUILDLOG_*.txt + **/CI_*.md + **/CI_*.txt + retention-days: 7 + if-no-files-found: ignore + + - name: Prepare Env Data for CodeQL Upload + id: env_data + env: + PACKAGE_NAME: ${{ matrix.Package }} + shell: python + run: | + import os + + package =3D os.environ['PACKAGE_NAME'].strip().lower() + directory_name =3D 'codeql-analysis-' + package + '-debug' + file_name =3D 'codeql-db-' + package + '-debug-0.sarif' + sarif_path =3D os.path.join('Build', directory_name, file_name) + + with open(os.environ['GITHUB_OUTPUT'], 'a') as fh: + if os.path.isfile(sarif_path): + print(f'upload_sarif_file=3Dtrue', file=3Dfh) + print(f'sarif_file_path=3D{sarif_path}', file=3Dfh) + else: + print(f'upload_sarif_file=3Dfalse', file=3Dfh) + + - name: Upload CodeQL Results (SARIF) As An Artifact + uses: actions/upload-artifact@v3 + if: steps.env_data.outputs.upload_sarif_file =3D=3D 'true' + with: + name: ${{ matrix.Package }}-CodeQL-SARIF + path: ${{ steps.env_data.outputs.sarif_file_path }} + retention-days: 14 + if-no-files-found: warn + + - name: Upload CodeQL Results (SARIF) To GitHub Code Scanning + uses: github/codeql-action/upload-sarif@v2 + if: steps.env_data.outputs.upload_sarif_file =3D=3D 'true' + with: + # Path to SARIF file relative to the root of the repository. + sarif_file: ${{ steps.env_data.outputs.sarif_file_path }} + # Optional category for the results. Used to differentiate multipl= e results for one commit. + # Each package is a separate category. + category: ${{ matrix.Package }} --=20 2.42.0.windows.2 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#109699): https://edk2.groups.io/g/devel/message/109699 Mute This Topic: https://groups.io/mt/102031061/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Fri May 17 10:44:22 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) client-ip=66.175.222.108; envelope-from=bounce+27952+109700+1787277+3901457@groups.io; helo=mail02.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+109700+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=linux.microsoft.com ARC-Seal: i=1; a=rsa-sha256; t=1697591128; cv=none; d=zohomail.com; s=zohoarc; b=ASEk/qFTvHbSWLx21w5e6CVyKCYCljxsldOSn1POHry2SKgxHJYynN6TVW3FU08rYzQEkXhf+SahH2fdbx8dQC0+TyNJ2v7CU63q9w+Y/lmbw7lY5f+OYfWJD/yEslT5kKcjxwbQ7wN/Skds3rnfca0EHryKG0TvAlAh6VaTuA0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1697591128; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:References:Sender:Subject:Subject:To:To:Message-Id; bh=fCGkyNUaKtwwTVjdWNPzqLkz6cVuKJeO4IteWEJqYTQ=; b=MU02VEGlB6OksaNfL0p3tKfOzL1sdxPaQf/wA68jtPvsfzUG4PtvMD134lZxrrgYXMEQDFLr/9PwvngMuZ+rgJ4dybrqrVz6zpS0Z/yK9usfZ122abAQ8N6lA/Jig3j3z07bFVF5D0q8LpY71Hor6P4EC6LO+h8eT9334BSLrQk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+109700+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by mx.zohomail.com with SMTPS id 1697591128955538.6444439902655; Tue, 17 Oct 2023 18:05:28 -0700 (PDT) Return-Path: DKIM-Signature: a=rsa-sha256; bh=lYHSs81rB7FApv213khW3qcFuJdbLnP7hc4ttM4geWk=; c=relaxed/simple; d=groups.io; h=DKIM-Filter:From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References:MIME-Version:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Transfer-Encoding; s=20140610; t=1697591128; v=1; b=TkE9q0MAkOFeo8zM8b3T1jJMH68vBRaptDcMeHPCWW70XDteUwv8bTw3OkliBvNAM+Fxn3Q4 0lGulFm5P6Q+Zqx/EaQ5CvKFHdBP+olYUr1qxe24X1wvsYN7ZdoPwzJ6a5hgdBNwG3nTUoEQl6e R0wMFYPziHfUAIhas9SpsiA4= X-Received: by 127.0.0.2 with SMTP id Q8kvYY1788612xSboMAtvnVx; Tue, 17 Oct 2023 18:05:28 -0700 X-Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by mx.groups.io with SMTP id smtpd.web11.271884.1697591128062697923 for ; Tue, 17 Oct 2023 18:05:28 -0700 X-Received: from localhost.localdomain (unknown [47.201.241.95]) by linux.microsoft.com (Postfix) with ESMTPSA id 35A2E20B74C0; Tue, 17 Oct 2023 18:05:27 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 35A2E20B74C0 From: "Michael Kubacki" To: devel@edk2.groups.io Cc: Sean Brogan , Michael D Kinney , Liming Gao Subject: [edk2-devel] [PATCH v3 6/7] .pytool/CISettings: Enable CodeQL audit mode Date: Tue, 17 Oct 2023 21:04:44 -0400 Message-ID: <20231018010445.528-7-mikuback@linux.microsoft.com> In-Reply-To: <20231018010445.528-1-mikuback@linux.microsoft.com> References: <20231018010445.528-1-mikuback@linux.microsoft.com> MIME-Version: 1.0 Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,mikuback@linux.microsoft.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: VXXruxWGNe8A3hZzBl5oMKFrx1787277AA= Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @groups.io) X-ZM-MESSAGEID: 1697591130493100001 Content-Type: text/plain; charset="utf-8" From: Michael Kubacki Since a large number of CodeQL queries are being enabled to identify issues that the community can collectively resolve, audit mode needs to be enabled to prevent the build from failing. In the future, this global audit mode can be disabled and individual packages can enable/disable audit mode in their package CI YAML file using the instructions in the CodeQL plugin readme. Cc: Sean Brogan Cc: Michael D Kinney Cc: Liming Gao Signed-off-by: Michael Kubacki --- .pytool/CISettings.py | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/.pytool/CISettings.py b/.pytool/CISettings.py index b8b8080439c1..ec3beb0dcf9d 100644 --- a/.pytool/CISettings.py +++ b/.pytool/CISettings.py @@ -196,6 +196,12 @@ class Settings(CiBuildSettingsManager, UpdateSettingsM= anager, SetupSettingsManag =20 try: scopes +=3D codeql_helpers.get_scopes(self.codeql) + + if self.codeql: + shell_environment.GetBuildVars().SetValue( + "STUART_CODEQL_AUDIT_ONLY", + "TRUE", + "Set in CISettings.py") except NameError: pass =20 --=20 2.42.0.windows.2 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#109700): https://edk2.groups.io/g/devel/message/109700 Mute This Topic: https://groups.io/mt/102031064/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- From nobody Fri May 17 10:44:22 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) client-ip=66.175.222.108; envelope-from=bounce+27952+109701+1787277+3901457@groups.io; helo=mail02.groups.io; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+109701+1787277+3901457@groups.io; dmarc=fail(p=none dis=none) header.from=linux.microsoft.com ARC-Seal: i=1; a=rsa-sha256; t=1697591132; cv=none; d=zohomail.com; s=zohoarc; b=Jt+zVLTt+f4xI11cAODnfrfxbYybljp0xNZDoxKwfHGRaYPnUXIOogXxhnoSF5RxSB2KFo+rL+Dz5uEj0lAI12HR0QuFIke5mH4PEoaVa8oNegAELC/lzT3VCZRwAlKhMAMhGmx5AebvYrG58+A2EGg+YcCti9IBf2FAFddxpuU= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1697591132; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Id:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Reply-To:Reply-To:References:Sender:Subject:Subject:To:To:Message-Id; bh=KnhbJS5kK3+bLSWOZ/JmkVjSX5nP2+oV+aTqAcJBK0I=; b=eSPsFZMfwUTs0hy3w+E2xJNdZxyI4QRdCOPTA2850sAgeQ5HbJ66jhmQlP6waUQERL0rCfDj9iYE12KarsLhaoFFjn9Usu6itYh5IIzvfQbAb94p9znD/7BTwi056SoMbHSE1k33YCEfq9wYho3xAk4mk7AHEor3j+H14eGDcik= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce+27952+109701+1787277+3901457@groups.io; dmarc=fail header.from= (p=none dis=none) Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by mx.zohomail.com with SMTPS id 1697591132884421.42970488444826; Tue, 17 Oct 2023 18:05:32 -0700 (PDT) Return-Path: DKIM-Signature: a=rsa-sha256; bh=tnkXvUehD/ZJ8NjVLICzEjgauh9PMfQFkmLFig+dctE=; c=relaxed/simple; d=groups.io; h=DKIM-Filter:From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References:MIME-Version:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Transfer-Encoding; s=20140610; t=1697591132; v=1; b=MTNHBo+caNsPwH9Ha5CByvzBgguoQZWNEshrLnNuwycndtE3sJNy+CQZUQU/+TTEF/ANOqdn IpWqkAmT9sP4IjZ3asifg9gaT4j4LOv5Ftcw5m52peRGFJPgIC1PZEMgVAv6l+abuLr7fSdKAwH axodrAFEuyUkNaVeKdUTkzXM= X-Received: by 127.0.0.2 with SMTP id 9AvcYY1788612x5duUaweu7G; Tue, 17 Oct 2023 18:05:32 -0700 X-Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by mx.groups.io with SMTP id smtpd.web10.272343.1697591132006577573 for ; Tue, 17 Oct 2023 18:05:32 -0700 X-Received: from localhost.localdomain (unknown [47.201.241.95]) by linux.microsoft.com (Postfix) with ESMTPSA id C069520B74C1; Tue, 17 Oct 2023 18:05:30 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com C069520B74C1 From: "Michael Kubacki" To: devel@edk2.groups.io Cc: Bob Feng , Liming Gao , Michael D Kinney , Rebecca Cran , Sean Brogan , Yuwei Chen Subject: [edk2-devel] [PATCH v3 7/7] BaseTools/Plugin/CodeQL: Enable 30 queries Date: Tue, 17 Oct 2023 21:04:45 -0400 Message-ID: <20231018010445.528-8-mikuback@linux.microsoft.com> In-Reply-To: <20231018010445.528-1-mikuback@linux.microsoft.com> References: <20231018010445.528-1-mikuback@linux.microsoft.com> MIME-Version: 1.0 Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,mikuback@linux.microsoft.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: ncK0oooazJg5vGFUCSblkZMsx1787277AA= Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @groups.io) X-ZM-MESSAGEID: 1697591134500100001 Content-Type: text/plain; charset="utf-8" From: Michael Kubacki Updates the CodeQL queries opted into by edk2 to a set of queries from the standard CodeQL query package `codeql/cpp-queries`. After testing a large number of queries the included set here were found to be the most useful with the least number of false positives. Some queries had a number of issues that led to them being placed on the exclusion list so that they are not considered in the future without the notes there being taken into account. General details about queries available in the pack are available here: https://codeql.github.com/codeql-query-help/cpp/ The issues found by these queries will need to be fixed over time. In the meantime, the results will show to those that have permission in the repo's GitHub Code Scanning area. The build will not fail due to CodeQL issues (since they are not all fixed) but that can be enabled in the future. Cc: Bob Feng Cc: Liming Gao Cc: Michael D Kinney Cc: Rebecca Cran Cc: Sean Brogan Cc: Yuwei Chen Signed-off-by: Michael Kubacki --- BaseTools/Plugin/CodeQL/CodeQlQueries.qls | 57 +++++++++++++++++--- 1 file changed, 50 insertions(+), 7 deletions(-) diff --git a/BaseTools/Plugin/CodeQL/CodeQlQueries.qls b/BaseTools/Plugin/C= odeQL/CodeQlQueries.qls index 3f97bcd583d5..1a5098322193 100644 --- a/BaseTools/Plugin/CodeQL/CodeQlQueries.qls +++ b/BaseTools/Plugin/CodeQL/CodeQlQueries.qls @@ -8,28 +8,71 @@ # Queries ##########################################################################= ################ =20 -## Enable When Time is Available to Fix Issues -# Hundreds of issues. Most appear valid. Type: Recommendation. -#- include: -# id: cpp/missing-null-test - ## Errors - include: - id: cpp/overrunning-write + id: cpp/badoverflowguard - include: - id: cpp/overrunning-write-with-float + id: cpp/infiniteloop +- include: + id: cpp/likely-bugs/memory-management/v2/conditionally-uninitialized-v= ariable +- include: + id: cpp/missing-null-test +- include: + id: cpp/missing-return +- include: + id: cpp/no-space-for-terminator - include: id: cpp/pointer-overflow-check +- include: + id: cpp/redundant-null-check-simple +- include: + id: cpp/sizeof/const-int-argument +- include: + id: cpp/sizeof/sizeof-or-operation-as-argument +- include: + id: cpp/unguardednullreturndereferenc - include: id: cpp/very-likely-overrunning-write =20 ## Warnings +- include: + id: cpp/comparison-with-wider-type - include: id: cpp/conditionallyuninitializedvariable +- include: + id: cpp/comparison-precedence +- include: + id: cpp/implicit-bitfield-downcast - include: id: cpp/infinite-loop-with-unsatisfiable-exit-condition +- include: + id: cpp/offset-use-before-range-check - include: id: cpp/overflow-buffer +- include: + id: cpp/overflow-calculated +- include: + id: cpp/overflow-destination +- include: + id: cpp/paddingbyteinformationdisclosure +- include: + id: cpp/return-stack-allocated-memory +- include: + id: cpp/static-buffer-overflow +- include: + id: cpp/unsigned-comparison-zero +- include: + id: cpp/uselesstest + +## Recommendations +- include: + id: cpp/missing-header-guard +- include: + id: cpp/unused-local-variable +- include: + id: cpp/unused-static-function +- include: + id: cpp/unused-static-variable =20 # Note: Some queries above are not active by default with the below filter. # Update the filter and run the queries again to get all results. --=20 2.42.0.windows.2 -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#109701): https://edk2.groups.io/g/devel/message/109701 Mute This Topic: https://groups.io/mt/102031065/1787277 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-