1. Patchew
  2. EDK2
  3. View series

[edk2-devel] [PATCH v3] BaseTools/Plugin: Too many execute filess cause "cmd too long" failure

Guo, Gua posted 1 patch 11 months, 3 weeks ago
Failed in applying to current master (apply log)
.../HostBasedUnitTestRunner.py                | 31 ++++++++++++++++---
1 file changed, 27 insertions(+), 4 deletions(-)
[edk2-devel] [PATCH v3] BaseTools/Plugin: Too many execute filess cause "cmd too long" failure
Posted by Guo, Gua 11 months, 3 weeks ago 
From: Gua Guo <gua.guo@intel.com>

Windows command prompt have 8191 characters limitation,
enhance it to make command too long can be resloved.

Provide an example, if have too many cov files, it causes to run single
command over the 8191 characters limitation.
> OpenCppCoverage
>  --export_type binary:coverage.cov
>  --working_dir={workspace}Build
>  --input_coverage=AAA.cov
>  ...
>  --input_coverage=NNN.cov

The solution is passing many coverage files in single command line to
breaking it up into many command lines with one coverage file per
command line in order to prevent single line is over to 8191 characters.

- Command Line 1
> OpenCppCoverage
>  --export_type binary:coverage.cov
>  --working_dir={workspace}Build
>  --input_coverage=AAA.cov
>  --input_coverage=coverage.cov
...

- Command Line N
> OpenCppCoverage
>  --export_type binary:coverage.cov
>  --working_dir={workspace}Build
>  --input_coverage=NNN.cov
>  --input_coverage=coverage.cov

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Michael Kubacki <mikuback@linux.microsoft.com>
Signed-off-by: Gua Guo <gua.guo@intel.com>
---
 .../HostBasedUnitTestRunner.py                | 31 ++++++++++++++++---
 1 file changed, 27 insertions(+), 4 deletions(-)

diff --git a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
index d993de9412..05bb6da50a 100644
--- a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
+++ b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
@@ -209,13 +209,25 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
         coverageFile = ""
         for testFile in testList:
             ret = RunCmd("OpenCppCoverage", f"--source {workspace} --export_type binary:{testFile}.cov -- {testFile}")
-            coverageFile += " --input_coverage=" + testFile + ".cov"
+            if ret != 0:
+                logging.error("UnitTest Coverage: Failed to collect coverage data.")
+                return 1
+
+            coverageFile  = f" --input_coverage={testFile}.cov"
+            if (os.path.isfile(f"{os.path.join(buildOutputBase, 'coverage.cov')}")):
+                coverageFile += f" --input_coverage={os.path.join(buildOutputBase, 'coverage.cov')}"
+            ret = RunCmd("OpenCppCoverage", f"--export_type binary:{os.path.join(buildOutputBase, 'coverage.cov')} --working_dir={workspace}Build {coverageFile}")
             if ret != 0:
                 logging.error("UnitTest Coverage: Failed to collect coverage data.")
                 return 1
 
         # Generate and XML file if requested.by each package
-        ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{os.path.join(buildOutputBase, 'coverage.xml')} --working_dir={workspace}Build {coverageFile}")
+        ret = RunCmd(
+            "OpenCppCoverage",
+            f"--export_type cobertura:{os.path.join(buildOutputBase, 'coverage.xml')} " +
+            f"--working_dir={workspace}Build " +
+            f"--input_coverage={os.path.join(buildOutputBase, 'coverage.cov')}"
+            )
         if ret != 0:
             logging.error("UnitTest Coverage: Failed to generate cobertura format xml in single package.")
             return 1
@@ -224,9 +236,20 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
         testCoverageList = glob.glob(os.path.join(workspace, "Build", "**","*Test*.exe.cov"), recursive=True)
         coverageFile = ""
         for testCoverage in testCoverageList:
-            coverageFile += " --input_coverage=" + testCoverage
+            coverageFile  = f" --input_coverage={testCoverage}"
+            if (os.path.isfile(f"{workspace}Build/coverage.cov")):
+                coverageFile += f" --input_coverage={workspace}Build/coverage.cov"
+            ret = RunCmd("OpenCppCoverage", f"--export_type binary:{workspace}Build/coverage.cov --working_dir={workspace}Build {coverageFile}")
+            if ret != 0:
+                logging.error("UnitTest Coverage: Failed to collect coverage data.")
+                return 1
 
-        ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{workspace}Build/coverage.xml --working_dir={workspace}Build {coverageFile}")
+        ret = RunCmd(
+            "OpenCppCoverage",
+            f"--export_type cobertura:{workspace}Build/coverage.xml " +
+            f"--working_dir={workspace}Build " +
+            f"--input_coverage={workspace}Build/coverage.cov"
+            )
         if ret != 0:
             logging.error("UnitTest Coverage: Failed to generate cobertura format xml.")
             return 1
-- 
2.39.2.windows.1



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#104288): https://edk2.groups.io/g/devel/message/104288
Mute This Topic: https://groups.io/mt/98769942/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [edk2-devel] [PATCH v3] BaseTools/Plugin: Too many execute filess cause "cmd too long" failure
Posted by Michael Kubacki 11 months, 3 weeks ago 
I understand it was common in some parts of the code base at one point 
to wrap all expressions in parentheses, but can it be avoided in newly 
added code?

For example make:

if (os.path.isfile(f"{os.path.join(buildOutputBase, 'coverage.cov')}")):

Be:

if os.path.isfile(f"{os.path.join(buildOutputBase, 'coverage.cov')}"):

The first style is not consistent with what you added elsewhere:

if ret != 0:

And the parentheses are unnecessary in this case and don't follow 
typical Python style.

---

Due to the number of references, it would be easier to follow if this 
path were assigned to a variable:

os.path.join(buildOutputBase, 'coverage.cov')

---

For consistency/compatibility, please use os.path.join() (or a similar 
abstraction) to build paths in cases like the following:

if (os.path.isfile(f"{workspace}Build/coverage.cov")):

---

Did you see any significant performance impact with this change?

Thanks,
Michael

On 5/8/2023 4:18 PM, gua.guo@intel.com wrote:
> From: Gua Guo <gua.guo@intel.com>
> 
> Windows command prompt have 8191 characters limitation,
> enhance it to make command too long can be resloved.
> 
> Provide an example, if have too many cov files, it causes to run single
> command over the 8191 characters limitation.
>> OpenCppCoverage
>>   --export_type binary:coverage.cov
>>   --working_dir={workspace}Build
>>   --input_coverage=AAA.cov
>>   ...
>>   --input_coverage=NNN.cov
> 
> The solution is passing many coverage files in single command line to
> breaking it up into many command lines with one coverage file per
> command line in order to prevent single line is over to 8191 characters.
> 
> - Command Line 1
>> OpenCppCoverage
>>   --export_type binary:coverage.cov
>>   --working_dir={workspace}Build
>>   --input_coverage=AAA.cov
>>   --input_coverage=coverage.cov
> ...
> 
> - Command Line N
>> OpenCppCoverage
>>   --export_type binary:coverage.cov
>>   --working_dir={workspace}Build
>>   --input_coverage=NNN.cov
>>   --input_coverage=coverage.cov
> 
> Cc: Michael D Kinney <michael.d.kinney@intel.com>
> Cc: Sean Brogan <sean.brogan@microsoft.com>
> Cc: Michael Kubacki <mikuback@linux.microsoft.com>
> Signed-off-by: Gua Guo <gua.guo@intel.com>
> ---
>   .../HostBasedUnitTestRunner.py                | 31 ++++++++++++++++---
>   1 file changed, 27 insertions(+), 4 deletions(-)
> 
> diff --git a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
> index d993de9412..05bb6da50a 100644
> --- a/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
> +++ b/BaseTools/Plugin/HostBasedUnitTestRunner/HostBasedUnitTestRunner.py
> @@ -209,13 +209,25 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
>           coverageFile = ""
> 
>           for testFile in testList:
> 
>               ret = RunCmd("OpenCppCoverage", f"--source {workspace} --export_type binary:{testFile}.cov -- {testFile}")
> 
> -            coverageFile += " --input_coverage=" + testFile + ".cov"
> 
> +            if ret != 0:
> 
> +                logging.error("UnitTest Coverage: Failed to collect coverage data.")
> 
> +                return 1
> 
> +
> 
> +            coverageFile  = f" --input_coverage={testFile}.cov"
> 
> +            if (os.path.isfile(f"{os.path.join(buildOutputBase, 'coverage.cov')}")):
> 
> +                coverageFile += f" --input_coverage={os.path.join(buildOutputBase, 'coverage.cov')}"
> 
> +            ret = RunCmd("OpenCppCoverage", f"--export_type binary:{os.path.join(buildOutputBase, 'coverage.cov')} --working_dir={workspace}Build {coverageFile}")
> 
>               if ret != 0:
> 
>                   logging.error("UnitTest Coverage: Failed to collect coverage data.")
> 
>                   return 1
> 
>   
> 
>           # Generate and XML file if requested.by each package
> 
> -        ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{os.path.join(buildOutputBase, 'coverage.xml')} --working_dir={workspace}Build {coverageFile}")
> 
> +        ret = RunCmd(
> 
> +            "OpenCppCoverage",
> 
> +            f"--export_type cobertura:{os.path.join(buildOutputBase, 'coverage.xml')} " +
> 
> +            f"--working_dir={workspace}Build " +
> 
> +            f"--input_coverage={os.path.join(buildOutputBase, 'coverage.cov')}"
> 
> +            )
> 
>           if ret != 0:
> 
>               logging.error("UnitTest Coverage: Failed to generate cobertura format xml in single package.")
> 
>               return 1
> 
> @@ -224,9 +236,20 @@ class HostBasedUnitTestRunner(IUefiBuildPlugin):
>           testCoverageList = glob.glob(os.path.join(workspace, "Build", "**","*Test*.exe.cov"), recursive=True)
> 
>           coverageFile = ""
> 
>           for testCoverage in testCoverageList:
> 
> -            coverageFile += " --input_coverage=" + testCoverage
> 
> +            coverageFile  = f" --input_coverage={testCoverage}"
> 
> +            if (os.path.isfile(f"{workspace}Build/coverage.cov")):
> 
> +                coverageFile += f" --input_coverage={workspace}Build/coverage.cov"
> 
> +            ret = RunCmd("OpenCppCoverage", f"--export_type binary:{workspace}Build/coverage.cov --working_dir={workspace}Build {coverageFile}")
> 
> +            if ret != 0:
> 
> +                logging.error("UnitTest Coverage: Failed to collect coverage data.")
> 
> +                return 1
> 
>   
> 
> -        ret = RunCmd("OpenCppCoverage", f"--export_type cobertura:{workspace}Build/coverage.xml --working_dir={workspace}Build {coverageFile}")
> 
> +        ret = RunCmd(
> 
> +            "OpenCppCoverage",
> 
> +            f"--export_type cobertura:{workspace}Build/coverage.xml " +
> 
> +            f"--working_dir={workspace}Build " +
> 
> +            f"--input_coverage={workspace}Build/coverage.cov"
> 
> +            )
> 
>           if ret != 0:
> 
>               logging.error("UnitTest Coverage: Failed to generate cobertura format xml.")
> 
>               return 1
> 


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#104399): https://edk2.groups.io/g/devel/message/104399
Mute This Topic: https://groups.io/mt/98769942/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

Patchew 2.1-devel-0870f84

© 2016 - 2024 Red Hat, Inc.