[edk2-devel] [Patch 00/10 V3] Enable multiple process AutoGen

Bob Feng posted 10 patches 4 years, 8 months ago
Failed in applying to current master (apply log)
There is a newer version of this series
BaseTools/Source/Python/AutoGen/AutoGen.py    | 4227 +----------------
.../Source/Python/AutoGen/AutoGenWorker.py    |  222 +
.../Source/Python/AutoGen/BuildEngine.py      |   22 +
BaseTools/Source/Python/AutoGen/DataPipe.py   |  153 +
BaseTools/Source/Python/AutoGen/GenC.py       |    6 +-
BaseTools/Source/Python/AutoGen/GenMake.py    |    3 +-
.../Source/Python/AutoGen/ModuleAutoGen.py    | 1903 ++++++++
.../Python/AutoGen/ModuleAutoGenHelper.py     |  616 +++
.../Source/Python/AutoGen/PlatformAutoGen.py  | 1493 ++++++
.../Source/Python/AutoGen/WorkspaceAutoGen.py |  905 ++++
BaseTools/Source/Python/Common/EdkLogger.py   |  119 +-
BaseTools/Source/Python/Common/Misc.py        |    1 -
.../Python/Common/TargetTxtClassObject.py     |   28 +-
.../Python/Common/ToolDefClassObject.py       |    6 +-
BaseTools/Source/Python/GenFds/GenFds.py      |    4 +-
.../Python/GenFds/GenFdsGlobalVariable.py     |   54 +-
.../Python/PatchPcdValue/PatchPcdValue.py     |    1 -
.../Source/Python/Workspace/DscBuildData.py   |   38 +-
.../Source/Python/Workspace/InfBuildData.py   |   39 +
.../Python/Workspace/WorkspaceCommon.py       |    4 +
.../Python/Workspace/WorkspaceDatabase.py     |    3 +
BaseTools/Source/Python/build/BuildReport.py  |    4 +-
BaseTools/Source/Python/build/build.py        |  333 +-
BaseTools/Source/Python/build/buildoptions.py |   92 +
24 files changed, 5834 insertions(+), 4442 deletions(-)
create mode 100644 BaseTools/Source/Python/AutoGen/AutoGenWorker.py
create mode 100644 BaseTools/Source/Python/AutoGen/DataPipe.py
create mode 100644 BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
create mode 100644 BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py
create mode 100644 BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
create mode 100644 BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py
create mode 100644 BaseTools/Source/Python/build/buildoptions.py
[edk2-devel] [Patch 00/10 V3] Enable multiple process AutoGen
Posted by Bob Feng 4 years, 8 months ago
BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875

In order to improve the build performance, we implemented
multiple-processes AutoGen. This change will reduce 20% time
for AutoGen phase.

The design document can be got from:
https://edk2.groups.io/g/devel/files/Designs/2019/0627/Multiple-thread-AutoGen.pdf

This patch serial pass the build of Ovmf, MinKabylake, MinPurley, packages
under Edk2 repository and intel client and server platforms.

V3:
1. Fixed incremental build issue.
2. Set AutoGen worker number to be align with "-n THREADNUMBER"
3. Enable block log queue.
V2: 
1. The first version missed autogen related commit 
from e812a812c1a0800c49e11507cb46222351520cc7. V2 add those commit
back.
2. Move CreateAsBuildInf into AutoGenWorker process
3. Save GlobalVar_<platform guid>_<arch>.bin to build folder.
4. Regenerate patches based on master bb824f685d


Feng, Bob C (10):
  BaseTools: Singleton the object to handle build conf file
  BaseTools: Split WorkspaceAutoGen._InitWorker into multiple functions
  BaseTools: Add functions to get platform scope build options
  BaseTools: Decouple AutoGen Objects
  BaseTools: Enable Multiple Process AutoGen
  BaseTools: Add shared data for processes
  BaseTools: Add LogAgent to support multiple process Autogen
  BaseTools: Move BuildOption parser out of build.py
  BaseTools: Add the support for python 2
  BaseTools: Enable block queue log agent.

 BaseTools/Source/Python/AutoGen/AutoGen.py    | 4227 +----------------
 .../Source/Python/AutoGen/AutoGenWorker.py    |  222 +
 .../Source/Python/AutoGen/BuildEngine.py      |   22 +
 BaseTools/Source/Python/AutoGen/DataPipe.py   |  153 +
 BaseTools/Source/Python/AutoGen/GenC.py       |    6 +-
 BaseTools/Source/Python/AutoGen/GenMake.py    |    3 +-
 .../Source/Python/AutoGen/ModuleAutoGen.py    | 1903 ++++++++
 .../Python/AutoGen/ModuleAutoGenHelper.py     |  616 +++
 .../Source/Python/AutoGen/PlatformAutoGen.py  | 1493 ++++++
 .../Source/Python/AutoGen/WorkspaceAutoGen.py |  905 ++++
 BaseTools/Source/Python/Common/EdkLogger.py   |  119 +-
 BaseTools/Source/Python/Common/Misc.py        |    1 -
 .../Python/Common/TargetTxtClassObject.py     |   28 +-
 .../Python/Common/ToolDefClassObject.py       |    6 +-
 BaseTools/Source/Python/GenFds/GenFds.py      |    4 +-
 .../Python/GenFds/GenFdsGlobalVariable.py     |   54 +-
 .../Python/PatchPcdValue/PatchPcdValue.py     |    1 -
 .../Source/Python/Workspace/DscBuildData.py   |   38 +-
 .../Source/Python/Workspace/InfBuildData.py   |   39 +
 .../Python/Workspace/WorkspaceCommon.py       |    4 +
 .../Python/Workspace/WorkspaceDatabase.py     |    3 +
 BaseTools/Source/Python/build/BuildReport.py  |    4 +-
 BaseTools/Source/Python/build/build.py        |  333 +-
 BaseTools/Source/Python/build/buildoptions.py |   92 +
 24 files changed, 5834 insertions(+), 4442 deletions(-)
 create mode 100644 BaseTools/Source/Python/AutoGen/AutoGenWorker.py
 create mode 100644 BaseTools/Source/Python/AutoGen/DataPipe.py
 create mode 100644 BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
 create mode 100644 BaseTools/Source/Python/AutoGen/ModuleAutoGenHelper.py
 create mode 100644 BaseTools/Source/Python/AutoGen/PlatformAutoGen.py
 create mode 100644 BaseTools/Source/Python/AutoGen/WorkspaceAutoGen.py
 create mode 100644 BaseTools/Source/Python/build/buildoptions.py

-- 
2.20.1.windows.1


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#44199): https://edk2.groups.io/g/devel/message/44199
Mute This Topic: https://groups.io/mt/32567312/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [Patch 00/10 V3] Enable multiple process AutoGen
Posted by Laszlo Ersek 4 years, 8 months ago
On 07/23/19 05:58, Bob Feng wrote:
> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875
> 
> In order to improve the build performance, we implemented
> multiple-processes AutoGen. This change will reduce 20% time
> for AutoGen phase.
> 
> The design document can be got from:
> https://edk2.groups.io/g/devel/files/Designs/2019/0627/Multiple-thread-AutoGen.pdf
> 
> This patch serial pass the build of Ovmf, MinKabylake, MinPurley, packages
> under Edk2 repository and intel client and server platforms.
> 
> V3:
> 1. Fixed incremental build issue.
> 2. Set AutoGen worker number to be align with "-n THREADNUMBER"
> 3. Enable block log queue.
> V2: 
> 1. The first version missed autogen related commit 
> from e812a812c1a0800c49e11507cb46222351520cc7. V2 add those commit
> back.
> 2. Move CreateAsBuildInf into AutoGenWorker process
> 3. Save GlobalVar_<platform guid>_<arch>.bin to build folder.
> 4. Regenerate patches based on master bb824f685d

Tested this series on top of commit cf2d8d4978e8.

(1) The GlobalVar*.bin files are now located under the Build directory,
so that's OK.

(2) Rebuilding OVMF (in a standalone edk2 clone, not as a QEMU
submodule) with different -D flags works fine now. The fw binary hash
matches a similar rebuild without the series applied.

(3) The blocking log queue works fine with Ctrl-S / Ctrl-Q.

Please see my note on the queue size (currently: constant 60) under
patch#10. I think the factor 10 that I suggested makes sense as well,
because I tested V3 with 4 threads (meaning room for 60/4=15 messages
per thread), and I felt that the compilation threads got blocked just a
bit too late after I pressed Ctrl-S. Therefore, a factor of 10 (rather
than the effective 15 that I tested now) will make for a more direct
user experience.

(4) I tested "-n1" and "-n2"; they seem to work fine.

(5) Restarting a build (incrementally) after Ctrl-C seems to work fine.

(6) The build seems to notice changes to .c files. OK.

(7) This version works fine as part of QEMU's edk2 submodule, with "make
-j 4 -C roms efi". All six firmware binaries built there are functional.


* For patches 01 through 09:

Tested-by: Laszlo Ersek <lersek@redhat.com>

* For patch 10: I'd like to test v4, with the proportional log queue
size, and give my T-b then.

* If you change other patches in v4 (01-09), then please drop my
Tested-by from those.

* Note that I didn't test:
- python3
- the "--hash" option
- the "--binary-destination" / "--binary-source" options
  (I'm not sure what those options do in the first place)
- the "--re-parse" option
- the "--no-cache" option
- the "--genfds-multi-thread" option

Hopefully others can test those.

Thanks
Laszlo

-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#44246): https://edk2.groups.io/g/devel/message/44246
Mute This Topic: https://groups.io/mt/32567312/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] [Patch 00/10 V3] Enable multiple process AutoGen
Posted by Bob Feng 4 years, 8 months ago
Hi Laszlo,

Thanks a lot for your testing. V4 and the future V5 change patch 4#, 5# and 10#.

For the items you did not test, I tested python3, the "--genfds-multi-thread" option. I think Steven Shi tested the "--hash" option and the "--binary-destination" / "--binary-source" options

I think the "--re-parse" option and the "--no-cache" option need to be removed from build options, because I did not see any code to handle those two. 

Thanks,
Bob

-----Original Message-----
From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of Laszlo Ersek
Sent: Tuesday, July 23, 2019 7:31 PM
To: devel@edk2.groups.io; Feng, Bob C <bob.c.feng@intel.com>
Subject: Re: [edk2-devel] [Patch 00/10 V3] Enable multiple process AutoGen

On 07/23/19 05:58, Bob Feng wrote:
> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=1875
> 
> In order to improve the build performance, we implemented 
> multiple-processes AutoGen. This change will reduce 20% time for 
> AutoGen phase.
> 
> The design document can be got from:
> https://edk2.groups.io/g/devel/files/Designs/2019/0627/Multiple-thread
> -AutoGen.pdf
> 
> This patch serial pass the build of Ovmf, MinKabylake, MinPurley, 
> packages under Edk2 repository and intel client and server platforms.
> 
> V3:
> 1. Fixed incremental build issue.
> 2. Set AutoGen worker number to be align with "-n THREADNUMBER"
> 3. Enable block log queue.
> V2: 
> 1. The first version missed autogen related commit from 
> e812a812c1a0800c49e11507cb46222351520cc7. V2 add those commit back.
> 2. Move CreateAsBuildInf into AutoGenWorker process 3. Save 
> GlobalVar_<platform guid>_<arch>.bin to build folder.
> 4. Regenerate patches based on master bb824f685d

Tested this series on top of commit cf2d8d4978e8.

(1) The GlobalVar*.bin files are now located under the Build directory, so that's OK.

(2) Rebuilding OVMF (in a standalone edk2 clone, not as a QEMU
submodule) with different -D flags works fine now. The fw binary hash matches a similar rebuild without the series applied.

(3) The blocking log queue works fine with Ctrl-S / Ctrl-Q.

Please see my note on the queue size (currently: constant 60) under patch#10. I think the factor 10 that I suggested makes sense as well, because I tested V3 with 4 threads (meaning room for 60/4=15 messages per thread), and I felt that the compilation threads got blocked just a bit too late after I pressed Ctrl-S. Therefore, a factor of 10 (rather than the effective 15 that I tested now) will make for a more direct user experience.

(4) I tested "-n1" and "-n2"; they seem to work fine.

(5) Restarting a build (incrementally) after Ctrl-C seems to work fine.

(6) The build seems to notice changes to .c files. OK.

(7) This version works fine as part of QEMU's edk2 submodule, with "make -j 4 -C roms efi". All six firmware binaries built there are functional.


* For patches 01 through 09:

Tested-by: Laszlo Ersek <lersek@redhat.com>

* For patch 10: I'd like to test v4, with the proportional log queue size, and give my T-b then.

* If you change other patches in v4 (01-09), then please drop my Tested-by from those.

* Note that I didn't test:
- python3
- the "--hash" option
- the "--binary-destination" / "--binary-source" options
  (I'm not sure what those options do in the first place)
- the "--re-parse" option
- the "--no-cache" option
- the "--genfds-multi-thread" option

Hopefully others can test those.

Thanks
Laszlo




-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.

View/Reply Online (#44627): https://edk2.groups.io/g/devel/message/44627
Mute This Topic: https://groups.io/mt/32567312/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub  [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-