PR: https://github.com/tianocore/edk2/pull/2711

Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in a custom
Fedora 35 container. The image provides gcc 11 from Fedora for all
architectures. The external dependencies for gcc have been removed
so stuart does not download them. iasl and nasm are also
included in the image, but remain as ext-dep for now (CI jobs on
Windows need those).

The Dockerfiles are here:
https://github.com/osteffenrh/edk2-build-images, but they should, of
course, then move to a repo in the Tianocore group, or something like
that. The images are built automatically via GitHub Actions and then
pushed to ghcr.io.

Oliver Steffen (4):
  CI: don't force python verison (Linux only)
  CI: add ~/.local/bin to PATH (Linux only)
  CI: Use Fedora 35 container (Linux only)
  BaseTools: Remove ext. gcc dependencies (Linux only)

 .../templates/basetools-build-steps.yml       |  9 --------
 .../templates/platform-build-run-steps.yml    |  7 ++++++
 .../templates/pr-gate-build-job.yml           |  3 +++
 .azurepipelines/templates/pr-gate-steps.yml   |  7 ++++++
 .../.azurepipelines/Ubuntu-GCC5.yml           |  6 ++---
 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml  | 21 ------------------
 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml      | 21 ------------------
 .../Bin/gcc_riscv64_unknown_ext_dep.yaml      | 22 -------------------
 .../.azurepipelines/Ubuntu-GCC5.yml           |  2 ++
 .../.azurepipelines/Ubuntu-GCC5.yml           |  6 ++---
 10 files changed, 23 insertions(+), 81 deletions(-)
 delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
 delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml

-- 
2.35.1



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#89058): https://edk2.groups.io/g/devel/message/89058
Mute This Topic: https://groups.io/mt/90557209/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

On Tue, 19 Apr 2022 at 09:35, Oliver Steffen <osteffen@redhat.com> wrote:
>
> PR: https://github.com/tianocore/edk2/pull/2711
>
> Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in a custom
> Fedora 35 container.

Why?

> The image provides gcc 11 from Fedora for all
> architectures. The external dependencies for gcc have been removed
> so stuart does not download them. iasl and nasm are also
> included in the image, but remain as ext-dep for now (CI jobs on
> Windows need those).
>
> The Dockerfiles are here:
> https://github.com/osteffenrh/edk2-build-images, but they should, of
> course, then move to a repo in the Tianocore group, or something like
> that. The images are built automatically via GitHub Actions and then
> pushed to ghcr.io.
>

It would help tremendously if you could include why doing any of this is neces
> Oliver Steffen (4):
>   CI: don't force python verison (Linux only)
>   CI: add ~/.local/bin to PATH (Linux only)
>   CI: Use Fedora 35 container (Linux only)
>   BaseTools: Remove ext. gcc dependencies (Linux only)
>
>  .../templates/basetools-build-steps.yml       |  9 --------
>  .../templates/platform-build-run-steps.yml    |  7 ++++++
>  .../templates/pr-gate-build-job.yml           |  3 +++
>  .azurepipelines/templates/pr-gate-steps.yml   |  7 ++++++
>  .../.azurepipelines/Ubuntu-GCC5.yml           |  6 ++---
>  BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml  | 21 ------------------
>  BaseTools/Bin/gcc_arm_linux_ext_dep.yaml      | 21 ------------------
>  .../Bin/gcc_riscv64_unknown_ext_dep.yaml      | 22 -------------------
>  .../.azurepipelines/Ubuntu-GCC5.yml           |  2 ++
>  .../.azurepipelines/Ubuntu-GCC5.yml           |  6 ++---
>  10 files changed, 23 insertions(+), 81 deletions(-)
>  delete mode 100644 BaseTools/Bin/gcc_aarch64_linux_ext_dep.yaml
>  delete mode 100644 BaseTools/Bin/gcc_arm_linux_ext_dep.yaml
>  delete mode 100644 BaseTools/Bin/gcc_riscv64_unknown_ext_dep.yaml
>
> --
> 2.35.1
>


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#89088): https://edk2.groups.io/g/devel/message/89088
Mute This Topic: https://groups.io/mt/90557209/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

On Tue, Apr 19, 2022 at 11:54:38PM +0200, Ard Biesheuvel wrote:
> On Tue, 19 Apr 2022 at 09:35, Oliver Steffen <osteffen@redhat.com> wrote:
> >
> > PR: https://github.com/tianocore/edk2/pull/2711
> >
> > Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in a custom
> > Fedora 35 container.
> 
> Why?
> 
> > The image provides gcc 11 from Fedora for all
> > architectures. The external dependencies for gcc have been removed
> > so stuart does not download them. iasl and nasm are also
> > included in the image, but remain as ext-dep for now (CI jobs on
> > Windows need those).

^^^ this for example.

The idea is to (a) switch linux CI builds from vm images to container
images, and (b) use our own, custom container images.

This allows to roll the dependencies needed for CI into the container
images, so there is no need to download them in CI jobs.  Fixes
temporary CI failures due to network problems and download errors.

This also allows to remove any distribution-specific commands (apt-get
for example) from CI jobs.  This will be handled in the Dockerfiles used
to build our custom container images instead.

Oliver is also working on ubuntu containers, look here:

> > The Dockerfiles are here:
> > https://github.com/osteffenrh/edk2-build-images,

Fedora is a bit faster on updating stuff.  For example: ubuntu 22.04
uses gcc 11 whereas Fedora 36 uses gcc 12 (both distros are in beta
right now).  Whenever we want:

  (a) stick to ubuntu, or
  (b) switch to fedora, or
  (c) use both

is up for debate.

> > but they should, of
> > course, then move to a repo in the Tianocore group, or something like
> > that. The images are built automatically via GitHub Actions and then
> > pushed to ghcr.io.

That is the next question: where should we host the Dockerfiles and
container images?

For the Dockerfiles that should probably a git repo in the tianocore
group.

For the container images the obvious choices are github and azure.
Given that microsoft owns github I'd expect both github actions and
azure pipelines are running on the same cloud infrastructure and it
doesn't make much of a difference whenever github or azure container
registry is used to host container images.

take care,
  Gerd



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#89116): https://edk2.groups.io/g/devel/message/89116
Mute This Topic: https://groups.io/mt/90557209/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

On Wed, 20 Apr 2022 at 09:30, Gerd Hoffmann <kraxel@redhat.com> wrote:
>
> On Tue, Apr 19, 2022 at 11:54:38PM +0200, Ard Biesheuvel wrote:
> > On Tue, 19 Apr 2022 at 09:35, Oliver Steffen <osteffen@redhat.com> wrote:
> > >
> > > PR: https://github.com/tianocore/edk2/pull/2711
> > >
> > > Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in a custom
> > > Fedora 35 container.
> >
> > Why?
> >
> > > The image provides gcc 11 from Fedora for all
> > > architectures. The external dependencies for gcc have been removed
> > > so stuart does not download them. iasl and nasm are also
> > > included in the image, but remain as ext-dep for now (CI jobs on
> > > Windows need those).
>
> ^^^ this for example.
>
> The idea is to (a) switch linux CI builds from vm images to container
> images, and (b) use our own, custom container images.
>
> This allows to roll the dependencies needed for CI into the container
> images, so there is no need to download them in CI jobs.  Fixes
> temporary CI failures due to network problems and download errors.
>
> This also allows to remove any distribution-specific commands (apt-get
> for example) from CI jobs.  This will be handled in the Dockerfiles used
> to build our custom container images instead.
>
> Oliver is also working on ubuntu containers, look here:
>
> > > The Dockerfiles are here:
> > > https://github.com/osteffenrh/edk2-build-images,
>
> Fedora is a bit faster on updating stuff.  For example: ubuntu 22.04
> uses gcc 11 whereas Fedora 36 uses gcc 12 (both distros are in beta
> right now).  Whenever we want:
>
>   (a) stick to ubuntu, or
>   (b) switch to fedora, or
>   (c) use both
>
> is up for debate.
>
> > > but they should, of
> > > course, then move to a repo in the Tianocore group, or something like
> > > that. The images are built automatically via GitHub Actions and then
> > > pushed to ghcr.io.
>
> That is the next question: where should we host the Dockerfiles and
> container images?
>
> For the Dockerfiles that should probably a git repo in the tianocore
> group.
>
> For the container images the obvious choices are github and azure.
> Given that microsoft owns github I'd expect both github actions and
> azure pipelines are running on the same cloud infrastructure and it
> doesn't make much of a difference whenever github or azure container
> registry is used to host container images.
>

Thanks for the explanation.

Please put information like this in the commit logs, rather than
assuming it will obvious to everyone without providing any background.

For these changes,

Acked-by: Ard Biesheuvel <ardb@kernel.org>


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#89131): https://edk2.groups.io/g/devel/message/89131
Mute This Topic: https://groups.io/mt/90557209/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-

On 20.04.2022 11:16, Ard Biesheuvel wrote:
>On Wed, 20 Apr 2022 at 09:30, Gerd Hoffmann <kraxel@redhat.com> wrote:
>>
>> On Tue, Apr 19, 2022 at 11:54:38PM +0200, Ard Biesheuvel wrote:
>> > On Tue, 19 Apr 2022 at 09:35, Oliver Steffen <osteffen@redhat.com> wrote:
>> > >
>> > > PR: https://github.com/tianocore/edk2/pull/2711
>> > >
>> > > Update CI, run all Linux (aka Ubuntu-GCC5) based jobs in a custom
>> > > Fedora 35 container.
>> >
>> > Why?
>> >
>> > > The image provides gcc 11 from Fedora for all
>> > > architectures. The external dependencies for gcc have been removed
>> > > so stuart does not download them. iasl and nasm are also
>> > > included in the image, but remain as ext-dep for now (CI jobs on
>> > > Windows need those).
>>
>> ^^^ this for example.
>>
>> The idea is to (a) switch linux CI builds from vm images to container
>> images, and (b) use our own, custom container images.
>>
>> This allows to roll the dependencies needed for CI into the container
>> images, so there is no need to download them in CI jobs.  Fixes
>> temporary CI failures due to network problems and download errors.
>>
>> This also allows to remove any distribution-specific commands (apt-get
>> for example) from CI jobs.  This will be handled in the Dockerfiles used
>> to build our custom container images instead.
>>
>> Oliver is also working on ubuntu containers, look here:
>>
>> > > The Dockerfiles are here:
>> > > https://github.com/osteffenrh/edk2-build-images,
>>
>> Fedora is a bit faster on updating stuff.  For example: ubuntu 22.04
>> uses gcc 11 whereas Fedora 36 uses gcc 12 (both distros are in beta
>> right now).  Whenever we want:
>>
>>   (a) stick to ubuntu, or
>>   (b) switch to fedora, or
>>   (c) use both
>>
>> is up for debate.
>>
>> > > but they should, of
>> > > course, then move to a repo in the Tianocore group, or something like
>> > > that. The images are built automatically via GitHub Actions and then
>> > > pushed to ghcr.io.
>>
>> That is the next question: where should we host the Dockerfiles and
>> container images?
>>
>> For the Dockerfiles that should probably a git repo in the tianocore
>> group.
>>
>> For the container images the obvious choices are github and azure.
>> Given that microsoft owns github I'd expect both github actions and
>> azure pipelines are running on the same cloud infrastructure and it
>> doesn't make much of a difference whenever github or azure container
>> registry is used to host container images.
>>
>
>Thanks for the explanation.
>
>Please put information like this in the commit logs, rather than
>assuming it will obvious to everyone without providing any background.

Will do.

There is a related discussion about CI and containers here:
https://github.com/tianocore/edk2/discussions/2614#discussioncomment-2595253
as well as in the meeting recording attached in that post.

Cheers,
  Oliver



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#89133): https://edk2.groups.io/g/devel/message/89133
Mute This Topic: https://groups.io/mt/90557209/1787277
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [importer@patchew.org]
-=-=-=-=-=-=-=-=-=-=-=-