From: Victor Lira <victorm.lira@amd.com>

Add x86_64 hardware test that creates a Xen Argo communication
connection between two PVH domains.

To accomplish this, add new build artifacts for Linux and Argo, and
update the xilinx x86_64 test script.

Victor Lira (3):
  automation: add linux 6.6.56 artifact
  automation: add linux argo test artifacts
  automation: add x86_64 test (linux argo)

 automation/gitlab-ci/build.yaml               | 34 +++++++
 automation/gitlab-ci/test.yaml                | 10 ++
 .../scripts/xilinx-smoke-dom0-x86_64.sh       | 76 ++++++++++-----
 .../tests-artifacts/argo/6.6.56.dockerfile    | 95 +++++++++++++++++++
 .../tests-artifacts/kernel/6.6.56.dockerfile  | 54 +++++++++++
 5 files changed, 244 insertions(+), 25 deletions(-)
 create mode 100644 automation/tests-artifacts/argo/6.6.56.dockerfile
 create mode 100644 automation/tests-artifacts/kernel/6.6.56.dockerfile

--
2.25.1

On 17/10/2024 6:18 pm, victorm.lira@amd.com wrote:
> From: Victor Lira <victorm.lira@amd.com>
>
> Add x86_64 hardware test that creates a Xen Argo communication
> connection between two PVH domains.
>
> To accomplish this, add new build artifacts for Linux and Argo, and
> update the xilinx x86_64 test script.
>
> Victor Lira (3):
>   automation: add linux 6.6.56 artifact
>   automation: add linux argo test artifacts
>   automation: add x86_64 test (linux argo)
>
>  automation/gitlab-ci/build.yaml               | 34 +++++++
>  automation/gitlab-ci/test.yaml                | 10 ++
>  .../scripts/xilinx-smoke-dom0-x86_64.sh       | 76 ++++++++++-----
>  .../tests-artifacts/argo/6.6.56.dockerfile    | 95 +++++++++++++++++++
>  .../tests-artifacts/kernel/6.6.56.dockerfile  | 54 +++++++++++
>  5 files changed, 244 insertions(+), 25 deletions(-)
>  create mode 100644 automation/tests-artifacts/argo/6.6.56.dockerfile
>  create mode 100644 automation/tests-artifacts/kernel/6.6.56.dockerfile

I'm afraid that we need to stop doing this test-artefacts like this.

The *-export pattern is nonsense (wasting lots of runner network&disk
and time just to copy data back into Github's artefact store), and the
main reason why we're at ~2T of data total for Xen-project.

We need a project wide expires_in: setting, which will prune all of our
not-most-recent content, and probably make most of that 2T disappear.

But, the test jobs stating what artefacts they want are perfectly
capable of pulling artefacts from somewhere other than an earlier nop
build job, meaning that we could get away with having one artefact
shared by everything, not one artefact copied per user per pipeline into
the artefact store.


I was thinking of experimenting with a separate top-level repo that does
nothing but has a few manual runs to populate artefacts, and having the
Xen tests pull artefacts from here rather than from earlier build jobs.

But, I've not had a chance to play yet, so I don't know for sure if this
is a workable direction.

Alternatively, if anyone has a better idea, I'm all ears.  But, it is
very clear that the *-export pattern is a giant bodge and there are
better ways.


Other notes.  Your dockerfiles use syntax:1 but not heredocs, and
they're also root containers.

Please follow the pattern in
https://lore.kernel.org/xen-devel/178560672106e37648304f5e597cc0b16c8db6db.1729170005.git.javi.merino@cloud.com/T/#u
or one of plenty other containers I've already converted in the past few
months.

Please do not add a new build job just to get a new minor variation of
Xen.  Just enable ARGO in general Xen build.

Although, given that you are now putting in testing from an unsupported
feature, we really need to bump the suggestion to move it to at least
Tech Preview.  (CC Dan/Chris).

~Andrew

On Thu, Oct 17, 2024 at 08:25:57PM +0100, Andrew Cooper wrote:
> I was thinking of experimenting with a separate top-level repo that does
> nothing but has a few manual runs to populate artefacts, and having the
> Xen tests pull artefacts from here rather than from earlier build jobs.

This sounds like a good idea. It may also help with rebuilding
containers when needed?

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

On Thu, 17 Oct 2024, Andrew Cooper wrote:
> On 17/10/2024 6:18 pm, victorm.lira@amd.com wrote:
> > From: Victor Lira <victorm.lira@amd.com>
> >
> > Add x86_64 hardware test that creates a Xen Argo communication
> > connection between two PVH domains.
> >
> > To accomplish this, add new build artifacts for Linux and Argo, and
> > update the xilinx x86_64 test script.
> >
> > Victor Lira (3):
> >   automation: add linux 6.6.56 artifact
> >   automation: add linux argo test artifacts
> >   automation: add x86_64 test (linux argo)
> >
> >  automation/gitlab-ci/build.yaml               | 34 +++++++
> >  automation/gitlab-ci/test.yaml                | 10 ++
> >  .../scripts/xilinx-smoke-dom0-x86_64.sh       | 76 ++++++++++-----
> >  .../tests-artifacts/argo/6.6.56.dockerfile    | 95 +++++++++++++++++++
> >  .../tests-artifacts/kernel/6.6.56.dockerfile  | 54 +++++++++++
> >  5 files changed, 244 insertions(+), 25 deletions(-)
> >  create mode 100644 automation/tests-artifacts/argo/6.6.56.dockerfile
> >  create mode 100644 automation/tests-artifacts/kernel/6.6.56.dockerfile
> 
> I'm afraid that we need to stop doing this test-artefacts like this.
> 
> The *-export pattern is nonsense (wasting lots of runner network&disk
> and time just to copy data back into Github's artefact store), and the
> main reason why we're at ~2T of data total for Xen-project.
> 
> We need a project wide expires_in: setting, which will prune all of our
> not-most-recent content, and probably make most of that 2T disappear.

I looked at this for an hour and couldn't find a way to do it
unfortunately. expires_in can only be changed if you are an Admin of the
Gitlab instance, meaning an Admin of gitlab.com.

However, there is a setting in each repository (not one global at the
gitlab.com/xen-project level) called "Keep artifacts from most recent
successful jobs" that can be disabled. I did it manually for my repo.

The big offender is patchew. I went and disabled "Keep artifacts from
most recent successful jobs" but I didn't see any immediate changes in
storage utilization. But it is expected to take time to propagate.


> But, the test jobs stating what artefacts they want are perfectly
> capable of pulling artefacts from somewhere other than an earlier nop
> build job, meaning that we could get away with having one artefact
> shared by everything, not one artefact copied per user per pipeline into
> the artefact store.
> 
> 
> I was thinking of experimenting with a separate top-level repo that does
> nothing but has a few manual runs to populate artefacts, and having the
> Xen tests pull artefacts from here rather than from earlier build jobs.
> 
> But, I've not had a chance to play yet, so I don't know for sure if this
> is a workable direction.
>
> Alternatively, if anyone has a better idea, I'm all ears.  But, it is
> very clear that the *-export pattern is a giant bodge and there are
> better ways.
> 
> Other notes.  Your dockerfiles use syntax:1 but not heredocs, and
> they're also root containers.
> 
> Please follow the pattern in
> https://lore.kernel.org/xen-devel/178560672106e37648304f5e597cc0b16c8db6db.1729170005.git.javi.merino@cloud.com/T/#u
> or one of plenty other containers I've already converted in the past few
> months.
> 
> Please do not add a new build job just to get a new minor variation of
> Xen.  Just enable ARGO in general Xen build.

+1