There is an infelicity in handling of FreeBSD's "memdisk" test.

The use of this step is as part of host examination.  There, it may
fail on hosts which cannot cope with the FreeBSD version in use.  That
is OK (from a general osstest/CI POV) because each host has its own
job name and therefore its own regression tracking.  So this test
failing on those hosts is not troublesome.

But as part of the tests for osstest itself, and Xen and Linux, we
also check that host examination still works.  This is appropriate
because host examination also checks that we can retrieve accurate
serial logs, etc.

To this end many general flights have test-*-examine jobs.

Like any job, these are peripatetic.  The x86 ones might run on BIOS
hosts, or UEFI hosts.  But our support for FreeBSD does not cope with
UEFI yet.  The result is that whether this test passes or fails
depends on the host, but this is not a heisenbug which should be fixed
right away.

Sort this out as follows:

* Have the examine test only run the FreeBSD step when the host
  firmware is BIOS.  On non-FreeBSD hosts the test step will go from
  "fail" to "skip", which is appropriate.

* To make sure we catch actual regressions in this step, make a copy
  of the examine job which is always run on a BIOS host.

* While we are here, explicitly check that the examine job works on
  x86 UEFI too.


4.16 release implications:

There are two sets of changes here:

I. flight composition:

The runvar dump shows that this only adds new jobs.  I think these new
tests are good, but of course they might be broken somehow.  If they
are broken, they are new jobs which won't count as regressions, and if
they flap we can force push while we disable them again.

II. behavioural change to ts-memdisk-try-append

This change is *intended* to make the test "skip" in cases where it is
currently "fail".  I have done an ad-hoc check that this seems to DTRT
but it might be bad somehow.

However, this test is currently actually-failing on the xen-unstable
branch.  And it could be force pushed if necessary.

III. Benefits

The benefit of all this is to improve the testing for 4.16.  In
particular, the examine test ought now to run properly, and no longer
cause blockages.

I think at this stage of the release this change to osstest ought to
have a release-ack but I think I have convinced myself it's OK, so:
Release-Acked-By: Ian Jackson <iwj@xenproject.org>


Ian Jackson (5):
  make-flight: do_examine_one: add firmware argument
  ts-memdisk-try-append: Reindent (nfc)
  ts-memdisk-try-append: Enable perl warnings
  ts-memdisk-try-append: More defensive case test
  examination: skip memdisk on non-BIOS hosts, run on some BIOS hosts

 make-flight           | 23 ++++++++++++++++++++--
 mfi-common            |  1 +
 ts-memdisk-try-append | 45 +++++++++++++++++++++++++++++--------------
 3 files changed, 53 insertions(+), 16 deletions(-)

-- 
2.20.1