Question about SEV* guests and automatic firmware selection

Jim Fehlig via Devel posted 1 patch 10 months, 2 weeks ago
Failed in applying to current master (apply log)
Question about SEV* guests and automatic firmware selection
Posted by Jim Fehlig via Devel 10 months, 2 weeks ago
Hi All,

While investigating an internal bug report, we noticed that a minimal firmware 
auto-selection configuration along with SEV* fails to find a match. E.g. the 
following config

<domain type="kvm">
   <os firmware="efi">
     <type arch="x86_64" machine="q35">hvm</type>
     <boot dev="hd"/>
   </os>
   <launchSecurity type="sev">
     <policy>0x07</policy>
   </launchSecurity>
...
</domain>

Fails with "Unable to find 'efi' firmware that is compatible with the current 
configuration". A firmware that should match has the following json description

{
     "description": "UEFI firmware for x86_64, with AMD SEV",
     "interface-types": [
         "uefi"
     ],
     "mapping": {
         "device": "flash",
	"mode": "stateless",
         "executable": {
             "filename": "/usr/share/qemu/ovmf-x86_64-sev.bin",
             "format": "raw"
         }
     },
     "targets": [
         {
             "architecture": "x86_64",
             "machines": [
                 "pc-q35-*"
             ]
         }
     ],
     "features": [
         "acpi-s4",
	"amd-sev",
	"amd-sev-es",
	"amd-sev-snp",
         "verbose-dynamic"
     ],
     "tags": [

     ]
}

Auto-selection works fine if I specify a 'stateless' firmware, e.g. amend the 
above config with

   <os firmware="efi">
     <type arch="x86_64" machine="q35">hvm</type>
     <loader stateless="yes"/>
     <boot dev="hd"/>
   </os>

Being unfamiliar with the firmware auto-selection code, I tried the below naive 
hack, which only led to test failures and the subsequent runtime error "unable 
to find any master var store for loader: /usr/share/qemu/ovmf-x86_64-sev.bin". 
Should auto-selection work with the minimal config, or is it expected that user 
also specify a stateless firmware?

Regards,
Jim

diff --git a/src/qemu/qemu_firmware.c b/src/qemu/qemu_firmware.c
index 2d0ec0b4fa..660b74141a 100644
--- a/src/qemu/qemu_firmware.c
+++ b/src/qemu/qemu_firmware.c
@@ -1293,15 +1293,17 @@ qemuFirmwareMatchDomain(const virDomainDef *def,
              return false;
          }

-        if (loader && loader->stateless == VIR_TRISTATE_BOOL_YES) {
-            if (flash->mode != QEMU_FIRMWARE_FLASH_MODE_STATELESS) {
-                VIR_DEBUG("Discarding loader without stateless flash");
-                return false;
-            }
-        } else {
-            if (flash->mode != QEMU_FIRMWARE_FLASH_MODE_SPLIT) {
-                VIR_DEBUG("Discarding loader without split flash");
-                return false;
+        if (loader) {
+            if (loader->stateless == VIR_TRISTATE_BOOL_YES) {
+                if (flash->mode != QEMU_FIRMWARE_FLASH_MODE_STATELESS) {
+                    VIR_DEBUG("Discarding loader without stateless flash");
+                    return false;
+                }
+            } else if (loader->stateless == VIR_TRISTATE_BOOL_NO) {
+                if (flash->mode != QEMU_FIRMWARE_FLASH_MODE_SPLIT) {
+                    VIR_DEBUG("Discarding loader without split flash");
+                    return false;
+                }
              }
          }
Re: Question about SEV* guests and automatic firmware selection
Posted by Daniel P. Berrangé via Devel 10 months, 2 weeks ago
On Mon, Apr 21, 2025 at 01:38:35PM -0600, Jim Fehlig via Devel wrote:
> Hi All,
> 
> While investigating an internal bug report, we noticed that a minimal
> firmware auto-selection configuration along with SEV* fails to find a match.
> E.g. the following config
> 
> <domain type="kvm">
>   <os firmware="efi">
>     <type arch="x86_64" machine="q35">hvm</type>
>     <boot dev="hd"/>
>   </os>
>   <launchSecurity type="sev">
>     <policy>0x07</policy>
>   </launchSecurity>
> ...
> </domain>
> 
> Fails with "Unable to find 'efi' firmware that is compatible with the
> current configuration". A firmware that should match has the following json
> description
> 
> {
>     "description": "UEFI firmware for x86_64, with AMD SEV",
>     "interface-types": [
>         "uefi"
>     ],
>     "mapping": {
>         "device": "flash",
> 	"mode": "stateless",
>         "executable": {
>             "filename": "/usr/share/qemu/ovmf-x86_64-sev.bin",
>             "format": "raw"
>         }
>     },
>     "targets": [
>         {
>             "architecture": "x86_64",
>             "machines": [
>                 "pc-q35-*"
>             ]
>         }
>     ],
>     "features": [
>         "acpi-s4",
> 	"amd-sev",
> 	"amd-sev-es",
> 	"amd-sev-snp",
>         "verbose-dynamic"
>     ],
>     "tags": [
> 
>     ]
> }
> 
> Auto-selection works fine if I specify a 'stateless' firmware, e.g. amend
> the above config with
> 
>   <os firmware="efi">
>     <type arch="x86_64" machine="q35">hvm</type>
>     <loader stateless="yes"/>
>     <boot dev="hd"/>
>   </os>
> 
> Being unfamiliar with the firmware auto-selection code, I tried the below
> naive hack, which only led to test failures and the subsequent runtime error
> "unable to find any master var store for loader:
> /usr/share/qemu/ovmf-x86_64-sev.bin". Should auto-selection work with the
> minimal config, or is it expected that user also specify a stateless
> firmware?

I don't have any SEV/SNP installation available to test with current,
but on Fedora/RHEL, AFAIK, we successfully install with

virt-install  \
--name snp  \
--launchSecurity sev-snp,policy=0x30000  \
--machine q35
--boot uefi

which will NOT result in 'stateless' attribute being set and our
firwmare descriptors match what you show above.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Re: Question about SEV* guests and automatic firmware selection
Posted by Jim Fehlig via Devel 10 months, 2 weeks ago
On 4/24/25 04:59, Daniel P. Berrangé wrote:
> On Mon, Apr 21, 2025 at 01:38:35PM -0600, Jim Fehlig via Devel wrote:
>> Hi All,
>>
>> While investigating an internal bug report, we noticed that a minimal
>> firmware auto-selection configuration along with SEV* fails to find a match.
>> E.g. the following config
>>
>> <domain type="kvm">
>>    <os firmware="efi">
>>      <type arch="x86_64" machine="q35">hvm</type>
>>      <boot dev="hd"/>
>>    </os>
>>    <launchSecurity type="sev">
>>      <policy>0x07</policy>
>>    </launchSecurity>
>> ...
>> </domain>
>>
>> Fails with "Unable to find 'efi' firmware that is compatible with the
>> current configuration". A firmware that should match has the following json
>> description
>>
>> {
>>      "description": "UEFI firmware for x86_64, with AMD SEV",
>>      "interface-types": [
>>          "uefi"
>>      ],
>>      "mapping": {
>>          "device": "flash",
>> 	"mode": "stateless",
>>          "executable": {
>>              "filename": "/usr/share/qemu/ovmf-x86_64-sev.bin",
>>              "format": "raw"
>>          }
>>      },
>>      "targets": [
>>          {
>>              "architecture": "x86_64",
>>              "machines": [
>>                  "pc-q35-*"
>>              ]
>>          }
>>      ],
>>      "features": [
>>          "acpi-s4",
>> 	"amd-sev",
>> 	"amd-sev-es",
>> 	"amd-sev-snp",
>>          "verbose-dynamic"
>>      ],
>>      "tags": [
>>
>>      ]
>> }
>>
>> Auto-selection works fine if I specify a 'stateless' firmware, e.g. amend
>> the above config with
>>
>>    <os firmware="efi">
>>      <type arch="x86_64" machine="q35">hvm</type>
>>      <loader stateless="yes"/>
>>      <boot dev="hd"/>
>>    </os>
>>
>> Being unfamiliar with the firmware auto-selection code, I tried the below
>> naive hack, which only led to test failures and the subsequent runtime error
>> "unable to find any master var store for loader:
>> /usr/share/qemu/ovmf-x86_64-sev.bin". Should auto-selection work with the
>> minimal config, or is it expected that user also specify a stateless
>> firmware?
> 
> I don't have any SEV/SNP installation available to test with current,
> but on Fedora/RHEL, AFAIK, we successfully install with
> 
> virt-install  \
> --name snp  \
> --launchSecurity sev-snp,policy=0x30000  \
> --machine q35
> --boot uefi

I see the same failure when using '--boot uefi' or '--boot firmware=efi'

ERROR    operation failed: Unable to find 'efi' firmware that is compatible with 
the current configuration

Works fine with '--boot firmware=efi,loader.stateless=yes'.

> which will NOT result in 'stateless' attribute being set and our
> firwmare descriptors match what you show above.

Nod. The rawhide descriptor '60-edk2-ovmf-x64-amdsev.json' is nearly identical 
to the one I posted, with exception of the missing acpi-s4 feature. But that 
shouldn't be there anyhow. It's a bug that has since been fixed in the openSUSE 
descriptor.

Regards,
Jim
Re: Question about SEV* guests and automatic firmware selection
Posted by Jim Fehlig via Devel 9 months, 3 weeks ago
On 4/24/25 14:18, Jim Fehlig wrote:
> On 4/24/25 04:59, Daniel P. Berrangé wrote:
>> On Mon, Apr 21, 2025 at 01:38:35PM -0600, Jim Fehlig via Devel wrote:
>>> Hi All,
>>>
>>> While investigating an internal bug report, we noticed that a minimal
>>> firmware auto-selection configuration along with SEV* fails to find a match.
>>> E.g. the following config
>>>
>>> <domain type="kvm">
>>>    <os firmware="efi">
>>>      <type arch="x86_64" machine="q35">hvm</type>
>>>      <boot dev="hd"/>
>>>    </os>
>>>    <launchSecurity type="sev">
>>>      <policy>0x07</policy>
>>>    </launchSecurity>
>>> ...
>>> </domain>
>>>
>>> Fails with "Unable to find 'efi' firmware that is compatible with the
>>> current configuration". A firmware that should match has the following json
>>> description
>>>
>>> {
>>>      "description": "UEFI firmware for x86_64, with AMD SEV",
>>>      "interface-types": [
>>>          "uefi"
>>>      ],
>>>      "mapping": {
>>>          "device": "flash",
>>>     "mode": "stateless",
>>>          "executable": {
>>>              "filename": "/usr/share/qemu/ovmf-x86_64-sev.bin",
>>>              "format": "raw"
>>>          }
>>>      },
>>>      "targets": [
>>>          {
>>>              "architecture": "x86_64",
>>>              "machines": [
>>>                  "pc-q35-*"
>>>              ]
>>>          }
>>>      ],
>>>      "features": [
>>>          "acpi-s4",
>>>     "amd-sev",
>>>     "amd-sev-es",
>>>     "amd-sev-snp",
>>>          "verbose-dynamic"
>>>      ],
>>>      "tags": [
>>>
>>>      ]
>>> }
>>>
>>> Auto-selection works fine if I specify a 'stateless' firmware, e.g. amend
>>> the above config with
>>>
>>>    <os firmware="efi">
>>>      <type arch="x86_64" machine="q35">hvm</type>
>>>      <loader stateless="yes"/>
>>>      <boot dev="hd"/>
>>>    </os>
>>>
>>> Being unfamiliar with the firmware auto-selection code, I tried the below
>>> naive hack, which only led to test failures and the subsequent runtime error
>>> "unable to find any master var store for loader:
>>> /usr/share/qemu/ovmf-x86_64-sev.bin". Should auto-selection work with the
>>> minimal config, or is it expected that user also specify a stateless
>>> firmware?

Andrea,

Having spent a fair bit of time in the firmware auto-selection code, perhaps you 
have an opinion about this?

Regards,
Jim

>>
>> I don't have any SEV/SNP installation available to test with current,
>> but on Fedora/RHEL, AFAIK, we successfully install with
>>
>> virt-install  \
>> --name snp  \
>> --launchSecurity sev-snp,policy=0x30000  \
>> --machine q35
>> --boot uefi
> 
> I see the same failure when using '--boot uefi' or '--boot firmware=efi'
> 
> ERROR    operation failed: Unable to find 'efi' firmware that is compatible with 
> the current configuration
> 
> Works fine with '--boot firmware=efi,loader.stateless=yes'.
> 
>> which will NOT result in 'stateless' attribute being set and our
>> firwmare descriptors match what you show above.
> 
> Nod. The rawhide descriptor '60-edk2-ovmf-x64-amdsev.json' is nearly identical 
> to the one I posted, with exception of the missing acpi-s4 feature. But that 
> shouldn't be there anyhow. It's a bug that has since been fixed in the openSUSE 
> descriptor.
> 
> Regards,
> Jim
> 
Re: Question about SEV* guests and automatic firmware selection
Posted by Jim Fehlig via Devel 7 months, 1 week ago
On 5/13/25 14:25, Jim Fehlig wrote:
> On 4/24/25 14:18, Jim Fehlig wrote:
>> On 4/24/25 04:59, Daniel P. Berrangé wrote:
>>> On Mon, Apr 21, 2025 at 01:38:35PM -0600, Jim Fehlig via Devel wrote:
>>>> Hi All,
>>>>
>>>> While investigating an internal bug report, we noticed that a minimal
>>>> firmware auto-selection configuration along with SEV* fails to find a match.
>>>> E.g. the following config
>>>>
>>>> <domain type="kvm">
>>>>    <os firmware="efi">
>>>>      <type arch="x86_64" machine="q35">hvm</type>
>>>>      <boot dev="hd"/>
>>>>    </os>
>>>>    <launchSecurity type="sev">
>>>>      <policy>0x07</policy>
>>>>    </launchSecurity>
>>>> ...
>>>> </domain>
>>>>
>>>> Fails with "Unable to find 'efi' firmware that is compatible with the
>>>> current configuration". A firmware that should match has the following json
>>>> description
>>>>
>>>> {
>>>>      "description": "UEFI firmware for x86_64, with AMD SEV",
>>>>      "interface-types": [
>>>>          "uefi"
>>>>      ],
>>>>      "mapping": {
>>>>          "device": "flash",
>>>>     "mode": "stateless",
>>>>          "executable": {
>>>>              "filename": "/usr/share/qemu/ovmf-x86_64-sev.bin",
>>>>              "format": "raw"
>>>>          }
>>>>      },
>>>>      "targets": [
>>>>          {
>>>>              "architecture": "x86_64",
>>>>              "machines": [
>>>>                  "pc-q35-*"
>>>>              ]
>>>>          }
>>>>      ],
>>>>      "features": [
>>>>          "acpi-s4",
>>>>     "amd-sev",
>>>>     "amd-sev-es",
>>>>     "amd-sev-snp",
>>>>          "verbose-dynamic"
>>>>      ],
>>>>      "tags": [
>>>>
>>>>      ]
>>>> }
>>>>
>>>> Auto-selection works fine if I specify a 'stateless' firmware, e.g. amend
>>>> the above config with
>>>>
>>>>    <os firmware="efi">
>>>>      <type arch="x86_64" machine="q35">hvm</type>
>>>>      <loader stateless="yes"/>
>>>>      <boot dev="hd"/>
>>>>    </os>
>>>>
>>>> Being unfamiliar with the firmware auto-selection code, I tried the below
>>>> naive hack, which only led to test failures and the subsequent runtime error
>>>> "unable to find any master var store for loader:
>>>> /usr/share/qemu/ovmf-x86_64-sev.bin". Should auto-selection work with the
>>>> minimal config, or is it expected that user also specify a stateless
>>>> firmware?
> 
> Andrea,
> 
> Having spent a fair bit of time in the firmware auto-selection code, perhaps you 
> have an opinion about this?

Sorry to keep nagging about this, but I still see the issue with latest git 
master using a json descriptor identical to the latest Fedora one. When omitting 
the 'stateless' attribute, autoselection fails with

operation failed: Unable to find 'efi' firmware that is compatible with the 
current configuration

With debug enabled, I can see the firmware is not even considered since it's not 
"split"

2025-07-29 21:02:05.517+0000: 32187: debug : qemuFirmwareMatchDomain:1310 : 
Discarding loader without split flash

I would need to install Fedora and verify myself, but I'd be surprised if it 
didn't encounter the same issue. Without the 'stateless' attribute, the 
following test would fail

https://gitlab.com/libvirt/libvirt/-/blob/master/src/qemu/qemu_firmware.c?ref_type=heads#L1303

executing the else condition, and failing similarly since the firmware is not split.

Regards,
Jim
Re: Question about SEV* guests and automatic firmware selection
Posted by Daniel P. Berrangé via Devel 7 months, 1 week ago
On Tue, Jul 29, 2025 at 03:28:49PM -0600, Jim Fehlig wrote:
> On 5/13/25 14:25, Jim Fehlig wrote:
> > On 4/24/25 14:18, Jim Fehlig wrote:
> > > On 4/24/25 04:59, Daniel P. Berrangé wrote:
> > > > On Mon, Apr 21, 2025 at 01:38:35PM -0600, Jim Fehlig via Devel wrote:
> > > > > Hi All,
> > > > > 
> > > > > While investigating an internal bug report, we noticed that a minimal
> > > > > firmware auto-selection configuration along with SEV* fails to find a match.
> > > > > E.g. the following config
> > > > > 
> > > > > <domain type="kvm">
> > > > >    <os firmware="efi">
> > > > >      <type arch="x86_64" machine="q35">hvm</type>
> > > > >      <boot dev="hd"/>
> > > > >    </os>
> > > > >    <launchSecurity type="sev">
> > > > >      <policy>0x07</policy>
> > > > >    </launchSecurity>
> > > > > ...
> > > > > </domain>
> > > > > 
> > > > > Fails with "Unable to find 'efi' firmware that is compatible with the
> > > > > current configuration". A firmware that should match has the following json
> > > > > description
> > > > > 
> > > > > {
> > > > >      "description": "UEFI firmware for x86_64, with AMD SEV",
> > > > >      "interface-types": [
> > > > >          "uefi"
> > > > >      ],
> > > > >      "mapping": {
> > > > >          "device": "flash",
> > > > >     "mode": "stateless",
> > > > >          "executable": {
> > > > >              "filename": "/usr/share/qemu/ovmf-x86_64-sev.bin",
> > > > >              "format": "raw"
> > > > >          }
> > > > >      },
> > > > >      "targets": [
> > > > >          {
> > > > >              "architecture": "x86_64",
> > > > >              "machines": [
> > > > >                  "pc-q35-*"
> > > > >              ]
> > > > >          }
> > > > >      ],
> > > > >      "features": [
> > > > >          "acpi-s4",
> > > > >     "amd-sev",
> > > > >     "amd-sev-es",
> > > > >     "amd-sev-snp",
> > > > >          "verbose-dynamic"
> > > > >      ],
> > > > >      "tags": [
> > > > > 
> > > > >      ]
> > > > > }
> > > > > 
> > > > > Auto-selection works fine if I specify a 'stateless' firmware, e.g. amend
> > > > > the above config with
> > > > > 
> > > > >    <os firmware="efi">
> > > > >      <type arch="x86_64" machine="q35">hvm</type>
> > > > >      <loader stateless="yes"/>
> > > > >      <boot dev="hd"/>
> > > > >    </os>
> > > > > 
> > > > > Being unfamiliar with the firmware auto-selection code, I tried the below
> > > > > naive hack, which only led to test failures and the subsequent runtime error
> > > > > "unable to find any master var store for loader:
> > > > > /usr/share/qemu/ovmf-x86_64-sev.bin". Should auto-selection work with the
> > > > > minimal config, or is it expected that user also specify a stateless
> > > > > firmware?
> > 
> > Andrea,
> > 
> > Having spent a fair bit of time in the firmware auto-selection code,
> > perhaps you have an opinion about this?
> 
> Sorry to keep nagging about this, but I still see the issue with latest git
> master using a json descriptor identical to the latest Fedora one. When
> omitting the 'stateless' attribute, autoselection fails with
> 
> operation failed: Unable to find 'efi' firmware that is compatible with the
> current configuration
> 
> With debug enabled, I can see the firmware is not even considered since it's
> not "split"
> 
> 2025-07-29 21:02:05.517+0000: 32187: debug : qemuFirmwareMatchDomain:1310 :
> Discarding loader without split flash
> 
> I would need to install Fedora and verify myself, but I'd be surprised if it
> didn't encounter the same issue. Without the 'stateless' attribute, the
> following test would fail
> 
> https://gitlab.com/libvirt/libvirt/-/blob/master/src/qemu/qemu_firmware.c?ref_type=heads#L1303
> 
> executing the else condition, and failing similarly since the firmware is not split.

So I've tested the scenario I mentioned before, and in fact see the
same behaviour as you describe

error: operation failed: Unable to find 'efi' firmware that is compatible with the current configuration

which confuses me, because I'm sure I used virt-install to provision
SNP guests before, without expanding the args to

   --boot uefi,loader.stateless=on

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
Re: Question about SEV* guests and automatic firmware selection
Posted by Andrea Bolognani via Devel 7 months, 1 week ago
On Wed, Jul 30, 2025 at 12:50:25PM +0100, Daniel P. Berrangé wrote:
> On Tue, Jul 29, 2025 at 03:28:49PM -0600, Jim Fehlig wrote:
> > > Andrea,
> > >
> > > Having spent a fair bit of time in the firmware auto-selection code,
> > > perhaps you have an opinion about this?
> >
> > Sorry to keep nagging about this

No worries. The original message fell through the cracks somehow, so
the ping is appreciated.

> > I would need to install Fedora and verify myself, but I'd be surprised if it
> > didn't encounter the same issue. Without the 'stateless' attribute, the
> > following test would fail
> >
> > https://gitlab.com/libvirt/libvirt/-/blob/master/src/qemu/qemu_firmware.c?ref_type=heads#L1303
> >
> > executing the else condition, and failing similarly since the firmware is not split.
>
> So I've tested the scenario I mentioned before, and in fact see the
> same behaviour as you describe
>
> error: operation failed: Unable to find 'efi' firmware that is compatible with the current configuration
>
> which confuses me, because I'm sure I used virt-install to provision
> SNP guests before, without expanding the args to
>
>    --boot uefi,loader.stateless=on

I can confirm that the issue reproduces on Fedora too, and that
you've correctly identified the problem area.

Patches hopefully addressing the issue are on the list now[1].
Thanks again for bringing this to my attention!


[1] https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/thread/RKQ3ANKDNJEFJSKQR4FMDU7PVHWYKYSF/
-- 
Andrea Bolognani / Red Hat / Virtualization