There are two scenarios identified after the recent firewall backend
selection was introduced, which result in libvirtd failing to startup
due to an inability to find either iptables/nftables

 - On Linux if running unprivileged with $PATH lacking the dir
   containing iptables/nftables
 - On non-Linux where iptables/nftables never existed

In the former case, it is preferrable to restore the behaviour whereby
the driver starts successfully. Users will get an error reported when
attempting to start any virtual network, due to the lack of permissions
needed to create bridge devices. This makes the missing firewall backend
irrelevant.

In the latter case, the network driver calls the 'nop' platform
implementation which does not attempt to implement any firewall logic,
just allowing the network to start without firewall rules.

To solve this are number of changes are required

 * Introduce VIR_FIREWALL_BACKEND_NONE, which does nothing except
   report a fatal error from virFirewallApply(). This code path
   is unreachable, since we'll never create a virFirewall
   object with with VIR_FIREWALL_BACKEND_NONE, so the error reporting
   is just a sanity check.

 * Ignore the compile time backend defaults and assume use of
   the 'none' backend if running unprivileged.

   This fixes the first regression, avoiding the failure to start
   libvirtd on Linux in unprivileged context, instead allowing use
   of the driver and expecting a permission denied when creating a
   bridge.

 * Reject the use of compile time backend defaults no non-Linux
   and hardcode the 'none' backend. The non-Linux platforms have
   no firewall implementation at all currently, so there's no
   reason to permit the use of 'firewall_backend_priority'
   meson option.

   This fixes the second regression, avoiding the failure to start
   libvirtd on non-Linux hosts due to non-existant Linux binaries.

 * Change the Linux platform backend to raise an error if the
   firewall backend is 'none'. Again this code path is unreachable
   by default since we'll fail to create the bridge before getting
   here, but if someone modified network.conf to request the 'none'
   backend, this will stop further progress.

 * Change the nop platform backend to raise an error if the
   firewall backend is 'iptables' or 'nftables'. Again this code
   path is unreachable, since we should already have failed to
   find the iptables/nftables binaries on non-Linux hosts, so
   this is just a sanity check.

 * 'none' is not permited as a value in 'firewall_backend_priority'
   meson option, since it is conceptually meaningless to ask for
   that on Linux.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---

Changed in v2:

 - Fix build problems on FreeBSD (changes proposed by Roman)

 meson.build                       | 26 +++++++++++++++++++-------
 meson_options.txt                 |  2 +-
 src/network/bridge_driver_conf.c  | 19 ++++++++++++++-----
 src/network/bridge_driver_linux.c | 10 ++++++++++
 src/network/bridge_driver_nop.c   | 15 ++++++++++++++-
 src/util/virfirewall.c            |  6 ++++++
 src/util/virfirewall.h            |  1 +
 7 files changed, 65 insertions(+), 14 deletions(-)

diff --git a/meson.build b/meson.build
index 5c7cd7ec2e..2e8b87280d 100644
--- a/meson.build
+++ b/meson.build
@@ -1647,15 +1647,27 @@ if not get_option('driver_network').disabled() and conf.has('WITH_LIBVIRTD')
   conf.set('WITH_NETWORK', 1)
 
   firewall_backend_priority = get_option('firewall_backend_priority')
-  if (not firewall_backend_priority.contains('nftables') or
-      not firewall_backend_priority.contains('iptables') or
-      firewall_backend_priority.length() != 2)
-    error('invalid value for firewall_backend_priority option')
+  if firewall_backend_priority.length() == 0
+      if host_machine.system() == 'linux'
+          firewall_backend_priority = ['nftables', 'iptables']
+      else
+          # No firewall impl on non-Linux so far, so force 'none'
+          # as placeholder
+          firewall_backend_priority = ['none']
+      endif
+  else
+      if host_machine.system() != 'linux'
+          error('firewall backend priority only supported on linux hosts')
+      endif
   endif
 
-  conf.set('FIREWALL_BACKEND_PRIORITY_0', 'VIR_FIREWALL_BACKEND_' + firewall_backend_priority[0].to_upper())
-  conf.set('FIREWALL_BACKEND_PRIORITY_1', 'VIR_FIREWALL_BACKEND_' + firewall_backend_priority[1].to_upper())
-  conf.set('FIREWALL_BACKEND_PRIORITY_NUM', firewall_backend_priority.length())
+  backends = []
+  foreach backend: firewall_backend_priority
+      backend = 'VIR_FIREWALL_BACKEND_' + backend.to_upper()
+      backends += backend
+  endforeach
+
+  conf.set('FIREWALL_BACKENDS', ', '.join(backends))
 elif get_option('driver_network').enabled()
   error('libvirtd must be enabled to build the network driver')
 endif
diff --git a/meson_options.txt b/meson_options.txt
index 50d71427cb..2d440c63d8 100644
--- a/meson_options.txt
+++ b/meson_options.txt
@@ -117,7 +117,7 @@ option('dtrace', type: 'feature', value: 'auto', description: 'use dtrace for st
 option('firewalld', type: 'feature', value: 'auto', description: 'firewalld support')
 # dep:firewalld
 option('firewalld_zone', type: 'feature', value: 'auto', description: 'whether to install firewalld libvirt zone')
-option('firewall_backend_priority', type: 'array', choices: ['nftables', 'iptables'], description: 'order in which to try firewall backends')
+option('firewall_backend_priority', type: 'array', choices: ['nftables', 'iptables'], value: [], description: 'order in which to try firewall backends')
 option('host_validate', type: 'feature', value: 'auto', description: 'build virt-host-validate')
 option('init_script', type: 'combo', choices: ['systemd', 'openrc', 'check', 'none'], value: 'check', description: 'Style of init script to install')
 option('loader_nvram', type: 'string', value: '', description: 'Pass list of pairs of <loader>:<nvram> paths. Both pairs and list items are separated by a colon.')
diff --git a/src/network/bridge_driver_conf.c b/src/network/bridge_driver_conf.c
index e2f3613a41..9da5e790b7 100644
--- a/src/network/bridge_driver_conf.c
+++ b/src/network/bridge_driver_conf.c
@@ -61,6 +61,7 @@ networkGetDnsmasqCaps(virNetworkDriverState *driver)
 
 static int
 virNetworkLoadDriverConfig(virNetworkDriverConfig *cfg G_GNUC_UNUSED,
+                           bool privileged,
                            const char *filename)
 {
     g_autoptr(virConf) conf = NULL;
@@ -68,13 +69,17 @@ virNetworkLoadDriverConfig(virNetworkDriverConfig *cfg G_GNUC_UNUSED,
     bool fwBackendSelected = false;
     size_t i;
     int fwBackends[] = {
-        FIREWALL_BACKEND_PRIORITY_0,
-        FIREWALL_BACKEND_PRIORITY_1,
+        FIREWALL_BACKENDS
     };
-    G_STATIC_ASSERT(G_N_ELEMENTS(fwBackends) == VIR_FIREWALL_BACKEND_LAST);
-    G_STATIC_ASSERT(G_N_ELEMENTS(fwBackends) == FIREWALL_BACKEND_PRIORITY_NUM);
+    G_STATIC_ASSERT(G_N_ELEMENTS(fwBackends) > 0 &&
+                    G_N_ELEMENTS(fwBackends) <= VIR_FIREWALL_BACKEND_LAST);
     int nFwBackends = G_N_ELEMENTS(fwBackends);
 
+    if (!privileged) {
+        fwBackends[0] = VIR_FIREWALL_BACKEND_NONE;
+        nFwBackends = 1;
+    }
+
     if (access(filename, R_OK) == 0) {
 
         conf = virConfReadFile(filename, 0);
@@ -104,6 +109,10 @@ virNetworkLoadDriverConfig(virNetworkDriverConfig *cfg G_GNUC_UNUSED,
     for (i = 0; i < nFwBackends && !fwBackendSelected; i++) {
 
         switch ((virFirewallBackend)fwBackends[i]) {
+        case VIR_FIREWALL_BACKEND_NONE:
+            fwBackendSelected = true;
+            break;
+
         case VIR_FIREWALL_BACKEND_IPTABLES: {
             g_autofree char *iptablesInPath = virFindFileInPath(IPTABLES);
 
@@ -187,7 +196,7 @@ virNetworkDriverConfigNew(bool privileged)
 
     configfile = g_strconcat(configdir, "/network.conf", NULL);
 
-    if (virNetworkLoadDriverConfig(cfg, configfile) < 0)
+    if (virNetworkLoadDriverConfig(cfg, privileged, configfile) < 0)
         return NULL;
 
     if (g_mkdir_with_parents(cfg->stateDir, 0777) < 0) {
diff --git a/src/network/bridge_driver_linux.c b/src/network/bridge_driver_linux.c
index 35e6bd1154..fe7c6e193c 100644
--- a/src/network/bridge_driver_linux.c
+++ b/src/network/bridge_driver_linux.c
@@ -47,6 +47,11 @@ networkFirewallSetupPrivateChains(virFirewallBackend backend,
                                   virFirewallLayer layer)
 {
     switch (backend) {
+    case VIR_FIREWALL_BACKEND_NONE:
+        virReportError(VIR_ERR_NO_SUPPORT, "%s",
+                       _("No firewall backend is available"));
+        return -1;
+
     case VIR_FIREWALL_BACKEND_IPTABLES:
         return iptablesSetupPrivateChains(layer);
 
@@ -417,6 +422,11 @@ networkAddFirewallRules(virNetworkDef *def,
     }
 
     switch (firewallBackend) {
+    case VIR_FIREWALL_BACKEND_NONE:
+        virReportError(VIR_ERR_NO_SUPPORT, "%s",
+                       _("No firewall backend is available"));
+        return -1;
+
     case VIR_FIREWALL_BACKEND_IPTABLES:
         return iptablesAddFirewallRules(def, fwRemoval);
 
diff --git a/src/network/bridge_driver_nop.c b/src/network/bridge_driver_nop.c
index 537b9234f8..2114d521d1 100644
--- a/src/network/bridge_driver_nop.c
+++ b/src/network/bridge_driver_nop.c
@@ -19,6 +19,8 @@
 
 #include <config.h>
 
+#define VIR_FROM_THIS VIR_FROM_NETWORK
+
 void networkPreReloadFirewallRules(virNetworkDriverState *driver G_GNUC_UNUSED,
                                    bool startup G_GNUC_UNUSED,
                                    bool force G_GNUC_UNUSED)
@@ -37,9 +39,20 @@ int networkCheckRouteCollision(virNetworkDef *def G_GNUC_UNUSED)
 }
 
 int networkAddFirewallRules(virNetworkDef *def G_GNUC_UNUSED,
-                            virFirewallBackend firewallBackend G_GNUC_UNUSED,
+                            virFirewallBackend firewallBackend,
                             virFirewall **fwRemoval G_GNUC_UNUSED)
 {
+    /*
+     * Shouldn't be possible, since virNetworkLoadDriverConfig
+     * ought to fail to find the required binaries when loading,
+     * so this is just a sanity check
+     */
+    if (firewallBackend != VIR_FIREWALL_BACKEND_NONE) {
+        virReportError(VIR_ERR_NO_SUPPORT,
+                       _("Firewall backend '%s' not available on this platform"),
+                       virFirewallBackendTypeToString(firewallBackend));
+        return -1;
+    }
     return 0;
 }
 
diff --git a/src/util/virfirewall.c b/src/util/virfirewall.c
index 2219506b18..d374f54b64 100644
--- a/src/util/virfirewall.c
+++ b/src/util/virfirewall.c
@@ -37,6 +37,7 @@ VIR_LOG_INIT("util.firewall");
 
 VIR_ENUM_IMPL(virFirewallBackend,
               VIR_FIREWALL_BACKEND_LAST,
+              "none",
               "iptables",
               "nftables");
 
@@ -815,6 +816,11 @@ virFirewallApplyCmd(virFirewall *firewall,
     }
 
     switch (virFirewallGetBackend(firewall)) {
+    case VIR_FIREWALL_BACKEND_NONE:
+        virReportError(VIR_ERR_NO_SUPPORT,
+                       _("Firewall backend is not implemented"));
+        return -1;
+
     case VIR_FIREWALL_BACKEND_IPTABLES:
         if (virFirewallCmdIptablesApply(firewall, fwCmd, &output) < 0)
             return -1;
diff --git a/src/util/virfirewall.h b/src/util/virfirewall.h
index 302a6a4e5b..bce51259d2 100644
--- a/src/util/virfirewall.h
+++ b/src/util/virfirewall.h
@@ -44,6 +44,7 @@ typedef enum {
 } virFirewallLayer;
 
 typedef enum {
+    VIR_FIREWALL_BACKEND_NONE, /* Always fails */
     VIR_FIREWALL_BACKEND_IPTABLES,
     VIR_FIREWALL_BACKEND_NFTABLES,
 
-- 
2.45.1

On Fri, Jun 14, 2024 at 03:43:53PM GMT, Daniel P. Berrangé wrote:
>  meson.build                       | 26 +++++++++++++++++++-------
>  meson_options.txt                 |  2 +-
>  src/network/bridge_driver_conf.c  | 19 ++++++++++++++-----
>  src/network/bridge_driver_linux.c | 10 ++++++++++
>  src/network/bridge_driver_nop.c   | 15 ++++++++++++++-
>  src/util/virfirewall.c            |  6 ++++++
>  src/util/virfirewall.h            |  1 +
>  7 files changed, 65 insertions(+), 14 deletions(-)

The test suite no longer passes after applying this. At the very
least, you need to squash in the diff at the bottom of this message.

>    firewall_backend_priority = get_option('firewall_backend_priority')
> -  if (not firewall_backend_priority.contains('nftables') or
> -      not firewall_backend_priority.contains('iptables') or
> -      firewall_backend_priority.length() != 2)
> -    error('invalid value for firewall_backend_priority option')
> +  if firewall_backend_priority.length() == 0
> +      if host_machine.system() == 'linux'
> +          firewall_backend_priority = ['nftables', 'iptables']
> +      else
> +          # No firewall impl on non-Linux so far, so force 'none'
> +          # as placeholder
> +          firewall_backend_priority = ['none']
> +      endif
> +  else
> +      if host_machine.system() != 'linux'
> +          error('firewall backend priority only supported on linux hosts')
> +      endif
>    endif

This implementation allows things such as

  -Dfirewall_backend_priority=nftables

and

  -Dfirewall_backend_priority=iptables,iptables

At least

  -Dfirewall_backend_priority=iptables,nftables,iptables

will be blocked, but only because it results in a compilation error:
meson will happily accept it.

Are we okay with that? It's IMO inferior to the much stricter
checking that's performed today.

> @@ -815,6 +816,11 @@ virFirewallApplyCmd(virFirewall *firewall,
>      switch (virFirewallGetBackend(firewall)) {
> +    case VIR_FIREWALL_BACKEND_NONE:
> +        virReportError(VIR_ERR_NO_SUPPORT,
> +                       _("Firewall backend is not implemented"));
> +        return -1;

This check (and the other ones you've introduced) are running too
late, which leads to this behavior:

  $ cat default-session.xml
  <network>
    <name>default-session</name>
    <forward mode='nat'/>
    <bridge name='virbr0' stp='on' delay='0'/>
    <ip address='192.168.123.1' netmask='255.255.255.0'>
      <dhcp>
        <range start='192.168.123.2' end='192.168.123.254'/>
      </dhcp>
    </ip>
  </network>

  $ virsh -c qemu:///session net-define default-session.xml
  Network default-session defined from default-session.xml

  $ virsh -c qemu:///session net-start default-session
  error: Failed to start network default-session
  error: error creating bridge interface virbr0: Operation not permitted

Since <forward mode='nat'/> requires firewall rules, we shouldn't
allow a network that uses it to be defined in the first place.



diff --git a/po/POTFILES b/po/POTFILES
index 4bfbb91164..1ed4086d2c 100644
--- a/po/POTFILES
+++ b/po/POTFILES
@@ -143,6 +143,7 @@ src/lxc/lxc_process.c
 src/network/bridge_driver.c
 src/network/bridge_driver_conf.c
 src/network/bridge_driver_linux.c
+src/network/bridge_driver_nop.c
 src/network/leaseshelper.c
 src/network/network_iptables.c
 src/network/network_nftables.c
diff --git a/src/network/bridge_driver_nop.c b/src/network/bridge_driver_nop.c
index 2114d521d1..323990cf17 100644
--- a/src/network/bridge_driver_nop.c
+++ b/src/network/bridge_driver_nop.c
@@ -49,8 +49,8 @@ int networkAddFirewallRules(virNetworkDef *def G_GNUC_UNUSED,
      */
     if (firewallBackend != VIR_FIREWALL_BACKEND_NONE) {
         virReportError(VIR_ERR_NO_SUPPORT,
-                       _("Firewall backend '%s' not available on this
platform"),
-                       virFirewallBackendTypeToString(firewallBackend));
+                       _("Firewall backend '%1$s' not available on
this platform"),
+                       virFirewallBackendTypeToSTring(firewallBackend));
         return -1;
     }
     return 0;
diff --git a/src/util/virfirewall.c b/src/util/virfirewall.c
index d374f54b64..090dbcdbed 100644
--- a/src/util/virfirewall.c
+++ b/src/util/virfirewall.c
@@ -817,7 +817,7 @@ virFirewallApplyCmd(virFirewall *firewall,

     switch (virFirewallGetBackend(firewall)) {
     case VIR_FIREWALL_BACKEND_NONE:
-        virReportError(VIR_ERR_NO_SUPPORT,
+        virReportError(VIR_ERR_NO_SUPPORT, "%s",
                        _("Firewall backend is not implemented"));
         return -1;

-- 
Andrea Bolognani / Red Hat / Virtualization

On Fri, Jun 14, 2024 at 12:22:50PM -0400, Andrea Bolognani wrote:
> On Fri, Jun 14, 2024 at 03:43:53PM GMT, Daniel P. Berrangé wrote:
> >  meson.build                       | 26 +++++++++++++++++++-------
> >  meson_options.txt                 |  2 +-
> >  src/network/bridge_driver_conf.c  | 19 ++++++++++++++-----
> >  src/network/bridge_driver_linux.c | 10 ++++++++++
> >  src/network/bridge_driver_nop.c   | 15 ++++++++++++++-
> >  src/util/virfirewall.c            |  6 ++++++
> >  src/util/virfirewall.h            |  1 +
> >  7 files changed, 65 insertions(+), 14 deletions(-)
> 
> The test suite no longer passes after applying this. At the very
> least, you need to squash in the diff at the bottom of this message.
> 
> >    firewall_backend_priority = get_option('firewall_backend_priority')
> > -  if (not firewall_backend_priority.contains('nftables') or
> > -      not firewall_backend_priority.contains('iptables') or
> > -      firewall_backend_priority.length() != 2)
> > -    error('invalid value for firewall_backend_priority option')
> > +  if firewall_backend_priority.length() == 0
> > +      if host_machine.system() == 'linux'
> > +          firewall_backend_priority = ['nftables', 'iptables']
> > +      else
> > +          # No firewall impl on non-Linux so far, so force 'none'
> > +          # as placeholder
> > +          firewall_backend_priority = ['none']
> > +      endif
> > +  else
> > +      if host_machine.system() != 'linux'
> > +          error('firewall backend priority only supported on linux hosts')
> > +      endif
> >    endif
> 
> This implementation allows things such as
> 
>   -Dfirewall_backend_priority=nftables
> 
> and
> 
>   -Dfirewall_backend_priority=iptables,iptables
> 
> At least
> 
>   -Dfirewall_backend_priority=iptables,nftables,iptables
> 
> will be blocked, but only because it results in a compilation error:
> meson will happily accept it.
> 
> Are we okay with that? It's IMO inferior to the much stricter
> checking that's performed today.

I found that if you try this with meson you'll see this

DEPRECATION: Duplicated values in array option is deprecated. This will become a hard error in the future.

I think we're fine to delegate this to Meson, given its intent to turn
this into a hard error eventually, since duplication is harmless for
us in the short term.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Mon, Jun 17, 2024 at 09:36:15AM GMT, Daniel P. Berrangé wrote:
> On Fri, Jun 14, 2024 at 12:22:50PM -0400, Andrea Bolognani wrote:
> > This implementation allows things such as
> >
> >   -Dfirewall_backend_priority=nftables
> >
> > and
> >
> >   -Dfirewall_backend_priority=iptables,iptables
> >
> > At least
> >
> >   -Dfirewall_backend_priority=iptables,nftables,iptables
> >
> > will be blocked, but only because it results in a compilation error:
> > meson will happily accept it.
> >
> > Are we okay with that? It's IMO inferior to the much stricter
> > checking that's performed today.
>
> I found that if you try this with meson you'll see this
>
> DEPRECATION: Duplicated values in array option is deprecated. This will become a hard error in the future.
>
> I think we're fine to delegate this to Meson, given its intent to turn
> this into a hard error eventually, since duplication is harmless for
> us in the short term.

I didn't realize that. Sounds good!

-- 
Andrea Bolognani / Red Hat / Virtualization

On 6/14/24 12:22 PM, Andrea Bolognani wrote:
> On Fri, Jun 14, 2024 at 03:43:53PM GMT, Daniel P. Berrangé wrote:
>>   meson.build                       | 26 +++++++++++++++++++-------
>>   meson_options.txt                 |  2 +-
>>   src/network/bridge_driver_conf.c  | 19 ++++++++++++++-----
>>   src/network/bridge_driver_linux.c | 10 ++++++++++
>>   src/network/bridge_driver_nop.c   | 15 ++++++++++++++-
>>   src/util/virfirewall.c            |  6 ++++++
>>   src/util/virfirewall.h            |  1 +
>>   7 files changed, 65 insertions(+), 14 deletions(-)
> 
> The test suite no longer passes after applying this. At the very
> least, you need to squash in the diff at the bottom of this message.
> 
>>     firewall_backend_priority = get_option('firewall_backend_priority')
>> -  if (not firewall_backend_priority.contains('nftables') or
>> -      not firewall_backend_priority.contains('iptables') or
>> -      firewall_backend_priority.length() != 2)
>> -    error('invalid value for firewall_backend_priority option')
>> +  if firewall_backend_priority.length() == 0
>> +      if host_machine.system() == 'linux'
>> +          firewall_backend_priority = ['nftables', 'iptables']
>> +      else
>> +          # No firewall impl on non-Linux so far, so force 'none'
>> +          # as placeholder
>> +          firewall_backend_priority = ['none']
>> +      endif
>> +  else
>> +      if host_machine.system() != 'linux'
>> +          error('firewall backend priority only supported on linux hosts')
>> +      endif
>>     endif

I suppose this is true for now, but will hopefully some day become false :-)

> 
> This implementation allows things such as
> 
>    -Dfirewall_backend_priority=nftables
> 
> and
> 
>    -Dfirewall_backend_priority=iptables,iptables
> 
> At least
> 
>    -Dfirewall_backend_priority=iptables,nftables,iptables
> 
> will be blocked, but only because it results in a compilation error:
> meson will happily accept it.
> 
> Are we okay with that? It's IMO inferior to the much stricter
> checking that's performed today.

I haven't had the time to test this, or even look at it in detail, since 
I'm leaving for a multi-week vacation on Sunday, but I thought I should 
at least give my opinion on how I think it *should* work - maybe it 
already *does* work as I describe)

What I would most prefer would be if you could list 1 or more backends 
(i.e. not all backends need to be listed, and if a particular backend 
isn't listed then it isn't checked for automatically (maybe isn't even 
allowed to be set explicitly), and ideally wouldn't even be compiled in).

It would be nice if it gave an error if the same backend was given more 
than once in the list, but since you can probably count the number of 
people who will ever mess with this setting with the fingers on your two 
hands (and maybe the addition of your toes), and they're hopefully 
mentally aware enough to realize that naming the same backend multiple 
time would be pointless, I don't think a check to prevent duplicates is 
absolutely necessary (if fixing it would mean that you can't list fewer 
than the total available backends - someone on Linux might want to only 
enable nftables).

To say it by example, I think on a Linux system it should work to define 
firewall_backend_prioty as:

    nftables,iptables
    iptables,nftables
    nftables
    iptables
    null

Currently on FreeBSD the only acceptable setting would be "null", but in 
the future it could be  "ipfw,pf", or the opposite order, or just one or 
the other.

> 
>> @@ -815,6 +816,11 @@ virFirewallApplyCmd(virFirewall *firewall,
>>       switch (virFirewallGetBackend(firewall)) {
>> +    case VIR_FIREWALL_BACKEND_NONE:
>> +        virReportError(VIR_ERR_NO_SUPPORT,
>> +                       _("Firewall backend is not implemented"));
>> +        return -1;
> 
> This check (and the other ones you've introduced) are running too
> late, which leads to this behavior:
> 
>    $ cat default-session.xml
>    <network>
>      <name>default-session</name>
>      <forward mode='nat'/>
>      <bridge name='virbr0' stp='on' delay='0'/>
>      <ip address='192.168.123.1' netmask='255.255.255.0'>
>        <dhcp>
>          <range start='192.168.123.2' end='192.168.123.254'/>
>        </dhcp>
>      </ip>
>    </network>
> 
>    $ virsh -c qemu:///session net-define default-session.xml
>    Network default-session defined from default-session.xml
> 
>    $ virsh -c qemu:///session net-start default-session
>    error: Failed to start network default-session
>    error: error creating bridge interface virbr0: Operation not permitted
> 
> Since <forward mode='nat'/> requires firewall rules, we shouldn't
> allow a network that uses it to be defined in the first place.
> 

I agree with Dan that we shouldn't prevent anyone from *defining* such a 
network, we should just log an error and fail if they try to start such 
a network on a system that doesn't have a non-null firewall backend.

> 
> 
> diff --git a/po/POTFILES b/po/POTFILES
> index 4bfbb91164..1ed4086d2c 100644
> --- a/po/POTFILES
> +++ b/po/POTFILES
> @@ -143,6 +143,7 @@ src/lxc/lxc_process.c
>   src/network/bridge_driver.c
>   src/network/bridge_driver_conf.c
>   src/network/bridge_driver_linux.c
> +src/network/bridge_driver_nop.c
>   src/network/leaseshelper.c
>   src/network/network_iptables.c
>   src/network/network_nftables.c
> diff --git a/src/network/bridge_driver_nop.c b/src/network/bridge_driver_nop.c
> index 2114d521d1..323990cf17 100644
> --- a/src/network/bridge_driver_nop.c
> +++ b/src/network/bridge_driver_nop.c
> @@ -49,8 +49,8 @@ int networkAddFirewallRules(virNetworkDef *def G_GNUC_UNUSED,
>        */
>       if (firewallBackend != VIR_FIREWALL_BACKEND_NONE) {
>           virReportError(VIR_ERR_NO_SUPPORT,
> -                       _("Firewall backend '%s' not available on this
> platform"),
> -                       virFirewallBackendTypeToString(firewallBackend));
> +                       _("Firewall backend '%1$s' not available on
> this platform"),
> +                       virFirewallBackendTypeToSTring(firewallBackend));
>           return -1;
>       }
>       return 0;
> diff --git a/src/util/virfirewall.c b/src/util/virfirewall.c
> index d374f54b64..090dbcdbed 100644
> --- a/src/util/virfirewall.c
> +++ b/src/util/virfirewall.c
> @@ -817,7 +817,7 @@ virFirewallApplyCmd(virFirewall *firewall,
> 
>       switch (virFirewallGetBackend(firewall)) {
>       case VIR_FIREWALL_BACKEND_NONE:
> -        virReportError(VIR_ERR_NO_SUPPORT,
> +        virReportError(VIR_ERR_NO_SUPPORT, "%s",
>                          _("Firewall backend is not implemented"));
>           return -1;
> 

On Fri, Jun 14, 2024 at 12:22:50PM -0400, Andrea Bolognani wrote:
> On Fri, Jun 14, 2024 at 03:43:53PM GMT, Daniel P. Berrangé wrote:
> >  meson.build                       | 26 +++++++++++++++++++-------
> >  meson_options.txt                 |  2 +-
> >  src/network/bridge_driver_conf.c  | 19 ++++++++++++++-----
> >  src/network/bridge_driver_linux.c | 10 ++++++++++
> >  src/network/bridge_driver_nop.c   | 15 ++++++++++++++-
> >  src/util/virfirewall.c            |  6 ++++++
> >  src/util/virfirewall.h            |  1 +
> >  7 files changed, 65 insertions(+), 14 deletions(-)
> 
> The test suite no longer passes after applying this. At the very
> least, you need to squash in the diff at the bottom of this message.

Well that's very wierd as I ran a CI pipeline with this commit
which worked !

https://gitlab.com/berrange/libvirt/-/pipelines/1332837446

but you are correct that it fails tests, so huh ?

> 
> >    firewall_backend_priority = get_option('firewall_backend_priority')
> > -  if (not firewall_backend_priority.contains('nftables') or
> > -      not firewall_backend_priority.contains('iptables') or
> > -      firewall_backend_priority.length() != 2)
> > -    error('invalid value for firewall_backend_priority option')
> > +  if firewall_backend_priority.length() == 0
> > +      if host_machine.system() == 'linux'
> > +          firewall_backend_priority = ['nftables', 'iptables']
> > +      else
> > +          # No firewall impl on non-Linux so far, so force 'none'
> > +          # as placeholder
> > +          firewall_backend_priority = ['none']
> > +      endif
> > +  else
> > +      if host_machine.system() != 'linux'
> > +          error('firewall backend priority only supported on linux hosts')
> > +      endif
> >    endif
> 
> This implementation allows things such as
> 
>   -Dfirewall_backend_priority=nftables

That's good, it allows preventing any fallback to iptables
by default.

> 
> and
> 
>   -Dfirewall_backend_priority=iptables,iptables
> 
> At least
> 
>   -Dfirewall_backend_priority=iptables,nftables,iptables
> 
> will be blocked, but only because it results in a compilation error:
> meson will happily accept it.
> 
> Are we okay with that? It's IMO inferior to the much stricter
> checking that's performed today.

I don't think it really matters - if someone wants to pass
a wierd config, so be it.

> 
> > @@ -815,6 +816,11 @@ virFirewallApplyCmd(virFirewall *firewall,
> >      switch (virFirewallGetBackend(firewall)) {
> > +    case VIR_FIREWALL_BACKEND_NONE:
> > +        virReportError(VIR_ERR_NO_SUPPORT,
> > +                       _("Firewall backend is not implemented"));
> > +        return -1;
> 
> This check (and the other ones you've introduced) are running too
> late, which leads to this behavior:
> 
>   $ cat default-session.xml
>   <network>
>     <name>default-session</name>
>     <forward mode='nat'/>
>     <bridge name='virbr0' stp='on' delay='0'/>
>     <ip address='192.168.123.1' netmask='255.255.255.0'>
>       <dhcp>
>         <range start='192.168.123.2' end='192.168.123.254'/>
>       </dhcp>
>     </ip>
>   </network>
> 
>   $ virsh -c qemu:///session net-define default-session.xml
>   Network default-session defined from default-session.xml
> 
>   $ virsh -c qemu:///session net-start default-session
>   error: Failed to start network default-session
>   error: error creating bridge interface virbr0: Operation not permitted
> 
> Since <forward mode='nat'/> requires firewall rules, we shouldn't
> allow a network that uses it to be defined in the first place.

This is the behaviour we already had before the nftables backend
was created, and its not been a problem. There's no functional
reason why we should refuse to allow it to be defined, if the
binaries aren't needed until startup.

> diff --git a/po/POTFILES b/po/POTFILES
> index 4bfbb91164..1ed4086d2c 100644
> --- a/po/POTFILES
> +++ b/po/POTFILES
> @@ -143,6 +143,7 @@ src/lxc/lxc_process.c
>  src/network/bridge_driver.c
>  src/network/bridge_driver_conf.c
>  src/network/bridge_driver_linux.c
> +src/network/bridge_driver_nop.c
>  src/network/leaseshelper.c
>  src/network/network_iptables.c
>  src/network/network_nftables.c
> diff --git a/src/network/bridge_driver_nop.c b/src/network/bridge_driver_nop.c
> index 2114d521d1..323990cf17 100644
> --- a/src/network/bridge_driver_nop.c
> +++ b/src/network/bridge_driver_nop.c
> @@ -49,8 +49,8 @@ int networkAddFirewallRules(virNetworkDef *def G_GNUC_UNUSED,
>       */
>      if (firewallBackend != VIR_FIREWALL_BACKEND_NONE) {
>          virReportError(VIR_ERR_NO_SUPPORT,
> -                       _("Firewall backend '%s' not available on this
> platform"),
> -                       virFirewallBackendTypeToString(firewallBackend));
> +                       _("Firewall backend '%1$s' not available on
> this platform"),
> +                       virFirewallBackendTypeToSTring(firewallBackend));
>          return -1;
>      }
>      return 0;
> diff --git a/src/util/virfirewall.c b/src/util/virfirewall.c
> index d374f54b64..090dbcdbed 100644
> --- a/src/util/virfirewall.c
> +++ b/src/util/virfirewall.c
> @@ -817,7 +817,7 @@ virFirewallApplyCmd(virFirewall *firewall,
> 
>      switch (virFirewallGetBackend(firewall)) {
>      case VIR_FIREWALL_BACKEND_NONE:
> -        virReportError(VIR_ERR_NO_SUPPORT,
> +        virReportError(VIR_ERR_NO_SUPPORT, "%s",
>                         _("Firewall backend is not implemented"));
>          return -1;
> 
> -- 
> Andrea Bolognani / Red Hat / Virtualization
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Fri, Jun 14, 2024 at 07:52:55PM GMT, Daniel P. Berrangé wrote:
> On Fri, Jun 14, 2024 at 12:22:50PM -0400, Andrea Bolognani wrote:
> > On Fri, Jun 14, 2024 at 03:43:53PM GMT, Daniel P. Berrangé wrote:
> > > @@ -815,6 +816,11 @@ virFirewallApplyCmd(virFirewall *firewall,
> > >      switch (virFirewallGetBackend(firewall)) {
> > > +    case VIR_FIREWALL_BACKEND_NONE:
> > > +        virReportError(VIR_ERR_NO_SUPPORT,
> > > +                       _("Firewall backend is not implemented"));
> > > +        return -1;
> >
> > This check (and the other ones you've introduced) are running too
> > late, which leads to this behavior:
> >
> >   $ cat default-session.xml
> >   <network>
> >     <name>default-session</name>
> >     <forward mode='nat'/>
> >     <bridge name='virbr0' stp='on' delay='0'/>
> >     <ip address='192.168.123.1' netmask='255.255.255.0'>
> >       <dhcp>
> >         <range start='192.168.123.2' end='192.168.123.254'/>
> >       </dhcp>
> >     </ip>
> >   </network>
> >
> >   $ virsh -c qemu:///session net-define default-session.xml
> >   Network default-session defined from default-session.xml
> >
> >   $ virsh -c qemu:///session net-start default-session
> >   error: Failed to start network default-session
> >   error: error creating bridge interface virbr0: Operation not permitted
> >
> > Since <forward mode='nat'/> requires firewall rules, we shouldn't
> > allow a network that uses it to be defined in the first place.
>
> This is the behaviour we already had before the nftables backend
> was created, and its not been a problem. There's no functional
> reason why we should refuse to allow it to be defined, if the
> binaries aren't needed until startup.

But in the case of qemu:///session, or when we're not on Linux, we
already know that there is no way for the network that's being
defined to ever work.

To me that's the same as allowing a guest to be defined, when the
corresponding QEMU binary doesn't support some of the devices. Or
when a firmware binary satisfying the constraints isn't available. We
reject such configurations for guests, and it would just be
consistent to do the same for networks.

Besides, even if we decide that we want to allow such networks to be
defined, we should report a more meaningful error for the failed
startup, one which points out that mode=nat will never work. It
shouldn't be the same error message that I was getting on FreeBSD
because of a kernel driver that wasn't loaded, as the user's ability
to make the error go away is completely different in the two
scenarios.

-- 
Andrea Bolognani / Red Hat / Virtualization

On Mon, Jun 17, 2024 at 04:44:01AM -0400, Andrea Bolognani wrote:
> On Fri, Jun 14, 2024 at 07:52:55PM GMT, Daniel P. Berrangé wrote:
> > On Fri, Jun 14, 2024 at 12:22:50PM -0400, Andrea Bolognani wrote:
> > > On Fri, Jun 14, 2024 at 03:43:53PM GMT, Daniel P. Berrangé wrote:
> > > > @@ -815,6 +816,11 @@ virFirewallApplyCmd(virFirewall *firewall,
> > > >      switch (virFirewallGetBackend(firewall)) {
> > > > +    case VIR_FIREWALL_BACKEND_NONE:
> > > > +        virReportError(VIR_ERR_NO_SUPPORT,
> > > > +                       _("Firewall backend is not implemented"));
> > > > +        return -1;
> > >
> > > This check (and the other ones you've introduced) are running too
> > > late, which leads to this behavior:
> > >
> > >   $ cat default-session.xml
> > >   <network>
> > >     <name>default-session</name>
> > >     <forward mode='nat'/>
> > >     <bridge name='virbr0' stp='on' delay='0'/>
> > >     <ip address='192.168.123.1' netmask='255.255.255.0'>
> > >       <dhcp>
> > >         <range start='192.168.123.2' end='192.168.123.254'/>
> > >       </dhcp>
> > >     </ip>
> > >   </network>
> > >
> > >   $ virsh -c qemu:///session net-define default-session.xml
> > >   Network default-session defined from default-session.xml
> > >
> > >   $ virsh -c qemu:///session net-start default-session
> > >   error: Failed to start network default-session
> > >   error: error creating bridge interface virbr0: Operation not permitted
> > >
> > > Since <forward mode='nat'/> requires firewall rules, we shouldn't
> > > allow a network that uses it to be defined in the first place.
> >
> > This is the behaviour we already had before the nftables backend
> > was created, and its not been a problem. There's no functional
> > reason why we should refuse to allow it to be defined, if the
> > binaries aren't needed until startup.
> 
> But in the case of qemu:///session, or when we're not on Linux, we
> already know that there is no way for the network that's being
> defined to ever work.
> 
> To me that's the same as allowing a guest to be defined, when the
> corresponding QEMU binary doesn't support some of the devices. Or
> when a firmware binary satisfying the constraints isn't available. We
> reject such configurations for guests, and it would just be
> consistent to do the same for networks.

FWIW, I find the QEMU behaviour really unpleasant / hostile, when I'm
working with guests that reference a self-built QEMU, as the binary
often "doesn't exist", if I've cleaned by build temporarily. We're
forced into this practice in QEMU, because we need to expand the
XML with defaults to fix the guest ABI. I don't like the idea of
propogating this practice elsewhere though.

Ultimately I think we should not provide the network driver at all
when running as non-root, but my attempt todo that exposed wierd
edgecases, as we've assumed the secondary drivers are always present
and cleaning that up properly is quite a bit of work

> Besides, even if we decide that we want to allow such networks to be
> defined, we should report a more meaningful error for the failed
> startup, one which points out that mode=nat will never work. It
> shouldn't be the same error message that I was getting on FreeBSD
> because of a kernel driver that wasn't loaded, as the user's ability
> to make the error go away is completely different in the two
> scenarios.

Overall, there's lots that could be improved in the network driver,
but for this, I'd really just like to focus on fixing the regression,
suc that we return to the behaviour we had historically.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

On Mon, Jun 17, 2024 at 01:40:16PM GMT, Daniel P. Berrangé wrote:
> On Mon, Jun 17, 2024 at 04:44:01AM -0400, Andrea Bolognani wrote:
> > On Fri, Jun 14, 2024 at 07:52:55PM GMT, Daniel P. Berrangé wrote:
> > > This is the behaviour we already had before the nftables backend
> > > was created, and its not been a problem. There's no functional
> > > reason why we should refuse to allow it to be defined, if the
> > > binaries aren't needed until startup.
> >
> > But in the case of qemu:///session, or when we're not on Linux, we
> > already know that there is no way for the network that's being
> > defined to ever work.
> >
> > To me that's the same as allowing a guest to be defined, when the
> > corresponding QEMU binary doesn't support some of the devices. Or
> > when a firmware binary satisfying the constraints isn't available. We
> > reject such configurations for guests, and it would just be
> > consistent to do the same for networks.
>
> FWIW, I find the QEMU behaviour really unpleasant / hostile, when I'm
> working with guests that reference a self-built QEMU, as the binary
> often "doesn't exist", if I've cleaned by build temporarily.

That makes things slightly annoying for developers, sure, but it's
much more friendly towards users IMO. Getting an error sooner
(define) rather than later (start) for situations where we know in
advance that things can't possibly ever work is a lot better, which
is also why libvirt erroring out is more user friendly than running
QEMU and let if fail instead.

> Overall, there's lots that could be improved in the network driver,
> but for this, I'd really just like to focus on fixing the regression,
> suc that we return to the behaviour we had historically.

Clearly I'm not entirely happy with the behavior you're implementing
with this, but I agree that it's still much better than leaving
several common configurations completely broken.

Reviewed-by: Andrea Bolognani <abologna@redhat.com>

-- 
Andrea Bolognani / Red Hat / Virtualization

On 6/14/24 16:43, Daniel P. Berrangé wrote:
> There are two scenarios identified after the recent firewall backend
> selection was introduced, which result in libvirtd failing to startup
> due to an inability to find either iptables/nftables
> 
>  - On Linux if running unprivileged with $PATH lacking the dir
>    containing iptables/nftables
>  - On non-Linux where iptables/nftables never existed
> 
> In the former case, it is preferrable to restore the behaviour whereby
> the driver starts successfully. Users will get an error reported when
> attempting to start any virtual network, due to the lack of permissions
> needed to create bridge devices. This makes the missing firewall backend
> irrelevant.
> 
> In the latter case, the network driver calls the 'nop' platform
> implementation which does not attempt to implement any firewall logic,
> just allowing the network to start without firewall rules.
> 
> To solve this are number of changes are required
> 
>  * Introduce VIR_FIREWALL_BACKEND_NONE, which does nothing except
>    report a fatal error from virFirewallApply(). This code path
>    is unreachable, since we'll never create a virFirewall
>    object with with VIR_FIREWALL_BACKEND_NONE, so the error reporting
>    is just a sanity check.
> 
>  * Ignore the compile time backend defaults and assume use of
>    the 'none' backend if running unprivileged.
> 
>    This fixes the first regression, avoiding the failure to start
>    libvirtd on Linux in unprivileged context, instead allowing use
>    of the driver and expecting a permission denied when creating a
>    bridge.
> 
>  * Reject the use of compile time backend defaults no non-Linux
>    and hardcode the 'none' backend. The non-Linux platforms have
>    no firewall implementation at all currently, so there's no
>    reason to permit the use of 'firewall_backend_priority'
>    meson option.
> 
>    This fixes the second regression, avoiding the failure to start
>    libvirtd on non-Linux hosts due to non-existant Linux binaries.
> 
>  * Change the Linux platform backend to raise an error if the
>    firewall backend is 'none'. Again this code path is unreachable
>    by default since we'll fail to create the bridge before getting
>    here, but if someone modified network.conf to request the 'none'
>    backend, this will stop further progress.
> 
>  * Change the nop platform backend to raise an error if the
>    firewall backend is 'iptables' or 'nftables'. Again this code
>    path is unreachable, since we should already have failed to
>    find the iptables/nftables binaries on non-Linux hosts, so
>    this is just a sanity check.
> 
>  * 'none' is not permited as a value in 'firewall_backend_priority'
>    meson option, since it is conceptually meaningless to ask for
>    that on Linux.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
> 
> Changed in v2:
> 
>  - Fix build problems on FreeBSD (changes proposed by Roman)
> 
>  meson.build                       | 26 +++++++++++++++++++-------
>  meson_options.txt                 |  2 +-
>  src/network/bridge_driver_conf.c  | 19 ++++++++++++++-----
>  src/network/bridge_driver_linux.c | 10 ++++++++++
>  src/network/bridge_driver_nop.c   | 15 ++++++++++++++-
>  src/util/virfirewall.c            |  6 ++++++
>  src/util/virfirewall.h            |  1 +
>  7 files changed, 65 insertions(+), 14 deletions(-)

Reviewed-by: Michal Privoznik <mprivozn@redhat.com>

Michal