From nobody Sat Nov 23 18:07:26 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.libvirt.org designates 8.43.85.245 as permitted sender) client-ip=8.43.85.245; envelope-from=devel-bounces@lists.libvirt.org; helo=lists.libvirt.org; Authentication-Results: mx.zohomail.com; dkim=fail; spf=pass (zohomail.com: domain of lists.libvirt.org designates 8.43.85.245 as permitted sender) smtp.mailfrom=devel-bounces@lists.libvirt.org; dmarc=fail(p=none dis=none) header.from=redhat.com Return-Path: Received: from lists.libvirt.org (lists.libvirt.org [8.43.85.245]) by mx.zohomail.com with SMTPS id 1723051329627750.7023580110387; Wed, 7 Aug 2024 10:22:09 -0700 (PDT) Received: by lists.libvirt.org (Postfix, from userid 996) id 76178828; Wed, 7 Aug 2024 13:22:08 -0400 (EDT) Received: from lists.libvirt.org (localhost [IPv6:::1]) by lists.libvirt.org (Postfix) with ESMTP id 166331324; Wed, 7 Aug 2024 13:16:24 -0400 (EDT) Received: by lists.libvirt.org (Postfix, from userid 996) id 5725BE1E; Wed, 7 Aug 2024 13:16:12 -0400 (EDT) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.libvirt.org (Postfix) with ESMTPS id 5269DC92 for ; Wed, 7 Aug 2024 13:16:11 -0400 (EDT) Received: from mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-365-MkK3L2mNPbaXm0cDF4itZg-1; Wed, 07 Aug 2024 13:16:08 -0400 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-03.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 3A2181955F38 for ; Wed, 7 Aug 2024 17:16:08 +0000 (UTC) Received: from vhost3.router.laine.org (unknown [10.22.32.31]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id BABB319560A3 for ; Wed, 7 Aug 2024 17:16:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on lists.libvirt.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=5.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, RCVD_IN_MSPIKE_H3,RCVD_IN_MSPIKE_WL,SPF_HELO_NONE autolearn=unavailable autolearn_force=no version=3.4.4 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723050971; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OiqXoWCm0Lvs1z6CoyPHVF2P+mMYmj9e7xF/OCDr2v8=; b=ATozukfXRUSGPfTb2iI0QxBaa8/2OB/LZZtEkNkMLsgSXqd4XLoKdOGndvEXFKvTNOTGTQ doAfDeZD7RPidR0sL5TuSpzsjl3p5fRlzcSGkP+IPQBZdZXpSmp1tNRWmWA2NWeEJIE7jy Sazbr4scCi0/8TZyWjJe+x3nl7KKyIg= X-MC-Unique: MkK3L2mNPbaXm0cDF4itZg-1 From: Laine Stump To: devel@lists.libvirt.org Subject: [PATCH 5/7] network: NetworkManager script to monitor/resolve conflicts with new interfaces Date: Wed, 7 Aug 2024 13:16:01 -0400 Message-ID: <20240807171603.218784-6-laine@redhat.com> In-Reply-To: <20240807171603.218784-1-laine@redhat.com> References: <20240807171603.218784-1-laine@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: quoted-printable Message-ID-Hash: I36WSLKO3SNG457IM5DTQKD6CBD4CCGI X-Message-ID-Hash: I36WSLKO3SNG457IM5DTQKD6CBD4CCGI X-MailFrom: laine@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; header-match-config-1; header-match-config-2; header-match-config-3; header-match-devel.lists.libvirt.org-0; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; suspicious-header X-Mailman-Version: 3.2.2 Precedence: list List-Id: Development discussions about the libvirt library & tools Archived-At: List-Archive: List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-ZohoMail-DKIM: fail (Header signature does not verify) X-ZM-MESSAGEID: 1723051331841116600 Content-Type: text/plain; charset="utf-8"; x-default="true" There has been a problem for several years with libvirt's default virtual network conflicting with the host physical network connection on new installs, particularly when the "host" is actually a virtual machine that is, itself, connected to the libvirt default network on its respective host. If the two default networks use the same subnet, and if the nested host's libvirt happens to start its network before the system networking connects to the L0 host, then network connectivity to the L1 guest is just silently non-working. We've tried several things over the years to eliminate this problem, including: 1) Checking for conflicting routes/interfaces during installation of the libvirt-daemon-config-network package (the one containing the default network config file) which tries different subnets until it finds one that doesn't conflict. This helps in some cases, but fails on 2 points: a) if the installation environment is different from the environment where the system is actually run (e.g. a "live CD" image of a distro, which is built in a container by the distro maintainers, then could later run in any number of places, and b) it modifies the installed files during the rpm installation %post script, which is now discouraged because people building container images don't like dealing with that. 2) When starting a libvirt network, we now check for any route or interface that conflicts with the new network's IP's and routes. This doesn't fix the problem, but does notify the user of the problem *as long as libvirt starts its networks after the host has started its system networks*. 3) New code (in the commits immediately previous to this one) add support for an "autoaddr" attribute in each virtual network element; when autoaddr is set, the network driver goes one step beyond (2) and actually finds an unused subnet and sets the new virtual network's addresses accordingly. These are all nice in their own ways, but none of them helps in the situation where libvirt's networks are started first (before the host's own network connections are all up). This led to this patch, which does the following: 4) Using a NetworkManager facility (dispatcher.d pscripts, which are run whenever any interface is brought up or down), check for any libvirt networks that conflict with a newly started NetworkManager interface, and if a conflict is found then log a message and destroy the libvirt network. Most usefully, though, if this destroyed network has autoaddr=3D'yes' then the script will immediately restart the network, which will find a new, unused subnet. Once this is in place, the only issues are: 1) It only works with NetworkManager. But of course almost all of the cases where this problem has been an issue, networking is managed by NetworkManager. 2) If there are guests already running and connected to the network, they will be disconnected, and won't be reconnected until libvirtd/virtqemud is restarted (one of the things the QEMU driver does when rereading the status of active guests is to make sure all their interfaces are connected to their respective networks). Signed-off-by: Laine Stump --- libvirt.spec.in | 2 + src/network/meson.build | 6 + src/network/nm-dispatcher-check-nets.py | 196 ++++++++++++++++++++++++ 3 files changed, 204 insertions(+) create mode 100755 src/network/nm-dispatcher-check-nets.py diff --git a/libvirt.spec.in b/libvirt.spec.in index 29101e74fe..51cecfa598 100644 --- a/libvirt.spec.in +++ b/libvirt.spec.in @@ -603,6 +603,7 @@ Network filter configuration files for cleaning guest t= raffic Summary: Network driver plugin for the libvirtd daemon Requires: libvirt-daemon-common =3D %{version}-%{release} Requires: libvirt-libs =3D %{version}-%{release} +Requires: python3-libxml2 Requires: dnsmasq >=3D 2.41 %if %{prefer_nftables} Requires: nftables @@ -2151,6 +2152,7 @@ exit 0 %dir %attr(0755, root, root) %{_localstatedir}/lib/libvirt/dnsmasq/ %attr(0755, root, root) %{_libexecdir}/libvirt_leaseshelper %{_libdir}/libvirt/connection-driver/libvirt_driver_network.so +%{_prefix}/lib/NetworkManager/dispatcher.d/50-libvirt-check-nets %{_mandir}/man8/virtnetworkd.8* %if %{with_firewalld_zone} %{_prefix}/lib/firewalld/zones/libvirt.xml diff --git a/src/network/meson.build b/src/network/meson.build index 8faff6eb1c..f620407759 100644 --- a/src/network/meson.build +++ b/src/network/meson.build @@ -169,4 +169,10 @@ if conf.has('WITH_NETWORK') rename: [ 'libvirt-routed-in.xml' ], ) endif + + install_data( + 'nm-dispatcher-check-nets.py', + install_dir: prefix / 'lib' / 'NetworkManager' / 'dispatcher.d', + rename: [ '50-libvirt-check-nets' ], + ) endif diff --git a/src/network/nm-dispatcher-check-nets.py b/src/network/nm-dispa= tcher-check-nets.py new file mode 100755 index 0000000000..454c434c88 --- /dev/null +++ b/src/network/nm-dispatcher-check-nets.py @@ -0,0 +1,196 @@ +#!/usr/bin/env python3 +# +# Copyright (C) 2012-2019 Red Hat, Inc. +# +# This library is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2.1 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see +# . + +import libvirt +import sys +import os +import libxml2 +from ipaddress import ip_network + +# This script should be installed in +# /usr/lib/NetworkManager/dispatcher.d/50-libvirt-check-nets. It will be +# called by NetworkManager every time a network interface is taken up +# or down. When a new network comes up, it checks the libvirt virtual +# networks to see if their IP address(es) (including any static +# routes) are in conflict with the IP address(es) (or static routes) +# of the newly added interface. If so, the libvirt network is +# disabled. It is assumed that the user will notice that their guests +# no longer have network connectvity (and/or the message logged by +# this script), see that the network has been disabled, and then +# realize the conflict when they try to restart it. +# +# set checkDefaultOnly=3DFalse to check *all* active libvirt networks +# for conflicts with the new interface. Set to True to check only the +# libvirt default network (since most networks other than the default +# network are added post-install at a time when all of the hosts other +# networks are already active, it may be overkill to check all of the +# libvirt networks for conflict here (and instead just add more +# needless overheard to bringing up a new host interface). +# +checkDefaultOnly =3D False + +# NB: since this file is installed in /usr/lib, it really shouldn't be +# modified by the user, but instead should be copied to +# /etc/NetworkManager/dispatcher.d, where it will override the copy in +# /usr/lib. Even that isn't a proper solution though - if we're going +# to actually have this config knob, perhaps we should check for it in +# the environment, and if someone wants to modify it they can put a +# short script in /etc that exports and environment variable and then +# calls this script? Just thinking out loud here. + + +def checkconflict(conn, netname, hostnets, hostif): + + # ignore if the network has been brought down or removed since we + # got the list + try: + net =3D conn.networkLookupByName(netname) + except libvirt.libvirtError: + return + + if not net.isActive(): + return + + xml =3D net.XMLDesc() + doc =3D libxml2.parseDoc(xml) + ctx =3D doc.xpathNewContext() + + # see if NetworkManager is informing us that this libvirt network + # itself is coming online + bridge =3D ctx.xpathEval("/network/bridge/@name") + if bridge and bridge[0].content =3D=3D hostif: + return + + # check *all* the addresses of this network + addrs =3D ctx.xpathEval("/network/*[@address]") + for ip in addrs: + ctx.setContextNode(ip) + address =3D ctx.xpathEval("@address") + prefix =3D ctx.xpathEval("@prefix") + netmask =3D ctx.xpathEval("@netmask") + autoaddr =3D ctx.xpathEval("@autoaddr") + + isAutoaddr =3D False + if autoaddr and len(autoaddr[0].content): + isAutoaddr =3D (autoaddr[0].content =3D=3D "yes") + + if not (address and len(address[0].content)): + continue + + addrstr =3D address[0].content + if not (prefix and len(prefix[0].content)): + # check for a netmask + if not (netmask and len(netmask[0].content)): + # this element has address, but no prefix or netmask + # probably it is so we can ignore it + continue + # convert netmask to prefix + prefixstr =3D str(ip_network("0.0.0.0/%s" % netmask[0].content= ).prefixlen) + else: + prefixstr =3D prefix[0].content + + virtnetaddress =3D ip_network("%s/%s" % (addrstr, prefixstr), stri= ct=3DFalse) + + for hostnet in hostnets: + if virtnetaddress =3D=3D hostnet: + # There is a conflict with this libvirt network and the sp= ecified + # net, so we need to disable the libvirt network + print("Stopping libvirt network '%s' because its subnet %s= conflicts with newly started interface '%s'')" + % (netname, str(hostnet), hostif)) + try: + net.destroy() + except libvirt.libvirtError: + print("Failed to destroy network %s" % netname) + return + + if isAutoaddr: + print("Restarting autoaddr libvirt network '%s'with ne= w subnet" % (netname)) + try: + net.create() + except libvirt.libvirtError: + print("Failed to restart network '%s'" % netname) + return + return + + +def addHostNets(hostnets, countenv, addrenv): + + count =3D os.getenv(countenv) + if not count or count =3D=3D 0: + return + + for num in range(int(count)): + addrstr =3D os.getenv("%s_%d" % (addrenv, num)) + if not addrstr or addrstr =3D=3D "": + continue + + net =3D ip_network(addrstr.split()[0], strict=3DFalse) + if net: + hostnets.append(net) + return + + +############################################################ + +if sys.argv[2] !=3D "up": + sys.exit(0) + +hostif =3D sys.argv[1] + +try: + conn =3D libvirt.open(None) +except libvirt.libvirtError: + print('Failed to open connection to the hypervisor') + sys.exit(0) + +if checkDefaultOnly: + nets =3D [] + net =3D conn.networkLookupByName("default") + if not (net and net.isActive()): + sys.exit(0) + nets.append(net) +else: + nets =3D conn.listAllNetworks(libvirt.VIR_CONNECT_LIST_NETWORKS_ACTIVE) + if not nets: + sys.exit(0) + +# We have at least one active network. Build a list of all network +# routes added by the new interface, and compare that list to the list +# of all networks used by each active libvirt network. If any are an +# exact match, then we have a conflict and need to shut down the +# libvirt network to avoid killing host networking. + +# When NetworkManager calls scripts in /etc/NetworkManager/dispatcher.d +# it will have all IP addresses and routes associated with the interface +# that is going up or down in the following environment variables: +# +# IP4_NUM_ADDRESSES - number of IPv4 addresses +# IP4_ADDRESS_N - one variable for each address, starting at _0 +# IP4_NUM_ROUTES - number of IPv5 routes +# IP4_ROUTE_N - one for each route, starting at _0 +# (replace "IP4" with "IP6" and repeat) +# +hostnets =3D [] +addHostNets(hostnets, "IP4_NUM_ADDRESSES", "IP4_ADDRESS") +addHostNets(hostnets, "IP4_NUM_ROUTES", "IP4_ROUTE") +addHostNets(hostnets, "IP6_NUM_ADDRESSES", "IP6_ADDRESS") +addHostNets(hostnets, "IP6_NUM_ROUTES", "IP6_ROUTE") + +for net in nets: + + checkconflict(conn, net.name(), hostnets, hostif) --=20 2.45.2