There have been some reports that, due to libvirt always trying to
assign the lowest numbered macvtap / tap device name possible, a new
guest would sometimes be started using the same tap device name as
previously used by another guest that is in the process of being destroyed *as the new guest is starting.
In some cases this has led to, for example, the old guest's
qemuProcessStop() code deleting a port from an OVS switch that had
just been re-added by the new guest (because the port name is based on
only the device name using the port). Similar problems can happen (and
I believe have) with nwfilter rules and bandwidth rules (which are
both instantiated based on the name of the tap device).
A couple patches have been previously proposed to change the ordering
of startup and shutdown processing, or to put a mutex around
everything related to the tap/macvtap device name usage, but in the
end no matter what you do there will still be possible holes, because
the device could be deleted outside libvirt's control (for example,
regular tap devices are automatically deleted when the qemu process
terminates, and that isn't always initiated by libvirt but could
instead happen completely asynchronously - libvirt then has no control
over the ordering of shutdown operations, and no opportunity to
protect it with a mutex.)
But this only happens if a new device is created at the same time as
one is being deleted. We can effectively eliminate the chance of this
happening if we end the practice of always looking for the lowest
numbered available device name, and instead just keep an integer that
is incremented each time we need a new device name. At some point it
will need to wrap back around to 0 (in order to avoid the IFNAMSIZ 15
character limit if nothing else), and we can't guarantee that the new
name really will be the *least* recently used name, but "math"
suggests that it will be *much* less common that we'll try to re-use
the *most* recently used name.
This patch implements such a counter for macvtap/macvlan only. It does
it on top of the existing "ID reservation" system (I'm thinking about
making a followup that gets rid of the bitmap, as it's now overkill
and just serves to make the code more confusing).
Signed-off-by: Laine Stump <laine@redhat.com>
---
src/util/virnetdevmacvlan.c | 67 +++++++++++++++++++++++++++----------
1 file changed, 50 insertions(+), 17 deletions(-)
diff --git a/src/util/virnetdevmacvlan.c b/src/util/virnetdevmacvlan.c
index 69a9c784bb..c086aa3eb0 100644
--- a/src/util/virnetdevmacvlan.c
+++ b/src/util/virnetdevmacvlan.c
@@ -74,6 +74,8 @@ VIR_LOG_INIT("util.netdevmacvlan");
virMutex virNetDevMacVLanCreateMutex = VIR_MUTEX_INITIALIZER;
virBitmapPtr macvtapIDs = NULL;
virBitmapPtr macvlanIDs = NULL;
+static int macvtapLastID = -1;
+static int macvlanLastID = -1;
static int
virNetDevMacVLanOnceInit(void)
@@ -108,12 +110,18 @@ virNetDevMacVLanReserveID(int id, unsigned int flags,
bool quietFail, bool nextFree)
{
virBitmapPtr bitmap;
+ int *lastID;
if (virNetDevMacVLanInitialize() < 0)
return -1;
- bitmap = (flags & VIR_NETDEV_MACVLAN_CREATE_WITH_TAP) ?
- macvtapIDs : macvlanIDs;
+ if (flags & VIR_NETDEV_MACVLAN_CREATE_WITH_TAP) {
+ bitmap = macvtapIDs;
+ lastID = &macvtapLastID;
+ } else {
+ bitmap = macvlanIDs;
+ lastID = &macvlanLastID;
+ }
if (id > MACVLAN_MAX_ID) {
virReportError(VIR_ERR_INTERNAL_ERROR,
@@ -122,24 +130,49 @@ virNetDevMacVLanReserveID(int id, unsigned int flags,
return -1;
}
- if ((id < 0 || nextFree) &&
- (id = virBitmapNextClearBit(bitmap, id)) < 0) {
- virReportError(VIR_ERR_INTERNAL_ERROR,
- _("no unused %s names available"),
- VIR_NET_GENERATED_PREFIX);
- return -1;
- }
+ if (id < 0 || nextFree) {
+ /* starting with *lastID + 1, do a loop looking for an unused
+ * device name, wrapping around at MACVLAN_MAX_ID.
+ */
+ int start = (++(*lastID)) % (MACVLAN_MAX_ID + 1);
+ bool found = false;
- if (virBitmapIsBitSet(bitmap, id)) {
- if (quietFail) {
- VIR_INFO("couldn't reserve name %s%d - already in use",
- VIR_NET_GENERATED_PREFIX, id);
- } else {
+ for (id = start;
+ id + 1 != start;
+ id = (++(*lastID)) % (MACVLAN_MAX_ID + 1)) {
+
+ if (!virBitmapIsBitSet(bitmap, id)) {
+ found = true;
+ break;
+ }
+ }
+ if (!found) {
virReportError(VIR_ERR_INTERNAL_ERROR,
- _("couldn't reserve name %s%d - already in use"),
- VIR_NET_GENERATED_PREFIX, id);
+ _("no unused %s names available"),
+ VIR_NET_GENERATED_PREFIX);
+ return -1;
}
- return -1;
+ } else {
+ /* A specific ID was requested, we just fail if that
+ * ID isn't available
+ */
+ if (virBitmapIsBitSet(bitmap, id)) {
+ if (quietFail) {
+ VIR_INFO("couldn't reserve name %s%d - already in use",
+ VIR_NET_GENERATED_PREFIX, id);
+ } else {
+ virReportError(VIR_ERR_INTERNAL_ERROR,
+ _("couldn't reserve name %s%d - already in use"),
+ VIR_NET_GENERATED_PREFIX, id);
+ }
+ return -1;
+ }
+ /* adjust lastID to not look below this ID (even though
+ * eventually we will wrap around and look below it - this is
+ * just a delay tactic
+ */
+ if (*lastID % (MACVLAN_MAX_ID + 1) < id)
+ *lastID = id;
}
if (virBitmapSetBit(bitmap, id) < 0) {
--
2.26.2
On 8/24/20 6:23 AM, Laine Stump wrote: > There have been some reports that, due to libvirt always trying to > assign the lowest numbered macvtap / tap device name possible, a new > guest would sometimes be started using the same tap device name as > previously used by another guest that is in the process of being destroyed *as the new guest is starting. Realign please :-) > > In some cases this has led to, for example, the old guest's > qemuProcessStop() code deleting a port from an OVS switch that had > just been re-added by the new guest (because the port name is based on > only the device name using the port). Similar problems can happen (and > I believe have) with nwfilter rules and bandwidth rules (which are > both instantiated based on the name of the tap device). > > A couple patches have been previously proposed to change the ordering > of startup and shutdown processing, or to put a mutex around > everything related to the tap/macvtap device name usage, but in the > end no matter what you do there will still be possible holes, because > the device could be deleted outside libvirt's control (for example, > regular tap devices are automatically deleted when the qemu process > terminates, and that isn't always initiated by libvirt but could > instead happen completely asynchronously - libvirt then has no control > over the ordering of shutdown operations, and no opportunity to > protect it with a mutex.) > > But this only happens if a new device is created at the same time as > one is being deleted. We can effectively eliminate the chance of this > happening if we end the practice of always looking for the lowest > numbered available device name, and instead just keep an integer that > is incremented each time we need a new device name. At some point it > will need to wrap back around to 0 (in order to avoid the IFNAMSIZ 15 > character limit if nothing else), and we can't guarantee that the new > name really will be the *least* recently used name, but "math" > suggests that it will be *much* less common that we'll try to re-use > the *most* recently used name. > > This patch implements such a counter for macvtap/macvlan only. It does > it on top of the existing "ID reservation" system (I'm thinking about > making a followup that gets rid of the bitmap, as it's now overkill > and just serves to make the code more confusing). > > Signed-off-by: Laine Stump <laine@redhat.com> > --- > src/util/virnetdevmacvlan.c | 67 +++++++++++++++++++++++++++---------- > 1 file changed, 50 insertions(+), 17 deletions(-) Michal
© 2016 - 2024 Red Hat, Inc.