From nobody Sat Sep 26 22:16:24 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1788376915; cv=none; d=zohomail.com; s=zohoarc; b=TmbqyQiHZ6D/xB5e42WTIMlqEwzvIETNXKXxrHV4uUC3/3MTydVKlyY0qiewk9HUFqHVtXL+G8cJ0UF9gRImNcTUSnO4vGTcfWQWy+oJ09nLYo4yTcSacbSQZTl6yLQpFHLQvNxF90dHe4zFDkT+Ck5wVaIsujcCtb1gcQwDzUk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1788376915; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=1xuq+U6rfrk8iRaF04vxMhoo21C+Y+dWpxjx6CV/2Bk=; b=DAANCoP59hpgoGfY1l4iGOwZlJvtjgwJccjTTh3CeAZA2MbuxllOVuMHVhB1VwuGzMyTcmlMyrZunWo/m8ZkjpJUw982ysMU/w4du3oiQdnPf0z8ug7v4vztl1PAYkUDsG+f6mE0By/TPAh0Az5qVtPRUD7xWx6imevmpN6xPMY= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 17883769155451022.5569428683295; Wed, 2 Sep 2026 12:21:55 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x1qWH-0004Hx-HA; Wed, 02 Sep 2026 15:21:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x1qWE-0004HQ-KC for qemu-devel@nongnu.org; Wed, 02 Sep 2026 15:21:10 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x1qWC-00025W-Gk for qemu-devel@nongnu.org; Wed, 02 Sep 2026 15:21:10 -0400 Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-696-pK2BIEWGPZm_V0j_FWA76A-1; Wed, 02 Sep 2026 15:21:05 -0400 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id E4158197753D; Wed, 2 Sep 2026 19:21:03 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.32.5]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 6564F3000DA3; Wed, 2 Sep 2026 19:21:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1788376867; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1xuq+U6rfrk8iRaF04vxMhoo21C+Y+dWpxjx6CV/2Bk=; b=BhsYdppGCyrWD5OaThkb1IfSL1zLMTOGr8M1EzW89tXCooxCY+74BkbGuvATSYvpqsthcJ QY3sKMED3OVfPJoMuMTYTsgoZOfVYD+azyb+5rhwuMMNUSEDeVXxmBtm3fQ6ReJfxkR6NZ wV90rSyGyB/LW5gc1CdyaYfFRl88bYE= X-MC-Unique: pK2BIEWGPZm_V0j_FWA76A-1 X-Mimecast-MFC-AGG-ID: pK2BIEWGPZm_V0j_FWA76A_1788376864 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Akihiko Odaki , Sriram Yagnaraman , Jason Wang , Alex Williamson , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [RFC PATCH v2 1/9] igb: Add x-vf-migration property and DVSEC extended capability Date: Wed, 2 Sep 2026 21:20:46 +0200 Message-ID: <20260902192054.3329753-2-clg@redhat.com> In-Reply-To: <20260902192054.3329753-1-clg@redhat.com> References: <20260902192054.3329753-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: 12 X-Spam_score: 1.2 X-Spam_bar: + X-Spam_report: (1.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_SBL_CSS=3.335, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1788376919461154100 Add an "x-vf-migration" property to the IGB PF device and expose a DVSEC (Designated Vendor-Specific Extended Capability) at offset 0x160 in VF extended config space when migration is enabled. The DVSEC provides the register interface for VF live migration: - CAPS: supported features (state migration) - CTRL: command doorbell - STATUS: state and error reporting - BUF_ADDR: shared DMA buffer address (GPA) Move IgbVfState from igbvf.c to igb_common.h so it can be shared with the migration module, and add the migration state field. AI-used-for: code (prototype) Signed-off-by: C=C3=A9dric Le Goater --- MAINTAINERS | 5 ++ hw/net/igb_common.h | 16 +++++++ hw/net/igb_migration.h | 72 ++++++++++++++++++++++++++++ hw/net/igb.c | 2 + hw/net/igb_migration.c | 106 +++++++++++++++++++++++++++++++++++++++++ hw/net/igbvf.c | 52 ++++++++++++++++---- hw/net/meson.build | 2 +- 7 files changed, 245 insertions(+), 10 deletions(-) create mode 100644 hw/net/igb_migration.h create mode 100644 hw/net/igb_migration.c diff --git a/MAINTAINERS b/MAINTAINERS index 4a49a40294eb..f88b526be238 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2816,6 +2816,11 @@ F: tests/functional/x86_64/test_netdev_ethtool.py F: tests/qtest/igb-test.c F: tests/qtest/libqos/igb.c =20 +igb VF migration +M: C=C3=A9dric Le Goater +S: Maintained +F: hw/net/igb_migration.* + eepro100 M: Stefan Weil S: Maintained diff --git a/hw/net/igb_common.h b/hw/net/igb_common.h index b316a5bcfa5c..f0e2529e757b 100644 --- a/hw/net/igb_common.h +++ b/hw/net/igb_common.h @@ -26,7 +26,9 @@ #ifndef HW_NET_IGB_COMMON_H #define HW_NET_IGB_COMMON_H =20 +#include "hw/pci/pci_device.h" #include "igb_regs.h" +#include "igb_migration.h" =20 #define TYPE_IGBVF "igbvf" =20 @@ -154,4 +156,18 @@ uint64_t igb_mmio_read(void *opaque, hwaddr addr, unsi= gned size); void igb_mmio_write(void *opaque, hwaddr addr, uint64_t val, unsigned size= ); void igb_vf_reset(void *opaque, uint16_t vfn); =20 +OBJECT_DECLARE_SIMPLE_TYPE(IgbVfState, IGBVF) + +struct IgbVfState { + PCIDevice parent_obj; + + uint16_t vfn; + bool migration_enabled; + + MemoryRegion mmio; + MemoryRegion msix; + + IgbVfMigState mig; +}; + #endif diff --git a/hw/net/igb_migration.h b/hw/net/igb_migration.h new file mode 100644 index 000000000000..3da28e11e49e --- /dev/null +++ b/hw/net/igb_migration.h @@ -0,0 +1,72 @@ +/* + * QEMU Intel 82576 SR/IOV VF Migration Support + * + * Copyright (c) 2026 Red Hat, Inc. + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#ifndef HW_NET_IGB_MIGRATION_H +#define HW_NET_IGB_MIGRATION_H + +#include "hw/pci/pci_device.h" + +/* + * Migration interface exposed as a DVSEC (Designated Vendor-Specific + * Extended Capability) in VF extended config space. + * + * DVSEC layout at IGB_MIG_DVSEC_OFFSET (0x160): + * + * +0x00 PCIe extended cap header (cap_id=3D0x23, ver=3D1, next) + * +0x04 DVSEC header 1 (len | rev | vendor_id) + * +0x08 DVSEC header 2 (DVSEC ID) + * +0x0A Reserved (padding for DWORD alignment) + * +0x0C CAPS (RO: F_STATE[0]) + * +0x10 CTRL (WO: doorbell command) + * +0x14 STATUS (RO: state[7:0], error_code[15:8]) + * +0x18 BUF_ADDR_LO (RW: shared buffer GPA low) + * +0x1C BUF_ADDR_HI (RW: shared buffer GPA high) + */ + +#define IGB_MIG_DVSEC_OFFSET 0x160 +#define IGB_MIG_DVSEC_SIZE 0x20 +#define IGB_MIG_DVSEC_VER 1 +#define IGB_MIG_DVSEC_ID 1 + +/* Register offsets relative to DVSEC base */ +#define IGB_MIG_CAPS 0x0C +#define IGB_MIG_CTRL 0x10 +#define IGB_MIG_STATUS 0x14 +#define IGB_MIG_BUF_ADDR_LO 0x18 +#define IGB_MIG_BUF_ADDR_HI 0x1C + +/* CAPS register layout */ +#define IGB_MIG_CAP_F_STATE (1u << 0) + +/* STATUS register: state in [7:0], error code [15:8] */ +#define IGB_MIG_STATUS_STATE_MASK 0xFF +#define IGB_MIG_STATUS_ERROR_CODE_SHIFT 8 +#define IGB_MIG_STATUS_ERR(code) \ + ((uint32_t)(code) << IGB_MIG_STATUS_ERROR_CODE_SHIFT) + +/* Device states (based on VFIO migration v2) */ +#define IGB_MIG_STATE_ERROR 0 +#define IGB_MIG_STATE_STOP 1 +#define IGB_MIG_STATE_RUNNING 2 +#define IGB_MIG_STATE_STOP_COPY 3 +#define IGB_MIG_STATE_RESUMING 4 + +typedef struct IgbVfMigState { + uint32_t mig_state; + uint64_t mig_data_buf_addr; +} IgbVfMigState; + +typedef struct IgbVfState IgbVfState; + +bool igbvf_add_migration_dvsec(PCIDevice *dev, Error **errp); +void igbvf_mig_state_reset(IgbVfState *s); +uint32_t igbvf_mig_config_read(IgbVfState *s, uint32_t addr, int size); +bool igbvf_mig_config_write(IgbVfState *s, uint32_t addr, uint32_t val, + int size); + +#endif diff --git a/hw/net/igb.c b/hw/net/igb.c index c076807e7110..7268e5473fc3 100644 --- a/hw/net/igb.c +++ b/hw/net/igb.c @@ -79,6 +79,7 @@ struct IGBState { =20 IGBCore core; bool has_flr; + bool vf_migration; }; =20 #define IGB_CAP_SRIOV_OFFSET (0x160) @@ -597,6 +598,7 @@ static const VMStateDescription igb_vmstate =3D { static const Property igb_properties[] =3D { DEFINE_NIC_PROPERTIES(IGBState, conf), DEFINE_PROP_BOOL("x-pcie-flr-init", IGBState, has_flr, true), + DEFINE_PROP_BOOL("x-vf-migration", IGBState, vf_migration, false), }; =20 static void igb_class_init(ObjectClass *class, const void *data) diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c new file mode 100644 index 000000000000..4dfebd82344c --- /dev/null +++ b/hw/net/igb_migration.c @@ -0,0 +1,106 @@ +/* + * QEMU Intel 82576 SR/IOV VF Migration Support + * + * Copyright (c) 2026 Red Hat, Inc. + * + * SPDX-License-Identifier: GPL-2.0-or-later + */ + +#include "qemu/osdep.h" +#include "hw/pci/pci_device.h" +#include "hw/pci/pcie.h" +#include "igb_common.h" +#include "igb_migration.h" + +static void igbvf_mig_update_status(IgbVfState *s, uint8_t err) +{ + IgbVfMigState *ms =3D &s->mig; + PCIDevice *dev =3D PCI_DEVICE(s); + uint32_t status; + + status =3D ms->mig_state & IGB_MIG_STATUS_STATE_MASK; + + if (err) { + status =3D IGB_MIG_STATE_ERROR | IGB_MIG_STATUS_ERR(err); + } + + pci_set_long(dev->config + IGB_MIG_DVSEC_OFFSET + IGB_MIG_STATUS, stat= us); +} + + +bool igbvf_add_migration_dvsec(PCIDevice *dev, Error **errp) +{ + uint16_t offset =3D IGB_MIG_DVSEC_OFFSET; + uint32_t caps; + + pcie_add_capability(dev, PCI_EXT_CAP_ID_DVSEC, 1, offset, + IGB_MIG_DVSEC_SIZE); + + /* DVSEC header 1: length[31:20] | rev[19:16] | vendor_id[15:0] */ + pci_set_long(dev->config + offset + 0x4, + (IGB_MIG_DVSEC_SIZE << 20) | + (IGB_MIG_DVSEC_VER << 16) | + PCI_VENDOR_ID_INTEL); + + /* DVSEC header 2: DVSEC ID */ + pci_set_word(dev->config + offset + 0x8, IGB_MIG_DVSEC_ID); + + /* CAPS: features (state migration only) */ + caps =3D IGB_MIG_CAP_F_STATE; + pci_set_long(dev->config + offset + IGB_MIG_CAPS, caps); + + /* STATUS: initial state is RUNNING */ + pci_set_long(dev->config + offset + IGB_MIG_STATUS, + IGB_MIG_STATE_RUNNING); + + /* BUF_ADDR_LO and BUF_ADDR_HI are writable */ + memset(dev->wmask + offset + IGB_MIG_BUF_ADDR_LO, 0xff, 4); + memset(dev->wmask + offset + IGB_MIG_BUF_ADDR_HI, 0xff, 4); + + return true; +} + +uint32_t igbvf_mig_config_read(IgbVfState *s, uint32_t addr, int size) +{ + PCIDevice *dev =3D PCI_DEVICE(s); + + return pci_default_read_config(dev, addr, size); +} + +bool igbvf_mig_config_write(IgbVfState *s, uint32_t addr, uint32_t val, + int size) +{ + PCIDevice *dev =3D PCI_DEVICE(s); + uint32_t offset =3D addr - IGB_MIG_DVSEC_OFFSET; + + switch (offset) { + case IGB_MIG_CTRL: + /* Command handling will be added in a later commit */ + break; + + case IGB_MIG_BUF_ADDR_LO: + case IGB_MIG_BUF_ADDR_HI: + pci_default_write_config(dev, addr, val, size); + break; + + default: + break; + } + + return true; +} + +void igbvf_mig_state_reset(IgbVfState *s) +{ + IgbVfMigState *ms =3D &s->mig; + + ms->mig_state =3D IGB_MIG_STATE_RUNNING; + ms->mig_data_buf_addr =3D 0; + + pci_set_long(PCI_DEVICE(s)->config + + IGB_MIG_DVSEC_OFFSET + IGB_MIG_BUF_ADDR_LO, 0); + pci_set_long(PCI_DEVICE(s)->config + + IGB_MIG_DVSEC_OFFSET + IGB_MIG_BUF_ADDR_HI, 0); + + igbvf_mig_update_status(s, 0); +} diff --git a/hw/net/igbvf.c b/hw/net/igbvf.c index 9a165c7063ee..30dfdb574ac7 100644 --- a/hw/net/igbvf.c +++ b/hw/net/igbvf.c @@ -38,27 +38,21 @@ */ =20 #include "qemu/osdep.h" +#include "qemu/range.h" #include "hw/core/hw-error.h" #include "hw/net/mii.h" #include "hw/pci/pci_device.h" #include "hw/pci/pcie.h" +#include "hw/pci/pcie_sriov.h" #include "hw/pci/msix.h" #include "net/eth.h" #include "net/net.h" #include "igb_common.h" #include "igb_core.h" +#include "igb_migration.h" #include "trace.h" #include "qapi/error.h" =20 -OBJECT_DECLARE_SIMPLE_TYPE(IgbVfState, IGBVF) - -struct IgbVfState { - PCIDevice parent_obj; - - MemoryRegion mmio; - MemoryRegion msix; -}; - static hwaddr vf_to_pf_addr(hwaddr addr, uint16_t vfn, bool write) { switch (addr) { @@ -199,10 +193,35 @@ static hwaddr vf_to_pf_addr(hwaddr addr, uint16_t vfn= , bool write) return HWADDR_MAX; } =20 +static bool igbvf_addr_in_dvsec(uint32_t addr, int len) +{ + return ranges_overlap(addr, len, + IGB_MIG_DVSEC_OFFSET, IGB_MIG_DVSEC_SIZE); +} + +static uint32_t igbvf_read_config(PCIDevice *dev, uint32_t addr, int size) +{ + IgbVfState *s =3D IGBVF(dev); + + if (s->migration_enabled && igbvf_addr_in_dvsec(addr, size)) { + return igbvf_mig_config_read(s, addr, size); + } + + return pci_default_read_config(dev, addr, size); +} + static void igbvf_write_config(PCIDevice *dev, uint32_t addr, uint32_t val, int len) { + IgbVfState *s =3D IGBVF(dev); + trace_igbvf_write_config(addr, val, len); + + if (s->migration_enabled && igbvf_addr_in_dvsec(addr, len)) { + igbvf_mig_config_write(s, addr, val, len); + return; + } + pci_default_write_config(dev, addr, val, len); if (object_property_get_bool(OBJECT(pcie_sriov_get_pf(dev)), "x-pcie-flr-init", &error_abort)) { @@ -282,13 +301,27 @@ static void igbvf_pci_realize(PCIDevice *dev, Error *= *errp) } =20 pcie_ari_init(dev, 0x150); + + if (object_property_get_bool(OBJECT(pcie_sriov_get_pf(dev)), + "x-vf-migration", &error_abort)) { + s->vfn =3D pcie_sriov_vf_number(dev); + s->migration_enabled =3D true; + if (!igbvf_add_migration_dvsec(dev, errp)) { + return; + } + } } =20 static void igbvf_qdev_reset_hold(Object *obj, ResetType type) { PCIDevice *vf =3D PCI_DEVICE(obj); + IgbVfState *s =3D IGBVF(vf); =20 igb_vf_reset(pcie_sriov_get_pf(vf), pcie_sriov_vf_number(vf)); + + if (s->migration_enabled) { + igbvf_mig_state_reset(s); + } } =20 static void igbvf_pci_uninit(PCIDevice *dev) @@ -309,6 +342,7 @@ static void igbvf_class_init(ObjectClass *class, const = void *data) =20 c->realize =3D igbvf_pci_realize; c->exit =3D igbvf_pci_uninit; + c->config_read =3D igbvf_read_config; c->vendor_id =3D PCI_VENDOR_ID_INTEL; c->device_id =3D E1000_DEV_ID_82576_VF; c->revision =3D 1; diff --git a/hw/net/meson.build b/hw/net/meson.build index 84f142df222a..bb4b449b25ba 100644 --- a/hw/net/meson.build +++ b/hw/net/meson.build @@ -11,7 +11,7 @@ system_ss.add(when: 'CONFIG_E1000_PCI', if_true: files('e= 1000.c', 'e1000x_common system_ss.add(when: 'CONFIG_E1000E_PCI_EXPRESS', if_true: files('net_tx_pk= t.c', 'net_rx_pkt.c')) system_ss.add(when: 'CONFIG_E1000E_PCI_EXPRESS', if_true: files('e1000e.c'= , 'e1000e_core.c', 'e1000x_common.c')) system_ss.add(when: 'CONFIG_IGB_PCI_EXPRESS', if_true: files('net_tx_pkt.c= ', 'net_rx_pkt.c')) -system_ss.add(when: 'CONFIG_IGB_PCI_EXPRESS', if_true: files('igb.c', 'igb= vf.c', 'igb_core.c')) +system_ss.add(when: 'CONFIG_IGB_PCI_EXPRESS', if_true: files('igb.c', 'igb= vf.c', 'igb_core.c', 'igb_migration.c')) system_ss.add(when: 'CONFIG_RTL8139_PCI', if_true: files('rtl8139.c')) system_ss.add(when: 'CONFIG_TULIP', if_true: files('tulip.c')) system_ss.add(when: 'CONFIG_VMXNET3_PCI', if_true: files('net_tx_pkt.c', '= net_rx_pkt.c')) --=20 2.55.0 From nobody Sat Sep 26 22:16:24 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1788376953; cv=none; d=zohomail.com; s=zohoarc; b=Ux8zaK71IWRcpCadN7Ak7iJMjzc+VFllzXBGezqVJHiXEWHtJI1RNCzoMMV5EBscm11xb06ibSPzTf07df3vWZKYkgj6B++kNwICNj/h5H+fGpKO11OLHL/mBJZTFt8Capw6SXmBURlLNsLmrMuVToOvkXvv9CSRHHSQRlaANz4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1788376953; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=UaTye5haqYCaE8xliP1SDd1GpMqNLr5kKcfdf8dTdyo=; b=Mj93syqQNfLTG6dcg+b/zcC4gXzXGvA1PU8U7V0f/YL7NSA3QsXe5Vp6NPDkRZkm1yPE4v23q3sOOFsU5YZTW3/CpQvJxuOhwXKjiaRJq8aT7pxKqfH8+K2xDdu9Zw5Bdk6czVniXxlds+E6ZKWs9T7RgEixeiaSqDnLlbBhEYw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1788376953888817.6825578776167; Wed, 2 Sep 2026 12:22:33 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x1qWd-0004MF-1e; Wed, 02 Sep 2026 15:21:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x1qWZ-0004LO-4m for qemu-devel@nongnu.org; Wed, 02 Sep 2026 15:21:32 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x1qWW-00028u-M2 for qemu-devel@nongnu.org; Wed, 02 Sep 2026 15:21:30 -0400 Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-127-hqUZH9ljOCSqJvq1C-gjtA-1; Wed, 02 Sep 2026 15:21:11 -0400 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D317F1801310; Wed, 2 Sep 2026 19:21:06 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.32.5]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 8BCC23000DA3; Wed, 2 Sep 2026 19:21:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1788376885; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UaTye5haqYCaE8xliP1SDd1GpMqNLr5kKcfdf8dTdyo=; b=EPGk9IWw/Aa2T/nFgJdMKKy8x8xxkI4/UnEJJf+aBi1AOFOlOHthtaW/jSDw1chd+BNGYV c2kwHgFsWx90H5oawx9YD1ALhUftFE4Di2IWeyiMor2CeYdTIizNbSj1GQUnqjR9OwGR9v 8kInSbTKODfXgWFqiJ8TE5XMXT9CVIE= X-MC-Unique: hqUZH9ljOCSqJvq1C-gjtA-1 X-Mimecast-MFC-AGG-ID: hqUZH9ljOCSqJvq1C-gjtA_1788376867 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Akihiko Odaki , Sriram Yagnaraman , Jason Wang , Alex Williamson , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [RFC PATCH v2 2/9] igb: Add migration state machine via extended config space Date: Wed, 2 Sep 2026 21:20:47 +0200 Message-ID: <20260902192054.3329753-3-clg@redhat.com> In-Reply-To: <20260902192054.3329753-1-clg@redhat.com> References: <20260902192054.3329753-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1788376957133154100 Implement the VFIO migration v2 state machine and command dispatch through the CTRL doorbell register in the DVSEC capability. Add three CTRL commands: - SET_STATE: transitions follow the VFIO migration protocol RUNNING <-> STOP STOP <-> STOP_COPY STOP <-> RESUMING ERROR -> STOP - SAVE: DMA-writes the device state to a guest buffer - LOAD: DMA-reads it back Add a DATA_SIZE register at +0x20 to report the state blob size. The save/load stubs are filled in by the next patch. AI-used-for: code (prototype) Signed-off-by: C=C3=A9dric Le Goater --- hw/net/igb_migration.h | 25 ++++- hw/net/igb_migration.c | 217 ++++++++++++++++++++++++++++++++++++++++- hw/net/trace-events | 6 ++ 3 files changed, 246 insertions(+), 2 deletions(-) diff --git a/hw/net/igb_migration.h b/hw/net/igb_migration.h index 3da28e11e49e..ea40ac65c54b 100644 --- a/hw/net/igb_migration.h +++ b/hw/net/igb_migration.h @@ -26,10 +26,11 @@ * +0x14 STATUS (RO: state[7:0], error_code[15:8]) * +0x18 BUF_ADDR_LO (RW: shared buffer GPA low) * +0x1C BUF_ADDR_HI (RW: shared buffer GPA high) + * +0x20 DATA_SIZE (RO: max state blob size in bytes) */ =20 #define IGB_MIG_DVSEC_OFFSET 0x160 -#define IGB_MIG_DVSEC_SIZE 0x20 +#define IGB_MIG_DVSEC_SIZE 0x24 #define IGB_MIG_DVSEC_VER 1 #define IGB_MIG_DVSEC_ID 1 =20 @@ -39,10 +40,20 @@ #define IGB_MIG_STATUS 0x14 #define IGB_MIG_BUF_ADDR_LO 0x18 #define IGB_MIG_BUF_ADDR_HI 0x1C +#define IGB_MIG_DATA_SIZE 0x20 =20 /* CAPS register layout */ #define IGB_MIG_CAP_F_STATE (1u << 0) =20 +/* CTRL register: command in [7:0] */ +#define IGB_MIG_CTRL_CMD_MASK 0xFF +#define IGB_MIG_CTRL_ARG_SHIFT 8 + +/* CTRL commands */ +#define IGB_MIG_CMD_SET_STATE 1 +#define IGB_MIG_CMD_SAVE 2 +#define IGB_MIG_CMD_LOAD 3 + /* STATUS register: state in [7:0], error code [15:8] */ #define IGB_MIG_STATUS_STATE_MASK 0xFF #define IGB_MIG_STATUS_ERROR_CODE_SHIFT 8 @@ -56,8 +67,20 @@ #define IGB_MIG_STATE_STOP_COPY 3 #define IGB_MIG_STATE_RESUMING 4 =20 +/* Error codes */ +#define IGB_MIG_ERR_UNK_CMD 1 +#define IGB_MIG_ERR_BAD_STATE 2 +#define IGB_MIG_ERR_NO_BUFFER 3 +#define IGB_MIG_ERR_DMA_FAILED 4 +#define IGB_MIG_ERR_BAD_SIZE 5 + +/* Shared buffer constants */ +#define IGB_VF_STATE_MAX_SIZE 4096 + typedef struct IgbVfMigState { uint32_t mig_state; + uint32_t mig_data[IGB_VF_STATE_MAX_SIZE / sizeof(uint32_t)]; + uint32_t mig_data_size; uint64_t mig_data_buf_addr; } IgbVfMigState; =20 diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c index 4dfebd82344c..8e7e6fac9b9b 100644 --- a/hw/net/igb_migration.c +++ b/hw/net/igb_migration.c @@ -7,10 +7,177 @@ */ =20 #include "qemu/osdep.h" +#include "qemu/log.h" #include "hw/pci/pci_device.h" #include "hw/pci/pcie.h" #include "igb_common.h" #include "igb_migration.h" +#include "system/address-spaces.h" +#include "trace.h" + +/* + * Per-VF state serialization / deserialization + */ + +static int igb_core_vf_save_state(IgbVfState *s, void *buf, size_t buf_siz= e) +{ + int size =3D 0; + + trace_igbvf_mig_save_state(s->vfn, size); + return size; +} + +static int igb_core_vf_max_data_size(IgbVfState *s) +{ + return sizeof(s->mig.mig_data); +} + +static int igb_core_vf_load_state(IgbVfState *s, const void *buf, size_t s= ize) +{ + trace_igbvf_mig_load_state(s->vfn, (uint32_t)size); + return 0; +} + +static int igbvf_mig_load(IgbVfState *s, const void *buf, size_t size) +{ + int ret; + + ret =3D igb_core_vf_load_state(s, buf, size); + if (ret < 0) { + return ret; + } + + return 0; +} + +/* + * Migration command handlers + */ + +static void igbvf_mig_update_data_size(IgbVfState *s, uint32_t size) +{ + IgbVfMigState *ms =3D &s->mig; + + ms->mig_data_size =3D size; + pci_set_long(PCI_DEVICE(s)->config + + IGB_MIG_DVSEC_OFFSET + IGB_MIG_DATA_SIZE, size); +} + +static uint8_t igbvf_mig_cmd_save(IgbVfState *s) +{ + IgbVfMigState *ms =3D &s->mig; + MemTxResult r; + int ret; + + if (ms->mig_state !=3D IGB_MIG_STATE_STOP_COPY) { + return IGB_MIG_ERR_BAD_STATE; + } + + if (!ms->mig_data_buf_addr) { + return IGB_MIG_ERR_NO_BUFFER; + } + + ret =3D igb_core_vf_save_state(s, ms->mig_data, sizeof(ms->mig_data)); + if (ret < 0) { + return -ret; + } + igbvf_mig_update_data_size(s, ret); + + r =3D address_space_write(&address_space_memory, ms->mig_data_buf_addr, + MEMTXATTRS_UNSPECIFIED, + ms->mig_data, ms->mig_data_size); + if (r !=3D MEMTX_OK) { + return IGB_MIG_ERR_DMA_FAILED; + } + + return 0; +} + +static uint8_t igbvf_mig_cmd_load(IgbVfState *s) +{ + IgbVfMigState *ms =3D &s->mig; + MemTxResult r; + int ret; + + if (ms->mig_state !=3D IGB_MIG_STATE_RESUMING) { + return IGB_MIG_ERR_BAD_STATE; + } + + if (!ms->mig_data_buf_addr) { + return IGB_MIG_ERR_NO_BUFFER; + } + + if (ms->mig_data_size =3D=3D 0 || + ms->mig_data_size > sizeof(ms->mig_data)) { + return IGB_MIG_ERR_BAD_SIZE; + } + + r =3D address_space_read(&address_space_memory, ms->mig_data_buf_addr, + MEMTXATTRS_UNSPECIFIED, + ms->mig_data, ms->mig_data_size); + if (r !=3D MEMTX_OK) { + return IGB_MIG_ERR_DMA_FAILED; + } + + ret =3D igbvf_mig_load(s, ms->mig_data, ms->mig_data_size); + if (ret < 0) { + return -ret; + } + + return 0; +} + +static uint8_t igbvf_mig_set_state(IgbVfState *s, uint32_t new_state) +{ + IgbVfMigState *ms =3D &s->mig; + uint32_t old =3D ms->mig_state; + int ret; + + switch (new_state) { + case IGB_MIG_STATE_STOP: + if (old !=3D IGB_MIG_STATE_RUNNING && + old !=3D IGB_MIG_STATE_STOP_COPY && + old !=3D IGB_MIG_STATE_RESUMING && + old !=3D IGB_MIG_STATE_ERROR) { + return IGB_MIG_ERR_BAD_STATE; + } + /* Restore DATA_SIZE to max, same as at reset */ + igbvf_mig_update_data_size(s, igb_core_vf_max_data_size(s)); + break; + + case IGB_MIG_STATE_RUNNING: + if (old !=3D IGB_MIG_STATE_STOP) { + return IGB_MIG_ERR_BAD_STATE; + } + break; + + case IGB_MIG_STATE_STOP_COPY: + if (old !=3D IGB_MIG_STATE_STOP) { + return IGB_MIG_ERR_BAD_STATE; + } + ret =3D igb_core_vf_save_state(s, ms->mig_data, sizeof(ms->mig_dat= a)); + if (ret < 0) { + return -ret; + } + igbvf_mig_update_data_size(s, ret); + break; + + case IGB_MIG_STATE_RESUMING: + if (old !=3D IGB_MIG_STATE_STOP) { + return IGB_MIG_ERR_BAD_STATE; + } + memset(ms->mig_data, 0, sizeof(ms->mig_data)); + igbvf_mig_update_data_size(s, 0); + break; + + default: + return IGB_MIG_ERR_BAD_STATE; + } + + ms->mig_state =3D new_state; + trace_igbvf_mig_set_state(s->vfn, old, new_state); + return 0; +} =20 static void igbvf_mig_update_status(IgbVfState *s, uint8_t err) { @@ -27,6 +194,38 @@ static void igbvf_mig_update_status(IgbVfState *s, uint= 8_t err) pci_set_long(dev->config + IGB_MIG_DVSEC_OFFSET + IGB_MIG_STATUS, stat= us); } =20 +static void igbvf_mig_cmd_ctrl(IgbVfState *s, uint32_t val) +{ + uint32_t cmd =3D val & IGB_MIG_CTRL_CMD_MASK; + uint32_t arg =3D val >> IGB_MIG_CTRL_ARG_SHIFT; + uint8_t err =3D 0; + + switch (cmd) { + case IGB_MIG_CMD_SET_STATE: + err =3D igbvf_mig_set_state(s, arg); + break; + + case IGB_MIG_CMD_SAVE: + err =3D igbvf_mig_cmd_save(s); + break; + + case IGB_MIG_CMD_LOAD: + igbvf_mig_update_data_size(s, arg); + err =3D igbvf_mig_cmd_load(s); + break; + + default: + err =3D IGB_MIG_ERR_UNK_CMD; + break; + } + + if (err) { + qemu_log_mask(LOG_GUEST_ERROR, + "igbvf: VF%u CTRL cmd %u failed (error %u)\n", + s->vfn, cmd, err); + } + igbvf_mig_update_status(s, err); +} =20 bool igbvf_add_migration_dvsec(PCIDevice *dev, Error **errp) { @@ -57,6 +256,8 @@ bool igbvf_add_migration_dvsec(PCIDevice *dev, Error **e= rrp) memset(dev->wmask + offset + IGB_MIG_BUF_ADDR_LO, 0xff, 4); memset(dev->wmask + offset + IGB_MIG_BUF_ADDR_HI, 0xff, 4); =20 + /* DATA_SIZE is set by igbvf_mig_state_reset() */ + return true; } =20 @@ -67,6 +268,16 @@ uint32_t igbvf_mig_config_read(IgbVfState *s, uint32_t = addr, int size) return pci_default_read_config(dev, addr, size); } =20 +static uint64_t igbvf_mig_get_buf_addr(IgbVfState *s) +{ + PCIDevice *dev =3D PCI_DEVICE(s); + uint32_t lo, hi; + + lo =3D pci_get_long(dev->config + IGB_MIG_DVSEC_OFFSET + IGB_MIG_BUF_A= DDR_LO); + hi =3D pci_get_long(dev->config + IGB_MIG_DVSEC_OFFSET + IGB_MIG_BUF_A= DDR_HI); + return ((uint64_t)hi << 32) | lo; +} + bool igbvf_mig_config_write(IgbVfState *s, uint32_t addr, uint32_t val, int size) { @@ -75,7 +286,8 @@ bool igbvf_mig_config_write(IgbVfState *s, uint32_t addr= , uint32_t val, =20 switch (offset) { case IGB_MIG_CTRL: - /* Command handling will be added in a later commit */ + s->mig.mig_data_buf_addr =3D igbvf_mig_get_buf_addr(s); + igbvf_mig_cmd_ctrl(s, val); break; =20 case IGB_MIG_BUF_ADDR_LO: @@ -94,8 +306,11 @@ void igbvf_mig_state_reset(IgbVfState *s) { IgbVfMigState *ms =3D &s->mig; =20 + trace_igbvf_mig_reset(s->vfn); ms->mig_state =3D IGB_MIG_STATE_RUNNING; ms->mig_data_buf_addr =3D 0; + igbvf_mig_update_data_size(s, igb_core_vf_max_data_size(s)); + memset(ms->mig_data, 0, sizeof(ms->mig_data)); =20 pci_set_long(PCI_DEVICE(s)->config + IGB_MIG_DVSEC_OFFSET + IGB_MIG_BUF_ADDR_LO, 0); diff --git a/hw/net/trace-events b/hw/net/trace-events index 001a20b0e2ac..7057fbe5f16a 100644 --- a/hw/net/trace-events +++ b/hw/net/trace-events @@ -295,6 +295,12 @@ igb_wrn_rx_desc_modes_not_supp(int desc_type) "Not sup= ported descriptor type: %d # igbvf.c igbvf_wrn_io_addr_unknown(uint64_t addr) "IO unknown register 0x%"PRIx64 =20 +# igb_migration.c +igbvf_mig_set_state(uint16_t vfn, uint32_t old_state, uint32_t new_state) = "VF%u: state %u -> %u" +igbvf_mig_save_state(uint16_t vfn, uint32_t size) "VF%u: saved %u bytes of= device state" +igbvf_mig_load_state(uint16_t vfn, uint32_t size) "VF%u: loaded %u bytes o= f device state" +igbvf_mig_reset(uint16_t vfn) "VF%u: migration state reset" + # spapr_llan.c spapr_vlan_get_rx_bd_from_pool_found(int pool, int32_t count, uint32_t rx_= bufs) "pool=3D%d count=3D%"PRId32" rxbufs=3D%"PRIu32 spapr_vlan_get_rx_bd_from_page(int buf_ptr, uint64_t bd) "use_buf_ptr=3D%d= bd=3D0x%016"PRIx64 --=20 2.55.0 From nobody Sat Sep 26 22:16:24 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1788376926; cv=none; d=zohomail.com; s=zohoarc; b=h3s9QctNeKpEGseq1sAduep0Gj8IdxW4E+zjxFknfss/KM1UuRN/ikkdmIYH2V9cEp75vz/2x46iyxKJ9jJz4FAbEVDCWvE0I2P6qyTEomALE7pb+58caePu83sMyJcUu5R1TcejO4clCRyRTo6jG5OcvLfFh2uHfc79eli4V9s= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1788376926; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=PVKZPlCHc9YqmFTv++EH7vALHKlkjgXDUbmgLVOuh0g=; b=auhoqN/hEP/LJyPcVuCaijvkQDIRmfe8Uqr/fJev/C6brNTTDFK3dtWmNDTawTCXa2nNI13CExJP74j3VcBTbq6Y+FMxwiKT4wgGWXI6g2awWRKf/DGYq/K8VWVT8mLOrjpazGA85roBC0Yj1TeXLE3UmeAEJ/fvDkdbZ12lCj0= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1788376926698287.89763148445286; Wed, 2 Sep 2026 12:22:06 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x1qWO-0004JM-FZ; Wed, 02 Sep 2026 15:21:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x1qWN-0004JC-NC for qemu-devel@nongnu.org; Wed, 02 Sep 2026 15:21:19 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x1qWK-00026c-P3 for qemu-devel@nongnu.org; Wed, 02 Sep 2026 15:21:19 -0400 Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-86-P5N1iiMbMO62YE-D1nabow-1; Wed, 02 Sep 2026 15:21:11 -0400 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id DDF7A1805A23; Wed, 2 Sep 2026 19:21:09 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.32.5]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 726B53000DA3; Wed, 2 Sep 2026 19:21:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1788376876; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PVKZPlCHc9YqmFTv++EH7vALHKlkjgXDUbmgLVOuh0g=; b=UE6qIuYy9E831PUYTMrp31/ol9PlbK45NpJqFE9FYQ/BITJFW0b2bAwhAxQjH7TKvS2E3X 4uikiwFOH5aDykUL3nWh/Op23ZdFKsUPtSEkOrmNd8LAe/+Xfj3DCqnlfsvrGnPbP9Ys7D CCr10HYQLw07thW/tDUVB8T+DI6EhDM= X-MC-Unique: P5N1iiMbMO62YE-D1nabow-1 X-Mimecast-MFC-AGG-ID: P5N1iiMbMO62YE-D1nabow_1788376870 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Akihiko Odaki , Sriram Yagnaraman , Jason Wang , Alex Williamson , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [RFC PATCH v2 3/9] igb: Add VF state serialization for live migration Date: Wed, 2 Sep 2026 21:20:48 +0200 Message-ID: <20260902192054.3329753-4-clg@redhat.com> In-Reply-To: <20260902192054.3329753-1-clg@redhat.com> References: <20260902192054.3329753-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1788376929350154100 Implement per-VF state serialization and deserialization for the SAVE and LOAD commands. The wire format consists of a header (magic, version, VF number, register count), per-VF register offset/value pairs from a whitelist, RA table entries owned by the VF, and TX queue contexts. Register offsets are relocated on load so a VF can migrate to a different VF number on the destination. RA pool ownership bits are swapped accordingly. AI-used-for: analysis, code (prototype) Signed-off-by: C=C3=A9dric Le Goater --- hw/net/igb_core.h | 2 + hw/net/igb_migration.h | 2 + hw/net/igb.c | 5 + hw/net/igb_migration.c | 347 ++++++++++++++++++++++++++++++++++++++++- 4 files changed, 354 insertions(+), 2 deletions(-) diff --git a/hw/net/igb_core.h b/hw/net/igb_core.h index d70b54e318f1..60724e2824ab 100644 --- a/hw/net/igb_core.h +++ b/hw/net/igb_core.h @@ -143,4 +143,6 @@ igb_receive_iov(IGBCore *core, const struct iovec *iov,= int iovcnt); void igb_start_recv(IGBCore *core); =20 +IGBCore *igb_pf_get_core(void *pf); + #endif diff --git a/hw/net/igb_migration.h b/hw/net/igb_migration.h index ea40ac65c54b..b2f601e74346 100644 --- a/hw/net/igb_migration.h +++ b/hw/net/igb_migration.h @@ -73,6 +73,8 @@ #define IGB_MIG_ERR_NO_BUFFER 3 #define IGB_MIG_ERR_DMA_FAILED 4 #define IGB_MIG_ERR_BAD_SIZE 5 +#define IGB_MIG_ERR_BAD_MAGIC 6 +#define IGB_MIG_ERR_BAD_VERSION 7 =20 /* Shared buffer constants */ #define IGB_VF_STATE_MAX_SIZE 4096 diff --git a/hw/net/igb.c b/hw/net/igb.c index 7268e5473fc3..f39f2bc3a04e 100644 --- a/hw/net/igb.c +++ b/hw/net/igb.c @@ -133,6 +133,11 @@ void igb_vf_reset(void *opaque, uint16_t vfn) igb_core_vf_reset(&s->core, vfn); } =20 +IGBCore *igb_pf_get_core(void *pf) +{ + return &IGB(pf)->core; +} + static bool igb_io_get_reg_index(IGBState *s, uint32_t *idx) { diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c index 8e7e6fac9b9b..c34035974620 100644 --- a/hw/net/igb_migration.c +++ b/hw/net/igb_migration.c @@ -10,18 +10,241 @@ #include "qemu/log.h" #include "hw/pci/pci_device.h" #include "hw/pci/pcie.h" +#include "net/eth.h" +#include "net/net.h" #include "igb_common.h" +#include "igb_core.h" #include "igb_migration.h" #include "system/address-spaces.h" #include "trace.h" =20 +static IGBCore *igbvf_get_core(IgbVfState *s) +{ + return igb_pf_get_core(pcie_sriov_get_pf(PCI_DEVICE(s))); +} + /* * Per-VF state serialization / deserialization */ =20 +#define IGB_MIG_BLOB_MAGIC 0x4D494742 /* "MIGB" */ +#define IGB_MIG_BLOB_VERSION 1 + +typedef struct IgbMigRegPair { + uint32_t offset; + uint32_t value; +} IgbMigRegPair; + +typedef struct IgbMigTxCtx { + uint32_t ctx_desc[8]; /* 2 =C3=97 adv_tx_context_desc (4 dword= s each) */ + uint32_t first_cmd_type_len; + uint32_t first_olinfo_status; + uint32_t first; + uint32_t skip_cp; +} IgbMigTxCtx; + +#define IGB_VF_MAX_FIXED_REGS 64 +#define IGB_VF_MAX_RA_REGS 48 /* (16 + 8) RA entries =C3=97 2 (RAL= +RAH) */ + +typedef struct IgbMigBlob { + uint32_t magic; + uint32_t version; + uint32_t vfn; + uint32_t num_regs; + IgbMigRegPair regs[IGB_VF_MAX_FIXED_REGS]; + uint32_t num_ra; + IgbMigRegPair ra[IGB_VF_MAX_RA_REGS]; + uint32_t num_tx_ctx; + IgbMigTxCtx tx_ctx[2]; +} IgbMigBlob; + +#define IGB_MIG_BLOB_SIZE sizeof(IgbMigBlob) + +QEMU_BUILD_BUG_ON(IGB_MIG_BLOB_SIZE > IGB_VF_STATE_MAX_SIZE); + +/* Register offsets that constitute a VF's state slice */ +static int igb_vf_reg_list(uint16_t vfn, uint32_t *offsets) +{ + int n =3D 0; + int q0 =3D vfn; + int q1 =3D vfn + IGB_NUM_VM_POOLS; + + /* Per-VF control and interrupt registers */ + offsets[n++] =3D E1000_PVTCTRL(vfn) >> 2; + offsets[n++] =3D E1000_PVTEICS(vfn) >> 2; + offsets[n++] =3D E1000_PVTEIMS(vfn) >> 2; + offsets[n++] =3D E1000_PVTEIMC(vfn) >> 2; + offsets[n++] =3D E1000_PVTEIAC(vfn) >> 2; + offsets[n++] =3D E1000_PVTEIAM(vfn) >> 2; + offsets[n++] =3D E1000_PVTEICR(vfn) >> 2; + + /* Per-VF statistics */ + offsets[n++] =3D E1000_PVFGPRC(vfn) >> 2; + offsets[n++] =3D E1000_PVFGPTC(vfn) >> 2; + offsets[n++] =3D E1000_PVFGORC(vfn) >> 2; + offsets[n++] =3D E1000_PVFGOTC(vfn) >> 2; + offsets[n++] =3D E1000_PVFMPRC(vfn) >> 2; + offsets[n++] =3D E1000_PVFGPRLBC(vfn) >> 2; + offsets[n++] =3D E1000_PVFGPTLBC(vfn) >> 2; + offsets[n++] =3D E1000_PVFGORLBC(vfn) >> 2; + offsets[n++] =3D E1000_PVFGOTLBC(vfn) >> 2; + + /* + * Mailbox control registers only - the 16-dword payload buffer + * (VMBMEM) is transient and drained on quiesce. + */ + offsets[n++] =3D E1000_V2PMAILBOX(vfn) >> 2; + offsets[n++] =3D E1000_P2VMAILBOX(vfn) >> 2; + + /* Per-VF config */ + offsets[n++] =3D E1000_VMOLR(vfn) >> 2; + offsets[n++] =3D E1000_VMVIR(vfn) >> 2; + offsets[n++] =3D E1000_PSRTYPE(vfn) >> 2; + + /* + * VF receive addresses (RA/RA2) are saved dynamically in + * igb_core_vf_save_state by scanning for entries whose pool + * bits match this VF - the PF driver chooses the RA slot. + */ + + /* Interrupt routing */ + offsets[n++] =3D (E1000_VTIVAR + vfn * 4) >> 2; + offsets[n++] =3D (E1000_VTIVAR_MISC + vfn * 4) >> 2; + + /* + * EITR (Extended Interrupt Throttle Register) - 3 vectors per VF. + * Each VF has 3 MSI-X vectors, each with its own EITR controlling + * interrupt coalescing. Without saving these, interrupt + * throttling resets to zero after migration which can cause + * interrupt storms or latency changes. VF N uses PF EITR indices + * (22 - N*3) .. (24 - N*3). + */ + { + int eitr_base =3D 22 - vfn * 3; + offsets[n++] =3D E1000_EITR(eitr_base) >> 2; + offsets[n++] =3D E1000_EITR(eitr_base + 1) >> 2; + offsets[n++] =3D E1000_EITR(eitr_base + 2) >> 2; + } + + /* RX and TX queue registers for queues q0 and q1 */ +#define ADD_QUEUE_REGS(q) do { \ + offsets[n++] =3D E1000_RDBAL(q) >> 2; \ + offsets[n++] =3D E1000_RDBAH(q) >> 2; \ + offsets[n++] =3D E1000_RDLEN(q) >> 2; \ + offsets[n++] =3D E1000_SRRCTL(q) >> 2; \ + offsets[n++] =3D E1000_RDH(q) >> 2; \ + offsets[n++] =3D E1000_RDT(q) >> 2; \ + offsets[n++] =3D E1000_RXDCTL(q) >> 2; \ + offsets[n++] =3D E1000_RXCTL(q) >> 2; \ + offsets[n++] =3D E1000_RQDPC(q) >> 2; \ + offsets[n++] =3D E1000_TDBAL(q) >> 2; \ + offsets[n++] =3D E1000_TDBAH(q) >> 2; \ + offsets[n++] =3D E1000_TDLEN(q) >> 2; \ + offsets[n++] =3D E1000_TDH(q) >> 2; \ + offsets[n++] =3D E1000_TDT(q) >> 2; \ + offsets[n++] =3D E1000_TXDCTL(q) >> 2; \ + offsets[n++] =3D E1000_TXCTL(q) >> 2; \ + offsets[n++] =3D E1000_TDWBAL(q) >> 2; \ + offsets[n++] =3D E1000_TDWBAH(q) >> 2; \ +} while (0) + + ADD_QUEUE_REGS(q0); + ADD_QUEUE_REGS(q1); +#undef ADD_QUEUE_REGS + + g_assert(n <=3D IGB_VF_MAX_FIXED_REGS); + return n; +} + +/* + * Scan RA and RA2 arrays for receive address entries assigned to + * this VF. The PF driver picks the RA slot, so we cannot use a + * fixed index - instead check each entry's pool bits. + */ +static int igb_core_vf_save_ra(IGBCore *core, uint16_t vfn, + IgbMigRegPair *regs) +{ + uint32_t vf_pool_bit =3D E1000_RAH_POOL_1 << vfn; + int n =3D 0; + static const struct { + uint32_t base; + int count; + } ra_banks[] =3D { + { RA, 16 }, + { RA2, 8 }, + }; + + for (int i =3D 0; i < ARRAY_SIZE(ra_banks); i++) { + for (int j =3D 0; j < ra_banks[i].count; j++) { + uint32_t ral_off =3D ra_banks[i].base + j * 2; + uint32_t rah_off =3D ra_banks[i].base + j * 2 + 1; + uint32_t rah_val =3D core->mac[rah_off]; + + if ((rah_val & E1000_RAH_AV) && (rah_val & vf_pool_bit)) { + regs[n].offset =3D cpu_to_le32(ral_off); + regs[n].value =3D cpu_to_le32(core->mac[ral_off]); + n++; + regs[n].offset =3D cpu_to_le32(rah_off); + regs[n].value =3D cpu_to_le32(rah_val); + n++; + } + } + } + return n; +} + +static void igb_core_vf_save_tx_ctx(IGBCore *core, int queue, + IgbMigTxCtx *tx) +{ + struct igb_tx *src =3D &core->tx[queue]; + + memcpy(tx->ctx_desc, src->ctx, sizeof(tx->ctx_desc)); + tx->first_cmd_type_len =3D cpu_to_le32(src->first_cmd_type_len); + tx->first_olinfo_status =3D cpu_to_le32(src->first_olinfo_status); + tx->first =3D cpu_to_le32(src->first); + tx->skip_cp =3D cpu_to_le32(src->skip_cp); +} + static int igb_core_vf_save_state(IgbVfState *s, void *buf, size_t buf_siz= e) { - int size =3D 0; + int size =3D IGB_MIG_BLOB_SIZE; + IGBCore *core =3D igbvf_get_core(s); + IgbMigBlob *blob =3D buf; + uint32_t offsets[IGB_VF_MAX_FIXED_REGS]; + int num_regs; + int q0 =3D s->vfn; + int q1 =3D s->vfn + IGB_NUM_VM_POOLS; + + /* + * Save PVT shadow registers (PVTEIMS/PVTEIAC/PVTEIAM) instead of + * extracting from PF aggregates - the L1 PF driver may have + * transiently cleared EIMS via EIMC. The load path ORs them back. + */ + num_regs =3D igb_vf_reg_list(s->vfn, offsets); + + if (!buf) { + return size; + } + + if (size > buf_size) { + return -IGB_MIG_ERR_BAD_SIZE; + } + + blob->magic =3D cpu_to_le32(IGB_MIG_BLOB_MAGIC); + blob->version =3D cpu_to_le32(IGB_MIG_BLOB_VERSION); + blob->vfn =3D cpu_to_le32(s->vfn); + + blob->num_regs =3D cpu_to_le32(num_regs); + for (int i =3D 0; i < num_regs; i++) { + blob->regs[i].offset =3D cpu_to_le32(offsets[i]); + blob->regs[i].value =3D cpu_to_le32(core->mac[offsets[i]]); + } + + blob->num_ra =3D cpu_to_le32(igb_core_vf_save_ra(core, s->vfn, blob->r= a)); + + blob->num_tx_ctx =3D cpu_to_le32(2); + igb_core_vf_save_tx_ctx(core, q0, &blob->tx_ctx[0]); + igb_core_vf_save_tx_ctx(core, q1, &blob->tx_ctx[1]); =20 trace_igbvf_mig_save_state(s->vfn, size); return size; @@ -29,11 +252,131 @@ static int igb_core_vf_save_state(IgbVfState *s, void= *buf, size_t buf_size) =20 static int igb_core_vf_max_data_size(IgbVfState *s) { - return sizeof(s->mig.mig_data); + int size =3D igb_core_vf_save_state(s, NULL, 0); + + g_assert(size > 0 && size <=3D IGB_VF_STATE_MAX_SIZE); + return size; +} + +static void igb_core_vf_load_tx_ctx(IGBCore *core, int queue, + const IgbMigTxCtx *tx) +{ + struct igb_tx *dst =3D &core->tx[queue]; + + /* + * Preserve the destination's tx_pkt - it's a host-side object, + * not guest state + */ + memcpy(dst->ctx, tx->ctx_desc, sizeof(dst->ctx)); + dst->first_cmd_type_len =3D le32_to_cpu(tx->first_cmd_type_len); + dst->first_olinfo_status =3D le32_to_cpu(tx->first_olinfo_status); + dst->first =3D le32_to_cpu(tx->first); + dst->skip_cp =3D le32_to_cpu(tx->skip_cp); +} + +static uint32_t igb_vf_relocate_offset(uint32_t offset, + const uint32_t *src_offsets, + const uint32_t *dst_offsets, + int num_offsets) +{ + for (int i =3D 0; i < num_offsets; i++) { + if (src_offsets[i] =3D=3D offset) { + return dst_offsets[i]; + } + } + return 0; } =20 static int igb_core_vf_load_state(IgbVfState *s, const void *buf, size_t s= ize) { + IGBCore *core =3D igbvf_get_core(s); + uint32_t src_offsets[IGB_VF_MAX_FIXED_REGS]; + uint32_t dst_offsets[IGB_VF_MAX_FIXED_REGS]; + int q0 =3D s->vfn; + int q1 =3D s->vfn + IGB_NUM_VM_POOLS; + + if (size < IGB_MIG_BLOB_SIZE) { + return -IGB_MIG_ERR_BAD_SIZE; + } + + const IgbMigBlob *blob =3D buf; + + uint32_t magic =3D le32_to_cpu(blob->magic); + uint32_t version =3D le32_to_cpu(blob->version); + uint32_t saved_vfn =3D le32_to_cpu(blob->vfn); + uint32_t num_regs =3D le32_to_cpu(blob->num_regs); + + if (magic !=3D IGB_MIG_BLOB_MAGIC) { + return -IGB_MIG_ERR_BAD_MAGIC; + } + if (version !=3D IGB_MIG_BLOB_VERSION) { + return -IGB_MIG_ERR_BAD_VERSION; + } + if (num_regs > IGB_VF_MAX_FIXED_REGS) { + return -IGB_MIG_ERR_BAD_SIZE; + } + + uint32_t num_ra =3D le32_to_cpu(blob->num_ra); + if (num_ra > IGB_VF_MAX_RA_REGS) { + return -IGB_MIG_ERR_BAD_SIZE; + } + + int num_offsets =3D igb_vf_reg_list(saved_vfn, src_offsets); + igb_vf_reg_list(s->vfn, dst_offsets); + + for (uint32_t i =3D 0; i < num_regs; i++) { + uint32_t src_off =3D le32_to_cpu(blob->regs[i].offset); + uint32_t value =3D le32_to_cpu(blob->regs[i].value); + uint32_t offset =3D igb_vf_relocate_offset(src_off, + src_offsets, dst_offsets, num_offsets); + if (!offset) { + return -IGB_MIG_ERR_BAD_SIZE; + } + + core->mac[offset] =3D value; + + /* + * Sync EITR to eitr_guest_value[] shadow array, stripping + * E1000_EITR_CNT_IGNR so guest register readback returns the + * correct value. + */ + if (offset >=3D EITR0 && offset < EITR0 + IGB_INTR_NUM) { + core->eitr_guest_value[offset - EITR0] =3D + value & ~E1000_EITR_CNT_IGNR; + } + } + + /* + * MSI-X table/PBA is not saved - L1's VFIO reprograms it with + * destination-specific IRTE references after migration. + */ + + uint32_t src_pool =3D E1000_RAH_POOL_1 << saved_vfn; + uint32_t dst_pool =3D E1000_RAH_POOL_1 << s->vfn; + + for (uint32_t i =3D 0; i < num_ra; i++) { + uint32_t offset =3D le32_to_cpu(blob->ra[i].offset); + uint32_t value =3D le32_to_cpu(blob->ra[i].value); + + /* RAH entries: swap pool ownership bits */ + if (offset >=3D RA && offset < RA + 32 && (offset - RA) % 2 =3D=3D= 1) { + value =3D (value & ~src_pool) | dst_pool; + } + if (offset >=3D RA2 && offset < RA2 + 16 && (offset - RA2) % 2 =3D= =3D 1) { + value =3D (value & ~src_pool) | dst_pool; + } + + core->mac[offset] =3D value; + } + + uint32_t num_tx =3D le32_to_cpu(blob->num_tx_ctx); + if (num_tx !=3D 2) { + return -IGB_MIG_ERR_BAD_SIZE; + } + + igb_core_vf_load_tx_ctx(core, q0, &blob->tx_ctx[0]); + igb_core_vf_load_tx_ctx(core, q1, &blob->tx_ctx[1]); + trace_igbvf_mig_load_state(s->vfn, (uint32_t)size); return 0; } --=20 2.55.0 From nobody Sat Sep 26 22:16:24 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1788376937; cv=none; d=zohomail.com; s=zohoarc; b=EEmVGYFyARYpa8beFeKmDW6Q6jiYCfe0sW/9MpUqZT7RKebvUbOWTgnGB6IGPsWtb3Bxigzo1BaDy48xCSMtWMzCtu8PgplNBCcaSfHKhggSgacHe0J14sQsbySeHAIuZ4evLKuOdsgFXlsF1BcSsc+DKRynZszaEoUA19xeyzg= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1788376937; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=xbLCMWNmPBdl2Wk4lrVEwwfNX8cyJ+dKTeNJ77eaiLQ=; b=NOd2O6+MVOdLEWA6d2Qty1EyI7HJNxItlPMGP1h4V39LkUrAoFPHOFDb5ZAKSGUSv6HpMjpgpN+fwAc1bJ4OD0RDO1Di1DmNslLTGYNw0xyFMGSCXwWYjOdzN4RSSb9I/dMop7UOwKSQ2lW7SkvLCx1jXjQsmxJpi9lWnXPcljw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1788376937588213.93427656435506; Wed, 2 Sep 2026 12:22:17 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x1qWT-0004K9-6P; Wed, 02 Sep 2026 15:21:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x1qWQ-0004Jl-1T for qemu-devel@nongnu.org; Wed, 02 Sep 2026 15:21:22 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x1qWO-000271-37 for qemu-devel@nongnu.org; Wed, 02 Sep 2026 15:21:21 -0400 Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-436-Q5NGRpAlOpCH3bTL0JeS9A-1; Wed, 02 Sep 2026 15:21:14 -0400 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 264151828B1D; Wed, 2 Sep 2026 19:21:13 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.32.5]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 8A2E23001D39; Wed, 2 Sep 2026 19:21:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1788376879; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xbLCMWNmPBdl2Wk4lrVEwwfNX8cyJ+dKTeNJ77eaiLQ=; b=O+CVTjsxWrX6YIQwt76uvxp+aztNnQw5hT+VSeHPwtuhAcdxtb6TLnwRj8/qvPRi2nJqyP HRLcjF6V4ji3WyByNwfQHpTyG5GtctVWrqEMAZw1eIk+4vNwHz2TSfM/EMiberVgzKqQ9m 1nDzYf6EFpOED0DQD+i0LoGHD7Nmo18= X-MC-Unique: Q5NGRpAlOpCH3bTL0JeS9A-1 X-Mimecast-MFC-AGG-ID: Q5NGRpAlOpCH3bTL0JeS9A_1788376873 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Akihiko Odaki , Sriram Yagnaraman , Jason Wang , Alex Williamson , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [RFC PATCH v2 4/9] igb: Add VF post-load fixups for live migration Date: Wed, 2 Sep 2026 21:20:49 +0200 Message-ID: <20260902192054.3329753-5-clg@redhat.com> In-Reply-To: <20260902192054.3329753-1-clg@redhat.com> References: <20260902192054.3329753-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1788376940388158500 After restoring per-VF register state, propagate the VF's PVT shadow values back into the PF's aggregate EIMS/EIAC/EIAM registers and re-apply the VTIVAR interrupt vector routing to the shared IVAR0. AI-used-for: analysis, code (prototype) Signed-off-by: C=C3=A9dric Le Goater --- hw/net/igb_core.h | 3 ++ hw/net/igb_core.c | 66 ++++++++++++++++++++++++++++++++++++++++++ hw/net/igb_migration.c | 7 +++++ 3 files changed, 76 insertions(+) diff --git a/hw/net/igb_core.h b/hw/net/igb_core.h index 60724e2824ab..22e10e4e6d0b 100644 --- a/hw/net/igb_core.h +++ b/hw/net/igb_core.h @@ -145,4 +145,7 @@ igb_start_recv(IGBCore *core); =20 IGBCore *igb_pf_get_core(void *pf); =20 +void igb_core_vf_propagate_irqs(IGBCore *core, uint16_t vfn); +void igb_core_vf_propagate_ivar(IGBCore *core, uint16_t vfn); + #endif diff --git a/hw/net/igb_core.c b/hw/net/igb_core.c index 2a4883907353..01745fe756d0 100644 --- a/hw/net/igb_core.c +++ b/hw/net/igb_core.c @@ -4552,3 +4552,69 @@ igb_core_post_load(IGBCore *core) =20 return 0; } + +/* + * Propagate VF interrupt state to PF aggregates after loading VF + * registers. The load path writes directly to mac[] bypassing the + * register handlers that OR VF bits into EIMS/EIAC/EIAM. Also clear + * stale VF bits in EICR that may have been set by packets arriving + * between PF vmstate restore and VF state load. + */ +void igb_core_vf_propagate_irqs(IGBCore *core, uint16_t vfn) +{ + uint32_t shift =3D 22 - vfn * IGBVF_MSIX_VEC_NUM; + uint32_t vf_mask =3D 0x7 << shift; + uint32_t pvt_idx; + + core->mac[EIMS] &=3D ~vf_mask; + pvt_idx =3D PVTEIMS0 + vfn * 0x40; + core->mac[EIMS] |=3D (core->mac[pvt_idx] & 0x7) << shift; + + core->mac[EIAC] &=3D ~vf_mask; + pvt_idx =3D PVTEIAC0 + vfn * 0x40; + core->mac[EIAC] |=3D (core->mac[pvt_idx] & 0x7) << shift; + + core->mac[EIAM] &=3D ~vf_mask; + pvt_idx =3D PVTEIAM0 + vfn * 0x40; + core->mac[EIAM] |=3D (core->mac[pvt_idx] & 0x7) << shift; + + core->mac[EICR] &=3D ~vf_mask; +} + +/* + * Re-apply VTIVAR -> IVAR0 interrupt routing. The L1 PF driver + * may have overwritten the shared IVAR0 entries with its own + * queue routing after L0 vmstate restore. + */ +void igb_core_vf_propagate_ivar(IGBCore *core, uint16_t vfn) +{ + uint32_t vtivar =3D core->mac[VTIVAR + vfn]; + int n; + uint8_t ent; + uint32_t mask; + + n =3D igb_ivar_entry_rx(vfn); + mask =3D 0xffU << (8 * (n % 4)); + if (vtivar & E1000_IVAR_VALID) { + ent =3D E1000_IVAR_VALID | + (24 - vfn * IGBVF_MSIX_VEC_NUM - (2 - (vtivar & 0x7))); + core->mac[IVAR0 + n / 4] =3D + (core->mac[IVAR0 + n / 4] & ~mask) | + ((uint32_t)ent << (8 * (n % 4))); + } else { + core->mac[IVAR0 + n / 4] &=3D ~mask; + } + + n =3D igb_ivar_entry_tx(vfn); + mask =3D 0xffU << (8 * (n % 4)); + ent =3D vtivar >> 8; + if (ent & E1000_IVAR_VALID) { + ent =3D E1000_IVAR_VALID | + (24 - vfn * IGBVF_MSIX_VEC_NUM - (2 - (ent & 0x7))); + core->mac[IVAR0 + n / 4] =3D + (core->mac[IVAR0 + n / 4] & ~mask) | + ((uint32_t)ent << (8 * (n % 4))); + } else { + core->mac[IVAR0 + n / 4] &=3D ~mask; + } +} diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c index c34035974620..af7257ccc4fd 100644 --- a/hw/net/igb_migration.c +++ b/hw/net/igb_migration.c @@ -384,12 +384,19 @@ static int igb_core_vf_load_state(IgbVfState *s, cons= t void *buf, size_t size) static int igbvf_mig_load(IgbVfState *s, const void *buf, size_t size) { int ret; + IGBCore *core =3D igbvf_get_core(s); =20 ret =3D igb_core_vf_load_state(s, buf, size); if (ret < 0) { return ret; } =20 + /* + * Post-load: sync VF interrupt and routing state to PF aggregates + */ + igb_core_vf_propagate_irqs(core, s->vfn); + igb_core_vf_propagate_ivar(core, s->vfn); + return 0; } =20 --=20 2.55.0 From nobody Sat Sep 26 22:16:24 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1788376959; cv=none; d=zohomail.com; s=zohoarc; b=PJExwGjTi7qVF0mQnjc422dPFk6EQRWGksOZ+/L/tfbjTea1QKUrUor3bBmeskNIdOXtS3PphdJWPRftJD+RlsEnV7p8ffv0W7d9VaceoXlpkAFXTK+kRcuQGIYNRVblnBXJhpaRWXdEaKZVDXiVEJKqJykIiPGVoeFSo95PgY0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1788376959; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=it5nOrSIqoiFDHGBXn0G01L3wXPGQ+o1sF2yHr3yFqc=; b=fkKUDt3VxjeE5X3OAXabJCZY9GQH4osNz/a5otu9XcSYBYTDePmBmKOoIFZoUh1dWL1mi1CMc1/TGQFDgW+8CnNVNWfMM5dAYe8AdJ1qB61SY+1iuuvmJg8Q4XUrynWwI6r/mAbZh9oDUbClrMdMiqGEervijagwLj8qd7ZW9ic= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1788376959410199.50678331384086; Wed, 2 Sep 2026 12:22:39 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x1qWV-0004KP-0y; Wed, 02 Sep 2026 15:21:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x1qWS-0004KB-Mi for qemu-devel@nongnu.org; Wed, 02 Sep 2026 15:21:25 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x1qWP-00027M-72 for qemu-devel@nongnu.org; Wed, 02 Sep 2026 15:21:24 -0400 Received: from mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-124-7Ky-Nou3NLmh2WgB0yMPrg-1; Wed, 02 Sep 2026 15:21:17 -0400 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-08.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id E41311830652; Wed, 2 Sep 2026 19:21:15 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.32.5]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 922B73000DA3; Wed, 2 Sep 2026 19:21:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1788376880; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=it5nOrSIqoiFDHGBXn0G01L3wXPGQ+o1sF2yHr3yFqc=; b=cRTy6KgT+IfVqh/fcakoetsgNRJ+Zy0G/h2A4peXfUtkcjCXcX87sGPiF9e2ja1OtiXvo9 ADzgyS9D13FqMU+QqS0tOvO9wlRrrOjb6ToHozkNU9MZGBn37Vu5nUOCYuXVHm5MBMWxig joQr/EJdvBDy2tQnn0WW0P56kyqcwMU= X-MC-Unique: 7Ky-Nou3NLmh2WgB0yMPrg-1 X-Mimecast-MFC-AGG-ID: 7Ky-Nou3NLmh2WgB0yMPrg_1788376876 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Akihiko Odaki , Sriram Yagnaraman , Jason Wang , Alex Williamson , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [RFC PATCH v2 5/9] igb: Add dirty page tracking for IGBVF migration Date: Wed, 2 Sep 2026 21:20:50 +0200 Message-ID: <20260902192054.3329753-6-clg@redhat.com> In-Reply-To: <20260902192054.3329753-1-clg@redhat.com> References: <20260902192054.3329753-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1788376961339154100 Add per-VF dirty page tracking using bitmaps allocated per IOVA range. The DMA data path is instrumented via igb_core_dirty_track_dma() calls at a= ll five DMA write sites: TX descriptor writeback (head pointer and status), RX descriptor writeback, RX header fragment, and RX payload fragment. The migration DVSEC exposes DIRTY_ENABLE, DIRTY_DISABLE and DIRTY_QUERY commands. Range parameters and query results are exchanged through the shared DMA buffer. Add the PRE_COPY device state to support pre-copy live migration with concurrent dirty tracking. PRE_COPY transitions: - RUNNING -> PRE_COPY - PRE_COPY -> STOP, RUNNING, STOP_COPY Dirty tracking is automatically disabled when leaving PRE_COPY or STOP_COPY. AI-used-for: code (prototype) Signed-off-by: C=C3=A9dric Le Goater --- hw/net/igb_core.h | 4 + hw/net/igb_migration.h | 65 +++++++- hw/net/igb_core.c | 42 ++++-- hw/net/igb_migration.c | 326 ++++++++++++++++++++++++++++++++++++++++- hw/net/trace-events | 5 + 5 files changed, 422 insertions(+), 20 deletions(-) diff --git a/hw/net/igb_core.h b/hw/net/igb_core.h index 22e10e4e6d0b..7fdc41690b36 100644 --- a/hw/net/igb_core.h +++ b/hw/net/igb_core.h @@ -40,6 +40,8 @@ #ifndef HW_NET_IGB_CORE_H #define HW_NET_IGB_CORE_H =20 +#include "igb_migration.h" + #define E1000E_MAC_SIZE (0x8000) #define IGB_EEPROM_SIZE (1024) =20 @@ -99,6 +101,8 @@ struct IGBCore { void (*owner_start_recv)(PCIDevice *d); =20 int64_t timadj; + + IGBVfDirtyState vf_dirty[IGB_MAX_VF_FUNCTIONS]; }; =20 void diff --git a/hw/net/igb_migration.h b/hw/net/igb_migration.h index b2f601e74346..996de2d2f50b 100644 --- a/hw/net/igb_migration.h +++ b/hw/net/igb_migration.h @@ -21,7 +21,8 @@ * +0x04 DVSEC header 1 (len | rev | vendor_id) * +0x08 DVSEC header 2 (DVSEC ID) * +0x0A Reserved (padding for DWORD alignment) - * +0x0C CAPS (RO: F_STATE[0]) + * +0x0C CAPS (RO: F_STATE[0], F_DIRTY[1], + * max_ranges[11:8], pgsize[16:12]) * +0x10 CTRL (WO: doorbell command) * +0x14 STATUS (RO: state[7:0], error_code[15:8]) * +0x18 BUF_ADDR_LO (RW: shared buffer GPA low) @@ -44,6 +45,12 @@ =20 /* CAPS register layout */ #define IGB_MIG_CAP_F_STATE (1u << 0) +#define IGB_MIG_CAP_F_DIRTY (1u << 1) +#define IGB_MIG_CAPS_MAX_RANGES_SHIFT 8 +#define IGB_MIG_CAPS_MAX_RANGES 4 +#define IGB_MIG_CAPS_PGSIZE_SHIFT 12 +#define IGB_MIG_CAPS_PGSIZE_4K (1u << 12) +#define IGB_MIG_CAPS_PGSIZE_64K (1u << 16) =20 /* CTRL register: command in [7:0] */ #define IGB_MIG_CTRL_CMD_MASK 0xFF @@ -53,6 +60,9 @@ #define IGB_MIG_CMD_SET_STATE 1 #define IGB_MIG_CMD_SAVE 2 #define IGB_MIG_CMD_LOAD 3 +#define IGB_MIG_CMD_DIRTY_ENABLE 4 +#define IGB_MIG_CMD_DIRTY_DISABLE 5 +#define IGB_MIG_CMD_DIRTY_QUERY 6 =20 /* STATUS register: state in [7:0], error code [15:8] */ #define IGB_MIG_STATUS_STATE_MASK 0xFF @@ -66,6 +76,7 @@ #define IGB_MIG_STATE_RUNNING 2 #define IGB_MIG_STATE_STOP_COPY 3 #define IGB_MIG_STATE_RESUMING 4 +#define IGB_MIG_STATE_PRE_COPY 5 =20 /* Error codes */ #define IGB_MIG_ERR_UNK_CMD 1 @@ -75,10 +86,29 @@ #define IGB_MIG_ERR_BAD_SIZE 5 #define IGB_MIG_ERR_BAD_MAGIC 6 #define IGB_MIG_ERR_BAD_VERSION 7 +#define IGB_MIG_ERR_TOO_MANY_RANGES 8 +#define IGB_MIG_ERR_BAD_RANGE 9 +#define IGB_MIG_ERR_BAD_PGSIZE 10 +#define IGB_MIG_ERR_NOT_ENABLED 11 =20 /* Shared buffer constants */ #define IGB_VF_STATE_MAX_SIZE 4096 =20 +#define IGB_MIG_DIRTY_DEFAULT_PGSIZE 4096 + +typedef struct IGBVfDirtyRange { + uint64_t iova; + uint64_t size; + uint64_t page_size; + unsigned long *bitmap; + uint64_t nbits; +} IGBVfDirtyRange; + +typedef struct IGBVfDirtyState { + IGBVfDirtyRange ranges[IGB_MIG_CAPS_MAX_RANGES]; + uint32_t num_ranges; +} IGBVfDirtyState; + typedef struct IgbVfMigState { uint32_t mig_state; uint32_t mig_data[IGB_VF_STATE_MAX_SIZE / sizeof(uint32_t)]; @@ -86,6 +116,36 @@ typedef struct IgbVfMigState { uint64_t mig_data_buf_addr; } IgbVfMigState; =20 +/* + * DMA buffer layouts for dirty tracking commands. + * + * DIRTY_ENABLE: driver writes igb_mig_dirty_enable_req to buffer + * before cmd. + * DIRTY_QUERY: driver writes iova/size fields, device writes + * response + bitmap. + */ +struct igb_mig_dirty_enable_req { + uint32_t len; + uint32_t flags; + uint64_t pgsize; + uint64_t range_iova; + uint64_t range_size; + uint32_t reserved[4]; +}; + +struct igb_mig_dirty_query { + uint32_t len; + uint32_t flags; + uint64_t iova; + uint64_t size; + uint32_t bitmap_size; + uint32_t dirty_page_count; + uint64_t dma_writes; + uint32_t reserved[6]; + uint8_t bitmap[]; +}; + +typedef struct IGBCore IGBCore; typedef struct IgbVfState IgbVfState; =20 bool igbvf_add_migration_dvsec(PCIDevice *dev, Error **errp); @@ -94,4 +154,7 @@ uint32_t igbvf_mig_config_read(IgbVfState *s, uint32_t a= ddr, int size); bool igbvf_mig_config_write(IgbVfState *s, uint32_t addr, uint32_t val, int size); =20 +void igb_core_dirty_track_dma(IGBCore *core, int vfn, + dma_addr_t addr, dma_addr_t len); + #endif diff --git a/hw/net/igb_core.c b/hw/net/igb_core.c index 01745fe756d0..12329a1aff58 100644 --- a/hw/net/igb_core.c +++ b/hw/net/igb_core.c @@ -824,6 +824,16 @@ igb_rx_ring_init(IGBCore *core, E1000E_RxRing *rxr, in= t idx) rxr->i =3D &i[idx]; } =20 +static inline void +igb_pci_dma_write(IGBCore *core, PCIDevice *dev, + dma_addr_t addr, const void *buf, dma_addr_t len) +{ + pci_dma_write(dev, addr, buf, len); + if (pci_is_vf(dev)) { + igb_core_dirty_track_dma(core, pcie_sriov_vf_number(dev), addr, le= n); + } +} + static uint32_t igb_txdesc_writeback(IGBCore *core, dma_addr_t base, union e1000_adv_tx_desc *tx_desc, @@ -847,13 +857,15 @@ igb_txdesc_writeback(IGBCore *core, dma_addr_t base, =20 if (tdwba & 1) { uint32_t buffer =3D cpu_to_le32(core->mac[txi->dh]); - pci_dma_write(d, tdwba & ~3, &buffer, sizeof(buffer)); + igb_pci_dma_write(core, d, + tdwba & ~3, &buffer, sizeof(buffer)); } else { uint32_t status =3D le32_to_cpu(tx_desc->wb.status) | E1000_TXD_ST= AT_DD; =20 tx_desc->wb.status =3D cpu_to_le32(status); - pci_dma_write(d, base + offsetof(union e1000_adv_tx_desc, wb), - &tx_desc->wb, sizeof(tx_desc->wb)); + igb_pci_dma_write(core, d, + base + offsetof(union e1000_adv_tx_desc, wb), + &tx_desc->wb, sizeof(tx_desc->wb)); } =20 return igb_tx_wb_eic(core, txi->idx); @@ -1598,11 +1610,12 @@ igb_pci_dma_write_rx_desc(IGBCore *core, PCIDevice = *dev, dma_addr_t addr, uint8_t status =3D d->status; =20 d->status &=3D ~E1000_RXD_STAT_DD; - pci_dma_write(dev, addr, desc, len); + igb_pci_dma_write(core, dev, addr, desc, len); =20 if (status & E1000_RXD_STAT_DD) { d->status =3D status; - pci_dma_write(dev, addr + offset, &status, sizeof(status)); + igb_pci_dma_write(core, dev, + addr + offset, &status, sizeof(status)); } } else { union e1000_adv_rx_desc *d =3D &desc->adv; @@ -1611,11 +1624,12 @@ igb_pci_dma_write_rx_desc(IGBCore *core, PCIDevice = *dev, dma_addr_t addr, uint32_t status =3D d->wb.upper.status_error; =20 d->wb.upper.status_error &=3D ~E1000_RXD_STAT_DD; - pci_dma_write(dev, addr, desc, len); + igb_pci_dma_write(core, dev, addr, desc, len); =20 if (status & E1000_RXD_STAT_DD) { d->wb.upper.status_error =3D status; - pci_dma_write(dev, addr + offset, &status, sizeof(status)); + igb_pci_dma_write(core, dev, + addr + offset, &status, sizeof(status)); } } } @@ -1737,9 +1751,9 @@ igb_write_hdr_frag_to_rx_buffers(IGBCore *core, { assert(data_len <=3D pdma_st->rx_desc_header_buf_size - pdma_st->bastate.written[0]); - pci_dma_write(d, - pdma_st->ba[0] + pdma_st->bastate.written[0], - data, data_len); + igb_pci_dma_write(core, d, + pdma_st->ba[0] + pdma_st->bastate.written[0], + data, data_len); pdma_st->bastate.written[0] +=3D data_len; pdma_st->bastate.cur_idx =3D 1; } @@ -1804,10 +1818,10 @@ igb_write_payload_frag_to_rx_buffers(IGBCore *core, data, bytes_to_write); =20 - pci_dma_write(d, - pdma_st->ba[pdma_st->bastate.cur_idx] + - pdma_st->bastate.written[pdma_st->bastate.cur_idx], - data, bytes_to_write); + igb_pci_dma_write(core, d, + pdma_st->ba[pdma_st->bastate.cur_idx] + + pdma_st->bastate.written[pdma_st->bastate.cur_id= x], + data, bytes_to_write); =20 pdma_st->bastate.written[pdma_st->bastate.cur_idx] +=3D bytes_to_w= rite; data +=3D bytes_to_write; diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c index af7257ccc4fd..cfa73673e39a 100644 --- a/hw/net/igb_migration.c +++ b/hw/net/igb_migration.c @@ -8,6 +8,8 @@ =20 #include "qemu/osdep.h" #include "qemu/log.h" +#include "qemu/bitmap.h" +#include "qemu/units.h" #include "hw/pci/pci_device.h" #include "hw/pci/pcie.h" #include "net/eth.h" @@ -400,6 +402,285 @@ static int igbvf_mig_load(IgbVfState *s, const void *= buf, size_t size) return 0; } =20 +/* + * Per-VF dirty page tracking + * + * All VF DMA writes in igb_core.c go through igb_pci_dma_write(), + * which calls igb_core_dirty_track_dma() to mark the target page in a + * per-range bitmap before performing the actual DMA. + * + * The IGBCore::vf_dirty[] bitmaps live in IGBCore so they are easily + * accessible from the core TX and RX paths without reaching back into + * VF state. + */ + +void igb_core_dirty_track_dma(IGBCore *core, int vfn, + dma_addr_t addr, dma_addr_t len) +{ + IGBVfDirtyState *ds =3D &core->vf_dirty[vfn]; + bool matched =3D false; + uint32_t i; + + if (!ds->num_ranges) { + return; + } + + trace_igb_core_dirty_track_dma(vfn, addr, len); + + for (i =3D 0; i < ds->num_ranges; i++) { + IGBVfDirtyRange *r =3D &ds->ranges[i]; + uint64_t r_end =3D r->iova + r->size; + uint64_t dma_end =3D addr + len; + uint64_t start, end, start_page, end_page, page; + + if (addr >=3D r_end || dma_end <=3D r->iova) { + continue; + } + + matched =3D true; + start =3D MAX(addr, r->iova); + end =3D MIN(dma_end, r_end); + + start_page =3D (start - r->iova) / r->page_size; + end_page =3D (end - 1 - r->iova) / r->page_size; + + for (page =3D start_page; page <=3D end_page; page++) { + if (page < r->nbits) { + set_bit(page, r->bitmap); + } + } + } + + if (!matched) { + trace_igb_core_dirty_track_dma_drop(vfn, addr, len); + } +} + +static IGBVfDirtyState *igb_core_vf_dirty_state(IgbVfState *s) +{ + IGBCore *core =3D igbvf_get_core(s); + return &core->vf_dirty[s->vfn]; +} + +#define IGB_MIG_DIRTY_MAX_PAGES ((256ULL * GiB) / (4 * KiB)) + +static uint32_t igb_core_vf_dirty_enable(IgbVfState *s, uint64_t pgsize, + uint64_t range_iova, + uint64_t range_size) +{ + uint32_t caps =3D pci_get_long(PCI_DEVICE(s)->config + + IGB_MIG_DVSEC_OFFSET + IGB_MIG_CAPS); + IGBVfDirtyState *ds =3D igb_core_vf_dirty_state(s); + IGBVfDirtyRange *r; + + if (ds->num_ranges >=3D IGB_MIG_CAPS_MAX_RANGES) { + return IGB_MIG_ERR_TOO_MANY_RANGES; + } + + if (!range_size) { + return IGB_MIG_ERR_BAD_RANGE; + } + + /* Validate page size against CAPS supported page size bitmask */ + if (!is_power_of_2(pgsize) || !(pgsize & caps)) { + return IGB_MIG_ERR_BAD_PGSIZE; + } + + if ((range_iova % pgsize) || (range_size % pgsize)) { + return IGB_MIG_ERR_BAD_PGSIZE; + } + + if (range_size / pgsize > IGB_MIG_DIRTY_MAX_PAGES) { + return IGB_MIG_ERR_BAD_RANGE; + } + + r =3D &ds->ranges[ds->num_ranges]; + r->iova =3D range_iova; + r->size =3D range_size; + r->page_size =3D pgsize; + r->nbits =3D range_size / pgsize; + r->bitmap =3D bitmap_new(r->nbits); + ds->num_ranges++; + return 0; +} + +static void igb_core_vf_dirty_disable(IgbVfState *s) +{ + IGBVfDirtyState *ds =3D igb_core_vf_dirty_state(s); + uint32_t i; + + for (i =3D 0; i < ds->num_ranges; i++) { + IGBVfDirtyRange *r =3D &ds->ranges[i]; + + g_free(r->bitmap); + r->bitmap =3D NULL; + r->nbits =3D 0; + } + ds->num_ranges =3D 0; + trace_igbvf_mig_dirty_disable(s->vfn); +} + +static bool igb_core_vf_dirty_enabled(IgbVfState *s) +{ + return igb_core_vf_dirty_state(s)->num_ranges > 0; +} + +static void igb_core_vf_dirty_query(IGBVfDirtyRange *r, + uint64_t range_iova, uint64_t range_si= ze, + void *buf, size_t buf_size, + size_t *out_size) +{ + uint64_t start_page =3D (range_iova - r->iova) / r->page_size; + uint64_t range_pages =3D range_size / r->page_size; + uint64_t count =3D MIN(range_pages, (uint64_t)buf_size * 8); + + memset(buf, 0, buf_size); + + if (start_page < r->nbits) { + uint64_t avail =3D r->nbits - start_page; + uint64_t n =3D MIN(count, avail); + + bitmap_copy_with_src_offset(buf, r->bitmap, start_page, n); + } + *out_size =3D bitmap_empty(buf, count) ? 0 : DIV_ROUND_UP(count, 8); +} + +static void igb_core_vf_dirty_query_commit(IGBVfDirtyRange *r, + uint64_t range_iova, + uint64_t range_size) +{ + uint64_t start_page =3D (range_iova - r->iova) / r->page_size; + + if (start_page < r->nbits) { + uint64_t avail =3D r->nbits - start_page; + uint64_t range_pages =3D range_size / r->page_size; + uint64_t n =3D MIN(range_pages, avail); + + bitmap_clear(r->bitmap, start_page, n); + } +} + +static uint8_t igbvf_mig_cmd_dirty_enable(IgbVfState *s) +{ + IgbVfMigState *ms =3D &s->mig; + struct igb_mig_dirty_enable_req req; + MemTxResult r; + uint32_t status; + + if (!ms->mig_data_buf_addr) { + return IGB_MIG_ERR_NO_BUFFER; + } + + r =3D address_space_read(&address_space_memory, ms->mig_data_buf_addr, + MEMTXATTRS_UNSPECIFIED, &req, sizeof(req)); + if (r !=3D MEMTX_OK) { + return IGB_MIG_ERR_DMA_FAILED; + } + + /* TODO: validate req.len and req.flags */ + + status =3D igb_core_vf_dirty_enable(s, le64_to_cpu(req.pgsize), + le64_to_cpu(req.range_iova), + le64_to_cpu(req.range_size)); + if (status !=3D 0) { + return status; + } + trace_igbvf_mig_dirty_enable(s->vfn, le64_to_cpu(req.pgsize), + le64_to_cpu(req.range_size) / + le64_to_cpu(req.pgsize)); + return 0; +} + +static IGBVfDirtyRange *igb_core_vf_dirty_range_valid(IGBVfDirtyState *ds, + uint64_t range_iova, + uint64_t range_size) +{ + if (!range_size) { + return NULL; + } + + for (uint32_t i =3D 0; i < ds->num_ranges; i++) { + IGBVfDirtyRange *r =3D &ds->ranges[i]; + + if (range_iova >=3D r->iova && + range_iova + range_size <=3D r->iova + r->size && + QEMU_IS_ALIGNED(range_iova, r->page_size) && + QEMU_IS_ALIGNED(range_size, r->page_size)) { + return r; + } + } + return NULL; +} + +static uint8_t igbvf_mig_cmd_dirty_query(IgbVfState *s) +{ + IgbVfMigState *ms =3D &s->mig; + IGBVfDirtyState *ds =3D &igbvf_get_core(s)->vf_dirty[s->vfn]; + uint64_t buf_addr =3D ms->mig_data_buf_addr; + uint64_t range_iova =3D 0, range_size =3D 0; + IGBVfDirtyRange *range; + uint32_t bmp_bytes, dirty_pages; + uint32_t val32; + size_t out_size; + g_autofree void *bitmap =3D NULL; + + if (!buf_addr) { + return IGB_MIG_ERR_NO_BUFFER; + } + + if (!igb_core_vf_dirty_enabled(s)) { + return IGB_MIG_ERR_NOT_ENABLED; + } + + address_space_read(&address_space_memory, + buf_addr + offsetof(struct igb_mig_dirty_query, iov= a), + MEMTXATTRS_UNSPECIFIED, &range_iova, sizeof(range_i= ova)); + range_iova =3D le64_to_cpu(range_iova); + address_space_read(&address_space_memory, + buf_addr + offsetof(struct igb_mig_dirty_query, siz= e), + MEMTXATTRS_UNSPECIFIED, &range_size, sizeof(range_s= ize)); + range_size =3D le64_to_cpu(range_size); + + range =3D igb_core_vf_dirty_range_valid(ds, range_iova, range_size); + if (!range) { + return IGB_MIG_ERR_BAD_RANGE; + } + + bmp_bytes =3D BITS_TO_LONGS(range_size / range->page_size) * + sizeof(unsigned long); + bitmap =3D g_malloc0(bmp_bytes); + + igb_core_vf_dirty_query(range, range_iova, range_size, + bitmap, bmp_bytes, &out_size); + + if (out_size) { + if (address_space_write(&address_space_memory, + buf_addr + + offsetof(struct igb_mig_dirty_query, bitma= p), + MEMTXATTRS_UNSPECIFIED, bitmap, out_size))= { + return IGB_MIG_ERR_DMA_FAILED; + } + } + + igb_core_vf_dirty_query_commit(range, range_iova, range_size); + + dirty_pages =3D bitmap_count_one(bitmap, range_size / range->page_size= ); + + val32 =3D cpu_to_le32(out_size); + address_space_write(&address_space_memory, + buf_addr + offsetof(struct igb_mig_dirty_query, + bitmap_size), + MEMTXATTRS_UNSPECIFIED, &val32, sizeof(val32)); + val32 =3D cpu_to_le32(dirty_pages); + address_space_write(&address_space_memory, + buf_addr + offsetof(struct igb_mig_dirty_query, + dirty_page_count), + MEMTXATTRS_UNSPECIFIED, &val32, sizeof(val32)); + + trace_igbvf_mig_dirty_query(s->vfn, (uint64_t)out_size, dirty_pages); + return 0; +} + /* * Migration command handlers */ @@ -419,7 +700,8 @@ static uint8_t igbvf_mig_cmd_save(IgbVfState *s) MemTxResult r; int ret; =20 - if (ms->mig_state !=3D IGB_MIG_STATE_STOP_COPY) { + if (ms->mig_state !=3D IGB_MIG_STATE_STOP_COPY && + ms->mig_state !=3D IGB_MIG_STATE_PRE_COPY) { return IGB_MIG_ERR_BAD_STATE; } =20 @@ -487,22 +769,33 @@ static uint8_t igbvf_mig_set_state(IgbVfState *s, uin= t32_t new_state) case IGB_MIG_STATE_STOP: if (old !=3D IGB_MIG_STATE_RUNNING && old !=3D IGB_MIG_STATE_STOP_COPY && + old !=3D IGB_MIG_STATE_PRE_COPY && old !=3D IGB_MIG_STATE_RESUMING && old !=3D IGB_MIG_STATE_ERROR) { return IGB_MIG_ERR_BAD_STATE; } + if (old =3D=3D IGB_MIG_STATE_PRE_COPY || + old =3D=3D IGB_MIG_STATE_STOP_COPY || + old =3D=3D IGB_MIG_STATE_ERROR) { + igb_core_vf_dirty_disable(s); + } /* Restore DATA_SIZE to max, same as at reset */ igbvf_mig_update_data_size(s, igb_core_vf_max_data_size(s)); break; =20 case IGB_MIG_STATE_RUNNING: - if (old !=3D IGB_MIG_STATE_STOP) { + if (old !=3D IGB_MIG_STATE_STOP && + old !=3D IGB_MIG_STATE_PRE_COPY) { return IGB_MIG_ERR_BAD_STATE; } + if (old =3D=3D IGB_MIG_STATE_PRE_COPY) { + igb_core_vf_dirty_disable(s); + } break; =20 case IGB_MIG_STATE_STOP_COPY: - if (old !=3D IGB_MIG_STATE_STOP) { + if (old !=3D IGB_MIG_STATE_STOP && + old !=3D IGB_MIG_STATE_PRE_COPY) { return IGB_MIG_ERR_BAD_STATE; } ret =3D igb_core_vf_save_state(s, ms->mig_data, sizeof(ms->mig_dat= a)); @@ -520,6 +813,12 @@ static uint8_t igbvf_mig_set_state(IgbVfState *s, uint= 32_t new_state) igbvf_mig_update_data_size(s, 0); break; =20 + case IGB_MIG_STATE_PRE_COPY: + if (old !=3D IGB_MIG_STATE_RUNNING) { + return IGB_MIG_ERR_BAD_STATE; + } + break; + default: return IGB_MIG_ERR_BAD_STATE; } @@ -564,6 +863,18 @@ static void igbvf_mig_cmd_ctrl(IgbVfState *s, uint32_t= val) err =3D igbvf_mig_cmd_load(s); break; =20 + case IGB_MIG_CMD_DIRTY_ENABLE: + err =3D igbvf_mig_cmd_dirty_enable(s); + break; + + case IGB_MIG_CMD_DIRTY_DISABLE: + igb_core_vf_dirty_disable(s); + break; + + case IGB_MIG_CMD_DIRTY_QUERY: + err =3D igbvf_mig_cmd_dirty_query(s); + break; + default: err =3D IGB_MIG_ERR_UNK_CMD; break; @@ -594,8 +905,10 @@ bool igbvf_add_migration_dvsec(PCIDevice *dev, Error *= *errp) /* DVSEC header 2: DVSEC ID */ pci_set_word(dev->config + offset + 0x8, IGB_MIG_DVSEC_ID); =20 - /* CAPS: features (state migration only) */ - caps =3D IGB_MIG_CAP_F_STATE; + /* CAPS: features | max_ranges | supported page sizes (4K) */ + caps =3D IGB_MIG_CAP_F_STATE | IGB_MIG_CAP_F_DIRTY | + (IGB_MIG_CAPS_MAX_RANGES << IGB_MIG_CAPS_MAX_RANGES_SHIFT) | + IGB_MIG_CAPS_PGSIZE_4K; pci_set_long(dev->config + offset + IGB_MIG_CAPS, caps); =20 /* STATUS: initial state is RUNNING */ @@ -657,6 +970,9 @@ void igbvf_mig_state_reset(IgbVfState *s) IgbVfMigState *ms =3D &s->mig; =20 trace_igbvf_mig_reset(s->vfn); + + igb_core_vf_dirty_disable(s); + ms->mig_state =3D IGB_MIG_STATE_RUNNING; ms->mig_data_buf_addr =3D 0; igbvf_mig_update_data_size(s, igb_core_vf_max_data_size(s)); diff --git a/hw/net/trace-events b/hw/net/trace-events index 7057fbe5f16a..c606191df2ef 100644 --- a/hw/net/trace-events +++ b/hw/net/trace-events @@ -300,6 +300,11 @@ igbvf_mig_set_state(uint16_t vfn, uint32_t old_state, = uint32_t new_state) "VF%u: igbvf_mig_save_state(uint16_t vfn, uint32_t size) "VF%u: saved %u bytes of= device state" igbvf_mig_load_state(uint16_t vfn, uint32_t size) "VF%u: loaded %u bytes o= f device state" igbvf_mig_reset(uint16_t vfn) "VF%u: migration state reset" +igbvf_mig_dirty_enable(uint16_t vfn, uint64_t pgsize, uint64_t nbits) "VF%= u: dirty tracking enabled pgsize=3D%"PRIu64" nbits=3D%"PRIu64 +igbvf_mig_dirty_disable(uint16_t vfn) "VF%u: dirty tracking disabled" +igbvf_mig_dirty_query(uint16_t vfn, uint64_t size, uint32_t dirty_pages) "= VF%u: dirty query returned %"PRIu64" bytes, %u dirty pages" +igb_core_dirty_track_dma(int vfn, uint64_t addr, uint64_t len) "VF%d: dirt= y DMA addr=3D0x%"PRIx64" len=3D%"PRIu64 +igb_core_dirty_track_dma_drop(int vfn, uint64_t addr, uint64_t len) "VF%d:= dirty DMA dropped addr=3D0x%"PRIx64" len=3D%"PRIu64" no matching range" =20 # spapr_llan.c spapr_vlan_get_rx_bd_from_pool_found(int pool, int32_t count, uint32_t rx_= bufs) "pool=3D%d count=3D%"PRId32" rxbufs=3D%"PRIu32 --=20 2.55.0 From nobody Sat Sep 26 22:16:24 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1788376920; cv=none; d=zohomail.com; s=zohoarc; b=HEEpZTtObCqfNvi3H3IapqToZPe090xbgpQ4ulTmLBB0bu43EYdgXg4uanxXLW/eDNYzP+mfm2EepcZliF1ndu5+H76jgEL4ntXfqPvQVK4VIWwb+DUjaUTZ+laQb1x1AbVf5QG9y2klKlTrzWL21kx1CqcDhiQCisDSaJfEqws= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1788376920; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=B2cjxw29/2X2+wcUsPPPggkx+sFbB/T9xSUUV0h+cG0=; b=OjJUCY7xxmxKfOfXdKV0cCyjfu58R6xS8Hk4PyvYOo3yEM78SKZ94EXfGRh9B9esvso9PcwFeetvZ9I2CEgVVlFdUKzkHrROXJWw7O582njoBGtQutOIPpSvYmwPqmqlZ1xe+O9E3EhumgC9DXhHn9aEK5DnWaGWfZ9inEgff+w= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1788376920186992.982199075862; Wed, 2 Sep 2026 12:22:00 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x1qWY-0004Kz-Uj; Wed, 02 Sep 2026 15:21:30 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x1qWV-0004KQ-7s for qemu-devel@nongnu.org; Wed, 02 Sep 2026 15:21:27 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x1qWS-00027w-1N for qemu-devel@nongnu.org; Wed, 02 Sep 2026 15:21:26 -0400 Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-711-U7eqF4t_OHqX2d_q07Y_BA-1; Wed, 02 Sep 2026 15:21:20 -0400 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id EB5A019540C1; Wed, 2 Sep 2026 19:21:18 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.32.5]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 5D5163000DA3; Wed, 2 Sep 2026 19:21:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1788376883; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=B2cjxw29/2X2+wcUsPPPggkx+sFbB/T9xSUUV0h+cG0=; b=FB5ShBmeOrHNRP2MM7m2Qdo2tuyFkmqIln9ysHnSN4pkvUItPcV8FXqqMQA4/bL1qGtn1X 7VE3oK5aHDaCUiRJBlfdftGMDTqk0i3cD6fpkV+jwzRuJVkhI9vrVc39u8CThxn+NNIpP3 D8y0MKkBK+5+y4Ua90U6mHyEunp4/Bk= X-MC-Unique: U7eqF4t_OHqX2d_q07Y_BA-1 X-Mimecast-MFC-AGG-ID: U7eqF4t_OHqX2d_q07Y_BA_1788376879 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Akihiko Odaki , Sriram Yagnaraman , Jason Wang , Alex Williamson , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [RFC PATCH v2 6/9] igb: Quiesce VFs on STOP and include PF enable state in migration Date: Wed, 2 Sep 2026 21:20:51 +0200 Message-ID: <20260902192054.3329753-7-clg@redhat.com> In-Reply-To: <20260902192054.3329753-1-clg@redhat.com> References: <20260902192054.3329753-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=170.10.129.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: 12 X-Spam_score: 1.2 X-Spam_bar: + X-Spam_report: (1.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=0.001, RCVD_IN_SBL_CSS=3.335, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1788376922243154100 Quiesce VFs by clearing their VFRE/VFTE bits on transitions to STOP (from RUNNING, PRE_COPY, or ERROR) and to STOP_COPY (from PRE_COPY). This prevents further DMA while device state is being serialized. Restore the saved VFRE/VFTE state on STOP->RUNNING so the VF can resume normal operation. Include the per-VF VFRE/VFTE enable bits in the migration blob so the destination knows whether receive and transmit were active before quiesce. On reset, both default to true. Add a QUIESCED bit to the STATUS register so the driver can confirm DMA has been drained before reading device state. Clear the VFLRE (VF Level Reset Event) bit before restoring state on the destination to prevent the PF watchdog from seeing a stale reset indication and overwriting the loaded registers. AI-used-for: analysis, code (prototype) Signed-off-by: C=C3=A9dric Le Goater --- hw/net/igb_migration.h | 6 ++- hw/net/igb_migration.c | 86 +++++++++++++++++++++++++++++++++++++++++- hw/net/trace-events | 6 ++- 3 files changed, 93 insertions(+), 5 deletions(-) diff --git a/hw/net/igb_migration.h b/hw/net/igb_migration.h index 996de2d2f50b..2bb9a0ed36ce 100644 --- a/hw/net/igb_migration.h +++ b/hw/net/igb_migration.h @@ -24,7 +24,8 @@ * +0x0C CAPS (RO: F_STATE[0], F_DIRTY[1], * max_ranges[11:8], pgsize[16:12]) * +0x10 CTRL (WO: doorbell command) - * +0x14 STATUS (RO: state[7:0], error_code[15:8]) + * +0x14 STATUS (RO: state[7:0], error_code[15:8], + * QUIESCED[16]) * +0x18 BUF_ADDR_LO (RW: shared buffer GPA low) * +0x1C BUF_ADDR_HI (RW: shared buffer GPA high) * +0x20 DATA_SIZE (RO: max state blob size in bytes) @@ -69,6 +70,7 @@ #define IGB_MIG_STATUS_ERROR_CODE_SHIFT 8 #define IGB_MIG_STATUS_ERR(code) \ ((uint32_t)(code) << IGB_MIG_STATUS_ERROR_CODE_SHIFT) +#define IGB_MIG_STATUS_QUIESCED (1u << 16) =20 /* Device states (based on VFIO migration v2) */ #define IGB_MIG_STATE_ERROR 0 @@ -114,6 +116,8 @@ typedef struct IgbVfMigState { uint32_t mig_data[IGB_VF_STATE_MAX_SIZE / sizeof(uint32_t)]; uint32_t mig_data_size; uint64_t mig_data_buf_addr; + bool mig_saved_vfre; + bool mig_saved_vfte; } IgbVfMigState; =20 /* diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c index cfa73673e39a..435bd0d1623b 100644 --- a/hw/net/igb_migration.c +++ b/hw/net/igb_migration.c @@ -58,6 +58,8 @@ typedef struct IgbMigBlob { IgbMigRegPair ra[IGB_VF_MAX_RA_REGS]; uint32_t num_tx_ctx; IgbMigTxCtx tx_ctx[2]; + uint32_t vfre; + uint32_t vfte; } IgbMigBlob; =20 #define IGB_MIG_BLOB_SIZE sizeof(IgbMigBlob) @@ -210,6 +212,7 @@ static void igb_core_vf_save_tx_ctx(IGBCore *core, int = queue, static int igb_core_vf_save_state(IgbVfState *s, void *buf, size_t buf_siz= e) { int size =3D IGB_MIG_BLOB_SIZE; + IgbVfMigState *ms =3D &s->mig; IGBCore *core =3D igbvf_get_core(s); IgbMigBlob *blob =3D buf; uint32_t offsets[IGB_VF_MAX_FIXED_REGS]; @@ -248,7 +251,12 @@ static int igb_core_vf_save_state(IgbVfState *s, void = *buf, size_t buf_size) igb_core_vf_save_tx_ctx(core, q0, &blob->tx_ctx[0]); igb_core_vf_save_tx_ctx(core, q1, &blob->tx_ctx[1]); =20 - trace_igbvf_mig_save_state(s->vfn, size); + blob->vfre =3D cpu_to_le32(ms->mig_saved_vfre); + blob->vfte =3D cpu_to_le32(ms->mig_saved_vfte); + + trace_igbvf_mig_save_state(s->vfn, size, ms->mig_saved_vfre, + ms->mig_saved_vfte, + core->mac[VFRE]); return size; } =20 @@ -291,6 +299,7 @@ static uint32_t igb_vf_relocate_offset(uint32_t offset, =20 static int igb_core_vf_load_state(IgbVfState *s, const void *buf, size_t s= ize) { + IgbVfMigState *ms =3D &s->mig; IGBCore *core =3D igbvf_get_core(s); uint32_t src_offsets[IGB_VF_MAX_FIXED_REGS]; uint32_t dst_offsets[IGB_VF_MAX_FIXED_REGS]; @@ -379,7 +388,12 @@ static int igb_core_vf_load_state(IgbVfState *s, const= void *buf, size_t size) igb_core_vf_load_tx_ctx(core, q0, &blob->tx_ctx[0]); igb_core_vf_load_tx_ctx(core, q1, &blob->tx_ctx[1]); =20 - trace_igbvf_mig_load_state(s->vfn, (uint32_t)size); + ms->mig_saved_vfre =3D !!le32_to_cpu(blob->vfre); + ms->mig_saved_vfte =3D !!le32_to_cpu(blob->vfte); + + trace_igbvf_mig_load_state(s->vfn, (uint32_t)size, + ms->mig_saved_vfre, + ms->mig_saved_vfte); return 0; } =20 @@ -388,6 +402,12 @@ static int igbvf_mig_load(IgbVfState *s, const void *b= uf, size_t size) int ret; IGBCore *core =3D igbvf_get_core(s); =20 + /* + * Pre-load: Clear the VFLRE bit before restoring state so the PF + * watchdog does not overwrite what we are about to load. + */ + core->mac[VFLRE] &=3D ~BIT(s->vfn); + ret =3D igb_core_vf_load_state(s, buf, size); if (ret < 0) { return ret; @@ -759,6 +779,45 @@ static uint8_t igbvf_mig_cmd_load(IgbVfState *s) return 0; } =20 +/* Quiesce a VF by disabling its RX and TX at the PF level. */ +static void igb_core_vf_quiesce(IgbVfState *s) +{ + IgbVfMigState *ms =3D &s->mig; + IGBCore *core =3D igbvf_get_core(s); + + ms->mig_saved_vfre =3D !!(core->mac[VFRE] & BIT(s->vfn)); + ms->mig_saved_vfte =3D !!(core->mac[VFTE] & BIT(s->vfn)); + + core->mac[VFRE] &=3D ~BIT(s->vfn); + core->mac[VFTE] &=3D ~BIT(s->vfn); + trace_igbvf_mig_quiesce(s->vfn, core->mac[VFRE], core->mac[VFTE]); +} + +static void igb_core_vf_unquiesce(IgbVfState *s) +{ + IgbVfMigState *ms =3D &s->mig; + IGBCore *core =3D igbvf_get_core(s); + bool re =3D ms->mig_saved_vfre; + bool te =3D ms->mig_saved_vfte; + + if (re) { + core->mac[VFRE] |=3D BIT(s->vfn); + } else { + core->mac[VFRE] &=3D ~BIT(s->vfn); + } + if (te) { + core->mac[VFTE] |=3D BIT(s->vfn); + } else { + core->mac[VFTE] &=3D ~BIT(s->vfn); + } + + trace_igbvf_mig_unquiesce(s->vfn, core->mac[VFRE], core->mac[VFTE]); + + if (re) { + igb_start_recv(core); + } +} + static uint8_t igbvf_mig_set_state(IgbVfState *s, uint32_t new_state) { IgbVfMigState *ms =3D &s->mig; @@ -779,6 +838,11 @@ static uint8_t igbvf_mig_set_state(IgbVfState *s, uint= 32_t new_state) old =3D=3D IGB_MIG_STATE_ERROR) { igb_core_vf_dirty_disable(s); } + if (old =3D=3D IGB_MIG_STATE_RUNNING || + old =3D=3D IGB_MIG_STATE_PRE_COPY || + old =3D=3D IGB_MIG_STATE_ERROR) { + igb_core_vf_quiesce(s); + } /* Restore DATA_SIZE to max, same as at reset */ igbvf_mig_update_data_size(s, igb_core_vf_max_data_size(s)); break; @@ -791,6 +855,9 @@ static uint8_t igbvf_mig_set_state(IgbVfState *s, uint3= 2_t new_state) if (old =3D=3D IGB_MIG_STATE_PRE_COPY) { igb_core_vf_dirty_disable(s); } + if (old =3D=3D IGB_MIG_STATE_STOP) { + igb_core_vf_unquiesce(s); + } break; =20 case IGB_MIG_STATE_STOP_COPY: @@ -798,6 +865,9 @@ static uint8_t igbvf_mig_set_state(IgbVfState *s, uint3= 2_t new_state) old !=3D IGB_MIG_STATE_PRE_COPY) { return IGB_MIG_ERR_BAD_STATE; } + if (old =3D=3D IGB_MIG_STATE_PRE_COPY) { + igb_core_vf_quiesce(s); + } ret =3D igb_core_vf_save_state(s, ms->mig_data, sizeof(ms->mig_dat= a)); if (ret < 0) { return -ret; @@ -840,6 +910,16 @@ static void igbvf_mig_update_status(IgbVfState *s, uin= t8_t err) status =3D IGB_MIG_STATE_ERROR | IGB_MIG_STATUS_ERR(err); } =20 + /* + * QUIESCED tells the driver it is safe to read device state. + * In STOP and STOP_COPY, igb_core_vf_quiesce() has already + * cleared VFRE/VFTE so no further VF DMA can occur. + */ + if (ms->mig_state =3D=3D IGB_MIG_STATE_STOP || + ms->mig_state =3D=3D IGB_MIG_STATE_STOP_COPY) { + status |=3D IGB_MIG_STATUS_QUIESCED; + } + pci_set_long(dev->config + IGB_MIG_DVSEC_OFFSET + IGB_MIG_STATUS, stat= us); } =20 @@ -977,6 +1057,8 @@ void igbvf_mig_state_reset(IgbVfState *s) ms->mig_data_buf_addr =3D 0; igbvf_mig_update_data_size(s, igb_core_vf_max_data_size(s)); memset(ms->mig_data, 0, sizeof(ms->mig_data)); + ms->mig_saved_vfre =3D true; + ms->mig_saved_vfte =3D true; =20 pci_set_long(PCI_DEVICE(s)->config + IGB_MIG_DVSEC_OFFSET + IGB_MIG_BUF_ADDR_LO, 0); diff --git a/hw/net/trace-events b/hw/net/trace-events index c606191df2ef..eb5c61a8f6fb 100644 --- a/hw/net/trace-events +++ b/hw/net/trace-events @@ -297,14 +297,16 @@ igbvf_wrn_io_addr_unknown(uint64_t addr) "IO unknown = register 0x%"PRIx64 =20 # igb_migration.c igbvf_mig_set_state(uint16_t vfn, uint32_t old_state, uint32_t new_state) = "VF%u: state %u -> %u" -igbvf_mig_save_state(uint16_t vfn, uint32_t size) "VF%u: saved %u bytes of= device state" -igbvf_mig_load_state(uint16_t vfn, uint32_t size) "VF%u: loaded %u bytes o= f device state" +igbvf_mig_save_state(uint16_t vfn, int size, bool vfre, bool vfte, uint32_= t reg_vfre) "VF%u: saved %d bytes vfre=3D%d vfte=3D%d VFRE=3D0x%x" +igbvf_mig_load_state(uint16_t vfn, uint32_t size, bool vfre, bool vfte) "V= F%u: loaded %u bytes vfre=3D%d vfte=3D%d" igbvf_mig_reset(uint16_t vfn) "VF%u: migration state reset" igbvf_mig_dirty_enable(uint16_t vfn, uint64_t pgsize, uint64_t nbits) "VF%= u: dirty tracking enabled pgsize=3D%"PRIu64" nbits=3D%"PRIu64 igbvf_mig_dirty_disable(uint16_t vfn) "VF%u: dirty tracking disabled" igbvf_mig_dirty_query(uint16_t vfn, uint64_t size, uint32_t dirty_pages) "= VF%u: dirty query returned %"PRIu64" bytes, %u dirty pages" igb_core_dirty_track_dma(int vfn, uint64_t addr, uint64_t len) "VF%d: dirt= y DMA addr=3D0x%"PRIx64" len=3D%"PRIu64 igb_core_dirty_track_dma_drop(int vfn, uint64_t addr, uint64_t len) "VF%d:= dirty DMA dropped addr=3D0x%"PRIx64" len=3D%"PRIu64" no matching range" +igbvf_mig_quiesce(uint16_t vfn, uint32_t vfre, uint32_t vfte) "VF%u: quies= ce VFRE=3D0x%x VFTE=3D0x%x" +igbvf_mig_unquiesce(uint16_t vfn, uint32_t vfre, uint32_t vfte) "VF%u: unq= uiesce VFRE=3D0x%x VFTE=3D0x%x" =20 # spapr_llan.c spapr_vlan_get_rx_bd_from_pool_found(int pool, int32_t count, uint32_t rx_= bufs) "pool=3D%d count=3D%"PRId32" rxbufs=3D%"PRIu32 --=20 2.55.0 From nobody Sat Sep 26 22:16:24 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1788376916; cv=none; d=zohomail.com; s=zohoarc; b=EgXy+jUKTH7soDIUFWxd+lMP6IK2ph95iowwiUTKRIMo+C8CsLs5zKfv7lSO+Gxpt4gdqMLaIqTUuxi5mzrvCMAloRFTzTFpFH0D4yvdktPX0nKa+LhD7Uh4eiTLdDYar4VN3Vq2SETwWrltdB8eN86oVzmrCHTrPC+/W/v++b4= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1788376916; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=6ZSKL6HWR0rPEMkkB/ou9RK4eePiGyNILtJrIWNEFfo=; b=R14BmRyxjyWgJ9nFLQRBouGyfD0l+K3BGTkiG8oDzgVoqlC4u2fl0ANfDZREYmoHJaIBsGhuFbE1GixBDresKQBhzceDmct+zN9AvJ0H29M4TEWO8vqQ7khGpHF7NN7FsRLdhwkyTwCZxqJs8aY4/2ffLgmKwLEKN2Fb7AnWV4k= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1788376916238906.036873598014; Wed, 2 Sep 2026 12:21:56 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x1qWc-0004Lh-E6; Wed, 02 Sep 2026 15:21:34 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x1qWX-0004Kr-Bs for qemu-devel@nongnu.org; Wed, 02 Sep 2026 15:21:29 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x1qWV-0002Az-R4 for qemu-devel@nongnu.org; Wed, 02 Sep 2026 15:21:29 -0400 Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-642-okXES1p9PJWrsEI6hrI1yA-1; Wed, 02 Sep 2026 15:21:22 -0400 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 7CA0B19560A2; Wed, 2 Sep 2026 19:21:21 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.32.5]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 633E93000DA3; Wed, 2 Sep 2026 19:21:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1788376887; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6ZSKL6HWR0rPEMkkB/ou9RK4eePiGyNILtJrIWNEFfo=; b=SjJa7ug0KIJ1mojy8XWAaS8oT6NfTkGXUNYOTQ4DvPWfcmfey/uZP/m73w4Shtlwprf9Hc G1eXeK0ENpELx6K5Pi2TdmRfBvgpuEMWe1hpvcVzVsFpt798c0IDfcrLungkYFwSTNwNFQ qT4Ls1Uu5m0SxFhafi3CLivJdbskTF0= X-MC-Unique: okXES1p9PJWrsEI6hrI1yA-1 X-Mimecast-MFC-AGG-ID: okXES1p9PJWrsEI6hrI1yA_1788376881 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Akihiko Odaki , Sriram Yagnaraman , Jason Wang , Alex Williamson , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [RFC PATCH v2 7/9] igb: Fix post-migration RX ring deadlock Date: Wed, 2 Sep 2026 21:20:52 +0200 Message-ID: <20260902192054.3329753-8-clg@redhat.com> In-Reply-To: <20260902192054.3329753-1-clg@redhat.com> References: <20260902192054.3329753-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: 12 X-Spam_score: 1.2 X-Spam_bar: + X-Spam_report: (1.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_SBL_CSS=3.335, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1788376919372158500 After restoring VFRE/VFTE on the destination, the VF's RX rings may be stalled because the guest driver is waiting for an interrupt that was pending at the time of migration. Re-raise any pending EICR causes for the VF (both RX and TX) via igb_set_eics() to break the deadlock. AI-used-for: analysis, code (prototype) Signed-off-by: C=C3=A9dric Le Goater --- hw/net/igb_core.h | 1 + hw/net/igb_core.c | 16 ++++++++++++++++ hw/net/igb_migration.c | 4 ++-- 3 files changed, 19 insertions(+), 2 deletions(-) diff --git a/hw/net/igb_core.h b/hw/net/igb_core.h index 7fdc41690b36..7d6f44b7ffe9 100644 --- a/hw/net/igb_core.h +++ b/hw/net/igb_core.h @@ -150,6 +150,7 @@ igb_start_recv(IGBCore *core); IGBCore *igb_pf_get_core(void *pf); =20 void igb_core_vf_propagate_irqs(IGBCore *core, uint16_t vfn); +void igb_core_vf_rearm_irqs(IGBCore *core, uint16_t vfn); void igb_core_vf_propagate_ivar(IGBCore *core, uint16_t vfn); =20 #endif diff --git a/hw/net/igb_core.c b/hw/net/igb_core.c index 12329a1aff58..1de8d95102e4 100644 --- a/hw/net/igb_core.c +++ b/hw/net/igb_core.c @@ -4632,3 +4632,19 @@ void igb_core_vf_propagate_ivar(IGBCore *core, uint1= 6_t vfn) core->mac[IVAR0 + n / 4] &=3D ~mask; } } + +/* + * Re-apply VF interrupt enables to PF aggregates and raise the + * pending causes so the guest driver resumes polling after migration. + */ +void igb_core_vf_rearm_irqs(IGBCore *core, uint16_t vfn) +{ + uint32_t shift =3D 22 - vfn * IGBVF_MSIX_VEC_NUM; + uint32_t pvt_idx =3D PVTEICR0 + vfn * 0x40; + uint32_t causes =3D (core->mac[pvt_idx] & 0x7) << shift; + + igb_core_vf_propagate_irqs(core, vfn); + if (causes) { + igb_set_eics(core, EICS, causes); + } +} diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c index 435bd0d1623b..46299ff77ab9 100644 --- a/hw/net/igb_migration.c +++ b/hw/net/igb_migration.c @@ -813,8 +813,8 @@ static void igb_core_vf_unquiesce(IgbVfState *s) =20 trace_igbvf_mig_unquiesce(s->vfn, core->mac[VFRE], core->mac[VFTE]); =20 - if (re) { - igb_start_recv(core); + if (re || te) { + igb_core_vf_rearm_irqs(core, s->vfn); } } =20 --=20 2.55.0 From nobody Sat Sep 26 22:16:24 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1788376959; cv=none; d=zohomail.com; s=zohoarc; b=WmqZuesBOmsJXlubWvON/I6I8GZ1Pw90X7/ev8y0z52o2gflhqJLMSexjFdQAXiv8qp0MeytoQLh5pozMq6nCQgzkL3b4ggXxcvl06BUz1ifj97mH5njtSuOkNmHTe7zTc4MNeGqq1jPoQn92dSm8O4cYlL+OKaiBUXAkU2NmbY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1788376959; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=KqAuP8/AUKso6DNaTPVzD0YdIlJ4B9ZHtCScu1u/1Ls=; b=CpDg0MCHa/2JKpT/3gbe99QSOBOMuktNRZPZiqPP5cxY2Nnep4HaJ2DH14YIpyWb6tprq6ffTD5QckkdgseLLnE5y7nQ0UhO9om/WITR0071HfQepuO5dj7AeTEkK+HhXopTam6k/clRRE6W4sltS7/0W9gEdHp7wZs/+gBuAYk= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1788376959920126.28887294774279; Wed, 2 Sep 2026 12:22:39 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x1qWd-0004MG-8R; Wed, 02 Sep 2026 15:21:35 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x1qWb-0004Li-2A for qemu-devel@nongnu.org; Wed, 02 Sep 2026 15:21:34 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x1qWY-0002Bz-TE for qemu-devel@nongnu.org; Wed, 02 Sep 2026 15:21:32 -0400 Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-590-E4QF9d7KOMqpbu34inp1dw-1; Wed, 02 Sep 2026 15:21:25 -0400 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 109361801315; Wed, 2 Sep 2026 19:21:24 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.32.5]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id EA4DB3000DA3; Wed, 2 Sep 2026 19:21:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1788376889; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KqAuP8/AUKso6DNaTPVzD0YdIlJ4B9ZHtCScu1u/1Ls=; b=Tc16akwaUDqi0YYpQ+GWj5qwF5TDLu+zAfvj4jN2DnggJK33re3APGI7O3UXLrLcmbjuMf WW7dCXQmW+exyjq7ATw7hYqj/Vvc1NDznAXdUC8IjSGBMgTKrtBbWl1A9VO1MSJgyQMrNz FJSywiZS/bePB2yvg+KiUva7SG2Wf4Q= X-MC-Unique: E4QF9d7KOMqpbu34inp1dw-1 X-Mimecast-MFC-AGG-ID: E4QF9d7KOMqpbu34inp1dw_1788376884 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Akihiko Odaki , Sriram Yagnaraman , Jason Wang , Alex Williamson , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [RFC PATCH v2 8/9] igb: Add dirty page tracking statistics Date: Wed, 2 Sep 2026 21:20:53 +0200 Message-ID: <20260902192054.3329753-9-clg@redhat.com> In-Reply-To: <20260902192054.3329753-1-clg@redhat.com> References: <20260902192054.3329753-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1788376962822154100 Add a GET_STATS command (cmd 7) to the migration DVSEC for monitoring dirty page tracking and DMA activity per VF. The device DMA-writes an igb_mig_stats_resp struct to the shared buffer. The dma_writes counter is also reported in the dirty query buffer so the driver gets it alongside the bitmap without an extra config read. Counters are reset on first DIRTY_ENABLE or device reset, so the driver can read final values after DIRTY_DISABLE. AI-used-for: code (prototype) Signed-off-by: C=C3=A9dric Le Goater --- hw/net/igb_core.h | 1 + hw/net/igb_migration.h | 22 ++++++++++++++ hw/net/igb_migration.c | 65 ++++++++++++++++++++++++++++++++++++++++-- hw/net/trace-events | 2 +- 4 files changed, 86 insertions(+), 4 deletions(-) diff --git a/hw/net/igb_core.h b/hw/net/igb_core.h index 7d6f44b7ffe9..380cc88b7959 100644 --- a/hw/net/igb_core.h +++ b/hw/net/igb_core.h @@ -103,6 +103,7 @@ struct IGBCore { int64_t timadj; =20 IGBVfDirtyState vf_dirty[IGB_MAX_VF_FUNCTIONS]; + IgbVfMigStats vf_mig_stats[IGB_MAX_VF_FUNCTIONS]; }; =20 void diff --git a/hw/net/igb_migration.h b/hw/net/igb_migration.h index 2bb9a0ed36ce..b334bb8f78ba 100644 --- a/hw/net/igb_migration.h +++ b/hw/net/igb_migration.h @@ -64,6 +64,7 @@ #define IGB_MIG_CMD_DIRTY_ENABLE 4 #define IGB_MIG_CMD_DIRTY_DISABLE 5 #define IGB_MIG_CMD_DIRTY_QUERY 6 +#define IGB_MIG_CMD_GET_STATS 7 =20 /* STATUS register: state in [7:0], error code [15:8] */ #define IGB_MIG_STATUS_STATE_MASK 0xFF @@ -111,6 +112,15 @@ typedef struct IGBVfDirtyState { uint32_t num_ranges; } IGBVfDirtyState; =20 +typedef struct IgbVfMigStats { + uint64_t dma_writes; + uint64_t dma_bytes; + uint32_t dirty_pages_set; + uint32_t dirty_pages_cleared; + uint32_t dirty_page_count; + uint32_t dirty_query_count; +} IgbVfMigStats; + typedef struct IgbVfMigState { uint32_t mig_state; uint32_t mig_data[IGB_VF_STATE_MAX_SIZE / sizeof(uint32_t)]; @@ -149,6 +159,18 @@ struct igb_mig_dirty_query { uint8_t bitmap[]; }; =20 +/* + * GET_STATS: device writes igb_mig_stats_resp to buffer. + */ +struct igb_mig_stats_resp { + uint64_t dma_writes; + uint64_t dma_bytes; + uint32_t dirty_pages_set; + uint32_t dirty_pages_cleared; + uint32_t dirty_page_count; + uint32_t dirty_query_count; +}; + typedef struct IGBCore IGBCore; typedef struct IgbVfState IgbVfState; =20 diff --git a/hw/net/igb_migration.c b/hw/net/igb_migration.c index 46299ff77ab9..2504d5bd308b 100644 --- a/hw/net/igb_migration.c +++ b/hw/net/igb_migration.c @@ -438,6 +438,7 @@ void igb_core_dirty_track_dma(IGBCore *core, int vfn, dma_addr_t addr, dma_addr_t len) { IGBVfDirtyState *ds =3D &core->vf_dirty[vfn]; + IgbVfMigStats *stats =3D &core->vf_mig_stats[vfn]; bool matched =3D false; uint32_t i; =20 @@ -447,6 +448,9 @@ void igb_core_dirty_track_dma(IGBCore *core, int vfn, =20 trace_igb_core_dirty_track_dma(vfn, addr, len); =20 + stats->dma_writes++; + stats->dma_bytes +=3D len; + for (i =3D 0; i < ds->num_ranges; i++) { IGBVfDirtyRange *r =3D &ds->ranges[i]; uint64_t r_end =3D r->iova + r->size; @@ -466,7 +470,10 @@ void igb_core_dirty_track_dma(IGBCore *core, int vfn, =20 for (page =3D start_page; page <=3D end_page; page++) { if (page < r->nbits) { - set_bit(page, r->bitmap); + if (!test_and_set_bit(page, r->bitmap)) { + stats->dirty_pages_set++; + stats->dirty_page_count++; + } } } } @@ -493,6 +500,15 @@ static uint32_t igb_core_vf_dirty_enable(IgbVfState *s= , uint64_t pgsize, IGBVfDirtyState *ds =3D igb_core_vf_dirty_state(s); IGBVfDirtyRange *r; =20 + /* + * Reset stats on first enable so the driver can read them after + * disable + */ + if (ds->num_ranges =3D=3D 0) { + memset(&igbvf_get_core(s)->vf_mig_stats[s->vfn], 0, + sizeof(IgbVfMigStats)); + } + if (ds->num_ranges >=3D IGB_MIG_CAPS_MAX_RANGES) { return IGB_MIG_ERR_TOO_MANY_RANGES; } @@ -635,12 +651,14 @@ static IGBVfDirtyRange *igb_core_vf_dirty_range_valid= (IGBVfDirtyState *ds, static uint8_t igbvf_mig_cmd_dirty_query(IgbVfState *s) { IgbVfMigState *ms =3D &s->mig; + IgbVfMigStats *stats =3D &igbvf_get_core(s)->vf_mig_stats[s->vfn]; IGBVfDirtyState *ds =3D &igbvf_get_core(s)->vf_dirty[s->vfn]; uint64_t buf_addr =3D ms->mig_data_buf_addr; uint64_t range_iova =3D 0, range_size =3D 0; IGBVfDirtyRange *range; uint32_t bmp_bytes, dirty_pages; uint32_t val32; + uint64_t val64; size_t out_size; g_autofree void *bitmap =3D NULL; =20 @@ -686,6 +704,10 @@ static uint8_t igbvf_mig_cmd_dirty_query(IgbVfState *s) =20 dirty_pages =3D bitmap_count_one(bitmap, range_size / range->page_size= ); =20 + stats->dirty_pages_cleared +=3D dirty_pages; + stats->dirty_page_count -=3D MIN(stats->dirty_page_count, dirty_pages); + stats->dirty_query_count++; + val32 =3D cpu_to_le32(out_size); address_space_write(&address_space_memory, buf_addr + offsetof(struct igb_mig_dirty_query, @@ -696,8 +718,13 @@ static uint8_t igbvf_mig_cmd_dirty_query(IgbVfState *s) buf_addr + offsetof(struct igb_mig_dirty_query, dirty_page_count), MEMTXATTRS_UNSPECIFIED, &val32, sizeof(val32)); - - trace_igbvf_mig_dirty_query(s->vfn, (uint64_t)out_size, dirty_pages); + val64 =3D cpu_to_le64(stats->dma_writes); + address_space_write(&address_space_memory, + buf_addr + offsetof(struct igb_mig_dirty_query, + dma_writes), + MEMTXATTRS_UNSPECIFIED, &val64, sizeof(val64)); + trace_igbvf_mig_dirty_query(s->vfn, (uint64_t)out_size, dirty_pages, + stats->dma_writes); return 0; } =20 @@ -898,6 +925,32 @@ static uint8_t igbvf_mig_set_state(IgbVfState *s, uint= 32_t new_state) return 0; } =20 +static uint8_t igbvf_mig_cmd_get_stats(IgbVfState *s) +{ + IgbVfMigState *ms =3D &s->mig; + IgbVfMigStats *stats =3D &igbvf_get_core(s)->vf_mig_stats[s->vfn]; + struct igb_mig_stats_resp resp; + MemTxResult r; + + if (!ms->mig_data_buf_addr) { + return IGB_MIG_ERR_NO_BUFFER; + } + + resp.dma_writes =3D cpu_to_le64(stats->dma_writes); + resp.dma_bytes =3D cpu_to_le64(stats->dma_bytes); + resp.dirty_pages_set =3D cpu_to_le32(stats->dirty_pages_set); + resp.dirty_pages_cleared =3D cpu_to_le32(stats->dirty_pages_cleared); + resp.dirty_page_count =3D cpu_to_le32(stats->dirty_page_count); + resp.dirty_query_count =3D cpu_to_le32(stats->dirty_query_count); + + r =3D address_space_write(&address_space_memory, ms->mig_data_buf_addr, + MEMTXATTRS_UNSPECIFIED, &resp, sizeof(resp)); + if (r !=3D MEMTX_OK) { + return IGB_MIG_ERR_DMA_FAILED; + } + return 0; +} + static void igbvf_mig_update_status(IgbVfState *s, uint8_t err) { IgbVfMigState *ms =3D &s->mig; @@ -955,6 +1008,10 @@ static void igbvf_mig_cmd_ctrl(IgbVfState *s, uint32_= t val) err =3D igbvf_mig_cmd_dirty_query(s); break; =20 + case IGB_MIG_CMD_GET_STATS: + err =3D igbvf_mig_cmd_get_stats(s); + break; + default: err =3D IGB_MIG_ERR_UNK_CMD; break; @@ -1059,6 +1116,8 @@ void igbvf_mig_state_reset(IgbVfState *s) memset(ms->mig_data, 0, sizeof(ms->mig_data)); ms->mig_saved_vfre =3D true; ms->mig_saved_vfte =3D true; + memset(&igbvf_get_core(s)->vf_mig_stats[s->vfn], 0, + sizeof(IgbVfMigStats)); =20 pci_set_long(PCI_DEVICE(s)->config + IGB_MIG_DVSEC_OFFSET + IGB_MIG_BUF_ADDR_LO, 0); diff --git a/hw/net/trace-events b/hw/net/trace-events index eb5c61a8f6fb..6c2c5493094a 100644 --- a/hw/net/trace-events +++ b/hw/net/trace-events @@ -302,7 +302,7 @@ igbvf_mig_load_state(uint16_t vfn, uint32_t size, bool = vfre, bool vfte) "VF%u: l igbvf_mig_reset(uint16_t vfn) "VF%u: migration state reset" igbvf_mig_dirty_enable(uint16_t vfn, uint64_t pgsize, uint64_t nbits) "VF%= u: dirty tracking enabled pgsize=3D%"PRIu64" nbits=3D%"PRIu64 igbvf_mig_dirty_disable(uint16_t vfn) "VF%u: dirty tracking disabled" -igbvf_mig_dirty_query(uint16_t vfn, uint64_t size, uint32_t dirty_pages) "= VF%u: dirty query returned %"PRIu64" bytes, %u dirty pages" +igbvf_mig_dirty_query(uint16_t vfn, uint64_t size, uint32_t dirty_pages, u= int64_t dma_writes) "VF%u: dirty query returned %"PRIu64" bytes, %u dirty p= ages (dma_writes=3D%"PRIu64")" igb_core_dirty_track_dma(int vfn, uint64_t addr, uint64_t len) "VF%d: dirt= y DMA addr=3D0x%"PRIx64" len=3D%"PRIu64 igb_core_dirty_track_dma_drop(int vfn, uint64_t addr, uint64_t len) "VF%d:= dirty DMA dropped addr=3D0x%"PRIx64" len=3D%"PRIu64" no matching range" igbvf_mig_quiesce(uint16_t vfn, uint32_t vfre, uint32_t vfte) "VF%u: quies= ce VFRE=3D0x%x VFTE=3D0x%x" --=20 2.55.0 From nobody Sat Sep 26 22:16:24 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=quarantine dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1788376945; cv=none; d=zohomail.com; s=zohoarc; b=S/0Sk9Okn+xCNH76MbK1RTX06LHEVGyPN4QmwdhSF/HuSHcE0VwzZWaf9E4qtpZLi2IXDwLYbjR3gOZth7gX6D5/yatKbr2YbkI1RVSNNsFDF82HxG0ZjTZ157beJqYw+OCr/ao+QoJURG8wRmv8JmvrRL5ssO805uTx9OUsReM= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1788376945; h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=F8Bn1fNYQ+/YxdFtQWMoHH/UqkbpvtBnA9tUBkcPzi0=; b=NCUqi9T7i7UjH260gIS7cVi5o8kisBvGLuYCWsMtrMKLh2gi4CRQilaTG/iObzL8zTBczWGHbYDweoYdb2PT611WvEIRMQDzm1UxNdlt8D8AFF0CLRaF7AodGgJD1Zzxm0+Ssye1C5j/xCpNaN5og+J6KuRnAoPYEcvSx1YT41Q= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=quarantine dis=none) Return-Path: Received: from lists1p.gnu.org (lists1p.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1788376945611413.3398124279688; Wed, 2 Sep 2026 12:22:25 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists1p.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1x1qWe-0004Mo-En; Wed, 02 Sep 2026 15:21:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists1p.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x1qWd-0004MH-8s for qemu-devel@nongnu.org; Wed, 02 Sep 2026 15:21:35 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1x1qWa-0002CE-9h for qemu-devel@nongnu.org; Wed, 02 Sep 2026 15:21:35 -0400 Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-320-Q2niTL3BNFGixVMQ-m77Og-1; Wed, 02 Sep 2026 15:21:28 -0400 Received: from mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D02A41955E96; Wed, 2 Sep 2026 19:21:26 +0000 (UTC) Received: from corto.redhat.com (unknown [10.44.32.5]) by mx-prod-int-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 7EA0E3000DA3; Wed, 2 Sep 2026 19:21:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1788376891; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=F8Bn1fNYQ+/YxdFtQWMoHH/UqkbpvtBnA9tUBkcPzi0=; b=f8k6pGpR6SyTdQoSJfSaeOMKenRPqfDQZEiMeAKaokwSRfKw7fbjny4QUPQgPyh1eLg82E m6UXxceeVuy/x/ax7qL4ZzKSW599/MWPt/UhgaiFKaHmhgId9xbxXpYnRj/ENI9k/fh15N U7hXXv70ejPiCrnMbdR0ZqzRXAptrlg= X-MC-Unique: Q2niTL3BNFGixVMQ-m77Og-1 X-Mimecast-MFC-AGG-ID: Q2niTL3BNFGixVMQ-m77Og_1788376886 From: =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= To: qemu-devel@nongnu.org Cc: Akihiko Odaki , Sriram Yagnaraman , Jason Wang , Alex Williamson , Peter Xu , =?UTF-8?q?C=C3=A9dric=20Le=20Goater?= Subject: [RFC PATCH v2 9/9] docs: Add igb VF migration testing setup guide Date: Wed, 2 Sep 2026 21:20:54 +0200 Message-ID: <20260902192054.3329753-10-clg@redhat.com> In-Reply-To: <20260902192054.3329753-1-clg@redhat.com> References: <20260902192054.3329753-1-clg@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.4 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists1p.gnu.org; Received-SPF: pass client-ip=170.10.133.124; envelope-from=clg@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: 12 X-Spam_score: 1.2 X-Spam_bar: + X-Spam_report: (1.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_SBL_CSS=3.335, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1788376949111154100 Document the igb VF migration interface: DVSEC register layout, dirty page tracking, testing setup with nested virtualization. AI-used-for: docs Signed-off-by: C=C3=A9dric Le Goater --- MAINTAINERS | 1 + docs/system/device-emulation.rst | 1 + docs/system/devices/igb-migration.rst | 417 ++++++++++++++++++++++++++ docs/system/devices/igb.rst | 6 + 4 files changed, 425 insertions(+) create mode 100644 docs/system/devices/igb-migration.rst diff --git a/MAINTAINERS b/MAINTAINERS index f88b526be238..4a2f1357e936 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -2820,6 +2820,7 @@ igb VF migration M: C=C3=A9dric Le Goater S: Maintained F: hw/net/igb_migration.* +F: docs/system/devices/igb-migration.rst =20 eepro100 M: Stefan Weil diff --git a/docs/system/device-emulation.rst b/docs/system/device-emulatio= n.rst index 40054bb7dfcc..75f423b795cd 100644 --- a/docs/system/device-emulation.rst +++ b/docs/system/device-emulation.rst @@ -90,6 +90,7 @@ Emulated Devices devices/cxl.rst devices/emmc.rst devices/igb.rst + devices/igb-migration.rst devices/ivshmem-flat.rst devices/ivshmem.rst devices/keyboard.rst diff --git a/docs/system/devices/igb-migration.rst b/docs/system/devices/ig= b-migration.rst new file mode 100644 index 000000000000..d72f14f5fe63 --- /dev/null +++ b/docs/system/devices/igb-migration.rst @@ -0,0 +1,417 @@ +.. SPDX-License-Identifier: GPL-2.0-or-later +.. _igb-migration: + +igb VF Migration +---------------- + +Live migration of VFIO-passthrough devices (SR-IOV VFs, vGPUs) is a +growing requirement, but real hardware with migration support is scarce +and hard to debug. An emulated device provides a fully controlled +testbed for developing and validating the entire software stack -- +vfio-pci variant drivers, VFIO core migration v2 framework, QEMU, +libvirt -- and for tuning complex migration policies such as downtime +convergence. It also serves as an educational reference for +understanding VFIO migration end-to-end, from device state +serialization to dirty page tracking. + +The igb device supports an experimental VF migration interface that allows +the `igb-vfio-pci`_ variant driver to migrate VF state during live +migration using the standard VFIO migration v2 protocol with stop-copy +and pre-copy support. + +This is enabled with the ``x-vf-migration`` property:: + + -device igb,x-vf-migration=3Don,... + +Each emulated VF then advertises a DVSEC discovered by the +`igb-vfio-pci`_ variant driver at bind time. This feature is +experimental (``x-`` prefix, default off). + +Architecture +~~~~~~~~~~~~ + +The target scenario is nested virtualization:: + + L0 QEMU + igb PF with x-vf-migration=3Don + =E2=94=94=E2=94=80=E2=94=80 VFs with migration DVSEC + + L1 kernel + igb-vfio-pci variant driver + translates VFIO migration v2 ioctls =E2=86=92 DVSEC config writes + + L1 QEMU (stock, unmodified) + vfio-pci device model, standard migration fd + + L2 guest + standard igbvf driver, unaware of migration + +The L1 QEMU is completely unmodified -- it sees a standard VFIO +migratable device and uses the normal migration fd path. The +`igb-vfio-pci`_ variant driver handles the translation between +VFIO migration v2 ioctls and DVSEC config writes. + +Design +~~~~~~ + +The migration interface is exposed through a DVSEC at offset 0x160 +in VF extended config space (see `DVSEC register layout`_ below for +the full register map). + +Device state is serialized as a versioned blob of per-VF register +(offset, value) pairs covering control, interrupt, RX/TX queue, +receive address (RA/RA2), etc. plus TX context descriptors and +VFRE/VFTE enable bits. The buffer address is a guest physical address +(GPA) written by the driver via ``virt_to_phys``; the device accesses +guest RAM directly through the system address space. + +Dirty page tracking is implemented with per-range bitmaps maintained +in IGBCore. All VF DMA paths in ``igb_core.c`` (TX data, RX data, +descriptor writeback) are instrumented to record touched pages. The +`igb-vfio-pci`_ variant driver registers tracked IOVA ranges and +queries dirty bitmaps through a shared buffer. Buffer structures +include len, flags, and reserved fields for future extensibility. + +The dirty bitmaps are maintained inside the device, which is not +realistic for discrete NICs without on-chip DRAM. + +DVSEC register layout +~~~~~~~~~~~~~~~~~~~~~ + +The migration DVSEC (36 bytes at offset ``0x160``) uses a command +doorbell model. All commands are synchronous -- the device completes +the operation before the config write returns:: + + Offset Name Access Description + +0x00 ExtCap Hdr RO PCIe extended cap (id=3D0x23, ver=3D1) + +0x04 DVSEC Hdr 1 RO length[31:20] | rev[19:16] | vendor_id[15:= 0] + +0x08 DVSEC Hdr 2 RO DVSEC ID (1) + +0x0A Reserved - Padding for DWORD alignment + +0x0C CAPS RO F_STATE[0], F_DIRTY[1], max_ranges[11:8], = pgsize[16:12] + +0x10 CTRL WO Doorbell: cmd[7:0], arg[31:8] + +0x14 STATUS RO state[7:0], error_code[15:8], quiesced[16] + +0x18 BUF_ADDR_LO RW Shared DMA buffer GPA (low 32 bits) + +0x1C BUF_ADDR_HI RW Shared DMA buffer GPA (high 32 bits) + +0x20 DATA_SIZE RO State blob size in bytes + +CTRL commands:: + + Cmd Name Arg Description + 1 SET_STATE state[31:8] Set migration state + 2 SAVE - DMA-write state to buffer + 3 LOAD size[31:8] DMA-read state from buffer + 4 DIRTY_ENABLE - Enable dirty tracking (params in DM= A buffer) + 5 DIRTY_DISABLE - Disable dirty tracking + 6 DIRTY_QUERY - Query dirty bitmap (via DMA buffer) + 7 GET_STATS - Query statistics (via DMA buffer) + +The driver sets ``BUF_ADDR_LO/HI`` before issuing commands that use a +DMA buffer (SAVE, LOAD, DIRTY_ENABLE, DIRTY_QUERY, GET_STATS). The +buffer address is latched per CTRL write, so the driver can use +different buffers for different commands. The buffer address is a +guest physical address (GPA). + +State transitions follow the VFIO migration v2 state machine. The +driver issues ``SET_STATE`` with the target state in the arg field and +reads ``STATUS`` to confirm the transition. Device states:: + + 0 ERROR 1 STOP 2 RUNNING + 3 STOP_COPY 4 RESUMING 5 PRE_COPY + +``DATA_SIZE`` reflects the state blob size. At reset and in ``STOP`` +state it holds the maximum size the driver should allocate. After +``SET_STATE(STOP_COPY)`` or ``SAVE`` it holds the actual serialized +size. The driver reads it after entering ``STOP_COPY`` to allocate an +exact-sized DMA buffer before issuing ``SAVE``. + +The state blob is a versioned sequence of register (offset, value) +pairs with magic ``0x4D494742`` ("MIGB"). + +When ``STATUS`` state is ``ERROR`` (0), bits [15:8] contain an error +code identifying the failure:: + + 1 UNK_CMD Unknown CTRL command + 2 BAD_STATE Command issued in wrong migration state + 3 NO_BUFFER Command requires buffer but BUF_ADDR not set + 4 DMA_FAILED DMA transfer to/from buffer failed + 5 BAD_SIZE State blob too large or empty + 6 BAD_MAGIC State blob magic mismatch + 7 BAD_VERSION State blob version mismatch + 8 TOO_MANY_RANGES Exceeds max_ranges from CAPS + 9 BAD_RANGE Invalid range (zero size, misaligned, not containe= d) + 10 BAD_PGSIZE Invalid or misaligned page size + 11 NOT_ENABLED Dirty query without prior enable + +Dirty page tracking +~~~~~~~~~~~~~~~~~~~ + +The migration interface supports per-VF dirty page tracking, advertised +by the ``F_DIRTY`` flag (bit 1) in ``CAPS``. This allows the variant +driver to enter ``PRE_COPY`` state while the VM continues to run, +iterating on dirty pages to reduce the final stop-and-copy window. + +The device maintains one dirty tracking engine per range, each with its +own bitmap scoped to the range boundaries. The ``CAPS`` register +advertises the maximum number of ranges in bits [11:8]. + +Dirty tracking is controlled through CTRL commands: + +- **DIRTY_ENABLE** (4): the driver fills an ``igb_mig_dirty_enable_req`` + struct in the DMA buffer with the page size, range IOVA, and range + size, then issues the command. The device allocates a bitmap for the + range and begins recording pages touched by DMA. Supported page sizes + are advertised in ``CAPS`` bits [16:12] (bit N =3D 2^N bytes). The driver + checks ``STATUS`` for errors after the command completes. +- **DIRTY_DISABLE** (5): tears down all ranges and stops tracking. +- **DIRTY_QUERY** (6): the driver writes (iova, size) into the + ``igb_mig_dirty_query`` DMA buffer, then issues the command. The + device validates the range, copies the dirty bitmap into the buffer, + and clears the tracked bits after successful DMA. The driver checks + ``STATUS`` for errors after the command. + +Dirty enable DMA buffer +~~~~~~~~~~~~~~~~~~~~~~~ + +The ``DIRTY_ENABLE`` command reads its parameters from the DMA buffer. +The ``len`` field holds the total structure size (including reserved +bytes) so the device can detect newer formats. ``flags`` and +``reserved`` must be zero:: + + Offset Field Type Description + 0x00 len uint32 Structure size in bytes + 0x04 flags uint32 Reserved, must be 0 + 0x08 pgsize uint64 Page granularity (must match a CAPS pgsiz= e bit) + 0x10 range_iova uint64 Tracked range start address + 0x18 range_size uint64 Tracked range size in bytes + 0x20 reserved[4] uint32 Reserved, must be 0 + +Dirty query DMA buffer +~~~~~~~~~~~~~~~~~~~~~~ + +The ``DIRTY_QUERY`` command uses a shared DMA buffer for both request +and response. The ``len`` field holds the total buffer size (header + +bitmap). ``flags`` and ``reserved`` must be zero:: + + Offset Field Written by Description + 0x00 len driver Total buffer size in bytes + 0x04 flags driver Reserved, must be 0 + 0x08 iova driver Query range start + 0x10 size driver Query range size + 0x18 bitmap_size device Bytes written to bitmap + 0x1C dirty_page_count device Number of set bits + 0x20 dma_writes device DMA write count (diagnostic) + 0x28 reserved[6] - Reserved, must be 0 + 0x40 bitmap[] device Dirty page bitmap + +Migration statistics +~~~~~~~~~~~~~~~~~~~~ + +The ``GET_STATS`` command DMA-writes a statistics response into the +driver-provided buffer. The driver sets ``BUF_ADDR_LO/HI`` and issues +the command; the device writes the response and returns:: + + Offset Field Type Description + 0x00 dma_writes uint64 DMA write operations tracked + 0x08 dma_bytes uint64 DMA bytes written + 0x10 dirty_pages_set uint32 Dirty pages marked since enable + 0x14 dirty_pages_cleared uint32 Dirty pages cleared by queries + 0x18 dirty_page_count uint32 Current dirty pages (set - cleare= d) + 0x1C dirty_query_count uint32 Number of QUERY operations + +The variant driver exposes these via debugfs at +``/sys/kernel/debug/vfio//migration/dirty/stats``. + +Testing setup +~~~~~~~~~~~~~ + +The target scenario is nested virtualization: L0 runs QEMU with an +igb PF (``x-vf-migration=3Don``), L1 runs the `igb-vfio-pci`_ variant +driver and an unmodified QEMU, and L2 runs a standard igbvf driver. +See `Architecture`_ above for the full stack diagram. + +NetworkManager configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In a nested setup, the L1 VMs (source and destination) have emulated +igb PFs connected to the L0 bridge. By default, NetworkManager +acquires DHCP leases on those PF interfaces and on any igb VFs created +later. This causes the VF MAC address to be learned on the L0 bridge, +which can misdirect iperf3 traffic after migration. + +To prevent this, configure NetworkManager on **both L1 VMs** and on the +**L2 guest disk image**. + +L1 VMs (source and destination) +............................... + +1. Prevent NetworkManager from managing igbvf interfaces: + +.. code-block:: bash + + cat > /etc/NetworkManager/conf.d/99-no-igbvf.conf <" ipv4.method disabled ipv6= .method disabled + +3. Reload NetworkManager: + +.. code-block:: bash + + nmcli general reload + +L2 guest disk image +................... + +Use ``virt-customize`` to add the igbvf unmanaged config to the guest +image (offline, before any test run): + +.. code-block:: bash + + virt-customize -a /srv/migration/rhel10.qcow2 \ + --write /etc/NetworkManager/conf.d/99-no-igbvf.conf:'[keyfile] + unmanaged-devices=3Ddriver:igbvf' + +Network diagram +^^^^^^^^^^^^^^^ + +The diagram below shows the nested setup where the source and +destination hosts are themselves VMs (L1) running on a physical +host (L0) that emulates the igb NIC:: + + =E2=94=8C=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=90 + =E2=94=82 L0: physical host = =E2=94=82 + =E2=94=82 = =E2=94=82 + =E2=94=82 virbr0 192.168.199.1/24 = =E2=94=82 + =E2=94=82 =E2=94=9C=E2=94=80=E2=94=80 NFS server: /srv/migration = =E2=94=82 + =E2=94=82 =E2=94=94=E2=94=80=E2=94=80 iperf3 client: iperf3 -c 192.168.= 199.200 -t 60 -i 1 =E2=94=82 + =E2=94=82 =E2=94=82 = =E2=94=82 + =E2=94=82 =E2=94=82 L0 virbr0 bridge (192.168.199.0/24) = =E2=94=82 + =E2=94=82 =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=BC=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=AC=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=AC=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 = =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=8C=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=B4=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=90 =E2=94=8C= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=B4=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=90 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 virtio =E2=94=82 =E2=94= =82 emulated=E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82c0:ff:ee:=E2=94=82 =E2=94= =82 igb PF =E2=94=82 L0 QEMU (vm6) =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 :00:06 =E2=94=82 =E2=94= =82 + igbvf =E2=94=82 tracks DMA dirty pages =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=94=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=AC=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=98 =E2=94=94= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=AC=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=98 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 = =E2=94=82 + =E2=94=82 =E2=94=8C=E2=94=80=E2=94=80=E2=94=80=E2=94=BC=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=BC=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=BC=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=90 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 L1: vm6 (source) =E2=94=82 = =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 enp1s0: 192.168.199.6 =E2=94=82 = =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 (management) =E2=94=82 = =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 enp8s0 (igb PF, = no IP) =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 = =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 igb VF0 =E2=94= =80=E2=94=80=E2=96=BA igb-vfio-pci (VFIO) =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 = dirty_sync =E2=86=92 L0 igbvf =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 = =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 virbr0 =E2=94=82 V= FIO passthrough =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 192.168.200.1/24 =E2=94=82 = =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 =E2= =94=82 =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=8C=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=BC=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=BC=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=90 =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 L2: rhel10 gue= st =E2=94=82 =E2=94=82 =E2=94=82 =E2=94= =82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 = =E2=94=82 =E2=94=82 =E2=94=82 =E2=94= =82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 virtio NIC igb VF= (enp7s0) =E2=94=82 =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 192.168.200.130/24 192.16= 8.199.200/24 =E2=94=82 =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 (SSH login) (iperf= 3 data path) =E2=94=82 =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 =E2= =94=82 =E2=94=82 =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 iperf3 -s -D =E2= =94=82 (listens on 0.0.0.0) =E2=94=82 =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=94=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=BC=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=98 =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 = =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 virsh migrate --live =E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=BC=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=96=BA vm7 = =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 = =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=94=E2=94=80=E2=94=80=E2=94=80=E2=94=BC=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=BC=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=98 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 = =E2=94=82 + =E2=94=82 =E2=94=82 iperf3 traffic =E2=94=82 = =E2=94=82 + =E2=94=82 =E2=94=94=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=98 = =E2=94=82 + =E2=94=82 = =E2=94=82 + =E2=94=82 =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80 = =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 = =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=8C=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=90 =E2=94=82 =E2=94=8C=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=90 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 virtio =E2=94=82 =E2=94=82 = =E2=94=82emulated =E2=94=82 L0 QEMU (vm7) =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82c0:ff:ee=E2=94=82 =E2=94=82 = =E2=94=82 igb PF =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 :00:07 =E2=94=82 =E2=94=82 = =E2=94=82 + igbvf =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=94=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=AC=E2=94=80=E2=94=80=E2=94=80=E2=94=98 =E2=94=82 =E2=94=94=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=AC=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=98 =E2=94=82 + =E2=94=82 =E2=94=8C=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=BC=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=BC=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=BC=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=90 =E2=94=82 + =E2=94=82 =E2=94=82 L1: vm7 (destination) =E2=94=82 = =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 enp1s0: 192.168.199.7 =E2=94=82 = =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 (management) enp8s0 (igb PF, no IP) = =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 = =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 igb VF0 =E2=94=80=E2=94=80= =E2=96=BA igb-vfio-pci (VFIO) =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 = =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 virbr0 =E2=94=82 VFIO passth= rough =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 192.168.200.1/24 =E2=94=82 = =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 = =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=8C=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=BC=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=BC=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=90 =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 L2: rhel10 (after migratio= n) =E2=94=82 =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 =E2= =94=82 =E2=94=82 =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 virtio NIC igb VF (enp7s0) = =E2=94=82 =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 192.168.200.130/24 192.168.199.200/= 24 =E2=94=82 =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 =E2=94=82 = =E2=94=82 =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=82 iperf3 -s -D =E2=94=82 (co= nnection survives) =E2=94=82 =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=82 =E2=94=94=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=BC=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=98 =E2=94=82 =E2=94=82 + =E2=94=82 =E2=94=94=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=BC=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=98 =E2=94=82 + =E2=94=82 =E2=94=82 = =E2=94=82 + =E2=94=82 iperf3 traffic resumes =E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=98 =E2=94=82 + =E2=94=82 (same IP, same MAC, same L2 segment =E2=86=92 transparent= to client) =E2=94=82 + =E2=94=94=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80= =E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2= =94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94=80=E2=94= =80=E2=94=80=E2=94=80=E2=94=98 + +Migration under iperf3 load works correctly: dirty page tracking +converges (from ~2000 pages per PRE_COPY iteration down to ~280 at +STOP_COPY), and STOP_COPY stays under 250ms. + +Todo +~~~~ + +1. Add migration blocker when ``x-vf-migration=3Don`` (no VMState yet) or + add VMState support for L0 migration (dirty bitmaps, tracking + engines, DVSEC registers, stats) +2. Add PRE_COPY state transfer to validate device INIT data (magic, + version, etc.) +3. Add qtests for migration state machine transitions, dirty page + tracking + +Ideas +~~~~~ + +1. **RX bandwidth throttle** (``x-mig-rx-limit``, uint32, default 0) + + Return false from ``can_receive`` when the per-VF packet count in the + current tracking interval exceeds the limit. Reduces DMA writes and + dirty pages realistically. + +2. **Migration phase timing** (GET_STATS extension) + + Add per-VF timestamps: ``precopy_start_ns``, ``stopcopy_start_ns``, + ``precopy_duration_ns``, ``stopcopy_duration_ns``, + ``state_transition_count``. Expose via GET_STATS. + +3. **Hot page simulation** (``x-mig-hot-pages``, uint32, default 0) + + Re-set the first N bitmap bits after each DIRTY_QUERY, simulating + workloads with hot pages that prevent convergence. + +4. **Error injection** (``x-mig-inject-error``, uint32, default 0) + + One-shot error code injection before command dispatch. A separate + ``x-mig-inject-dma-fail`` (bool) for persistent DMA failure testing. + +AI disclaimer +~~~~~~~~~~~~~ + +Claude was used to analyze the IGB PF and VF internal state and +identify the pain points of a working live migration of such devices. +The generated code served as a starting point but *significant* time +was then spent cleaning up, reworking, and shaping it into a clear, +reviewable IGB model extension. + +.. _igb-vfio-pci: https://github.com/legoater/vfio-pci-extras diff --git a/docs/system/devices/igb.rst b/docs/system/devices/igb.rst index 50f625fd77e4..00271dbc92c3 100644 --- a/docs/system/devices/igb.rst +++ b/docs/system/devices/igb.rst @@ -64,6 +64,12 @@ command: =20 pyvenv/bin/meson test --suite thorough func-x86_64-netdev_ethtool =20 +VF Migration (experimental) +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D + +See :ref:`igb-migration` for details on the experimental VF live migration +interface. + References =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 --=20 2.55.0