From nobody Tue May 14 13:02:52 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) client-ip=170.10.133.124; envelope-from=libvir-list-bounces@redhat.com; helo=us-smtp-delivery-124.mimecast.com; Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass(p=none dis=none) header.from=redhat.com ARC-Seal: i=1; a=rsa-sha256; t=1657643273; cv=none; d=zohomail.com; s=zohoarc; b=mAPVqfRUsa0MNjlgJeY+2S3v8iU8W1uKXFWgIc9DHvbOsvAKZXUmMu5+KFYuyi7GAqHKCrN/Bjrr8qSHOscQTkLU71Wu3k7/Omru+0SETCDz8WG+L2hH/UWCJT9HVSL6gC+lBPWMJ8uRqUPJiqstGk8wh1IDH4CFVbywGqgapuA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1657643273; h=Content-Type:Content-Transfer-Encoding:Date:From:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:Sender:Subject:To; bh=g3NnWEV0B1EOeuDqUfjRhiTfEJXI5t+ycw0yXtYpTgI=; b=BwBZBO9knqznHxCR3XHdv40qBlpAlhT+MKVSaYzunX4LOi7vad+G3k3PTLnmsS5/mB/mu4C5nNogyQPon5wr7eI5hUa/JKPIygcGo36kiKcBYX7WnhSDXeM8GmSyg3qzGI5pzUPk9dreaD6VynEUHAM6z6X374HLvequJmVZFg8= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=libvir-list-bounces@redhat.com; dmarc=pass header.from= (p=none dis=none) Return-Path: Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mx.zohomail.com with SMTPS id 165764327311474.55223981979725; Tue, 12 Jul 2022 09:27:53 -0700 (PDT) Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-654-HhklAJomMjWmgNs3YGnwPw-1; Tue, 12 Jul 2022 12:27:41 -0400 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9E014821BB2; Tue, 12 Jul 2022 16:27:37 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (unknown [10.30.29.100]) by smtp.corp.redhat.com (Postfix) with ESMTP id C07102026D2D; Tue, 12 Jul 2022 16:27:36 +0000 (UTC) Received: from mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (localhost [IPv6:::1]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id 889CE1947060; Tue, 12 Jul 2022 16:27:36 +0000 (UTC) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) by mm-prod-listman-01.mail-001.prod.us-east-1.aws.redhat.com (Postfix) with ESMTP id F0FAC1947058 for ; Tue, 12 Jul 2022 16:27:35 +0000 (UTC) Received: by smtp.corp.redhat.com (Postfix) id D7FA92026D07; Tue, 12 Jul 2022 16:27:35 +0000 (UTC) Received: from localhost.localdomain.com (unknown [10.33.36.104]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5AFD02026D64; Tue, 12 Jul 2022 16:27:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1657643272; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=g3NnWEV0B1EOeuDqUfjRhiTfEJXI5t+ycw0yXtYpTgI=; b=h52OYfNL4JBx4h0dvNJ3W0t2n5dhAHMkWbAoeJ7l0RSZldnHYxs301ij/55Lx8V2ow6qPa bY3c16J+HjK1+8/hmjqdTpL11dGqDNczUuOyz2/B8KokjcLOYDcsTNrc/0RNRsi37sOChK 3cfnefQnbCoRPLWxElhB5/uR+w1Ntd0= X-MC-Unique: HhklAJomMjWmgNs3YGnwPw-1 X-Original-To: libvir-list@listman.corp.redhat.com From: =?UTF-8?q?Daniel=20P=2E=20Berrang=C3=A9?= To: libvir-list@redhat.com Subject: [libvirt PATCH v2] docs: add info about factors affecting CPU compatibility Date: Tue, 12 Jul 2022 17:27:33 +0100 Message-Id: <20220712162733.1586844-1-berrange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-BeenThere: libvir-list@redhat.com X-Mailman-Version: 2.1.29 Precedence: list List-Id: Development discussions about the libvirt library & tools List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: libvir-list-bounces@redhat.com Sender: "libvir-list" X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=libvir-list-bounces@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-ZohoMail-DKIM: pass (identity @redhat.com) X-ZM-MESSAGEID: 1657643274238100001 While libvirt solves most of the problem of ensuring compatibility, when there is incompatibility it can be hard for users to track down the cause. Everything knows to check the physical CPU model, but there are a surprisingly large number of other factors influencing compatibility. Signed-off-by: Daniel P. Berrang=C3=A9 Reviewed-by: Michal Privoznik --- In v2: - Also mention nested virt docs/drvqemu.rst | 83 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/docs/drvqemu.rst b/docs/drvqemu.rst index c174bdb4cf..c9f2432268 100644 --- a/docs/drvqemu.rst +++ b/docs/drvqemu.rst @@ -416,6 +416,89 @@ following command should be run as root, prior to star= ting libvirtd libvirt will then place each virtual machine in a cgroup at ``/dev/cgroup/libvirt/qemu/$VMNAME/`` =20 + +Live migration compatibility +---------------------------- + +Many factors can affect the ability to live migrate a guest between a pair +of hosts. It is critical that when QEMU is started on the destination host, +the exposed guest machine ABI matches what was exposed by the existing QEMU +process on the source host. To facilitate this, when libvirt receives a +guest configuration document, it will attempt to expand any features that +were not specified, to ensure a stable guest machine ABI. Mostly this invo= lves +adding address information to all devices, and adding controllers to attach +the devices to. + +Certain features that affect the guest ABI, however, may only be known at = the +time the guest is started and can be influenced by features of the host OS +and its hardware. This means that even if the guest XML configuration is t= he +same, it may still be impossible to migrate the guest between two hosts. + +Migration CPU model compatibility +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The most common problems with migration compatibility surround the use of +the guest CPU ``host-model`` or ``host-passthrough`` modes. Both of these +modes attempt to expose the full host CPU featureset to the guest. The +``host-model`` mode attempts to expose as many features as possible while +retaining the ability to accurately check compatibility between hosts prior +to migration running. The ``host-passthrough`` mode attempts to expose the +host CPU as precisely as possible, but with the cost that it is not possib= le +for libvirt to check compatibility prior to migration. + +If using ``host-model`` the target host hardware and software deployment +must expose a superset of the features of the source host CPU. If using +``host-passthrough`` the target host CPU and software deployment must +always expose a superset of the fetures, however, it is further strongly +recommended that the source and destination hosts be identical in every +way. + +In both cases, there are a number of factors that will influence the CPU +features available to the guest + +* **Physical CPU model** - the core constraint on what features are availa= ble. + Check ``/proc/cpuinfo`` for CPU model name. +* **Firmware revision (BIOS/UEFI/etc)** - firmware updates may bundle micr= ocode + updates which arbitrarily add or remove CPU features, typically in respo= nse + to new hardware vulnerabilities. Check ``dmidecode`` for details on ``x8= 6`` + and ``aarch64`` platforms for firmware version, and ``/proc/cpuinfo`` for + associated microcode version (if not updated by the OS). +* **Firmware settings** - certain firmware settings can affect accessibili= ty of + features. For example, turning on/off SMT/HT not only affects the number + of logical CPUs available to the OS, but can indirectly influence other + factors such as the number of performance counters available for use. Ch= eck + the firmware specific configuration interface. +* **Host kernel version** - the host kernel software version may have a + need to block certain physical CPU features from use in the guest. It can + also emulate certain features that may not exist in the silicon, for exa= mple, + x2apic. Check ``uname -r`` output for kernel version. +* **Host kernel settings** - the kernel command line options can be used to + block certain physical CPU features from use in the guest, for example, + ``tsx=3Doff``, ``l1tf=3D...`` or ``nosmt``. Check ``/proc/cmdline`` and + ``/etc/modprobe.d/*.conf``. +* **microcode update version** - while the firmware will load the initial + microcode in to the CPU, the OS may ship packages providing newer microc= ode + updates since these can be deployed on a more timely manner than firmware + updates. These updates can arbitrarily load add or remove CPU features. + Check ``/proc/cpuinfo`` for microcode version. +* **QEMU version** - even when the kernel supports exposing a CPU feature = to + the guest, an update in the QEMU emulator version will be required to un= lock + its usage with a guest, except with ``host-passthrough``. Check the outp= ut + of ``$QEMU -version``. +* **libvirt version** - even when the kernel and QEMU support exposing a C= PU + feature to the guest, an update in the libvirt version will be required = to + unlock its usage with a guest, except with ``host-passthrough``. Check + ``virsh version``. +* **Nested virtualization** - due to the limitations of nested virtualizat= ion, + a L1 nested host may not be able to expose the same featureset as a bare + metal host, even if everything else is the same. + +The ``virsh capabilities`` output will provide information on the high lev= el +CPU model, its features, microcode version. Most of the time this will pro= vide +enough information to know whether the CPUs of two hosts will be compatibl= e. +If there are unexpected differences though, checking the above list of +influencing factors can reveal where the difference arises from. + Import and export of libvirt domain XML configs ----------------------------------------------- =20 --=20 2.36.1