From nobody Wed Nov 27 11:37:12 2024 Delivered-To: importer@patchew.org Received-SPF: pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) client-ip=192.237.175.120; envelope-from=xen-devel-bounces@lists.xenproject.org; helo=lists.xenproject.org; Authentication-Results: mx.zohomail.com; spf=pass (zohomail.com: domain of lists.xenproject.org designates 192.237.175.120 as permitted sender) smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Return-Path: Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) by mx.zohomail.com with SMTPS id 1698746881593553.5452924854391; Tue, 31 Oct 2023 03:08:01 -0700 (PDT) Received: from list by lists.xenproject.org with outflank-mailman.625584.975000 (Exim 4.92) (envelope-from ) id 1qxleo-000180-OU; Tue, 31 Oct 2023 10:07:34 +0000 Received: by outflank-mailman (output) from mailman id 625584.975000; Tue, 31 Oct 2023 10:07:34 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qxleo-00017t-Ln; Tue, 31 Oct 2023 10:07:34 +0000 Received: by outflank-mailman (input) for mailman id 625584; Tue, 31 Oct 2023 10:07:33 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qxlen-00017n-Bp for xen-devel@lists.xenproject.org; Tue, 31 Oct 2023 10:07:33 +0000 Received: from smarthost01b.sbp.mail.zen.net.uk (smarthost01b.sbp.mail.zen.net.uk [212.23.1.3]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 4d33a229-77d5-11ee-98d6-6d05b1d4d9a1; Tue, 31 Oct 2023 11:07:30 +0100 (CET) Received: from [217.155.64.189] (helo=mail0.xen.dingwall.me.uk) by smarthost01b.sbp.mail.zen.net.uk with esmtpsa (TLS1.0) tls TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (Exim 4.95) (envelope-from ) id 1qxlej-005oZv-Ts for xen-devel@lists.xenproject.org; Tue, 31 Oct 2023 10:07:29 +0000 Received: from localhost (localhost [IPv6:::1]) by mail0.xen.dingwall.me.uk (Postfix) with ESMTP id 913E188396D for ; Tue, 31 Oct 2023 10:07:29 +0000 (GMT) Received: from mail0.xen.dingwall.me.uk ([127.0.0.1]) by localhost (mail0.xen.dingwall.me.uk [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pMEm6G2DdbAN for ; Tue, 31 Oct 2023 10:07:29 +0000 (GMT) Received: from ghoul.dingwall.me.uk (ghoul.dingwall.me.uk [IPv6:2a02:8010:698e:302::c0a8:1c8]) by dingwall.me.uk (Postfix) with ESMTP id 6847088396A for ; Tue, 31 Oct 2023 10:07:29 +0000 (GMT) Received: by ghoul.dingwall.me.uk (Postfix, from userid 1000) id 5BD79488; Tue, 31 Oct 2023 10:07:29 +0000 (GMT) X-Outflank-Mailman: Message body and most headers restored to incoming version X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 4d33a229-77d5-11ee-98d6-6d05b1d4d9a1 X-Virus-Scanned: Debian amavisd-new at dingwall.me.uk Date: Tue, 31 Oct 2023 10:07:29 +0000 From: James Dingwall To: xen-devel@lists.xenproject.org Subject: live migration fails: qemu placing pci devices at different locations Message-ID: MIME-Version: 1.0 Content-Disposition: inline X-Originating-smarthost01b-IP: [217.155.64.189] Feedback-ID: 217.155.64.189 X-ZM-MESSAGEID: 1698746882965100001 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Hi, I'm having a bit of trouble performing live migration between hvm guests. = The sending side is xen 4.14.5 (qemu 5.0), receiving 4.15.5 (qemu 5.1). The er= ror message recorded in qemu-dm---incoming.log: qemu-system-i386: Unknown savevm section or instance '0000:00:04.0/vga' 0. = Make sure that your current VM setup matches your saved VM setup, including= any hotplugged devices I have patched libxl_dm.c to explicitly assign `addr=3Dxx` values for vario= us devices and when these are correct the domain migrates correctly. However the configuration differences between guests means that the values are not consistent. The domain config file doesn't allow the pci address to be expressed in the configuration for, e.g. `soundhw=3D"DEVICE"` e.g.=20 diff --git a/tools/libs/light/libxl_dm.c b/tools/libs/light/libxl_dm.c index 6e531863ac0..daa7c49846f 100644 --- a/tools/libs/light/libxl_dm.c +++ b/tools/libs/light/libxl_dm.c @@ -1441,7 +1441,7 @@ static int libxl__build_device_model_args_new(libxl__= gc *gc, flexarray_append(dm_args, "-spice"); flexarray_append(dm_args, spiceoptions); if (libxl_defbool_val(b_info->u.hvm.spice.vdagent)) { - flexarray_vappend(dm_args, "-device", "virtio-serial", + flexarray_vappend(dm_args, "-device", "virtio-serial,addr= =3D04", "-chardev", "spicevmc,id=3Dvdagent,name=3Dvdagent", "-= device", "virtserialport,chardev=3Dvdagent,name=3Dcom.redhat.sp= ice.0", NULL); The order of devices on the qemu command line (below) appears to be the same so my assumption is that the internals of qemu have resulted in things being connected in a different order. The output of a Windows `lspci` tool is also included. Could anyone make any additional suggestions on how I could try to gain consistency between the different qemu versions? Thanks, James xen 4.14.5 /usr/lib/xen/bin/qemu-system-i386 -xen-domid 19 -no-shutdown -chardev socket,id=3Dlibxl-cmd,fd=3D19,server,nowait -S=20 -mon chardev=3Dlibxl-cmd,mode=3Dcontrol -chardev socket,id=3Dlibxenstat-cmd,path=3D/var/run/xen/qmp-libxenstat-19= ,server,nowait -mon chardev=3Dlibxenstat-cmd,mode=3Dcontrol -nodefaults -no-user-config -name -vnc 0.0.0.0:93 -display none -k en-us -spice port=3D35993,tls-port=3D0,addr=3D127.0.0.1,disable-ticketing,agent= -mouse=3Don,disable-copy-paste,image-compression=3Dauto_glz=20 -device virtio-serial -chardev spicevmc,id=3Dvdagent,name=3Dvdagent -device virtserialport,chardev=3Dvdagent,name=3Dcom.redhat.spice.0 -device VGA,vgamem_mb=3D16 -boot order=3Dcn -usb -usbdevice tablet -soundhw hda -smp 2,maxcpus=3D2 -device rtl8139,id=3Dnic0,netdev=3Dnet0,mac=3D00:16:3e:64:c8:68 -netdev type=3Dtap,id=3Dnet0,ifname=3Dvif19.0-emu,script=3Dno,downscript= =3Dno -object tls-creds-x509,id=3Dtls0,endpoint=3Dclient,dir=3D/etc/certificate= s/usbredir,verify-peer=3Dyes -chardev socket,id=3Dcharredir_serial0,host=3D127.0.0.1,port=3D48052,reco= nnect=3D2,nodelay,keepalive=3Don,user-timeout=3D5 -device isa-serial,chardev=3Dcharredir_serial0 -chardev socket,id=3Dcharredir_serial1,host=3D127.0.0.1,port=3D48054,reco= nnect=3D2,nodelay,keepalive=3Don,user-timeout=3D5 -device isa-serial,chardev=3Dcharredir_serial1 -chardev socket,id=3Dcharredir_serial2,host=3D127.0.0.1,port=3D48055,reco= nnect=3D2,nodelay,keepalive=3Don,user-timeout=3D5 -device pci-serial,chardev=3Dcharredir_serial2 -trace events=3D/etc/xen/qemu-trace-options -machine xenfv -m 2032 -drive file=3D/dev/drbd1002,if=3Dide,index=3D0,media=3Ddisk,format=3Draw,= cache=3Dwriteback -drive file=3D/dev/drbd1003,if=3Dide,index=3D1,media=3Ddisk,format=3Draw,= cache=3Dwriteback -runas 131091:131072 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton I= I] 00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton = II] (rev 01) 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) 00:02.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 0= 1) 00:03.0 Audio device: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family)= High Definition Audio Controller (rev 01) 00:04.0 Communication controller: Red Hat, Inc Virtio console 00:05.0 VGA compatible controller: Device 1234:1111 (rev 02) 00:07.0 Serial controller: Red Hat, Inc. QEMU PCI 16550A Adapter (rev 01) xen 4.15.5 /usr/lib/xen/bin/qemu-system-i386 -xen-domid 15 -no-shutdown -chardev socket,id=3Dlibxl-cmd,fd=3D19,server=3Don,wait=3Doff -S -mon chardev=3Dlibxl-cmd,mode=3Dcontrol -chardev socket,id=3Dlibxenstat-cmd,path=3D/var/run/xen/qmp-libxenstat-15= ,server=3Don,wait=3Doff -mon chardev=3Dlibxenstat-cmd,mode=3Dcontrol -nodefaults -no-user-config -name -vnc 0.0.0.0:93 -display none -k en-us -spice port=3D35993,tls-port=3D0,addr=3D127.0.0.1,disable-ticketing=3Don,= agent-mouse=3Don,disable-copy-paste=3Don,image-compression=3Dauto_glz -device virtio-serial -chardev spicevmc,id=3Dvdagent,name=3Dvdagent -device virtserialport,chardev=3Dvdagent,name=3Dcom.redhat.spice.0 -device VGA,vgamem_mb=3D16 -boot order=3Dcn -usb -usbdevice tablet -device intel-hda -device hda-duplex -smp 2,maxcpus=3D2 -device rtl8139,id=3Dnic0,netdev=3Dnet0,mac=3D00:16:3e:64:c8:68 -netdev type=3Dtap,id=3Dnet0,ifname=3Dvif15.0-emu,br=3Dwan0-00,script=3Dn= o,downscript=3Dno -object tls-creds-x509,id=3Dtls0,endpoint=3Dclient,dir=3D/etc/certificate= s/usbredir,verify-peer=3Dyes -chardev socket,id=3Dcharredir_serial0,host=3D127.0.0.1,port=3D48052,reco= nnect=3D2,nodelay,keepalive=3Don,user-timeout=3D5 -device isa-serial,chardev=3Dcharredir_serial0 -chardev socket,id=3Dcharredir_serial1,host=3D127.0.0.1,port=3D48054,reco= nnect=3D2,nodelay,keepalive=3Don,user-timeout=3D5 -device isa-serial,chardev=3Dcharredir_serial1 -chardev socket,id=3Dcharredir_serial2,host=3D127.0.0.1,port=3D48055,reco= nnect=3D2,nodelay,keepalive=3Don,user-timeout=3D5 -device pci-serial,chardev=3Dcharredir_serial2 -trace events=3D/etc/xen/qemu-trace-options -machine xenfv,suppress-vmdes= c=3Don -m 2032 -drive file=3D/dev/drbd1002,if=3Dide,index=3D0,media=3Ddisk,format=3Draw,= cache=3Dwriteback -drive file=3D/dev/drbd1003,if=3Dide,index=3D1,media=3Ddisk,format=3Draw,= cache=3Dwriteback -runas 131087:131072 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02) 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton I= I] 00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton = II] (rev 01) 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03) 00:02.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 0= 1) 00:03.0 Communication controller: Red Hat, Inc Virtio console 00:04.0 VGA compatible controller: Device 1234:1111 (rev 02) 00:05.0 Audio device: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family)= High Definition Audio Controller (rev 01) 00:07.0 Serial controller: Red Hat, Inc. QEMU PCI 16550A Adapter (rev 01)