From nobody Sat Apr 11 20:14:09 2026 Delivered-To: importer@patchew.org Authentication-Results: mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass(p=reject dis=none) header.from=linux.ibm.com ARC-Seal: i=1; a=rsa-sha256; t=1775097102; cv=none; d=zohomail.com; s=zohoarc; b=OGNik/HPUknR+oZIysRAZJy16tncpj2JTf0zjnKPgA8du4qVI4+TkfGZ1ALRDicSLA58469gd4j1MMEPFAZ+AH+Zfmy+MME7me00bHqrGtUKavC+PEP0MLciBVEP4c/CS4yX6LbutDuULj8R6MQAtkv44yDw2vbh6cxoHVsN/JY= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1775097102; h=Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:List-Subscribe:List-Post:List-Id:List-Archive:List-Help:List-Unsubscribe:MIME-Version:Message-ID:References:Sender:Subject:Subject:To:To:Message-Id:Reply-To; bh=t5cVYkxBAOpmPSSXdFdsVxwtAJBjRxBXKstytrtL+qc=; b=h81hagxdmJ+G5wpGlPnLRwv/QTJacEQcNRR+k4eSKilJmVjxvAhuGE9Ua5Igi/IQ7icNKFEkjJTLvDeJivQgmnxinROykZctHY3ks3Xrd6fvDAADqi2U6mblWi92EvTkbl0MrvscdGeICia4gUv+dawyq0n4ROKvSkNAdjWS5KI= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass; spf=pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=qemu-devel-bounces+importer=patchew.org@nongnu.org; dmarc=pass header.from= (p=reject dis=none) Return-Path: Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) by mx.zohomail.com with SMTPS id 1775097102646962.3616047560287; Wed, 1 Apr 2026 19:31:42 -0700 (PDT) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1w87pU-0004kQ-E2; Wed, 01 Apr 2026 22:30:44 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1w87ov-0004Eh-6k; Wed, 01 Apr 2026 22:30:10 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1w87ot-0001yn-1z; Wed, 01 Apr 2026 22:30:08 -0400 Received: from pps.filterd (m0360083.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 631M4hsB560642; Thu, 2 Apr 2026 02:30:03 GMT Received: from ppma11.dal12v.mail.ibm.com (db.9e.1632.ip4.static.sl-reverse.com [50.22.158.219]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4d66g231pu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 02 Apr 2026 02:30:02 +0000 (GMT) Received: from pps.filterd (ppma11.dal12v.mail.ibm.com [127.0.0.1]) by ppma11.dal12v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 6321iueg008698; Thu, 2 Apr 2026 02:30:02 GMT Received: from smtprelay03.dal12v.mail.ibm.com ([172.16.1.5]) by ppma11.dal12v.mail.ibm.com (PPS) with ESMTPS id 4d6v11r449-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 02 Apr 2026 02:30:02 +0000 Received: from smtpav06.wdc07v.mail.ibm.com (smtpav06.wdc07v.mail.ibm.com [10.39.53.233]) by smtprelay03.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 6322U03N29491796 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 2 Apr 2026 02:30:01 GMT Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C26C8580F4; Thu, 2 Apr 2026 02:30:00 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 39DB3580F6; Thu, 2 Apr 2026 02:30:00 +0000 (GMT) Received: from WIN-DU0DFC9G5VV.ibm.com (unknown [9.61.247.119]) by smtpav06.wdc07v.mail.ibm.com (Postfix) with ESMTP; Thu, 2 Apr 2026 02:30:00 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=t5cVYkxBAOpmPSSXd FdsVxwtAJBjRxBXKstytrtL+qc=; b=U9h/5jsV5fZ38DkQM8nGtw9voZEy30wuf 0l+2LXyUOLfC7OoA7MIhgM2zZf+CgBRyqfWru2PoItC7qkI+MoY6FRB7zkNyVBIK xVSmnP/7N9YxHae6gySPuRhmPsllgxCG1Pq8J1jrU7+nvWu0mBjGyD33xulJ/3Ly LqhNfXVJkFyCz/fnUzTaQxfDjRIHWKcsf8OjYZEKnxzjZDiBdCakNdJLknSCxGPe 4Id87xLeVdJhOTAuFKjfD8YFvQOMxVYM5RqFL5ZNyCAXoh09tzRU40HyxIpokV+F ePuoJ0puOmtIG/Kn/K5auhVOH7ZljY2wD3UBnvqDhfuz6smy3xiZA== From: Konstantin Shkolnyy To: mjrosato@linux.ibm.com Cc: alifm@linux.ibm.com, richard.henderson@linaro.org, iii@linux.ibm.com, david@kernel.org, cohuck@redhat.com, pasic@linux.ibm.com, borntraeger@linux.ibm.com, qemu-s390x@nongnu.org, qemu-devel@nongnu.org, Konstantin Shkolnyy Subject: [PATCH 14/15] s390x/pci: Implement migration for emulated devices Date: Wed, 1 Apr 2026 21:29:20 -0500 Message-Id: <20260402022921.298818-15-kshk@linux.ibm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20260402022921.298818-1-kshk@linux.ibm.com> References: <20260402022921.298818-1-kshk@linux.ibm.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Authority-Analysis: v=2.4 cv=Fdo6BZ+6 c=1 sm=1 tr=0 ts=69cdd4aa cx=c_pps a=aDMHemPKRhS1OARIsFnwRA==:117 a=aDMHemPKRhS1OARIsFnwRA==:17 a=A5OVakUREuEA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=iQ6ETzBq9ecOQQE5vZCe:22 a=VnNF1IyMAAAA:8 a=1WZqjCYO3q-P4G2YYo4A:9 X-Proofpoint-ORIG-GUID: oRdHDHgOv2BWWHXZ973_1EsBb1UfJMUD X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNDAyMDAxOSBTYWx0ZWRfX/Z0hZxiogMlK A07AO9zBdtKJuQnFveUdp1kP61DHQXVuKKNktKgcnvY/TMphbSfrJNa1j42thT6A1xFJ+cAZ1P2 egPQiQNpwJw+JkjEbO+OFZGztnMme9pfTR3CscTyxIBkz6ylcmQAi/vC1zG4+62SFlBaolL0VS3 HVkSteYwL9CvIJJvWxL2B5efwizzmveWZDUY6xp/XTvhZjyN2DXaRG4ETAEKmdC65+DKMFh9UxF jHV6zik3fGoxkewFMgfiP562g1y06s0TRA87+yj1Y3Jx6MV9gxsDayG7yA4lU4GneHORj6PIpwg ObekpDNmtwolZk0lcLUOz97F5ZLOP6vOk4MsRn6b51bWwroK+9Xvr26+Jp3OC6MufqoOZKBZF76 k0BHOplINqx+neIQgE/9egWJ9JshsvQNg0DXJcNyFFumCb/Fs6iB2xvPQAebnfeej3ptD8fiakj nMY2Eh28lzuZVB1U0HA== X-Proofpoint-GUID: oRdHDHgOv2BWWHXZ973_1EsBb1UfJMUD X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.51,FMLib:17.12.100.49 definitions=2026-04-01_05,2026-04-01_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 clxscore=1015 bulkscore=0 suspectscore=0 priorityscore=1501 adultscore=0 malwarescore=0 phishscore=0 spamscore=0 lowpriorityscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2603050001 definitions=main-2604020019 Received-SPF: pass (zohomail.com: domain of gnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; envelope-from=qemu-devel-bounces+importer=patchew.org@nongnu.org; helo=lists.gnu.org; Received-SPF: pass client-ip=148.163.156.1; envelope-from=kshk@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -6 X-Spam_score: -0.7 X-Spam_bar: / X-Spam_report: (-0.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_VALIDITY_CERTIFIED_BLOCKED=1, RCVD_IN_VALIDITY_RPBL_BLOCKED=1, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: qemu development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+importer=patchew.org@nongnu.org Sender: qemu-devel-bounces+importer=patchew.org@nongnu.org X-ZohoMail-DKIM: pass (identity @ibm.com) X-ZM-MESSAGEID: 1775097105268154100 Content-Type: text/plain; charset="utf-8" Implement zPCI device state migration, consequently enabling migration of VMs that have emulated PCI devices, whether virtio or not. Migration is allowed for devices whose function handle has the FH_SHM_EMUL bit set. For these devices QEMU will save and restore the state of its zPCI emulator. Passthrough devices will continue to block migration. Signed-off-by: Konstantin Shkolnyy --- hw/s390x/s390-pci-bus.c | 112 ++++++++++++++++++++++++++++++-- hw/s390x/s390-pci-inst.c | 2 +- include/hw/s390x/s390-pci-bus.h | 2 + 3 files changed, 110 insertions(+), 6 deletions(-) diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c index f339a0fd94..b3037b9a8a 100644 --- a/hw/s390x/s390-pci-bus.c +++ b/hw/s390x/s390-pci-bus.c @@ -26,6 +26,7 @@ #include "hw/pci/pci_bridge.h" #include "hw/pci/msi.h" #include "exec/cpu-common.h" +#include "migration/blocker.h" #include "qemu/error-report.h" #include "qemu/module.h" #include "system/reset.h" @@ -1116,6 +1117,28 @@ static int s390_pci_interp_plug(S390pciState *s, S39= 0PCIBusDevice *pbdev) return 0; } =20 +static int s390_set_passthrough_migration_blocker(S390PCIBusDevice *pbdev, + Error **errp) +{ + pbdev->passthrough_migr_blocker =3D NULL; + + if (pbdev->fh & FH_SHM_EMUL) { + return 0; + } + error_setg(&pbdev->passthrough_migr_blocker, + "Migration blocked by passthrough zPCI device " + "fh 0x%x uid %d fid %d", pbdev->fh, pbdev->uid, pbdev->fid); + + return migrate_add_blocker(&pbdev->passthrough_migr_blocker, errp); +} + +static void s390_clear_passthrough_migration_blocker(S390PCIBusDevice *pbd= ev) +{ + if (pbdev->passthrough_migr_blocker) { + migrate_del_blocker(&pbdev->passthrough_migr_blocker); + } +} + static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *de= v, Error **errp) { @@ -1235,6 +1258,11 @@ static void s390_pcihost_plug(HotplugHandler *hotplu= g_dev, DeviceState *dev, return; } =20 + if (s390_set_passthrough_migration_blocker(pbdev, errp)) { + s390_pci_msix_free(pbdev); + return; + } + if (dev->hotplugged) { s390_pci_generate_plug_event(HP_EVENT_TO_CONFIGURED , pbdev->fh, pbdev->fid); @@ -1271,6 +1299,8 @@ static void s390_pcihost_unplug(HotplugHandler *hotpl= ug_dev, DeviceState *dev, return; } =20 + s390_clear_passthrough_migration_blocker(pbdev); + s390_pci_generate_plug_event(HP_EVENT_STANDBY_TO_RESERVED, pbdev->fh, pbdev->fid); bus =3D pci_get_bus(pci_dev); @@ -1600,13 +1630,85 @@ static const Property s390_pci_device_properties[] = =3D { true), }; =20 -static const VMStateDescription s390_pci_device_vmstate =3D { - .name =3D TYPE_S390_PCI_DEVICE, +static int s390_pci_device_post_load(void *opaque, int version_id) +{ + S390PCIBusDevice *pbdev =3D S390_PCI_DEVICE(opaque); + + /* + * Now that S390PCIBusDevice fields have been restored, regenerate IOM= MU + * state - that includes IOTLB contents and QEMU memory regions. + */ + if (pbdev->iommu_enabled) { + assert(pbdev->iommu); + if (s390_pci_is_translation_enabled(pbdev->g_iota)) { + s390_pci_iommu_enable(pbdev); + s390_pci_ioat_replay(pbdev); + } else { + s390_pci_iommu_direct_map_enable(pbdev); + } + } + /* - * TODO: add state handling here, so migration works at least with - * emulated pci devices on s390x + * Guest sets fmb_addr by mpcifc.ZPCI_MOD_FC_SET_MEASURE instruction. + * The handler consequently starts fmb_timer. Now that fmb_addr has be= en + * restored, we may need to restart fmb_timer. */ - .unmigratable =3D 1, + if (pbdev->fmb_addr) { + assert(!pbdev->fmb_timer); + assert(pbdev->pci_group); + pbdev->fmb_timer =3D timer_new_ms(QEMU_CLOCK_VIRTUAL, + fmb_update, pbdev); + timer_mod(pbdev->fmb_timer, + qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL) + + pbdev->pci_group->zpci_group.mui); + } + return 0; +} + +static const VMStateDescription s390_pci_device_vmstate =3D { + .name =3D TYPE_S390_PCI_DEVICE, + .version_id =3D 1, + .minimum_version_id =3D 1, + .post_load =3D s390_pci_device_post_load, + .fields =3D (const VMStateField[]) { + VMSTATE_UINT32(state, S390PCIBusDevice), + VMSTATE_UINT16(uid, S390PCIBusDevice), + VMSTATE_UINT32(idx, S390PCIBusDevice), + VMSTATE_UINT32(fh, S390PCIBusDevice), + VMSTATE_UINT32(fid, S390PCIBusDevice), + VMSTATE_BOOL(fid_defined, S390PCIBusDevice), + VMSTATE_UINT64(fmb_addr, S390PCIBusDevice), + VMSTATE_UINT32(fmb.format, S390PCIBusDevice), + VMSTATE_UINT32(fmb.sample, S390PCIBusDevice), + VMSTATE_UINT64(fmb.last_update, S390PCIBusDevice), + VMSTATE_UINT64_ARRAY(fmb.counter, S390PCIBusDevice, + ARRAY_SIZE(((S390PCIBusDevice *)0)->fmb.counter)), + VMSTATE_UINT64(fmb.fmt0.dma_rbytes, S390PCIBusDevice), + VMSTATE_UINT64(fmb.fmt0.dma_wbytes, S390PCIBusDevice), + VMSTATE_UINT8(isc, S390PCIBusDevice), + VMSTATE_UINT16(noi, S390PCIBusDevice), + VMSTATE_UINT8(sum, S390PCIBusDevice), + VMSTATE_UINT8(pft, S390PCIBusDevice), + VMSTATE_UINT64(routes.adapter.ind_addr, S390PCIBusDevice), + VMSTATE_UINT64(routes.adapter.summary_addr, S390PCIBusDevice), + VMSTATE_UINT64(routes.adapter.ind_offset, S390PCIBusDevice), + VMSTATE_UINT32(routes.adapter.summary_offset, S390PCIBusDevice), + VMSTATE_UINT32(routes.adapter.adapter_id, S390PCIBusDevice), + VMSTATE_BOOL(iommu_enabled, S390PCIBusDevice), + VMSTATE_UINT64(g_iota, S390PCIBusDevice), + VMSTATE_UINT64(pba, S390PCIBusDevice), + VMSTATE_UINT64(pal, S390PCIBusDevice), + VMSTATE_UINT64(max_dma_limit, S390PCIBusDevice), + VMSTATE_PTR_TO_IND_ADDR(summary_ind, S390PCIBusDevice), + VMSTATE_PTR_TO_IND_ADDR(indicator, S390PCIBusDevice), + VMSTATE_BOOL(pci_unplug_request_processed, S390PCIBusDevice), + VMSTATE_BOOL(unplug_requested, S390PCIBusDevice), + VMSTATE_BOOL(interp, S390PCIBusDevice), + VMSTATE_BOOL(forwarding_assist, S390PCIBusDevice), + VMSTATE_BOOL(aif, S390PCIBusDevice), + VMSTATE_BOOL(rtr_avail, S390PCIBusDevice), + VMSTATE_END_OF_LIST() + } }; =20 static void s390_pci_device_class_init(ObjectClass *klass, const void *dat= a) diff --git a/hw/s390x/s390-pci-inst.c b/hw/s390x/s390-pci-inst.c index 191f549ba7..a9c0a4effb 100644 --- a/hw/s390x/s390-pci-inst.c +++ b/hw/s390x/s390-pci-inst.c @@ -1094,7 +1094,7 @@ static int fmb_do_update(S390PCIBusDevice *pbdev, int= offset, uint64_t val, return ret; } =20 -static void fmb_update(void *opaque) +void fmb_update(void *opaque) { S390PCIBusDevice *pbdev =3D opaque; int64_t t =3D qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL); diff --git a/include/hw/s390x/s390-pci-bus.h b/include/hw/s390x/s390-pci-bu= s.h index 2fab28e6e0..2edb023112 100644 --- a/include/hw/s390x/s390-pci-bus.h +++ b/include/hw/s390x/s390-pci-bus.h @@ -337,6 +337,7 @@ struct S390PCIBusDevice { uint16_t uid; uint32_t idx; uint32_t fh; + Error *passthrough_migr_blocker; uint32_t fid; bool fid_defined; uint64_t fmb_addr; @@ -415,5 +416,6 @@ S390PCIBusDevice *s390_pci_find_dev_by_pci(S390pciState= *s, S390PCIBusDevice *s390_pci_find_next_avail_dev(S390pciState *s, S390PCIBusDevice *pbdev); void s390_pci_ism_reset(void); +void fmb_update(void *opaque); =20 #endif --=20 2.34.1