From nobody Sun Jun 14 07:35:01 2026 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EDB323F787E; Fri, 1 May 2026 19:26:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777663608; cv=none; b=OPwp8VPa+9ANPiQK+sVtTM7OM8U/xn7eAvlr3XiFk7QosLRzQjYBnomaYpf6cL2oWV8uxqWxxr5bPtKq+h4Tl3BIRPnivHdPrHS+ndAga5qorZ+SRe3vkBNqQ6mUhjJllS74SnBiCVw/o38HysY5JYrb0FPR/GI/GLVXiXgJKAE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777663608; c=relaxed/simple; bh=RWHLEYTvACOIkJcpQeZF7m+uxDKtXoN4/UoRkmCEtfI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=DfM+a30p+A5JQ5d6U+FEmTi72v27z/qafPDs0S18H+1hryOezzs2BJanz1cIumxqs8JRNLbLGu6u3zvgCEBGx9ThOsvd7oWT4QjbIIINKG5tc/uZPFwOL1VUeCIyT0igdIlXC9RKuOMtbfqAymD58+80bYLNofTWAEVQBcgXJGA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=VnssU3Y9; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="VnssU3Y9" Received: from pps.filterd (m0360083.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 6417ctWb3961718; Fri, 1 May 2026 19:26:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=6dhk6oPMjcd/tLceq t102Cix8j/Hvs1RPLzVjDwP/Ro=; b=VnssU3Y9dDQn1y+Sk3L8p9shyBkT45JUw h9wbeI4mbTPdc/+CRbqxFuBXNUd7q7nU5Dm7dgllysJX43IaKyxaTTvNJOpIoiqV lpNVxV1AxdHMHH6AMdNsn/mXqhtm89rP8kp6VxWnw0RNwwk+mUK3yQ62TbC1PcfE Ol4LWx1GEI30xG7ZM8cBExnks6zjklEuQub/xaw3NsIrNZFgrO1/bRRg5rSuZVZF 5Xmbcsg4jPLEIwLfCwk39UKiPwKu+/26IZiJI8NJ2lbwkowx3LqY9QK6giSwP4F7 J5lTPnGzlKlwUMaHt+a5JoNyZbl1uj8WYXQZ6Paq2Nwivhqpt0DvA== Received: from ppma12.dal12v.mail.ibm.com (dc.9e.1632.ip4.static.sl-reverse.com [50.22.158.220]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4drn4568ak-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 01 May 2026 19:26:44 +0000 (GMT) Received: from pps.filterd (ppma12.dal12v.mail.ibm.com [127.0.0.1]) by ppma12.dal12v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 641JNtHv013160; Fri, 1 May 2026 19:26:43 GMT Received: from smtprelay05.wdc07v.mail.ibm.com ([172.16.1.72]) by ppma12.dal12v.mail.ibm.com (PPS) with ESMTPS id 4ds7xqryda-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 01 May 2026 19:26:43 +0000 (GMT) Received: from smtpav05.wdc07v.mail.ibm.com (smtpav05.wdc07v.mail.ibm.com [10.39.53.232]) by smtprelay05.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 641JQd4S18219632 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 1 May 2026 19:26:40 GMT Received: from smtpav05.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E212958043; Fri, 1 May 2026 19:26:39 +0000 (GMT) Received: from smtpav05.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4120158053; Fri, 1 May 2026 19:26:38 +0000 (GMT) Received: from Mac.ibm.com (unknown [9.61.247.25]) by smtpav05.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 1 May 2026 19:26:38 +0000 (GMT) From: Omar Elghoul To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: oelghoul@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, schnelle@linux.ibm.com, mjrosato@linux.ibm.com, alifm@linux.ibm.com, farman@linux.ibm.com, gbayer@linux.ibm.com, alex@shazbot.org Subject: [PATCH v1 1/3] s390/pci: Preserve FMB state in device re-enablement Date: Fri, 1 May 2026 15:25:28 -0400 Message-ID: <20260501192530.9429-2-oelghoul@linux.ibm.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260501192530.9429-1-oelghoul@linux.ibm.com> References: <20260501192530.9429-1-oelghoul@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: odcMXheFGBk5onxttqMqV8hLu7h7lQXo X-Authority-Analysis: v=2.4 cv=Ft81OWrq c=1 sm=1 tr=0 ts=69f4fe74 cx=c_pps a=bLidbwmWQ0KltjZqbj+ezA==:117 a=bLidbwmWQ0KltjZqbj+ezA==:17 a=NGcC8JguVDcA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=iQ6ETzBq9ecOQQE5vZCe:22 a=VnNF1IyMAAAA:8 a=sdm-MloC9ubkM-Nm6RcA:9 a=0bXxn9q0MV6snEgNplNhOjQmxlI=:19 X-Proofpoint-GUID: odcMXheFGBk5onxttqMqV8hLu7h7lQXo X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNTAxMDE5MCBTYWx0ZWRfXw/EwYTVk+90b BCOhoBxNFYWGYrCA7C9W12YNAVIkLBmUgdZ/IQ1wcW2tm7nIy38LSdcWOAg4bm1KMmmk2KVHIZN Qwx0XBjPAmOI6u+XqeCKctw9K/twgxnxlJvhoZrwyIOtHZHbIHEp2lUdtl8U5B6pGs1wZ81o65+ md+SmQ9kpRz+a4ulVoIDfDRPHq8rROrR3Q2N16x1PVaO6Xcd4PVbB8hWyugs4GSsjtICYGCCVdS KOgwYWSga9A/5wcz1iTVP919C+DepXLy5PklSBAtKZn0I+syNn6rMIHjLE8LKlViRXJZc9hThy6 /t6P7qtzEdV9cCuaSE0wb7P8fsfeUmtHq878wryvbgDL9dntYI5GB0S6W/csjjheZkfFulDkF1b k9KPj6X88WlOxz5osC5mgNB8p58tKvRO+fOAQ+UPENvXzC+S6dux41P3jJRg+OWppgCS1IQIVjS i98lBvh6Z+zUTIhKFSA== X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.51,FMLib:17.12.100.49 definitions=2026-05-01_05,2026-04-30_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 lowpriorityscore=0 bulkscore=0 spamscore=0 impostorscore=0 clxscore=1011 malwarescore=0 phishscore=0 suspectscore=0 adultscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2604200000 definitions=main-2605010190 Content-Type: text/plain; charset="utf-8" Introduce a function zpci_fmb_reenable_device() that checks for the state of the FMB and reuses the same buffer where appropriate. If FMB was not previously enabled, it enables it for the device. Call this function during a zPCI device re-enablement, which in turn implicitly ensures that the FMB is enabled for host devices during their KVM registration. This function also clears out the software counters, so that a program resetting an FMB would see all its counters restart from zero as expected. The function to clear the software counters is also separated into a static function as it is now reused in both zpci_fmb_enable_device() and zpci_fmb_reenable_device(). Signed-off-by: Omar Elghoul Reviewed-by: Niklas Schnelle --- arch/s390/include/asm/pci.h | 1 + arch/s390/pci/pci.c | 71 ++++++++++++++++++++++++++++++------- 2 files changed, 59 insertions(+), 13 deletions(-) diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h index 5dcf35f0f325..65014e52d559 100644 --- a/arch/s390/include/asm/pci.h +++ b/arch/s390/include/asm/pci.h @@ -323,6 +323,7 @@ void zpci_remove_parent_msi_domain(struct zpci_bus *zbu= s); /* FMB */ int zpci_fmb_enable_device(struct zpci_dev *); int zpci_fmb_disable_device(struct zpci_dev *); +int zpci_fmb_reenable_device(struct zpci_dev *zdev); =20 /* Debug */ int zpci_debug_init(void); diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c index 39bd2adfc240..9bc38e041130 100644 --- a/arch/s390/pci/pci.c +++ b/arch/s390/pci/pci.c @@ -164,6 +164,24 @@ int zpci_unregister_ioat(struct zpci_dev *zdev, u8 dma= as) return cc; } =20 +static void zpci_fmb_clear_iommu_ctrs(struct zpci_dev *zdev) +{ + struct zpci_iommu_ctrs *ctrs; + unsigned long flags =3D 0; + + /* reset software counters */ + spin_lock_irqsave(&zdev->dom_lock, flags); + ctrs =3D zpci_get_iommu_ctrs(zdev); + if (ctrs) { + atomic64_set(&ctrs->mapped_pages, 0); + atomic64_set(&ctrs->unmapped_pages, 0); + atomic64_set(&ctrs->global_rpcits, 0); + atomic64_set(&ctrs->sync_map_rpcits, 0); + atomic64_set(&ctrs->sync_rpcits, 0); + } + spin_unlock_irqrestore(&zdev->dom_lock, flags); +} + /* Modify PCI: Set PCI function measurement parameters */ int zpci_fmb_enable_device(struct zpci_dev *zdev) { @@ -181,18 +199,7 @@ int zpci_fmb_enable_device(struct zpci_dev *zdev) return -ENOMEM; WARN_ON((u64) zdev->fmb & 0xf); =20 - /* reset software counters */ - spin_lock_irqsave(&zdev->dom_lock, flags); - ctrs =3D zpci_get_iommu_ctrs(zdev); - if (ctrs) { - atomic64_set(&ctrs->mapped_pages, 0); - atomic64_set(&ctrs->unmapped_pages, 0); - atomic64_set(&ctrs->global_rpcits, 0); - atomic64_set(&ctrs->sync_map_rpcits, 0); - atomic64_set(&ctrs->sync_rpcits, 0); - } - spin_unlock_irqrestore(&zdev->dom_lock, flags); - + zpci_fmb_clear_iommu_ctrs(zdev); =20 fib.fmb_addr =3D virt_to_phys(zdev->fmb); fib.gd =3D zdev->gisa; @@ -227,6 +234,41 @@ int zpci_fmb_disable_device(struct zpci_dev *zdev) } return cc ? -EIO : 0; } +EXPORT_SYMBOL_GPL(zpci_fmb_disable_device); + +int zpci_fmb_reenable_device(struct zpci_dev *zdev) +{ + u64 req =3D ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_SET_MEASURE); + struct zpci_iommu_ctrs *ctrs; + struct zpci_fib fib =3D {0}; + unsigned long flags; + u8 cc, status; + + if (!zdev->fmb) + return zpci_fmb_enable_device(zdev); + + fib.gd =3D zdev->gisa; + cc =3D zpci_mod_fc(req, &fib, &status); /* Disable function measurement */ + + /* Unlike in zpci_fmb_disable_device(), cc =3D=3D 3 is not a valid state = here + * because we are re-enabling function measurement for the same function + * handle. + */ + if (cc) + return -EIO; + + zpci_fmb_clear_iommu_ctrs(zdev); + + fib.fmb_addr =3D virt_to_phys(zdev->fmb); + cc =3D zpci_mod_fc(req, &fib, &status); /* Re-enable function measurement= */ + if (cc) { + kmem_cache_free(zdev_fmb_cache, zdev->fmb); + zdev->fmb =3D NULL; + return -EIO; + } + return 0; +} +EXPORT_SYMBOL_GPL(zpci_fmb_reenable_device); =20 static int zpci_cfg_load(struct zpci_dev *zdev, int offset, u32 *val, u8 l= en) { @@ -729,9 +771,12 @@ int zpci_reenable_device(struct zpci_dev *zdev) } =20 rc =3D zpci_iommu_register_ioat(zdev, &status); - if (rc) + if (rc) { zpci_disable_device(zdev); + return rc; + } =20 + zpci_fmb_reenable_device(zdev); return rc; } EXPORT_SYMBOL_GPL(zpci_reenable_device); --=20 2.52.0 From nobody Sun Jun 14 07:35:01 2026 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D15541C2FF; Fri, 1 May 2026 19:27:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777663631; cv=none; b=ciQYp0iZWL5hygl3ZvbSFHanihzcN+gc3Ribkm+xm6TzPTd637NGjI47ap1I9L0MkV1j8Dtibv5dUFyYsycb1n0lj5RSZnpkvfduNG5IKBRtrxeFTU9iyqYgXufX/m+GKYsgxu6w9Sztu4bGyj64AuFVgVnolW5y+Th3MSIFljI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777663631; c=relaxed/simple; bh=05Y7MgNIeyX9AUD2z6JcZ4EOO1ulTxQAE361xC5/bSI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YeeAizb9Hb9HXF7K0TwQeUXcde8E91f41hPeCquj97LSsOD73tvOIM5WVWO1irjzxKznxFSbl3F6HMpcAWJkrAiz5r21is+sdWsmif5U+7MBy0WM/OipEtN6UDIsQvm+oq2RgF+YIcuLKhyIEew/ccIXQF44bKNg5fCf7l2lWvk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=IN6x86/O; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="IN6x86/O" Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 641FFWCX2766617; Fri, 1 May 2026 19:27:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=48W2QeYenMYp1w8Jg 4KOVhhMg71kyh3w/0Kg1u+Sw/w=; b=IN6x86/Oq4JF4FOhwtxteup6o0z4nRLYq 2xLYzxkwEmtUo9XcGnD+0rOgZ1MFhko9KfrOyqfHTJI5Q5GI96OZIRuvi6oMxDtc 1opo2h+WX2CDvv3r3uuWJjjxdLOh7vqUq6ls6VCTtoQVEjqQfNK+MeKkj+9MP8FM 8c5k9NQKbKSIUiTCFMqrEsNKO1/MhG7XJUari48l/0WtI5h9DgH2LwvEGWATBfh/ cW7kIugVXWrfTwSbICMaPTQXe68pVWaai+kFXdg9bC5nDwBFBlXKZgjxKqVjgTFe p5krSsU4GHwz75tV9TV+XyAFtpW6sbNeI9FHvQvOxRIqjwlaRKvsQ== Received: from ppma21.wdc07v.mail.ibm.com (5b.69.3da9.ip4.static.sl-reverse.com [169.61.105.91]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4drn9rp2vf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 01 May 2026 19:27:07 +0000 (GMT) Received: from pps.filterd (ppma21.wdc07v.mail.ibm.com [127.0.0.1]) by ppma21.wdc07v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 641JNjn2006833; Fri, 1 May 2026 19:27:06 GMT Received: from smtprelay03.wdc07v.mail.ibm.com ([172.16.1.70]) by ppma21.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4ds8xkgujn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 01 May 2026 19:27:06 +0000 (GMT) Received: from smtpav05.wdc07v.mail.ibm.com (smtpav05.wdc07v.mail.ibm.com [10.39.53.232]) by smtprelay03.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 641JQaKk54722922 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 1 May 2026 19:26:36 GMT Received: from smtpav05.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9338558053; Fri, 1 May 2026 19:27:03 +0000 (GMT) Received: from smtpav05.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E55D858043; Fri, 1 May 2026 19:27:01 +0000 (GMT) Received: from Mac.ibm.com (unknown [9.61.247.25]) by smtpav05.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 1 May 2026 19:27:01 +0000 (GMT) From: Omar Elghoul To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: oelghoul@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, schnelle@linux.ibm.com, mjrosato@linux.ibm.com, alifm@linux.ibm.com, farman@linux.ibm.com, gbayer@linux.ibm.com, alex@shazbot.org Subject: [PATCH v1 2/3] vfio-pci/zdev: Add VFIO FMB device feature Date: Fri, 1 May 2026 15:25:29 -0400 Message-ID: <20260501192530.9429-3-oelghoul@linux.ibm.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260501192530.9429-1-oelghoul@linux.ibm.com> References: <20260501192530.9429-1-oelghoul@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-GUID: pWHRyhQxLrGFaO1OrcM-lvFk2gIopmpd X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNTAxMDE5MCBTYWx0ZWRfX6RYOw1hmaRMD uYAs7JO8H6i5ORUizu/iSw4v6rj9xFdBteQCK/NHEPK+XA3n3Dz6KNeAvWbJ4tn+IJh1DiuiqIz 8k9Bjpd2kjecVwg/XDlUdkyfJxuCeZF1eaiIM8RdIEsNEfLIAtnFxslukJpx5x/pfu4jopZktbC oYiJGSPtE2w/qxqSgN9j5Qm5gWAhuRp+uZj7B5k/x/3Mn62nQ2i+mEF+lY/eSQR4G55g/N8wIR0 e6QzP1wUWg6oPeVxJHs6wkIOuz/wzTL/4vdb5xYLiDgAaI7Gq8Ad+4jrzG9IWVA+AAT4nOfCknh 7s5jYcH2ULpuj0nUsxNEopT8+nDh/2tv8/5NbvtJWp9qtfd3LCvI2yqsnTDQzSZPNE1hXLW8oV3 uKS3aaOlJoF3LyXcMjA9QzrOYhe37lHMadHqjyHAMpP8hSmzRzbq+sQsNrRnrmRe8oBg24XHGwn X1tz+dXxH7ZbJfE1v3w== X-Authority-Analysis: v=2.4 cv=Kc7idwYD c=1 sm=1 tr=0 ts=69f4fe8b cx=c_pps a=GFwsV6G8L6GxiO2Y/PsHdQ==:117 a=GFwsV6G8L6GxiO2Y/PsHdQ==:17 a=NGcC8JguVDcA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=uAbxVGIbfxUO_5tXvNgY:22 a=VnNF1IyMAAAA:8 a=qYbwNa5VYoetIufaySgA:9 X-Proofpoint-ORIG-GUID: pWHRyhQxLrGFaO1OrcM-lvFk2gIopmpd X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.51,FMLib:17.12.100.49 definitions=2026-05-01_05,2026-04-30_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1011 phishscore=0 bulkscore=0 adultscore=0 spamscore=0 malwarescore=0 impostorscore=0 priorityscore=1501 lowpriorityscore=0 suspectscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2604200000 definitions=main-2605010190 Content-Type: text/plain; charset="utf-8" Set up a new VFIO feature for zPCI devices to share the latest FMB snapshot with userspace. This feature supports the same 4 FMB formats (0 through 3) that are already supported by the kernel. With VFIO_DEVICE_FEATURE_GET, allow the user driver to read the latest FMB snapshot as well as query whether the FMB is currently enabled on the function, itself indicating whether the FMB snapshot is valid. On the other hand, with VFIO_DEVICE_FEATURE_SET, the userspace driver can enable or disable the FMB. Signed-off-by: Omar Elghoul --- drivers/vfio/pci/vfio_pci_core.c | 2 + drivers/vfio/pci/vfio_pci_priv.h | 9 ++++ drivers/vfio/pci/vfio_pci_zdev.c | 77 ++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 43 ++++++++++++++++++ 4 files changed, 131 insertions(+) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_c= ore.c index 3f8d093aacf8..63e80b6fa0dc 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1534,6 +1534,8 @@ int vfio_pci_core_ioctl_feature(struct vfio_device *d= evice, u32 flags, return vfio_pci_core_feature_token(vdev, flags, arg, argsz); case VFIO_DEVICE_FEATURE_DMA_BUF: return vfio_pci_core_feature_dma_buf(vdev, flags, arg, argsz); + case VFIO_DEVICE_FEATURE_ZPCI_FMB: + return vfio_pci_zdev_feature_fmb(vdev, flags, arg, argsz); default: return -ENOTTY; } diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_p= riv.h index fca9d0dfac90..208e05942b48 100644 --- a/drivers/vfio/pci/vfio_pci_priv.h +++ b/drivers/vfio/pci/vfio_pci_priv.h @@ -93,6 +93,8 @@ int vfio_pci_info_zdev_add_caps(struct vfio_pci_core_devi= ce *vdev, struct vfio_info_cap *caps); int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev); void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev); +int vfio_pci_zdev_feature_fmb(struct vfio_pci_core_device *vdev, u32 flags, + void __user *arg, size_t argsz); #else static inline int vfio_pci_info_zdev_add_caps(struct vfio_pci_core_device = *vdev, struct vfio_info_cap *caps) @@ -107,6 +109,13 @@ static inline int vfio_pci_zdev_open_device(struct vfi= o_pci_core_device *vdev) =20 static inline void vfio_pci_zdev_close_device(struct vfio_pci_core_device = *vdev) {} + +static inline int vfio_pci_zdev_feature_fmb(struct vfio_pci_core_device *v= dev, + u32 flags, void __user *arg, + size_t argsz) +{ + return -ENOTTY; +} #endif =20 static inline bool vfio_pci_is_vga(struct pci_dev *pdev) diff --git a/drivers/vfio/pci/vfio_pci_zdev.c b/drivers/vfio/pci/vfio_pci_z= dev.c index 0990fdb146b7..1e9efe2bee69 100644 --- a/drivers/vfio/pci/vfio_pci_zdev.c +++ b/drivers/vfio/pci/vfio_pci_zdev.c @@ -167,3 +167,80 @@ void vfio_pci_zdev_close_device(struct vfio_pci_core_d= evice *vdev) if (zpci_kvm_hook.kvm_unregister) zpci_kvm_hook.kvm_unregister(zdev); } + +int vfio_pci_zdev_feature_fmb(struct vfio_pci_core_device *vdev, u32 flags, + void __user *arg, size_t argsz) +{ + struct zpci_dev *zdev; + struct vfio_device_feature_zpci_fmb fmb =3D {0}; + u32 ops =3D VFIO_DEVICE_FEATURE_GET | VFIO_DEVICE_FEATURE_SET; + int ret; + + ret =3D vfio_check_feature(flags, argsz, ops, sizeof(fmb)); + if (ret !=3D 1) + return ret; + + zdev =3D to_zpci(vdev->pdev); + if (!zdev) + return -ENODEV; + + mutex_lock(&zdev->fmb_lock); + if (flags & VFIO_DEVICE_FEATURE_SET) { + if (copy_from_user(&fmb, arg, sizeof(fmb))) { + ret =3D -EFAULT; + goto release_lock; + } + + if (fmb.flags & VFIO_DEVICE_FEATURE_ZPCI_FMB_FLAGS_ENABLED) + ret =3D zpci_fmb_reenable_device(zdev); + else + ret =3D zpci_fmb_disable_device(zdev); + goto release_lock; + } + + ret =3D 0; + if (zdev->fmb) { + fmb.flags |=3D VFIO_DEVICE_FEATURE_ZPCI_FMB_FLAGS_ENABLED; + } else { + fmb.flags &=3D ~VFIO_DEVICE_FEATURE_ZPCI_FMB_FLAGS_ENABLED; + goto release_lock; + } + + fmb.format =3D zdev->fmb->format; + fmb.fmt_ind =3D zdev->fmb->fmt_ind; + fmb.samples =3D zdev->fmb->samples; + fmb.last_update =3D zdev->fmb->last_update; + fmb.ld_ops =3D zdev->fmb->ld_ops; + fmb.st_ops =3D zdev->fmb->st_ops; + fmb.stb_ops =3D zdev->fmb->stb_ops; + fmb.rpcit_ops =3D zdev->fmb->rpcit_ops; + + switch (zdev->fmb->format) { + case 0: + if (zdev->fmb->fmt_ind & ZPCI_FMB_DMA_COUNTER_VALID) { + fmb.fmt0.dma_rbytes =3D zdev->fmb->fmt0.dma_rbytes; + fmb.fmt0.dma_wbytes =3D zdev->fmb->fmt0.dma_wbytes; + } + break; + case 1: + fmb.fmt1.rx_bytes =3D zdev->fmb->fmt1.rx_bytes; + fmb.fmt1.rx_packets =3D zdev->fmb->fmt1.rx_packets; + fmb.fmt1.tx_bytes =3D zdev->fmb->fmt1.tx_bytes; + fmb.fmt1.tx_packets =3D zdev->fmb->fmt1.tx_packets; + break; + case 2: + fmb.fmt2.consumed_work_units =3D zdev->fmb->fmt2.consumed_work_units; + fmb.fmt2.max_work_units =3D zdev->fmb->fmt2.max_work_units; + break; + case 3: + fmb.fmt3.tx_bytes =3D zdev->fmb->fmt3.tx_bytes; + break; + } + + if (copy_to_user(arg, &fmb, sizeof(fmb))) + ret =3D -EFAULT; + +release_lock: + mutex_unlock(&zdev->fmb_lock); + return ret; +} diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 5de618a3a5ee..6cbc34ff063e 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -1534,6 +1534,49 @@ struct vfio_device_feature_dma_buf { */ #define VFIO_DEVICE_FEATURE_MIG_PRECOPY_INFOv2 12 =20 +/** + * Upon VFIO_DEVICE_FEATURE_GET, provide FMB passthrough for VFIO zPCI dev= ices. + * + * Upon VFIO_DEVICE_FEATURE_SET, only the flags field is read while the + * remainder of the structure is ignored. This allows the driver to enable= or + * disable the FMB while also leaving reserved bits for future flag expans= ion. + * All reserved fields should be zero for future compatibility. + */ +#define VFIO_DEVICE_FEATURE_ZPCI_FMB 13 +#define VFIO_DEVICE_FEATURE_ZPCI_FMB_FLAGS_ENABLED 0x1 + +struct vfio_device_feature_zpci_fmb { + __u64 flags; + __u32 format: 8; + __u32 fmt_ind: 24; + __u32 samples; + __u64 last_update; + __u64 ld_ops; + __u64 st_ops; + __u64 stb_ops; + __u64 rpcit_ops; + union { + struct { + __u64 dma_rbytes; + __u64 dma_wbytes; + } fmt0; + struct { + __u64 rx_bytes; + __u64 rx_packets; + __u64 tx_bytes; + __u64 tx_packets; + } fmt1; + struct { + __u64 consumed_work_units; + __u64 max_work_units; + } fmt2; + struct { + __u64 tx_bytes; + } fmt3; + }; + __u64 reserved[16]; +}; + /* -------- API for Type1 VFIO IOMMU -------- */ =20 /** --=20 2.52.0 From nobody Sun Jun 14 07:35:01 2026 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CCA8622F74A; Fri, 1 May 2026 19:27:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777663679; cv=none; b=kjQ9lTA50CuPANrbu8pCzG+2+BWmGsHGae4ahXTjbFI7THi4wyLDQgV7k4jL+0w3o9arG443RUXm18CObd3VaNrYjI/1+D7my4lM6DNGGskSiv5rgqwwzGlQioIZoJGwZZC+4GZpUrklzu0C0k8UkF2pmT6ELNiDjuOPaTB2sdQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1777663679; c=relaxed/simple; bh=Pqe3GwHtSZcK2T21A0SqmnIEyCF0+g9fH8a3c/J1xaY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=niVyHdStvjt4JQqU6OX0wg183w42jWbHquyhggawCzqtTMs6p5BQH+ITAXdHy5NV5U5MiJOo3ttj5PSZmdCYIBopngS6ny/4mmtnckFT5ShiCuY3BhMe7g7aHz8FmV5RcvilIHUggGfuWxgwtjfc1p7QapP8SJsYuizMfYVJoj4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=at7idcfj; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="at7idcfj" Received: from pps.filterd (m0356516.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 641AS1oe1381532; Fri, 1 May 2026 19:27:55 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=jN438DebSZsUDDnLH E5asg0DK+u7w1VCo7Yf7QJkQG0=; b=at7idcfj/jY+wRBb5IsHVY2Uabm9zCrgG CWtmokQI0Ib/j10y4rnyTb4VcSoRhCfzEQAXFT8TRcxQcC2/CAYxub0+G1uY5ytc ML5Nap9s6ujAoBuurZQDmJln8PbAipnEDNRZKJhFSQ1OtUyybRvGxX8dThyyNsYc y00lgVDHhHIRk+A0+oFc2ELER1oy3FvDOLROdzQd+W51C2sCbL4dV4+lax3l23vH S2grJfAulFMBYBND/Z1iemmKQm1j/Gn4vokarrwpT4Rm0tUUfvBrK9VNsuw4dsQo 3dPLubk0vuXhoF297iXe/MzFA5if91ncrJqxujE99O122xrLzyt6Q== Received: from ppma23.wdc07v.mail.ibm.com (5d.69.3da9.ip4.static.sl-reverse.com [169.61.105.93]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4drk1k49yt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 01 May 2026 19:27:54 +0000 (GMT) Received: from pps.filterd (ppma23.wdc07v.mail.ibm.com [127.0.0.1]) by ppma23.wdc07v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 641JNnGI004255; Fri, 1 May 2026 19:27:54 GMT Received: from smtprelay05.dal12v.mail.ibm.com ([172.16.1.7]) by ppma23.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4ds9ehrrb2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 01 May 2026 19:27:54 +0000 (GMT) Received: from smtpav05.wdc07v.mail.ibm.com (smtpav05.wdc07v.mail.ibm.com [10.39.53.232]) by smtprelay05.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 641JRovL25100942 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 1 May 2026 19:27:51 GMT Received: from smtpav05.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AF89358059; Fri, 1 May 2026 19:27:50 +0000 (GMT) Received: from smtpav05.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 27CDC58053; Fri, 1 May 2026 19:27:49 +0000 (GMT) Received: from Mac.ibm.com (unknown [9.61.247.25]) by smtpav05.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 1 May 2026 19:27:49 +0000 (GMT) From: Omar Elghoul To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: oelghoul@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, schnelle@linux.ibm.com, mjrosato@linux.ibm.com, alifm@linux.ibm.com, farman@linux.ibm.com, gbayer@linux.ibm.com, alex@shazbot.org Subject: [PATCH v1 3/3] s390/pci: Fence FMB enable/disable via sysfs for passthrough devices Date: Fri, 1 May 2026 15:25:30 -0400 Message-ID: <20260501192530.9429-4-oelghoul@linux.ibm.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260501192530.9429-1-oelghoul@linux.ibm.com> References: <20260501192530.9429-1-oelghoul@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNTAxMDE5MCBTYWx0ZWRfX4r2AKb//1Byy /q8Z0yH83gWHAj70TtsqKBp3c2xDHad2b3W/Va2xGmWCIiriTqSfx7nicc82aDQv5SYQaS8RJlX bPYLVTr53pag+iQLlkyqsnuEmz4ufuVVpfjmNA3+Q79b4kXSQfaIQKsSycyHDad/X7CkwLi2Wls BPpZ1y+/PtJRXKAaIWWDmTD1hRtUvX/e1DCbu4llh4ISv/Ja0yqCpr4By/LgVK6ivXRbvS3nDIL Pjdl5C/Z6Nyk0m1ZRWowR8HHX0XaV3nh2E0wtAbu1OuwC9hiOFdLNbhTSkMu8Vgy/kZQVoW8J1l ilx4dghH6uc3yrGSIUoXF8J6fx7T2QuGYt+oKCsus0YQiTSaH2g3JISEbp86xjJ8b8jJDQAJpkU qG7K3lcuuoNZdGptqHaaPsS7KZQvOM0yeZkC0d1/eGIgmmvV7OwwCETs+on3IpfJPINTVOJqVR8 xpI1y/GQJtS1sVYSGoQ== X-Proofpoint-GUID: 9z_iCkJRxIl3kSXIM-RI_xs5uyML9Yge X-Proofpoint-ORIG-GUID: 9z_iCkJRxIl3kSXIM-RI_xs5uyML9Yge X-Authority-Analysis: v=2.4 cv=MohiLWae c=1 sm=1 tr=0 ts=69f4feba cx=c_pps a=3Bg1Hr4SwmMryq2xdFQyZA==:117 a=3Bg1Hr4SwmMryq2xdFQyZA==:17 a=NGcC8JguVDcA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=Y2IxJ9c9Rs8Kov3niI8_:22 a=VnNF1IyMAAAA:8 a=LQOIlC5QHTpW1RRPx_YA:9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.51,FMLib:17.12.100.49 definitions=2026-05-01_05,2026-04-30_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 adultscore=0 suspectscore=0 bulkscore=0 spamscore=0 priorityscore=1501 malwarescore=0 clxscore=1015 lowpriorityscore=0 impostorscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2604200000 definitions=main-2605010190 Content-Type: text/plain; charset="utf-8" Introduce a fence over enabling or disabling FMB via sysfs when the zPCI device is associated with a KVM. This will allow a KVM guest to use FMB passthrough and avoid the edge-case where the host disables FMB while the guest is still using it, which may cause partial counter resets and inconsistent reads which have no parallel in the architecture. With this patch, the userspace driver, likely QEMU, is still able to enable or disable the FMB using the VFIO device feature introduced in the previous patch, effectively securing what is associated with the VM state and isolating it from other processes on the host. For VFIO devices that are not associated with a KVM (i.e., for userspace drivers other than QEMU), this fence does not take effect. Signed-off-by: Omar Elghoul --- arch/s390/pci/pci_debug.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/arch/s390/pci/pci_debug.c b/arch/s390/pci/pci_debug.c index c7ed7bf254b5..b28923395d03 100644 --- a/arch/s390/pci/pci_debug.c +++ b/arch/s390/pci/pci_debug.c @@ -149,6 +149,9 @@ static ssize_t pci_perf_seq_write(struct file *file, co= nst char __user *ubuf, if (!zdev) return 0; =20 + if (zdev->kzdev) + return -EPERM; + rc =3D kstrtoul_from_user(ubuf, count, 10, &val); if (rc) return rc; --=20 2.52.0