From nobody Mon May 25 01:15:21 2026 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B67E18FDDE; Tue, 19 May 2026 22:43:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779230614; cv=none; b=AWPTS0/s2WOCump7RxKE8eufubVPSGG2B0DbK0frifT6VSTUKrS4hlp0mRji1yl8iHeTn/uXpHPZF5Qwr3v8cba84pCUzbJ8LyWQJ4Bq7Gaz0XPeqr+pXil6wHeO8YFprGv0a/Xycn2nqWS9yJUxmNKGCV71+kN7Lv7jrRRKStE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779230614; c=relaxed/simple; bh=4k+l6wcQdft/kmjIwmrM7v8gHVMIFPxq+8dlBg5+zhI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Sk1CjlfY2ZT2G/HEHPG8jwQm+1tb41IWuziJUfAbkfh3LSIsXmiapisyCTg2BmMa7UA+6ONs4A925vs2BjlhxwZsZReYf5FHkhRb3ftwBWASA7TM/ml6JVMpbar/XzApAE95KwKHkMJYa1eH6pnDyW0cKW5ZLKsgHtY3G3nYLTQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=VU8bxkb4; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="VU8bxkb4" Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 64JF13Ev795743; Tue, 19 May 2026 22:43:30 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=F9i8DssIb9Mj1M/nY CLIOk2AEZCuF7njsq9xNmy7Snw=; b=VU8bxkb4cwPOo1Ozi3oD8qKlN0GapdKNk +eWDrYpbjhcLknzpBZGbbBlFaAqwet0LLh5wQt/r5H9Igyl6sI2QeW2W4u1/ErM3 b5vYoREKFmHwK4QtdAyQwOd4jdK+HoJk4fygc0AF0NsSp6e8RmiLMcz/mcW+2tB1 XL9oIkYHHcT4h1432Vyf1l01I9nJo36LuH4bl+NT5Un7sb+Sk9rGAjqW2qqAoCW/ 35RV2zXVNkElWCsrGTlsV2RQD48xfhWOmQNKcCOlnsulhM2+rt7g+gGmAYuOreg9 ko+pwMO8CcFv4VIfWbwJhr237yWGp+3MbICZlET3Sy3DI1B8batMg== Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4e6h8mqd4c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 19 May 2026 22:43:29 +0000 (GMT) Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 64JMd8vL004605; Tue, 19 May 2026 22:43:28 GMT Received: from smtprelay03.wdc07v.mail.ibm.com ([172.16.1.70]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 4e754gckjv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 19 May 2026 22:43:28 +0000 (GMT) Received: from smtpav04.wdc07v.mail.ibm.com (smtpav04.wdc07v.mail.ibm.com [10.39.53.231]) by smtprelay03.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 64JMguie10945164 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 19 May 2026 22:42:56 GMT Received: from smtpav04.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DBE7B58052; Tue, 19 May 2026 22:43:24 +0000 (GMT) Received: from smtpav04.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 38FD158050; Tue, 19 May 2026 22:43:23 +0000 (GMT) Received: from Mac.ibm.com (unknown [9.61.253.120]) by smtpav04.wdc07v.mail.ibm.com (Postfix) with ESMTP; Tue, 19 May 2026 22:43:23 +0000 (GMT) From: Omar Elghoul To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: oelghoul@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, schnelle@linux.ibm.com, mjrosato@linux.ibm.com, alifm@linux.ibm.com, farman@linux.ibm.com, gbayer@linux.ibm.com, alex@shazbot.org Subject: [PATCH v2 1/3] s390/pci: Preserve FMB state in device re-enablement Date: Tue, 19 May 2026 18:42:02 -0400 Message-ID: <20260519224204.19154-2-oelghoul@linux.ibm.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260519224204.19154-1-oelghoul@linux.ibm.com> References: <20260519224204.19154-1-oelghoul@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-GUID: gIQAz5Qtc2YG8-4XWRs2AVuhcZn0csfV X-Authority-Analysis: v=2.4 cv=GYMnWwXL c=1 sm=1 tr=0 ts=6a0ce791 cx=c_pps a=AfN7/Ok6k8XGzOShvHwTGQ==:117 a=AfN7/Ok6k8XGzOShvHwTGQ==:17 a=NGcC8JguVDcA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=uAbxVGIbfxUO_5tXvNgY:22 a=VnNF1IyMAAAA:8 a=sdm-MloC9ubkM-Nm6RcA:9 X-Proofpoint-ORIG-GUID: gIQAz5Qtc2YG8-4XWRs2AVuhcZn0csfV X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNTE5MDIyNyBTYWx0ZWRfX0uNyiwkQmMNo mQRKOMk57GpMiVULJ8u1ctRPcZGjLTpYzpg+3qJtya9iqbdtlEB7fE7cyJVS+25iknPvhv2Uovb bMV87MsnKRyBhtftt16AjADnZK0U5cHOi3Mz0U8ChGzlG4YlHlTHTGxMTU7EmgHACxKyoZPH009 4cJnpuPTQGjrE87I1cujDqUUTKgrNYt0EKOvtHVJkLYIZA8OfJ2FG3nQRsrdUrHBk2BSVXdRPze 2DXJ+8PsLrR/OnWMw9jooWPYtZlirmr95O8TnWx5FH0W7kfCjQManluWOPBenBLpi6DqM8DevVL YsANpJrrbzTwarhaYF94gNFsaZCb7PWVRMhgOQMUk1QW6VIL8x1VhT8FwTYAgTz8n4Gw3BI0uMa 3gx6IQqgtOKoYlhZDwi+ddaIw952E6ycIkQkFu4FvTWnneyNYrvLPk+LJlyPBVLr1TqMP7p2guO E56wTPVitJuXb0ZDMBQ== X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.51,FMLib:17.12.100.49 definitions=2026-05-19_06,2026-05-18_01,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 malwarescore=0 lowpriorityscore=0 priorityscore=1501 bulkscore=0 adultscore=0 suspectscore=0 spamscore=0 clxscore=1015 impostorscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2605130000 definitions=main-2605190227 Content-Type: text/plain; charset="utf-8" Introduce a function zpci_fmb_reenable_device() that checks for the state of the FMB and reuses the same buffer where appropriate. If FMB was not previously enabled, it enables it for the device. Call this function during a zPCI device re-enablement, which in turn implicitly ensures that the FMB is enabled for host devices during their KVM registration. This function also clears out the software counters, so that a program resetting an FMB would see all its counters restart from zero as expected. The function to clear the software counters is also separated into a static function as it is now reused in both zpci_fmb_enable_device() and zpci_fmb_reenable_device(). Signed-off-by: Omar Elghoul --- arch/s390/include/asm/pci.h | 1 + arch/s390/pci/pci.c | 75 +++++++++++++++++++++++++++++-------- 2 files changed, 61 insertions(+), 15 deletions(-) diff --git a/arch/s390/include/asm/pci.h b/arch/s390/include/asm/pci.h index 5dcf35f0f325..65014e52d559 100644 --- a/arch/s390/include/asm/pci.h +++ b/arch/s390/include/asm/pci.h @@ -323,6 +323,7 @@ void zpci_remove_parent_msi_domain(struct zpci_bus *zbu= s); /* FMB */ int zpci_fmb_enable_device(struct zpci_dev *); int zpci_fmb_disable_device(struct zpci_dev *); +int zpci_fmb_reenable_device(struct zpci_dev *zdev); =20 /* Debug */ int zpci_debug_init(void); diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c index 39bd2adfc240..56cabb2dc291 100644 --- a/arch/s390/pci/pci.c +++ b/arch/s390/pci/pci.c @@ -164,22 +164,10 @@ int zpci_unregister_ioat(struct zpci_dev *zdev, u8 dm= aas) return cc; } =20 -/* Modify PCI: Set PCI function measurement parameters */ -int zpci_fmb_enable_device(struct zpci_dev *zdev) +static void zpci_fmb_clear_iommu_ctrs(struct zpci_dev *zdev) { - u64 req =3D ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_SET_MEASURE); struct zpci_iommu_ctrs *ctrs; - struct zpci_fib fib =3D {0}; - unsigned long flags; - u8 cc, status; - - if (zdev->fmb || sizeof(*zdev->fmb) < zdev->fmb_length) - return -EINVAL; - - zdev->fmb =3D kmem_cache_zalloc(zdev_fmb_cache, GFP_KERNEL); - if (!zdev->fmb) - return -ENOMEM; - WARN_ON((u64) zdev->fmb & 0xf); + unsigned long flags =3D 0; =20 /* reset software counters */ spin_lock_irqsave(&zdev->dom_lock, flags); @@ -192,7 +180,24 @@ int zpci_fmb_enable_device(struct zpci_dev *zdev) atomic64_set(&ctrs->sync_rpcits, 0); } spin_unlock_irqrestore(&zdev->dom_lock, flags); +} + +/* Modify PCI: Set PCI function measurement parameters */ +int zpci_fmb_enable_device(struct zpci_dev *zdev) +{ + u64 req =3D ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_SET_MEASURE); + struct zpci_fib fib =3D {0}; + u8 cc, status; + + if (zdev->fmb || sizeof(*zdev->fmb) < zdev->fmb_length) + return -EINVAL; + + zdev->fmb =3D kmem_cache_zalloc(zdev_fmb_cache, GFP_KERNEL); + if (!zdev->fmb) + return -ENOMEM; + WARN_ON((u64) zdev->fmb & 0xf); =20 + zpci_fmb_clear_iommu_ctrs(zdev); =20 fib.fmb_addr =3D virt_to_phys(zdev->fmb); fib.gd =3D zdev->gisa; @@ -227,6 +232,41 @@ int zpci_fmb_disable_device(struct zpci_dev *zdev) } return cc ? -EIO : 0; } +EXPORT_SYMBOL_GPL(zpci_fmb_disable_device); + +int zpci_fmb_reenable_device(struct zpci_dev *zdev) +{ + u64 req =3D ZPCI_CREATE_REQ(zdev->fh, 0, ZPCI_MOD_FC_SET_MEASURE); + struct zpci_fib fib =3D {0}; + u8 cc, status; + + lockdep_assert_held(&zdev->fmb_lock); + + if (!zdev->fmb) + return zpci_fmb_enable_device(zdev); + + fib.gd =3D zdev->gisa; + cc =3D zpci_mod_fc(req, &fib, &status); /* Disable function measurement */ + + /* Unlike in zpci_fmb_disable_device(), cc =3D=3D 3 is not a valid state = here + * because we are re-enabling function measurement for the same function + * handle. + */ + if (cc) + return -EIO; + + zpci_fmb_clear_iommu_ctrs(zdev); + + fib.fmb_addr =3D virt_to_phys(zdev->fmb); + cc =3D zpci_mod_fc(req, &fib, &status); /* Re-enable function measurement= */ + if (cc) { + kmem_cache_free(zdev_fmb_cache, zdev->fmb); + zdev->fmb =3D NULL; + return -EIO; + } + return 0; +} +EXPORT_SYMBOL_GPL(zpci_fmb_reenable_device); =20 static int zpci_cfg_load(struct zpci_dev *zdev, int offset, u32 *val, u8 l= en) { @@ -729,9 +769,14 @@ int zpci_reenable_device(struct zpci_dev *zdev) } =20 rc =3D zpci_iommu_register_ioat(zdev, &status); - if (rc) + if (rc) { zpci_disable_device(zdev); + return rc; + } =20 + mutex_lock(&zdev->fmb_lock); + zpci_fmb_reenable_device(zdev); + mutex_unlock(&zdev->fmb_lock); return rc; } EXPORT_SYMBOL_GPL(zpci_reenable_device); --=20 2.52.0 From nobody Mon May 25 01:15:21 2026 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 48EF330E0E9; Tue, 19 May 2026 22:44:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779230641; cv=none; b=flE4JuEMrX2bIZEZEfrUaaLCk1pGLOOCPu3DqVa81NWLZNT8pC/lCPEw/yVr0xbTr6zOaDK7dz/aZTUGBGyXb1FEFs0daGeZ4qpi60jauCTwy1cYMD6wWuHFZJZTREqSEzJsENwtRd+Z+0m/w5veQPMbxCSYWkIUIoe4do/8Q+o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779230641; c=relaxed/simple; bh=+7P+Zxjc27LNoZRx+6u2Gq3BRm7S50AZo68qn3qaZ1I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=rBLJTmbGbaDaFyck7LZ59SMXIB2w57B6gplT9tKDihRNU03mtuWhgQIUXWJUwHjPcL7FgjySPJ7D7D3B9fB2BrEfp+1IVbrJFD1Ospl1FPUBcHXfM75XYtPIxIGtBQYpWNFI/Nu6ylJV/QaOYvEhSRpb30KkJTZUmctcAyTENew= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=GaN9N8pz; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="GaN9N8pz" Received: from pps.filterd (m0353725.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 64JF5ScD2090331; Tue, 19 May 2026 22:43:57 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=aSO7UjV/fLOz3VGoq 09K8sYOp7i2RoKtQloXaq4gnQg=; b=GaN9N8pz512FA98LyTxSM3FLqLeNE9Sj6 1arW1kJv9j7/pHZKBy9IC8yceGna4FHkiaBiH4wBYW+dPu9NJQlXtaQTzApRcR3A lRgejA7QzPg3Oc/UbYezTh7YyWeqtt6ioBehT1JfmJGr6NwdQyq6YhKpiFD49YQc d98gKhOquKt9Vr3q7PeXLIwBNEvDgZHuvfLZezntdshUyHl5IAI0rZ2lZc8jfrvK oKA+NGHP3ONwsruMRpm5+I2N1Y/72ddSO9yJIbtPFA+T2vaSjPT8vS9fNPTZ/PEy QyeheFxwgb9Q7Fh7YISJeywlJfsULXY/iNFIfYNb50z+LsENFKr2Q== Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4e6h88ek0y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 19 May 2026 22:43:57 +0000 (GMT) Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 64JMdBRC004654; Tue, 19 May 2026 22:43:56 GMT Received: from smtprelay06.wdc07v.mail.ibm.com ([172.16.1.73]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 4e754gckma-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 19 May 2026 22:43:56 +0000 (GMT) Received: from smtpav04.wdc07v.mail.ibm.com (smtpav04.wdc07v.mail.ibm.com [10.39.53.231]) by smtprelay06.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 64JMhrkJ30409440 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 19 May 2026 22:43:53 GMT Received: from smtpav04.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3A15958052; Tue, 19 May 2026 22:43:53 +0000 (GMT) Received: from smtpav04.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8DC3658050; Tue, 19 May 2026 22:43:51 +0000 (GMT) Received: from Mac.ibm.com (unknown [9.61.253.120]) by smtpav04.wdc07v.mail.ibm.com (Postfix) with ESMTP; Tue, 19 May 2026 22:43:51 +0000 (GMT) From: Omar Elghoul To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: oelghoul@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, schnelle@linux.ibm.com, mjrosato@linux.ibm.com, alifm@linux.ibm.com, farman@linux.ibm.com, gbayer@linux.ibm.com, alex@shazbot.org Subject: [PATCH v2 2/3] vfio-pci/zdev: Add VFIO FMB device feature Date: Tue, 19 May 2026 18:42:03 -0400 Message-ID: <20260519224204.19154-3-oelghoul@linux.ibm.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260519224204.19154-1-oelghoul@linux.ibm.com> References: <20260519224204.19154-1-oelghoul@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 7vlDda0YqiyAceI13I593rq6awvVncl_ X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNTE5MDIyNyBTYWx0ZWRfX1x4yRgjdEYp2 TuSUUulWQClYcrekollfesWh82Wq8sr8XcKW95wrSbnt00CL/FLPoeMvvRjbvCKr8MrmjBPSOms +2cCakdjNH1ikSu0h1e7nZ0bvc+C1LYYQyCTGtgqQh2o9+8ZgMgLHH5axQg/kd0Z4+PEzw/9aex HKUc2sTxDV7PGWeFUQDOZ1WFno5Xmbl15g69DvDoJvDL7/f9ewYJNfxkKIgwT8nAd0HYq76g94r MAP/8AmVEIkvglaV/FFGmn2i6cmWfisHOAQihwaQvA6OiDylj0jIpz35RlWr7sZFnnZ+G363M4i U+VeZ+vNGy0uudZxgDjcydiqaAndJ1TAy1ykKk5KsrrS1qZJQ7LGTMdgeQ0fEC+doXGKGcXUNZ9 TcTZj34NqKJP2cfkZ8GMcay2PUwowmZKSy9NHITwDIhLqifC82EMybXOaHD825U+NSKmmcDxXqs ScVHSv0b63icVdcg9xQ== X-Proofpoint-GUID: 7vlDda0YqiyAceI13I593rq6awvVncl_ X-Authority-Analysis: v=2.4 cv=apyCzyZV c=1 sm=1 tr=0 ts=6a0ce7ad cx=c_pps a=AfN7/Ok6k8XGzOShvHwTGQ==:117 a=AfN7/Ok6k8XGzOShvHwTGQ==:17 a=NGcC8JguVDcA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=V8glGbnc2Ofi9Qvn3v5h:22 a=VnNF1IyMAAAA:8 a=qYbwNa5VYoetIufaySgA:9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.51,FMLib:17.12.100.49 definitions=2026-05-19_06,2026-05-18_01,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 malwarescore=0 lowpriorityscore=0 priorityscore=1501 impostorscore=0 bulkscore=0 suspectscore=0 adultscore=0 spamscore=0 phishscore=0 clxscore=1015 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2605130000 definitions=main-2605190227 Content-Type: text/plain; charset="utf-8" Set up a new VFIO feature for zPCI devices to share the latest FMB snapshot with userspace. This feature supports the same 4 FMB formats (0 through 3) that are already supported by the kernel. With VFIO_DEVICE_FEATURE_GET, allow the user driver to read the latest FMB snapshot as well as query whether the FMB is currently enabled on the function, itself indicating whether the FMB snapshot is valid. On the other hand, with VFIO_DEVICE_FEATURE_SET, the userspace driver can enable or disable the FMB. Signed-off-by: Omar Elghoul --- drivers/vfio/pci/vfio_pci_core.c | 2 + drivers/vfio/pci/vfio_pci_priv.h | 9 ++++ drivers/vfio/pci/vfio_pci_zdev.c | 77 ++++++++++++++++++++++++++++++++ include/uapi/linux/vfio.h | 43 ++++++++++++++++++ 4 files changed, 131 insertions(+) diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_c= ore.c index 050e7542952e..07e13667d66a 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -1569,6 +1569,8 @@ int vfio_pci_core_ioctl_feature(struct vfio_device *d= evice, u32 flags, return vfio_pci_core_feature_token(vdev, flags, arg, argsz); case VFIO_DEVICE_FEATURE_DMA_BUF: return vfio_pci_core_feature_dma_buf(vdev, flags, arg, argsz); + case VFIO_DEVICE_FEATURE_ZPCI_FMB: + return vfio_pci_zdev_feature_fmb(vdev, flags, arg, argsz); default: return -ENOTTY; } diff --git a/drivers/vfio/pci/vfio_pci_priv.h b/drivers/vfio/pci/vfio_pci_p= riv.h index fca9d0dfac90..208e05942b48 100644 --- a/drivers/vfio/pci/vfio_pci_priv.h +++ b/drivers/vfio/pci/vfio_pci_priv.h @@ -93,6 +93,8 @@ int vfio_pci_info_zdev_add_caps(struct vfio_pci_core_devi= ce *vdev, struct vfio_info_cap *caps); int vfio_pci_zdev_open_device(struct vfio_pci_core_device *vdev); void vfio_pci_zdev_close_device(struct vfio_pci_core_device *vdev); +int vfio_pci_zdev_feature_fmb(struct vfio_pci_core_device *vdev, u32 flags, + void __user *arg, size_t argsz); #else static inline int vfio_pci_info_zdev_add_caps(struct vfio_pci_core_device = *vdev, struct vfio_info_cap *caps) @@ -107,6 +109,13 @@ static inline int vfio_pci_zdev_open_device(struct vfi= o_pci_core_device *vdev) =20 static inline void vfio_pci_zdev_close_device(struct vfio_pci_core_device = *vdev) {} + +static inline int vfio_pci_zdev_feature_fmb(struct vfio_pci_core_device *v= dev, + u32 flags, void __user *arg, + size_t argsz) +{ + return -ENOTTY; +} #endif =20 static inline bool vfio_pci_is_vga(struct pci_dev *pdev) diff --git a/drivers/vfio/pci/vfio_pci_zdev.c b/drivers/vfio/pci/vfio_pci_z= dev.c index 0990fdb146b7..1e9efe2bee69 100644 --- a/drivers/vfio/pci/vfio_pci_zdev.c +++ b/drivers/vfio/pci/vfio_pci_zdev.c @@ -167,3 +167,80 @@ void vfio_pci_zdev_close_device(struct vfio_pci_core_d= evice *vdev) if (zpci_kvm_hook.kvm_unregister) zpci_kvm_hook.kvm_unregister(zdev); } + +int vfio_pci_zdev_feature_fmb(struct vfio_pci_core_device *vdev, u32 flags, + void __user *arg, size_t argsz) +{ + struct zpci_dev *zdev; + struct vfio_device_feature_zpci_fmb fmb =3D {0}; + u32 ops =3D VFIO_DEVICE_FEATURE_GET | VFIO_DEVICE_FEATURE_SET; + int ret; + + ret =3D vfio_check_feature(flags, argsz, ops, sizeof(fmb)); + if (ret !=3D 1) + return ret; + + zdev =3D to_zpci(vdev->pdev); + if (!zdev) + return -ENODEV; + + mutex_lock(&zdev->fmb_lock); + if (flags & VFIO_DEVICE_FEATURE_SET) { + if (copy_from_user(&fmb, arg, sizeof(fmb))) { + ret =3D -EFAULT; + goto release_lock; + } + + if (fmb.flags & VFIO_DEVICE_FEATURE_ZPCI_FMB_FLAGS_ENABLED) + ret =3D zpci_fmb_reenable_device(zdev); + else + ret =3D zpci_fmb_disable_device(zdev); + goto release_lock; + } + + ret =3D 0; + if (zdev->fmb) { + fmb.flags |=3D VFIO_DEVICE_FEATURE_ZPCI_FMB_FLAGS_ENABLED; + } else { + fmb.flags &=3D ~VFIO_DEVICE_FEATURE_ZPCI_FMB_FLAGS_ENABLED; + goto release_lock; + } + + fmb.format =3D zdev->fmb->format; + fmb.fmt_ind =3D zdev->fmb->fmt_ind; + fmb.samples =3D zdev->fmb->samples; + fmb.last_update =3D zdev->fmb->last_update; + fmb.ld_ops =3D zdev->fmb->ld_ops; + fmb.st_ops =3D zdev->fmb->st_ops; + fmb.stb_ops =3D zdev->fmb->stb_ops; + fmb.rpcit_ops =3D zdev->fmb->rpcit_ops; + + switch (zdev->fmb->format) { + case 0: + if (zdev->fmb->fmt_ind & ZPCI_FMB_DMA_COUNTER_VALID) { + fmb.fmt0.dma_rbytes =3D zdev->fmb->fmt0.dma_rbytes; + fmb.fmt0.dma_wbytes =3D zdev->fmb->fmt0.dma_wbytes; + } + break; + case 1: + fmb.fmt1.rx_bytes =3D zdev->fmb->fmt1.rx_bytes; + fmb.fmt1.rx_packets =3D zdev->fmb->fmt1.rx_packets; + fmb.fmt1.tx_bytes =3D zdev->fmb->fmt1.tx_bytes; + fmb.fmt1.tx_packets =3D zdev->fmb->fmt1.tx_packets; + break; + case 2: + fmb.fmt2.consumed_work_units =3D zdev->fmb->fmt2.consumed_work_units; + fmb.fmt2.max_work_units =3D zdev->fmb->fmt2.max_work_units; + break; + case 3: + fmb.fmt3.tx_bytes =3D zdev->fmb->fmt3.tx_bytes; + break; + } + + if (copy_to_user(arg, &fmb, sizeof(fmb))) + ret =3D -EFAULT; + +release_lock: + mutex_unlock(&zdev->fmb_lock); + return ret; +} diff --git a/include/uapi/linux/vfio.h b/include/uapi/linux/vfio.h index 5de618a3a5ee..6cbc34ff063e 100644 --- a/include/uapi/linux/vfio.h +++ b/include/uapi/linux/vfio.h @@ -1534,6 +1534,49 @@ struct vfio_device_feature_dma_buf { */ #define VFIO_DEVICE_FEATURE_MIG_PRECOPY_INFOv2 12 =20 +/** + * Upon VFIO_DEVICE_FEATURE_GET, provide FMB passthrough for VFIO zPCI dev= ices. + * + * Upon VFIO_DEVICE_FEATURE_SET, only the flags field is read while the + * remainder of the structure is ignored. This allows the driver to enable= or + * disable the FMB while also leaving reserved bits for future flag expans= ion. + * All reserved fields should be zero for future compatibility. + */ +#define VFIO_DEVICE_FEATURE_ZPCI_FMB 13 +#define VFIO_DEVICE_FEATURE_ZPCI_FMB_FLAGS_ENABLED 0x1 + +struct vfio_device_feature_zpci_fmb { + __u64 flags; + __u32 format: 8; + __u32 fmt_ind: 24; + __u32 samples; + __u64 last_update; + __u64 ld_ops; + __u64 st_ops; + __u64 stb_ops; + __u64 rpcit_ops; + union { + struct { + __u64 dma_rbytes; + __u64 dma_wbytes; + } fmt0; + struct { + __u64 rx_bytes; + __u64 rx_packets; + __u64 tx_bytes; + __u64 tx_packets; + } fmt1; + struct { + __u64 consumed_work_units; + __u64 max_work_units; + } fmt2; + struct { + __u64 tx_bytes; + } fmt3; + }; + __u64 reserved[16]; +}; + /* -------- API for Type1 VFIO IOMMU -------- */ =20 /** --=20 2.52.0 From nobody Mon May 25 01:15:21 2026 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC476370D67; Tue, 19 May 2026 22:44:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779230663; cv=none; b=o7LckScQX5anOsq8Vi9oab/G91+CWz3AHTZoIpjhgQGF8PUwVq6DcUlL5aE+aqdOf6lx8pYAYyzO13G7efN4D1I34Nw0LvbZISV6+ob/JZlvfwlGCNc2w91X3Eiucl9HNainleIsroiYnblY3c78hDxrc3pJY0rK99EuGT4k5nU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1779230663; c=relaxed/simple; bh=ADBuxpAHmfGju7MPPidYfUdnyuM/6W+pFUa5x6o1zfw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=jqEGF1unuvBaNaWBt+2T24fGT++tGv/mLMKhW7Lgfla9D3Pe34r8+sNLHeglJUHK3pvRsR4zQRcIGkr+tk3M0XuhqyJa/UDNqRVr7vuwcLd6XXTRdDyyyO861goEuLkD5drudmAe5NFvWX3PelGcXASLWLAmg779vhY6nMG+ABA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=EHd5fQdt; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="EHd5fQdt" Received: from pps.filterd (m0360083.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 64JMFtiU4104744; Tue, 19 May 2026 22:44:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=+BUYnlGN3UPyA3ORn JYnT7dxYFPIvdJZstvzf7IoxRU=; b=EHd5fQdtCZMF1EsbZHBQ0S6q8oCzSUxCP DGl6R0RTKtqXXGda7AYmD/MANYQG+vkqVHFxPsmUM8whqQrng3JqbaR6YYIR2wxB oO2kx6N13dsiM/F0qs8hr8aF4wkQUrDbDT5Rmbxz0EK3IebVQ6F9ihYsdjf9HcnP DyN83G82+8XprJwq+GlKpdgy9S9lCkK7C/eC5oqkCP1e3Kqf8holyMQAiLgFM2tZ qkkdv1tZgmjEmdY0/3Dtd7Kw9c5bgsbpgNNB5h7DG8RP8H9EOL0woidN20oYMAb6 QGKuY+Wlu/ey14nRvQyMbXwuikL/O9/LK6rTl79ij+8mwUe6iFIgA== Received: from ppma21.wdc07v.mail.ibm.com (5b.69.3da9.ip4.static.sl-reverse.com [169.61.105.91]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4e6h9xyfeb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 19 May 2026 22:44:19 +0000 (GMT) Received: from pps.filterd (ppma21.wdc07v.mail.ibm.com [127.0.0.1]) by ppma21.wdc07v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 64JMdMxD021176; Tue, 19 May 2026 22:44:18 GMT Received: from smtprelay02.wdc07v.mail.ibm.com ([172.16.1.69]) by ppma21.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4e73wk4twb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 19 May 2026 22:44:18 +0000 (GMT) Received: from smtpav04.wdc07v.mail.ibm.com (smtpav04.wdc07v.mail.ibm.com [10.39.53.231]) by smtprelay02.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 64JMiFcM21103232 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 19 May 2026 22:44:15 GMT Received: from smtpav04.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CA1B258045; Tue, 19 May 2026 22:44:15 +0000 (GMT) Received: from smtpav04.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2A54758050; Tue, 19 May 2026 22:44:14 +0000 (GMT) Received: from Mac.ibm.com (unknown [9.61.253.120]) by smtpav04.wdc07v.mail.ibm.com (Postfix) with ESMTP; Tue, 19 May 2026 22:44:14 +0000 (GMT) From: Omar Elghoul To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: oelghoul@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, schnelle@linux.ibm.com, mjrosato@linux.ibm.com, alifm@linux.ibm.com, farman@linux.ibm.com, gbayer@linux.ibm.com, alex@shazbot.org Subject: [PATCH v2 3/3] s390/pci: Fence FMB enable/disable via sysfs for passthrough devices Date: Tue, 19 May 2026 18:42:04 -0400 Message-ID: <20260519224204.19154-4-oelghoul@linux.ibm.com> X-Mailer: git-send-email 2.52.0 In-Reply-To: <20260519224204.19154-1-oelghoul@linux.ibm.com> References: <20260519224204.19154-1-oelghoul@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNTE5MDIyNyBTYWx0ZWRfX7NMznHLmPat1 i+gz7SKX4M5VVBOdNXHuerH9kGjOYdI+Ksp+1tTTzpSla3fDIpAdZcc5bDAW85WuGKZTxkdGcv5 WYQ7qDrvYcVC4gZTetV7+ostADIwJuwCGTyeu8DtNMno0xgHJ7Nbej/Qc5x399JAG4w81oBtnGO 7pb7GxlZDU7f4QRSjnxzrrXaCZ05JCN65/vGcLyn5x2VdWvzI4PN2K+t9my6p1kxgHEQsUY5zbE t1Eh2Z7nZFlOPulSOY9tiHBVy0H59orar/YjMlAr1knC6sNhbQxH4CsB41o9q/oY4+W8rfH6SBv /XZUEMjlch2Onqh0cTvhnMbGboMLXcAEHJJssPvjCkKXzmxYDsZe8QT2MA/rvZwGkuUxbD7hYJE OjEpwPC+vSsXEUB0fjT7FhBmILcA9ke0a3cJMMYnXVh6otrrW8UxvLPZV2TpoCVbLd0PE1P5nN5 m0DOt9FWYjlyAKEn9vg== X-Authority-Analysis: v=2.4 cv=BNuDalQG c=1 sm=1 tr=0 ts=6a0ce7c3 cx=c_pps a=GFwsV6G8L6GxiO2Y/PsHdQ==:117 a=GFwsV6G8L6GxiO2Y/PsHdQ==:17 a=NGcC8JguVDcA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=iQ6ETzBq9ecOQQE5vZCe:22 a=VnNF1IyMAAAA:8 a=LQOIlC5QHTpW1RRPx_YA:9 X-Proofpoint-ORIG-GUID: gpqo4L0SjI6NG4NwGvtlU8cOZYMuyNWy X-Proofpoint-GUID: gpqo4L0SjI6NG4NwGvtlU8cOZYMuyNWy X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.51,FMLib:17.12.100.49 definitions=2026-05-19_06,2026-05-18_01,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 priorityscore=1501 malwarescore=0 impostorscore=0 suspectscore=0 lowpriorityscore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2605130000 definitions=main-2605190227 Content-Type: text/plain; charset="utf-8" Introduce a fence over enabling or disabling FMB via sysfs when the zPCI device is associated with a KVM. This will allow a KVM guest to use FMB passthrough and avoid the edge-case where the host disables FMB while the guest is still using it, which may cause partial counter resets and inconsistent reads which have no parallel in the architecture. With this patch, the userspace driver, likely QEMU, is still able to enable or disable the FMB using the VFIO device feature introduced in the previous patch, effectively securing what is associated with the VM state and isolating it from other processes on the host. For VFIO devices that are not associated with a KVM (i.e., for userspace drivers other than QEMU), this fence does not take effect. Signed-off-by: Omar Elghoul --- arch/s390/pci/pci_debug.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/arch/s390/pci/pci_debug.c b/arch/s390/pci/pci_debug.c index c7ed7bf254b5..2601614b919b 100644 --- a/arch/s390/pci/pci_debug.c +++ b/arch/s390/pci/pci_debug.c @@ -149,9 +149,15 @@ static ssize_t pci_perf_seq_write(struct file *file, c= onst char __user *ubuf, if (!zdev) return 0; =20 + mutex_lock(&zdev->kzdev_lock); + if (zdev->kzdev) { + rc =3D -EPERM; + goto release_kzdev_and_out; + } + rc =3D kstrtoul_from_user(ubuf, count, 10, &val); if (rc) - return rc; + goto release_kzdev_and_out; =20 mutex_lock(&zdev->fmb_lock); switch (val) { @@ -163,6 +169,9 @@ static ssize_t pci_perf_seq_write(struct file *file, co= nst char __user *ubuf, break; } mutex_unlock(&zdev->fmb_lock); + +release_kzdev_and_out: + mutex_unlock(&zdev->kzdev_lock); return rc ? rc : count; } =20 --=20 2.52.0