From nobody Fri Jul 24 20:51:09 2026 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 863353B47E0; Fri, 24 Jul 2026 16:14:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784909698; cv=none; b=uHGvw+ZlP9ZcqXaMoW2NIorl3QnhH22XtnPuCx+oM23InLy8G5RkAh5f9SyTV8oruU4zgGaGV+NpX3B/AdbnPLuF2hEnNte/NM5Z1gAZZREuy9JOo5yz9rn2vtwTZrJRVTGuJaryzwVNJcOLYyi4NegvQNx3Dyo755E91EqOX00= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784909698; c=relaxed/simple; bh=1s4beF7AUJtgUKAJTio1L984Gbc4oaNU5NglgSK1RX8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PQqg05/P4oDxGa2alQylhAmlod7mmUX/PRSRLoyo8HeW8akddHV8zMJ1+9LUGnFL3QIMfTzvA104CWXTu7nCme6e4xeeYDsH7Wz8JuCXxzkZei3x0gEF2ghbXg2/Dv8EePQA9JEwzs2bgYTq/xMjeWBkFLDr/aP2nAfv0xPbzvI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=Mt1KpOCx; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="Mt1KpOCx" Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 66ODfeMI100436; Fri, 24 Jul 2026 16:13:58 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=qRRr17gzgKP7yb8yZ FVvzVBAWOspssSggHtlOCAu1DQ=; b=Mt1KpOCxoVde+6NGGOqYPUTvz4wB3OD+9 HPL/slwMq0QOvblAXNaARPyl32inCiuZZE0A8tmUg4kjwtGwBTPMhxJ1zVZ6ugoD pUBDqI4FvMx9gZiS2sxaxpDIC952JCDWDr6UM44SSPNsYuk4MGVNmo2SS86zdvhQ pqooL216Sb0dIFfQR/w9RjINgJdww/9NP21Ls+9EkLe8gSKb5swnYjZKT3lZOgoZ idZWNUGa3FWdHXt+tL4IcJYgYgjVrbDplmIVPwg8apxd57FSANveB0alwFWk1/q+ B+7P9cS0urSuP3/hjQ5XqC04LtLuBBvjIq4qLuGgPR6/95/WqX0Pw== Received: from ppma23.wdc07v.mail.ibm.com (5d.69.3da9.ip4.static.sl-reverse.com [169.61.105.93]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4fm8gs8u49-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:13:58 +0000 (GMT) Received: from pps.filterd (ppma23.wdc07v.mail.ibm.com [127.0.0.1]) by ppma23.wdc07v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 66OG4jmW025395; Fri, 24 Jul 2026 16:13:57 GMT Received: from smtprelay04.dal12v.mail.ibm.com ([172.16.1.6]) by ppma23.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4fgnahhknq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:13:57 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (smtpav06.wdc07v.mail.ibm.com [10.39.53.233]) by smtprelay04.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 66OGDt9p8585738 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 24 Jul 2026 16:13:56 GMT Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C84C658054; Fri, 24 Jul 2026 16:13:55 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 45FB15803F; Fri, 24 Jul 2026 16:13:54 +0000 (GMT) Received: from li-4c4c4544-004d-4810-8043-b7c04f423534.ibm.com.com (unknown [9.61.50.28]) by smtpav06.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 24 Jul 2026 16:13:54 +0000 (GMT) From: Anthony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex@shazbot.org, kwankhede@nvidia.com, fiuczy@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, agordeev@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com Subject: [PATCH v5 01/15] s390/vfio-ap: Provide function to get the number of queues assigned to mdev Date: Fri, 24 Jul 2026 12:13:37 -0400 Message-ID: <20260724161351.1802644-2-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260724161351.1802644-1-akrowiak@linux.ibm.com> References: <20260724161351.1802644-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-Spam-Info: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX/oFwh4a/IM6a hIY3U4GYe+/K3qGtAZRTxph6Qrb/sI8HpOXPhWNssSUGkj6AcyPTsOuYNlezERZCXMJQ+4N9qto G/46PjXyx4Nd2z5H+4/84cuFeTRR6bA= X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX9EuBhDF0+aa0 0/FSJ1jZYXvOLe+h4DE3riiFeFg3RNWMXlp2uDne3m8r5n/X9ka84MeUc3jaiOJYklM9alfdNbh AktLFVGiL6+V/BIXBh4EJ1BcCR60UXhlTGcePyrpdoqonavoq8HuW36EeTtPtZuxyMgozg7EUDJ oPJChAwNwrdbGJZ92XuKP0ArD+P97qyiDLflGKRxf2Am+l+w/VENLzQYLkgyACBxaSsWD2iw0FB q+/aALl9J/K/64wfcid3t9Qfhufd5FIlbZratH6s24k3YReOEbps3efAdRbXNTYqX+zPhFEwto+ 18HTZApu4mGjjg2CvjUHzGzIIwesoXzcpKCOVQ0C/OWw8T33IIY1QJN9Wiv7h8cdBEyjbu/qqZ6 8e9xPGRKQblVN0U1W3fzAgroLcrzNWW3KxxRWOPfMQWGysoMJClIlIAQ1PWijddCb8PilL+CI6F AaHjchbiOFDw+2y/Qhw== X-Authority-Analysis: v=2.4 cv=Q9LiJY2a c=1 sm=1 tr=0 ts=6a638f46 cx=c_pps a=3Bg1Hr4SwmMryq2xdFQyZA==:117 a=3Bg1Hr4SwmMryq2xdFQyZA==:17 a=RAioF0-LDSMA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=uAbxVGIbfxUO_5tXvNgY:22 a=VnNF1IyMAAAA:8 a=X0LuQLNI_Dy_l-hxe5EA:9 X-Proofpoint-ORIG-GUID: ORJxeYqobaHOx1ToSqZpoQdiABO5jaPe X-Proofpoint-GUID: ORJxeYqobaHOx1ToSqZpoQdiABO5jaPe X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-07-24_03,2026-07-24_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 malwarescore=0 adultscore=0 phishscore=0 bulkscore=0 spamscore=0 priorityscore=1501 impostorscore=0 suspectscore=0 lowpriorityscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2607240145 Content-Type: text/plain; charset="utf-8" Provides a function that returns the number of queues that are or will be passed through to a guest when the mdev is attached to a guest. Signed-off-by: Anthony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 14 ++++++++++++++ drivers/s390/crypto/vfio_ap_private.h | 2 ++ 2 files changed, 16 insertions(+) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_a= p_ops.c index 44b3a1dcc1b3..71d3e2c2e07c 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -2098,6 +2098,20 @@ static void vfio_ap_mdev_request(struct vfio_device = *vdev, unsigned int count) release_update_locks_for_mdev(matrix_mdev); } =20 +int vfio_ap_mdev_get_num_queues(struct ap_matrix *ap_matrix) +{ + unsigned long apid, apqi; + int num_queues =3D 0; + + lockdep_assert_held(&matrix_dev->mdevs_lock); + + for_each_set_bit_inv(apid, ap_matrix->apm, AP_DEVICES) + for_each_set_bit_inv(apqi, ap_matrix->aqm, AP_DOMAINS) + num_queues++; + + return num_queues; +} + static int vfio_ap_mdev_get_device_info(unsigned long arg) { unsigned long minsz; diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vf= io_ap_private.h index 9bff666b0b35..9677e49554d7 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -152,6 +152,8 @@ struct vfio_ap_queue { struct work_struct reset_work; }; =20 +int vfio_ap_mdev_get_num_queues(struct ap_matrix *ap_matrix); + int vfio_ap_mdev_register(void); void vfio_ap_mdev_unregister(void); =20 --=20 2.53.0 From nobody Fri Jul 24 20:51:09 2026 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC2D6446046; Fri, 24 Jul 2026 16:14:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784909673; cv=none; b=JlyMwt63jUGgFvcPGaAYbQjZQnfv3AcMPkqjpyutnmEP9EHbCBsdAwlo8ekp47Q/v5bRExriYQIgKeSBppvpiOASugMNRspn95c5U+uRdILLoJt21nuImhrwufw5Dzt30FOYOxgAv9SzwRNoxHAxAkdSJrNrvwdICDl8BwvcbvE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784909673; c=relaxed/simple; bh=21oo8kBO+QCMUCXwJWr+B7ciA23W070MyMeW9pAYobE=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=F4s0QlGcdjTctudPRHE8CE+UuEZ5QgxYN8VXi/6o4mHC00PPHNbCd65NW+q9ErXU02C3Zw/Y09e5LeogHBVsfy/iwAQX4Ntqd1r7NKuHwfAgJyCVCQXL8cPAxojwvX5Ttzg0qETiGpvinJaIyg1Rtqwi9XZbKBOpNL1+BFEGjm8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=mVJZlSbu; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="mVJZlSbu" Received: from pps.filterd (m0356517.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 66ODfflI1992823; Fri, 24 Jul 2026 16:14:00 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=KgXRALvjK5k7/xaiU QIfj1cT+67a6jgKOa4/Uhj4cRo=; b=mVJZlSbuFjQpKh5wt9kkqhXiCTSwAkh2K nla8VAxVbWqigGuEt/MUY8ZcispxoqDgPJf6U7bU0Qt/PhSNAE1nU0DYVwOiNbac vzjECbVTomdVYkLx5OysHyh6bqAUYAno8TYzXHpdbcwX36olZw9isy8ecpt7LBKg XIFNPBMWYKiEPozk8PpxWF0eLD/gbk4EiOZhttrSR4ixfmY5wmWuk7zr5BYlNT9k f5FXSUfp2Bx4493ClIBvA8oPTrPowUJHNMjxyRkhArz0Zj/yNvLErBOkv1vGgKdO SRCxr4gFZ0JKOXZPyvu/yYq1Yd0YNl7INL8eoG8gz5tYvCPnzeSLg== Received: from ppma21.wdc07v.mail.ibm.com (5b.69.3da9.ip4.static.sl-reverse.com [169.61.105.91]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4fm8h00tyj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:00 +0000 (GMT) Received: from pps.filterd (ppma21.wdc07v.mail.ibm.com [127.0.0.1]) by ppma21.wdc07v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 66OG4dtQ028407; Fri, 24 Jul 2026 16:13:59 GMT Received: from smtprelay06.dal12v.mail.ibm.com ([172.16.1.8]) by ppma21.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4fgmtk9n1y-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:13:59 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (smtpav06.wdc07v.mail.ibm.com [10.39.53.233]) by smtprelay06.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 66OGDvio17302094 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 24 Jul 2026 16:13:57 GMT Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 813445804E; Fri, 24 Jul 2026 16:13:57 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id F204B5803F; Fri, 24 Jul 2026 16:13:55 +0000 (GMT) Received: from li-4c4c4544-004d-4810-8043-b7c04f423534.ibm.com.com (unknown [9.61.50.28]) by smtpav06.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 24 Jul 2026 16:13:55 +0000 (GMT) From: Anthony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex@shazbot.org, kwankhede@nvidia.com, fiuczy@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, agordeev@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com Subject: [PATCH v5 02/15] s390/vfio-ap: Data structures for facilitating vfio device migration Date: Fri, 24 Jul 2026 12:13:38 -0400 Message-ID: <20260724161351.1802644-3-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260724161351.1802644-1-akrowiak@linux.ibm.com> References: <20260724161351.1802644-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Authority-Analysis: v=2.4 cv=LdcMLDfi c=1 sm=1 tr=0 ts=6a638f48 cx=c_pps a=GFwsV6G8L6GxiO2Y/PsHdQ==:117 a=GFwsV6G8L6GxiO2Y/PsHdQ==:17 a=RAioF0-LDSMA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=U7nrCbtTmkRpXpFmAIza:22 a=VnNF1IyMAAAA:8 a=OK_kerOuralICY-O2F0A:9 X-Proofpoint-ORIG-GUID: kcm2AjmpbNW78laN-33WfTsWCCXj7Gqg X-Proofpoint-GUID: kcm2AjmpbNW78laN-33WfTsWCCXj7Gqg X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfXzjkky5xWrlhm 6YKql4enAzTf0zyuthftBOiw6b6siDsik4E1ijNBnX9OWUI0bJX3vkJbG9Qu11XVczH21WLOMy1 6gRs/uHft5aklLC3mhrQ7q0qpGWdcuqJfdMNMMPN/AvZ8aDVdypt9puCxVx+P/aoQv1UhjdDQpn 7pLz3+HDRBdA9NcXlyiqb+c4NhXoUYw94dK/IwUQ91Cww/UojEiYUzo1gsYztqikresJ7FzdBJW jF+OtL/QOzR9XZLcTuh7sC+zlbwfbmgdPSo9GpIPaiGWbEHxQZ0Do8YDRRWztRM8lisvqi8dzAi t7ivNljl+gyIRqFaZ+i+LlduowmJOK9KH0dv8y1/hepllij/UU7GQCz9acO8gd4G3YxKqHjcHpb qnLcVuxLm/HKF5U1N5/XP5/LjQfzX+Ee2b3QK5NcuEAZiOa8aeDIPAHDWt74h07XJrAjLnboLvE R4waR+WMKXtGOUcVtPQ== X-Proofpoint-Spam-Info: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX/aAaLWzT6i9h W7juj2Ts9UAkD1NBKzDqt8e3sjxX28Lgx0mYoY1IaLi9TTbpC/PWL5rALSm44I1qruW8lywSDYc HCgxCwda939nyoWN4v9a9QDfGoFa9Os= X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-07-24_03,2026-07-24_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 suspectscore=0 lowpriorityscore=0 spamscore=0 malwarescore=0 clxscore=1015 adultscore=0 priorityscore=1501 phishscore=0 bulkscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2607240145 Content-Type: text/plain; charset="utf-8" Creates the data structures used to facilitate state transitions during vfio device migration. Signed-off-by: Anthony Krowiak --- drivers/s390/crypto/Makefile | 2 +- drivers/s390/crypto/vfio_ap_migration.c | 72 +++++++++++++++++++++++++ drivers/s390/crypto/vfio_ap_private.h | 5 ++ 3 files changed, 78 insertions(+), 1 deletion(-) create mode 100644 drivers/s390/crypto/vfio_ap_migration.c diff --git a/drivers/s390/crypto/Makefile b/drivers/s390/crypto/Makefile index e83c6603c858..20f29184825a 100644 --- a/drivers/s390/crypto/Makefile +++ b/drivers/s390/crypto/Makefile @@ -34,5 +34,5 @@ pkey-uv-objs :=3D pkey_uv.o obj-$(CONFIG_PKEY_UV) +=3D pkey-uv.o =20 # adjunct processor matrix -vfio_ap-objs :=3D vfio_ap_drv.o vfio_ap_ops.o +vfio_ap-objs :=3D vfio_ap_drv.o vfio_ap_ops.o vfio_ap_migration.o obj-$(CONFIG_VFIO_AP) +=3D vfio_ap.o diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/= vfio_ap_migration.c new file mode 100644 index 000000000000..1b0856aacb69 --- /dev/null +++ b/drivers/s390/crypto/vfio_ap_migration.c @@ -0,0 +1,72 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Drives vfio_ap mdev migration. + * + * Copyright IBM Corp. 2025 + */ +#include "vfio_ap_private.h" + +/** + * vfio_ap_migration_file + * + * This object is used for chunk processing of multiple reads and writes of + * AP configuration information. + * + * @filp: file stream used to read or write AP configuration data + * @ap_config: object used to store AP configuration data between read or = write + * calls + * @config_sz: the size (in bytes) of @ap_config + */ +struct vfio_ap_migration_file { + struct file *filp; + struct vfio_ap_config *ap_config; + unsigned long config_sz; +}; + +/** + * vfio_ap_migration_data + * + * Manages the migration state for the VFIO device that maintains the AP + * configuration of the guest being migrated. + * + * @mig_state: the current migration state + * @resuming_mig_file: the object used to restore the state of the vfio-ap + * device the destination host: + * @stop_copy_mig_file: the object used to store the AP configuration of t= he + * source guest for transfer to the destination host. + */ +struct vfio_ap_migration_data { + enum vfio_device_mig_state mig_state; + struct vfio_ap_migration_file resuming_mig_file; + struct vfio_ap_migration_file stop_copy_mig_file; +}; + +/** + * vfio_ap_queue_info - the information for an AP queue + * + * @data: contains the queue information returned in GR2 from the PQAP(TAP= Q) + * command + * @apqn: the APQN of the queue + * @reserved: padding to ensure consistent structure size across platforms + */ +struct vfio_ap_queue_info { + u64 data; + u16 apqn; + u8 reserved[6]; +}; + +/** + * vfio_ap_config - the guest's AP configuration + * + * @num_queues: the number of queues passed through to the guest + * @reserved: padding to ensure proper alignment of @cntl_domains + * @adm: bitmap specifying the control domains in the AP configuration + * @qinfo: an array of vfio_ap_queue_info objects, each specifying the + * queue information for a queue passed through to the guest + */ +struct vfio_ap_config { + u32 num_queues; + u8 reserved[4]; + u64 adm[DIV_ROUND_UP(AP_DOMAINS, 64)]; + struct vfio_ap_queue_info qinfo[] __counted_by(num_queues); +}; diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vf= io_ap_private.h index 9677e49554d7..2b542648964b 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -91,6 +91,9 @@ struct ap_queue_table { DECLARE_HASHTABLE(queues, 8); }; =20 +/* Forward declaration for migration data structure */ +struct vfio_ap_migration_data; + /** * struct ap_matrix_mdev - Contains the data associated with a matrix medi= ated * device. @@ -110,6 +113,7 @@ struct ap_queue_table { * @aqm_add: bitmap of APQIs added to the host's AP configuration * @adm_add: bitmap of control domain numbers added to the host's AP * configuration + * @mig_data: vfio device migration data */ struct ap_matrix_mdev { struct vfio_device vdev; @@ -125,6 +129,7 @@ struct ap_matrix_mdev { DECLARE_BITMAP(apm_add, AP_DEVICES); DECLARE_BITMAP(aqm_add, AP_DOMAINS); DECLARE_BITMAP(adm_add, AP_DOMAINS); + struct vfio_ap_migration_data *mig_data; }; =20 /** --=20 2.53.0 From nobody Fri Jul 24 20:51:09 2026 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A8F24448BB2; Fri, 24 Jul 2026 16:14:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784909685; cv=none; b=V+9kKu86DW17AYiHUu2dbVGli+vUoe217NyX1qg3mryBFM17G58asd/SjIDpKj6EicdvjR9pcLfh83/34w0vz83tQdKpf9V55xJYX98Yqw6kNSvRfEWIspNwYON2ksG7qDZ+qEsTweG7gOQNQ/Oiv4rXjpmoYFgOaFToMD9V81k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784909685; c=relaxed/simple; bh=prBGKywt3nO+gLZxeQMk8BAqBKDxaGFtE4uAgURKlVQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=B2n7AMbzdcqbOTHKJWC0NXW/Efv2gQdMXVG1fFq/ktQ4E+gHXnUX7kJcJN9Mmgvki99/+0vreaP5jTU6Qc8r4eSvuNChxl8P/ZHkEYycp/vmdhbWA/LvVYsbaFUUYIZRVYRQCLHopktXDD/GFUsYkHhYqvhFAjZ3nmsWPGPSAzc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=XA6G1JEA; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="XA6G1JEA" Received: from pps.filterd (m0356517.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 66ODftwR1993432; Fri, 24 Jul 2026 16:14:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=8sTowYIP8NOuNegnI cX3MrPthDqBd+ZzSw77G+6519I=; b=XA6G1JEAoXX2OAKRfwL5VeNBSKF82pPfY u78AymZA2jJAAYzPwCtH4eWLnR/fFAfpr+RqZripywn542s9rXvvDGESnvoSwAZe HBtyqpVpTD6Vf8/MX8msma/Y9+l2prqM3HMWAgeetJulfgvk8cEWP90vm83ytPES KfqLthw+ukti8KRRavHcjSpZENNmt/jpf3V7dzql1mAmVvFiIkN945PO3tLAE5BN rdOODee5UGu8msC0dYWnJlXI+OOmg+FPa8h/0VDAO1eTYfXfioeRjVooUvpaZVxJ c0Sj4pi246WAaYSJjfsES7WoCjILc7B6oBkIYY1HCB26ks6wr+/SQ== Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4fm8h00tyt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:01 +0000 (GMT) Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 66OG4bYD010711; Fri, 24 Jul 2026 16:14:00 GMT Received: from smtprelay01.wdc07v.mail.ibm.com ([172.16.1.68]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 4fgp1gsf7x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:00 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (smtpav06.wdc07v.mail.ibm.com [10.39.53.233]) by smtprelay01.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 66OGDx4D4784908 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 24 Jul 2026 16:13:59 GMT Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 4C00158056; Fri, 24 Jul 2026 16:13:59 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id ABFF25803F; Fri, 24 Jul 2026 16:13:57 +0000 (GMT) Received: from li-4c4c4544-004d-4810-8043-b7c04f423534.ibm.com.com (unknown [9.61.50.28]) by smtpav06.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 24 Jul 2026 16:13:57 +0000 (GMT) From: Anthony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex@shazbot.org, kwankhede@nvidia.com, fiuczy@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, agordeev@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com Subject: [PATCH v5 03/15] s390/vfio-ap: Functions to initialize/release vfio device migration data Date: Fri, 24 Jul 2026 12:13:39 -0400 Message-ID: <20260724161351.1802644-4-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260724161351.1802644-1-akrowiak@linux.ibm.com> References: <20260724161351.1802644-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Authority-Analysis: v=2.4 cv=LdcMLDfi c=1 sm=1 tr=0 ts=6a638f4a cx=c_pps a=AfN7/Ok6k8XGzOShvHwTGQ==:117 a=AfN7/Ok6k8XGzOShvHwTGQ==:17 a=RAioF0-LDSMA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=U7nrCbtTmkRpXpFmAIza:22 a=VnNF1IyMAAAA:8 a=jy3f7vMduTvd68GZpnMA:9 X-Proofpoint-ORIG-GUID: cQYELzbJSLKR1A2jYfnA-Aqj4yo1-lxb X-Proofpoint-GUID: cQYELzbJSLKR1A2jYfnA-Aqj4yo1-lxb X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfXwhKb/mXVDkaq FSk5xS7zhUbHuLX9/xg7fnKky4JADhmIj7Akz+WMpuKHfk8K10e6j3r1pO6OsNOvwrSDJPwJLJn eLvbuLqL+DF98oLn+pZURP+VEz120R+gBxt01GRAJx9AtlxbrFHWLaefTRlJJyPl/zYtYFsP+G0 S2sB0YvQLtvGRayIYj43CXVf42hS20Q84Fmtqycm/frSfU6sIO6t+HBCihU8xWd6Y0JvTkL4BmN vQpRANlNhz7mNsiAAfEZ3/6MqNxnOHSb42nPj2mfSCe+gmzMhPP7msJl6CkQfms00laVJB84HvH IlpK6u64tBSMDJIc10GrZtydzIAJM225lx8OC4WxqEhoDlVYgbTlxiZR5Nb2bd10RtRvrgZhctW ginOxi+hzXuOcQDGq86gyofju0jqD2t2wWRVpGqDbkM3CZd2Wh+rqXYRIjzszT0YEF3CcE3HVU+ Bln9iUlYegG+JcQxtKQ== X-Proofpoint-Spam-Info: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX5M3NYqVgCZMM lPMEr967BwpsXxfmcJtkfXt3o0+SfR4dPcK1rD0WmMBK6D17XNI6id65/PnJ/Ac4U0uX40f0iZC +ktP+1QJpxSSs6E3Lm9SEA0yVfBlB0k= X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-07-24_03,2026-07-24_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 suspectscore=0 lowpriorityscore=0 spamscore=0 malwarescore=0 clxscore=1015 adultscore=0 priorityscore=1501 phishscore=0 bulkscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2607240145 Content-Type: text/plain; charset="utf-8" Provides the functions that initialize and release the data structures used during live guest migration: * vfio_ap_init_migration_capabilities Sets the migration flags and vfio_migration_ops structure into the vfio_device object when the mdev is probed. * vfio_ap_init_migration_data Allocates and initializes the object used to maintain the state of the VFIO migration. It is called when the VFIO device is opened. * vfio_ap_release_migration_data Frees the memory of the object used to maintain the VFIO migration state. It is called when the VFIO device release callback is invoked and when the VFIO device is closed. Signed-off-by: Anthony Krowiak --- drivers/s390/crypto/vfio_ap_migration.c | 80 +++++++++++++++++++++++++ drivers/s390/crypto/vfio_ap_ops.c | 60 ++++++++++++++++--- drivers/s390/crypto/vfio_ap_private.h | 4 ++ 3 files changed, 136 insertions(+), 8 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/= vfio_ap_migration.c index 1b0856aacb69..9187cfe26da7 100644 --- a/drivers/s390/crypto/vfio_ap_migration.c +++ b/drivers/s390/crypto/vfio_ap_migration.c @@ -4,6 +4,7 @@ * * Copyright IBM Corp. 2025 */ +#include #include "vfio_ap_private.h" =20 /** @@ -70,3 +71,82 @@ struct vfio_ap_config { u64 adm[DIV_ROUND_UP(AP_DOMAINS, 64)]; struct vfio_ap_queue_info qinfo[] __counted_by(num_queues); }; + +static struct file *vfio_ap_set_state(struct vfio_device *vdev, + enum vfio_device_mig_state new_state) +{ + return NULL; +} + +static int vfio_ap_get_state(struct vfio_device *vdev, + enum vfio_device_mig_state *current_state) +{ + return -EOPNOTSUPP; +} + +static int vfio_ap_get_data_size(struct vfio_device *vdev, + unsigned long *stop_copy_length) +{ + return -EOPNOTSUPP; +} + +static const struct vfio_migration_ops vfio_ap_migration_ops =3D { + .migration_set_state =3D vfio_ap_set_state, + .migration_get_state =3D vfio_ap_get_state, + .migration_get_data_size =3D vfio_ap_get_data_size, +}; + +/** + * vfio_ap_init_migrations_capabilities - initialize migration capabilities + * + * @matrix_mdev: pointer to object containing the mdev state + */ +void vfio_ap_init_migration_capabilities(struct ap_matrix_mdev *matrix_mde= v) +{ + if (ap_is_se_guest()) + return; + + matrix_mdev->vdev.migration_flags =3D VFIO_MIGRATION_STOP_COPY; + matrix_mdev->vdev.mig_ops =3D &vfio_ap_migration_ops; +} + +/** + * vfio_ap_init_migration_data - initialize migration data and functions + * + * @matrix_mdev: pointer to object containing the mdev state + * + * Return: zero if initialization is successful; otherwise, returns a erro= r. + */ +int vfio_ap_init_migration_data(struct ap_matrix_mdev *matrix_mdev) +{ + struct vfio_ap_migration_data *mig_data; + + lockdep_assert_held(&matrix_dev->mdevs_lock); + + mig_data =3D kzalloc_obj(struct vfio_ap_migration_data, GFP_KERNEL); + if (!mig_data) + return -ENOMEM; + + mig_data->mig_state =3D VFIO_DEVICE_STATE_RUNNING; + matrix_mdev->mig_data =3D mig_data; + + return 0; +} + +/** + * vfio_ap_release_migration_data: reclaim private migration data + * + * @vdev: pointer to the mdev + */ +void vfio_ap_release_migration_data(struct ap_matrix_mdev *matrix_mdev) +{ + lockdep_assert_held(&matrix_dev->mdevs_lock); + + if (!matrix_mdev->mig_data) + return; + + kfree(matrix_mdev->mig_data->resuming_mig_file.ap_config); + kfree(matrix_mdev->mig_data->stop_copy_mig_file.ap_config); + kfree(matrix_mdev->mig_data); + matrix_mdev->mig_data =3D NULL; +} diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_a= p_ops.c index 71d3e2c2e07c..47aca908701d 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -775,18 +775,30 @@ static bool vfio_ap_mdev_filter_matrix(struct ap_matr= ix_mdev *matrix_mdev, =20 static int vfio_ap_mdev_init_dev(struct vfio_device *vdev) { - struct ap_matrix_mdev *matrix_mdev =3D - container_of(vdev, struct ap_matrix_mdev, vdev); + struct ap_matrix_mdev *matrix_mdev; =20 + mutex_lock(&matrix_dev->mdevs_lock); + matrix_mdev =3D container_of(vdev, struct ap_matrix_mdev, vdev); matrix_mdev->mdev =3D to_mdev_device(vdev->dev); vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->matrix); matrix_mdev->pqap_hook =3D handle_pqap; vfio_ap_matrix_init(&matrix_dev->info, &matrix_mdev->shadow_apcb); hash_init(matrix_mdev->qtable.queues); + mutex_unlock(&matrix_dev->mdevs_lock); =20 return 0; } =20 +static void vfio_ap_mdev_release_dev(struct vfio_device *vdev) +{ + struct ap_matrix_mdev *matrix_mdev; + + mutex_lock(&matrix_dev->mdevs_lock); + matrix_mdev =3D container_of(vdev, struct ap_matrix_mdev, vdev); + vfio_ap_release_migration_data(matrix_mdev); + mutex_unlock(&matrix_dev->mdevs_lock); +} + static int vfio_ap_mdev_probe(struct mdev_device *mdev) { struct ap_matrix_mdev *matrix_mdev; @@ -797,18 +809,29 @@ static int vfio_ap_mdev_probe(struct mdev_device *mde= v) if (IS_ERR(matrix_mdev)) return PTR_ERR(matrix_mdev); =20 + mutex_lock(&matrix_dev->mdevs_lock); + + /* + * Migration capabilities must be initialized before calling + * vfio_register_emulated_iommu_dev; otherwise, the VFIO core + * will see mig_ops as NULL during the registration. This could + * prevent the VFIO core from properly setting up migration + * infrastructure like debugfs entries. + */ + vfio_ap_init_migration_capabilities(matrix_mdev); + ret =3D vfio_register_emulated_iommu_dev(&matrix_mdev->vdev); if (ret) goto err_put_vdev; matrix_mdev->req_trigger =3D NULL; matrix_mdev->cfg_chg_trigger =3D NULL; dev_set_drvdata(&mdev->dev, matrix_mdev); - mutex_lock(&matrix_dev->mdevs_lock); list_add(&matrix_mdev->node, &matrix_dev->mdev_list); mutex_unlock(&matrix_dev->mdevs_lock); return 0; =20 err_put_vdev: + mutex_unlock(&matrix_dev->mdevs_lock); vfio_put_device(&matrix_mdev->vdev); return ret; } @@ -2052,19 +2075,39 @@ static int vfio_ap_mdev_reset_qlist(struct list_hea= d *qlist) =20 static int vfio_ap_mdev_open_device(struct vfio_device *vdev) { - struct ap_matrix_mdev *matrix_mdev =3D - container_of(vdev, struct ap_matrix_mdev, vdev); + struct ap_matrix_mdev *matrix_mdev; + int ret; =20 if (!vdev->kvm) return -EINVAL; =20 - return vfio_ap_mdev_set_kvm(matrix_mdev, vdev->kvm); + mutex_lock(&matrix_dev->mdevs_lock); + matrix_mdev =3D container_of(vdev, struct ap_matrix_mdev, vdev); + ret =3D vfio_ap_init_migration_data(matrix_mdev); + mutex_unlock(&matrix_dev->mdevs_lock); + + if (ret) + return ret; + + ret =3D vfio_ap_mdev_set_kvm(matrix_mdev, vdev->kvm); + if (ret) { + /* Clean up migration data on failure */ + mutex_lock(&matrix_dev->mdevs_lock); + vfio_ap_release_migration_data(matrix_mdev); + mutex_unlock(&matrix_dev->mdevs_lock); + } + + return ret; } =20 static void vfio_ap_mdev_close_device(struct vfio_device *vdev) { - struct ap_matrix_mdev *matrix_mdev =3D - container_of(vdev, struct ap_matrix_mdev, vdev); + struct ap_matrix_mdev *matrix_mdev; + + mutex_lock(&matrix_dev->mdevs_lock); + matrix_mdev =3D container_of(vdev, struct ap_matrix_mdev, vdev); + vfio_ap_release_migration_data(matrix_mdev); + mutex_unlock(&matrix_dev->mdevs_lock); =20 vfio_ap_mdev_unset_kvm(matrix_mdev); } @@ -2374,6 +2417,7 @@ static const struct attribute_group vfio_queue_attr_g= roup =3D { =20 static const struct vfio_device_ops vfio_ap_matrix_dev_ops =3D { .init =3D vfio_ap_mdev_init_dev, + .release =3D vfio_ap_mdev_release_dev, .open_device =3D vfio_ap_mdev_open_device, .close_device =3D vfio_ap_mdev_close_device, .ioctl =3D vfio_ap_mdev_ioctl, diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vf= io_ap_private.h index 2b542648964b..a2a713f93674 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -172,4 +172,8 @@ void vfio_ap_on_cfg_changed(struct ap_config_info *new_= config_info, void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info, struct ap_config_info *old_config_info); =20 +void vfio_ap_init_migration_capabilities(struct ap_matrix_mdev *matrix_mde= v); +int vfio_ap_init_migration_data(struct ap_matrix_mdev *matrix_mdev); +void vfio_ap_release_migration_data(struct ap_matrix_mdev *matrix_mdev); + #endif /* _VFIO_AP_PRIVATE_H_ */ --=20 2.53.0 From nobody Fri Jul 24 20:51:09 2026 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D224A3CB555; Fri, 24 Jul 2026 16:14:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784909704; cv=none; b=sJTcqYN6TopvdyzkrqFBoB+kPtv/9QisBjcbavVY0xCxi23rqQIulw1RapYdq52juDd6+wHjfQnFQ7L2GIBya6NcCHbD7Q2FsvjXylG0PMnySUySQABTcpjef0JiiMkPRySH1UXGUqiS8IvanPc9b8/Ryp+zLdTp81zEc5+vhw0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784909704; c=relaxed/simple; bh=PYvMsVDT5j5Fd7fs4GG0JdBqUvchh83Nm+RFk7knaBg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZRkNHeOZgIQN1RweaUrEVqvuY3As0rqG/OXx4oT177eMLYGOV1ZipTNGeqmrJn2SGD/2NIT7zDNXsDb/z6DabCuB4zXoV8GdsgDmlDm5M1ex9VLOKe2H8OwDnbiUkHude6ORUxBjTTPkcDTOJpeRIssdYvSwsHn9zPLa1iLJUZo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=TUnnATw4; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="TUnnATw4" Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 66ODfuOa100791; Fri, 24 Jul 2026 16:14:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=MvXu2hCasoSpvnk7j M+cM4Hjnh0euR1nL0euL3S06S0=; b=TUnnATw4Een3/6E07HzZk21FxLNaRtvmS FA69NpZExr3yaVPeB1km068gb5IHTqJsYQjp652s7QMKeGbHz+wHGdwZLsG6qqHT UGDvcEAmWHfxH9kp9W5Mk3mmYM4earXmQGRsHH78iLvRNrN5+3Dhjcj4vW00TN2C 8NwFy0xX4KDTn+H72nsOFqRTljkCYK8kCtwzS+diw3pnOkhr9omFVJPY1i58gOrr U/tf02+j7nLKkTAv58impISEKG6XxGkxruNf0ImywgY+IuHLo1NBdRVuKpGfZUwg CgdYpRWcTk2PVGVRGyKETPg6bZM+QvxCVWjRisvlYQf+CqFrnJ6ww== Received: from ppma22.wdc07v.mail.ibm.com (5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4fm8gs8u4x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:03 +0000 (GMT) Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1]) by ppma22.wdc07v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 66OG4piR026084; Fri, 24 Jul 2026 16:14:02 GMT Received: from smtprelay03.wdc07v.mail.ibm.com ([172.16.1.70]) by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4fgm6whr1u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:02 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (smtpav06.wdc07v.mail.ibm.com [10.39.53.233]) by smtprelay03.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 66OGDPgK8520320 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 24 Jul 2026 16:13:25 GMT Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 066F65804E; Fri, 24 Jul 2026 16:14:01 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 74A985803F; Fri, 24 Jul 2026 16:13:59 +0000 (GMT) Received: from li-4c4c4544-004d-4810-8043-b7c04f423534.ibm.com.com (unknown [9.61.50.28]) by smtpav06.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 24 Jul 2026 16:13:59 +0000 (GMT) From: Anthony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex@shazbot.org, kwankhede@nvidia.com, fiuczy@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, agordeev@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com Subject: [PATCH v5 04/15] s390/vfio-ap: Reset migration state in VFIO_DEVICE_RESET ioctl handler Date: Fri, 24 Jul 2026 12:13:40 -0400 Message-ID: <20260724161351.1802644-5-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260724161351.1802644-1-akrowiak@linux.ibm.com> References: <20260724161351.1802644-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-Spam-Info: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfXyIEhGJ5D2ekq lBaPtxr+MPoYRRnfOO85Pv+1wESwxxFL8lXyOK8t9HsgvdRobHHCjEtrsQmj6yUWPTxzH2fZmdY Bduj4dgHUFE8kXNPec3uQeqe44MtWvc= X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX3/JEIe8+GBcq fsq8bU+2B8/3DmmYnJOWSXlj/6TOI/ZOUamj2j23prkxpOto55zgUKRQ9Xh/jh4me29pyRdhfaX CZF+7TSxsGf5I6hCbYCHWyR5cT4+yWJPJWH+i0eYk/vywecoDSRK8Lw5Yw42nXlhZjclSbv0Gc3 gGX/w+8/R73pOB2nF0sKHKkZst+zgBpEZcyvXdGiIATXQDHuIsAwHj9o2+URPQq5mVhB7CAtb1n WMX11cwaIqV4qEfwpe4ZOY86YNjc47DgMN6/A+2KfkheXu2JJJvSkzd3dR5LEzY7NITsutJl4J3 efbeIhzrfuZ1K6eucNvSbODCj1fVVlPbnv/Lj0rHJRRlvPfKgUdcy38IR8qdS9+ALsBMKWBRLfx qk15q7VnZOSQZ8Cp/YWVHVoyXQARAIxYSENB4tswXgyDI+SUI5TXtZPxy50+Li3TiQGGjx7QVWj feZ8o6ygaeJ1MtEdvEQ== X-Authority-Analysis: v=2.4 cv=Q9LiJY2a c=1 sm=1 tr=0 ts=6a638f4b cx=c_pps a=5BHTudwdYE3Te8bg5FgnPg==:117 a=5BHTudwdYE3Te8bg5FgnPg==:17 a=RAioF0-LDSMA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=uAbxVGIbfxUO_5tXvNgY:22 a=VnNF1IyMAAAA:8 a=yK-hndWHgRgZV9HjYSkA:9 X-Proofpoint-ORIG-GUID: b2eZW33DvAJqcJVyCWlnNCQWcA0TB9_J X-Proofpoint-GUID: b2eZW33DvAJqcJVyCWlnNCQWcA0TB9_J X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-07-24_03,2026-07-24_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 malwarescore=0 adultscore=0 phishscore=0 bulkscore=0 spamscore=0 priorityscore=1501 impostorscore=0 suspectscore=0 lowpriorityscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2607240145 Content-Type: text/plain; charset="utf-8" According to the VFIO migration specifications, userspace must issue a VFIO_DEVICE_RESET ioctl if a vfio device enters an error state or fails a state transition to recover the device and return its migration state to RUNNING. This patch adds a function that is called when a VFIO_DEVICE_RESET is issued that releases the migration files and frees the storage allocated for maintaining the migration state. Signed-off-by: Anthony Krowiak --- drivers/s390/crypto/vfio_ap_migration.c | 54 +++++++++++++++++++++++++ drivers/s390/crypto/vfio_ap_ops.c | 4 ++ drivers/s390/crypto/vfio_ap_private.h | 1 + 3 files changed, 59 insertions(+) diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/= vfio_ap_migration.c index 9187cfe26da7..45557f433652 100644 --- a/drivers/s390/crypto/vfio_ap_migration.c +++ b/drivers/s390/crypto/vfio_ap_migration.c @@ -150,3 +150,57 @@ void vfio_ap_release_migration_data(struct ap_matrix_m= dev *matrix_mdev) kfree(matrix_mdev->mig_data); matrix_mdev->mig_data =3D NULL; } + +static void vfio_ap_release_mig_files(struct ap_matrix_mdev *matrix_mdev) +{ + struct vfio_ap_migration_data *mig_data; + + /* + * The fput call does not call .release synchronously while the + * mdevs_lock mutex is held, so there is no problem with incurring a + * deadlock situation if fput is executed in this function. + */ + lockdep_assert_held(&matrix_dev->mdevs_lock); + + mig_data =3D matrix_mdev->mig_data; + if (!mig_data) + return; + + if (mig_data->stop_copy_mig_file.filp) { + fput(mig_data->stop_copy_mig_file.filp); + mig_data->stop_copy_mig_file.filp =3D NULL; + } + + kfree(mig_data->stop_copy_mig_file.ap_config); + mig_data->stop_copy_mig_file.ap_config =3D NULL; + mig_data->stop_copy_mig_file.config_sz =3D 0; + + if (mig_data->resuming_mig_file.filp) { + fput(mig_data->resuming_mig_file.filp); + mig_data->resuming_mig_file.filp =3D NULL; + } + + kfree(mig_data->resuming_mig_file.ap_config); + mig_data->resuming_mig_file.ap_config =3D NULL; + mig_data->resuming_mig_file.config_sz =3D 0; +} + +/** + * vfio_ap_reset_migration_state - Reset the vfio-ap migration state + * + * @matrix_mdev: pointer to the object maintaining the vfio-ap device state + * + * Called during VFIO_DEVICE_RESET to clean up any active migration + * state and reset the device to RUNNING state as required by the VFIO + * migration specification. + */ +void vfio_ap_reset_migration_state(struct ap_matrix_mdev *matrix_mdev) +{ + lockdep_assert_held(&matrix_dev->mdevs_lock); + + if (!matrix_mdev->mig_data) + return; + + vfio_ap_release_mig_files(matrix_mdev); + matrix_mdev->mig_data->mig_state =3D VFIO_DEVICE_STATE_RUNNING; +} diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_a= p_ops.c index 47aca908701d..678bdd4c82e9 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -2332,6 +2332,10 @@ static ssize_t vfio_ap_mdev_ioctl(struct vfio_device= *vdev, break; case VFIO_DEVICE_RESET: ret =3D vfio_ap_mdev_reset_queues(matrix_mdev); + if (ret =3D=3D 0) { + /* Reset migration state per VFIO migration spec */ + vfio_ap_reset_migration_state(matrix_mdev); + } break; case VFIO_DEVICE_GET_IRQ_INFO: ret =3D vfio_ap_get_irq_info(arg); diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vf= io_ap_private.h index a2a713f93674..1fbdfcce5a11 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -175,5 +175,6 @@ void vfio_ap_on_scan_complete(struct ap_config_info *ne= w_config_info, void vfio_ap_init_migration_capabilities(struct ap_matrix_mdev *matrix_mde= v); int vfio_ap_init_migration_data(struct ap_matrix_mdev *matrix_mdev); void vfio_ap_release_migration_data(struct ap_matrix_mdev *matrix_mdev); +void vfio_ap_reset_migration_state(struct ap_matrix_mdev *matrix_mdev); =20 #endif /* _VFIO_AP_PRIVATE_H_ */ --=20 2.53.0 From nobody Fri Jul 24 20:51:09 2026 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE3BC448BBA; Fri, 24 Jul 2026 16:14:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784909685; cv=none; b=OtoP3uMOQh5CgxbR5ryvA85dtDWYmIvmlOdujSEqF+kqd+4QULC381SFGAPuYilptmQqVfwOW1q1ahO8Y/nFNcUpqC8g4GCaCL54+mMY8CacQvNk91ms0UahCcJomjBhrdJqGrl9DcPMFaFqKUQWHzKvw2bZbMcYppX63UIl4ks= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784909685; c=relaxed/simple; bh=NWMi0eJtJBtMi76rIS9kKRRoC6G9dOPI4tANZfdfEkM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Af7gFtKZ9+GQNB186bjh3Xk3YNsPmMY1d6cvgJEVhV8S/N8s/2hn4qrAY6sxXihjy5qls1lsaCjW9ZCYylf+7L9iuOF41wAkmnt4ugD84+rdO6PbKBUrY2prL91vEd/alY64p7Pcd/bDwOZObJpwDD76yaazAOhrxSuH3kVcze8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=nzWbAKnz; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="nzWbAKnz" Received: from pps.filterd (m0356517.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 66ODfnak1993261; Fri, 24 Jul 2026 16:14:05 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=R0YT2UCTeh+GvzwDC JPDjRmneAQePusPPX9NL0NZ3cA=; b=nzWbAKnzRpHKatWuAo4+CvnzGgvt6opJk KzXDPdmGEGpctdCG2Nr2Au7WQuWJKNmKezYwc83fl+44NbMl+o4dhUJruxt0aSvb BY6m53Zp0x3wJOaisOhgrrer1CB7Sz6Ox1punRLSBsa9o57t73DB+/i2xzUNri/C iV/FO2GNoBunBnRQbGIIXm5a3ZY3+g3p6RWP4DVCAgTooXd4FGIY8DtIJH+Mger5 /XnKiYS7rhvmjWzzcUPbfsRPvENSloPHToHIFYawiBoqIYG28wZuZvYfxzWqbeaF WB2Gkz+LaF5lvrzPNX5+Nduf/j8wEuZDiO8nsVy7aB88QA4Y9kZ2Q== Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4fm8h00u0b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:05 +0000 (GMT) Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 66OG4btX010716; Fri, 24 Jul 2026 16:14:04 GMT Received: from smtprelay04.wdc07v.mail.ibm.com ([172.16.1.71]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 4fgp1gsf86-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:04 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (smtpav06.wdc07v.mail.ibm.com [10.39.53.233]) by smtprelay04.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 66OGE2Ho51970550 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 24 Jul 2026 16:14:02 GMT Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B18A35804E; Fri, 24 Jul 2026 16:14:02 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2F5845803F; Fri, 24 Jul 2026 16:14:01 +0000 (GMT) Received: from li-4c4c4544-004d-4810-8043-b7c04f423534.ibm.com.com (unknown [9.61.50.28]) by smtpav06.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 24 Jul 2026 16:14:01 +0000 (GMT) From: Anthony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex@shazbot.org, kwankhede@nvidia.com, fiuczy@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, agordeev@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com Subject: [PATCH v5 05/15] s390-vfio-ap: Callback to get/set vfio device mig state during guest migration Date: Fri, 24 Jul 2026 12:13:41 -0400 Message-ID: <20260724161351.1802644-6-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260724161351.1802644-1-akrowiak@linux.ibm.com> References: <20260724161351.1802644-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Authority-Analysis: v=2.4 cv=LdcMLDfi c=1 sm=1 tr=0 ts=6a638f4d cx=c_pps a=AfN7/Ok6k8XGzOShvHwTGQ==:117 a=AfN7/Ok6k8XGzOShvHwTGQ==:17 a=RAioF0-LDSMA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=U7nrCbtTmkRpXpFmAIza:22 a=VnNF1IyMAAAA:8 a=-jbGNgXFjY_vrUuEAzMA:9 X-Proofpoint-ORIG-GUID: MbDOdJfjYGY6zraWX5WHLLBgSLG9pSrP X-Proofpoint-GUID: MbDOdJfjYGY6zraWX5WHLLBgSLG9pSrP X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfXx+j9T0cHDj5H 7bG/yiHHqaK6m7oCzV3LIfhliy2gYADL0d/g1uwDR0Wr3Yw6n1dXNFEzNLr5vLRYzQdcmSHUefO A89ToJhzcqiaTuyKj+OokS+/YWZ8Mp6dRhCCJdvvu4XgpQrb6+5BNrt6GZdDCKMHvBg8TElwDyO pZReWoy8e8O+eMer11DFiBaSqH/YT/imAQ6cKvWe1odndiOXICiB8MlEqX4+XNxOBkj5j/C27pf sXwZ5cYJeW+q3EVTt9rSe1DgP0ku/KAT6o2qaWzrUkWjI8k2ngritmLuMx4ziIdNmrgmprNu+fl RRfKRK/4MFkYWFpQjRGBIQrNY6bJT+tDvufjsTxpOONPQ/f1zzznCTHI7QgDEN1Q10pmlgudps/ zFR4SbC8yOZoz9XppGlpVTUPpzKaCRpo3r4JejPqUA82F8LuOHXnTvvC/EzlXFRFf9vnnb5A4gS CJ5f9OI60bVoDwAAbfA== X-Proofpoint-Spam-Info: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfXxEwL1LBaa8SJ BfMwCjceI007xveEH9i1oNXWrNKw+tUTZW/HEhDbdU5nKOG2RD7nnH8aYeXXwH/JaSMoi0xE5To QmNamRvuCn8OgmnhIdUkUtWIQbO1xtw= X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-07-24_03,2026-07-24_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 suspectscore=0 lowpriorityscore=0 spamscore=0 malwarescore=0 clxscore=1015 adultscore=0 priorityscore=1501 phishscore=0 bulkscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2607240145 Content-Type: text/plain; charset="utf-8" Implements two callbacks related to the current state of a vfio-ap device: * The callback that sets a new migration state of a vfio-ap device during live migration of guests with pass-through access to AP devices. This callback is mandatory for VFIO_DEVICE_FEATURE_MIGRATION support. The function pointer for this callback is specified via the 'migration_set_state' field of the 'vfio_migration_ops' structure which is stored with the VFIO device when the 'vfio_device' structure representing the mediated device is initialized. * The callback that returns the current vfio device migration state during live migration of guests with pass-through access to AP devices. The function pointer for this callback is specified via the 'migration_get_state' field of the 'vfio_migration_ops' structure which is stored with the VFIO device when the 'vfio_device' structure representing the mediated device is initialized. Signed-off-by: Anthony Krowiak --- drivers/s390/crypto/vfio_ap_migration.c | 130 +++++++++++++++++++++++- 1 file changed, 128 insertions(+), 2 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/= vfio_ap_migration.c index 45557f433652..17cabd7541e7 100644 --- a/drivers/s390/crypto/vfio_ap_migration.c +++ b/drivers/s390/crypto/vfio_ap_migration.c @@ -72,16 +72,141 @@ struct vfio_ap_config { struct vfio_ap_queue_info qinfo[] __counted_by(num_queues); }; =20 +static struct file * +vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev, + enum vfio_device_mig_state new_state) +{ + struct vfio_ap_migration_data *mig_data; + enum vfio_device_mig_state cur_state; + + lockdep_assert_held(&matrix_dev->mdevs_lock); + mig_data =3D matrix_mdev->mig_data; + cur_state =3D mig_data->mig_state; + dev_dbg(matrix_mdev->vdev.dev, "%s: %d -> %d\n", __func__, cur_state, + new_state); + + if (cur_state =3D=3D VFIO_DEVICE_STATE_STOP && + new_state =3D=3D VFIO_DEVICE_STATE_STOP_COPY) { + /* TODO */ + return ERR_PTR(-EOPNOTSUPP); + } + + if (cur_state =3D=3D VFIO_DEVICE_STATE_STOP && + new_state =3D=3D VFIO_DEVICE_STATE_RESUMING) { + /* TODO */ + return ERR_PTR(-EOPNOTSUPP); + } + + if ((cur_state =3D=3D VFIO_DEVICE_STATE_RESUMING && + new_state =3D=3D VFIO_DEVICE_STATE_STOP) || + (cur_state =3D=3D VFIO_DEVICE_STATE_STOP_COPY && + new_state =3D=3D VFIO_DEVICE_STATE_STOP)) { + /* TODO */ + return ERR_PTR(-EOPNOTSUPP); + } + + if ((cur_state =3D=3D VFIO_DEVICE_STATE_STOP && + new_state =3D=3D VFIO_DEVICE_STATE_RUNNING) || + (cur_state =3D=3D VFIO_DEVICE_STATE_RUNNING && + new_state =3D=3D VFIO_DEVICE_STATE_STOP)) { + /* TODO */ + return ERR_PTR(-EOPNOTSUPP); + } + + /* vfio_mig_get_next_state() does not use arcs other than the above */ + WARN_ON(true); + + return ERR_PTR(-EINVAL); +} + static struct file *vfio_ap_set_state(struct vfio_device *vdev, enum vfio_device_mig_state new_state) { - return NULL; + int ret; + struct file *filp =3D NULL; + struct ap_matrix_mdev *matrix_mdev; + enum vfio_device_mig_state next_state; + struct vfio_ap_migration_data *mig_data; + + matrix_mdev =3D container_of(vdev, struct ap_matrix_mdev, vdev); + + mutex_lock(&matrix_dev->mdevs_lock); + if (ap_is_se_guest()) { + dev_err_once(matrix_mdev->vdev.dev, + "Migration not allowed from or to a Secure Execution guest\n"); + mutex_unlock(&matrix_dev->mdevs_lock); + return ERR_PTR(-EPERM); + } + + mig_data =3D matrix_mdev->mig_data; + + /* + * The mig_data pointer is set in the vfio_ap_init_migration_data + * function which is called when the vfio-ap device fd is opened. + * Since the implicit pre-open state is RUNNING, a request to set + * RUNNING is a no-op. Any other state transition is invalid before + * open_device. + */ + if (!mig_data) { + mutex_unlock(&matrix_dev->mdevs_lock); + if (new_state =3D=3D VFIO_DEVICE_STATE_RUNNING) + return NULL; + return ERR_PTR(-ENODEV); + } + + dev_dbg(vdev->dev, "%s -> %d\n", __func__, new_state); + + while (mig_data->mig_state !=3D VFIO_DEVICE_STATE_ERROR && + mig_data->mig_state !=3D new_state) { + ret =3D vfio_mig_get_next_state(vdev, mig_data->mig_state, + new_state, &next_state); + if (ret) { + filp =3D ERR_PTR(ret); + break; + } + + filp =3D vfio_ap_transition_to_state(matrix_mdev, next_state); + if (IS_ERR(filp)) + break; + + mig_data->mig_state =3D next_state; + + if (WARN_ON(filp && new_state !=3D next_state)) { + fput(filp); + filp =3D ERR_PTR(-EINVAL); + break; + } + } + + mutex_unlock(&matrix_dev->mdevs_lock); + + return filp; } =20 static int vfio_ap_get_state(struct vfio_device *vdev, enum vfio_device_mig_state *current_state) { - return -EOPNOTSUPP; + struct ap_matrix_mdev *matrix_mdev; + struct vfio_ap_migration_data *mig_data; + + mutex_lock(&matrix_dev->mdevs_lock); + + matrix_mdev =3D container_of(vdev, struct ap_matrix_mdev, vdev); + mig_data =3D matrix_mdev->mig_data; + + /* + * The mig_data pointer is set in the vfio_ap_init_migration_data + * function which is called when the vfio-ap device fd is opened. + * If mig_data is NULL, report RUNNING as the implicit pre-open state + * so userspace doesn't need to perform any state transition before the + * device becomes active. + */ + *current_state =3D (mig_data) ? mig_data->mig_state : + VFIO_DEVICE_STATE_RUNNING; + + mutex_unlock(&matrix_dev->mdevs_lock); + + return 0; } =20 static int vfio_ap_get_data_size(struct vfio_device *vdev, @@ -103,6 +228,7 @@ static const struct vfio_migration_ops vfio_ap_migratio= n_ops =3D { */ void vfio_ap_init_migration_capabilities(struct ap_matrix_mdev *matrix_mde= v) { + /* Live guest migration is not supported for SE guests */ if (ap_is_se_guest()) return; =20 --=20 2.53.0 From nobody Fri Jul 24 20:51:09 2026 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 03694446846; Fri, 24 Jul 2026 16:14:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784909690; cv=none; b=qD5DGgFr7aCNXvx7HasRyWPXZoX1pDLo6ntjpu1B8ZGwnwdVleOWJas0cCKC+3X9b3DnXpKIfjTWfBvFSH3YpBCTv96gI2VIUShLwlrSvS4RlSoSVscf3RQiduCCAWDDUoQSkDPFGh+2h17JDyr5FqqiLGvuUBoqnWOiGs++V8E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784909690; c=relaxed/simple; bh=8Dj+xd0+jnPwTkc+A3bHLiSPi4ZOQkbWayvAXbT/yVc=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hwZnm8sBqyIOkxti3T4cMWxEb0FDEUemUry7ysY0Vy+VBAWjYpyK1btSRWYMxT8ajcWcQl566ArtaqFCMcPEf1woMNi0y+XBQ+0ZKWCgI5XIJ2g5F67/yTasz289NPOrPDRUekF903Xn/ZuIZkn69axf3oJZtSamXktVgVQq1Ac= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=AYIm7Ah0; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="AYIm7Ah0" Received: from pps.filterd (m0360083.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 66ODfmTQ1974221; Fri, 24 Jul 2026 16:14:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=9OdWTJLCK1SsinkPp F/b/ps6OBLAcICdGKWvbMKT99I=; b=AYIm7Ah0CW/TEhbvUkbnj0oT127G5jh6w WcleF5FN6efixWNbyuZ8SyXHCmgjEY5GiZAK7j2tpI3ECE5Lw8pevtYOcXDEkjmj HyVYv07vYjl0hroNquKHJep6G35KLgFYu1xaNX7sY3PdwL7lbgBcSbIryZqv5bfA YfoicsTdXtHlzcFxTdWOnaVQETZGAMlaKIgNjDnMYEwusW6mFDX28H/lHxhzyAjN nqF+hoAMcGPVhjAFpk75aygaw6s0GvapC3FpYV+eJrF3I02nqvzqQESAaRJjRjND krx85BqV/uacv+EwzKL+yEpw4PGLdsCOxPhNxu/iy+5hEN88oPy/A== Received: from ppma22.wdc07v.mail.ibm.com (5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4fm8h50u4v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:06 +0000 (GMT) Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1]) by ppma22.wdc07v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 66OG4bNs026009; Fri, 24 Jul 2026 16:14:05 GMT Received: from smtprelay06.wdc07v.mail.ibm.com ([172.16.1.73]) by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4fgm6whr22-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:05 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (smtpav06.wdc07v.mail.ibm.com [10.39.53.233]) by smtprelay06.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 66OGE4KI25887382 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 24 Jul 2026 16:14:04 GMT Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6B2635804E; Fri, 24 Jul 2026 16:14:04 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DB4615803F; Fri, 24 Jul 2026 16:14:02 +0000 (GMT) Received: from li-4c4c4544-004d-4810-8043-b7c04f423534.ibm.com.com (unknown [9.61.50.28]) by smtpav06.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 24 Jul 2026 16:14:02 +0000 (GMT) From: Anthony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex@shazbot.org, kwankhede@nvidia.com, fiuczy@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, agordeev@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com Subject: [PATCH v5 06/15] s390/vfio-ap: Transition guest migration state from STOP to STOP_COPY Date: Fri, 24 Jul 2026 12:13:42 -0400 Message-ID: <20260724161351.1802644-7-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260724161351.1802644-1-akrowiak@linux.ibm.com> References: <20260724161351.1802644-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-Spam-Info: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX9OGv7dG8TKvL TliOEqUU8mFugQnDfZ6OsGXp9v8IvLl6Zf3hcttW0J+6jjoKnP6J8APf7tFtc2xhS6ceufbNM6Q lRuSZZyM/JUk/qXXevuJtL0qffosAWQ= X-Proofpoint-ORIG-GUID: Aaz-aIQvjSR2qYontx5WDXtX-6ktncZA X-Proofpoint-GUID: Aaz-aIQvjSR2qYontx5WDXtX-6ktncZA X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX1gmQUCYg1T8O 1ySsARd1pHe8J+BJL+P1ki9jEdk2DoyYvXQ4CkpaXH5NVZC1vt0wxD6ThLSHuz8CnWvxVaujmXV fbIGfJpQ9m/ZWjiPDux/DVJTfdW4g/rd7D0nznCW2S96Ga+Ck0hWjwYKzYW12qENlErTodLNxFm a4jX5tUxu7ld+06pbvs2oW7KPvnmGAnulnqzM1r8TrxUoBKO6qNYX+Y2DdESE6reJlSiaodtcQw aekmffAJqJ/S4NRp/+W/bF2VfTP0Q0XbCJ49hCosikQvP26ctS84kylaym49KVSHgyWtWxu/GSh D+ZGX5sgK8QwhdWgifFiKdoVhxSs+mh9dMZL/CKlKv9ONvPNcOBb8pUFhD4dRH6pJpdnq7aNSLB 7vayrV749P7q3ckS1zGwoaitTdoeCcqqYtCYV6UHcGElPoGpZCBH6nExuqxDlhYfzF+bGNMsXS6 eMJp2uiIyisF1C+Pe9A== X-Authority-Analysis: v=2.4 cv=du3rzVg4 c=1 sm=1 tr=0 ts=6a638f4e cx=c_pps a=5BHTudwdYE3Te8bg5FgnPg==:117 a=5BHTudwdYE3Te8bg5FgnPg==:17 a=RAioF0-LDSMA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=iQ6ETzBq9ecOQQE5vZCe:22 a=VnNF1IyMAAAA:8 a=_8NKb_9EqzLue2x-dRcA:9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-07-24_03,2026-07-24_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 clxscore=1015 malwarescore=0 spamscore=0 impostorscore=0 phishscore=0 bulkscore=0 priorityscore=1501 lowpriorityscore=0 suspectscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2607240145 Content-Type: text/plain; charset="utf-8" The transition of the guest migration state from VFIO_DEVICE_STATE_STOP to VFIO_DEVICE_STATE_STOP_COPY begins the process of saving the vfio device state. The vfio device state is comprised of the guest's AP configuration which specifies the adapters, domains and control domains to which the guest will be given access. To begin the process of saving the vfio device state, the VFIO AP device driver opens a file stream that will be used to read the source guest's AP configuration. Note: This commit introduces the file operations structure for migration state files with a stub implementation of the read operation. The complete implementation is provided in the subsequent commit. This incremental approach allows for easier review of the migration infrastructure setup separate from the actual implementation logic. Signed-off-by: Anthony Krowiak --- drivers/s390/crypto/vfio_ap_migration.c | 78 ++++++++++++++++++++++++- 1 file changed, 76 insertions(+), 2 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/= vfio_ap_migration.c index 17cabd7541e7..e566250a88ea 100644 --- a/drivers/s390/crypto/vfio_ap_migration.c +++ b/drivers/s390/crypto/vfio_ap_migration.c @@ -4,6 +4,7 @@ * * Copyright IBM Corp. 2025 */ +#include #include #include "vfio_ap_private.h" =20 @@ -72,6 +73,67 @@ struct vfio_ap_config { struct vfio_ap_queue_info qinfo[] __counted_by(num_queues); }; =20 +static void +vfio_ap_release_stop_copy_file(struct vfio_ap_migration_data *mig_data) +{ + kfree(mig_data->stop_copy_mig_file.ap_config); + mig_data->stop_copy_mig_file.ap_config =3D NULL; + mig_data->stop_copy_mig_file.config_sz =3D 0; + mig_data->stop_copy_mig_file.filp =3D NULL; +} + +static ssize_t +vfio_ap_stop_copy_read(struct file *, char __user *, size_t, loff_t *) +{ + /* TODO */ + return -EOPNOTSUPP; +} + +static int vfio_ap_release_mig_file(struct inode *file_inode, struct file = *filp) +{ + struct ap_matrix_mdev *matrix_mdev; + int ret =3D 0; + + mutex_lock(&matrix_dev->mdevs_lock); + + matrix_mdev =3D filp->private_data; + if (!matrix_mdev || !matrix_mdev->mig_data) { + ret =3D -ENODEV; + goto done; + } + + if (filp =3D=3D matrix_mdev->mig_data->stop_copy_mig_file.filp) + vfio_ap_release_stop_copy_file(matrix_mdev->mig_data); + else + ret =3D -ENOENT; + +done: + mutex_unlock(&matrix_dev->mdevs_lock); + return ret; +} + +static const struct file_operations vfio_ap_stop_copy_fops =3D { + .owner =3D THIS_MODULE, + .read =3D vfio_ap_stop_copy_read, + .compat_ioctl =3D compat_ptr_ioctl, + .release =3D vfio_ap_release_mig_file, +}; + +static struct file *vfio_ap_open_file_stream(struct ap_matrix_mdev *matrix= _mdev, + const struct file_operations *fops, + int flags) +{ + struct file *filp; + + lockdep_assert_held(&matrix_dev->mdevs_lock); + + filp =3D anon_inode_getfile("vfio_ap_mig_file", fops, matrix_mdev, flags); + if (!IS_ERR(filp)) + stream_open(filp->f_inode, filp); + + return filp; +} + static struct file * vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev, enum vfio_device_mig_state new_state) @@ -85,10 +147,22 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *mat= rix_mdev, dev_dbg(matrix_mdev->vdev.dev, "%s: %d -> %d\n", __func__, cur_state, new_state); =20 + /* + * Begins the process of saving the vfio device state by creating and + * returning a streaming data_fd to be used to read out the internal + * state of the vfio-ap device on the source host. + */ if (cur_state =3D=3D VFIO_DEVICE_STATE_STOP && new_state =3D=3D VFIO_DEVICE_STATE_STOP_COPY) { - /* TODO */ - return ERR_PTR(-EOPNOTSUPP); + struct file *filp =3D vfio_ap_open_file_stream(matrix_mdev, + &vfio_ap_stop_copy_fops, + O_RDONLY); + if (IS_ERR(filp)) + return ERR_CAST(filp); + + mig_data->stop_copy_mig_file.filp =3D filp; + + return filp; } =20 if (cur_state =3D=3D VFIO_DEVICE_STATE_STOP && --=20 2.53.0 From nobody Fri Jul 24 20:51:09 2026 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2F7274457C8; Fri, 24 Jul 2026 16:38:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784911091; cv=none; b=oIw8lgBYW84Z2tf8cqObUuSCi015SFEHEBC/IbRuAruTyKp4xvQGTLY6dt4PSctWN1zUJt1brgpwkG9aUsVlUOklBnop2Aj0rr9VJo67QHM8bllcoSawZLYQQPBExVUq5n91Y7/A7FWrq1Z7kB9rsD1OMb6wHWZp7NS+I5xCC3k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784911091; c=relaxed/simple; bh=TULad7R9slwNmvRdtguAk+p+AqxRE0QwPurttrZzOow=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mUwHi9q+73xjWjSJb4aJT4E4oJyv4IEMK8P7v4pXxfSVHDuj/TFlL2o46M/6nWmLD5TRjwM3p10ibmWqeXAf8NcwNfgN//X/eZiYwW3GCAjKK9tJX6xJKOHUH4qHYc25dYIO945erAiIu07lOzYFbblWWtxk8bXSsTJexpBKG2w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=mzV/+Q47; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="mzV/+Q47" Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 66ODff40100453; Fri, 24 Jul 2026 16:14:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=GoLpGVR92LHFURtw1 08mfduIBpm8QMxdF4xAURJWoqU=; b=mzV/+Q47aYu3uQSIwS+/MHQ6uSSi2URx8 U2hO5D4/oKlwQajjx0DHLutv/IqYuNqAUtS5fK2bRS1PRAkx4ukpT59PoRQzP2ER Yty8PRtJ5Z7L9sDqMZHzTJC1XBHdGTdmULOrFZqaG9uYSsxoW3ik4jJFh09aXSOR EqBTryXjFX8cL0Xu3dq0OyUAnGyLsGMe/JNKAy19W6P+EHJ/jwX4ECk5lbX2LAT4 bq5tvdq771sCaCVitrLCFT2JytQJwS7JgE0mfEm0whajuBcv9C9M5NLKnvHAXET1 HKgEpMNKri1ENIAds/hGVrzVaJ+gTDPNBOgPlZBXmY4fwH4/V+fpQ== Received: from ppma21.wdc07v.mail.ibm.com (5b.69.3da9.ip4.static.sl-reverse.com [169.61.105.91]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4fm8gs8u5k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:08 +0000 (GMT) Received: from pps.filterd (ppma21.wdc07v.mail.ibm.com [127.0.0.1]) by ppma21.wdc07v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 66OG4bQg028388; Fri, 24 Jul 2026 16:14:07 GMT Received: from smtprelay02.dal12v.mail.ibm.com ([172.16.1.4]) by ppma21.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4fgmtk9n2b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:07 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (smtpav06.wdc07v.mail.ibm.com [10.39.53.233]) by smtprelay02.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 66OGE6Pv32768724 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 24 Jul 2026 16:14:06 GMT Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 286655804E; Fri, 24 Jul 2026 16:14:06 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 956245803F; Fri, 24 Jul 2026 16:14:04 +0000 (GMT) Received: from li-4c4c4544-004d-4810-8043-b7c04f423534.ibm.com.com (unknown [9.61.50.28]) by smtpav06.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 24 Jul 2026 16:14:04 +0000 (GMT) From: Anthony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex@shazbot.org, kwankhede@nvidia.com, fiuczy@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, agordeev@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com Subject: [PATCH v5 07/15] s390/vfio-ap: File ops called to save the vfio device migration state Date: Fri, 24 Jul 2026 12:13:43 -0400 Message-ID: <20260724161351.1802644-8-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260724161351.1802644-1-akrowiak@linux.ibm.com> References: <20260724161351.1802644-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-Spam-Info: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfXyufsAFunsASw 8gNj0MAC0FeIKci9xeK8fIIl/EGF3EZ0IptZiEagnhBSE8K6w3NWF3bap+FuTKFZX5QkUjYYJp0 NNOb35Tq7bQRZ+uxtSD4Gv/MZuomRZE= X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX2wbzz3ftkNeW VYV6J6R5dhNs44HrUWTXvl+pLQsRigZVqeq0TXikc5CTmvr1O/c6R2jt1wQtc5AJUbKaLA9+81k uPzwbdW8qqac6P4HwXmA7+Rbusr3VXXMZBdlsKHvsNduYvETi9uwhlMdHugzpaOhmmcx5acQCIF hdnFjP1gTR0KM19U/+7IyAZ0ggrUpDPpqZWF8+35yaqdIfJFLNdPWV2OCZmHzhnTiTcpfx2u1VZ thzrmAYxmAKCPK/4RMYIDwCcgZzCD3bdofVNL/L5aVw/O2BPKipDOnk+xch3YrrJBPjMDJfyu2b HxxZmstTK0+8tIqv38svH2gTDRbuIrBA3txuZ1SiYtk+DX/4jdzJutcjhywWPPrXX5hGPuWhgt3 Po6uqzhYsvLZ95j/fP7n96rfAtHrovhnhvUT+jH7HSp32dlCrrorgWTFzb30DBbvvImICCVJvRO VPEouK8bKhnbOasogNA== X-Authority-Analysis: v=2.4 cv=Q9LiJY2a c=1 sm=1 tr=0 ts=6a638f50 cx=c_pps a=GFwsV6G8L6GxiO2Y/PsHdQ==:117 a=GFwsV6G8L6GxiO2Y/PsHdQ==:17 a=RAioF0-LDSMA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=uAbxVGIbfxUO_5tXvNgY:22 a=VnNF1IyMAAAA:8 a=BK3xoX9RFYtmYAf9xMEA:9 X-Proofpoint-ORIG-GUID: W2FGDCto2-6TbH9HKhc7VOzjyegxvyRJ X-Proofpoint-GUID: W2FGDCto2-6TbH9HKhc7VOzjyegxvyRJ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-07-24_03,2026-07-24_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 malwarescore=0 adultscore=0 phishscore=0 bulkscore=0 spamscore=0 priorityscore=1501 impostorscore=0 suspectscore=0 lowpriorityscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2607240145 Content-Type: text/plain; charset="utf-8" Implements the read callback function that was added to the file_operations structure for the file created to save the state of the vfio-ap device when the migration state transitioned from STOP to to the STOP_COPY state. This function copies the guest's AP configuration information to userspace. The information copied is comprised of the APQN of each queue device passed through to the guest along with its hardware information. This state data will be transferred to the vfio_ap device driver on the destination host when the state is transitioned to RESUMING. Signed-off-by: Anthony Krowiak --- drivers/s390/crypto/vfio_ap_migration.c | 278 +++++++++++++++++++++++- 1 file changed, 271 insertions(+), 7 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/= vfio_ap_migration.c index e566250a88ea..0d949ec205de 100644 --- a/drivers/s390/crypto/vfio_ap_migration.c +++ b/drivers/s390/crypto/vfio_ap_migration.c @@ -82,13 +82,6 @@ vfio_ap_release_stop_copy_file(struct vfio_ap_migration_= data *mig_data) mig_data->stop_copy_mig_file.filp =3D NULL; } =20 -static ssize_t -vfio_ap_stop_copy_read(struct file *, char __user *, size_t, loff_t *) -{ - /* TODO */ - return -EOPNOTSUPP; -} - static int vfio_ap_release_mig_file(struct inode *file_inode, struct file = *filp) { struct ap_matrix_mdev *matrix_mdev; @@ -112,6 +105,277 @@ static int vfio_ap_release_mig_file(struct inode *fil= e_inode, struct file *filp) return ret; } =20 +/** + * validate_stop_copy_read_parms: Validate the input parameters to the + * vfio_ap_stop_copy_read function + * + * @matrix_mdev: The object device containing the state to be read + * @filp: Pointer to the file stream used to read the vfio-ap device state + * @pos: The file offset from which to start reading data + * @len: The length of the data to be read + * + * Verify the following: + * - @filp private data is an ap_matrix_mdev instance + * - @filp is the instance opened when state transitioned from STOP to STO= P_COPY + * - @pos + @len does not cause integer overflow + * + * Returns: 0 if the parameters pass validation; otherwise returns an error + */ +static int validate_stop_copy_read_parms(struct file *filp, loff_t *pos, + size_t len) +{ + struct vfio_ap_migration_data *mig_data; + struct ap_matrix_mdev *matrix_mdev; + loff_t total_len; + + lockdep_assert_held(&matrix_dev->mdevs_lock); + + if (check_add_overflow((loff_t)len, *pos, &total_len)) + return -EIO; + + matrix_mdev =3D filp->private_data; + + if (!matrix_mdev || !matrix_mdev->mig_data) + return -ENODEV; + + mig_data =3D matrix_mdev->mig_data; + + if (mig_data->stop_copy_mig_file.filp !=3D filp) + return -EINVAL; + + return 0; +} + +static size_t vfio_ap_config_size(struct ap_matrix_mdev *matrix_mdev, + int *num_queues) +{ + size_t qinfo_size; + + lockdep_assert_held(&matrix_dev->mdevs_lock); + + *num_queues =3D vfio_ap_mdev_get_num_queues(&matrix_mdev->shadow_apcb); + qinfo_size =3D *num_queues * sizeof(struct vfio_ap_queue_info); + + return qinfo_size + sizeof(struct vfio_ap_config); +} + +static int get_hardware_info_for_queue(const char *mdev_name, + struct ap_tapq_hwinfo *hwinfo, + unsigned long apqn) +{ + struct ap_queue_status status; + + status =3D ap_tapq(apqn, hwinfo); + + switch (status.response_code) { + case AP_RESPONSE_NORMAL: + case AP_RESPONSE_RESET_IN_PROGRESS: + case AP_RESPONSE_DECONFIGURED: + case AP_RESPONSE_CHECKSTOPPED: + case AP_RESPONSE_BUSY: + /* For all these RCs the tapq info should be available */ + return 0; + case AP_RESPONSE_Q_NOT_AVAIL: + pr_err("vfio_ap_mdev %s: Failed to get hwinfo for queue %02lx.%04lx: TAP= Q rc=3D%d", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn), + status.response_code); + return -ENODEV; + default: + /* + * Without a pending async error, the tapq info should be + * available + */ + if (status.async) + return 0; + + pr_err("vfio_ap_mdev %s:Failed to get hwinfo for queue %02lx.%04lx: TAPQ= rc=3D%d", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn), + status.response_code); + return -EIO; + } + + return -EINVAL; +} + +static int vfio_ap_store_queue_info(const char *mdev_name, + struct vfio_ap_config *ap_config) +{ + struct ap_tapq_hwinfo source_hwinfo; + unsigned long num_queues; + int ret; + + /* + * ap_tapq() is a hardware instruction that may take time to complete. + * It must be called without mdevs_lock held to avoid blocking other + * mdevs. The apqn list was already snapshotted into ap_config->qinfo[] + * by the caller under the lock. + */ + for (num_queues =3D 0; num_queues < ap_config->num_queues; num_queues++) { + ret =3D get_hardware_info_for_queue(mdev_name, &source_hwinfo, + ap_config->qinfo[num_queues].apqn); + if (ret) + return ret; + + ap_config->qinfo[num_queues].data =3D source_hwinfo.value; + } + + return 0; +} + +static int +vfio_ap_get_config(struct ap_matrix_mdev *matrix_mdev) +{ + unsigned long *apm, *aqm, apid, apqi, num_queues; + struct vfio_ap_config *ap_configuration; + const char *mdev_name; + size_t ap_config_size; + int ret; + + lockdep_assert_held(&matrix_dev->mdevs_lock); + + ap_config_size =3D vfio_ap_config_size(matrix_mdev, (int *)&num_queues); + + ap_configuration =3D kzalloc(ap_config_size, GFP_KERNEL_ACCOUNT); + if (!ap_configuration) + return -ENOMEM; + + /* + * Snapshot the APQN list from shadow_apcb under the lock so that + * ap_tapq() calls in vfio_ap_store_queue_info() can happen without it. + */ + apm =3D matrix_mdev->shadow_apcb.apm; + aqm =3D matrix_mdev->shadow_apcb.aqm; + num_queues =3D 0; + for_each_set_bit_inv(apid, apm, AP_DEVICES) { + for_each_set_bit_inv(apqi, aqm, AP_DOMAINS) { + ap_configuration->qinfo[num_queues].apqn =3D + AP_MKQID(apid, apqi); + num_queues +=3D 1; + } + } + ap_configuration->num_queues =3D num_queues; + memcpy(ap_configuration->adm, matrix_mdev->shadow_apcb.adm, + sizeof(ap_configuration->adm)); + mdev_name =3D dev_name(matrix_mdev->vdev.dev); + + /** + * Release the global mdevs_lock which guards access to all mdevs. + * Storing the queue information could take a while if there are a + * large number of queues. + */ + mutex_unlock(&matrix_dev->mdevs_lock); + + ret =3D vfio_ap_store_queue_info(mdev_name, ap_configuration); + if (ret) { + kfree(ap_configuration); + mutex_lock(&matrix_dev->mdevs_lock); + return ret; + } + + /* Retake the mdevs lock so we can safely make updates to the mdev */ + mutex_lock(&matrix_dev->mdevs_lock); + + /* + * Verify mig_data is still valid - mdevs_lock was dropped, so the + * device could have been closed concurrently. + */ + if (!matrix_mdev->mig_data) { + kfree(ap_configuration); + return -ENODEV; + } + + matrix_mdev->mig_data->stop_copy_mig_file.ap_config =3D ap_configuration; + matrix_mdev->mig_data->stop_copy_mig_file.config_sz =3D ap_config_size; + + return 0; +} + +static ssize_t vfio_ap_stop_copy_read(struct file *filp, char __user *buf, + size_t len, loff_t *pos) +{ + struct ap_matrix_mdev *matrix_mdev; + struct vfio_ap_config *ap_config; + ssize_t ret =3D 0; + size_t ap_config_size; + + /* + * When userspace calls read() with an explicit offset (pread), pos is + * non-NULL and the function rejects it with -ESPIPE (illegal seek). For + * normal read() calls, pos is NULL, so we'll use the file's internal + * position filp->f_pos + */ + if (pos) + return -ESPIPE; + + mutex_lock(&matrix_dev->mdevs_lock); + + pos =3D &filp->f_pos; + + ret =3D validate_stop_copy_read_parms(filp, pos, len); + if (ret) { + mutex_unlock(&matrix_dev->mdevs_lock); + return ret; + } + + matrix_mdev =3D filp->private_data; + if (!matrix_mdev->mig_data->stop_copy_mig_file.ap_config) { + ret =3D vfio_ap_get_config(matrix_mdev); + if (ret) { + mutex_unlock(&matrix_dev->mdevs_lock); + return ret; + } + } + + ap_config_size =3D matrix_mdev->mig_data->stop_copy_mig_file.config_sz; + + /* + * If the position exceeds the size of the AP configuration data, + * then indicate EOF; otherwise calculate the length of the data to + * read such that a buffer overrun is prevented. + */ + if (*pos >=3D ap_config_size) + len =3D 0; + else + len =3D min_t(size_t, ap_config_size - *pos, len); + + /* If we've reached an EOF condition, let the caller know */ + if (len =3D=3D 0) { + mutex_unlock(&matrix_dev->mdevs_lock); + return 0; + } + + /* + * Allocate ap_config and copy the content of the object caching the + * AP configuration data between calls to it so we can give up the + * global mdevs_lock while copying the data to userspace. + */ + ap_config =3D kzalloc(ap_config_size, GFP_KERNEL_ACCOUNT); + if (!ap_config) { + mutex_unlock(&matrix_dev->mdevs_lock); + return -ENOMEM; + } + + memcpy(ap_config, matrix_mdev->mig_data->stop_copy_mig_file.ap_config, + ap_config_size); + + /* + * Give up the global mdevs_lock while copying data to the user; it + * might be a long running operation and we don't want to prevent access + * to other mdevs for an inordinate amount of time. + */ + mutex_unlock(&matrix_dev->mdevs_lock); + + if (copy_to_user(buf, (char *)ap_config + *pos, len)) { + kfree(ap_config); + return -EFAULT; + } + + kfree(ap_config); + *pos +=3D len; + + return len; +} + static const struct file_operations vfio_ap_stop_copy_fops =3D { .owner =3D THIS_MODULE, .read =3D vfio_ap_stop_copy_read, --=20 2.53.0 From nobody Fri Jul 24 20:51:09 2026 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BC82B42048; Fri, 24 Jul 2026 16:34:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784910876; cv=none; b=JDxKjC2iBCAVHfCMgPbRSIXm9O7DPq4WeHe3ZdD05TCJMiRP5t0yV1cdZ+iSJJGBNiOrQLFC76CFqYJoh8p/cWh+a73gsrFPtOVu0hMp3KSthNjHWJKjIk5UgCeiv8FTLkfuzWWazXfyr792LfQCLeSobCIOscy084ikMplCX5E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784910876; c=relaxed/simple; bh=k+Obqj9iOgucfJURJ9rpqDocwWXy6ezMgS7pfDVZiYU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=h8/2Y9WiPBxzGxRmhFH8FAnjndXr3JHTbcwcRCKZNt5AQ0RZAMDY5sIZNHy1wD+8D5o5br/hxgeTT6c4vAnbkh5CdJv5Za+5gAWzgii8U3xt0BQI6i1FPS9aOm7LPX3PYrCfILZXLTq5imN6sSH2FhaahSMdX3Z8qMpvYkLnOco= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=fTUUEQ2u; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="fTUUEQ2u" Received: from pps.filterd (m0360072.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 66ODftnp1994137; Fri, 24 Jul 2026 16:14:10 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=h5nXIeSx8guEBHDGz wyPZyocq4zTuerMD3COLV/IQXc=; b=fTUUEQ2uGqI9np0Y0SgPsy4fcZUqFj/T5 FJMkdN/wZv2I31dsjgk81xQgTYZJkPGtajt4spdMyxE1BebNjsxWLVWxKjfhEdgo a4cUeOLiraitqNhLGZ1bUwQkpu4bMH+1bws60rFx58VvysW4b6gIkGyJhWp4A8MN FUXdTrXTPQfI1ahYmkCF5qwqlWSbFkN2+oCBYnsd052a5bJ3jDgxsbDyg+ePfrbO BQKaA8fGV/DwweosX4tz9H6HFdJ4lyrwZHw89xUrpWhuX8LrLpE7qxvkp9gYvN8i xBBwoaQm2OSWhw/ilrOpJfinCwSNcDJV48dCOIGdJoBzUgCIbp78g== Received: from ppma11.dal12v.mail.ibm.com (db.9e.1632.ip4.static.sl-reverse.com [50.22.158.219]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4fg7ahmwas-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:09 +0000 (GMT) Received: from pps.filterd (ppma11.dal12v.mail.ibm.com [127.0.0.1]) by ppma11.dal12v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 66OG4k70022313; Fri, 24 Jul 2026 16:14:09 GMT Received: from smtprelay03.dal12v.mail.ibm.com ([172.16.1.5]) by ppma11.dal12v.mail.ibm.com (PPS) with ESMTPS id 4fgpgyscg3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:09 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (smtpav06.wdc07v.mail.ibm.com [10.39.53.233]) by smtprelay03.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 66OGE8mR30212840 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 24 Jul 2026 16:14:08 GMT Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id DB8105804E; Fri, 24 Jul 2026 16:14:07 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 523855803F; Fri, 24 Jul 2026 16:14:06 +0000 (GMT) Received: from li-4c4c4544-004d-4810-8043-b7c04f423534.ibm.com.com (unknown [9.61.50.28]) by smtpav06.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 24 Jul 2026 16:14:06 +0000 (GMT) From: Anthony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex@shazbot.org, kwankhede@nvidia.com, fiuczy@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, agordeev@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com Subject: [PATCH v5 08/15] s390/vfio-ap: Transition device migration state from STOP to RESUMING Date: Fri, 24 Jul 2026 12:13:44 -0400 Message-ID: <20260724161351.1802644-9-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260724161351.1802644-1-akrowiak@linux.ibm.com> References: <20260724161351.1802644-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: 3mPnQELeimcjQRtQbZLICFGNah5evglK X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfXwZfPcsTZ4cD6 ZtywCqC8utBBBNO8WDUoVO0sHhHF71DMDe+HmNqcNSL/qe+Jn+I8i8/hcKpQcgT1s40Hm49gX9Z e72styOupCU6N+UkbwXKRwyw7QaNXFtptguUqK3fk85UelHGtNVxU37pSmTrYDT+L3C3T7EeDLp 7ozaXJCcfug5Vcdpzdyi0zMimnvZ8wx7UasqN9KBAD0jct+C6BAakmmKU1TZcJbr0x2oukX2Qlv yZ99PY1C0G+rYikZRe/ZW8/UE8Hvy2uusth2yJDbqbijaDyi9u4+zwesslEhmN8HyzalMWecMkh CS4Vf6x2eDxEnD9/VjvgyyYWmKJ6XW+92SVcv//fvlDeek8vuX39kEY+/HagumqhYxj9rlfZr5f yY6dgzXlUkJcghE8k/TqEXnGK2ffH5YJBqJ1xJTrUm9YuvuWmp6/kTqGSGFjrmS8Tixd8armq0Y KETK7N3UNoDTxstXukw== X-Proofpoint-Spam-Info: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX3sylQC/e59xv 0T2JBgLJgAwv+PooY6CmLwoFqdGMj78oI08McKoPYsgob/5PpF+hdQx3tYaCEKu83zd2SbbNQWr vTnQOY953v5w0Xzyoqo1Mo3Bhbl/4hA= X-Proofpoint-GUID: 3mPnQELeimcjQRtQbZLICFGNah5evglK X-Authority-Analysis: v=2.4 cv=SM5ykuvH c=1 sm=1 tr=0 ts=6a638f51 cx=c_pps a=aDMHemPKRhS1OARIsFnwRA==:117 a=aDMHemPKRhS1OARIsFnwRA==:17 a=RAioF0-LDSMA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=RzCfie-kr_QcCd8fBx8p:22 a=VnNF1IyMAAAA:8 a=hRsAbhqWHO6Cr0izqpsA:9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-07-24_03,2026-07-24_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 lowpriorityscore=0 bulkscore=0 suspectscore=0 adultscore=0 spamscore=0 malwarescore=0 priorityscore=1501 phishscore=0 impostorscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2607240145 Content-Type: text/plain; charset="utf-8" Transitioning from VFIO_DEVICE_STATE_STOP to VFIO_DEVICE_STATE_RESUMING starts a process of restoring the device state of the vfio device on the target system. To prepare for restoring the device state, open a file stream to receive the guest's AP configuration saved when the device state on the source system transitioned from VFIO_DEVICE_STATE_STOP to VFIO_DEVICE_STATE_STOP_COPY. Note: This commit introduces the file operations structure for migration state files with a stub implementation of the write operation. The complete implementation is provided in the subsequent commit. This incremental approach allows for easier review of the migration infrastructure setup separate from the actual implementation logic. Signed-off-by: Anthony Krowiak --- drivers/s390/crypto/vfio_ap_migration.c | 51 ++++++++++++++++++++++++- 1 file changed, 49 insertions(+), 2 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/= vfio_ap_migration.c index 0d949ec205de..4a489a7a05ff 100644 --- a/drivers/s390/crypto/vfio_ap_migration.c +++ b/drivers/s390/crypto/vfio_ap_migration.c @@ -82,6 +82,15 @@ vfio_ap_release_stop_copy_file(struct vfio_ap_migration_= data *mig_data) mig_data->stop_copy_mig_file.filp =3D NULL; } =20 +static void +vfio_ap_release_resuming_file(struct vfio_ap_migration_data *mig_data) +{ + kfree(mig_data->resuming_mig_file.ap_config); + mig_data->resuming_mig_file.ap_config =3D NULL; + mig_data->resuming_mig_file.config_sz =3D 0; + mig_data->resuming_mig_file.filp =3D NULL; +} + static int vfio_ap_release_mig_file(struct inode *file_inode, struct file = *filp) { struct ap_matrix_mdev *matrix_mdev; @@ -97,6 +106,8 @@ static int vfio_ap_release_mig_file(struct inode *file_i= node, struct file *filp) =20 if (filp =3D=3D matrix_mdev->mig_data->stop_copy_mig_file.filp) vfio_ap_release_stop_copy_file(matrix_mdev->mig_data); + else if (filp =3D=3D matrix_mdev->mig_data->resuming_mig_file.filp) + vfio_ap_release_resuming_file(matrix_mdev->mig_data); else ret =3D -ENOENT; =20 @@ -398,6 +409,32 @@ static struct file *vfio_ap_open_file_stream(struct ap= _matrix_mdev *matrix_mdev, return filp; } =20 +static ssize_t vfio_ap_resuming_write(struct file *filp, const char __user= *buf, + size_t len, loff_t *pos) +{ + /* TODO */ + return -EOPNOTSUPP; +} + +static const struct file_operations vfio_ap_resume_fops =3D { + .owner =3D THIS_MODULE, + .write =3D vfio_ap_resuming_write, + .release =3D vfio_ap_release_mig_file, +}; + +static struct file *vfio_ap_resuming_init(struct ap_matrix_mdev *matrix_md= ev) +{ + struct vfio_ap_migration_data *mig_data; + struct file *filp; + + lockdep_assert_held(&matrix_dev->mdevs_lock); + + mig_data =3D matrix_mdev->mig_data; + filp =3D vfio_ap_open_file_stream(matrix_mdev, &vfio_ap_resume_fops, O_WR= ONLY); + + return filp; +} + static struct file * vfio_ap_transition_to_state(struct ap_matrix_mdev *matrix_mdev, enum vfio_device_mig_state new_state) @@ -429,10 +466,20 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *ma= trix_mdev, return filp; } =20 + /* + * Begins the process of restoring the vfio device state by creating and + * returning a streaming data_fd to be used to read in the internal + * state of the vfio-ap device on the destination host. + */ if (cur_state =3D=3D VFIO_DEVICE_STATE_STOP && new_state =3D=3D VFIO_DEVICE_STATE_RESUMING) { - /* TODO */ - return ERR_PTR(-EOPNOTSUPP); + struct file *filp =3D vfio_ap_resuming_init(matrix_mdev); + + if (IS_ERR(filp)) + return ERR_CAST(filp); + + mig_data->resuming_mig_file.filp =3D filp; + return filp; } =20 if ((cur_state =3D=3D VFIO_DEVICE_STATE_RESUMING && --=20 2.53.0 From nobody Fri Jul 24 20:51:09 2026 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CEA1E4302F8; Fri, 24 Jul 2026 16:42:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784911359; cv=none; b=I2AeL6YDowJR6a7ASKjEe4rjlQI8quct9Rr78DCsok2ijqEukO11E0y4k4lYAmmhuFhwEoooTcAn8DI4w9c9SxcRorOuljjUlSC5e7aj2sV3nyG2bqs/ia0cuDx6pp5o12epo575xoMfm4J1LDR/8SudHgHVX+FbVFEHEp8QHVU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784911359; c=relaxed/simple; bh=YJM/x+AP0ugNAWHOYZQLjAh++KAF8+DUspJBolR7AaQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Pnnhgg6GC8EKj/IVspGYWuHUchW9uCgVUalB2hidFLsWOw5bsOXN3ExDPZeNGiRK/+f2fi+XtoslnS1Gsbs5ZxcMUCZdPVjU/h9Nmqg97VWYPzaMINhtI3MvxUPsiAXpY/c1pfqCyvAWywUzjzyMcgh1OII+BEXqA0X5jbPUCnQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=fzTfdcjt; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="fzTfdcjt" Received: from pps.filterd (m0360083.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 66ODfdVU1973588; Fri, 24 Jul 2026 16:14:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=ToddTH3EdKmswPEGd ZFs2hGvBoaxn59bPSaSVjb3PRs=; b=fzTfdcjt9CdCkM0IiUg8lV9PHnr9ld+m0 5MrQon4nmODeOAr0kNASMyj0M0zIMKzbJ8bckf9IBVkaV10gdrwHxM5tfhh55n6q 8eOqBTHTG/wSpguSTC6fZ4BRwPTdd3qjg2H7cNOwXTkFC9euwJ/TcvPa3l2EZ9Nr hlJhzdgJPiD+4PP1K2PUYptn0/KXuRZ8VaVcFYEdfXZprbmkkV4gaNtEF27LkbaS Yd+p3PVadkBKzDmz020lL5w9Eb8n+BU7PjBMcIeP+l2Cbjvzg/RiAsRcCBROaF8h 1j+IfypRHfQ9bJC4fRr5H8HOfCJcDsoOv52ktmmafLkOJ177TjkIA== Received: from ppma21.wdc07v.mail.ibm.com (5b.69.3da9.ip4.static.sl-reverse.com [169.61.105.91]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4fm8h50u5s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:12 +0000 (GMT) Received: from pps.filterd (ppma21.wdc07v.mail.ibm.com [127.0.0.1]) by ppma21.wdc07v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 66OG4bFM028391; Fri, 24 Jul 2026 16:14:11 GMT Received: from smtprelay05.dal12v.mail.ibm.com ([172.16.1.7]) by ppma21.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4fgmtk9n2j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:11 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (smtpav06.wdc07v.mail.ibm.com [10.39.53.233]) by smtprelay05.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 66OGE9Eb32113276 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 24 Jul 2026 16:14:10 GMT Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9EC5858054; Fri, 24 Jul 2026 16:14:09 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 13BC25803F; Fri, 24 Jul 2026 16:14:08 +0000 (GMT) Received: from li-4c4c4544-004d-4810-8043-b7c04f423534.ibm.com.com (unknown [9.61.50.28]) by smtpav06.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 24 Jul 2026 16:14:07 +0000 (GMT) From: Anthony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex@shazbot.org, kwankhede@nvidia.com, fiuczy@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, agordeev@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com Subject: [PATCH v5 09/15] s390/vfio-ap: Add method to set a new guest AP configuration Date: Fri, 24 Jul 2026 12:13:45 -0400 Message-ID: <20260724161351.1802644-10-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260724161351.1802644-1-akrowiak@linux.ibm.com> References: <20260724161351.1802644-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-Spam-Info: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX1juPOEU6TUpS xGvY3kp1Ot6P8xge5HnUIUjbHinlT8Mbs1Mp6we1ZGFHSd34rsyC6zUGSM0YJ1hUmsoW+HA07wh ibmT8tY5Am8lIpY+WyHVwwI4UphQDlY= X-Proofpoint-ORIG-GUID: zcDj5A-qTUimI64GASIbkdwhPfngudQv X-Proofpoint-GUID: zcDj5A-qTUimI64GASIbkdwhPfngudQv X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX22sI7/9429aE RIZMw1zAmftWw7TjCL+WB4O/DwopXnCovFQkosTOD1W0g29d+tTzamdQWpolTRxmm4U88SKNrbO hSMRBuH1kF3QZZ0WmxU5AmKo6REzI4kNjry/uXkBDIiaOJV9dBz295sX1gNnXEWDnyyqxwpcJeq hkoDKBAgNsLwBW9naMvhynuQigjUT5d3NJDgRm+zzru23z0nulYCiIOe8woMcuoVE0Guk4Xg/zb x8LyWdtonkVd6nfLJwfvrUPa/zH9wOP8hQqaP8Jlyxrr4ntzurkZgEArCKdO/KuPS+Yzek8a6hU 5sJbJYsIJmaslkDqsjpKyyfLxsxk4xxDAqElC1JsF+iG4+r3IWTXmgm5atnU4mvhL1fW/HowasY tIgoyR4WWOIaD0ip+L0n4qYkwAjQkRGvnz799E1QXzxrUATMbkGwThnjpPi5i461fkaNLGgufOH usQenRRlbhhcszA7xdA== X-Authority-Analysis: v=2.4 cv=du3rzVg4 c=1 sm=1 tr=0 ts=6a638f54 cx=c_pps a=GFwsV6G8L6GxiO2Y/PsHdQ==:117 a=GFwsV6G8L6GxiO2Y/PsHdQ==:17 a=7yNgoilTr2P9NN8D:21 a=RAioF0-LDSMA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=iQ6ETzBq9ecOQQE5vZCe:22 a=VnNF1IyMAAAA:8 a=N1Ivd2qBLINY2Vw9BGkA:9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-07-24_03,2026-07-24_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 clxscore=1015 malwarescore=0 spamscore=0 impostorscore=0 phishscore=0 bulkscore=0 priorityscore=1501 lowpriorityscore=0 suspectscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2607240145 Content-Type: text/plain; charset="utf-8" Adds a new vfio_ap_set_new_config function to set a guest's AP configuration. This is needed in order to set the state of the mdev when it is migrated from a remote host system during the RESUMING phase. Key changes: * Refactored code from the ap_config_store function - handles changes to the sysfs ap_config attribute - into a new, non-static function which is callable from the ap_config_store function as well as the live guest migration code. Signed-off-by: Anthony Krowiak --- drivers/s390/crypto/vfio_ap_ops.c | 227 ++++++++++++++------------ drivers/s390/crypto/vfio_ap_private.h | 61 +++++++ 2 files changed, 188 insertions(+), 100 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_a= p_ops.c index 678bdd4c82e9..454bb9fe1971 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -81,53 +81,6 @@ static inline void release_update_locks_for_kvm(struct k= vm *kvm) mutex_unlock(&matrix_dev->guests_lock); } =20 -/** - * get_update_locks_for_mdev: Acquire the locks required to dynamically up= date a - * KVM guest's APCB in the proper order. - * - * @matrix_mdev: a pointer to a struct ap_matrix_mdev object containing th= e AP - * configuration data to use to update a KVM guest's APCB. - * - * The proper locking order is: - * 1. matrix_dev->guests_lock: required to use the KVM pointer to update a= KVM - * guest's APCB. - * 2. matrix_mdev->kvm->lock: required to update a guest's APCB - * 3. matrix_dev->mdevs_lock: required to access data stored in a matrix_= mdev - * - * Note: If @matrix_mdev is NULL or is not attached to a KVM guest, the KVM - * lock will not be taken. - */ -static inline void get_update_locks_for_mdev(struct ap_matrix_mdev *matrix= _mdev) -{ - mutex_lock(&matrix_dev->guests_lock); - if (matrix_mdev && matrix_mdev->kvm) - mutex_lock(&matrix_mdev->kvm->lock); - mutex_lock(&matrix_dev->mdevs_lock); -} - -/** - * release_update_locks_for_mdev: Release the locks used to dynamically up= date a - * KVM guest's APCB in the proper order. - * - * @matrix_mdev: a pointer to a struct ap_matrix_mdev object containing th= e AP - * configuration data to use to update a KVM guest's APCB. - * - * The proper unlocking order is: - * 1. matrix_dev->mdevs_lock - * 2. matrix_mdev->kvm->lock - * 3. matrix_dev->guests_lock - * - * Note: If @matrix_mdev is NULL or is not attached to a KVM guest, the KVM - * lock will not be released. - */ -static inline void release_update_locks_for_mdev(struct ap_matrix_mdev *ma= trix_mdev) -{ - mutex_unlock(&matrix_dev->mdevs_lock); - if (matrix_mdev && matrix_mdev->kvm) - mutex_unlock(&matrix_mdev->kvm->lock); - mutex_unlock(&matrix_dev->guests_lock); -} - /** * get_update_locks_by_apqn: Find the mdev to which an APQN is assigned and * acquire the locks required to update the APCB of @@ -642,8 +595,7 @@ static int handle_pqap(struct kvm_vcpu *vcpu) return 0; } =20 -static void vfio_ap_matrix_init(struct ap_config_info *info, - struct ap_matrix *matrix) +void vfio_ap_matrix_init(struct ap_config_info *info, struct ap_matrix *ma= trix) { matrix->apm_max =3D info->apxa ? info->na : 63; matrix->aqm_max =3D info->apxa ? info->nd : 15; @@ -875,7 +827,7 @@ static void vfio_ap_mdev_unlink_fr_queues(struct ap_mat= rix_mdev *matrix_mdev) q =3D vfio_ap_mdev_get_queue(matrix_mdev, AP_MKQID(apid, apqi)); if (q) - q->matrix_mdev =3D NULL; + vfio_ap_mdev_link_queue(matrix_mdev, q); } } } @@ -1014,19 +966,20 @@ static void vfio_ap_mdev_link_adapter(struct ap_matr= ix_mdev *matrix_mdev, unsigned long apqi; =20 for_each_set_bit_inv(apqi, matrix_mdev->matrix.aqm, AP_DOMAINS) - vfio_ap_mdev_link_apqn(matrix_mdev, - AP_MKQID(apid, apqi)); + vfio_ap_mdev_link_apqn(matrix_mdev, AP_MKQID(apid, apqi)); } =20 -static void collect_queues_to_reset(struct ap_matrix_mdev *matrix_mdev, - unsigned long apid, - struct list_head *qlist) +static void collect_queues_by_apid(struct ap_matrix_mdev *matrix_mdev, + unsigned long apid, + struct list_head *qlist) { struct vfio_ap_queue *q; unsigned long apqi; =20 for_each_set_bit_inv(apqi, matrix_mdev->shadow_apcb.aqm, AP_DOMAINS) { - q =3D vfio_ap_mdev_get_queue(matrix_mdev, AP_MKQID(apid, apqi)); + q =3D matrix_mdev ? + vfio_ap_mdev_get_queue(matrix_mdev, AP_MKQID(apid, apqi)) : + vfio_ap_find_queue(AP_MKQID(apid, apqi)); if (q) list_add_tail(&q->reset_qnode, qlist); } @@ -1038,7 +991,7 @@ static void reset_queues_for_apid(struct ap_matrix_mde= v *matrix_mdev, struct list_head qlist; =20 INIT_LIST_HEAD(&qlist); - collect_queues_to_reset(matrix_mdev, apid, &qlist); + collect_queues_by_apid(matrix_mdev, apid, &qlist); vfio_ap_mdev_reset_qlist(&qlist); } =20 @@ -1054,7 +1007,7 @@ static int reset_queues_for_apids(struct ap_matrix_md= ev *matrix_mdev, INIT_LIST_HEAD(&qlist); =20 for_each_set_bit_inv(apid, apm_reset, AP_DEVICES) - collect_queues_to_reset(matrix_mdev, apid, &qlist); + collect_queues_by_apid(matrix_mdev, apid, &qlist); =20 return vfio_ap_mdev_reset_qlist(&qlist); } @@ -1725,54 +1678,100 @@ static void ap_matrix_copy(struct ap_matrix *dst, = struct ap_matrix *src) bitmap_copy(dst->adm, src->adm, AP_DOMAINS); } =20 -static ssize_t ap_config_store(struct device *dev, struct device_attribute= *attr, - const char *buf, size_t count) +static void get_removed_matrixes(struct ap_matrix *m_removed, + struct ap_matrix *m_old, + struct ap_matrix *m_new) { - struct ap_matrix_mdev *matrix_mdev =3D dev_get_drvdata(dev); - struct ap_matrix m_new, m_old, m_added, m_removed; + bitmap_andnot(m_removed->apm, m_old->apm, m_new->apm, AP_DEVICES); + bitmap_andnot(m_removed->aqm, m_old->aqm, m_new->aqm, AP_DOMAINS); + bitmap_andnot(m_removed->adm, m_old->adm, m_new->adm, AP_DOMAINS); +} + +static void get_added_matrixes(struct ap_matrix *m_added, + struct ap_matrix *m_old, + struct ap_matrix *m_new) +{ + bitmap_andnot(m_added->apm, m_new->apm, m_old->apm, AP_DEVICES); + bitmap_andnot(m_added->aqm, m_new->aqm, m_old->aqm, AP_DOMAINS); + bitmap_andnot(m_added->adm, m_new->adm, m_old->adm, AP_DOMAINS); +} + +static int validate_new_state(struct ap_matrix_mdev *matrix_mdev) +{ + int rc; + + /* Ensure new state is valid, else undo new state */ + rc =3D vfio_ap_mdev_validate_masks(matrix_mdev); + if (rc) + return rc; + + rc =3D ap_matrix_overflow_check(matrix_mdev); + if (rc) + return rc; + + return 0; +} + +static void link_new_queues(struct ap_matrix_mdev *matrix_mdev, + struct ap_matrix *m_added) +{ + unsigned long apid, apqi; + + for_each_set_bit_inv(apid, m_added->apm, AP_DEVICES) + vfio_ap_mdev_link_adapter(matrix_mdev, apid); + + for_each_set_bit_inv(apqi, m_added->aqm, AP_DOMAINS) + vfio_ap_mdev_link_domain(matrix_mdev, apqi); +} + +/** + * vfio_ap_set_new_guest_config: + * + * Set a new AP configuration for a guest. + * + * @matrix_mdev: Object used to maintain the AP configuration for a guest + * @m_new: Object used to set the new AP configuration + * + * Returns: zero (0) if the new AP configuration is successfully set; othe= rwise, + * returns an error: + * + * ~ EADDRNOTAVAIL One or more APQNs are reserved for host use + * ~ EADDRINUSE One or more APQNs are assigned to another mdev + * ~ ENODEV An adapter, domain or control domain in the new + * AP configuration exceeds the max architected value + */ +int vfio_ap_set_new_guest_config(struct ap_matrix_mdev *matrix_mdev, + struct ap_matrix *m_new) +{ + struct ap_matrix m_old, m_old_shadow, m_added, m_removed; DECLARE_BITMAP(apm_filtered, AP_DEVICES); - unsigned long newbit; - char *newbuf, *rest; - int rc =3D count; bool do_update; + int rc; =20 - newbuf =3D kstrndup(buf, AP_CONFIG_STRLEN, GFP_KERNEL); - if (!newbuf) - return -ENOMEM; - rest =3D newbuf; + lockdep_assert_held(&ap_attr_mutex); + assert_has_update_locks_for_mdev(matrix_mdev); =20 - mutex_lock(&ap_attr_mutex); - get_update_locks_for_mdev(matrix_mdev); - - /* Save old state */ + /* Save the old state */ ap_matrix_copy(&m_old, &matrix_mdev->matrix); - if (parse_bitmap(&rest, m_new.apm, AP_DEVICES) || - parse_bitmap(&rest, m_new.aqm, AP_DOMAINS) || - parse_bitmap(&rest, m_new.adm, AP_DOMAINS)) { - rc =3D -EINVAL; - goto out; - } + ap_matrix_copy(&m_old_shadow, &matrix_mdev->shadow_apcb); =20 - bitmap_andnot(m_removed.apm, m_old.apm, m_new.apm, AP_DEVICES); - bitmap_andnot(m_removed.aqm, m_old.aqm, m_new.aqm, AP_DOMAINS); - bitmap_andnot(m_added.apm, m_new.apm, m_old.apm, AP_DEVICES); - bitmap_andnot(m_added.aqm, m_new.aqm, m_old.aqm, AP_DOMAINS); + /* + * Get the adapters, domains and control domains added and/or removed + * from the existing configuration + */ + get_removed_matrixes(&m_removed, &m_old, m_new); + get_added_matrixes(&m_added, &m_old, m_new); =20 /* Need new bitmaps in matrix_mdev for validation */ - ap_matrix_copy(&matrix_mdev->matrix, &m_new); + ap_matrix_copy(&matrix_mdev->matrix, m_new); =20 /* Ensure new state is valid, else undo new state */ - rc =3D vfio_ap_mdev_validate_masks(matrix_mdev); - if (rc) { - ap_matrix_copy(&matrix_mdev->matrix, &m_old); - goto out; - } - rc =3D ap_matrix_overflow_check(matrix_mdev); + rc =3D validate_new_state(matrix_mdev); if (rc) { ap_matrix_copy(&matrix_mdev->matrix, &m_old); - goto out; + ap_matrix_copy(&matrix_mdev->shadow_apcb, &m_old_shadow); + return rc; } - rc =3D count; =20 /* Need old bitmaps in matrix_mdev for unplug/unlink */ ap_matrix_copy(&matrix_mdev->matrix, &m_old); @@ -1782,14 +1781,10 @@ static ssize_t ap_config_store(struct device *dev, = struct device_attribute *attr vfio_ap_mdev_hot_unplug_domains(matrix_mdev, m_removed.aqm); =20 /* Need new bitmaps in matrix_mdev for linking new adapters/domains */ - ap_matrix_copy(&matrix_mdev->matrix, &m_new); - - /* Link newly added adapters */ - for_each_set_bit_inv(newbit, m_added.apm, AP_DEVICES) - vfio_ap_mdev_link_adapter(matrix_mdev, newbit); + ap_matrix_copy(&matrix_mdev->matrix, m_new); =20 - for_each_set_bit_inv(newbit, m_added.aqm, AP_DOMAINS) - vfio_ap_mdev_link_domain(matrix_mdev, newbit); + /* Link queues associated with the newly added adapters and domains */ + link_new_queues(matrix_mdev, &m_added); =20 /* filter resources not bound to vfio-ap */ do_update =3D vfio_ap_mdev_filter_matrix(matrix_mdev, apm_filtered); @@ -1800,7 +1795,39 @@ static ssize_t ap_config_store(struct device *dev, s= truct device_attribute *attr vfio_ap_mdev_update_guest_apcb(matrix_mdev); reset_queues_for_apids(matrix_mdev, apm_filtered); } -out: + + return 0; +} + +static ssize_t ap_config_store(struct device *dev, struct device_attribute= *attr, + const char *buf, size_t count) +{ + struct ap_matrix_mdev *matrix_mdev =3D dev_get_drvdata(dev); + struct ap_matrix m_new; + char *newbuf, *rest; + ssize_t rc; + + newbuf =3D kstrndup(buf, AP_CONFIG_STRLEN, GFP_KERNEL); + if (!newbuf) + return -ENOMEM; + rest =3D newbuf; + + mutex_lock(&ap_attr_mutex); + get_update_locks_for_mdev(matrix_mdev); + + if (parse_bitmap(&rest, m_new.apm, AP_DEVICES) || + parse_bitmap(&rest, m_new.aqm, AP_DOMAINS) || + parse_bitmap(&rest, m_new.adm, AP_DOMAINS)) { + kfree(newbuf); + release_update_locks_for_mdev(matrix_mdev); + mutex_unlock(&ap_attr_mutex); + return -EINVAL; + } + + rc =3D vfio_ap_set_new_guest_config(matrix_mdev, &m_new); + if (!rc) + rc =3D count; + release_update_locks_for_mdev(matrix_mdev); mutex_unlock(&ap_attr_mutex); kfree(newbuf); diff --git a/drivers/s390/crypto/vfio_ap_private.h b/drivers/s390/crypto/vf= io_ap_private.h index 1fbdfcce5a11..150dfce8a674 100644 --- a/drivers/s390/crypto/vfio_ap_private.h +++ b/drivers/s390/crypto/vfio_ap_private.h @@ -157,6 +157,62 @@ struct vfio_ap_queue { struct work_struct reset_work; }; =20 +/** + * get_update_locks_for_mdev: Acquire the locks required to dynamically up= date a + * KVM guest's APCB in the proper order. + * + * @matrix_mdev: a pointer to a struct ap_matrix_mdev object containing th= e AP + * configuration data to use to update a KVM guest's APCB. + * + * The proper locking order is: + * 1. matrix_dev->guests_lock: required to use the KVM pointer to update a= KVM + * guest's APCB. + * 2. matrix_mdev->kvm->lock: required to update a guest's APCB + * 3. matrix_dev->mdevs_lock: required to access data stored in a matrix_= mdev + * + * Note: If @matrix_mdev is NULL or is not attached to a KVM guest, the KVM + * lock will not be taken. + */ +static inline void get_update_locks_for_mdev(struct ap_matrix_mdev *matrix= _mdev) +{ + mutex_lock(&matrix_dev->guests_lock); + if (matrix_mdev && matrix_mdev->kvm) + mutex_lock(&matrix_mdev->kvm->lock); + mutex_lock(&matrix_dev->mdevs_lock); +} + +/** + * release_update_locks_for_mdev: Release the locks used to dynamically up= date a + * KVM guest's APCB in the proper order. + * + * @matrix_mdev: a pointer to a struct ap_matrix_mdev object containing th= e AP + * configuration data to use to update a KVM guest's APCB. + * + * The proper unlocking order is: + * 1. matrix_dev->mdevs_lock + * 2. matrix_mdev->kvm->lock + * 3. matrix_dev->guests_lock + * + * Note: If @matrix_mdev is NULL or is not attached to a KVM guest, the KVM + * lock will not be released. + */ +static inline void release_update_locks_for_mdev(struct ap_matrix_mdev *ma= trix_mdev) +{ + mutex_unlock(&matrix_dev->mdevs_lock); + if (matrix_mdev && matrix_mdev->kvm) + mutex_unlock(&matrix_mdev->kvm->lock); + mutex_unlock(&matrix_dev->guests_lock); +} + +static inline void +assert_has_update_locks_for_mdev(struct ap_matrix_mdev *matrix_mdev) +{ + lockdep_assert_held(&matrix_dev->guests_lock); + if (matrix_mdev && matrix_mdev->kvm) + lockdep_assert_held(&matrix_mdev->kvm->lock); + lockdep_assert_held(&matrix_dev->mdevs_lock); +} + int vfio_ap_mdev_get_num_queues(struct ap_matrix *ap_matrix); =20 int vfio_ap_mdev_register(void); @@ -172,9 +228,14 @@ void vfio_ap_on_cfg_changed(struct ap_config_info *new= _config_info, void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info, struct ap_config_info *old_config_info); =20 +void vfio_ap_matrix_init(struct ap_config_info *info, struct ap_matrix *ma= trix); + void vfio_ap_init_migration_capabilities(struct ap_matrix_mdev *matrix_mde= v); int vfio_ap_init_migration_data(struct ap_matrix_mdev *matrix_mdev); void vfio_ap_release_migration_data(struct ap_matrix_mdev *matrix_mdev); void vfio_ap_reset_migration_state(struct ap_matrix_mdev *matrix_mdev); =20 +int vfio_ap_set_new_guest_config(struct ap_matrix_mdev *matrix_mdev, + struct ap_matrix *m_new); + #endif /* _VFIO_AP_PRIVATE_H_ */ --=20 2.53.0 From nobody Fri Jul 24 20:51:09 2026 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 43960446823; Fri, 24 Jul 2026 16:14:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784909670; cv=none; b=BLRYiriGUNh6GdQNILdd+O56POUUuQX6vXSw5f5/9fjHen5+WXPu8ws886VIRK7RimR8NWh8VK82Gzl0food+s39EOl1iiZWZWKkYjkNwmCQ7mHGA6Ip9QV03jMqmZG/XXPc2ywwlFXowQ1V/Zyz6Dr8r7nTEcjWnxMIaKRvimk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784909670; c=relaxed/simple; bh=WnfhDQrigzzu2s0wxYhtHgqIwjB45QseGEGDpb/t+IA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=UxPoQ8I8xmU4Wy+7549xuxhWe/ry6D+/UXZ0LbbqE4jzLQDVA1RD4psdOp0hl69EvzexYvU8jWtNeKV4fvxbybWxGC/Ge24g8Zro4/BB5hEZ1mx5V+eL+gn5B/9sNvjki+QXyYzWDGUBq8uiC+zYpUAYdRqcRoq3MA0K5iZl5JM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=ZemeS2YB; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="ZemeS2YB" Received: from pps.filterd (m0353725.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 66ODfnjE1869258; Fri, 24 Jul 2026 16:14:14 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=aoaX9VEI6UyC9mH8S WFVjb0z82tG2tTVRFFnTceFu8E=; b=ZemeS2YBbSgmHJoLUZylBsMeM2f7FmSKV VuIHULcpWmtndkRascxP0BiqmGMqr+BUES2Jb9J1a1xe6XlFcJ0W7aYiogSdcVnk Ec6HFTqP0iBufXavsNl+DTbfi9tbPDzb3agQ6ovv1mC4LT0UI+Idb7ZVl665zA44 kq95rXEUqQBuaxjCVDzbeWDVB2Ks6XZr6tlMUiswTe4d9qsTW+WW+HQtu56maJEZ 6KF4TpVeoXkK3d/D1y4U3QvwcJ6x1nWVn0AKu3OawNNT+6GbSFSfC8Fkz/dfLZtC J+aw9lPyI4mEyRgAG5r7YdF+54JzAtgR3vJ2TqzmS/0ewTB0WWC3g== Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4fg78gmvd8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:13 +0000 (GMT) Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 66OG4h2A010743; Fri, 24 Jul 2026 16:14:12 GMT Received: from smtprelay07.dal12v.mail.ibm.com ([172.16.1.9]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 4fgp1gsf8d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:12 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (smtpav06.wdc07v.mail.ibm.com [10.39.53.233]) by smtprelay07.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 66OGEB9s2818710 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 24 Jul 2026 16:14:11 GMT Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6F6355804E; Fri, 24 Jul 2026 16:14:11 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C71EC5803F; Fri, 24 Jul 2026 16:14:09 +0000 (GMT) Received: from li-4c4c4544-004d-4810-8043-b7c04f423534.ibm.com.com (unknown [9.61.50.28]) by smtpav06.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 24 Jul 2026 16:14:09 +0000 (GMT) From: Anthony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex@shazbot.org, kwankhede@nvidia.com, fiuczy@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, agordeev@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com Subject: [PATCH v5 10/15] s390/vfio-ap: File ops called to resume the vfio device migration Date: Fri, 24 Jul 2026 12:13:46 -0400 Message-ID: <20260724161351.1802644-11-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260724161351.1802644-1-akrowiak@linux.ibm.com> References: <20260724161351.1802644-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX+K1Z1U1veqNo 63GRw/izL2hVC9JzbhmlG1FlRyMbVlmRj/I7U35/70d3PB38phEqu9Uujwjf4hMgBpMoV9LQgF8 0Ykvz5xdG+JNxqyyGTE2J7TAHRIJzpnZdYyEv6sI3vVKRVOaS9NmFRNv243p4iCDyh0BDw8u6v3 9fKQExg5BFZkssXlZAZjAUsM7z/ZI+ZhGnAlbxLTyhDPBemKKf6bGFkteFfR0Wh0B3K9OrNfufN O3i6MEhTzr86/7P2QGQwhPHVNUxWISiyQXo4bjLPylqVz78V6Vv55vWRzhGQXOQUgdC6d2qPw/K zzUY7ABKqzvkiwJQOgT9mtIXeWkFUYN95cYFiqqYouYVANy3qCXlTYaBT73hGq5BgtjcOpfAfzJ RYPjBcH8GWG+5maDlzKxpeYj+dQSL2auvUqdrYAYf7Llo7hnISXELiaV5s8rFFF3R7XG2GJrGQ3 QfkltIStx36isGuXhrA== X-Proofpoint-GUID: udDULVX5dI33-QoZv8XxTOISFrBrnzfl X-Authority-Analysis: v=2.4 cv=MelcfZ/f c=1 sm=1 tr=0 ts=6a638f55 cx=c_pps a=AfN7/Ok6k8XGzOShvHwTGQ==:117 a=AfN7/Ok6k8XGzOShvHwTGQ==:17 a=RAioF0-LDSMA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=V8glGbnc2Ofi9Qvn3v5h:22 a=VnNF1IyMAAAA:8 a=dAFWD2Np9BBlLd7oX9IA:9 X-Proofpoint-Spam-Info: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX+YZ1lHYjfsNf fVlogN5QliwZb+tJjM8BjRuOqsMlN7Tiy72+yqo5HFj0Bx0YZ383PgA2KGnxd1IHYusJpunHBH7 94F7eT1h0EqesSNzG7NpBlfoIAZOc4Q= X-Proofpoint-ORIG-GUID: udDULVX5dI33-QoZv8XxTOISFrBrnzfl X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-07-24_03,2026-07-24_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 lowpriorityscore=0 priorityscore=1501 bulkscore=0 spamscore=0 clxscore=1015 malwarescore=0 phishscore=0 adultscore=0 suspectscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2607240145 Content-Type: text/plain; charset="utf-8" Implements the 'write' callback function that was added to the 'file_operations' structure for the file stream created to restore the state of the vfio-ap device on the destination system when the migration state transitioned from STOP to RESUMING The write callback retrieves the vfio device migration state saved to the file stream created when the vfio device state was transitioned from STOP to STOP_COPY. The saved state contains the source guest's AP configuration information. This data is copied from the userspace buffer passed to the 'write' callback and stored in the vfio_ap_config structure used to set the state of the vfio-ap device on the destination host. If the source guest's AP configuration is compatible with the AP configuration on the destination host, it will be hot plugged into the destination guest. In order for the source guest's and destination host's AP configurations to be considered compatible: * Each APQN in the source guest's AP configuration must also be in the destination host's AP configuration * Each matching APQN in the destination host's AP configuration must be bound to the vfio_ap device driver * Each matching APQN in the destination host's AP configuration must reference a queue device with compatible hardware: - The source and destination queues must have the same facilities installed: ~ APSC facility ~ APQKM facility ~ AP4KC facility - The source and destination queues must have the same mode: ~ Coprocessor-mode ~ Accelerator-mode ~ XCP-mode - The source and destination queues must have the same APXA facility setting ~ If the APXA facility is installed on source queue, it must also be installed on the destination queue and vice versa - The source and destination queues must have a compatible classification setting. If the source queue has full native card function, then the destination queue must also have full native card function. If the source queue has stateless functions, then the destination queue can have stateless functions or full native card function because the latter includes the stateless functions. - The binding and associated state for both the source and destination queues must indicate that the queue is usable for all messages (i.e., BS bits equal to 00). - The AP type of the destination queue must be the same as or newer than the source queue (backward compatibility) Note: The get_hardware_info_for_queue function that was created in a previous patch was modified to take a mediated device name rather than an ap_matrix_mdev object because that is what is needed for this patch so the function can be executed without holding the matrix_dev->mdevs_lock. Signed-off-by: Anthony Krowiak --- drivers/s390/crypto/vfio_ap_migration.c | 830 +++++++++++++++++++++++- 1 file changed, 826 insertions(+), 4 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/= vfio_ap_migration.c index 4a489a7a05ff..7a9a8f63a76d 100644 --- a/drivers/s390/crypto/vfio_ap_migration.c +++ b/drivers/s390/crypto/vfio_ap_migration.c @@ -8,6 +8,54 @@ #include #include "vfio_ap_private.h" =20 +/* + * Masks the fields of the queue information returned from the PQAP(TAPQ) + * command. In order to migrate a guest, it's AP configuration must be + * compatible with AP configuration assigned to the target guest's mdev. + * This mask is used to verify that the queue information for each source = and + * target queue is compatible. + * + * The following bits must match for the source device and the correspondi= ng + * destination device: + * -----------------------------------------------------------------------= -- + * S bit 0: APSC facility installed + * M bit 1: APQKM facility installed + * C bit 2: AP4KC facility installed + * Mode bits 3-5: + * D bit 3: CCA-mode facility + * A bit 4: accelerator-mode facility + * X bit 5: XCP-mode facility + * N bit 6: APXA facility installed + * SL bit 7: SLCF facility installed + * + * Either bit 8 or bit 9 will be set. If bit 8 is set for the source devic= e, + * then it must also be set for the corresponding destination device: + * -----------------------------------------------------------------------= -- + * Classification (functional capabilities) bits 8-16 + * bit 8: Native card function + * bit 9: Only stateless functions + * + * The BS bits must be set to 0 for both the source and corresponding + * destination device: + * -----------------------------------------------------------------------= -- + * BS bits 16-17: + * + * The AP type of the source device must be less than or equal to that of + * the corresponding destination device: + * -----------------------------------------------------------------------= -- + * AP Type bits 32-40: + */ +#define QINFO_DATA_MASK 0xffffc000ff000000 + +/* + * Masks the bit that indicates whether full native card function is avail= able + * from the 8 bits specifying the functional capabilities of a queue + */ +#define CLASSIFICATION_NATIVE_FCN_MASK 0x80 + +/* The maximum number of queues that can be installed in an s390 system */ +#define MAX_AP_QUEUES (AP_DEVICES * AP_DOMAINS) + /** * vfio_ap_migration_file * @@ -204,8 +252,6 @@ static int get_hardware_info_for_queue(const char *mdev= _name, status.response_code); return -EIO; } - - return -EINVAL; } =20 static int vfio_ap_store_queue_info(const char *mdev_name, @@ -409,11 +455,787 @@ static struct file *vfio_ap_open_file_stream(struct = ap_matrix_mdev *matrix_mdev, return filp; } =20 +static int validate_resuming_write_parms(struct file *filp, + size_t len, loff_t *pos) +{ + struct ap_matrix_mdev *matrix_mdev; + loff_t total_len; + int ret; + + lockdep_assert_held(&matrix_dev->mdevs_lock); + + matrix_mdev =3D filp->private_data; + if (!matrix_mdev || !matrix_mdev->mig_data) { + ret =3D -ENODEV; + goto done; + } + + if (filp !=3D matrix_mdev->mig_data->resuming_mig_file.filp) { + ret =3D -ENXIO; + goto done; + } + + if (*pos < 0) { + ret =3D -EINVAL; + goto done; + } + + if (check_add_overflow((loff_t)len, *pos, &total_len)) { + ret =3D -ERANGE; + goto done; + } + + /* + * If the ap_config has not yet been allocated and the file position + * indicates this is not the first write, or the ap_config has been alloc= ated + * but the file position indicates this is the first write, then this is = an + * error condition. + */ + if ((!matrix_mdev->mig_data->resuming_mig_file.ap_config && *pos !=3D 0) = || + (matrix_mdev->mig_data->resuming_mig_file.ap_config && *pos =3D=3D 0)= ) { + ret =3D -EFAULT; + + goto done; + } + + ret =3D 0; + +done: + return ret; +} + +static ssize_t calculate_ap_config_size(unsigned int num_queues) +{ + size_t qinfo_size; + + if (num_queues > MAX_AP_QUEUES) + return -EINVAL; + + qinfo_size =3D num_queues * sizeof(struct vfio_ap_queue_info); + return qinfo_size + sizeof(struct vfio_ap_config); +} + +/** + * allocate_ap_config: + * + * Allocate storage for the source guest's AP configuration data sent from + * userspace. + * + * @ap_config: The location in which to store the pointer to the storage + * allocated for the AP configuration data. + * @buf: The userspace buffer containing some or all of the source + * guest's AP configuration data + * @len: The number of bytes of data to copy from @buf + * + * Returns: The number of bytes of storage allocated for the config data or + * an error: + * + * + * -EIO: failed to copy data from @buf + * -EINVAL: the number of queues specified exceeds the max allowed + * -ENOMEM: the allocation of storage failed + */ +static ssize_t allocate_ap_config(struct vfio_ap_config **ap_config, + const char __user *buf, size_t len) +{ + struct vfio_ap_config tmp_ap_config; + ssize_t config_size; + size_t copy_size; + + /* + * If the length of the data sent exceeds the size of the vfio_ap_config + * structure, then we will copy enough data from userspace to get the + * number of queues which we can use to allocate enough space all of + * the queue information. + */ + copy_size =3D min(len, sizeof(tmp_ap_config)); + + if (copy_from_user(&tmp_ap_config, buf, copy_size)) + return -EIO; + + /* + * If the length of data sent includes the number of queues + * in the AP configuration, then calculate its size; otherwise + * set config_size to the length of data sent. + */ + if (len >=3D sizeof(struct vfio_ap_config)) { + config_size =3D calculate_ap_config_size(tmp_ap_config.num_queues); + + /* If the calculation returned an error */ + if (config_size < 0) + return config_size; + } else { + config_size =3D len; + } + + *ap_config =3D kzalloc(config_size, GFP_KERNEL_ACCOUNT); + if (!*ap_config) + return -ENOMEM; + + return config_size; +} + +static ssize_t reallocate_ap_config(struct vfio_ap_migration_file *mig_fil= e, + struct vfio_ap_config **new_ap_config, + size_t len) +{ + struct vfio_ap_config *cur_ap_config; + unsigned long cur_cfg_sz; + ssize_t new_cfg_sz; + + cur_ap_config =3D mig_file->ap_config; + cur_cfg_sz =3D mig_file->config_sz; + + /* + * If the current configuration size is greater than the + * size of a vfio_ap_config structure (i.e., contains the num_queues + * field), then there should already be enough storage allocated + * to store the source guest's AP configuration. Let's verify that the + * amount of storage allocated is what we expect based on the number of + * vfio_ap_queue_info objects that must be stored. + */ + if (cur_cfg_sz >=3D sizeof(struct vfio_ap_config)) { + new_cfg_sz =3D calculate_ap_config_size(cur_ap_config->num_queues); + if (cur_cfg_sz !=3D new_cfg_sz) + return -EINVAL; + } else { + if (check_add_overflow(cur_cfg_sz, len, &new_cfg_sz)) + return -ERANGE; + } + + *new_ap_config =3D kzalloc(new_cfg_sz, GFP_KERNEL_ACCOUNT); + if (!*new_ap_config) + return -ENOMEM; + + memcpy(*new_ap_config, cur_ap_config, cur_cfg_sz); + + return new_cfg_sz; +} + +/** + * qdev_is_bound_to_vfio_ap: + * + * Query to determine whether a queue with the specified APQN is available= on + * the host system and bound to the vfio_ap device driver. + * + * @apqn: The APQN of the queue device being queried + * + * Returns: True if there is a queue device with the specified @apqn insta= lled + * in the system and is bound to the vfio_ap device driver; otherwise, + * returns false. + */ +static bool qdev_is_bound_to_vfio_ap(unsigned int apqn) +{ + struct ap_queue *queue; + bool is_bound =3D true; + + queue =3D ap_get_qdev(apqn); + if (!queue) + return false; + + if (queue->ap_dev.device.driver !=3D &matrix_dev->vfio_ap_drv->driver) + is_bound =3D false; + + put_device(&queue->ap_dev.device); + + return is_bound; +} + +/** + * queues_available: + * + * Query whether each queue from the source guest's AP configuration is + * available and bound to the vfio_ap device driver; if not, log an error + * message. + * + * @mdev_name: The mdev name to use in error messages + * @source_config: The object specifying the source guest's AP configurati= on + * + * Returns: true if each queue identified in @source_config is available a= nd + * bound to the vfio_ap device driver; otherwise, returns false. + */ +static bool queues_available(const char *mdev_name, + struct vfio_ap_config *source_config) +{ + unsigned long apqn; + bool ret =3D true; + + for (int i =3D 0; i < source_config->num_queues; i++) { + apqn =3D source_config->qinfo[i].apqn; + + /* + * Find the queue device bound to the vfio_ap device driver. If it is + * not found, log an error and continue so users see all problems + * at once, not one-at-a-time through retries of the migration. + */ + if (!qdev_is_bound_to_vfio_ap(apqn)) { + pr_err("vfio_ap_mdev %s: Queue %02lx.%04lx not available to vfio_ap dri= ver on target host\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + ret =3D false; + } + } + + return ret; +} + +/** + * control_domains_available + * + * Query whether each control domain specified in the source guest's AP + * configuration is installed in the host system. + * + * @mdev_name: The name of the mdev to use when logging messages + * @source_config: The object specifying the source guest's AP config + * + * Returns: True if each control domain is installed; otherwise, logs an + * error message for each unavailable control domain and returns + * false. + */ +static bool control_domains_available(const char *mdev_name, + struct vfio_ap_config *source_config) +{ + unsigned long domain_num; + bool available =3D true; + + for_each_set_bit_inv(domain_num, (unsigned long *)source_config->adm, + AP_DOMAINS) { + if (!ap_test_config_ctrl_domain(domain_num)) { + pr_err("vfio_ap_mdev: %s: Control domain %04lx not available on the des= tination host", + mdev_name, domain_num); + available =3D false; + } + } + + return available; +} + +static void report_facilities_compatibility(const char *mdev_name, + unsigned long apqn, + struct ap_tapq_hwinfo *src_hwinfo, + struct ap_tapq_hwinfo *target_hwinfo) +{ + if (src_hwinfo->apsc !=3D target_hwinfo->apsc) { + if (src_hwinfo->apsc) { + pr_err("vfio_ap_mdev %s: APSC facility installed in source queue %02lx.= %04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + + pr_err("vfio_ap_mdev %s: APSC facility not installed in target queue %0= 2lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + } else { + pr_err("vfio_ap_mdev %s: APSC facility not installed in source queue %0= 2lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + + pr_err("vfio_ap_mdev %s APSC facility installed in target queue %02lx.%= 04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + } + } + + if (src_hwinfo->mex4k !=3D target_hwinfo->mex4k) { + if (src_hwinfo->mex4k) { + pr_err("vfio_ap_mdev %s: mex4k facility installed in source queue %02lx= .%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + + pr_err("vfio_ap_mdev %s: mex4k facility not installed in target queue %= 02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + } else { + pr_err("vfio_ap_mdev %s: mex4k facility not installed in source queue %= 02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + + pr_err("vfio_ap_mdev %s: mex4k facility installed in target queue %02lx= .%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + } + } + + if (src_hwinfo->crt4k !=3D target_hwinfo->crt4k) { + if (src_hwinfo->crt4k) { + pr_err("vfio_ap_mdev %s: crt4k facility installed in source queue %02lx= .%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + + pr_err("vfio_ap_mdev %s: crt4k facility not installed in target queue %= 02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + } else { + pr_err("vfio_ap_mdev %s: crt4k facility not installed in source queue %= 02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + + pr_err("vfio_ap_mdev %s: crt4k facility installed in target queue %02lx= .%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + } + } +} + +static void report_mode_compatibility(const char *mdev_name, + unsigned long apqn, + struct ap_tapq_hwinfo *src_hwinfo, + struct ap_tapq_hwinfo *target_hwinfo) +{ + if (src_hwinfo->cca !=3D target_hwinfo->cca) { + if (src_hwinfo->cca) { + pr_err("vfio_ap_mdev %s: Coprocessor-mode facility installed in source = queue %02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + + pr_err("vfio_ap_mdev %s: Coprocessor-mode facility not installed targe= t queue %02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + } else { + pr_err("vfio_ap_mdev %s: Coprocessor-mode facility not installed in sou= rce queue %02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + + pr_err("vfio_ap_mdev %s: Coprocessor-mode facility installed target qu= eue %02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + } + } + + if (src_hwinfo->accel !=3D target_hwinfo->accel) { + if (src_hwinfo->accel) { + pr_err("vfio_ap_mdev %s: Accelerator-mode facility installed source que= ue %02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + + pr_err("vfio_ap_mdev %s: Accelerator-mode facility not installed target= queue %02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + } else { + pr_err("vfio_ap_mdev %s: Accelerator-mode facility not installed source= queue %02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + + pr_err("vfio_ap_mdev %s: Accelerator-mode facility installed target que= ue %02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + } + } + + if (src_hwinfo->ep11 !=3D target_hwinfo->ep11) { + if (src_hwinfo->ep11) { + pr_err("vfio_ap_mdev %s: XCP-mode facility installed source queue %02lx= .%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + + pr_err("vfio_ap_mdev %s: XCP-mode facility not installed target queue %= 02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + } else { + pr_err("vfio_ap_mdev %s: XCP-mode facility not installed source queue %= 02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + + pr_err("vfio_ap_mdev %s: XCP-mode facility installed target queue %02lx= .%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + } + } +} + +static void report_apxa_compatibility(const char *mdev_name, + unsigned long apqn, + struct ap_tapq_hwinfo *src_hwinfo, + struct ap_tapq_hwinfo *target_hwinfo) +{ + if (src_hwinfo->apxa !=3D target_hwinfo->apxa) { + if (src_hwinfo->apxa) { + pr_err("vfio_ap_mdev %s: AP-extended-addressing (APXA) facility install= ed in source queue %02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + + pr_err("vfio_ap_mdev %s: AP-extended-addressing (APXA) facility not ins= talled in target queue %02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + } else { + pr_err("vfio_ap_mdev %s: AP-extended-addressing (APXA) facility not ins= talled in source queue %02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + + pr_err("vfio_ap_mdev %s: AP-extended-addressing (APXA) facility install= ed in target queue %02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + } + } +} + +static void report_slcf_compatibility(const char *mdev_name, + unsigned long apqn, + struct ap_tapq_hwinfo *src_hwinfo, + struct ap_tapq_hwinfo *target_hwinfo) +{ + if (src_hwinfo->slcf !=3D target_hwinfo->slcf) { + if (src_hwinfo->slcf) { + pr_err("vfio_ap_mdev %s: Stateless-command-filtering (SLCF) available i= n source queue %02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + + pr_err("vfio_ap_mdev %s: Stateless-command-filtering (SLCF) not availab= le in target queue %02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + } else { + pr_err("vfio_ap_mdev %s: Stateless-command-filtering (SLCF) not availab= le in source queue %02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + + pr_err("vfio_ap_mdev %s: Stateless-command-filtering (SLCF) available i= n target queue %02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + } + } +} + +static void report_bs_compatibility(const char *mdev_name, + unsigned long apqn, + struct ap_tapq_hwinfo *src_hwinfo, + struct ap_tapq_hwinfo *target_hwinfo) +{ + /* + * The BS field on both the source and destination must be 0, so if one + * of them is not, then report an error. + */ + if (src_hwinfo->bs || target_hwinfo->bs) { + pr_err("vfio_ap_mdev %s: Bind/associate state for source (%01x) and targ= et (%01x) queue %02lx.%04lx must be 0\n", + mdev_name, src_hwinfo->bs, target_hwinfo->bs, + AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + } +} + +static void report_aptype_compatibility(const char *mdev_name, + unsigned long apqn, + struct ap_tapq_hwinfo *src_hwinfo, + struct ap_tapq_hwinfo *target_hwinfo) +{ + if (src_hwinfo->at > target_hwinfo->at) { + pr_err("vfio_ap_mdev %s: AP type of source (%02x) not compatible with ta= rget (%02x)\n", + mdev_name, src_hwinfo->at, target_hwinfo->at); + } +} + +static bool classes_compatible(struct ap_tapq_hwinfo *src_hwinfo, + struct ap_tapq_hwinfo *target_hwinfo) +{ + unsigned long src_native, target_native; + + src_native =3D src_hwinfo->class & CLASSIFICATION_NATIVE_FCN_MASK; + target_native =3D target_hwinfo->class & CLASSIFICATION_NATIVE_FCN_MASK; + + /* + * If the source queue has full native card function and the + * target queue has only stateless functions available, then + * there may be instructions that will not execute on the + * target queue. This shall be reported as an error. + * + * If the source queue has only stateless card functions and the + * target queue has full native card function available, then + * we are okay because the target queue can run all stateless card + * functions. + */ + return (src_native !=3D target_native) ? !src_native : true; +} + +static void report_class_compatibility(const char *mdev_name, + unsigned long apqn, + struct ap_tapq_hwinfo *src_hwinfo, + struct ap_tapq_hwinfo *target_hwinfo) +{ + if (!classes_compatible(src_hwinfo, target_hwinfo)) { + pr_err("vfio_ap_mdev %s: Full native card function available on source q= ueue %02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + + pr_err("vfio_ap_mdev %s: Only stateless functions available on target qu= eue %02lx.%04lx\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + } +} + +/* + * Log a device error reporting that migration failed due to queue + * incompatibilities followed by a device error for each incompatible feat= ure. + */ +static void report_qinfo_incompatibilities(const char *mdev_name, + unsigned long apqn, + struct ap_tapq_hwinfo *src_hwinfo, + struct ap_tapq_hwinfo *target_hwinfo) +{ + pr_err("vfio_ap_mdev %s: Migration failed: Source and target queue (%02lx= .%04lx) not compatible\n", + mdev_name, AP_QID_CARD(apqn), AP_QID_QUEUE(apqn)); + + report_facilities_compatibility(mdev_name, apqn, src_hwinfo, target_hwinf= o); + report_mode_compatibility(mdev_name, apqn, src_hwinfo, target_hwinfo); + report_apxa_compatibility(mdev_name, apqn, src_hwinfo, target_hwinfo); + report_slcf_compatibility(mdev_name, apqn, src_hwinfo, target_hwinfo); + report_aptype_compatibility(mdev_name, apqn, src_hwinfo, target_hwinfo); + report_bs_compatibility(mdev_name, apqn, src_hwinfo, target_hwinfo); + report_class_compatibility(mdev_name, apqn, src_hwinfo, target_hwinfo); +} + +/** + * queue_hardware_info_is_compatible: + * + * Verify whether the hardware information for a source queue is compatibl= e with + * the hardware info for the corresponding queue on this system. + * + * In order to be compatible, the hardware information for each queue must + * meet the following requirements: + * + * 1. The hardware facilities bits much match + * 2. The AP type of the source queue must be the same as or older than th= at + * of the target queue (target is backwards compatible) + * 3. The classification bits must indicate: + * - Both queues have full native card function or both have stateless + * functions available + * - If the classification bits don't match, then the only acceptable + * configuration is stateless functions for the source queue and + * full native function for the target queue + * 4. The BS bits for both queues must be 0 (Queue usable for all messages + * supported by the adapter) + * + * @mdev_name: The mdev name to use in error messages + * @apqn: The APQN for the queues + * @src_hwinfo: The hardware info for the source queue + * @target_hwinfo: The hardware info for the corresponding queue on this s= ystem + * + * Returns: true if the hardware info for the two queues is compatible; + * otherwise, returns false. + */ +static bool queue_hardware_info_is_compatible(const char *mdev_name, + unsigned long apqn, + struct ap_tapq_hwinfo *src_hwinfo, + struct ap_tapq_hwinfo *target_hwinfo) +{ + unsigned long src_bits, target_bits; + + src_bits =3D src_hwinfo->value & QINFO_DATA_MASK; + target_bits =3D target_hwinfo->value & QINFO_DATA_MASK; + + /* If all bits match the queues are compatible */ + if (src_bits =3D=3D target_bits && + (src_hwinfo->bs =3D=3D 0 && target_hwinfo->bs =3D=3D 0)) + return true; + + if (src_hwinfo->apsc =3D=3D target_hwinfo->apsc && + src_hwinfo->mex4k =3D=3D target_hwinfo->mex4k && + src_hwinfo->crt4k =3D=3D target_hwinfo->crt4k && + src_hwinfo->cca =3D=3D target_hwinfo->cca && + src_hwinfo->accel =3D=3D target_hwinfo->accel && + src_hwinfo->ep11 =3D=3D target_hwinfo->ep11 && + src_hwinfo->slcf =3D=3D target_hwinfo->slcf && + src_hwinfo->apxa =3D=3D target_hwinfo->apxa && + src_hwinfo->at <=3D target_hwinfo->at && + classes_compatible(src_hwinfo, target_hwinfo) && + (src_hwinfo->bs =3D=3D 0 && target_hwinfo->bs =3D=3D 0)) + return true; + + report_qinfo_incompatibilities(mdev_name, apqn, src_hwinfo, target_hwinfo= ); + + return false; +} + +/** + * verify_ap_configs_are_compatible: + * + * Verifies that the queues in the source guest's AP configuration are + * compatible with the corresponding queues on this system. + * + * @mdev_name: The mdev name to use in error messages + * @source_config: The object specifying the source guest's AP configurati= on + * + * Returns: an error indicating either a failure to retrieve a queue's + * hardware information or one or more source queues are not + * compatible with the corresponding queue on this system; otherwise, + * returns zero to indicate compatibility. + */ +static int verify_ap_configs_are_compatible(const char *mdev_name, + struct vfio_ap_config *source_config) +{ + struct ap_tapq_hwinfo src_hwinfo, dest_hwinfo; + unsigned long apqn; + int ret =3D 0, rc; + + for (int i =3D 0; i < source_config->num_queues; i++) { + apqn =3D source_config->qinfo[i].apqn; + + /* + * If we can't get the hardware info for a particular queue, then let's + * capture the function return code and continue so we can log all + * errors to aid in debugging of migration. + */ + rc =3D get_hardware_info_for_queue(mdev_name, &dest_hwinfo, apqn); + if (rc) { + ret =3D rc; + continue; + } + + src_hwinfo.value =3D source_config->qinfo[i].data; + + if (!queue_hardware_info_is_compatible(mdev_name, apqn, + &src_hwinfo, + &dest_hwinfo)) + ret =3D -EINVAL; + } + + return ret; +} + +static int do_post_copy_validation(const char *mdev_name, + struct vfio_ap_config *source_config) +{ + if (!queues_available(mdev_name, source_config)) + return -ENODEV; + + if (!control_domains_available(mdev_name, source_config)) + return -ENODEV; + + return verify_ap_configs_are_compatible(mdev_name, source_config); +} + +/** + * setup_ap_matrix_from_ap_config: + * + * Set the bits corresponding to the adapters, domains and control domains + * from the source guest's AP configuration into an ap_matrix object to be + * used to update the destination guest to run on this host. + * + * @ap_config: The source guest's AP configuration + * @guest_matrix: The object to be used to update the destination guest's + * AP configuration + */ +static void setup_ap_matrix_from_ap_config(struct vfio_ap_config *ap_confi= g, + struct ap_matrix *guest_matrix) +{ + struct ap_config_info host_config_info =3D { 0 }; + unsigned long apid, apqi, *guest_adm; + struct vfio_ap_queue_info qinfo; + + ap_qci(&host_config_info); + vfio_ap_matrix_init(&host_config_info, guest_matrix); + + for (int i =3D 0; i < ap_config->num_queues; i++) { + qinfo =3D ap_config->qinfo[i]; + apid =3D AP_QID_CARD(qinfo.apqn); + apqi =3D AP_QID_QUEUE(qinfo.apqn); + + if (!test_bit_inv(apid, guest_matrix->apm)) + set_bit_inv(apid, guest_matrix->apm); + if (!test_bit_inv(apqi, guest_matrix->aqm)) + set_bit_inv(apqi, guest_matrix->aqm); + } + + guest_adm =3D (unsigned long *)ap_config->adm; + for_each_set_bit_inv(apqi, guest_adm, AP_DOMAINS) { + if (!test_bit_inv(apqi, guest_matrix->adm)) + set_bit_inv(apqi, guest_matrix->adm); + } +} + static ssize_t vfio_ap_resuming_write(struct file *filp, const char __user= *buf, size_t len, loff_t *pos) { - /* TODO */ - return -EOPNOTSUPP; + struct vfio_ap_migration_data *mig_data; + struct ap_matrix_mdev *matrix_mdev; + struct vfio_ap_config *ap_config; + struct ap_matrix guest_matrix; + ssize_t ret =3D 0, cfg_sz; + const char *mdev_name; + + /* + * When userspace calls write() with an explicit offset (pwrite), pos is + * non-NULL and the function rejects it with -ESPIPE (illegal seek). For + * normal write() calls, pos is NULL, so we'll use the file's internal + * position filp->f_pos + */ + if (pos) + return -ESPIPE; + + mutex_lock(&matrix_dev->mdevs_lock); + pos =3D &filp->f_pos; + + ret =3D validate_resuming_write_parms(filp, len, pos); + if (ret) { + mutex_unlock(&matrix_dev->mdevs_lock); + return ret; + } + + matrix_mdev =3D filp->private_data; + mig_data =3D matrix_mdev->mig_data; + mdev_name =3D dev_name(matrix_mdev->vdev.dev); + + /* + * If this is the first write operation, then allocate storage for the + * AP configuration information; otherwise, reallocate the + * struct vfio_ap_config object used to store the AP configuration data + * sent from userspace. + */ + if (*pos =3D=3D 0) { + ret =3D allocate_ap_config(&ap_config, buf, len); + + /* If the allocation failed, we'll return the error */ + if (ret < 0) { + mutex_unlock(&matrix_dev->mdevs_lock); + return ret; + } + + cfg_sz =3D ret; + } else { + ret =3D reallocate_ap_config(&mig_data->resuming_mig_file, + &ap_config, len); + if (ret < 0) { + mutex_unlock(&matrix_dev->mdevs_lock); + return ret; + } + + cfg_sz =3D ret; + } + + if (*pos + len > cfg_sz) { + ret =3D -EIO; + kfree(ap_config); + mutex_unlock(&matrix_dev->mdevs_lock); + return ret; + } + + /* + * We don't want to lock all mdevs while copying data from userspace so + * we don't block all other mdevs in case the I/O takes a long time. + * From here on out we don't need this lock because we are not + * accessing the matrix_mdev until we need to get the update + * locks to set the new destination guest's AP configuration in which + * case this lock will be taken then. + */ + mutex_unlock(&matrix_dev->mdevs_lock); + + if (copy_from_user((char *)ap_config + *pos, buf, len)) { + ret =3D -EIO; + kfree(ap_config); + + return ret; + } + + /* Check if we've completed writing the entire configuration */ + if (*pos + len =3D=3D cfg_sz) { + /* Acquire locks required to update the guest's AP config */ + mutex_lock(&ap_attr_mutex); + get_update_locks_for_mdev(matrix_mdev); + + /* + * Verify the device wasn't closed while mdevs_lock was dropped + * for the copy_from_user above. get_update_locks_for_mdev() + * reacquires mdevs_lock, so the check is safe here. + */ + if (!matrix_mdev->mig_data) { + release_update_locks_for_mdev(matrix_mdev); + mutex_unlock(&ap_attr_mutex); + kfree(ap_config); + return -ENODEV; + } + + ret =3D do_post_copy_validation(mdev_name, ap_config); + if (ret < 0) { + kfree(ap_config); + release_update_locks_for_mdev(matrix_mdev); + mutex_unlock(&ap_attr_mutex); + return ret; + } + + setup_ap_matrix_from_ap_config(ap_config, &guest_matrix); + + ret =3D vfio_ap_set_new_guest_config(matrix_mdev, &guest_matrix); + + release_update_locks_for_mdev(matrix_mdev); + mutex_unlock(&ap_attr_mutex); + + if (ret) { + kfree(ap_config); + return ret; + } + } + + mutex_lock(&matrix_dev->mdevs_lock); + kfree(mig_data->resuming_mig_file.ap_config); + mig_data->resuming_mig_file.ap_config =3D ap_config; + mig_data->resuming_mig_file.config_sz =3D cfg_sz; + mutex_unlock(&matrix_dev->mdevs_lock); + + ret =3D len; + *pos +=3D len; + + return ret; } =20 static const struct file_operations vfio_ap_resume_fops =3D { --=20 2.53.0 From nobody Fri Jul 24 20:51:09 2026 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B80FA4457CE; Fri, 24 Jul 2026 16:39:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784911150; cv=none; b=Rqbw17eyDpAxIZ8C9JYp+gAnn6YCfk5lX7tBlxJLRBvCiYf239DFTMyQHzDVN6JgJ1pQy1UEqHaArUKsncbKuAl3wC1YgyQKuZb9U5viFud52LaBLGVFjp0/3k8c25dNxXxhVJ4fyboXayyi6A7m48f30grBjgZQX18vedBO15o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784911150; c=relaxed/simple; bh=TphgFbBAAzdQNcdeXOt1dvVgb7T89uNXNp/rohtleMY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=IOSweUJEN7v6QlOdp+yrEtr/Sm9dB5sIqAC694xCeiVTVgX7qCNZdzBYTYrec/GYW0woQUJ6t37phyxD+p8BLPCaA7j3NUUbBZhTez2Ts+DMvD1C8tFQZalkPEKcWM0+2ujRVimvBvihYqvDBXjRm58RPGyy7rt7zVsRNI0dMV0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=db6FjRWc; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="db6FjRWc" Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 66ODfffG100480; Fri, 24 Jul 2026 16:14:16 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=Cewiy1c02KRJnFsfM b1roLRPbtF0Hw4p68nbusKGSwE=; b=db6FjRWczZzvxtVuOH/OYq+b9OCt7zFPE hxDHjfmyocSKOUH1ZNf8Qo5P3ki1UvGrNKTuLKXFrEcp12sh2Zbr9SMOQQfzxnIl ytbMFwLJWRqPf+7+DPLC5cBa24YvK0MuZTUuDoGMXJHK0pk9aO0qEjAKLUv4qRuv PoMECF7txYShyTPO0v5/yYc2kVgBPc7oJdJJtxZl6zz4f7F71ksdlWfnFAQey6ZU tDHUQNWTnuOpq+vwRsLa/VeCgXIQlTbCObf9p/UE4Jdf3pbLM7D1zmaTy85Vw5ul x/MVkC7XW11iT1FXt8cE3Vzim6RPT25DJue15dDkoGv7vtAU6T8Mg== Received: from ppma13.dal12v.mail.ibm.com (dd.9e.1632.ip4.static.sl-reverse.com [50.22.158.221]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4fm8gs8u6t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:15 +0000 (GMT) Received: from pps.filterd (ppma13.dal12v.mail.ibm.com [127.0.0.1]) by ppma13.dal12v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 66OG4iO8010748; Fri, 24 Jul 2026 16:14:14 GMT Received: from smtprelay02.wdc07v.mail.ibm.com ([172.16.1.69]) by ppma13.dal12v.mail.ibm.com (PPS) with ESMTPS id 4fgp1gsf8g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:14 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (smtpav06.wdc07v.mail.ibm.com [10.39.53.233]) by smtprelay02.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 66OGEDWR19858122 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 24 Jul 2026 16:14:13 GMT Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 290635804E; Fri, 24 Jul 2026 16:14:13 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 998755803F; Fri, 24 Jul 2026 16:14:11 +0000 (GMT) Received: from li-4c4c4544-004d-4810-8043-b7c04f423534.ibm.com.com (unknown [9.61.50.28]) by smtpav06.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 24 Jul 2026 16:14:11 +0000 (GMT) From: Anthony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex@shazbot.org, kwankhede@nvidia.com, fiuczy@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, agordeev@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com Subject: [PATCH v5 11/15] s390/vfio-ap: Transition device migration state to STOP Date: Fri, 24 Jul 2026 12:13:47 -0400 Message-ID: <20260724161351.1802644-12-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260724161351.1802644-1-akrowiak@linux.ibm.com> References: <20260724161351.1802644-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-Spam-Info: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX4cM2PRYoIRlN Nn5Yzjp5Pml9j6NP2CuYDZyKwIjWoTEgzqwBfTB/ogz+WbEo+J1DUzKzpxrMSZ6l/05iJneqI6e XUY+l7hYU/f8rTiav+4agZsusKokfSU= X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX1jVizn/WUp/G sgfjwtikk6LbIAbczKgnoL8qbHkn1o5PZWTupGpOi9QsHU9NdTqbJpfuCLonTGttj6tdcvR7Je2 YiLZpK7EbOsgyAlOT9YgWsbZq7iulb+PCyXPcs7uNhDMYknF8tPBeWDHFWuSWRCgtpmRNFAOchU eOsiWM10pYuC1zbTEMENRAPAZ8XixRiCVKAGeJ1/jUUP7bBxwr7kza7fYZLpTrqANcGiP8DuIa3 SAkt03WZePJ87mF9FWREYpSpzLPKrlx9Uw/gUuPXJ4q/DDojL6pdmX1KwKOt8Odsyo8ZKFQGZ3F TxMuyc+GQDTL/c7+BlMDBV8taZ/fcFVmUX9y0eJv8j4xJ6bbo0uGZ2SEwWsnENw7XbQUBmVpMT7 05w+xgwHMhF/j3smuEGB/8PN3Db2ZX1SXo6zWRGQp3Z9ZxcwTirwPTzKDQjkfrF6q6RD9DAE4Pu vEG2/ttzfrFlri8tIBA== X-Authority-Analysis: v=2.4 cv=Q9LiJY2a c=1 sm=1 tr=0 ts=6a638f57 cx=c_pps a=AfN7/Ok6k8XGzOShvHwTGQ==:117 a=AfN7/Ok6k8XGzOShvHwTGQ==:17 a=RAioF0-LDSMA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=uAbxVGIbfxUO_5tXvNgY:22 a=VnNF1IyMAAAA:8 a=LlGlNFsFWGapDO07ubkA:9 X-Proofpoint-ORIG-GUID: l9ghG5J66PFCOgvTAB-2RYJE_LLAoqM0 X-Proofpoint-GUID: l9ghG5J66PFCOgvTAB-2RYJE_LLAoqM0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-07-24_03,2026-07-24_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 malwarescore=0 adultscore=0 phishscore=0 bulkscore=0 spamscore=0 priorityscore=1501 impostorscore=0 suspectscore=0 lowpriorityscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2607240145 Content-Type: text/plain; charset="utf-8" Typically at this point, interrupt processing for the device, DMA and other changes to the device's external state must be prohibited. For VFIO AP, there is no physical DMA device virtualized by the vfio-ap device . It merely manages the guest's AP configuration that identifies the devices passed through to the guest, but does not have access to any of them. Those AP devices are passed through and controlled exclusively by the SIE program used to start the guest. There only thing to do at this point is return NULL to the caller since the file streams and associated data will get cleaned up as a result of the VFIO migration framework releasing the files. Signed-off-by: Anthony Krowiak --- drivers/s390/crypto/vfio_ap_migration.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/= vfio_ap_migration.c index 7a9a8f63a76d..21cd9810a18c 100644 --- a/drivers/s390/crypto/vfio_ap_migration.c +++ b/drivers/s390/crypto/vfio_ap_migration.c @@ -1304,12 +1304,17 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *= matrix_mdev, return filp; } =20 + /* + * Terminates the data transfer session of the vfio-ap device state + * between the source and target hosts. Since the vfio-ap device does + * not virtualize a DMA device, there is no internal device state to + * incorporate into the vfio-ap device on the target. + */ if ((cur_state =3D=3D VFIO_DEVICE_STATE_RESUMING && new_state =3D=3D VFIO_DEVICE_STATE_STOP) || (cur_state =3D=3D VFIO_DEVICE_STATE_STOP_COPY && new_state =3D=3D VFIO_DEVICE_STATE_STOP)) { - /* TODO */ - return ERR_PTR(-EOPNOTSUPP); + return NULL; } =20 if ((cur_state =3D=3D VFIO_DEVICE_STATE_STOP && --=20 2.53.0 From nobody Fri Jul 24 20:51:09 2026 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 419731E9906; Fri, 24 Jul 2026 17:20:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784913639; cv=none; b=OHkoPuZwQzFObB/C75FMAyNF3lN8yC9LUXh9jauS+yDunntADoWaf2cAbdwg2cJrk+3XJZsXO3rIj3SEnVAWXX7Ns/TY5gd7H3gHvkWsXNn2VlvT/pOVov80YD1//kC1z8VWuG5HTcdIhKI86bXXgL+1WV27Nf/mZzfMkeoZMhQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784913639; c=relaxed/simple; bh=F6EPhn4D7yRtGRsiR4G4CQeG9B089+lPX3nW/Giv13k=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KleUnjbdPkNY+HdrN6gP58bDP4YCnFK7p1D58R6V7w78vGc1vWdkTmDMr0bjNokG0KxfvPLy2nh7UHahHP0f8Kg9vkHKH6LGjdjqwyn0+3lrvQU8cDD0O/cYtZ8RHZ/C3LrB8waYdtvfPC+zX17bCpmB3m7pLRlHkkU+dH2f6eE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=ZFBOR8QF; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="ZFBOR8QF" Received: from pps.filterd (m0356517.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 66ODfgIG1992890; Fri, 24 Jul 2026 16:14:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=Q53JyTInkQbpzOZiA 29V6fcw0t5yC8ViDTnjFXo/AvY=; b=ZFBOR8QFeER3o7TE7eGxg8g28ARmGGcfa 1oEe3U5KHIGZ9IBejuOC/MHjup2BOeZO9cfsvvwUbjHSHtUrmCLtf9M6a8O/10Eb siQQgVei+rZ4hKmI2vF2K4I8xoen6e2IAtX3oLg6YxT0Dr1Fl2FNdRKmj1FRsC8E fbw51pbt57PhuQtFTuBH5BQVQwalyZqSHXTlG6Fnjp7s2zSnAtp+Yzfej/wk34SM CoTAflxOirYdE2Gv//ivJFDoEnrz750UfmWAKlB7DKYy5aQpMHjXONL8lQgmB6q5 sNzYNoBgHjTVWTDOYSlxpGKBhgTmitBX1vgtJZntNBrFAI1ZtdY6g== Received: from ppma21.wdc07v.mail.ibm.com (5b.69.3da9.ip4.static.sl-reverse.com [169.61.105.91]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4fm8h00u20-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:17 +0000 (GMT) Received: from pps.filterd (ppma21.wdc07v.mail.ibm.com [127.0.0.1]) by ppma21.wdc07v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 66OG4cRP028401; Fri, 24 Jul 2026 16:14:16 GMT Received: from smtprelay03.wdc07v.mail.ibm.com ([172.16.1.70]) by ppma21.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4fgmtk9n2v-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:16 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (smtpav06.wdc07v.mail.ibm.com [10.39.53.233]) by smtprelay03.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 66OGDc8f56164716 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 24 Jul 2026 16:13:39 GMT Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D604558055; Fri, 24 Jul 2026 16:14:14 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5313F58054; Fri, 24 Jul 2026 16:14:13 +0000 (GMT) Received: from li-4c4c4544-004d-4810-8043-b7c04f423534.ibm.com.com (unknown [9.61.50.28]) by smtpav06.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 24 Jul 2026 16:14:13 +0000 (GMT) From: Anthony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex@shazbot.org, kwankhede@nvidia.com, fiuczy@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, agordeev@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com Subject: [PATCH v5 12/15] s390/vfio-ap: Transition device migration state from STOP to RUNNING and vice versa Date: Fri, 24 Jul 2026 12:13:48 -0400 Message-ID: <20260724161351.1802644-13-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260724161351.1802644-1-akrowiak@linux.ibm.com> References: <20260724161351.1802644-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Authority-Analysis: v=2.4 cv=LdcMLDfi c=1 sm=1 tr=0 ts=6a638f59 cx=c_pps a=GFwsV6G8L6GxiO2Y/PsHdQ==:117 a=GFwsV6G8L6GxiO2Y/PsHdQ==:17 a=RAioF0-LDSMA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=U7nrCbtTmkRpXpFmAIza:22 a=VnNF1IyMAAAA:8 a=YI3AHy_hWZklpQ5N57sA:9 X-Proofpoint-ORIG-GUID: mUrjqAYuCsnLb4Ra9gPAAPWfrD0I4Tbq X-Proofpoint-GUID: mUrjqAYuCsnLb4Ra9gPAAPWfrD0I4Tbq X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfXwthOsP1p02yh cHrXXeR/X3AaIAr4PKfx3ESUYFSaCBiW/WpRJ0SRkQtjeCWPT1+3JUqRGh2h5oaLHGfVOqTkAFg sM7n0LX4de7bwq9eUto3axCfco1oeGDQ17Tq3SQgDPSLNA51A5PwQYDZl5zQMhdKRsmwrFlE3yM NA8zFVgK959ZzMJjanlqtWV+S2L1tkEc83io+gMEb56b2wf0oJSvWGMUfhP7H2wcMkbWl2JsszQ G69VBgHsR5+Jh/GSMSvKmTdSSoaP+k/pcBfJ2zi+M19GRpkxW9vNUlVisYYQPZdZ/iE/id4YUY1 2krXc0ZvpTW8dDfTdRBIyRJ79SYKsqmwN3b3CZ2uZTdiJh1O1bEL2M46diHE6KCi3RxTx2peiib RbetPKHVF1Ebq3TjY3n4DKQd5W2/0G05nbkf92e5MsbOk49/h6uAZ/J2uH7VcBI/QpXljBgCLNz O2cyV10kw5xI23KfCGg== X-Proofpoint-Spam-Info: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX9TqeoqycPO3a 0JzebIVapBoseLiUb25FHEHV+nWCayfIY3QPxyGwM/ef7UjD8zUGHUQz5oFeyEzoj/QccGYmMiB GHfQcBiUf8ftLqNeUxOmzSCMtKwSm44= X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-07-24_03,2026-07-24_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 suspectscore=0 lowpriorityscore=0 spamscore=0 malwarescore=0 clxscore=1015 adultscore=0 priorityscore=1501 phishscore=0 bulkscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2607240145 Content-Type: text/plain; charset="utf-8" The transition from STOP to RUNNING and vice versa indicates that the vfio device is operating normally and migration is not currently taking place. In this case, just set the new state. Signed-off-by: Anthony Krowiak --- drivers/s390/crypto/vfio_ap_migration.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/= vfio_ap_migration.c index 21cd9810a18c..8876338d2898 100644 --- a/drivers/s390/crypto/vfio_ap_migration.c +++ b/drivers/s390/crypto/vfio_ap_migration.c @@ -1317,12 +1317,17 @@ vfio_ap_transition_to_state(struct ap_matrix_mdev *= matrix_mdev, return NULL; } =20 + /* + * These states indicate migration has either not been initiated or + * has completed and the vfio-ap device is operating normally.Since the + * vfio-ap device does not virtualize a DMA device, there is no internal + * device state to incorporate into the vfio-ap device on the target. + */ if ((cur_state =3D=3D VFIO_DEVICE_STATE_STOP && new_state =3D=3D VFIO_DEVICE_STATE_RUNNING) || (cur_state =3D=3D VFIO_DEVICE_STATE_RUNNING && new_state =3D=3D VFIO_DEVICE_STATE_STOP)) { - /* TODO */ - return ERR_PTR(-EOPNOTSUPP); + return NULL; } =20 /* vfio_mig_get_next_state() does not use arcs other than the above */ --=20 2.53.0 From nobody Fri Jul 24 20:51:09 2026 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4E9BA331ED1; Fri, 24 Jul 2026 16:33:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784910802; cv=none; b=Tj19Fa/EhtWnyCEJ8i5m7snQa2xmBwSNhumpoJlN3zSMGXB1ygCWj0B2WKAZiHNx1f/lWjFXFnULm1r57OUN+U+bgL+DILovOmx0DHgGb+RgPsXMIK0pMtOuKaEAqfpcaUyDvr0ToX5wWzetyPIURplr4BcuqOAjItw49Z+/Zdc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784910802; c=relaxed/simple; bh=mOk33HJJS7pCPcxC4VL+fAK4OX3+1RGMBfBp9f5uA9A=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=YdMjcTLizyMAQEnT08ncsEB5QAPN9CpghYSNfcPzAgA/Qwr1Yt+H0uBH2bkCwOYrHtKhNeuPpqsb6Qlgmc3lI8J72N1+lbiMezShRzOcVJtJYkTCthE45wMh3miis4jAiZAFs/V/1kRczeMWhdU99afbL3xtKHYtOO6B6wW/8Rc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=hLwC/PZO; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="hLwC/PZO" Received: from pps.filterd (m0356516.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 66ODg8WR1899385; Fri, 24 Jul 2026 16:14:18 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=qG3IrEHveo+ZMR5w6 mHcbCXKA3r+HDZPyj36pkkSYWg=; b=hLwC/PZOg5UzIGSLW7sHhE9Z4bANzsZLX R5kUdzfAgIi7nV/Qyv+gSVC/Y3Fzw/nfKdZMzbAm61Mm7tYwQfPzuNPYymvN04as 26r1J8M0u+QvaYnjWkPWIFbCJqyYr6/qRHUHkspgtu+nMenNV/fIi/6A/ItsH4xs mbfJR0a5uro/7ge9LXx7dvaRmIsw1g1mJAdXI8UOD9rJcntdHl0VJZUrH1fqKdNH fYaJld7cgQNGddDa9D6utzhQESaFHz3AquvhAU21RfIL2jPAtbFacaGuvo/mmRvd hrJnma7C8udIwjjSXzX/DnAMI8X5hNQm93k3YB2scdtLDf5wm8C3Q== Received: from ppma23.wdc07v.mail.ibm.com (5d.69.3da9.ip4.static.sl-reverse.com [169.61.105.93]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4fg77kmy24-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:18 +0000 (GMT) Received: from pps.filterd (ppma23.wdc07v.mail.ibm.com [127.0.0.1]) by ppma23.wdc07v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 66OG4eQe025354; Fri, 24 Jul 2026 16:14:17 GMT Received: from smtprelay05.wdc07v.mail.ibm.com ([172.16.1.72]) by ppma23.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4fgnahhkpu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:17 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (smtpav06.wdc07v.mail.ibm.com [10.39.53.233]) by smtprelay05.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 66OGEGao24052362 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 24 Jul 2026 16:14:16 GMT Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 97A7058055; Fri, 24 Jul 2026 16:14:16 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 0B6DF58054; Fri, 24 Jul 2026 16:14:15 +0000 (GMT) Received: from li-4c4c4544-004d-4810-8043-b7c04f423534.ibm.com.com (unknown [9.61.50.28]) by smtpav06.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 24 Jul 2026 16:14:14 +0000 (GMT) From: Anthony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex@shazbot.org, kwankhede@nvidia.com, fiuczy@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, agordeev@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com Subject: [PATCH v5 13/15] s390/vfio-ap: Callback to get the size of data to be migrated during guest migration Date: Fri, 24 Jul 2026 12:13:49 -0400 Message-ID: <20260724161351.1802644-14-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260724161351.1802644-1-akrowiak@linux.ibm.com> References: <20260724161351.1802644-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Authority-Analysis: v=2.4 cv=HJXz0Itv c=1 sm=1 tr=0 ts=6a638f5a cx=c_pps a=3Bg1Hr4SwmMryq2xdFQyZA==:117 a=3Bg1Hr4SwmMryq2xdFQyZA==:17 a=vZ-Ft76KQvoWVpZ_:21 a=RAioF0-LDSMA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=Y2IxJ9c9Rs8Kov3niI8_:22 a=VnNF1IyMAAAA:8 a=lW0G1kjGILAuz2jbLQAA:9 X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX+n17kc+Vus30 YwF1AJbQ8hFVgux5F/+52DoIVLOC6Ob4yiPK2BgWxq0sIfIRxLwEoTLGFqkrJBAzVbsomqX3nxV /OlhjSOVg8SaDmAjZKtFvi+KbVPgIYxA+nnu+34ECOBS7EEX0jygpevjhXSfnq1i44H4Y9eTgAr E3nUPzS1Qlk+NbMxg0MHSmrRbI1sM/GBX76Y+Zw08Tyg97Rhz4qk7RZVeJMzO16IkTbN7T71LOp EbYMSHLXJ7ZPsXHjIcSr8YLjawWbFyly0e7IXnPLkaDCEd1gxHrmCJxl+Lkc/AJDNhxEf7rGtu8 mNRO1Nb54eeME51CmXWtmxWAn8FL8kWKZxPoKWdppo8ndGAXhkCCxCO+/ovcYg7+CnMmqSHTY/j Tp6aovja0bM/yF8MBfGAOouhLQ8R+T+T+4DpU++ZyolUdwFrtroOox+SRU+gtHXEfAhqseZPCpi 7EvFg6Pi9QGDChPOrlQ== X-Proofpoint-ORIG-GUID: LOSlXpvKHcMi9pADCR97esXXcaw6spQW X-Proofpoint-Spam-Info: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX33lZpkAg8eV9 Tz0cewa4DbnuXmv5CnmBgO26/Hxh8Qvtfp0/JxoAv4ap1W8kt+TxaoN0hkWe5lMin/dJc+w83ke /uaV25wM3aXIGT+OmAwyY4JDZJjj00Y= X-Proofpoint-GUID: LOSlXpvKHcMi9pADCR97esXXcaw6spQW X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-07-24_03,2026-07-24_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 priorityscore=1501 lowpriorityscore=0 suspectscore=0 malwarescore=0 impostorscore=0 clxscore=1015 phishscore=0 spamscore=0 adultscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2607240145 Content-Type: text/plain; charset="utf-8" Callback to get the estimated data length of the data that will be required to complete reading of the vfio device's internal state. It's mandatory for VFIO_DEVICE_FEATURE_MIGRATION migration support. The function pointer for this callback is specified via the 'migration_get_data_size' field of the 'vfio_migration_ops' structure which is stored with the VFIO device when the 'vfio_device' structure representing the mediated device is initialized. Signed-off-by: Anthony Krowiak --- drivers/s390/crypto/vfio_ap_migration.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/drivers/s390/crypto/vfio_ap_migration.c b/drivers/s390/crypto/= vfio_ap_migration.c index 8876338d2898..20b519f0898f 100644 --- a/drivers/s390/crypto/vfio_ap_migration.c +++ b/drivers/s390/crypto/vfio_ap_migration.c @@ -1429,7 +1429,18 @@ static int vfio_ap_get_state(struct vfio_device *vde= v, static int vfio_ap_get_data_size(struct vfio_device *vdev, unsigned long *stop_copy_length) { - return -EOPNOTSUPP; + struct ap_matrix_mdev *matrix_mdev; + size_t qinfo_sz; + int num_queues; + + mutex_lock(&matrix_dev->mdevs_lock); + matrix_mdev =3D container_of(vdev, struct ap_matrix_mdev, vdev); + num_queues =3D vfio_ap_mdev_get_num_queues(&matrix_mdev->shadow_apcb); + qinfo_sz =3D num_queues * sizeof(struct vfio_ap_queue_info); + *stop_copy_length =3D qinfo_sz + sizeof(struct vfio_ap_config); + mutex_unlock(&matrix_dev->mdevs_lock); + + return 0; } =20 static const struct vfio_migration_ops vfio_ap_migration_ops =3D { --=20 2.53.0 From nobody Fri Jul 24 20:51:09 2026 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F48A4483BC; Fri, 24 Jul 2026 16:14:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784909677; cv=none; b=l3IDTvtSv5QUHo4cUSzK6XIN1f9GR1+HbfPaZM1nH2dDzyshOHdGOJx+efod6KGNCU9jBVAQ0t+TfbiVGON24q5RMTixOFZQ69pRa+6h6F3MNluNdhDoWWHRIfYZWRBXmdMpWILzTG2ZwHyOkx360hvnLZUcDB3GYmZ7fPpG0Y0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784909677; c=relaxed/simple; bh=Hal2jLpW9jwI4RpxS7uNTsEG8gypNxlL9fhMbAkxgr8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=pqG42ZNzq5CVW0iGirFTU1v7ejJBKCw98FseGnFkAUt8yKfMdfLRkK7xV0VPg/169I0nTndR/fOh2KjS+5hbP1Ar4f18arlPXr5otv4FaUywV1XNsPbSbOm5ukT/NUzbg+fF+4EHol6Gg49GZGMmYYwIPDDNIE9jeM0d1poNmy4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=deY6cp7B; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="deY6cp7B" Received: from pps.filterd (m0360083.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 66ODfdll1973574; Fri, 24 Jul 2026 16:14:20 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=MuFmywDxoSDjBBAjm Z7MnYtpHXmbfOsODaDcZdMsvkg=; b=deY6cp7BJJ88IYs6Jusnuo+kN1wDOzyoL 4ZUHIjbQHv4VxYR0biZ5rQga6Hh1n3p7n3RPx+xzoRV72BY08yCn5hmyZQYIUGpK tqEvy5W/E3Q29No/LjoKnbzMwApcWSQZrwW8sI5hlb5UuSDGYECwPWnSa9mfKzti Rms5MS4J90vmJAyn1ZEmHBjAvURUKM/3vcKcFAwujcW3zMKOJ3H4bfLlY8XljSOw yBj6KfZf5NnmrXIuVCDIMDAiLdK7okHmSZKFPUjHV8N58FcYEUHLzAVOLkTpk8we Kl9E8BKTI7yUSZwuDd1RbumEApsoIHgZGXA4tKl4DqBgYVsg0F5Cg== Received: from ppma22.wdc07v.mail.ibm.com (5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4fm8h50u73-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:20 +0000 (GMT) Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1]) by ppma22.wdc07v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 66OG4bvc026008; Fri, 24 Jul 2026 16:14:19 GMT Received: from smtprelay07.wdc07v.mail.ibm.com ([172.16.1.74]) by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4fgm6whr2w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:19 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (smtpav06.wdc07v.mail.ibm.com [10.39.53.233]) by smtprelay07.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 66OGEIBa11534912 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 24 Jul 2026 16:14:18 GMT Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5E8C75804E; Fri, 24 Jul 2026 16:14:18 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id C10805803F; Fri, 24 Jul 2026 16:14:16 +0000 (GMT) Received: from li-4c4c4544-004d-4810-8043-b7c04f423534.ibm.com.com (unknown [9.61.50.28]) by smtpav06.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 24 Jul 2026 16:14:16 +0000 (GMT) From: Anthony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex@shazbot.org, kwankhede@nvidia.com, fiuczy@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, agordeev@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com Subject: [PATCH v5 14/15] s390/vfio-ap: Add 'migratable' feature to sysfs 'features' attribute Date: Fri, 24 Jul 2026 12:13:50 -0400 Message-ID: <20260724161351.1802644-15-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260724161351.1802644-1-akrowiak@linux.ibm.com> References: <20260724161351.1802644-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-Spam-Info: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX+Qtgjd5UjSTU PZMIPMAo5yrSkPgBLVFAtzmNQMi0SI1RpaP6hVWhZlZSUmwk+S5DoLB+TnwXCDPbT8Dxe9uIwHV 3Kvxj99HamuKhGddWHlUGszHS/ZplCs= X-Proofpoint-ORIG-GUID: GWsqgRMRj23gH4sJSL2tuHVJAKpCT6uy X-Proofpoint-GUID: GWsqgRMRj23gH4sJSL2tuHVJAKpCT6uy X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX3X4aeyZcCe9r 06fqQ4s8Wpy8JXGy1YnSQboKj0ZuB/sz7tkR9IYg/9ePCWrABphGAby1yRWCiuObNX+kvXa4eTX lriCCo2Je8PjXWwIzNTVksMWxjHzokqcgsjx62EEjhsNIsoSbFZ+gf6ZfAMdcgeoqpW7iULMOtt ibxGdh5SGDR6HAesIKzimykUMEFSkDCtz3ZcHX1WvTBaeOTCWHeesNxOzR9iJKQFlme2IJacpVk 4ewJnz2Uc95oquRRlS4upun3qkmQKmBHmQQMnbeGKwPQ+0JyILhErqAPt6Co1ckRhOts12tbXyN wNFD1ewvwcroFgnrAlWZ6KwZS+7F+96yvpXNVRSQ1dhll96Nj5SwHci7bDpj4e+40Tf0+9Zv8I2 adM8b0kw1LbtXAaJMaQE6QxaRvqdHrIxsJmN/LCELpDfi20xe2nQF63f2IKpvIHVje6RLfLcLQi f0qY1GHKd4guUmPd05g== X-Authority-Analysis: v=2.4 cv=du3rzVg4 c=1 sm=1 tr=0 ts=6a638f5c cx=c_pps a=5BHTudwdYE3Te8bg5FgnPg==:117 a=5BHTudwdYE3Te8bg5FgnPg==:17 a=RAioF0-LDSMA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=iQ6ETzBq9ecOQQE5vZCe:22 a=VnNF1IyMAAAA:8 a=CB_8iKbeFirw2v3l3xgA:9 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-07-24_03,2026-07-24_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 clxscore=1015 malwarescore=0 spamscore=0 impostorscore=0 phishscore=0 bulkscore=0 priorityscore=1501 lowpriorityscore=0 suspectscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2607240145 Content-Type: text/plain; charset="utf-8" Add the 'migratable' feature to the features provided by the sysfs /sys/devices/vfio_ap/matrix/feature attribute to indicate that migration of vfio devices is supported. Signed-off-by: Anthony Krowiak --- drivers/s390/crypto/vfio_ap_drv.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/drivers/s390/crypto/vfio_ap_drv.c b/drivers/s390/crypto/vfio_a= p_drv.c index 8e69ed286bb9..4e973736522b 100644 --- a/drivers/s390/crypto/vfio_ap_drv.c +++ b/drivers/s390/crypto/vfio_ap_drv.c @@ -26,9 +26,11 @@ MODULE_LICENSE("GPL v2"); struct ap_matrix_dev *matrix_dev; debug_info_t *vfio_ap_dbf_info; =20 +#define FEATURES "guest_matrix hotplug ap_config migratable" + static ssize_t features_show(struct device *dev, struct device_attribute *= attr, char *buf) { - return sysfs_emit(buf, "guest_matrix hotplug ap_config\n"); + return sysfs_emit(buf, "%s\n", FEATURES); } static DEVICE_ATTR_RO(features); =20 --=20 2.53.0 From nobody Fri Jul 24 20:51:09 2026 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1FED7448B93; Fri, 24 Jul 2026 16:14:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.156.1 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784909709; cv=none; b=MNSC/gFBszu7zE1jVZC0uFtMllMDrCJfYE4wMOSHIXItETrLnwSWDoF11wSx1U6uf/3ZhD9EqmWcPUaH8MbIqdrWuiaz557wODb6lRM5WtkA1TsBEJA8Ok/5TgZyDo9/gVtxFrYEpfh0KDkKFjd1lsPGmPsdsbkHN6OCKly6tCM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1784909709; c=relaxed/simple; bh=lHAAwkMu4ySYnvbvcYOQDayqZmqSLiSH+6AtILrug28=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=Mml/wivsBOeIGosnvRwwaMrZLtRm/ib0ASYhi38+bM3wsD/JaG0a1rNlW1LPTW8vEZB+rka2/FKE7XNoV3+KHwzfCRofgwB6VHiUtH4IFXwZr+VoMZMByeqyhTpeOW3FGoaY4agiR76bpfSJVTwTip6miJYKrmrkwPCfGL2eYHM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=lxwTVfoC; arc=none smtp.client-ip=148.163.156.1 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="lxwTVfoC" Received: from pps.filterd (m0353729.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.11/8.18.1.11) with ESMTP id 66ODfo3m100655; Fri, 24 Jul 2026 16:14:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=2bFwZwBmncIqk/Zef +MNjgmkis/BeEwp+wkQfIRG6Hg=; b=lxwTVfoCid+ARacAZs6z0z+fI/198WngW T1nziwJlrZUexsMYdQygJ7+7Sg8NdZwfib92ljtjKKfSbSx2+45rClrkA55H3eGt oaaVppzHKDsdsa0rAf5MHIBUaOZbxx4mR+Ab1K9a6dfwg52UbsUCOEK8Gh8iShTY gHksRVYlzYVfKiDwxvdvVE9rfGVd0UJfVDEHJ8A4kAE7+T0q//mUbLWpm+CRJN+w WsaOumk0FQ9CgDSvxdxfGM0/8LkTsd99keVcqEHPFdt25uwa2PT1+5sx2uc1/3I9 f5Gyjr8uH0WnBwgqfY6xJM7HXaBKavm5SKGy7MJzH5c8D+djNZOAg== Received: from ppma21.wdc07v.mail.ibm.com (5b.69.3da9.ip4.static.sl-reverse.com [169.61.105.91]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 4fm8gs8u7r-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:22 +0000 (GMT) Received: from pps.filterd (ppma21.wdc07v.mail.ibm.com [127.0.0.1]) by ppma21.wdc07v.mail.ibm.com (8.18.1.7/8.18.1.7) with ESMTP id 66OG4dGd028411; Fri, 24 Jul 2026 16:14:21 GMT Received: from smtprelay03.dal12v.mail.ibm.com ([172.16.1.5]) by ppma21.wdc07v.mail.ibm.com (PPS) with ESMTPS id 4fgmtk9n31-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 24 Jul 2026 16:14:21 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (smtpav06.wdc07v.mail.ibm.com [10.39.53.233]) by smtprelay03.dal12v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 66OGEKIn13697594 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 24 Jul 2026 16:14:20 GMT Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 186DF58054; Fri, 24 Jul 2026 16:14:20 +0000 (GMT) Received: from smtpav06.wdc07v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 88F475803F; Fri, 24 Jul 2026 16:14:18 +0000 (GMT) Received: from li-4c4c4544-004d-4810-8043-b7c04f423534.ibm.com.com (unknown [9.61.50.28]) by smtpav06.wdc07v.mail.ibm.com (Postfix) with ESMTP; Fri, 24 Jul 2026 16:14:18 +0000 (GMT) From: Anthony Krowiak To: linux-s390@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: jjherne@linux.ibm.com, borntraeger@de.ibm.com, mjrosato@linux.ibm.com, pasic@linux.ibm.com, alex@shazbot.org, kwankhede@nvidia.com, fiuczy@linux.ibm.com, pbonzini@redhat.com, frankja@linux.ibm.com, imbrenda@linux.ibm.com, agordeev@linux.ibm.com, hca@linux.ibm.com, gor@linux.ibm.com Subject: [PATCH v5 15/15] s390/vfio-ap: Add live guest migration chapter to vfio-ap.rst Date: Fri, 24 Jul 2026 12:13:51 -0400 Message-ID: <20260724161351.1802644-16-akrowiak@linux.ibm.com> X-Mailer: git-send-email 2.53.0 In-Reply-To: <20260724161351.1802644-1-akrowiak@linux.ibm.com> References: <20260724161351.1802644-1-akrowiak@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-Spam-Info: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX11LEAgpBqxhq z1DGuefuQRZfTTkC+nMl+fQ0DJrdUPFMMDz1XcW60nnGLdEsLWvF/Gu1M3wbzvlcxHq80L1WKhm MBz0Qj45ExSQbf7eg6YCUCdbERVrPog= X-Proofpoint-Spam-Details-Enc: AW1haW4tMjYwNzI0MDE0NSBTYWx0ZWRfX11DCw+sQgQm6 hSyD5+hxm2XTpj+SOSrurXOW2vGqp9C3DmxIiycXu0mjBtJaHZIZBMrsO0BpD6moZ/OMGMba4Ng UHN9Vryri6cUg+DHrVcG+hY7Gra3vLyXVR6qtj5Trk6j3XlWjmkTRTC/vat45H7S90j2nOBebFd gZKiFJyaXMeBiLzNVDEcl7Vny6MNQXdWfPtBv8p8mw0YaqlWAIi0oQN97Ijtek2lK4ghVf02cok T/zvvipXBKtT5UD5RvR3VJCs3Z9qVIXYXnxIUXnyW1bjfgobJ3nTHRbwPHeE+4FACJIzN9rh1q0 Pt4mwCGDZjLuVru+40ScmEmtNiAIgkr616S4lI5rI4ZkryTNqPQ6GA7O2r5O9x/zjvanFnadVMN btfUuC2EOqYCzeRdWWj00hnhKxsqd1iwZ/GH3thGyN3cBzg5P5G9IJJBisFGWBsDtY49HMmIlwH oqfdZNms3lpvdVWkATw== X-Authority-Analysis: v=2.4 cv=Q9LiJY2a c=1 sm=1 tr=0 ts=6a638f5f cx=c_pps a=GFwsV6G8L6GxiO2Y/PsHdQ==:117 a=GFwsV6G8L6GxiO2Y/PsHdQ==:17 a=RAioF0-LDSMA:10 a=VkNPw1HP01LnGYTKEx00:22 a=RnoormkPH1_aCDwRdu11:22 a=uAbxVGIbfxUO_5tXvNgY:22 a=VnNF1IyMAAAA:8 a=eqVRqZzbwv3K9eCZ3BEA:9 X-Proofpoint-ORIG-GUID: hg4hRIx-3RVfqStN2PS5uzVCo-pnajkE X-Proofpoint-GUID: hg4hRIx-3RVfqStN2PS5uzVCo-pnajkE X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1143,Hydra:6.1.134,FMLib:17.12.100.49 definitions=2026-07-24_03,2026-07-24_02,2025-10-01_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 malwarescore=0 adultscore=0 phishscore=0 bulkscore=0 spamscore=0 priorityscore=1501 impostorscore=0 suspectscore=0 lowpriorityscore=0 classifier=typeunknown authscore=0 authtc= authcc= route=outbound adjust=0 reason=mlx scancount=1 engine=8.22.0-2606150000 definitions=main-2607240145 Content-Type: text/plain; charset="utf-8" Add a 'Live Guest Migration' chapter to the Documentation/arch/s390/vfio-ap.rst document to describe the details for initiating live guest migration for a guest to which AP adapters, domains and control domains have been passed through. Signed-off-by: Anthony Krowiak --- Documentation/arch/s390/vfio-ap.rst | 616 +++++++++++++++++++++++----- 1 file changed, 517 insertions(+), 99 deletions(-) diff --git a/Documentation/arch/s390/vfio-ap.rst b/Documentation/arch/s390/= vfio-ap.rst index ac0c07f76ddd..02d7cbe5e479 100644 --- a/Documentation/arch/s390/vfio-ap.rst +++ b/Documentation/arch/s390/vfio-ap.rst @@ -1012,7 +1012,7 @@ guest_matrix dyn ap_config =20 the following features are advertised: =20 ----------------+----------------------------------------------------------= -----+ ++--------------+----------------------------------------------------------= -----+ | Flag | Description = | +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D+ | guest_matrix | guest_matrix attribute exists. It reports the matrix of = | @@ -1021,105 +1021,523 @@ the following features are advertised: +--------------+----------------------------------------------------------= -----+ | dyn | Indicates hot plug/unplug of AP adapters, domains and con= trol | | | domains for a guest to which the mdev is attached. = | -+------------+------------------------------------------------------------= -----+ ++--------------+----------------------------------------------------------= -----+ | ap_config | ap_config interface for one-shot modifications to mdev co= nfig | +--------------+----------------------------------------------------------= -----+ +| migratable | Indicates that live guest migration is supported for gues= ts | +| | to which crypto devices are passed through = | ++--------------+----------------------------------------------------------= -----+ =20 -Limitations -=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D -Live guest migration is not supported for guests using AP devices without -intervention by a system administrator. Before a KVM guest can be migrated, -the vfio_ap mediated device must be removed. Unfortunately, it can not be -removed manually (i.e., echo 1 > /sys/devices/vfio_ap/matrix/$UUID/remove)= while -the mdev is in use by a KVM guest. If the guest is being emulated by QEMU, -its mdev can be hot unplugged from the guest in one of two ways: - -1. If the KVM guest was started with libvirt, you can hot unplug the mdev = via - the following commands: - - virsh detach-device - - For example, to hot unplug mdev 62177883-f1bb-47f0-914d-32a22e3a8804= from - the guest named 'my-guest': - - virsh detach-device my-guest ~/config/my-guest-hostdev.xml - - The contents of my-guest-hostdev.xml: - -.. code-block:: xml - - - -
- - - - - virsh qemu-monitor-command --hmp "device-del " - - For example, to hot unplug the vfio_ap mediated device identified on= the - qemu command line with 'id=3Dhostdev0' from the guest named 'my-gues= t': - -.. code-block:: sh - - virsh qemu-monitor-command my-guest --hmp "device_del hostdev0" - -2. A vfio_ap mediated device can be hot unplugged by attaching the qemu mo= nitor - to the guest and using the following qemu monitor command: - - (QEMU) device-del id=3D - - For example, to hot unplug the vfio_ap mediated device that was spec= ified - on the qemu command line with 'id=3Dhostdev0' when the guest was sta= rted: - - (QEMU) device-del id=3Dhostdev0 - -After live migration of the KVM guest completes, an AP configuration can be -restored to the KVM guest by hot plugging a vfio_ap mediated device on the= target -system into the guest in one of two ways: - -1. If the KVM guest was started with libvirt, you can hot plug a matrix me= diated - device into the guest via the following virsh commands: - - virsh attach-device - - For example, to hot plug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 i= nto - the guest named 'my-guest': - - virsh attach-device my-guest ~/config/my-guest-hostdev.xml - - The contents of my-guest-hostdev.xml: - -.. code-block:: xml - - - -
- - - - - virsh qemu-monitor-command --hmp \ - "device_add vfio-ap,sysfsdev=3D,id=3D" - - For example, to hot plug the vfio_ap mediated device - 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest named 'my-guest'= with - device-id hostdev0: - - virsh qemu-monitor-command my-guest --hmp \ - "device_add vfio-ap,\ - sysfsdev=3D/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22= e3a8804,\ - id=3Dhostdev0" - -2. A vfio_ap mediated device can be hot plugged by attaching the qemu moni= tor - to the guest and using the following qemu monitor command: - - (qemu) device_add "vfio-ap,sysfsdev=3D,id=3D" - - For example, to plug the vfio_ap mediated device - 62177883-f1bb-47f0-914d-32a22e3a8804 into the guest with the device-= id - hostdev0: +Live Guest Migration +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D +The VFIO AP mediated device is not used to provide userspace with direct +access to device memory as is the case with other devices that use the VFIO +framework for device pass-through. Instead, it manages AP +configuration metadata identifying the adapters, domains, and control +domains to which a guest will be granted access. These AP resources are +configured by assigning them to a vfio-ap mediated device via its sysfs +assignment interfaces. When the guest is started, the vfio_ap device driver +sets the guest's AP configuration from the metadata stored with the mediat= ed +device. The AP devices are not accessed directly through the vfio_ap drive= r, +so the driver does not migrate the state of the AP devices themselves. +Instead, it migrates the AP configuration metadata. The vfio_ap device +driver on the destination host will then ensure that the AP configuration +metadata migrated from the source host is compatible with the AP +configuration available on the destination host and if +so, will set the destination guest's AP configuration accordingly. =20 - (QEMU) device-add "vfio-ap,\ - sysfsdev=3D/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32= a22e3a8804,\ - id=3Dhostdev0" +To be considered compatible, the AP configuration for the destination host= must +meet these requirements: + +* Each AP adapter, domain and control domain number assigned to the source + guest's AP configuration must be in the destination host's AP + configuration. This can be verified as follows: + + * On the source host, display the AP configuration of the source guest + with the following two commands: + + .. code-block:: bash + + cat /sys/devices/vfio_ap/matrix/$UUID/guest_matrix + cat /sys/devices/vfio_ap/matrix/$UUID/control_domains + + .. note:: + Replace ``$UUID`` with the actual UUID of your mediated device. + + The ``guest_matrix`` lists the APQNs of the queue devices assigned + to the mediated device; for example: + + .. code-block:: bash + + 00.0033 + 00.003a + + The ``control_domains`` lists the domain numbers of the control + domains; for example: + + .. code-block:: bash + + 0033 + 003a + + * Display the AP configuration of the destination host with the following + two commands: + + .. code-block:: bash + + lszcrypt -V + cat /sys/bus/ap/ap_control_domain_mask + + .. note:: + The ``lszcrypt -V`` command lists the AP cards (adapters) and queue + devices on the host system; for example: + + .. code-block:: bash + + =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D =3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + CARD.DOMAIN TYPE MODE DRIVER + =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D =3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + 00 CEX5C CCA-Coproc cex4card + 00.0033 CEX5C CCA-Coproc vfio_ap + 00.003a CEX5C CCA-Coproc vfio_ap + =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D =3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D + + Each bit in the ap_control_domain_mask corresponds to + domain numbers 0-255. If a bit is set to one, then + the control domain is available on the source host. For + example: + + .. code-block:: bash + + 0x00000000000010200000000000000000000000000000000000000000000000= 00 + + This mask indicates that AP control domains 51 (0x33) and 58 (0x3a)= are + in the AP configuration of the host system. + + * Verify the configurations: + + * Each APQN in the guest's sysfs ``guest_matrix`` file must match an + APQN returned from the ``lszcrypt -V`` command on the destination ho= st. It + is important to note the driver to which the queue is bound which + is displayed in the ``driver`` column. Queues not bound to the vfio_= ap + driver are not available to the destination guest. Even if the APQNs= are + available to the driver, that does not guarantee they can be used by= the + guest being migrated; for example, migration will fail if any of the= se APQNs + are assigned to a mediated device other than the one being migrated. + + * Each domain number from the source guest's sysfs + ``control_domains`` attribute must correspond to a bit set to one in + the destination host's ``ap_control_domain_mask``. + + +* Each APQN assigned to the source guest and destination guest must + reference a queue device with compatible hardware capabilities: + + +--------------+--------------------------------------------------------= -------+ + | Hardware | Description = | + | Capabilities | = | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D+ + | [1] | * AP special command facility (APSC) = | + | Facilities | * AP 4096-bit ME PKU commands facility (AP4KM) = | + | | * AP 4096-bit CRT PKU commands (AP4KC) = | + +--------------+--------------------------------------------------------= -------+ + | [1] Mode | * CCA-mode = | + | | * Accelerator-mode = | + | | * XCP-mode (EP11) = | + +--------------+--------------------------------------------------------= -------+ + | [1] AP | APXA installed = | + | extended | = | + | addressing | = | + +--------------+--------------------------------------------------------= -------+ + | [1] Command | Command filtering available = | + | filtering | = | + +--------------+--------------------------------------------------------= -------+ + | [2] | * Full native card function = | + | Functional | * Only stateless functions = | + | capabilities | = | + +--------------+--------------------------------------------------------= -------+ + | [3] AP type | * 10 (0x0a): CEX4 = | + | | * 11 (0x0b): CEX5 = | + | | * 12 (0x0c) CEX6 = | + | | * 13 (0x0d) CEX7 = | + | | * 14 (0x0e) CEX8 = | + +--------------+--------------------------------------------------------= -------+ + + +-----------------------------------------------------------------------= ------+ + | Table Keys = | + +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D+ + | = | + | [1] The hardware capability must be the same for both the sourc= e | + | and destination queue device. = | + | = | + | [2] The source and destination queues must either both have = | + | full native card function or both have stateless functions.= | + | If the functional capabilities don't match, then the source= | + | queue can have stateless capabilities since full native car= d | + | function can handle stateless functions; otherwise, migrati= on | + | will fail. = | + | = | + | [3] The AP type on the source and destination guests can = | + | differ if the queue passed through to the target guest is a= | + | newer model (backwards compatible) = | + +-----------------------------------------------------------------------= ------+ + + * To verify the hardware capabilities are compatible: + + * On both the source and destination hosts, display the hardware capab= ilities + for each AP adapter in the AP configuration for that host: + + .. code-block:: bash + + cat /sys/bus/ap/devices/card$APID/ap_functions + + .. note:: + The ``$APID`` is the two-character adapter number in hexadecimal = format; + for example, ``card02`` or ``card1f``. The ``/sys/bus/ap/devices`= ` directory + also lists the APQNs of the queue devices installed in the respec= tive + host system with the first two characters being the APID; for exa= mple: + ``02.0004`` or ``00.0033`` . + + * Verify the hardware capabilities for each AP adapter device on the s= ource + and destination host are compatible. The ``ap_functions`` attribute = is a + bitmask. The bits in the mask read from left to right starting with = bit 0. + Each bit that is set to one indicates the corresponding hardware cap= ability + is installed: + + * Bits 0-2 are the facilities bits. Each bit value must match for th= e AP + devices on the source and destination systems. The values indicate= the + following: + + * bit-0: APSC is installed + * bit-1: AP4KM is installed + * bit-2: AP4KC is installed + + * Bits 3-5 specify the adapter Modes. Only one of these bits will be= set to + one and must be the same for both the source and destination adapt= er + devices: + + * bit-3: CCA-mode + * bit-4: Accelerator-mode + * bit-5: XCP-mode (EP11) + + * Bit 6 specifies whether APXA is installed and must be the same for= both + the source and destination adapter devices. + + * Bit 7 specifies whether command filtering is installed + and must be the same for both the source and destination adapter d= evices. + + * Bits 8-15 are the classification bits. There are only two relevant + bits and only one of them will be set to one. For the source and d= estination + adapter devices to be considered compatible: + + * Bit 8 may be set to 1 on both hosts (full native card function) + * Bit 9 may be set to 1 on both hosts (only stateless functions av= ailable) + * Bit 9 may be set to 1 on the source host and bit 8 may be set to= 1 on + the destination host + + * Bits 16-17 are the binding state bits which must be 00 + + * To verify the AP types are compatible: + + * On both the source and destination hosts, display the hardware type + for each AP adapter in the AP configuration for that host to ensure = the + source and destination adapter types are the same, or the source typ= e is + less than the destination type:: + + cat /sys/bus/ap/devices/card$APID/hwtype + + .. note:: + The ``$APID`` is the two-character adapter number in hexadecimal = format; + for example, ``card02`` or ``card1f``. + +Setting up for live guest migration on the destination host +----------------------------------------------------------- +To set up a host as the destination for live guest migration, do the +following: + +1. Create a mediated device +~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Create a mediated device with the same name (UUID) as the mediated device +used to supply the AP configuration to the source guest. For example, if +the mediated device on the source guest is named +62177883-f1bb-47f0-914d-32a22e3a8804: + +.. code-block:: bash + + echo 62177883-f1bb-47f0-914d-32a22e3a8804 > \ + /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough/= create + +2. Reserve adapters and domains for the vfio_ap device driver's use +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Reserve the same adapters, and domains passed through to the source guest +for the vfio_ap device driver on the destination host by clearing the +adapter numbers from the ``/sys/bus/ap/apmask`` and/or the domain numbers +from the ``/sys/bus/ap/aqmask`` files. + +For example, suppose adapters 0, 20 (0x14) and 22 (0x16) as well as +domains 51 (0x33) and 58 (0x3a) are passed through to the source guest. To +ensure they can be passed through to the destination guest, they must be +made available to the vfio_ap device driver on the destination host. To +make them available, do one or both of the following on the destination ho= st: + +.. code-block:: bash + + echo -0,-0x14,-22 > /sys/bus/ap/apmask + echo -0,-0x33,-0x3a > /sys/bus/ap/aqmask + +.. note:: + + * It is not necessary to reserve both the adapters and domains. + Reserving an adapter implicitly allows the vfio_ap driver to assign + each of the domains available on the system along with the reserved + adapters to a guest's AP configuration and vice versa. + + * The Cartesian product of the adapters and domains reserved for the + vfio_ap device driver identifies the APQNs of the AP queue devices t= o be + bound to the vfio_ap device driver. If an APQN does not reference a + queue device installed in the system at the time of live guest migra= tion, + the migration will fail. + + * It is not necessary to configure the destination mediated device with + adapters, domains and control domains because the source guest's AP + configuration will overlay the destination guest's during migration. + + * To verify that the queue devices are reserved for the vfio_ap + device driver, issue the ``lszcrypt -V`` command and you should + see ``vfio_ap`` in the ``DRIVER`` column of the output. + +Live guest migration failures due to AP configuration errors +------------------------------------------------------------ +This section describes the errors that may occur during live guest migrati= on +when the AP configuration of the source guest and destination host are not +compatible and where to look for problem determination. + +The destination host is missing the mediated device +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This error occurs when a mediated device with the same name (UUID) as the +mediated device being migrated has not been created on the destination hos= t. + +**Source host** + ++--------------+----------------------------------------------------------= ------+ +| Log | Message = | ++=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D+ +| Console log: | error: device not found: mediated device '$UUID' not = | +| | found = | ++--------------+----------------------------------------------------------= ------+ +| QEMU log: | initiating migration = | +| | qemu-system-s390x: Sibling indicated error 1 = | ++--------------+----------------------------------------------------------= ------+ + +.. note:: + $UUID is the UUID of the mediated device being migrated from the source= host. + + +One or more queues are not bound to the vfio_ap driver on the destination = host +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~= ~~~~ +The queue devices passed through to the source guest are not installed in = the +destination host or are not bound to its vfio_ap device driver. + +**Source host** + ++--------------+----------------------------------------------------------= ------+ +| Log | Message = | ++=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D+ +| Console log: | error: operation failed: migration failed. Message from t= he | +| | source host: operation failed: job 'migration out' failed= : | +| | Sibling indicated error 1. Message from the destination h= ost: | +| | operation failed: job 'migration in' failed: load of migr= ation | +| | failed: No such device: Failed to load vmstate version_id= : 1, | +| | ret: -19 = | ++--------------+----------------------------------------------------------= ------+ + +**Destination host** + ++--------------+----------------------------------------------------------= ------+ +| Kernel log: | vfio_ap_mdev $UUID: Queue $APID.$APQI not available to = | +| | vfio_ap driver on target host = | ++--------------+----------------------------------------------------------= ------+ +| QEMU log: | shutting down, reason=3Dfailed = | +| | qemu-system-s390x: terminating on signal $SIGNAL_NUM from= | +| | pid $PID $DAEMON = | ++--------------+----------------------------------------------------------= ------+ + + +One or more control domains are not available on the destination host +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The control domains passed through to the source guest are not installed o= n the +destination host. + +**Source host** + ++--------------+----------------------------------------------------------= ------+ +| Log | Message = | ++=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D+ +| Console log: | error: operation failed: migration failed. Message from t= he | +| | source host: operation failed: job 'migration out' failed= : | +| | Sibling indicated error 1. Message from the destination h= ost: | +| | operation failed: job 'migration in' failed: load of migr= ation | +| | failed: No such device: Failed to load vmstate version_id= : 1, | +| | ret: -19 = | ++--------------+----------------------------------------------------------= ------+ + +**Destination host** + ++--------------+----------------------------------------------------------= ------+ +| Kernel log: | vfio_ap_mdev $UUID: Control domain $DOMAIN_NUM not availa= ble | +| | on the destination host = | ++--------------+----------------------------------------------------------= ------+ +| QEMU log: | shutting down, reason=3Dfailed = | +| | qemu-system-s390x: terminating on signal $SIGNAL_NUM from= | +| | pid $PID $DAEMON = | ++--------------+----------------------------------------------------------= ------+ + + +.. note:: + * $UUID is the UUID of the mediated device being migrated from the sour= ce host. + * $APID.$APQI is adapter number and queue index (domain number) compris= ing the + $APQN (for example, 00.0033) of the queue device that is not availabl= e. + * $SIGNAL_NUM is the signal number that caused the QEMU process to term= inate. + * $PID is the process ID of the QEMU process on the destination host. + * $DAEMON is the name of the QEMU daemon process. + * This message will be logged for each queue for which this error is de= tected. + +Failure to get hardware information for a queue device on destination host +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The vfio_ap device driver on the destination host will execute an AP instr= uction +to get the hardware information for a queue device to perform queue compat= ibility +checks. The instruction can fail for a number of reasons in which case the +migration will be terminated. + +**Source host** + ++--------------+----------------------------------------------------------= ------+ +| Log | Message = | ++=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D+ +| Console log: | error: operation failed: migration failed. Message from t= he | +| | source host: operation failed: job 'migration out' failed= : | +| | Sibling indicated error 1. Message from the destination h= ost: | +| | operation failed: job 'migration in' failed: load of migr= ation | +| | failed: failed to get hardware information for queue devi= ce | +| | $APQN = | ++--------------+----------------------------------------------------------= ------+ + +**Destination host** + ++--------------+----------------------------------------------------------= ------+ +| Log | Message = | ++=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D+ +| Console log: | vfio_ap_mdev $UUID: migration failed: Failed to get hwinf= o for | +| | queue $APQN on target host: rc=3D$RESPONSE-CODE = | ++--------------+----------------------------------------------------------= ------+ +| QEMU log: | shutting down, reason=3Dfailed = | +| | qemu-system-s390x: terminating on signal $SIGNAL-ID from = pid | +| | $PID $DAEMON = | ++--------------+----------------------------------------------------------= ------+ + +.. note:: + * $UUID is the UUID of the mediated device being migrated from the sour= ce host. + * $APQN (for example, 00.0033) of the queue device that is not compatib= le. + * $RESPONSE_CODE is the response code from the AP instruction. + * $SIGNAL_NUM is the signal number of the signal that caused the QEMU p= rocess + to terminate. + * $PID is the process ID of the QEMU process on the destination host. + * $DAEMON is the name of the QEMU daemon process. + * This message will be logged for each queue for which this error is de= tected. + +The AP configurations of the source guest and destination host not compati= ble +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~= ~~~ +The functional capabilities or facilities available on one or more of the = queues +passed through to the source guest are not compatible with the queue devic= e with +the same APQN on the destination system (see the Hardware Capabilities tab= le above) + +**Source host** + ++--------------+----------------------------------------------------------= ------+ +| Log | Message = | ++=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D+ +| Console log: | error: operation failed: migration failed. Message from t= he | +| | source host: operation failed: job 'migration out' failed= : | +| | Sibling indicated error 1. Message from the destination h= ost: | +| | operation failed: job 'migration in' failed: load of migr= ation | +| | failed: No such device: Failed to load vmstate version_id= : 1, | +| | ret: -19 = | ++--------------+----------------------------------------------------------= ------+ +| Kernel log: | N/A = | ++--------------+----------------------------------------------------------= ------+ +| QEMU log: | *initiating migration* = | +| | = | +| | *qemu-system-s390x: Sibling indicated error 1* = | ++--------------+----------------------------------------------------------= ------+ + +**Destination host** + ++--------------+----------------------------------------------------------= ------+ +| Log | Message = | ++=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D+=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D+ +| Console log: | N/A = | ++--------------+----------------------------------------------------------= ------+ +| Kernel log: | *vfio_ap_mdev $UUID: Migration failed: Source and target = | +| | queue ($APQN) not compatible* = | +| | = | +| | The message above will be followed by one or more message= s | +| | enumerating the incompatible features; for example: = | +| | = | +| | ``vfio_ap_mdev $UUID: APSC facility installed in source q= ueue | +| | $APQN`` = | +| | ``vfio_ap_mdev $UUID: APSC facility not installed in targ= et | +| | queue $APQN`` = | +| | = | +| | ``AP type of source ($APTYPE1) not compatible with target= | +| | ($APTYPE2)`` ... = | ++--------------+----------------------------------------------------------= ------+ +| QEMU log: | *initiating migration* = | +| | = | +| | *qemu-system-s390x: error while loading state section id = ...* | +| | = | +| | *shutting down, reason=3Dfailed* = | +| | = | +| | *terminating on signal $SIGNAL-ID from pid $PID $DAEMON* = | ++--------------+----------------------------------------------------------= ------+ + +.. note:: + * $UUID is the UUID of the mediated device being migrated from the sour= ce host. + * $APQN (for example, 00.0033) of the queue device that is incompatible. + * $APTYPE1 is the type of the source queue device. + * $APTYPE2 is the type of the target queue device. + * $SIGNAL-ID is the signal number of the signal that caused the QEMU pr= ocess + to terminate. + * $PROCESS-ID is the process ID of the QEMU process on the source host. + * $PID is the process ID of the QEMU process on the destination host. + * $DAEMON is the name of the QEMU daemon process. + * Each follow-on message logging the incompatibility will be logged for= each + error detected for the specified queue device. + +AP Configuration Management +--------------------------- +The AP configurations of the source guest and destination host must be kept +in synchronization or live guest migration will likely fail due to potenti= al +incompatibilities introduced by such changes. In particular, changing adap= ter +or domain reservations via the sysfs ``apmask`` or ``aqmask`` attributes on +either the source or destination host without coordinating such changes be= tween +the two hosts -especially while a migration is taking place - can result in +migration failures. + +Configuration stability is an orchestration-layer or system administrator +responsibility, which is consistent with other VFIO device types. The vfio= _ap +device driver will validate source and destination configurations and prov= ide +clear diagnostics when incompatibilities are detected, enabling orchestrat= ion +tools to implement appropriate policies; for example, blocking migration w= hen +such changes are to be made. + +Master Key administration +-------------------------- +It is imperative that s390 Common Cryptographic Architecture (CCA) master = key +administration be performed on both the source and destination AP +devices to synchronize the key values prior to allowing live guest +migration. If the master keys do not match, then crypto applications that +rely on secure keys wrapped by a CCA master key will fail when the guest on +which they are running is migrated to the destination host. Again, this is +an orchestration-layer or system administrator responsibility and migratio= n must +be blocked while master key administration is being performed. --=20 2.53.0