From nobody Sun Nov 24 01:46:02 2024 Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 295A02BAF9; Sat, 9 Nov 2024 06:33:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=148.163.158.5 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731134030; cv=none; b=jvbZt8+WWVHN7PofoXrSM9iYLAAABFoc1T98ShAA6BirX/7A80YM12XTAuKHyn8buXefyTWQZ5yCLo9LnrmZP0rZjRYmbkuVekEKVYy4peYWE89VXAgvBGuKnMnaHPXPsi7yAOGOHlYhbUIpRBbh0WXJP4YMynprPoSBfaTPYYs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731134030; c=relaxed/simple; bh=crBwZEzdt4P6Shs4T1zgayBcwvRsQgOF9Hco+fzKZmI=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WsKSj/sJsR/kkANAkF839mJqs9YS/2yDkwaxExqrhPMIdiEAlR7nigKhqFHDpHo56QooJgvqa/oDx8bBogfRaeb1FfEEnZW8gjVeb5wHtN8bS5GD/5Ihw4b9SRwCnnzTXPI7K4ukV1gjPeGlHUz+iziZ84HEl7TupoXRG0fYH+0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com; spf=pass smtp.mailfrom=linux.ibm.com; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b=W9fU1W3p; arc=none smtp.client-ip=148.163.158.5 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.ibm.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="W9fU1W3p" Received: from pps.filterd (m0356516.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 4A96AF2R013229; Sat, 9 Nov 2024 06:33:28 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=cc :content-transfer-encoding:date:from:in-reply-to:message-id :mime-version:references:subject:to; s=pp1; bh=Gz8RsGttr6chp4RjJ Ef/BGCjFYj8UEZxTPfBDTERkcI=; b=W9fU1W3p2RyHS2Rf7ZoJN41nAR7bzHDjY Q9+pz0Y7XYXZ+dix9Rs7AbJueUln0JnGmvloD3pMYpZf/6uu747b34LgnnM4Q2P2 5h6CPDhjyvsn84oXAPNBIC01flYb+XMhny731oe7hS6+V9ZitZVO8VqRHQ1UIKFU IbJol4iM9hJDJmZfxeDjeogZx0E1OiK61aR9Duzs+drAtlcGT77kHP6J5Je8m/jY xwrHjUtaocbPfaZiK8K5tlIiC59w71Eh/gPjwYRFllFra2BMeU3GGW6pgUWxpT5k 3A03RGJfo6Pj1A0tONW0r1GpGQUI3QDXEepprhuxZk7Iu61poK7Hg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 42t27r01w7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 09 Nov 2024 06:33:27 +0000 (GMT) Received: from m0356516.ppops.net (m0356516.ppops.net [127.0.0.1]) by pps.reinject (8.18.0.8/8.18.0.8) with ESMTP id 4A96XRG2023988; Sat, 9 Nov 2024 06:33:27 GMT Received: from ppma22.wdc07v.mail.ibm.com (5c.69.3da9.ip4.static.sl-reverse.com [169.61.105.92]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 42t27r01w5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 09 Nov 2024 06:33:27 +0000 (GMT) Received: from pps.filterd (ppma22.wdc07v.mail.ibm.com [127.0.0.1]) by ppma22.wdc07v.mail.ibm.com (8.18.1.2/8.18.1.2) with ESMTP id 4A94ajTN023983; Sat, 9 Nov 2024 06:33:26 GMT Received: from smtprelay05.fra02v.mail.ibm.com ([9.218.2.225]) by ppma22.wdc07v.mail.ibm.com (PPS) with ESMTPS id 42nxt0hw5j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 09 Nov 2024 06:33:26 +0000 Received: from smtpav04.fra02v.mail.ibm.com (smtpav04.fra02v.mail.ibm.com [10.20.54.103]) by smtprelay05.fra02v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 4A96XNkd53215510 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 9 Nov 2024 06:33:23 GMT Received: from smtpav04.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 03A3920043; Sat, 9 Nov 2024 06:33:23 +0000 (GMT) Received: from smtpav04.fra02v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 55C2D20040; Sat, 9 Nov 2024 06:33:20 +0000 (GMT) Received: from li-c6426e4c-27cf-11b2-a85c-95d65bc0de0e.ibm.com.com (unknown [9.124.214.93]) by smtpav04.fra02v.mail.ibm.com (Postfix) with ESMTP; Sat, 9 Nov 2024 06:33:20 +0000 (GMT) From: Gautam Menghani To: mpe@ellerman.id.au, npiggin@gmail.com, christophe.leroy@csgroup.eu, naveen@kernel.org, maddy@linux.ibm.com, vaibhav@linux.ibm.com Cc: Gautam Menghani , linuxppc-dev@lists.ozlabs.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/3] KVM: PPC: Book3S HV: Stop using vc->dpdes for nested KVM guests Date: Sat, 9 Nov 2024 12:02:56 +0530 Message-ID: <20241109063301.105289-3-gautam@linux.ibm.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241109063301.105289-1-gautam@linux.ibm.com> References: <20241109063301.105289-1-gautam@linux.ibm.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-TM-AS-GCONF: 00 X-Proofpoint-GUID: -xG-80Z3ZCcl4uSADp85sKTAuF6wSY-5 X-Proofpoint-ORIG-GUID: XKqvrCkzaXn5a7YRozyrrnSDMB9OqG7- X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1051,Hydra:6.0.680,FMLib:17.12.62.30 definitions=2024-10-15_01,2024-10-11_01,2024-09-30_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 impostorscore=0 suspectscore=0 bulkscore=0 clxscore=1015 spamscore=0 lowpriorityscore=0 phishscore=0 priorityscore=1501 adultscore=0 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.19.0-2409260000 definitions=main-2411090049 Content-Type: text/plain; charset="utf-8" commit 6398326b9ba1 ("KVM: PPC: Book3S HV P9: Stop using vc->dpdes") introduced an optimization to use only vcpu->doorbell_request for SMT emulation for Power9 and above guests, but the code for nested guests still relies on the old way of handling doorbells, due to which an L2 guest (see [1]) cannot be booted with XICS with SMT>1. The command to repro this issue is: // To be run in L1 qemu-system-ppc64 \ -drive file=3Drhel.qcow2,format=3Dqcow2 \ -m 20G \ -smp 8,cores=3D1,threads=3D8 \ -cpu host \ -nographic \ -machine pseries,ic-mode=3Dxics -accel kvm Fix the plumbing to utilize vcpu->doorbell_request instead of vcore->dpdes for nested KVM guests on P9 and above. [1] Terminology 1. L0 : PowerNV linux running with HV privileges 2. L1 : Pseries KVM guest running on top of L0 2. L2 : Nested KVM guest running on top of L1 Fixes: 6398326b9ba1 ("KVM: PPC: Book3S HV P9: Stop using vc->dpdes") Signed-off-by: Gautam Menghani --- arch/powerpc/kvm/book3s_hv.c | 9 +++++++++ arch/powerpc/kvm/book3s_hv_nested.c | 14 ++++++++++---- 2 files changed, 19 insertions(+), 4 deletions(-) diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c index 0a8f143e0a75..b93a93777237 100644 --- a/arch/powerpc/kvm/book3s_hv.c +++ b/arch/powerpc/kvm/book3s_hv.c @@ -4282,6 +4282,15 @@ static int kvmhv_vcpu_entry_p9_nested(struct kvm_vcp= u *vcpu, u64 time_limit, uns } hvregs.hdec_expiry =3D time_limit; =20 + /* + * hvregs has the doorbell status, so zero it here which + * enables us to receive doorbells when H_ENTER_NESTED is + * in progress for this vCPU + */ + + if (vcpu->arch.doorbell_request) + vcpu->arch.doorbell_request =3D 0; + /* * When setting DEC, we must always deal with irq_work_raise * via NMI vs setting DEC. The problem occurs right as we diff --git a/arch/powerpc/kvm/book3s_hv_nested.c b/arch/powerpc/kvm/book3s_= hv_nested.c index 05f5220960c6..125440a606ee 100644 --- a/arch/powerpc/kvm/book3s_hv_nested.c +++ b/arch/powerpc/kvm/book3s_hv_nested.c @@ -32,7 +32,7 @@ void kvmhv_save_hv_regs(struct kvm_vcpu *vcpu, struct hv_= guest_state *hr) struct kvmppc_vcore *vc =3D vcpu->arch.vcore; =20 hr->pcr =3D vc->pcr | PCR_MASK; - hr->dpdes =3D vc->dpdes; + hr->dpdes =3D vcpu->arch.doorbell_request; hr->hfscr =3D vcpu->arch.hfscr; hr->tb_offset =3D vc->tb_offset; hr->dawr0 =3D vcpu->arch.dawr0; @@ -105,7 +105,7 @@ static void save_hv_return_state(struct kvm_vcpu *vcpu, { struct kvmppc_vcore *vc =3D vcpu->arch.vcore; =20 - hr->dpdes =3D vc->dpdes; + hr->dpdes =3D vcpu->arch.doorbell_request; hr->purr =3D vcpu->arch.purr; hr->spurr =3D vcpu->arch.spurr; hr->ic =3D vcpu->arch.ic; @@ -143,7 +143,7 @@ static void restore_hv_regs(struct kvm_vcpu *vcpu, cons= t struct hv_guest_state * struct kvmppc_vcore *vc =3D vcpu->arch.vcore; =20 vc->pcr =3D hr->pcr | PCR_MASK; - vc->dpdes =3D hr->dpdes; + vcpu->arch.doorbell_request =3D hr->dpdes; vcpu->arch.hfscr =3D hr->hfscr; vcpu->arch.dawr0 =3D hr->dawr0; vcpu->arch.dawrx0 =3D hr->dawrx0; @@ -170,7 +170,13 @@ void kvmhv_restore_hv_return_state(struct kvm_vcpu *vc= pu, { struct kvmppc_vcore *vc =3D vcpu->arch.vcore; =20 - vc->dpdes =3D hr->dpdes; + /* + * This L2 vCPU might have received a doorbell while H_ENTER_NESTED was b= eing handled. + * Make sure we preserve the doorbell if it was either: + * a) Sent after H_ENTER_NESTED was called on this vCPU (arch.doorbell_= request would be 1) + * b) Doorbell was not handled and L2 exited for some other reason (hr-= >dpdes would be 1) + */ + vcpu->arch.doorbell_request =3D vcpu->arch.doorbell_request | hr->dpdes; vcpu->arch.hfscr =3D hr->hfscr; vcpu->arch.purr =3D hr->purr; vcpu->arch.spurr =3D hr->spurr; --=20 2.45.2