From nobody Sun Nov 24 22:30:11 2024 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D4DDC1BC9ED for ; Thu, 31 Oct 2024 17:06:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730394401; cv=none; b=omiPfM+v/vP8/Ep7fDDJ7Yf9xGf+grQJner9O/t3VaCSl4ZnivNCUTREIciqizekol4n4ra11RTIKx0R40H/rRdRavcA+W+0ROi8aWzinjQW60nWICqG6B8HGjIY5uO7dSz2JfZ2Hu9zQZtLYkpJFCw7EnNV6fM+ys4mh+eS7hw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730394401; c=relaxed/simple; bh=TmFubaHpoCAggLPaGKoJJzqPvhVGJlp46tYw+F5ne78=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=KxTMqXb8vaKbigfuYOROaUPdQJbxt3qOmm0bktFeGVPaVctWvSUfVowcEWgWqlmt7WzkexjJvGcUJc43sod0i2EM2EI4mFx8UVOr1LSzTE+BV8Y+TuR+UJqiiGZeVZgrddK1Abj9rehW1ZtJ50Axyncpdj81cS4zrDq2wgYhij4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=E7+Phsm4; arc=none smtp.client-ip=209.85.128.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="E7+Phsm4" Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-6ea258fe4b6so22743487b3.1 for ; Thu, 31 Oct 2024 10:06:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1730394399; x=1730999199; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=aPh8/kX/bvDxkNAaOErpld4gvqOrWlO93evgZwqYlRo=; b=E7+Phsm4OeSWrbJ5rJ7BojMhbGT0fpZIZbNIwuiF80O04Nnmuc47fCS0tRl+ZWLgEv dGe9nB5mnoIFTCprLKbr8SIfugcuwSxyl1fR4bo6MJsRf1jn9DVSAI0TMU/I6cq11EmR iaoPwlSMWwAcTQ7KjbZWhxsmOBTp0tBBFMKPfh0EPlFEfZGg8XFpbe7kJPIt7az6qh+7 9Qy2O4X++TL4FK+XlTGr3+Iu9Mq6RQ6ZWig+rM4rwJeocY3KW3HasbmXveHmjUVy3GQs GvbkWfM3ZkDK/CJVt/3zD/bUD9HJDQtP/qYNwDP+9NO3Pv75sCZ5EJCFU4DiYBtbFTye EcaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730394399; x=1730999199; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=aPh8/kX/bvDxkNAaOErpld4gvqOrWlO93evgZwqYlRo=; b=eCxXKrH6jjyvEO/3Rb/F6XbNhQE4/UUb19LjBH+lhtIllMqPWJa1KZ/VLy2NZ19wlM PcrwkixSdmZRE9q6fMMif2M5ZTSbm47V1bEL4m7w3Z6LnzIT5YZAhkwrMyJdvm6oZre9 jdnXhKguPS4+GSr8ga5iCoOpGUGaUmZP91xiKPB2jyuWREftxHIeQuiTo36Vzy+s2qeC slL4LkbN2YIQWq4/SaUaUsXrpUhqMWubxvPtVSS4i7R/lIB1Pcd1zyFNNcrdOz0uJz+o aQ4bsE4o331Yqh+FQdjQh47/Qkbx/lwSAxXxpVpPpJW4HUa9BJkfuAwt8JDI2AIkGh1b EmDg== X-Forwarded-Encrypted: i=1; AJvYcCXzGa5tPTEzdvxC7tdg+qSi3CcSuJTWpO5CFcKUkkbCe3xm7JQ+kQKScsWiJTuPjdtX7aN1mWIPNDTG8CQ=@vger.kernel.org X-Gm-Message-State: AOJu0YzYeS1awYeeLCkODQN+ylNzd00sE0new3Af/V/vJb+x6Lawkvze D3mQIwBpZzZBPoSB2qiDGvuYpgAICsv+LygfU1TiSBx59ce4jfsOTJk+IgCXQQwMIrfYmUUPGxR 2Yw== X-Google-Smtp-Source: AGHT+IGiTrpfLC5rBMx47TY3FTKwPiTPo+GH6RHeAuK/D3+uMT5K0Tm4GGSSfwK2VRsPn0K+jhIU4mRjfi4= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a05:690c:6811:b0:6e3:14c3:379a with SMTP id 00721157ae682-6ea6479f2f6mr87277b3.0.1730394398970; Thu, 31 Oct 2024 10:06:38 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 31 Oct 2024 10:06:32 -0700 In-Reply-To: <20241031170633.1502783-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241031170633.1502783-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.163.g1226f6d8fa-goog Message-ID: <20241031170633.1502783-2-seanjc@google.com> Subject: [PATCH 1/2] KVM: x86/mmu: Check yielded_gfn for forward progress iff resched is needed From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Swap the order of the checks in tdp_mmu_iter_cond_resched() so that KVM checks to see if a resched is needed _before_ checking to see if yielding must be disallowed to guarantee forward progress. Iterating over TDP MMU SPTEs is a hot path, e.g. tearing down a root can touch millions of SPTEs, and not needing to reschedule is by far the common case. On the other hand, disallowing yielding because forward progress has not been made is a very rare case. Returning early for the common case (no resched), effectively reduces the number of checks from 2 to 1 for the common case, and should make the code slightly more predictable for the CPU. To resolve a weird conundrum where the forward progress check currently returns false, but the need resched check subtly returns iter->yielded, which _should_ be false (enforced by a WARN), return false unconditionally (which might also help make the sequence more predictable). If KVM has a bug where iter->yielded is left danging, continuing to yield is neither right nor wrong, it was simply an artifact of how the original code was written. Unconditionally returning false when yielding is unnecessary or unwanted will also allow extracting the "should resched" logic to a separate helper in a future patch. Cc: David Matlack Signed-off-by: Sean Christopherson Reviewed-by: James Houghton --- arch/x86/kvm/mmu/tdp_mmu.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index 91caa73a905b..a06f3d5cb651 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -700,29 +700,29 @@ static inline bool __must_check tdp_mmu_iter_cond_res= ched(struct kvm *kvm, { WARN_ON_ONCE(iter->yielded); =20 + if (!need_resched() && !rwlock_needbreak(&kvm->mmu_lock)) + return false; + /* Ensure forward progress has been made before yielding. */ if (iter->next_last_level_gfn =3D=3D iter->yielded_gfn) return false; =20 - if (need_resched() || rwlock_needbreak(&kvm->mmu_lock)) { - if (flush) - kvm_flush_remote_tlbs(kvm); + if (flush) + kvm_flush_remote_tlbs(kvm); =20 - rcu_read_unlock(); + rcu_read_unlock(); =20 - if (shared) - cond_resched_rwlock_read(&kvm->mmu_lock); - else - cond_resched_rwlock_write(&kvm->mmu_lock); + if (shared) + cond_resched_rwlock_read(&kvm->mmu_lock); + else + cond_resched_rwlock_write(&kvm->mmu_lock); =20 - rcu_read_lock(); + rcu_read_lock(); =20 - WARN_ON_ONCE(iter->gfn > iter->next_last_level_gfn); + WARN_ON_ONCE(iter->gfn > iter->next_last_level_gfn); =20 - iter->yielded =3D true; - } - - return iter->yielded; + iter->yielded =3D true; + return true; } =20 static inline gfn_t tdp_mmu_max_gfn_exclusive(void) --=20 2.47.0.163.g1226f6d8fa-goog From nobody Sun Nov 24 22:30:11 2024 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F1E241BF311 for ; Thu, 31 Oct 2024 17:06:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730394405; cv=none; b=Li2txz02tZGkcWTmhJRij9V1A3kJPFOriOB9eA/tFn55ILcjGwnxqz88PaiwDXAWSFQZ45ywDh7JvBERqbGHLp3BLBSeG0dQ7pDVqy7WIM8zTCsoev0U3ogtLJ4/EQ7irdQodE6SulPUWhYXxS54J/yQDLyHwzXqnMwNb9Z8eXw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730394405; c=relaxed/simple; bh=CF9GNWAqKmVnELilmfxXohKaud+5EZ3rcAojpZDvqys=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=kNcOFZiopfd/OBMQwin01T2jezqmNgyLBXJqgGqooehII5jvo1UfW//fNdpxnAFaMm3UjLTzhw77P0cBsfou+xEPuFYVxI7sheLAJG+oilBSSficvoxF6eOkAoTaKK+Cq/7X+6Qbssxpo7rEvnt/GdLmSichcKIMTL7cfKD0FBk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=zOEMREqJ; arc=none smtp.client-ip=209.85.219.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="zOEMREqJ" Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-e29205f6063so1837218276.1 for ; Thu, 31 Oct 2024 10:06:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1730394401; x=1730999201; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=5mvSMrPyn0juvdY997mNRNyl5E1lyo12cgDG/LNUV4Q=; b=zOEMREqJhazgdbctpKNuHwfeHX1dIFH3dJ4GnBizcvRQGVHDCRndraNI2gwPd+L45W Tln524wSE3k9arg2/exxsSt+ETYzyCWXacidpTbBd2TkFlFFGLi/CrgYhJQzRnRq/ZtV lufNsckN92lOHIpSJbGoAXUd1poKo6583ldlxI+xKxsG9aLDyekEB0tB8k9Z98ConHlU H/1avGvS0ocL7ygv1VVMWPE98dVKZDcGQJxA/229PFv2ybe7IQbMfLUzH14d/vpf3iL1 DONs280DRVWLIPuYBmW3RYOD9BWEWUFH69dO82C/LXtjEjag32+Aed08jDodZKO1Ujdf kGAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730394401; x=1730999201; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=5mvSMrPyn0juvdY997mNRNyl5E1lyo12cgDG/LNUV4Q=; b=RL4SQWx9sLRoiftQO9PjyXM50PWYqG+IMPx3qinJ/NPjPdzIBWOBPUeNAXaDE2aHpL Ybfa+BufV+JSydMoiPpGHrI+ZztclNm4/itYSHbTKaxnxgABumBytR7AbWK+Ev/TQ6Xc 3jFsAEPMUpK6ITS6Wm7QDl0nroy24UBmEqE8o/csoyQFlw6tgeDEvFuq8Mon20lpF7JX BCMReLI+ltZmcD8J5YP0TYwK8J9UinHJo+d+nuWB/6GXOt0UJSVDlGwKUuLlEiGNJMN4 y9brJT8NqLXGpMYyizMYTYN9duaq9FImdkHqfQj0yTI/WFERZtIUnG4N6Grg3y4kr1i7 cLjA== X-Forwarded-Encrypted: i=1; AJvYcCUDhvRSt3msE9PAC0ToaL2VJwg92NS65QrICeCBv+hG+f/6W+ubwvWMn+GSUt0cFtBdLDMTSoT8QzAOjho=@vger.kernel.org X-Gm-Message-State: AOJu0YxXrkSHbiC2IyTti1eG3SzKvRXSJon5/mXsgb9ywJwnh15MgqVz Spznc3IJXoWlbX+pxFxNyrdcHEQmUl7g+4qTKtgCjeHXtk/x80AV1EmgKrkLnmrJRM0sIICv6jg cNQ== X-Google-Smtp-Source: AGHT+IHaNKkAsPzvMlrC7DsQB+BKEBKT51iYS8Lpz9uTMWn9JUZ5fiG1CamoxG8hwdIvdxbOzXOPUscKPTA= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:9d:3983:ac13:c240]) (user=seanjc job=sendgmr) by 2002:a25:b10e:0:b0:e30:d9d3:b87 with SMTP id 3f1490d57ef6-e33026aa6c5mr845276.8.1730394400585; Thu, 31 Oct 2024 10:06:40 -0700 (PDT) Reply-To: Sean Christopherson Date: Thu, 31 Oct 2024 10:06:33 -0700 In-Reply-To: <20241031170633.1502783-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20241031170633.1502783-1-seanjc@google.com> X-Mailer: git-send-email 2.47.0.163.g1226f6d8fa-goog Message-ID: <20241031170633.1502783-3-seanjc@google.com> Subject: [PATCH 2/2] KVM: x86/mmu: Demote the WARN on yielded in xxx_cond_resched() to KVM_MMU_WARN_ON From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, David Matlack Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Convert the WARN in tdp_mmu_iter_cond_resched() that the iterator hasn't already yielded to a KVM_MMU_WARN_ON() so the code is compiled out for production kernels (assuming production kernels disable KVM_PROVE_MMU). Checking for a needed reschedule is a hot path, and KVM sanity checks iter->yielded in several other less-hot paths, i.e. the odds of KVM not flagging that something went sideways are quite low. Furthermore, the odds of KVM not noticing *and* the WARN detecting something worth investigating are even lower. Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/tdp_mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index a06f3d5cb651..c158ef8c1a36 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -698,7 +698,7 @@ static inline bool __must_check tdp_mmu_iter_cond_resch= ed(struct kvm *kvm, struct tdp_iter *iter, bool flush, bool shared) { - WARN_ON_ONCE(iter->yielded); + KVM_MMU_WARN_ON(iter->yielded); =20 if (!need_resched() && !rwlock_needbreak(&kvm->mmu_lock)) return false; --=20 2.47.0.163.g1226f6d8fa-goog