From nobody Fri Sep 25 16:02:05 2026 Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EEBDA4FB9A3 for ; Thu, 10 Sep 2026 16:23:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.70 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057440; cv=none; b=NfsCKR7N5vtSpksyZUUFtJ+8YfVdPrhn9DlxNHkBEcm8pQxOXwVlL7TW6mWlQW8R0WmssZ1Hr49xlwNuG0A7hlvwWSFiOvdR/XbmRYQVXWmHiO7ZaxaKYNozl7ehDF0oW/fV5yeEl2f9+AquyD9BDEyoFINFsDaH0sAbfuFzXYU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057440; c=relaxed/simple; bh=vLBwpA/W09CA4nuBN8wyhZ23eWcSQH6kAchso1GU80E=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=pICupHP3D8UAaY5ecAC0WpcKLkzY2m4nRcoLYx2yvb9OScmF3DaVt5Ts64u3SCb4YCVXHHT6K0aXIqApgcNBPJBSQP5u3SDLIhfUF0LIXH4LXIljk4XqAXZJ0eDwYSWdpFD41x8ny1ZKKSsJC4pFstOwOZnEXHgsQC5xwKpPy2g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--elver.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Wf0Nuwce; arc=none smtp.client-ip=209.85.208.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--elver.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Wf0Nuwce" Received: by mail-ed1-f70.google.com with SMTP id 4fb4d7f45d1cf-6a5d682e372so1235124a12.0 for ; Thu, 10 Sep 2026 09:23:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1789057436; x=1789662236; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=i8SjnCZzsDgkEIH+7JYFlN1mRbLf5pSXtHbHm73zeYY=; b=Wf0NuwceSdk8Nu0ieEuDi/TN6+n2hk9pksr19hSpCfWRU/7FJSJsbmp+/ZMAV1Z8KF HJNnFxlalgIojjE0agHcme/zMEXSBEnsCUsVFESc6UVAo8uNqK4cee1tSr+EgHUiM7WC 1KaQU+gF+vHy/qjTprBpIDPjYkqLCLBRl2vHCgI5ZuFyre3dxKHnSno4SwRh8nFjBnUu saAsQakwmwytf2ff2NQwMefZeUOMFNQrZOd1BjIkPIcf3EUTKccnnVWsYu4cRAr7tIX2 E2X/MKwqptK89JGwN7EoKcGp2t7wOdV7RB6xT7jWH2QcuK/cYTos0vJyTKs88mxqVjuz 4Ghw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789057436; x=1789662236; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=i8SjnCZzsDgkEIH+7JYFlN1mRbLf5pSXtHbHm73zeYY=; b=X1WMb/jT5uYZOK+kr56JyMG2c3G8WsQBusL3jb4uoGw4znfgzzzrtkGJHkD6ldjC0X c/lTkBQCYzQGyHPr6BO/tKQyFDxTudcWMr4mSx5WsdRpvheP437pWp/Vo8X33JM54R0U b0WXUU/SNIDfHXzeZ8Ly1uvfvSB/3gpAWrEuXTE7mOasgTi2yAZHpAvDJhLwDqaHZIAZ e+WnUcWNtT5ajvT3jQS6OTd58ch7zhTupi4OWLuZ3eBmddU/C5vojAeP7/gVemyRLxnu eeAIY75S3Lm/ZN6gAjD4UVLAkhIMsIM7PKFPmP/KhFAqOH1ktvbFJQ8qCd24dQJAIJnq MeuQ== X-Forwarded-Encrypted: i=1; AKwUvBwUehfzJUZssjNno/nHqLZSmLPG2v/29c7UWHyYN1tDyj5h9srI7PvTIBWxPTQg+arGA30BAm8xfBH+XK4=@vger.kernel.org X-Gm-Message-State: AFuF++kmpD2G3kC8qOg4wi+8QLK5a8d3P5YRqNBlCbwLvRyoBnwG7vfc uFSR3xKfGofzq05wQSqnpUgOWl/EuqXaHE1LzTBTG/vUfWYbAtoaoUV+Q/VOZLg7NHvBqFnVpSt INg== X-Received: from edqr4.prod.google.com ([2002:aa7:d584:0:b0:6a9:853c:f2bd]) (user=elver job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6402:c28e:b0:6a6:75b7:4ff6 with SMTP id 4fb4d7f45d1cf-6a9a3136f44mr1906127a12.8.1789057435775; Thu, 10 Sep 2026 09:23:55 -0700 (PDT) Date: Thu, 10 Sep 2026 16:21:34 +0000 In-Reply-To: <20260910162343.4092060-1-elver@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260910162343.4092060-1-elver@google.com> X-Mailer: git-send-email 2.55.0.1003.g10538fe699-goog Message-ID: <20260910162343.4092060-2-elver@google.com> Subject: [PATCH RFC 01/10] KVM: x86/pmu: Acquire SRCU in pmc_is_event_allowed() to protect filter lookup From: Marco Elver To: elver@google.com Cc: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Vitaly Kuznetsov , Kiryl Shutsemau , Rick Edgecombe , David Hildenbrand , kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Dereferencing kvm->arch.pmu_event_filter via srcu_dereference() requires holding kvm->srcu to guard against concurrent filter replacement and freeing by kvm_vm_ioctl_set_pmu_event_filter(). Counter reprogramming can reach pmc_is_event_allowed() without holding kvm->srcu. Specifically, on AMD SVM, toggling EFER.SVME via KVM_SET_SREGS or KVM_SET_SREGS2 triggers synchronous counter reprogramming outside of any SRCU read-side critical section: kvm_vcpu_ioctl(KVM_SET_SREGS{,2}) kvm_vcpu_ioctl_x86_set_sregs{,2}() __set_sregs_common() kvm_x86_call(set_efer)() svm_set_efer() svm_pmu_handle_nested_transition() __svm_pmu_handle_nested_transition(..., defer=3Dfalse) __kvm_pmu_reprogram_counters() kvm_pmu_handle_event() reprogram_counter() pmc_is_event_allowed() srcu_dereference(kvm->arch.pmu_event_filter, &kvm->= srcu) If userspace concurrently updates the filter (KVM_SET_PMU_EVENT_FILTER), a concurrent free and subsequent use-after-free is possible. Protect filter lookups directly in pmc_is_event_allowed(): 1. check rcu_access_pointer() first for the common fast path; 2. acquire guard(srcu)(&kvm->srcu) only when a filter is present; 3. drop redundant outer srcu_read_lock() in kvm_pmu_trigger_event(). Found with Clang context analysis. Fixes: a02a25a65246 ("KVM: x86/pmu: Reprogram Host/Guest-Only counters on n= ested transitions") Signed-off-by: Marco Elver --- arch/x86/kvm/pmu.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/pmu.c b/arch/x86/kvm/pmu.c index a7d60c8785cd..3ad1e696edca 100644 --- a/arch/x86/kvm/pmu.c +++ b/arch/x86/kvm/pmu.c @@ -536,6 +536,11 @@ static bool pmc_is_event_allowed(struct kvm_pmc *pmc) struct kvm_x86_pmu_event_filter *filter; struct kvm *kvm =3D pmc->vcpu->kvm; =20 + if (!rcu_access_pointer(kvm->arch.pmu_event_filter)) + return true; + + guard(srcu)(&kvm->srcu); + filter =3D srcu_dereference(kvm->arch.pmu_event_filter, &kvm->srcu); if (!filter) return true; @@ -1132,7 +1137,7 @@ static void kvm_pmu_trigger_event(struct kvm_vcpu *vc= pu, DECLARE_BITMAP(bitmap, X86_PMC_IDX_MAX); struct kvm_pmu *pmu =3D vcpu_to_pmu(vcpu); struct kvm_pmc *pmc; - int i, idx; + int i; =20 BUILD_BUG_ON(sizeof(pmu->global_ctrl) * BITS_PER_BYTE !=3D X86_PMC_IDX_MA= X); =20 @@ -1145,14 +1150,12 @@ static void kvm_pmu_trigger_event(struct kvm_vcpu *= vcpu, (unsigned long *)&pmu->global_ctrl, X86_PMC_IDX_MAX)) return; =20 - idx =3D srcu_read_lock(&vcpu->kvm->srcu); kvm_for_each_pmc(pmu, pmc, i, bitmap) { if (!pmc_is_event_allowed(pmc) || !cpl_is_matched(pmc)) continue; =20 kvm_pmu_incr_counter(pmc); } - srcu_read_unlock(&vcpu->kvm->srcu, idx); } =20 void kvm_pmu_instruction_retired(struct kvm_vcpu *vcpu) --=20 2.55.0.1003.g10538fe699-goog From nobody Fri Sep 25 16:02:05 2026 Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CDEB54FC8E5 for ; Thu, 10 Sep 2026 16:23:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.70 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057442; cv=none; b=sX6jUKYWZMwQqfLb1K3BKk3KQ4gsmNZ/4LZVruUUz5MBB5bxKPhrAmjKmAp/YP5SAW/cYn+Y+nKg0OLUspRh/TR9hShGy5ryrNjMY7S0J2PBANXGxlzqGEIcIjYRbZY8skR8BtxSgmVjiQkfiSs4s0lJYp+6Y3KLCJSqP5JrF+0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057442; c=relaxed/simple; bh=FXlmfD+i35+iX4uCD+rWudM4SzaRO4YkxNrcJ/VTr5Q=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=VXijH3vCTzyoe6WmlDH+8PdZTg+Gh625MeI1SHnhcXA4YdWGTJebVue0X372efl8TFSnSR+GzMl7yz7s4ZXrh0YDTUPFLTHO2zoT6ntURYjIw8tdk0NTa1Df74ErfjTv4ZLw2maYe0wbeLc6PDnYnP3bj57BugJ+tAj/SNDlWYM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--elver.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=kMqlNF4H; arc=none smtp.client-ip=209.85.128.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--elver.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="kMqlNF4H" Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-495474a5fbcso161755e9.1 for ; Thu, 10 Sep 2026 09:23:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1789057437; x=1789662237; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=2noM4J4G4Z/91+d3z307VtID9oIWCOO73XBAg9AhrEM=; b=kMqlNF4Hn1J9tFyjo5gvrCqIBqsyI5THMEtF5+toeGriWCHILvYPeFgt/Z5+/7M704 DMjoYZfP4PwlndhEG4erxP75XeZUxbhu7qkXa2hv5u/Nf6uKS7OEZMmt2t2NZdnZTlz5 TuD6VudMzHOMGcX3zr+IDD0D0kI04z0PYLCWUcBa5cMjw9uATNasHG+FhR8ysSWhPjSx kTNnXwEMtRQi2Bs7TQ/J2mRrJH7nGTyXUKaVV5ocb/dK3sV3c000JB6lZT25Y8B3VSdk jInuhpYLHNkOWYVmwceAswTCwTC9LtmcS4N3pTLzTDeTzvsIJneX9p31r7YUJl5ZkXmz 3f8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789057437; x=1789662237; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=2noM4J4G4Z/91+d3z307VtID9oIWCOO73XBAg9AhrEM=; b=dQ2nuEhX9EB/hbULO/cK6opcORt2466yKMkeb4JT6skC8r/OhaCE0Xbpll2tsYx5oF dgec85ca6LloWRn/lH6S+5EFSBDHQj2Nh2fCvQXw66Y6G2w47IhKGMqfUCtR/EE4d6D3 LoM1tZgXei6gH9rbPf4mHt56ZCiaVQjKJBDYTOLUEvmplqzB2CfkgfqV5e/QZm6E0GJ/ JY5nC4scMuYfKmyCgjcF5FakKqSA6VIAINYCgOBgWPM1VZhUjuMXUnJOLLbIqekH3nGp h3kDLcUXWoHZDKVdlmrAWzdLnYcXqYVnxe2nylK03Y2/jHKMSCLY+4ERON+AJWs5KmnE sctg== X-Forwarded-Encrypted: i=1; AKwUvByBy55J/Q5ZnTRkYNwvhpxYK5g2xic3R5c16nplN7tIQQZniezeaK/BoEe+d7wJF8QYVk4DE+F5ByPEUkU=@vger.kernel.org X-Gm-Message-State: AFuF++lkXOZaPoBo7H79M3TEeLquE9b5GFxHnZdJPyPyqOmrmlIbd8iw WBA+7y8ph348w9q4XkZtJz5uffT7LMLWml8dnja3u5vg99z4kAi3PMAnUCAwngXNfPAhleETsHg O3A== X-Received: from wrzf4.prod.google.com ([2002:a05:6000:1b04:b0:484:36e2:923e]) (user=elver job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600d:848e:20b0:495:4d88:e630 with SMTP id 5b1f17b1804b1-49cf824f697mr311679985e9.10.1789057437113; Thu, 10 Sep 2026 09:23:57 -0700 (PDT) Date: Thu, 10 Sep 2026 16:21:35 +0000 In-Reply-To: <20260910162343.4092060-1-elver@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260910162343.4092060-1-elver@google.com> X-Mailer: git-send-email 2.55.0.1003.g10538fe699-goog Message-ID: <20260910162343.4092060-3-elver@google.com> Subject: [PATCH RFC 02/10] KVM: Allow reading memslots while holding slots_arch_lock From: Marco Elver To: elver@google.com Cc: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Vitaly Kuznetsov , Kiryl Shutsemau , Rick Edgecombe , David Hildenbrand , kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" kvm_swap_active_memslots() updates kvm->memslots[as_id] while holding both kvm->slots_lock and kvm->slots_arch_lock. Holding either lock guarantees that memslots cannot be concurrently modified. Allow reading memslots in __kvm_memslots() when kvm->slots_arch_lock is held. Signed-off-by: Marco Elver --- include/linux/kvm_host.h | 1 + 1 file changed, 1 insertion(+) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 03bfc92864b6..5ed8260ef01f 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1089,6 +1089,7 @@ static inline struct kvm_memslots *__kvm_memslots(str= uct kvm *kvm, int as_id) as_id =3D array_index_nospec(as_id, KVM_MAX_NR_ADDRESS_SPACES); return srcu_dereference_check(kvm->memslots[as_id], &kvm->srcu, lockdep_is_held(&kvm->slots_lock) || + lockdep_is_held(&kvm->slots_arch_lock) || !refcount_read(&kvm->users_count)); } =20 --=20 2.55.0.1003.g10538fe699-goog From nobody Fri Sep 25 16:02:05 2026 Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7D6D550EC09 for ; Thu, 10 Sep 2026 16:24:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.69 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057444; cv=none; b=oxpA4zBCcF+ACkeE83FuHmpNRi1gXPGr9V0RQZd0TU9/S12gmHpqtjmZ5PU3aKA/aK9EXF95CFNMME+j3oyqWcsHNtTgPeAGWBkrwpxL8VXe35VhMQMbsBrT093s2tZa/rgAC24v6y9TLv1CFjs/iOL4q8EZqTFUPb+PbLZ5sRU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057444; c=relaxed/simple; bh=MnMX/OSFrRzbTxG7baAXBumw8j8z3VdCQcDWmSfrIJ0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=qdvvJW1UlB2fkqelQ9yFkrQtqUHqNGZqyC0EL6wLQ1EtI5pdQfWZfkfhZ8G8omBaj4ovfCZhw+lG94t9Mq3kCJRstfXxBxxv51UxvrfVl4NtW+MBJDML0KGPUz3fHFXuYJi9pjkYtBX78pgpdiqfUoW9xQ6SGbBEp0gIDmixhiw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--elver.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WQ6am5kD; arc=none smtp.client-ip=209.85.128.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--elver.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WQ6am5kD" Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-49953abe51fso69845e9.1 for ; Thu, 10 Sep 2026 09:24:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1789057439; x=1789662239; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=RRWmScgPx+BLIgnYUF1mxiVgMRyBAcMFYt7T+8tBBLE=; b=WQ6am5kDkIfSMrePPt0m4Sa89K6912KJnDNj1ZlfI81/JHyfLf4XwCYtb+HyKQJqs+ bZvRRBKVB0zXhJu0WhZg1iFHFfWqGkDB1vM4wkmF9HLok3P2QNR5omVUfzkSaMxoHmM1 TW+4TYY8pHnDEehoY0z/iisB8AULso/6+NzHk/3IMYgLA4hHy1OInvRZkKxgZgGz13qH 7pW0BwjMQdKjgsyDnbHSlni36zWNX7NxUycUpwH89lmB3PUbgjFo2sRTghkCGKe6A9Tl cmgSfg+WkHrCqbqPvhreW7YsYL/N5QzQJcwRaYNtXPwGgETfL+y06Ov6rVP4+s7Blm6C oDJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789057439; x=1789662239; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=RRWmScgPx+BLIgnYUF1mxiVgMRyBAcMFYt7T+8tBBLE=; b=lgYyXo4X0GwhQq2bwIH1LHH1ifWgQpBulS5fDhyBwPFH6+22dfsxYDoCsEZffXwbqg snVQFf8G6RD92zPWof50IhAlryQtNeOEoGHXEKTsWeM42HDWr/a2Z6B7C1OGMgVkpFvO d2VScKIUP5dyvDxJ5osbBDnBvCREv5mhchxw1ZCc8H8KlSK/GiWfqyz3zAjavDWS+kcD YVA3g0c2jOaWi0qrGr1DSvZuCH49ApT5QYTAvM4aC/1/Ga72JySeM9Hz2FYEKg+6Dpuh 3dn9PvsrzMgtR9Z8E8DXfyFddCwm1FJyegiAlPA9V1+Q3iiCpUX5TOW4dJj0h1vh0hTF nNrA== X-Forwarded-Encrypted: i=1; AKwUvBwm9TtZiHiXA8GsEilukWBAmn9YlL5+JxsdBZnZQ+eoU09XpMp1DJ/7Kmxqaz5RdHaEm+7XecpeGH5LqVE=@vger.kernel.org X-Gm-Message-State: AFuF++mnGtGAoy+XAbjvmkrOhErG0k+zwv8v3/0rA4hLE/YOWjTwX32s UOnRltRaMzFPxcdTI0G39+mVJuaeTAEOvVgc09mys1TUrjyWyl+bpnnM2IrPGu4zYwGuYIaiy39 g0Q== X-Received: from wmoy5.prod.google.com ([2002:a05:600c:17c5:b0:499:d96d:63b2]) (user=elver job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:46d5:b0:49c:fc6c:be06 with SMTP id 5b1f17b1804b1-49cfc6cc142mr367514185e9.29.1789057438931; Thu, 10 Sep 2026 09:23:58 -0700 (PDT) Date: Thu, 10 Sep 2026 16:21:36 +0000 In-Reply-To: <20260910162343.4092060-1-elver@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260910162343.4092060-1-elver@google.com> X-Mailer: git-send-email 2.55.0.1003.g10538fe699-goog Message-ID: <20260910162343.4092060-4-elver@google.com> Subject: [PATCH RFC 03/10] KVM: guest_memfd: Avoid conditional mmu_lock acquisition From: Marco Elver To: elver@google.com Cc: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Vitaly Kuznetsov , Kiryl Shutsemau , Rick Edgecombe , David Hildenbrand , kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Refactor __kvm_gmem_invalidate_start() to check for present bindings upfront via xa_find() instead of tracking found memslots inside the range iteration with a 'found_memslot' flag and conditionally acquiring and releasing KVM's mmu_lock. Eliminating the conditional locking inside the loop simplifies control flow, ensures lock scoping is straightforward, and subsequently allows Clang context analysis to verify that mmu_lock is held unconditionally across kvm_mmu_unmap_gfn_range(). No functional change intended. Signed-off-by: Marco Elver --- virt/kvm/guest_memfd.c | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 625e62e1a031..896f3b076562 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -124,10 +124,16 @@ static void __kvm_gmem_invalidate_start(struct gmem_f= ile *f, pgoff_t start, pgoff_t end, enum kvm_gfn_range_filter attr_filter) { - bool flush =3D false, found_memslot =3D false; + bool flush =3D false; struct kvm_memory_slot *slot; struct kvm *kvm =3D f->kvm; - unsigned long index; + pgoff_t index =3D start; + + if (!xa_find(&f->bindings, &index, end - 1, XA_PRESENT)) + return; + + KVM_MMU_LOCK(kvm); + kvm_mmu_invalidate_start(kvm); =20 xa_for_each_range(&f->bindings, index, slot, start, end - 1) { pgoff_t pgoff =3D slot->gmem.pgoff; @@ -140,13 +146,6 @@ static void __kvm_gmem_invalidate_start(struct gmem_fi= le *f, pgoff_t start, .attr_filter =3D attr_filter, }; =20 - if (!found_memslot) { - found_memslot =3D true; - - KVM_MMU_LOCK(kvm); - kvm_mmu_invalidate_start(kvm); - } - flush |=3D kvm_mmu_unmap_gfn_range(kvm, &gfn_range); =20 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_INVALIDATE @@ -157,8 +156,7 @@ static void __kvm_gmem_invalidate_start(struct gmem_fil= e *f, pgoff_t start, if (flush) kvm_flush_remote_tlbs(kvm); =20 - if (found_memslot) - KVM_MMU_UNLOCK(kvm); + KVM_MMU_UNLOCK(kvm); } =20 static void kvm_gmem_invalidate_start(struct inode *inode, pgoff_t start, --=20 2.55.0.1003.g10538fe699-goog From nobody Fri Sep 25 16:02:05 2026 Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 682F74CA798 for ; Thu, 10 Sep 2026 16:24:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.71 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057445; cv=none; b=sszF6AX1z5iAd6CPMyKsMNb+n8aSU8fTMXvX2qMhxyzoPgnTVBt1g7Xn+j2aXmq83HHiyMCPTtYhNvZAsSEtjdWtYQqdcn5OhhE+cVONLoqODTrwo3132aYPdT3D755CLgCNLYXQiVd/j5fALa1/Ne93dj38rxeFEBVS9B80cr4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057445; c=relaxed/simple; bh=/1ZUt4dzZcbPmaM7S+5OTB49kLBy/Ei/YrHi3jIC1+w=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Y9oRd3Kt9SHWg95kZkzdmlTo1ParGcOQX10ghJ1pFqJzuToZzlXp3/6fXZDZ8OWUU0ZV3bmvo2Eg/+/60WR6s4EHKroMyM4mx1fRgGBGsCFedi+cUMx2vIAFoTNpe6kvDaWthm8z0C68m3VxfkgypjJf3PkEbYo2otOXuhR4VmA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--elver.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=s+z7JNL5; arc=none smtp.client-ip=209.85.128.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--elver.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="s+z7JNL5" Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-49cced8309bso25595e9.3 for ; Thu, 10 Sep 2026 09:24:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1789057440; x=1789662240; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=XwSVU+R2t/RzQm+JBBs/i0HD0WgNEqi2oht1lvIUFrI=; b=s+z7JNL5hyfM8LPmVrmuutw+5V81yCVHH70WrVk3eR0j202y/4ZY39QE2DGiZXmiT8 8cJTdyNMr8JEvPOmPT8Gs2xC1fGcDgphbYS7dRKWSsEAaRFkEjAOeuG6Q8vyt9KuvbBs 1YCLMdCNACzErNDteOhr2UJlGFNlTMqpXN03cVRnadiSFDfftI9MnycdWzacbg/h3dE4 fVLP1HxTwchUMCbJG7P4/bvDf2KznkGg0PYuJj+DEC6kv9sZMfJijZDFjH1woW5ls8YS OFG1MevnA2sbPuld4vdUIAfVih5l3Uq/AFNT9zs6O7ZJkUwjsWavas2XOqPC4ybMtKbp 6xMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789057440; x=1789662240; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=XwSVU+R2t/RzQm+JBBs/i0HD0WgNEqi2oht1lvIUFrI=; b=GpyOvCSThSfgkOWcDwcxOMEXdZsDgOIi6/CXQTUDg27gPPGP/PjRDeEywf0KVFjFo2 PpSl+BMM+sE4deBgKAiYlyvc/5BfWOknMHnkxOwX34Qbb6HobU74rI/WbMFMGHPzsJ0c XYSB7yegdZWdyCrMLW7VdwuTlAIgz1rZ3vaXWSvJmrvUcypJbgNi2srPlpa1Xx0P8Lth mdboOYGRrqmnrvixUo/PgGDHq3L9W0QlvI2hQR2JNt7V1i+880RkgGKQqyrqyjUNIjEK KbKTRloqmT4zmtboIFob+HHQ8/28wOGyeELY+3gc5bcpQNpmoxdm3Y2VZUJ9c8VeAI0W 7jag== X-Forwarded-Encrypted: i=1; AKwUvBz0GYKtQSOXlL9sDjwv5OFotBIXcbk21hyWvB9uqE0/8oY6FuTD9SnsvnsA3iC5cPrSWEdzygjtKE26Quw=@vger.kernel.org X-Gm-Message-State: AFuF++ng7uigTVWZivQPHNxA63pjfSYeB7doDCvxdqyEiG7ahIYpy3J9 Ro0ik1CvCpjmKoAfSFu6fSgzS5nLFTgpQSYZYa2E1lt1Lo0EOlC3K1COSXTJrKeVxVMuRzjlpka ELg== X-Received: from wmv11.prod.google.com ([2002:a05:600c:26cb:b0:499:4c8d:551f]) (user=elver job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:a00d:b0:49c:cfbe:5a76 with SMTP id 5b1f17b1804b1-49cf81e4a05mr405814905e9.2.1789057440169; Thu, 10 Sep 2026 09:24:00 -0700 (PDT) Date: Thu, 10 Sep 2026 16:21:37 +0000 In-Reply-To: <20260910162343.4092060-1-elver@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260910162343.4092060-1-elver@google.com> X-Mailer: git-send-email 2.55.0.1003.g10538fe699-goog Message-ID: <20260910162343.4092060-5-elver@google.com> Subject: [PATCH RFC 04/10] KVM: Refactor kvm_handle_hva_range() to avoid conditional mmu_lock From: Marco Elver To: elver@google.com Cc: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Vitaly Kuznetsov , Kiryl Shutsemau , Rick Edgecombe , David Hildenbrand , kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Refactor kvm_handle_hva_range() to check for overlapping memslots upfront via interval_tree_iter_first() instead of tracking found memslots inside the range iteration with a 'found_memslot' flag and conditionally acquiring and releasing mmu_lock. This simplifies the control flow by cleanly decoupling the search for overlapping memslots from the subsequent walk. It also separates the lockless path from the serialized path into distinct branches, eliminating the conditional locking, which subsequently enables Clang context analysis to validate locking in this function. No functional change intended. Signed-off-by: Marco Elver --- virt/kvm/kvm_main.c | 94 ++++++++++++++++++++++++++++----------------- 1 file changed, 58 insertions(+), 36 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 65eb26a0520d..f7bfa2d32507 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -558,31 +558,14 @@ static void kvm_null_fn(void) node; \ node =3D interval_tree_iter_next(node, start, last)) \ =20 -static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm, - const struct kvm_mmu_notifier_range *range) +static __always_inline bool __kvm_handle_hva_range_walk(struct kvm *kvm, + const struct kvm_mmu_notifier_range *range) { - struct kvm_mmu_notifier_return r =3D { - .ret =3D false, - .found_memslot =3D false, - }; struct kvm_gfn_range gfn_range; struct kvm_memory_slot *slot; struct kvm_memslots *slots; - int i, idx; - - if (WARN_ON_ONCE(range->end <=3D range->start)) - return r; - - /* A null handler is allowed if and only if on_lock() is provided. */ - if (WARN_ON_ONCE(IS_KVM_NULL_FN(range->on_lock) && - IS_KVM_NULL_FN(range->handler))) - return r; - - /* on_lock will never be called for lockless walks */ - if (WARN_ON_ONCE(range->lockless && !IS_KVM_NULL_FN(range->on_lock))) - return r; - - idx =3D srcu_read_lock(&kvm->srcu); + bool ret =3D false; + int i; =20 for (i =3D 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) { struct interval_tree_node *node; @@ -620,28 +603,67 @@ static __always_inline kvm_mn_ret_t kvm_handle_hva_ra= nge(struct kvm *kvm, gfn_range.slot =3D slot; gfn_range.lockless =3D range->lockless; =20 - if (!r.found_memslot) { - r.found_memslot =3D true; - if (!range->lockless) { - KVM_MMU_LOCK(kvm); - if (!IS_KVM_NULL_FN(range->on_lock)) - range->on_lock(kvm); + ret |=3D range->handler(kvm, &gfn_range); + } + } + + return ret; +} =20 - if (IS_KVM_NULL_FN(range->handler)) - goto mmu_unlock; - } - } - r.ret |=3D range->handler(kvm, &gfn_range); +static __always_inline kvm_mn_ret_t kvm_handle_hva_range(struct kvm *kvm, + const struct kvm_mmu_notifier_range *range) +{ + struct kvm_mmu_notifier_return r =3D { + .ret =3D false, + .found_memslot =3D false, + }; + struct kvm_memslots *slots; + int i, idx; + + if (WARN_ON_ONCE(range->end <=3D range->start)) + return r; + + /* A null handler is allowed if and only if on_lock() is provided. */ + if (WARN_ON_ONCE(IS_KVM_NULL_FN(range->on_lock) && + IS_KVM_NULL_FN(range->handler))) + return r; + + /* on_lock will never be called for lockless walks */ + if (WARN_ON_ONCE(range->lockless && !IS_KVM_NULL_FN(range->on_lock))) + return r; + + idx =3D srcu_read_lock(&kvm->srcu); + + for (i =3D 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) { + slots =3D __kvm_memslots(kvm, i); + if (interval_tree_iter_first(&slots->hva_tree, range->start, range->end = - 1)) { + r.found_memslot =3D true; + break; } } =20 - if (range->flush_on_ret && r.ret) - kvm_flush_remote_tlbs(kvm); + if (!r.found_memslot) + goto out; + + if (range->lockless) { + r.ret =3D __kvm_handle_hva_range_walk(kvm, range); + if (range->flush_on_ret && r.ret) + kvm_flush_remote_tlbs(kvm); + } else { + KVM_MMU_LOCK(kvm); + if (!IS_KVM_NULL_FN(range->on_lock)) + range->on_lock(kvm); + + if (!IS_KVM_NULL_FN(range->handler)) + r.ret =3D __kvm_handle_hva_range_walk(kvm, range); + + if (range->flush_on_ret && r.ret) + kvm_flush_remote_tlbs(kvm); =20 -mmu_unlock: - if (r.found_memslot && !range->lockless) KVM_MMU_UNLOCK(kvm); + } =20 +out: srcu_read_unlock(&kvm->srcu, idx); =20 return r; --=20 2.55.0.1003.g10538fe699-goog From nobody Fri Sep 25 16:02:05 2026 Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B8F85509EFB for ; Thu, 10 Sep 2026 16:24:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.69 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057446; cv=none; b=XYdTXfe2r4o2+WB2h9c3qL+aaImUkuTw9RHCirF2kE0VNQeVo0kwLDnGAvXWSRgf21CwVE0MSYYQE1JzNEFj9VZ/LtQuRY9WfRl26N6tkEBY4IDK4oK6oEPVwtMGC/v7vS5R13rs5AFyYKQgP6IM+zV0bWF3KZOdk5Ii2qCmyZw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057446; c=relaxed/simple; bh=Dwq4yWxNTOPewjvgBHCVt6k8PpiEcRg1OwsQvoXmnaE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=kZtxGjkfvgVkU7gI2mGV6skjSFihwvu9qyrU0rluC8LNqyLS6KkpynfKgkZ+ciR9X/mkgYvcNbh3+e29pyuBShRRTGalN3uYoTqX/UZVtg0FluvRy8rfW0pTRTZmQdSxSBhUmc55bw1IK6WgcRid1fRmllsCgrg5tw0VDJNNW5s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--elver.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Ph6IZwIM; arc=none smtp.client-ip=209.85.128.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--elver.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Ph6IZwIM" Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-49cd55ee2f7so211435e9.0 for ; Thu, 10 Sep 2026 09:24:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1789057441; x=1789662241; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=uC0obBsMpYIPwDDLkx/VVIyvnyMYMqgtwKGzNKgjqas=; b=Ph6IZwIM2tdp+/TbYjMKGDXDoCwP2Tv4UEYk7QjLyD6z9P48xnWpUikLyCtxVev1A6 LNSFBjAlBX65bS2jLV9fanv8Y+I2pIpML4O1fCo4OOq1KKyHbwmJGZgxd6ye13hXZ/5e iTeUjODlWl+sUVLL0yUsoKJNRbfN8Du74xk2z266tnKos0DPT3F6JJjPMsBCxKnOwYy3 AOr7adylDXU366qV8fLTJjUIMvX7ltVMHSQlDMcsgUdTctn4673bToeeP1fQxx+4brIW zUatn+Qnw19RSVSOT5HZJ5o3AJ4gC4EZ6F7nD/u6t2sjPARD74H5ycdXJ6Xa42uYlJjy K9hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789057441; x=1789662241; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=uC0obBsMpYIPwDDLkx/VVIyvnyMYMqgtwKGzNKgjqas=; b=kQIXs4+3WeJcx6jUGL/yqdQRn0wF2cywgTml/oXolczBdNWtK8VaiPiG3GYjQvqn+o 2nP6PpbVhjzQVUm602Frlbn1PMFvVNOC8NooVZ58zZdxYks6Z27izMe1FffE1SESnSIv 8hUlzaZHHlB3f4fpIe/i7Hh0R2T2QUegwJkQO0zDBOn6RFzgEN6QTVyxYA76UIQPwmkG cF++goI5gC8Z/n8VXcAg5izXAgUDoK6zXSvWcRN9uARLhY+f3oI947J3YiG2RMO28+dE d4yQQwgoEl+7ja4xOW8zDFWSUD2pvRa6wqjNyZg4bF9nprJrfOPYw4TAuKMIVuvyOKYL d/JA== X-Forwarded-Encrypted: i=1; AKwUvBxGNdkQ0PaNlu1zBtwkkc2RRyh3iY52VDOwRe9h17yNjQA+MgPqiFnfheZIsIMgHlMci78xl/qlkeW2AsM=@vger.kernel.org X-Gm-Message-State: AFuF++kVC8x9SI732Q31QNuVCot8kErnPFjaYztqV+Li+JG4nNMrQwLs WaIYWLCEDeDg1m59LViCsGmg6b4gTRiSZEMj0OhhgqChtlBy5pV94ysRGwmAbzJXaRVtrqIL5Km qIg== X-Received: from wmht20.prod.google.com ([2002:a05:600c:41d4:b0:49d:659:548e]) (user=elver job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:198d:b0:49c:fa21:1c7c with SMTP id 5b1f17b1804b1-49d1ed0b452mr135146635e9.17.1789057441100; Thu, 10 Sep 2026 09:24:01 -0700 (PDT) Date: Thu, 10 Sep 2026 16:21:38 +0000 In-Reply-To: <20260910162343.4092060-1-elver@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260910162343.4092060-1-elver@google.com> X-Mailer: git-send-email 2.55.0.1003.g10538fe699-goog Message-ID: <20260910162343.4092060-6-elver@google.com> Subject: [PATCH RFC 05/10] KVM: Refactor kvm_handle_gfn_range() to avoid conditional mmu_lock From: Marco Elver To: elver@google.com Cc: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Vitaly Kuznetsov , Kiryl Shutsemau , Rick Edgecombe , David Hildenbrand , kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Refactor kvm_handle_gfn_range() to check for overlapping memslots upfront via kvm_for_each_memslot_in_gfn_range() instead of tracking found memslots inside the range iteration with a 'found_memslot' flag and conditionally acquiring and releasing mmu_lock. This simplifies the control flow by cleanly decoupling the search for overlapping memslots from the subsequent walk, eliminating the conditional locking and establishing a clean, unconditional lock scope for mmu_lock, which subsequently enables Clang context analysis to validate locking in this function. No functional change intended. Signed-off-by: Marco Elver --- virt/kvm/kvm_main.c | 46 +++++++++++++++++++++++++++++++++------------ 1 file changed, 34 insertions(+), 12 deletions(-) diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index f7bfa2d32507..f86e690a1798 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -2485,14 +2485,13 @@ bool kvm_range_has_memory_attributes(struct kvm *kv= m, gfn_t start, gfn_t end, return true; } =20 -static __always_inline void kvm_handle_gfn_range(struct kvm *kvm, - struct kvm_mmu_notifier_range *range) +static __always_inline bool __kvm_handle_gfn_range_walk(struct kvm *kvm, + struct kvm_mmu_notifier_range *range) { struct kvm_gfn_range gfn_range; struct kvm_memory_slot *slot; struct kvm_memslots *slots; struct kvm_memslot_iter iter; - bool found_memslot =3D false; bool ret =3D false; int i; =20 @@ -2519,22 +2518,45 @@ static __always_inline void kvm_handle_gfn_range(st= ruct kvm *kvm, if (gfn_range.start >=3D gfn_range.end) continue; =20 - if (!found_memslot) { - found_memslot =3D true; - KVM_MMU_LOCK(kvm); - if (!IS_KVM_NULL_FN(range->on_lock)) - range->on_lock(kvm); - } - ret |=3D range->handler(kvm, &gfn_range); } } =20 + return ret; +} + +static __always_inline void kvm_handle_gfn_range(struct kvm *kvm, + struct kvm_mmu_notifier_range *range) +{ + struct kvm_memslot_iter iter; + struct kvm_memslots *slots; + bool found_memslot =3D false; + bool ret; + int i; + + for (i =3D 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) { + slots =3D __kvm_memslots(kvm, i); + kvm_for_each_memslot_in_gfn_range(&iter, slots, range->start, range->end= ) { + found_memslot =3D true; + break; + } + if (found_memslot) + break; + } + + if (!found_memslot) + return; + + KVM_MMU_LOCK(kvm); + if (!IS_KVM_NULL_FN(range->on_lock)) + range->on_lock(kvm); + + ret =3D __kvm_handle_gfn_range_walk(kvm, range); + if (range->flush_on_ret && ret) kvm_flush_remote_tlbs(kvm); =20 - if (found_memslot) - KVM_MMU_UNLOCK(kvm); + KVM_MMU_UNLOCK(kvm); } =20 static bool kvm_pre_set_memory_attributes(struct kvm *kvm, --=20 2.55.0.1003.g10538fe699-goog From nobody Fri Sep 25 16:02:05 2026 Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3652951E45E for ; Thu, 10 Sep 2026 16:24:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.70 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057450; cv=none; b=JqMNM3Te/sDNBcsxA6IrTOrCL0yWejgzaR7ns45e4qDFhYStLs63YsPCxx8uIKQFH43njWEhNIpsQ4XSnOUUugIBbyHy8wVu7XWlVbPncuq/GoZUy+68B3x9fZ33atRtd7WI2mKSgorFRt7gf0wWBQlHzbM5Yop481XKZmAcmh8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057450; c=relaxed/simple; bh=ikMfwhMtZU9paqXK7lBJlN27t/Eb/3ljigo2KEZ0wtw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=NJT58ibBuhBwV+pcAwFRjSqpFvsTqcUh4aleALfPVee/3LGhwcLuuZcqHO3ByL8R3oU87QqsVLDSTVUcxTrLrI5h3VuJF1h3iYGZSTNOykYjWbd08N3st6vGQZ4UiolwA88aAhZAyxjiIvjfAJuXGNJW1P1jVP7iImgNjFks1RM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--elver.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=hqfAG1IL; arc=none smtp.client-ip=209.85.128.70 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--elver.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="hqfAG1IL" Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-49cf9df1eadso68585e9.3 for ; Thu, 10 Sep 2026 09:24:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1789057443; x=1789662243; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=eg/KuATtTStoF6vYmBtiBei5rDkCm8qO1Fh60wlX3ho=; b=hqfAG1ILjIb05Zds96iPWj62EPV8ggngygUkJHEE8AXLQOHqI5QgOQbyzNIWU+oz3M pMAYNLxJQm85GQ7VxhLnmYFGMkhAVrhUPlRpNv28Al3tGkvlu1gTenkXGWEgKKZNjPt+ rXBEluwGK4usgpQ3twmsceseFtnORrBSkK8PygzA4L11HRMvdpeUVorxQB4U0JbD5rYv 2nCo5wWoXZJtSOVzexg75+ZQaVrDtQK3jWLvjzl4hGnyrk50sTaqrT8dHssC4+h5JaFd oFHPCC/fap1L8MGvkv2YWlhJZ6Rksvr9nZbGRNl3LXPR+qpXUF19isBal+9N1iWmYPFN K7HQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789057443; x=1789662243; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=eg/KuATtTStoF6vYmBtiBei5rDkCm8qO1Fh60wlX3ho=; b=JQoEYN5ZV3eZ3qTy9gNR9gTiWKx9a2M9yDi0TkRfW8yULpB/Fbr0fczyR+Izuenbtf lR02Zy7MtfZ3tEMT8ruHF3IUYBn9uosmJGrBkRQnfEilqW2M7eP1z9cDCI7LhHZnYjWs pB/uUq9xVlLLIGAolyKMGFTwgLR3mlGxgHDk28AEz171ckOqUZG4ZTLwATn1XWsMK5Cf u4IBurBKho2QrsuKsFres221irVpc3ATebEayO/7AdsmLvK6Fnbw9quyt4zYZURNZYy/ ellnH+iQvyCwgkqKKzQ988KHpx/H85Liqd1f2nzWCu/qBAJuxMqjcBNDlfitOPk9cUAu 9jBA== X-Forwarded-Encrypted: i=1; AKwUvByQBYk3Zr3zm/Hl3+Nl6vsv370UnJ/ViySbMm5FApulq05ZRuYJt/f3ivgU9Mqpb8069Z9Ju/g+bBy7iNA=@vger.kernel.org X-Gm-Message-State: AFuF++mI4ByDvspcXOa/qpTz5VIdTbiLSxy9cmyyKf34o3sXJdO5wP4t a5R3vYJXvAvs8VHxsRkrITfAW5i8bEpMl3nGQKlduRK6DdV9VI7eEFut2ExBguDo7dyVICf7la5 eew== X-Received: from wmnb18.prod.google.com ([2002:a05:600c:6d2:b0:49b:2006:9521]) (user=elver job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:4684:b0:49c:fc6c:be14 with SMTP id 5b1f17b1804b1-49cfc6cc10bmr386808385e9.26.1789057442334; Thu, 10 Sep 2026 09:24:02 -0700 (PDT) Date: Thu, 10 Sep 2026 16:21:39 +0000 In-Reply-To: <20260910162343.4092060-1-elver@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260910162343.4092060-1-elver@google.com> X-Mailer: git-send-email 2.55.0.1003.g10538fe699-goog Message-ID: <20260910162343.4092060-7-elver@google.com> Subject: [PATCH RFC 06/10] KVM: Add basic lock context annotations From: Marco Elver To: elver@google.com Cc: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Vitaly Kuznetsov , Kiryl Shutsemau , Rick Edgecombe , David Hildenbrand , kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add basic lock context annotations across common KVM sources. The only opted-out functions are kvm_trylock_all_vcpus(), kvm_lock_all_vcpus(), and kvm_unlock_all_vcpus(), which dynamically acquire and release an arbitrary number of vCPU mutexes that cannot be represented statically. No functional change intended. Signed-off-by: Marco Elver --- include/linux/kvm_host.h | 202 +++++++++++++++++++++++++++------------ virt/kvm/dirty_ring.c | 2 + virt/kvm/eventfd.c | 5 + virt/kvm/guest_memfd.c | 1 + virt/kvm/kvm_main.c | 40 +++++++- virt/kvm/pfncache.c | 3 + 6 files changed, 189 insertions(+), 64 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 5ed8260ef01f..12f241304228 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -220,18 +220,6 @@ enum kvm_bus { KVM_NR_BUSES }; =20 -int kvm_io_bus_write(struct kvm_vcpu *vcpu, enum kvm_bus bus_idx, gpa_t ad= dr, - int len, const void *val); -int kvm_io_bus_write_cookie(struct kvm_vcpu *vcpu, enum kvm_bus bus_idx, - gpa_t addr, int len, const void *val, long cookie); -int kvm_io_bus_read(struct kvm_vcpu *vcpu, enum kvm_bus bus_idx, gpa_t add= r, - int len, void *val); -int kvm_io_bus_register_dev(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t a= ddr, - int len, struct kvm_io_device *dev); -int kvm_io_bus_unregister_dev(struct kvm *kvm, enum kvm_bus bus_idx, - struct kvm_io_device *dev); -struct kvm_io_device *kvm_io_bus_get_dev(struct kvm *kvm, enum kvm_bus bus= _idx, - gpa_t addr); =20 #ifdef CONFIG_KVM_ASYNC_PF struct kvm_async_pf { @@ -245,12 +233,6 @@ struct kvm_async_pf { bool wakeup_all; bool notpresent_injected; }; - -void kvm_clear_async_pf_completion_queue(struct kvm_vcpu *vcpu); -void kvm_check_async_pf_completion(struct kvm_vcpu *vcpu); -bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, - unsigned long hva, struct kvm_arch_async_pf *arch); -int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu); #endif =20 union kvm_mmu_notifier_arg { @@ -905,6 +887,25 @@ struct kvm { #define vcpu_err(vcpu, fmt, ...) \ kvm_err("vcpu%i " fmt, (vcpu)->vcpu_id, ## __VA_ARGS__) =20 +int kvm_io_bus_write(struct kvm_vcpu *vcpu, enum kvm_bus bus_idx, gpa_t ad= dr, + int len, const void *val) + __must_hold_shared(&vcpu->kvm->srcu); +int kvm_io_bus_write_cookie(struct kvm_vcpu *vcpu, enum kvm_bus bus_idx, + gpa_t addr, int len, const void *val, long cookie) + __must_hold_shared(&vcpu->kvm->srcu); +int kvm_io_bus_read(struct kvm_vcpu *vcpu, enum kvm_bus bus_idx, gpa_t add= r, + int len, void *val) + __must_hold_shared(&vcpu->kvm->srcu); +int kvm_io_bus_register_dev(struct kvm *kvm, enum kvm_bus bus_idx, gpa_t a= ddr, + int len, struct kvm_io_device *dev) + __must_hold(&kvm->slots_lock); +int kvm_io_bus_unregister_dev(struct kvm *kvm, enum kvm_bus bus_idx, + struct kvm_io_device *dev) + __must_hold(&kvm->slots_lock); +struct kvm_io_device *kvm_io_bus_get_dev(struct kvm *kvm, enum kvm_bus bus= _idx, + gpa_t addr) + __must_hold_shared(&kvm->srcu); + static inline void kvm_vm_dead(struct kvm *kvm) { kvm->vm_dead =3D true; @@ -956,6 +957,7 @@ static inline void kvm_vm_bugged(struct kvm *kvm) }) =20 static inline void kvm_vcpu_srcu_read_lock(struct kvm_vcpu *vcpu) + __acquires_shared(&vcpu->kvm->srcu) { #ifdef CONFIG_PROVE_RCU WARN_ONCE(vcpu->srcu_depth++, @@ -965,6 +967,7 @@ static inline void kvm_vcpu_srcu_read_lock(struct kvm_v= cpu *vcpu) } =20 static inline void kvm_vcpu_srcu_read_unlock(struct kvm_vcpu *vcpu) + __releases_shared(&vcpu->kvm->srcu) { srcu_read_unlock(&vcpu->kvm->srcu, vcpu->____srcu_idx); =20 @@ -985,6 +988,7 @@ static inline bool kvm_dirty_log_manual_protect_and_ini= t_set(struct kvm *kvm) * registrations. */ static inline struct kvm_io_bus *kvm_get_bus(struct kvm *kvm, enum kvm_bus= idx) + __must_hold(&kvm->slots_lock) { return rcu_dereference_protected(kvm->buses[idx], lockdep_is_held(&kvm->slots_lock)); @@ -1047,9 +1051,9 @@ static inline bool kvm_is_vcpu_creation_in_progress(s= truct kvm *kvm) =20 void kvm_destroy_vcpus(struct kvm *kvm); =20 -int kvm_trylock_all_vcpus(struct kvm *kvm); -int kvm_lock_all_vcpus(struct kvm *kvm); -void kvm_unlock_all_vcpus(struct kvm *kvm); +int kvm_trylock_all_vcpus(struct kvm *kvm) __must_hold(&kvm->lock); +int kvm_lock_all_vcpus(struct kvm *kvm) __must_hold(&kvm->lock); +void kvm_unlock_all_vcpus(struct kvm *kvm) __must_hold(&kvm->lock); =20 void vcpu_load(struct kvm_vcpu *vcpu); void vcpu_put(struct kvm_vcpu *vcpu); @@ -1085,6 +1089,7 @@ bool file_is_kvm(struct file *file); void kvm_put_kvm_no_destroy(struct kvm *kvm); =20 static inline struct kvm_memslots *__kvm_memslots(struct kvm *kvm, int as_= id) + __must_hold_shared(&kvm->srcu) { as_id =3D array_index_nospec(as_id, KVM_MAX_NR_ADDRESS_SPACES); return srcu_dereference_check(kvm->memslots[as_id], &kvm->srcu, @@ -1094,11 +1099,13 @@ static inline struct kvm_memslots *__kvm_memslots(s= truct kvm *kvm, int as_id) } =20 static inline struct kvm_memslots *kvm_memslots(struct kvm *kvm) + __must_hold_shared(&kvm->srcu) { return __kvm_memslots(kvm, 0); } =20 static inline struct kvm_memslots *kvm_vcpu_memslots(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { int as_id =3D kvm_arch_vcpu_memslots_id(vcpu); =20 @@ -1110,7 +1117,8 @@ static inline bool kvm_memslots_empty(struct kvm_mems= lots *slots) return RB_EMPTY_ROOT(&slots->gfn_tree); } =20 -bool kvm_are_all_memslots_empty(struct kvm *kvm); +bool kvm_are_all_memslots_empty(struct kvm *kvm) + __must_hold(&kvm->slots_lock); =20 #define kvm_for_each_memslot(memslot, bkt, slots) \ hash_for_each(slots->id_hash, bkt, memslot, id_node[slots->node_idx]) \ @@ -1225,9 +1233,10 @@ static inline bool kvm_memslot_iter_is_valid(struct = kvm_memslot_iter *iter, gfn_ kvm_memslot_iter_is_valid(iter, end); \ kvm_memslot_iter_next(iter)) =20 -struct kvm_memory_slot *gfn_to_memslot(struct kvm *kvm, gfn_t gfn); -struct kvm_memslots *kvm_vcpu_memslots(struct kvm_vcpu *vcpu); -struct kvm_memory_slot *kvm_vcpu_gfn_to_memslot(struct kvm_vcpu *vcpu, gfn= _t gfn); +struct kvm_memory_slot *gfn_to_memslot(struct kvm *kvm, gfn_t gfn) + __must_hold_shared(&kvm->srcu); +struct kvm_memory_slot *kvm_vcpu_gfn_to_memslot(struct kvm_vcpu *vcpu, gfn= _t gfn) + __must_hold_shared(&vcpu->kvm->srcu); =20 /* * KVM_SET_USER_MEMORY_REGION ioctl allows the following operations: @@ -1248,7 +1257,8 @@ enum kvm_mr_change { }; =20 int kvm_set_internal_memslot(struct kvm *kvm, - const struct kvm_userspace_memory_region2 *mem); + const struct kvm_userspace_memory_region2 *mem) + __must_hold(&kvm->slots_lock); void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot); void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen); int kvm_arch_prepare_memory_region(struct kvm *kvm, @@ -1268,14 +1278,18 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, int kvm_prefetch_pages(struct kvm_memory_slot *slot, gfn_t gfn, struct page **pages, int nr_pages); =20 -struct page *__gfn_to_page(struct kvm *kvm, gfn_t gfn, bool write); +struct page *__gfn_to_page(struct kvm *kvm, gfn_t gfn, bool write) + __must_hold_shared(&kvm->srcu); static inline struct page *gfn_to_page(struct kvm *kvm, gfn_t gfn) + __must_hold_shared(&kvm->srcu) { return __gfn_to_page(kvm, gfn, true); } =20 -unsigned long gfn_to_hva(struct kvm *kvm, gfn_t gfn); -unsigned long gfn_to_hva_prot(struct kvm *kvm, gfn_t gfn, bool *writable); +unsigned long gfn_to_hva(struct kvm *kvm, gfn_t gfn) + __must_hold_shared(&kvm->srcu); +unsigned long gfn_to_hva_prot(struct kvm *kvm, gfn_t gfn, bool *writable) + __must_hold_shared(&kvm->srcu); unsigned long gfn_to_hva_memslot(struct kvm_memory_slot *slot, gfn_t gfn); unsigned long gfn_to_hva_memslot_prot(struct kvm_memory_slot *slot, gfn_t = gfn, bool *writable); @@ -1324,30 +1338,40 @@ kvm_pfn_t __kvm_faultin_pfn(const struct kvm_memory= _slot *slot, gfn_t gfn, static inline kvm_pfn_t kvm_faultin_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, bool write, bool *writable, struct page **refcounted_page) + __must_hold_shared(&vcpu->kvm->srcu) { return __kvm_faultin_pfn(kvm_vcpu_gfn_to_memslot(vcpu, gfn), gfn, write ? FOLL_WRITE : 0, writable, refcounted_page); } =20 int kvm_read_guest_page(struct kvm *kvm, gfn_t gfn, void *data, int offset, - int len); -int kvm_read_guest(struct kvm *kvm, gpa_t gpa, void *data, unsigned long l= en); + int len) + __must_hold_shared(&kvm->srcu); +int kvm_read_guest(struct kvm *kvm, gpa_t gpa, void *data, unsigned long l= en) + __must_hold_shared(&kvm->srcu); int kvm_read_guest_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc, - void *data, unsigned long len); + void *data, unsigned long len) + __must_hold_shared(&kvm->srcu); int kvm_read_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache = *ghc, void *data, unsigned int offset, - unsigned long len); + unsigned long len) + __must_hold_shared(&kvm->srcu); int kvm_write_guest_page(struct kvm *kvm, gfn_t gfn, const void *data, - int offset, int len); + int offset, int len) + __must_hold_shared(&kvm->srcu); int kvm_write_guest(struct kvm *kvm, gpa_t gpa, const void *data, - unsigned long len); + unsigned long len) + __must_hold_shared(&kvm->srcu); int kvm_write_guest_cached(struct kvm *kvm, struct gfn_to_hva_cache *ghc, - void *data, unsigned long len); + void *data, unsigned long len) + __must_hold_shared(&kvm->srcu); int kvm_write_guest_offset_cached(struct kvm *kvm, struct gfn_to_hva_cache= *ghc, void *data, unsigned int offset, - unsigned long len); + unsigned long len) + __must_hold_shared(&kvm->srcu); int kvm_gfn_to_hva_cache_init(struct kvm *kvm, struct gfn_to_hva_cache *gh= c, - gpa_t gpa, unsigned long len); + gpa_t gpa, unsigned long len) + __must_hold_shared(&kvm->srcu); =20 #define __kvm_get_guest(kvm, gfn, offset, v) \ ({ \ @@ -1391,32 +1415,42 @@ int kvm_gfn_to_hva_cache_init(struct kvm *kvm, stru= ct gfn_to_hva_cache *ghc, offset_in_page(__gpa), v); \ }) =20 -int kvm_clear_guest(struct kvm *kvm, gpa_t gpa, unsigned long len); -bool kvm_is_visible_gfn(struct kvm *kvm, gfn_t gfn); -bool kvm_vcpu_is_visible_gfn(struct kvm_vcpu *vcpu, gfn_t gfn); -unsigned long kvm_host_page_size(struct kvm_vcpu *vcpu, gfn_t gfn); +int kvm_clear_guest(struct kvm *kvm, gpa_t gpa, unsigned long len) + __must_hold_shared(&kvm->srcu); +bool kvm_is_visible_gfn(struct kvm *kvm, gfn_t gfn) + __must_hold_shared(&kvm->srcu); +bool kvm_vcpu_is_visible_gfn(struct kvm_vcpu *vcpu, gfn_t gfn) + __must_hold_shared(&vcpu->kvm->srcu); +unsigned long kvm_host_page_size(struct kvm_vcpu *vcpu, gfn_t gfn) + __must_hold_shared(&vcpu->kvm->srcu); void mark_page_dirty_in_slot(struct kvm *kvm, const struct kvm_memory_slot= *memslot, gfn_t gfn); -void mark_page_dirty(struct kvm *kvm, gfn_t gfn); -void kvm_vcpu_mark_page_dirty(struct kvm_vcpu *vcpu, gfn_t gfn); +void mark_page_dirty(struct kvm *kvm, gfn_t gfn) + __must_hold_shared(&kvm->srcu); +void kvm_vcpu_mark_page_dirty(struct kvm_vcpu *vcpu, gfn_t gfn) + __must_hold_shared(&vcpu->kvm->srcu); =20 int __kvm_vcpu_map(struct kvm_vcpu *vcpu, gfn_t gfn, struct kvm_host_map *= map, - bool writable); + bool writable) + __must_hold_shared(&vcpu->kvm->srcu); void kvm_vcpu_unmap(struct kvm_vcpu *vcpu, struct kvm_host_map *map); =20 static inline int kvm_vcpu_map(struct kvm_vcpu *vcpu, gfn_t gfn, struct kvm_host_map *map) + __must_hold_shared(&vcpu->kvm->srcu) { return __kvm_vcpu_map(vcpu, gfn, map, true); } =20 static inline int kvm_vcpu_map_readonly(struct kvm_vcpu *vcpu, gfn_t gfn, struct kvm_host_map *map) + __must_hold_shared(&vcpu->kvm->srcu) { return __kvm_vcpu_map(vcpu, gfn, map, false); } =20 static inline void kvm_vcpu_map_mark_dirty(struct kvm_vcpu *vcpu, struct kvm_host_map *map) + __must_hold_shared(&vcpu->kvm->srcu) { if (kvm_vcpu_mapped(map)) kvm_vcpu_mark_page_dirty(vcpu, map->gfn); @@ -1441,18 +1475,25 @@ DEFINE_CLASS(kvm_vcpu_map_local##ro, kvm_vcpu_local= _map_t, \ DEFINE_VCPU_MAP_CLASS(); DEFINE_VCPU_MAP_CLASS(_readonly); =20 -unsigned long kvm_vcpu_gfn_to_hva(struct kvm_vcpu *vcpu, gfn_t gfn); -unsigned long kvm_vcpu_gfn_to_hva_prot(struct kvm_vcpu *vcpu, gfn_t gfn, b= ool *writable); +unsigned long kvm_vcpu_gfn_to_hva(struct kvm_vcpu *vcpu, gfn_t gfn) + __must_hold_shared(&vcpu->kvm->srcu); +unsigned long kvm_vcpu_gfn_to_hva_prot(struct kvm_vcpu *vcpu, gfn_t gfn, b= ool *writable) + __must_hold_shared(&vcpu->kvm->srcu); int kvm_vcpu_read_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, void *data,= int offset, - int len); + int len) + __must_hold_shared(&vcpu->kvm->srcu); int kvm_vcpu_read_guest_atomic(struct kvm_vcpu *vcpu, gpa_t gpa, void *dat= a, - unsigned long len); + unsigned long len) + __must_hold_shared(&vcpu->kvm->srcu); int kvm_vcpu_read_guest(struct kvm_vcpu *vcpu, gpa_t gpa, void *data, - unsigned long len); + unsigned long len) + __must_hold_shared(&vcpu->kvm->srcu); int kvm_vcpu_write_guest_page(struct kvm_vcpu *vcpu, gfn_t gfn, const void= *data, - int offset, int len); + int offset, int len) + __must_hold_shared(&vcpu->kvm->srcu); int kvm_vcpu_write_guest(struct kvm_vcpu *vcpu, gpa_t gpa, const void *dat= a, - unsigned long len); + unsigned long len) + __must_hold_shared(&vcpu->kvm->srcu); =20 /** * kvm_gpc_init - initialize gfn_to_pfn_cache. @@ -1482,7 +1523,8 @@ void kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struc= t kvm *kvm); * invalidations to be processed. Callers are required to use kvm_gpc_che= ck() * to ensure that the cache is valid before accessing the target page. */ -int kvm_gpc_activate(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned lon= g len); +int kvm_gpc_activate(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsigned lon= g len) + __must_hold_shared(&gpc->kvm->srcu); =20 /** * kvm_gpc_activate_hva - prepare a cached kernel mapping and HPA for a gi= ven HVA. @@ -1498,7 +1540,8 @@ int kvm_gpc_activate(struct gfn_to_pfn_cache *gpc, gp= a_t gpa, unsigned long len) * The semantics of this function are the same as those of kvm_gpc_activat= e(). It * merely bypasses a layer of address translation. */ -int kvm_gpc_activate_hva(struct gfn_to_pfn_cache *gpc, unsigned long hva, = unsigned long len); +int kvm_gpc_activate_hva(struct gfn_to_pfn_cache *gpc, unsigned long hva, = unsigned long len) + __must_hold_shared(&gpc->kvm->srcu); =20 /** * kvm_gpc_check - check validity of a gfn_to_pfn_cache. @@ -1516,7 +1559,8 @@ int kvm_gpc_activate_hva(struct gfn_to_pfn_cache *gpc= , unsigned long hva, unsign * Callers in IN_GUEST_MODE may do so without locking, although they should * still hold a read lock on kvm->scru for the memslot checks. */ -bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len); +bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len) + __must_hold_shared(&gpc->kvm->srcu); =20 /** * kvm_gpc_refresh - update a previously initialized cache. @@ -1534,7 +1578,8 @@ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsi= gned long len); * still lock and check the cache status, as this function does not return * with the lock still held to permit access. */ -int kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, unsigned long len); +int kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, unsigned long len) + __must_hold_shared(&gpc->kvm->srcu); =20 /** * kvm_gpc_deactivate - deactivate and unlink a gfn_to_pfn_cache. @@ -1652,6 +1697,31 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu); void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu); void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu); =20 +#ifdef CONFIG_KVM_ASYNC_PF +void kvm_clear_async_pf_completion_queue(struct kvm_vcpu *vcpu); +void kvm_check_async_pf_completion(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu); +bool kvm_setup_async_pf(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, + unsigned long hva, struct kvm_arch_async_pf *arch) + __must_hold_shared(&vcpu->kvm->srcu); +int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu); + +bool kvm_arch_async_page_not_present(struct kvm_vcpu *vcpu, + struct kvm_async_pf *work) + __must_hold_shared(&vcpu->kvm->srcu); +void kvm_arch_async_page_present(struct kvm_vcpu *vcpu, + struct kvm_async_pf *work) + __must_hold_shared(&vcpu->kvm->srcu); +void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, + struct kvm_async_pf *work) + __must_hold_shared(&vcpu->kvm->srcu); +#ifndef CONFIG_KVM_ASYNC_PF_SYNC +void kvm_arch_async_page_present_queued(struct kvm_vcpu *vcpu); +#endif +bool kvm_arch_can_dequeue_async_page_present(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu); +#endif + #ifdef CONFIG_HAVE_KVM_PM_NOTIFIER int kvm_arch_pm_notifier(struct kvm *kvm, unsigned long state); #endif @@ -1824,8 +1894,10 @@ struct kvm_irq_ack_notifier { }; =20 int kvm_irq_map_gsi(struct kvm *kvm, - struct kvm_kernel_irq_routing_entry *entries, int gsi); -int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned irqchip, unsigned pin); + struct kvm_kernel_irq_routing_entry *entries, int gsi) + __must_hold_shared(&kvm->irq_srcu); +int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned irqchip, unsigned pin) + __must_hold_shared(&kvm->irq_srcu); =20 int kvm_set_irq(struct kvm *kvm, int irq_source_id, u32 irq, int level, bool line_status); @@ -1835,7 +1907,8 @@ int kvm_arch_set_irq_inatomic(struct kvm_kernel_irq_r= outing_entry *e, struct kvm *kvm, int irq_source_id, int level, bool line_status); bool kvm_irq_has_notifier(struct kvm *kvm, unsigned irqchip, unsigned pin); -void kvm_notify_acked_gsi(struct kvm *kvm, int gsi); +void kvm_notify_acked_gsi(struct kvm *kvm, int gsi) + __must_hold_shared(&kvm->irq_srcu); void kvm_notify_acked_irq(struct kvm *kvm, unsigned irqchip, unsigned pin); void kvm_register_irq_ack_notifier(struct kvm *kvm, struct kvm_irq_ack_notifier *kian); @@ -1937,6 +2010,7 @@ __gfn_to_hva_memslot(const struct kvm_memory_slot *sl= ot, gfn_t gfn) } =20 static inline int memslot_id(struct kvm *kvm, gfn_t gfn) + __must_hold_shared(&kvm->srcu) { return gfn_to_memslot(kvm, gfn)->id; } @@ -1965,6 +2039,7 @@ static inline hpa_t pfn_to_hpa(kvm_pfn_t pfn) } =20 static inline bool kvm_is_gpa_in_memslot(struct kvm *kvm, gpa_t gpa) + __must_hold_shared(&kvm->srcu) { unsigned long hva =3D gfn_to_hva(kvm, gpa_to_gfn(gpa)); =20 @@ -2265,7 +2340,8 @@ void kvm_irqfd_release(struct kvm *kvm); bool kvm_notify_irqfd_resampler(struct kvm *kvm, unsigned int irqchip, unsigned int pin); -void kvm_irq_routing_update(struct kvm *); +void kvm_irq_routing_update(struct kvm *kvm) + __must_hold(&kvm->irq_lock); #else static inline int kvm_irqfd(struct kvm *kvm, struct kvm_irqfd *args) { @@ -2282,7 +2358,8 @@ static inline bool kvm_notify_irqfd_resampler(struct = kvm *kvm, } #endif /* CONFIG_HAVE_KVM_IRQCHIP */ =20 -void kvm_arch_irq_routing_update(struct kvm *kvm); +void kvm_arch_irq_routing_update(struct kvm *kvm) + __must_hold(&kvm->irq_lock); =20 static inline void __kvm_make_request(int req, struct kvm_vcpu *vcpu) { @@ -2632,7 +2709,8 @@ typedef int (*kvm_gmem_populate_cb)(struct kvm *kvm, = gfn_t gfn, kvm_pfn_t pfn, =20 long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn, void __user *src, long npages, bool may_writeback_src, - kvm_gmem_populate_cb post_populate, void *opaque); + kvm_gmem_populate_cb post_populate, void *opaque) + __must_hold(&kvm->slots_lock); #endif =20 #ifdef CONFIG_HAVE_KVM_ARCH_GMEM_RECLAIM diff --git a/virt/kvm/dirty_ring.c b/virt/kvm/dirty_ring.c index 572b854edf74..11a6aba98b5e 100644 --- a/virt/kvm/dirty_ring.c +++ b/virt/kvm/dirty_ring.c @@ -51,6 +51,7 @@ static bool kvm_dirty_ring_full(struct kvm_dirty_ring *ri= ng) } =20 static void kvm_reset_dirty_gfn(struct kvm *kvm, u32 slot, u64 offset, u64= mask) + __must_hold(&kvm->slots_lock) { struct kvm_memory_slot *memslot; int as_id, id; @@ -61,6 +62,7 @@ static void kvm_reset_dirty_gfn(struct kvm *kvm, u32 slot= , u64 offset, u64 mask) if (as_id >=3D kvm_arch_nr_memslot_as_ids(kvm) || id >=3D KVM_USER_MEM_SL= OTS) return; =20 + __assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */ memslot =3D id_to_memslot(__kvm_memslots(kvm, as_id), id); =20 if (!memslot || offset >=3D memslot->npages || diff --git a/virt/kvm/eventfd.c b/virt/kvm/eventfd.c index 93ad2ebc963f..6701390336f6 100644 --- a/virt/kvm/eventfd.c +++ b/virt/kvm/eventfd.c @@ -271,6 +271,7 @@ irqfd_wakeup(wait_queue_entry_t *wait, unsigned mode, i= nt sync, void *key) } =20 static void irqfd_update(struct kvm *kvm, struct kvm_kernel_irqfd *irqfd) + __must_hold_shared(&kvm->irq_srcu) { struct kvm_kernel_irq_routing_entry *e; struct kvm_kernel_irq_routing_entry entries[KVM_NR_IRQCHIPS]; @@ -300,6 +301,7 @@ struct kvm_irqfd_pt { =20 static void kvm_irqfd_register(struct file *file, wait_queue_head_t *wqh, poll_table *pt) + __must_hold_shared(&container_of(pt, struct kvm_irqfd_pt, pt)->kvm->irq_s= rcu) { struct kvm_irqfd_pt *p =3D container_of(pt, struct kvm_irqfd_pt, pt); struct kvm_kernel_irqfd *irqfd =3D p->irqfd; @@ -660,6 +662,9 @@ void kvm_irq_routing_update(struct kvm *kvm) { struct kvm_kernel_irqfd *irqfd; =20 + /* Update-side mutex kvm->irq_lock is held. */ + __assume_shared_ctx_lock(&kvm->irq_srcu); + spin_lock_irq(&kvm->irqfds.lock); =20 list_for_each_entry(irqfd, &kvm->irqfds.items, list) { diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 896f3b076562..bc97e566559e 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -836,6 +836,7 @@ long kvm_gmem_populate(struct kvm *kvm, gfn_t start_gfn= , void __user *src, long i; =20 lockdep_assert_held(&kvm->slots_lock); + __assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */ =20 if (WARN_ON_ONCE(npages <=3D 0)) return -EINVAL; diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index f86e690a1798..2f22d5439d39 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -560,6 +560,7 @@ static void kvm_null_fn(void) =20 static __always_inline bool __kvm_handle_hva_range_walk(struct kvm *kvm, const struct kvm_mmu_notifier_range *range) + __must_hold_shared(&kvm->srcu) { struct kvm_gfn_range gfn_range; struct kvm_memory_slot *slot; @@ -1379,6 +1380,7 @@ static int kvm_vm_release(struct inode *inode, struct= file *filp) } =20 int kvm_trylock_all_vcpus(struct kvm *kvm) + __context_unsafe(/* multi-lock acquisition */) { struct kvm_vcpu *vcpu; unsigned long i, j; @@ -1401,6 +1403,7 @@ int kvm_trylock_all_vcpus(struct kvm *kvm) EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_trylock_all_vcpus); =20 int kvm_lock_all_vcpus(struct kvm *kvm) + __context_unsafe(/* multi-lock acquisition */) { struct kvm_vcpu *vcpu; unsigned long i, j; @@ -1426,6 +1429,7 @@ int kvm_lock_all_vcpus(struct kvm *kvm) EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_lock_all_vcpus); =20 void kvm_unlock_all_vcpus(struct kvm *kvm) + __context_unsafe(/* multi-lock release */) { struct kvm_vcpu *vcpu; unsigned long i; @@ -1453,7 +1457,9 @@ static int kvm_alloc_dirty_bitmap(struct kvm_memory_s= lot *memslot) } =20 static struct kvm_memslots *kvm_get_inactive_memslots(struct kvm *kvm, int= as_id) + __must_hold(&kvm->slots_lock) { + __assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */ struct kvm_memslots *active =3D __kvm_memslots(kvm, as_id); int node_idx_inactive =3D active->node_idx ^ 1; =20 @@ -1535,6 +1541,7 @@ static void kvm_replace_gfn_node(struct kvm_memslots = *slots, static void kvm_replace_memslot(struct kvm *kvm, struct kvm_memory_slot *old, struct kvm_memory_slot *new) + __must_hold(&kvm->slots_lock) { int as_id =3D kvm_memslots_get_as_id(old, new); struct kvm_memslots *slots =3D kvm_get_inactive_memslots(kvm, as_id); @@ -1621,9 +1628,11 @@ static int check_memory_region_flags(struct kvm *kvm, } =20 static void kvm_swap_active_memslots(struct kvm *kvm, int as_id) + __must_hold(&kvm->slots_lock) + __releases(&kvm->slots_arch_lock) { struct kvm_memslots *slots =3D kvm_get_inactive_memslots(kvm, as_id); - + __assume_shared_ctx_lock(&kvm->srcu); /* update-side locks are held */ /* Grab the generation from the activate memslots. */ u64 gen =3D __kvm_memslots(kvm, as_id)->generation; =20 @@ -1796,6 +1805,8 @@ static void kvm_commit_memory_region(struct kvm *kvm, static void kvm_activate_memslot(struct kvm *kvm, struct kvm_memory_slot *old, struct kvm_memory_slot *new) + __must_hold(&kvm->slots_lock) + __releases(&kvm->slots_arch_lock) { int as_id =3D kvm_memslots_get_as_id(old, new); =20 @@ -1821,6 +1832,8 @@ static void kvm_copy_memslot(struct kvm_memory_slot *= dest, static void kvm_invalidate_memslot(struct kvm *kvm, struct kvm_memory_slot *old, struct kvm_memory_slot *invalid_slot) + __must_hold(&kvm->slots_lock) + __must_hold(&kvm->slots_arch_lock) { /* * Mark the current slot INVALID. As with all memslot modifications, @@ -1862,6 +1875,8 @@ static void kvm_invalidate_memslot(struct kvm *kvm, =20 static void kvm_create_memslot(struct kvm *kvm, struct kvm_memory_slot *new) + __must_hold(&kvm->slots_lock) + __releases(&kvm->slots_arch_lock) { /* Add the new memslot to the inactive set and activate. */ kvm_replace_memslot(kvm, NULL, new); @@ -1871,6 +1886,8 @@ static void kvm_create_memslot(struct kvm *kvm, static void kvm_delete_memslot(struct kvm *kvm, struct kvm_memory_slot *old, struct kvm_memory_slot *invalid_slot) + __must_hold(&kvm->slots_lock) + __releases(&kvm->slots_arch_lock) { /* * Remove the old memslot (in the inactive memslots) by passing NULL as @@ -1884,6 +1901,8 @@ static void kvm_move_memslot(struct kvm *kvm, struct kvm_memory_slot *old, struct kvm_memory_slot *new, struct kvm_memory_slot *invalid_slot) + __must_hold(&kvm->slots_lock) + __releases(&kvm->slots_arch_lock) { /* * Replace the old memslot in the inactive slots, and then swap slots @@ -1896,6 +1915,8 @@ static void kvm_move_memslot(struct kvm *kvm, static void kvm_update_flags_memslot(struct kvm *kvm, struct kvm_memory_slot *old, struct kvm_memory_slot *new) + __must_hold(&kvm->slots_lock) + __releases(&kvm->slots_arch_lock) { /* * Similar to the MOVE case, but the slot doesn't need to be zapped as @@ -1910,6 +1931,7 @@ static int kvm_set_memslot(struct kvm *kvm, struct kvm_memory_slot *old, struct kvm_memory_slot *new, enum kvm_mr_change change) + __must_hold(&kvm->slots_lock) { struct kvm_memory_slot *invalid_slot; int r; @@ -2016,6 +2038,7 @@ static bool kvm_check_memslot_overlap(struct kvm_mems= lots *slots, int id, =20 static int kvm_set_memory_region(struct kvm *kvm, const struct kvm_userspace_memory_region2 *mem) + __must_hold(&kvm->slots_lock) { struct kvm_memory_slot *old, *new; struct kvm_memslots *slots; @@ -2026,6 +2049,7 @@ static int kvm_set_memory_region(struct kvm *kvm, int r; =20 lockdep_assert_held(&kvm->slots_lock); + __assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */ =20 r =3D check_memory_region_flags(kvm, mem); if (r) @@ -2242,6 +2266,7 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_dirty_log); * */ static int kvm_get_dirty_log_protect(struct kvm *kvm, struct kvm_dirty_log= *log) + __must_hold(&kvm->slots_lock) { struct kvm_memslots *slots; struct kvm_memory_slot *memslot; @@ -2260,6 +2285,7 @@ static int kvm_get_dirty_log_protect(struct kvm *kvm,= struct kvm_dirty_log *log) if (as_id >=3D kvm_arch_nr_memslot_as_ids(kvm) || id >=3D KVM_USER_MEM_SL= OTS) return -EINVAL; =20 + __assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */ slots =3D __kvm_memslots(kvm, as_id); memslot =3D id_to_memslot(slots, id); if (!memslot || !memslot->dirty_bitmap) @@ -2353,6 +2379,7 @@ static int kvm_vm_ioctl_get_dirty_log(struct kvm *kvm, */ static int kvm_clear_dirty_log_protect(struct kvm *kvm, struct kvm_clear_dirty_log *log) + __must_hold(&kvm->slots_lock) { struct kvm_memslots *slots; struct kvm_memory_slot *memslot; @@ -2375,6 +2402,7 @@ static int kvm_clear_dirty_log_protect(struct kvm *kv= m, if (log->first_page & 63) return -EINVAL; =20 + __assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */ slots =3D __kvm_memslots(kvm, as_id); memslot =3D id_to_memslot(slots, id); if (!memslot || !memslot->dirty_bitmap) @@ -2487,6 +2515,7 @@ bool kvm_range_has_memory_attributes(struct kvm *kvm,= gfn_t start, gfn_t end, =20 static __always_inline bool __kvm_handle_gfn_range_walk(struct kvm *kvm, struct kvm_mmu_notifier_range *range) + __must_hold(&kvm->slots_lock) { struct kvm_gfn_range gfn_range; struct kvm_memory_slot *slot; @@ -2506,6 +2535,7 @@ static __always_inline bool __kvm_handle_gfn_range_wa= lk(struct kvm *kvm, * if the private flag is being toggled, i.e. all mappings are in play. */ =20 + __assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */ for (i =3D 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) { slots =3D __kvm_memslots(kvm, i); =20 @@ -2527,6 +2557,7 @@ static __always_inline bool __kvm_handle_gfn_range_wa= lk(struct kvm *kvm, =20 static __always_inline void kvm_handle_gfn_range(struct kvm *kvm, struct kvm_mmu_notifier_range *range) + __must_hold(&kvm->slots_lock) { struct kvm_memslot_iter iter; struct kvm_memslots *slots; @@ -2534,6 +2565,7 @@ static __always_inline void kvm_handle_gfn_range(stru= ct kvm *kvm, bool ret; int i; =20 + __assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */ for (i =3D 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) { slots =3D __kvm_memslots(kvm, i); kvm_for_each_memslot_in_gfn_range(&iter, slots, range->start, range->end= ) { @@ -3201,8 +3233,10 @@ void kvm_vcpu_unmap(struct kvm_vcpu *vcpu, struct kv= m_host_map *map) memunmap(map->hva); #endif =20 - if (map->writable) + if (map->writable) { + __assume_shared_ctx_lock(&vcpu->kvm->srcu); /* srcu held or VM being des= troyed */ kvm_vcpu_mark_page_dirty(vcpu, map->gfn); + } =20 if (map->pinned_page) { if (map->writable) @@ -5075,6 +5109,7 @@ bool kvm_are_all_memslots_empty(struct kvm *kvm) int i; =20 lockdep_assert_held(&kvm->slots_lock); + __assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */ =20 for (i =3D 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) { if (!kvm_memslots_empty(__kvm_memslots(kvm, i))) @@ -5930,6 +5965,7 @@ static int __kvm_io_bus_write(struct kvm_vcpu *vcpu, = struct kvm_io_bus *bus, } =20 static struct kvm_io_bus *kvm_get_bus_srcu(struct kvm *kvm, enum kvm_bus i= dx) + __must_hold_shared(&kvm->srcu) { /* * Ensure that any updates to kvm_buses[] observed by the previous vCPU diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c index 728d2c1b488a..f883fbad3487 100644 --- a/virt/kvm/pfncache.c +++ b/virt/kvm/pfncache.c @@ -153,6 +153,7 @@ static inline bool mmu_notifier_retry_cache(struct kvm = *kvm, unsigned long mmu_s } =20 static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc) + __must_hold(&gpc->lock) { /* Note, the new page offset may be different than the old! */ void *old_khva =3D (void *)PAGE_ALIGN_DOWN((uintptr_t)gpc->khva); @@ -254,6 +255,7 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cac= he *gpc) } =20 static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, gpa_t gpa, unsi= gned long uhva) + __must_hold_shared(&gpc->kvm->srcu) { unsigned long page_offset; bool unmap_old =3D false; @@ -396,6 +398,7 @@ void kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct = kvm *kvm) =20 static int __kvm_gpc_activate(struct gfn_to_pfn_cache *gpc, gpa_t gpa, uns= igned long uhva, unsigned long len) + __must_hold_shared(&gpc->kvm->srcu) { struct kvm *kvm =3D gpc->kvm; =20 --=20 2.55.0.1003.g10538fe699-goog From nobody Fri Sep 25 16:02:05 2026 Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D0F652B1C4 for ; Thu, 10 Sep 2026 16:24:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.69 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057459; cv=none; b=WwuHRnO5DhvKwlcH7PHsjiJuM8RQ3LwAB8/vIsYfu/EGgXl72AskPFHFGg3aZK8cmL8zMmAMNB9A5SFgcNkTPwUdL0V7HlgwdFoiCnVirnyCjGAGPwXCwfLXfXYq1JJp6uA0GqdhIlJusXs6/xje86y1zBO2Pa0BjQgIViVKyTA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057459; c=relaxed/simple; bh=nyp1CjhE37fv/eNY3uNnSxmNlTELsiXYH+XE3jWaJT4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=micZOjqpIyq0KGtLiZ58HcWuVWa4QephTxsL6OwNEEjbke8E8dv1SKuh8FM722Wp5bfPhYn6y/FayvN5PeKwHkDJXxTaEHvUqQG5S/NieTvqjQYVppZw1kA6z17iChmwU7mMZlMnpBQsd8TS1hzRfIBEhZL79PisFDUG24aG0PQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--elver.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=evSoW0ip; arc=none smtp.client-ip=209.85.221.69 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--elver.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="evSoW0ip" Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-482e05af072so6375306f8f.3 for ; Thu, 10 Sep 2026 09:24:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1789057445; x=1789662245; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=rXcm2H9Hj3wZMgLwTNPt9AcEfZj6H1K1nw4RELODNwU=; b=evSoW0iprlYTU7SV9A2T1T7qQl5bGzQW8nFla1X4A1noLiDynzqw4xyCO9IdNxzLlN tatMR6wtkKw24BRYy7t2+JeeVlUgZHCXD5d67Wpl74I2xiYer/sNB1YU+qf///yHJsyL c3tEmu8SKJcE/aGMBH3CpjT1A3qFueUrjM837SYlqWgOqgZ+WMSq9ONaffMFZXDHQCPT G/MCMQpdE6b/HM1rWcWE+6rU/5n6PF1f4Zphx3WgTyjCSPUgDfgBGgRMUeomWAILoCs4 86ujp1ocJa29TbTq6QNX8mlmCF0sgeQVzUslTI2jQDXyIQUACYTF4o2T20agUSxSNeXS 6e3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789057445; x=1789662245; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=rXcm2H9Hj3wZMgLwTNPt9AcEfZj6H1K1nw4RELODNwU=; b=CWB/4gAq/L7a33N0msOjYRDwcI9IBL/24jQGmqbaLC5/dmt8BJkyjen8i9eaBJzVE5 lu3z3QXqCnswud/meqw+hMjEbJUOLjOYmrdtGwyaMlu8SKzW3Kqx/qHqc01I+rLEx1Hd s19YZiBx+ixMqRVc8/q3Mc/eE0IrVdOJJ5ZcNhJZhKaPs1rP9d4uqQMhfgFPIgON+3B3 UZUTZvDJOGy+PLh4IXvBxD0I/T2emvSyy84yJa6SzMngtqZNJi7l64MR1iLVFwJc8/Ll yuT3kVBalUTACRgHEH9vsr2Li8f6X0VjoPHThMO7V7zfnZsfPI1HVIV+yiXZfYQ3L8px AICQ== X-Forwarded-Encrypted: i=1; AKwUvBzYdaDeaOQOZ6gXFg6jgult/bkyruGE6nUGoCite3fVhMUQ4yTmZIiihM+cD6fmJ/jCb8tfaYPiWtKiqug=@vger.kernel.org X-Gm-Message-State: AFuF++m6+F6ah5E8g/0YKw7kkiyJLoI6Aon8oz9ERNbzk0BK+j+sPbxB UXYZelLW92ahxXPgSvIkm653UWBN7fuVYPYIBO2LvJh/4clqiYKu2oSINoIVKlTiFaA+E25/MpY 6ew== X-Received: from wmdp17.prod.google.com ([2002:a05:600c:5d1:b0:493:b636:4ee9]) (user=elver job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600d:6445:10b0:49c:fc6c:be09 with SMTP id 5b1f17b1804b1-49cfc6cc00emr286275605e9.32.1789057443617; Thu, 10 Sep 2026 09:24:03 -0700 (PDT) Date: Thu, 10 Sep 2026 16:21:40 +0000 In-Reply-To: <20260910162343.4092060-1-elver@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260910162343.4092060-1-elver@google.com> X-Mailer: git-send-email 2.55.0.1003.g10538fe699-goog Message-ID: <20260910162343.4092060-8-elver@google.com> Subject: [PATCH RFC 07/10] KVM: x86: Add basic lock context annotations From: Marco Elver To: elver@google.com Cc: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Vitaly Kuznetsov , Kiryl Shutsemau , Rick Edgecombe , David Hildenbrand , kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add basic lock context annotations across common x86, VMX, SVM, and Hyper-V sources in preparation for enabling context analysis. The only opted-out functions are tdx_acquire_vm_state_locks() and tdx_release_vm_state_locks(), which dynamically acquire and release an arbitrary number of locks that cannot be represented statically. Because Clang context analysis evaluates lock expressions syntactically, it cannot prove the equivalence of heap pointer aliases (e.g. `vcpu->kvm` vs. `to_svm(vcpu)->vcpu.kvm`). When a callee expects a lock expression through a different alias path than the caller, bridge the alias using lockdep_assert_held() on the target expression. This dynamically validates lock ownership under lockdep while satisfying the compiler's static analysi= s. No functional change intended. Signed-off-by: Marco Elver --- arch/x86/kvm/debugfs.c | 1 + arch/x86/kvm/hyperv.c | 21 +++++++++++++ arch/x86/kvm/hyperv.h | 16 +++++++--- arch/x86/kvm/ioapic.c | 24 ++++++++++++--- arch/x86/kvm/irq.c | 2 ++ arch/x86/kvm/lapic.c | 8 ++++- arch/x86/kvm/lapic.h | 15 ++++++--- arch/x86/kvm/mmu/page_track.c | 1 + arch/x86/kvm/mmu/page_track.h | 3 +- arch/x86/kvm/mmu/spte.h | 3 +- arch/x86/kvm/msrs.c | 5 +++ arch/x86/kvm/msrs.h | 3 +- arch/x86/kvm/regs.h | 15 ++++++--- arch/x86/kvm/smm.c | 6 ++++ arch/x86/kvm/smm.h | 2 +- arch/x86/kvm/svm/hyperv.c | 2 ++ arch/x86/kvm/svm/hyperv.h | 6 ++-- arch/x86/kvm/svm/nested.c | 30 ++++++++++++++++-- arch/x86/kvm/svm/sev.c | 18 +++++++++++ arch/x86/kvm/svm/svm.c | 32 +++++++++++++++++++ arch/x86/kvm/svm/svm.h | 28 +++++++++++------ arch/x86/kvm/vmx/hyperv.h | 6 ++-- arch/x86/kvm/vmx/main.c | 5 +++ arch/x86/kvm/vmx/nested.c | 54 ++++++++++++++++++++++++++++++-- arch/x86/kvm/vmx/nested.h | 19 ++++++++---- arch/x86/kvm/vmx/sgx.c | 3 ++ arch/x86/kvm/vmx/sgx.h | 3 +- arch/x86/kvm/vmx/tdx.c | 9 ++++++ arch/x86/kvm/vmx/vmx.c | 11 +++++++ arch/x86/kvm/vmx/x86_ops.h | 24 ++++++++++----- arch/x86/kvm/x86.c | 58 ++++++++++++++++++++++++++++++++--- arch/x86/kvm/x86.h | 37 +++++++++++++++------- 32 files changed, 398 insertions(+), 72 deletions(-) diff --git a/arch/x86/kvm/debugfs.c b/arch/x86/kvm/debugfs.c index 0074a56e45b4..0bf8c021ee89 100644 --- a/arch/x86/kvm/debugfs.c +++ b/arch/x86/kvm/debugfs.c @@ -109,6 +109,7 @@ static int kvm_mmu_rmaps_stat_show(struct seq_file *m, = void *v) } =20 mutex_lock(&kvm->slots_lock); + __assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */ write_lock(&kvm->mmu_lock); =20 for (i =3D 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) { diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c index 604651cb2739..cca7c98a3808 100644 --- a/arch/x86/kvm/hyperv.c +++ b/arch/x86/kvm/hyperv.c @@ -268,6 +268,8 @@ static int synic_set_msr(struct kvm_vcpu_hv_synic *syni= c, struct kvm_vcpu *vcpu =3D hv_synic_to_vcpu(synic); int ret; =20 + lockdep_assert_held(&vcpu->kvm->srcu); + if (!synic->active && (!host || data)) return 1; =20 @@ -780,6 +782,8 @@ static int synic_deliver_msg(struct kvm_vcpu_hv_synic *= synic, u32 sint, struct hv_message_header hv_hdr; int r; =20 + lockdep_assert_held(&vcpu->kvm->srcu); + if (!(synic->msg_page & HV_SYNIC_SIMP_ENABLE)) return -ENOENT; =20 @@ -1056,6 +1060,7 @@ static bool kvm_hv_msr_partition_wide(u32 msr) } =20 static int kvm_hv_msr_get_crash_data(struct kvm *kvm, u32 index, u64 *pdat= a) + __must_hold(&to_kvm_hv(kvm)->hv_lock) { struct kvm_hv *hv =3D to_kvm_hv(kvm); size_t size =3D ARRAY_SIZE(hv->hv_crash_param); @@ -1068,6 +1073,7 @@ static int kvm_hv_msr_get_crash_data(struct kvm *kvm,= u32 index, u64 *pdata) } =20 static int kvm_hv_msr_get_crash_ctl(struct kvm *kvm, u64 *pdata) + __must_hold(&to_kvm_hv(kvm)->hv_lock) { struct kvm_hv *hv =3D to_kvm_hv(kvm); =20 @@ -1076,6 +1082,7 @@ static int kvm_hv_msr_get_crash_ctl(struct kvm *kvm, = u64 *pdata) } =20 static int kvm_hv_msr_set_crash_ctl(struct kvm *kvm, u64 data) + __must_hold(&to_kvm_hv(kvm)->hv_lock) { struct kvm_hv *hv =3D to_kvm_hv(kvm); =20 @@ -1085,6 +1092,7 @@ static int kvm_hv_msr_set_crash_ctl(struct kvm *kvm, = u64 data) } =20 static int kvm_hv_msr_set_crash_data(struct kvm *kvm, u32 index, u64 data) + __must_hold(&to_kvm_hv(kvm)->hv_lock) { struct kvm_hv *hv =3D to_kvm_hv(kvm); size_t size =3D ARRAY_SIZE(hv->hv_crash_param); @@ -1175,6 +1183,7 @@ static bool compute_tsc_page_parameters(struct pvcloc= k_vcpu_time_info *hv_clock, * TSC scaling is unsupported). */ static inline bool tsc_page_update_unsafe(struct kvm_hv *hv) + __must_hold(&hv->hv_lock) { return (hv->hv_tsc_page_status !=3D HV_TSC_PAGE_GUEST_CHANGED) && hv->hv_tsc_emulation_control; @@ -1397,6 +1406,8 @@ void kvm_hv_xsaves_xsavec_maybe_warn(struct kvm_vcpu = *vcpu) =20 static int kvm_hv_set_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 data, bool host) + __must_hold(&to_kvm_hv(vcpu->kvm)->hv_lock) + __must_hold_shared(&vcpu->kvm->srcu) { struct kvm *kvm =3D vcpu->kvm; struct kvm_hv *hv =3D to_kvm_hv(kvm); @@ -1539,6 +1550,7 @@ static u64 current_task_runtime_100ns(void) } =20 static int kvm_hv_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data, bool h= ost) + __must_hold_shared(&vcpu->kvm->srcu) { struct kvm_vcpu_hv *hv_vcpu =3D to_hv_vcpu(vcpu); =20 @@ -1652,6 +1664,7 @@ static int kvm_hv_set_msr(struct kvm_vcpu *vcpu, u32 = msr, u64 data, bool host) =20 static int kvm_hv_get_msr_pw(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata, bool host) + __must_hold(&to_kvm_hv(vcpu->kvm)->hv_lock) { u64 data =3D 0; struct kvm *kvm =3D vcpu->kvm; @@ -1910,6 +1923,7 @@ struct kvm_hv_hcall { =20 static int kvm_hv_get_hc_data(struct kvm *kvm, struct kvm_hv_hcall *hc, u16 orig_cnt, u16 cnt_cap, u64 *data) + __must_hold_shared(&kvm->srcu) { /* * Preserve the original count when ignoring entries via a "cap", KVM @@ -1943,6 +1957,7 @@ static int kvm_hv_get_hc_data(struct kvm *kvm, struct= kvm_hv_hcall *hc, =20 static u64 kvm_get_sparse_vp_set(struct kvm *kvm, struct kvm_hv_hcall *hc, u64 *sparse_banks) + __must_hold_shared(&kvm->srcu) { if (hc->var_cnt > HV_MAX_SPARSE_VCPU_BANKS) return -EINVAL; @@ -1953,6 +1968,7 @@ static u64 kvm_get_sparse_vp_set(struct kvm *kvm, str= uct kvm_hv_hcall *hc, } =20 static int kvm_hv_get_tlb_flush_entries(struct kvm *kvm, struct kvm_hv_hca= ll *hc, u64 entries[]) + __must_hold_shared(&kvm->srcu) { return kvm_hv_get_hc_data(kvm, hc, hc->rep_cnt, hc->rep_cnt, entries); } @@ -2034,6 +2050,7 @@ int kvm_hv_vcpu_flush_tlb(struct kvm_vcpu *vcpu) } =20 static u64 kvm_hv_flush_tlb(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc) + __must_hold_shared(&vcpu->kvm->srcu) { struct kvm_vcpu_hv *hv_vcpu =3D to_hv_vcpu(vcpu); unsigned long *vcpu_mask =3D hv_vcpu->vcpu_mask; @@ -2243,6 +2260,7 @@ static void kvm_hv_send_ipi_to_many(struct kvm *kvm, = u32 vector, } =20 static u64 kvm_hv_send_ipi(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *hc) + __must_hold_shared(&vcpu->kvm->srcu) { struct kvm_vcpu_hv *hv_vcpu =3D to_hv_vcpu(vcpu); u64 *sparse_banks =3D hv_vcpu->sparse_banks; @@ -2405,6 +2423,7 @@ static void kvm_hv_hypercall_set_result(struct kvm_vc= pu *vcpu, u64 result) } =20 static int kvm_hv_hypercall_complete(struct kvm_vcpu *vcpu, u64 result) + __must_hold_shared(&vcpu->kvm->srcu) { u32 tlb_lock_count =3D 0; int ret; @@ -2428,11 +2447,13 @@ static int kvm_hv_hypercall_complete(struct kvm_vcp= u *vcpu, u64 result) } =20 static int kvm_hv_hypercall_complete_userspace(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { return kvm_hv_hypercall_complete(vcpu, vcpu->run->hyperv.u.hcall.result); } =20 static u16 kvm_hvcall_signal_event(struct kvm_vcpu *vcpu, struct kvm_hv_hc= all *hc) + __must_hold_shared(&vcpu->kvm->srcu) { struct kvm_hv *hv =3D to_kvm_hv(vcpu->kvm); struct eventfd_ctx *eventfd; diff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h index 622a6553e9ac..09dbab1fddb1 100644 --- a/arch/x86/kvm/hyperv.h +++ b/arch/x86/kvm/hyperv.h @@ -197,7 +197,8 @@ static inline u32 kvm_hv_get_vpindex(struct kvm_vcpu *v= cpu) return hv_vcpu ? hv_vcpu->vp_index : vcpu->vcpu_idx; } =20 -int kvm_hv_set_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 data, bool h= ost); +int kvm_hv_set_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 data, bool h= ost) + __must_hold_shared(&vcpu->kvm->srcu); int kvm_hv_get_msr_common(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata, bool= host); =20 static inline bool kvm_hv_hypercall_enabled(struct kvm_vcpu *vcpu) @@ -205,9 +206,11 @@ static inline bool kvm_hv_hypercall_enabled(struct kvm= _vcpu *vcpu) return vcpu->arch.hyperv_enabled && to_kvm_hv(vcpu->kvm)->hv_guest_os_id; } =20 -int kvm_hv_hypercall(struct kvm_vcpu *vcpu); +int kvm_hv_hypercall(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu); =20 -void kvm_hv_irq_routing_update(struct kvm *kvm); +void kvm_hv_irq_routing_update(struct kvm *kvm) + __must_hold_shared(&kvm->irq_srcu); int kvm_hv_synic_set_irq(struct kvm_kernel_irq_routing_entry *e, struct kv= m *kvm, int irq_source_id, int level, bool line_status); void kvm_hv_synic_send_eoi(struct kvm_vcpu *vcpu, int vector); @@ -227,7 +230,8 @@ static inline bool kvm_hv_synic_auto_eoi_set(struct kvm= _vcpu *vcpu, int vector) void kvm_hv_vcpu_uninit(struct kvm_vcpu *vcpu); =20 bool kvm_hv_assist_page_enabled(struct kvm_vcpu *vcpu); -int kvm_hv_get_assist_page(struct kvm_vcpu *vcpu); +int kvm_hv_get_assist_page(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu); =20 static inline struct kvm_vcpu_hv_stimer *to_hv_stimer(struct kvm_vcpu *vcp= u, int timer_index) @@ -285,7 +289,8 @@ static inline bool kvm_hv_invtsc_suppressed(struct kvm_= vcpu *vcpu) void kvm_hv_process_stimers(struct kvm_vcpu *vcpu); =20 void kvm_hv_setup_tsc_page(struct kvm *kvm, - struct pvclock_vcpu_time_info *hv_clock); + struct pvclock_vcpu_time_info *hv_clock) + __must_hold_shared(&kvm->srcu); void kvm_hv_request_tsc_page_update(struct kvm *kvm); =20 void kvm_hv_xsaves_xsavec_maybe_warn(struct kvm_vcpu *vcpu); @@ -352,6 +357,7 @@ static inline bool kvm_hv_is_tlb_flush_hcall(struct kvm= _vcpu *vcpu) } =20 static inline int kvm_hv_verify_vp_assist(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { if (!to_hv_vcpu(vcpu)) return 0; diff --git a/arch/x86/kvm/ioapic.c b/arch/x86/kvm/ioapic.c index 0d59b9c758c2..d6865e557abe 100644 --- a/arch/x86/kvm/ioapic.c +++ b/arch/x86/kvm/ioapic.c @@ -36,9 +36,11 @@ #include "x86.h" =20 static int ioapic_service(struct kvm_ioapic *vioapic, int irq, - bool line_status); + bool line_status) + __must_hold(&vioapic->lock); =20 static unsigned long ioapic_read_indirect(struct kvm_ioapic *ioapic) + __must_hold(&ioapic->lock) { unsigned long result =3D 0; =20 @@ -76,20 +78,24 @@ static unsigned long ioapic_read_indirect(struct kvm_io= apic *ioapic) } =20 static void rtc_irq_eoi_tracking_reset(struct kvm_ioapic *ioapic) + __must_hold(&ioapic->lock) { ioapic->rtc_status.pending_eoi =3D 0; bitmap_zero(ioapic->rtc_status.map, KVM_MAX_VCPU_IDS); } =20 -static void kvm_rtc_eoi_tracking_restore_all(struct kvm_ioapic *ioapic); +static void kvm_rtc_eoi_tracking_restore_all(struct kvm_ioapic *ioapic) + __must_hold(&ioapic->lock); =20 static void rtc_status_pending_eoi_check_valid(struct kvm_ioapic *ioapic) + __must_hold(&ioapic->lock) { if (WARN_ON_ONCE(ioapic->rtc_status.pending_eoi < 0)) kvm_rtc_eoi_tracking_restore_all(ioapic); } =20 static void __rtc_irq_eoi_tracking_restore_one(struct kvm_vcpu *vcpu) + __must_hold(&vcpu->kvm->arch.vioapic->lock) { bool new_val, old_val; struct kvm_ioapic *ioapic =3D vcpu->kvm->arch.vioapic; @@ -137,12 +143,15 @@ static void kvm_rtc_eoi_tracking_restore_all(struct k= vm_ioapic *ioapic) return; =20 rtc_irq_eoi_tracking_reset(ioapic); - kvm_for_each_vcpu(i, vcpu, ioapic->kvm) - __rtc_irq_eoi_tracking_restore_one(vcpu); + kvm_for_each_vcpu(i, vcpu, ioapic->kvm) { + lockdep_assert_held(&vcpu->kvm->arch.vioapic->lock); /* vcpu->kvm->arch.= vioapic =3D=3D ioapic */ + __rtc_irq_eoi_tracking_restore_one(vcpu); + } } =20 static void rtc_irq_eoi(struct kvm_ioapic *ioapic, struct kvm_vcpu *vcpu, int vector) + __must_hold(&ioapic->lock) { struct rtc_status *status =3D &ioapic->rtc_status; =20 @@ -156,6 +165,7 @@ static void rtc_irq_eoi(struct kvm_ioapic *ioapic, stru= ct kvm_vcpu *vcpu, } =20 static bool rtc_irq_check_coalesced(struct kvm_ioapic *ioapic) + __must_hold(&ioapic->lock) { if (ioapic->rtc_status.pending_eoi > 0) return true; /* coalesced */ @@ -164,6 +174,7 @@ static bool rtc_irq_check_coalesced(struct kvm_ioapic *= ioapic) } =20 static void ioapic_lazy_update_eoi(struct kvm_ioapic *ioapic, int irq) + __must_hold(&ioapic->lock) { unsigned long i; struct kvm_vcpu *vcpu; @@ -187,6 +198,7 @@ static void ioapic_lazy_update_eoi(struct kvm_ioapic *i= oapic, int irq) =20 static int ioapic_set_irq(struct kvm_ioapic *ioapic, unsigned int irq, int irq_level, bool line_status) + __must_hold(&ioapic->lock) { union kvm_ioapic_redirect_entry entry; u32 mask =3D 1 << irq; @@ -246,6 +258,7 @@ static int ioapic_set_irq(struct kvm_ioapic *ioapic, un= signed int irq, } =20 static void kvm_ioapic_inject_all(struct kvm_ioapic *ioapic, unsigned long= irr) + __must_hold(&ioapic->lock) { u32 idx; =20 @@ -330,6 +343,7 @@ void kvm_fire_mask_notifiers(struct kvm *kvm, unsigned = irqchip, unsigned pin, } =20 static void ioapic_write_indirect(struct kvm_ioapic *ioapic, u32 val) + __must_hold(&ioapic->lock) { unsigned index; bool mask_before, mask_after; @@ -541,6 +555,7 @@ static void kvm_ioapic_update_eoi_one(struct kvm_vcpu *= vcpu, struct kvm_ioapic *ioapic, int trigger_mode, int pin) + __must_hold(&ioapic->lock) { struct kvm_lapic *apic =3D vcpu->arch.apic; union kvm_ioapic_redirect_entry *ent =3D &ioapic->redirtbl[pin]; @@ -695,6 +710,7 @@ static int ioapic_mmio_write(struct kvm_vcpu *vcpu, str= uct kvm_io_device *this, } =20 static void kvm_ioapic_reset(struct kvm_ioapic *ioapic) + __must_hold(&ioapic->lock) { int i; =20 diff --git a/arch/x86/kvm/irq.c b/arch/x86/kvm/irq.c index 871977288272..5dd3bfe2aeb4 100644 --- a/arch/x86/kvm/irq.c +++ b/arch/x86/kvm/irq.c @@ -416,6 +416,8 @@ void kvm_scan_ioapic_routes(struct kvm_vcpu *vcpu, void kvm_arch_irq_routing_update(struct kvm *kvm) { #ifdef CONFIG_KVM_HYPERV + /* Update-side mutex kvm->irq_lock is held. */ + __assume_shared_ctx_lock(&kvm->irq_srcu); kvm_hv_irq_routing_update(kvm); #endif =20 diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index e1f3cea14765..f5022de46725 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -901,6 +901,7 @@ int kvm_pv_send_ipi(struct kvm *kvm, unsigned long ipi_= bitmap_low, } =20 static int pv_eoi_put_user(struct kvm_vcpu *vcpu, u8 val) + __must_hold_shared(&vcpu->kvm->srcu) { =20 return kvm_write_guest_cached(vcpu->kvm, &vcpu->arch.pv_eoi.data, &val, @@ -908,6 +909,7 @@ static int pv_eoi_put_user(struct kvm_vcpu *vcpu, u8 va= l) } =20 static int pv_eoi_get_user(struct kvm_vcpu *vcpu, u8 *val) + __must_hold_shared(&vcpu->kvm->srcu) { =20 return kvm_read_guest_cached(vcpu->kvm, &vcpu->arch.pv_eoi.data, val, @@ -920,6 +922,7 @@ static inline bool pv_eoi_enabled(struct kvm_vcpu *vcpu) } =20 static void pv_eoi_set_pending(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { if (pv_eoi_put_user(vcpu, KVM_PV_EOI_ENABLED) < 0) return; @@ -928,6 +931,7 @@ static void pv_eoi_set_pending(struct kvm_vcpu *vcpu) } =20 static bool pv_eoi_test_and_clr_pending(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { u8 val; =20 @@ -3378,6 +3382,7 @@ void __kvm_migrate_apic_timer(struct kvm_vcpu *vcpu) */ static void apic_sync_pv_eoi_from_guest(struct kvm_vcpu *vcpu, struct kvm_lapic *apic) + __must_hold_shared(&vcpu->kvm->srcu) { int vector; =20 @@ -3428,6 +3433,7 @@ void kvm_lapic_sync_from_vapic(struct kvm_vcpu *vcpu) */ static void apic_sync_pv_eoi_to_guest(struct kvm_vcpu *vcpu, struct kvm_lapic *apic) + __must_hold_shared(&vcpu->kvm->srcu) { if (!pv_eoi_enabled(vcpu) || /* IRR set or many bits in ISR: could be nested. */ @@ -3443,7 +3449,7 @@ static void apic_sync_pv_eoi_to_guest(struct kvm_vcpu= *vcpu, return; } =20 - pv_eoi_set_pending(apic->vcpu); + pv_eoi_set_pending(vcpu); } =20 void kvm_lapic_sync_to_vapic(struct kvm_vcpu *vcpu) diff --git a/arch/x86/kvm/lapic.h b/arch/x86/kvm/lapic.h index bd1098c89d99..a193f32a5688 100644 --- a/arch/x86/kvm/lapic.h +++ b/arch/x86/kvm/lapic.h @@ -148,7 +148,8 @@ int kvm_apic_set_irq(struct kvm_vcpu *vcpu, struct kvm_= lapic_irq *irq, int kvm_apic_local_deliver(struct kvm_lapic *apic, int lvt_type); void kvm_apic_update_apicv(struct kvm_vcpu *vcpu); int kvm_alloc_apic_access_page(struct kvm *kvm); -void kvm_inhibit_apic_access_page(struct kvm_vcpu *vcpu); +void kvm_inhibit_apic_access_page(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu); =20 bool kvm_irq_delivery_to_apic_fast(struct kvm *kvm, struct kvm_lapic *src, struct kvm_lapic_irq *irq, int *r); @@ -179,9 +180,12 @@ void kvm_set_lapic_tscdeadline_msr(struct kvm_vcpu *vc= pu, u64 data); void kvm_apic_write_nodecode(struct kvm_vcpu *vcpu, u32 offset); void kvm_apic_set_eoi_accelerated(struct kvm_vcpu *vcpu, int vector); =20 -int kvm_lapic_set_vapic_addr(struct kvm_vcpu *vcpu, gpa_t vapic_addr); -void kvm_lapic_sync_from_vapic(struct kvm_vcpu *vcpu); -void kvm_lapic_sync_to_vapic(struct kvm_vcpu *vcpu); +int kvm_lapic_set_vapic_addr(struct kvm_vcpu *vcpu, gpa_t vapic_addr) + __must_hold_shared(&vcpu->kvm->srcu); +void kvm_lapic_sync_from_vapic(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu); +void kvm_lapic_sync_to_vapic(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu); =20 int kvm_x2apic_icr_write_fast(struct kvm_lapic *apic, u64 data); int kvm_x2apic_msr_write(struct kvm_vcpu *vcpu, u32 msr, u64 data); @@ -190,7 +194,8 @@ int kvm_x2apic_msr_read(struct kvm_vcpu *vcpu, u32 msr,= u64 *data); int kvm_hv_vapic_msr_write(struct kvm_vcpu *vcpu, u32 msr, u64 data); int kvm_hv_vapic_msr_read(struct kvm_vcpu *vcpu, u32 msr, u64 *data); =20 -int kvm_lapic_set_pv_eoi(struct kvm_vcpu *vcpu, u64 data, unsigned long le= n); +int kvm_lapic_set_pv_eoi(struct kvm_vcpu *vcpu, u64 data, unsigned long le= n) + __must_hold_shared(&vcpu->kvm->srcu); void kvm_lapic_exit(void); =20 u64 kvm_x2apic_disable_read_intercept_reg_mask(struct kvm_vcpu *vcpu); diff --git a/arch/x86/kvm/mmu/page_track.c b/arch/x86/kvm/mmu/page_track.c index 7e8195a311bb..058a26d8065a 100644 --- a/arch/x86/kvm/mmu/page_track.c +++ b/arch/x86/kvm/mmu/page_track.c @@ -196,6 +196,7 @@ static int kvm_enable_external_write_tracking(struct kv= m *kvm) return -EOPNOTSUPP; =20 mutex_lock(&kvm->slots_arch_lock); + __assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */ =20 /* * Check for *any* write tracking user (not just external users) under diff --git a/arch/x86/kvm/mmu/page_track.h b/arch/x86/kvm/mmu/page_track.h index d4d72ed999b1..8f7e176af858 100644 --- a/arch/x86/kvm/mmu/page_track.h +++ b/arch/x86/kvm/mmu/page_track.h @@ -21,7 +21,8 @@ void __kvm_write_track_remove_gfn(struct kvm *kvm, struct kvm_memory_slot *slot, gfn_t gfn); =20 bool kvm_gfn_is_write_tracked(struct kvm *kvm, - const struct kvm_memory_slot *slot, gfn_t gfn); + const struct kvm_memory_slot *slot, gfn_t gfn) + __must_hold_shared(&kvm->srcu); =20 #ifdef CONFIG_KVM_EXTERNAL_WRITE_TRACKING int kvm_page_track_init(struct kvm *kvm); diff --git a/arch/x86/kvm/mmu/spte.h b/arch/x86/kvm/mmu/spte.h index e730717824b3..cf8fc077ce04 100644 --- a/arch/x86/kvm/mmu/spte.h +++ b/arch/x86/kvm/mmu/spte.h @@ -568,7 +568,8 @@ u64 make_small_spte(struct kvm *kvm, u64 huge_spte, union kvm_mmu_page_role role, int index); u64 make_huge_spte(struct kvm *kvm, u64 small_spte, int level); u64 make_nonleaf_spte(u64 *child_pt, bool ad_disabled); -u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsigned int access); +u64 make_mmio_spte(struct kvm_vcpu *vcpu, u64 gfn, unsigned int access) + __must_hold_shared(&vcpu->kvm->srcu); u64 mark_spte_for_access_track(u64 spte); =20 /* Restore an acc-track PTE back to a regular PTE */ diff --git a/arch/x86/kvm/msrs.c b/arch/x86/kvm/msrs.c index dd3bb04878ca..a90d6576e3cf 100644 --- a/arch/x86/kvm/msrs.c +++ b/arch/x86/kvm/msrs.c @@ -1222,6 +1222,7 @@ static void kvm_get_xstate_msr(struct kvm_vcpu *vcpu,= struct msr_data *msr_info) } =20 static void kvm_write_wall_clock(struct kvm *kvm, gpa_t wall_clock, int se= c_hi_ofs) + __must_hold_shared(&kvm->srcu) { int version; int r; @@ -1264,9 +1265,12 @@ static void kvm_write_wall_clock(struct kvm *kvm, gp= a_t wall_clock, int sec_hi_o =20 static void kvm_write_system_time(struct kvm_vcpu *vcpu, gpa_t system_time, bool old_msr, bool host_initiated) + __must_hold_shared(&vcpu->kvm->srcu) { struct kvm_arch *ka =3D &vcpu->kvm->arch; =20 + lockdep_assert_held(&vcpu->arch.pv_time.kvm->srcu); /* vcpu->arch.pv_time= .kvm =3D=3D vcpu->kvm */ + if (vcpu->vcpu_id =3D=3D 0 && !host_initiated) { if (ka->boot_vcpu_runs_old_kvmclock !=3D old_msr) kvm_make_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu); @@ -1382,6 +1386,7 @@ static int set_msr_mce(struct kvm_vcpu *vcpu, struct = msr_data *msr_info) } =20 static int kvm_pv_enable_async_pf(struct kvm_vcpu *vcpu, u64 data) + __must_hold_shared(&vcpu->kvm->srcu) { gpa_t gpa =3D data & ~0x3f; =20 diff --git a/arch/x86/kvm/msrs.h b/arch/x86/kvm/msrs.h index 7cc182a15b3b..0736b0f61873 100644 --- a/arch/x86/kvm/msrs.h +++ b/arch/x86/kvm/msrs.h @@ -74,7 +74,8 @@ fastpath_t handle_fastpath_wrmsr(struct kvm_vcpu *vcpu); fastpath_t handle_fastpath_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int r= eg); =20 int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr); -int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr); +int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr) + __must_hold_shared(&vcpu->kvm->srcu); =20 int kvm_add_user_return_msr(u32 msr); int kvm_find_user_return_msr(u32 msr); diff --git a/arch/x86/kvm/regs.h b/arch/x86/kvm/regs.h index 447f0ec3e63e..3f6d4a0ac97f 100644 --- a/arch/x86/kvm/regs.h +++ b/arch/x86/kvm/regs.h @@ -56,15 +56,20 @@ static_assert(!(KVM_POSSIBLE_CR0_GUEST_BITS & X86_CR0_P= DPTR_BITS)); =20 void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsign= ed long cr0); void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsign= ed long cr4); -int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0); -int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3); -int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4); +int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) + __must_hold_shared(&vcpu->kvm->srcu); +int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3) + __must_hold_shared(&vcpu->kvm->srcu); +int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) + __must_hold_shared(&vcpu->kvm->srcu); int kvm_set_cr8(struct kvm_vcpu *vcpu, unsigned long cr8); int kvm_set_dr(struct kvm_vcpu *vcpu, int dr, unsigned long val); unsigned long kvm_get_dr(struct kvm_vcpu *vcpu, int dr); unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu); -void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw); -int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3); +void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw) + __must_hold_shared(&vcpu->kvm->srcu); +int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3) + __must_hold_shared(&vcpu->kvm->srcu); =20 static inline bool is_long_mode(struct kvm_vcpu *vcpu) { diff --git a/arch/x86/kvm/smm.c b/arch/x86/kvm/smm.c index 656a38dad7e7..5f93257961c3 100644 --- a/arch/x86/kvm/smm.c +++ b/arch/x86/kvm/smm.c @@ -418,6 +418,7 @@ static int rsm_load_seg_64(struct kvm_vcpu *vcpu, =20 static int rsm_enter_protected_mode(struct kvm_vcpu *vcpu, u64 cr0, u64 cr3, u64 cr4) + __must_hold_shared(&vcpu->kvm->srcu) { int bad; u64 pcid; @@ -463,6 +464,7 @@ static int rsm_enter_protected_mode(struct kvm_vcpu *vc= pu, =20 static int rsm_load_state_32(struct x86_emulate_ctxt *ctxt, const struct kvm_smram_state_32 *smstate) + __must_hold_shared(&((struct kvm_vcpu *)ctxt->vcpu)->kvm->srcu) { struct kvm_vcpu *vcpu =3D ctxt->vcpu; struct desc_ptr dt; @@ -515,6 +517,7 @@ static int rsm_load_state_32(struct x86_emulate_ctxt *c= txt, #ifdef CONFIG_X86_64 static int rsm_load_state_64(struct x86_emulate_ctxt *ctxt, const struct kvm_smram_state_64 *smstate) + __must_hold_shared(&((struct kvm_vcpu *)ctxt->vcpu)->kvm->srcu) { struct kvm_vcpu *vcpu =3D ctxt->vcpu; struct desc_ptr dt; @@ -578,6 +581,9 @@ int emulator_leave_smm(struct x86_emulate_ctxt *ctxt) u64 smbase; int ret; =20 + /* Called via struct x86_emulate_ops callback; assert SRCU dynamically. */ + lockdep_assert_held(&vcpu->kvm->srcu); + smbase =3D vcpu->arch.smbase; =20 ret =3D kvm_vcpu_read_guest(vcpu, smbase + 0xfe00, smram.bytes, sizeof(sm= ram)); diff --git a/arch/x86/kvm/smm.h b/arch/x86/kvm/smm.h index db3c88f16138..6c15db642538 100644 --- a/arch/x86/kvm/smm.h +++ b/arch/x86/kvm/smm.h @@ -155,7 +155,7 @@ static inline bool is_smm(struct kvm_vcpu *vcpu) } =20 void kvm_smm_changed(struct kvm_vcpu *vcpu, bool in_smm); -void enter_smm(struct kvm_vcpu *vcpu); +void enter_smm(struct kvm_vcpu *vcpu) __must_hold_shared(&vcpu->kvm->srcu); int emulator_leave_smm(struct x86_emulate_ctxt *ctxt); void process_smi(struct kvm_vcpu *vcpu); #else diff --git a/arch/x86/kvm/svm/hyperv.c b/arch/x86/kvm/svm/hyperv.c index 4f24dcb45116..117c069c5bfa 100644 --- a/arch/x86/kvm/svm/hyperv.c +++ b/arch/x86/kvm/svm/hyperv.c @@ -10,6 +10,8 @@ void svm_hv_inject_synthetic_vmexit_post_tlb_flush(struct= kvm_vcpu *vcpu) { struct vcpu_svm *svm =3D to_svm(vcpu); =20 + lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm =3D=3D vcpu->= kvm */ + /* * The exit code used by Hyper-V for software-defined exits is reserved * by AMD specifically for such use cases. diff --git a/arch/x86/kvm/svm/hyperv.h b/arch/x86/kvm/svm/hyperv.h index f70d076911a6..e9990cabd332 100644 --- a/arch/x86/kvm/svm/hyperv.h +++ b/arch/x86/kvm/svm/hyperv.h @@ -48,14 +48,16 @@ static inline bool nested_svm_is_l2_tlb_flush_hcall(str= uct kvm_vcpu *vcpu) kvm_hv_is_tlb_flush_hcall(vcpu); } =20 -void svm_hv_inject_synthetic_vmexit_post_tlb_flush(struct kvm_vcpu *vcpu); +void svm_hv_inject_synthetic_vmexit_post_tlb_flush(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu); #else /* CONFIG_KVM_HYPERV */ static inline void nested_svm_hv_update_vm_vp_ids(struct kvm_vcpu *vcpu) {} static inline bool nested_svm_is_l2_tlb_flush_hcall(struct kvm_vcpu *vcpu) { return false; } -static inline void svm_hv_inject_synthetic_vmexit_post_tlb_flush(struct kv= m_vcpu *vcpu) {} +static inline void svm_hv_inject_synthetic_vmexit_post_tlb_flush(struct kv= m_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) {} #endif /* CONFIG_KVM_HYPERV */ =20 #endif /* __ARCH_X86_KVM_SVM_HYPERV_H__ */ diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 73f37b050d0a..d421d6807166 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -38,9 +38,12 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *vcpu, struct x86_exception *fault, bool from_hardware) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_svm *svm =3D to_svm(vcpu); struct vmcb *vmcb =3D svm->vmcb; + + lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm =3D=3D vcpu->= kvm */ u64 fault_stage; =20 /* @@ -72,6 +75,7 @@ static void nested_svm_inject_npf_exit(struct kvm_vcpu *v= cpu, } =20 static u64 nested_svm_get_tdp_pdptr(struct kvm_vcpu *vcpu, int index) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_svm *svm =3D to_svm(vcpu); u64 cr3 =3D svm->nested.ctl.nested_cr3; @@ -293,6 +297,7 @@ int __init nested_svm_init_msrpm_merge_offsets(void) * may contain zero bits. */ static bool nested_svm_merge_msrpm(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_svm *svm =3D to_svm(vcpu); nsvm_msrpm_merge_t *msrpm02 =3D svm->nested.msrpm; @@ -713,6 +718,7 @@ static void nested_svm_transition_tlb_flush(struct kvm_= vcpu *vcpu) */ static int nested_svm_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3, bool nested_npt, bool reload_pdptrs) + __must_hold_shared(&vcpu->kvm->srcu) { if (CC(!kvm_vcpu_is_legal_cr3(vcpu, cr3))) return -EINVAL; @@ -1065,7 +1071,7 @@ int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 v= mcb12_gpa, bool from_vmrun) nested_vmcb02_prepare_control(svm); nested_vmcb02_prepare_save(svm); =20 - ret =3D nested_svm_load_cr3(&svm->vcpu, svm->nested.save.cr3, + ret =3D nested_svm_load_cr3(vcpu, svm->nested.save.cr3, nested_npt_enabled(svm), from_vmrun); if (ret) return ret; @@ -1117,6 +1123,8 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu) u64 vmcb12_gpa; struct vmcb *vmcb01 =3D svm->vmcb01.ptr; =20 + lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm =3D=3D vcpu->= kvm */ + if (!svm->nested.hsave_msr) { kvm_inject_gp(vcpu, 0); return 1; @@ -1320,6 +1328,8 @@ void nested_svm_vmexit(struct vcpu_svm *svm) struct vmcb *vmcb01 =3D svm->vmcb01.ptr; struct vmcb *vmcb02 =3D svm->nested.vmcb02.ptr; =20 + lockdep_assert_held(&vcpu->kvm->srcu); /* vcpu =3D=3D &svm->vcpu */ + if (nested_svm_vmexit_update_vmcb12(vcpu)) kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu); =20 @@ -1467,14 +1477,17 @@ void nested_svm_vmexit(struct vcpu_svm *svm) } =20 static void nested_svm_triple_fault(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_svm *svm =3D to_svm(vcpu); =20 + lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm =3D=3D vcpu->= kvm */ + if (!vmcb12_is_intercept(&svm->nested.ctl, INTERCEPT_SHUTDOWN)) return; =20 kvm_clear_request(KVM_REQ_TRIPLE_FAULT, vcpu); - nested_svm_simple_vmexit(to_svm(vcpu), SVM_EXIT_SHUTDOWN); + nested_svm_simple_vmexit(svm, SVM_EXIT_SHUTDOWN); } =20 int svm_allocate_nested(struct vcpu_svm *svm) @@ -1562,6 +1575,7 @@ void svm_leave_nested(struct kvm_vcpu *vcpu) } =20 static int nested_svm_exit_handled_msr(struct vcpu_svm *svm) + __must_hold_shared(&svm->vcpu.kvm->srcu) { gpa_t base =3D svm->nested.ctl.msrpm_base_pa; int write, bit_nr; @@ -1587,6 +1601,7 @@ static int nested_svm_exit_handled_msr(struct vcpu_sv= m *svm) } =20 static int nested_svm_intercept_ioio(struct vcpu_svm *svm) + __must_hold_shared(&svm->vcpu.kvm->srcu) { unsigned port, size, iopm_len; u16 val, mask; @@ -1612,6 +1627,7 @@ static int nested_svm_intercept_ioio(struct vcpu_svm = *svm) } =20 static int nested_svm_intercept(struct vcpu_svm *svm) + __must_hold_shared(&svm->vcpu.kvm->srcu) { u64 exit_code =3D svm->vmcb->control.exit_code; int vmexit =3D NESTED_EXIT_HOST; @@ -1679,11 +1695,14 @@ static bool nested_svm_is_exception_vmexit(struct k= vm_vcpu *vcpu, u8 vector, } =20 static void nested_svm_inject_exception_vmexit(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct kvm_queued_exception *ex =3D &vcpu->arch.exception_vmexit; struct vcpu_svm *svm =3D to_svm(vcpu); struct vmcb *vmcb =3D svm->vmcb; =20 + lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm =3D=3D vcpu->= kvm */ + vmcb->control.exit_code =3D SVM_EXIT_EXCP_BASE + ex->vector; =20 if (ex->has_error_code) @@ -1719,9 +1738,12 @@ static inline bool nested_exit_on_init(struct vcpu_s= vm *svm) } =20 static int svm_check_nested_events(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct kvm_lapic *apic =3D vcpu->arch.apic; struct vcpu_svm *svm =3D to_svm(vcpu); + + lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm =3D=3D vcpu->= kvm */ /* * Only a pending nested run blocks a pending exception. If there is a * previously injected event, the pending exception occurred while said @@ -1946,6 +1968,7 @@ static int svm_get_nested_state(struct kvm_vcpu *vcpu, static int svm_set_nested_state(struct kvm_vcpu *vcpu, struct kvm_nested_state __user *user_kvm_nested_state, struct kvm_nested_state *kvm_state) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_svm *svm =3D to_svm(vcpu); struct vmcb __user *user_vmcb =3D (struct vmcb __user *) @@ -2086,7 +2109,7 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu, * thus MMU might not be initialized correctly. * Set it again to fix this. */ - ret =3D nested_svm_load_cr3(&svm->vcpu, vcpu->arch.cr3, + ret =3D nested_svm_load_cr3(vcpu, vcpu->arch.cr3, nested_npt_enabled(svm), false); if (ret) goto out_free; @@ -2105,6 +2128,7 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu, } =20 static bool svm_get_nested_state_pages(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { if (WARN_ON(!is_guest_mode(vcpu))) return true; diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index 5705723f1f41..ec9589cd0abe 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -1117,6 +1117,7 @@ static int __sev_launch_update_vmsa(struct kvm *kvm, = struct kvm_vcpu *vcpu, } =20 static int sev_launch_update_vmsa(struct kvm *kvm, struct kvm_sev_cmd *arg= p) + __must_hold(&kvm->lock) { struct kvm_vcpu *vcpu; unsigned long i; @@ -1981,6 +1982,8 @@ static bool is_cmd_allowed_from_mirror(u32 cmd_id) } =20 static int sev_lock_two_vms(struct kvm *dst_kvm, struct kvm *src_kvm) + __cond_acquires(0, &dst_kvm->lock) + __cond_acquires(0, &src_kvm->lock) { struct kvm_sev_info *dst_sev =3D to_kvm_sev_info(dst_kvm); struct kvm_sev_info *src_sev =3D to_kvm_sev_info(src_kvm); @@ -2016,6 +2019,8 @@ static int sev_lock_two_vms(struct kvm *dst_kvm, stru= ct kvm *src_kvm) } =20 static void sev_unlock_two_vms(struct kvm *dst_kvm, struct kvm *src_kvm) + __releases(&dst_kvm->lock) + __releases(&src_kvm->lock) { struct kvm_sev_info *dst_sev =3D to_kvm_sev_info(dst_kvm); struct kvm_sev_info *src_sev =3D to_kvm_sev_info(src_kvm); @@ -2473,6 +2478,7 @@ static int snp_launch_update(struct kvm *kvm, struct = kvm_sev_cmd *argp) * situations. */ guard(mutex)(&kvm->slots_lock); + __assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */ =20 memslot =3D gfn_to_memslot(kvm, params.gfn_start); if (!kvm_slot_has_gmem(memslot)) @@ -2503,6 +2509,7 @@ static int snp_launch_update(struct kvm *kvm, struct = kvm_sev_cmd *argp) } =20 static int snp_launch_update_vmsa(struct kvm *kvm, struct kvm_sev_cmd *arg= p) + __must_hold(&kvm->lock) { struct kvm_sev_info *sev =3D to_kvm_sev_info(kvm); struct sev_data_snp_launch_update data =3D {}; @@ -2560,6 +2567,7 @@ static int snp_launch_update_vmsa(struct kvm *kvm, st= ruct kvm_sev_cmd *argp) } =20 static int snp_launch_finish(struct kvm *kvm, struct kvm_sev_cmd *argp) + __must_hold(&kvm->lock) { struct kvm_sev_info *sev =3D to_kvm_sev_info(kvm); struct kvm_sev_snp_launch_finish params; @@ -3622,6 +3630,7 @@ int pre_sev_run(struct vcpu_svm *svm, int cpu) =20 #define GHCB_SCRATCH_AREA_LIMIT (16ULL * PAGE_SIZE) static int setup_vmgexit_scratch(struct vcpu_svm *svm, bool sync, u64 min_= len) + __must_hold_shared(&svm->vcpu.kvm->srcu) { struct vmcb_control_area *control =3D &svm->vmcb->control; u64 ghcb_scratch_beg, ghcb_scratch_end; @@ -4036,6 +4045,7 @@ static void __sev_snp_reload_vmsa(struct kvm_vcpu *vc= pu, gpa_t gpa) kvm_pfn_t pfn; =20 lockdep_assert_held(&svm->sev_es.snp_vmsa_mutex); + lockdep_assert_held(&vcpu->kvm->srcu); =20 /* * Clear use of the VMSA. Ensure snp_guest_vmsa_gpa is written exactly @@ -4216,6 +4226,7 @@ static int sev_snp_ap_creation(struct vcpu_svm *svm) } =20 static int snp_handle_guest_req(struct vcpu_svm *svm, gpa_t req_gpa, gpa_t= resp_gpa) + __must_hold_shared(&svm->vcpu.kvm->srcu) { struct sev_data_snp_guest_request data =3D {0}; struct kvm *kvm =3D svm->vcpu.kvm; @@ -4262,10 +4273,13 @@ static int snp_req_certs_err(struct vcpu_svm *svm, = u32 vmm_error) } =20 static int snp_complete_req_certs(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_svm *svm =3D to_svm(vcpu); struct vmcb_control_area *control =3D &svm->vmcb->control; =20 + lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm =3D=3D vcpu->= kvm */ + switch (READ_ONCE(vcpu->run->snp_req_certs.ret)) { case 0: return snp_handle_guest_req(svm, control->exit_info_1, @@ -4285,6 +4299,7 @@ static int snp_complete_req_certs(struct kvm_vcpu *vc= pu) } =20 static int snp_handle_ext_guest_req(struct vcpu_svm *svm, gpa_t req_gpa, g= pa_t resp_gpa) + __must_hold_shared(&svm->vcpu.kvm->srcu) { struct kvm_vcpu *vcpu =3D &svm->vcpu; struct kvm *kvm =3D vcpu->kvm; @@ -4347,6 +4362,7 @@ static int snp_handle_ext_guest_req(struct vcpu_svm *= svm, gpa_t req_gpa, gpa_t r } =20 static int sev_handle_vmgexit_msr_protocol(struct vcpu_svm *svm) + __must_hold_shared(&svm->vcpu.kvm->srcu) { struct vmcb_control_area *control =3D &svm->vmcb->control; struct kvm_vcpu *vcpu =3D &svm->vcpu; @@ -4509,6 +4525,8 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu) struct vmcb_control_area *control =3D &svm->vmcb->control; u64 ghcb_gpa; =20 + lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm =3D=3D vcpu->= kvm */ + /* Validate the GHCB */ ghcb_gpa =3D control->ghcb_gpa; if (ghcb_gpa & GHCB_MSR_INFO_MASK) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 7d59d301e1e5..c20b6a9fe8dd 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -1432,10 +1432,13 @@ static void svm_srso_vm_destroy(void) { } #endif =20 static void svm_prepare_switch_to_guest(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_svm *svm =3D to_svm(vcpu); struct svm_cpu_data *sd =3D per_cpu_ptr(&svm_data, vcpu->cpu); =20 + lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm =3D=3D vcpu->= kvm */ + if (is_sev_es_guest(vcpu)) sev_es_unmap_ghcb(svm); =20 @@ -1533,6 +1536,7 @@ static bool svm_get_if_flag(struct kvm_vcpu *vcpu) } =20 static void svm_cache_reg(struct kvm_vcpu *vcpu, enum kvm_reg reg) + __must_hold_shared(&vcpu->kvm->srcu) { kvm_register_mark_available(vcpu, reg); =20 @@ -1972,6 +1976,7 @@ static int svm_check_emulate_instruction(struct kvm_v= cpu *vcpu, int emul_type, void *insn, int insn_len); =20 static int npf_interception(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_svm *svm =3D to_svm(vcpu); int rc; @@ -2094,6 +2099,7 @@ static int icebp_interception(struct kvm_vcpu *vcpu) } =20 static int ud_interception(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { return handle_ud(vcpu); } @@ -2193,12 +2199,15 @@ static int shutdown_interception(struct kvm_vcpu *v= cpu) } =20 static int io_interception(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_svm *svm =3D to_svm(vcpu); u32 io_info =3D svm->vmcb->control.exit_info_1; /* address size bug? */ int size, in, string; unsigned port; =20 + lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm =3D=3D vcpu->= kvm */ + ++vcpu->stat.io_exits; string =3D (io_info & SVM_IOIO_STR_MASK) !=3D 0; in =3D (io_info & SVM_IOIO_TYPE_MASK) !=3D 0; @@ -2279,6 +2288,7 @@ static int vmsave_interception(struct kvm_vcpu *vcpu) } =20 static int vmrun_interception(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { if (nested_svm_check_permissions(vcpu)) return 1; @@ -2319,11 +2329,14 @@ static u64 svm_get_decoded_instr_exit_code(struct k= vm_vcpu *vcpu) * 2) VMware backdoor */ static int gp_interception(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_svm *svm =3D to_svm(vcpu); u32 error_code =3D svm->vmcb->control.exit_info_1; u64 svm_exit_code; =20 + lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm =3D=3D vcpu->= kvm */ + /* Both #GP cases have zero error_code */ if (error_code) goto reinject; @@ -2562,11 +2575,14 @@ static int rsm_interception(struct kvm_vcpu *vcpu) =20 static bool check_selective_cr0_intercepted(struct kvm_vcpu *vcpu, unsigned long val) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_svm *svm =3D to_svm(vcpu); unsigned long cr0 =3D vcpu->arch.cr0; bool ret =3D false; =20 + lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm =3D=3D vcpu->= kvm */ + if (!is_guest_mode(vcpu) || (!(vmcb12_is_intercept(&svm->nested.ctl, INTERCEPT_SELECTIVE_CR0)))) return false; @@ -2585,6 +2601,7 @@ static bool check_selective_cr0_intercepted(struct kv= m_vcpu *vcpu, #define CR_VALID (1ULL << 63) =20 static int cr_interception(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_svm *svm =3D to_svm(vcpu); int reg, cr; @@ -2733,6 +2750,7 @@ static int dr_interception(struct kvm_vcpu *vcpu) } =20 static int cr8_write_interception(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { u8 cr8_prev =3D kvm_get_cr8(vcpu); int r; @@ -2750,6 +2768,7 @@ static int cr8_write_interception(struct kvm_vcpu *vc= pu) } =20 static int efer_trap(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct msr_data msr_info; int ret; @@ -2988,6 +3007,7 @@ static int svm_set_vm_cr(struct kvm_vcpu *vcpu, u64 d= ata) } =20 static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_svm *svm =3D to_svm(vcpu); int ret =3D 0; @@ -3265,6 +3285,7 @@ static int pause_interception(struct kvm_vcpu *vcpu) } =20 static int invpcid_interception(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_svm *svm =3D to_svm(vcpu); unsigned long type; @@ -3329,6 +3350,7 @@ static int bus_lock_exit(struct kvm_vcpu *vcpu) } =20 static int vmmcall_interception(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { /* * Inject a #UD if L2 is active and the VMMCALL isn't a Hyper-V TLB @@ -3713,10 +3735,13 @@ static void svm_get_entry_info(struct kvm_vcpu *vcp= u, u32 *intr_info, } =20 static int svm_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_svm *svm =3D to_svm(vcpu); struct kvm_run *kvm_run =3D vcpu->run; =20 + lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm =3D=3D vcpu->= kvm */ + if (unlikely(exit_fastpath =3D=3D EXIT_FASTPATH_EXIT_USERSPACE)) return 0; =20 @@ -4832,12 +4857,15 @@ static int svm_check_intercept(struct kvm_vcpu *vcp= u, struct x86_instruction_info *info, enum x86_intercept_stage stage, struct x86_exception *exception) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_svm *svm =3D to_svm(vcpu); int vmexit, ret =3D X86EMUL_CONTINUE; struct __x86_intercept icpt_info; struct vmcb *vmcb =3D svm->vmcb; =20 + lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm =3D=3D vcpu->= kvm */ + if (info->intercept >=3D ARRAY_SIZE(x86_intercept_map)) goto out; =20 @@ -5011,9 +5039,12 @@ static int svm_smi_allowed(struct kvm_vcpu *vcpu, bo= ol for_injection) } =20 static int svm_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_svm *svm =3D to_svm(vcpu); =20 + lockdep_assert_held(&svm->vcpu.kvm->srcu); /* svm->vcpu.kvm =3D=3D vcpu->= kvm */ + if (!is_guest_mode(vcpu)) return 0; =20 @@ -5057,6 +5088,7 @@ static int svm_enter_smm(struct kvm_vcpu *vcpu, union= kvm_smram *smram) } =20 static int svm_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smr= am) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_svm *svm =3D to_svm(vcpu); struct vmcb *vmcb12; diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index e958943b8162..e62700fc678d 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -818,7 +818,8 @@ bool svm_smi_blocked(struct kvm_vcpu *vcpu); bool svm_nmi_blocked(struct kvm_vcpu *vcpu); bool svm_interrupt_blocked(struct kvm_vcpu *vcpu); void svm_set_gif(struct vcpu_svm *svm, bool value); -int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code); +int svm_invoke_exit_handler(struct kvm_vcpu *vcpu, u64 exit_code) + __must_hold_shared(&vcpu->kvm->srcu); void set_msr_interception(struct kvm_vcpu *vcpu, u32 *msrpm, u32 msr, int read, int write); void svm_complete_interrupt_delivery(struct kvm_vcpu *vcpu, int delivery_m= ode, @@ -870,17 +871,21 @@ static inline bool nested_exit_on_nmi(struct vcpu_svm= *svm) =20 int __init nested_svm_init_msrpm_merge_offsets(void); =20 -int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb_gpa, bool from_vm= run); +int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb_gpa, bool from_vm= run) + __must_hold_shared(&vcpu->kvm->srcu); void svm_leave_nested(struct kvm_vcpu *vcpu); void svm_free_nested(struct vcpu_svm *svm); int svm_allocate_nested(struct vcpu_svm *svm); -int nested_svm_vmrun(struct kvm_vcpu *vcpu); +int nested_svm_vmrun(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu); void svm_copy_vmrun_state(struct vmcb_save_area *to_save, struct vmcb_save_area *from_save); void svm_copy_vmloadsave_state(struct vmcb *to_vmcb, struct vmcb *from_vmc= b); -void nested_svm_vmexit(struct vcpu_svm *svm); +void nested_svm_vmexit(struct vcpu_svm *svm) + __must_hold_shared(&svm->vcpu.kvm->srcu); =20 static inline void nested_svm_simple_vmexit(struct vcpu_svm *svm, u32 exit= _code) + __must_hold_shared(&svm->vcpu.kvm->srcu) { svm->vmcb->control.exit_code =3D exit_code; svm->vmcb->control.exit_info_1 =3D 0; @@ -888,7 +893,8 @@ static inline void nested_svm_simple_vmexit(struct vcpu= _svm *svm, u32 exit_code) nested_svm_vmexit(svm); } =20 -int nested_svm_exit_handled(struct vcpu_svm *svm); +int nested_svm_exit_handled(struct vcpu_svm *svm) + __must_hold_shared(&svm->vcpu.kvm->srcu); int nested_svm_check_permissions(struct kvm_vcpu *vcpu); int nested_svm_check_cached_vmcb12(struct kvm_vcpu *vcpu); int nested_svm_check_exception(struct vcpu_svm *svm, unsigned nr, @@ -973,11 +979,13 @@ void avic_refresh_virtual_apic_mode(struct kvm_vcpu *= vcpu); int pre_sev_run(struct vcpu_svm *svm, int cpu); void sev_init_vmcb(struct vcpu_svm *svm, bool init_event); void sev_vcpu_after_set_cpuid(struct vcpu_svm *svm); -int sev_es_string_io(struct vcpu_svm *svm, int size, unsigned int port, in= t in); +int sev_es_string_io(struct vcpu_svm *svm, int size, unsigned int port, in= t in) + __must_hold_shared(&svm->vcpu.kvm->srcu); void sev_es_recalc_msr_intercepts(struct kvm_vcpu *vcpu); void sev_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector); void sev_es_prepare_switch_to_guest(struct vcpu_svm *svm, struct sev_es_sa= ve_area *hostsa); -void sev_es_unmap_ghcb(struct vcpu_svm *svm); +void sev_es_unmap_ghcb(struct vcpu_svm *svm) + __must_hold_shared(&svm->vcpu.kvm->srcu); =20 #ifdef CONFIG_KVM_AMD_SEV bool sev_vcpu_needs_initialization(struct kvm_vcpu *vcpu); @@ -989,7 +997,8 @@ int sev_mem_enc_unregister_region(struct kvm *kvm, int sev_vm_copy_enc_context_from(struct kvm *kvm, unsigned int source_fd); int sev_vm_move_enc_context_from(struct kvm *kvm, unsigned int source_fd); void sev_guest_memory_reclaimed(struct kvm *kvm); -int sev_handle_vmgexit(struct kvm_vcpu *vcpu); +int sev_handle_vmgexit(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu); =20 /* These symbols are used in common code and are stubbed below. */ =20 @@ -1010,7 +1019,8 @@ void sev_hardware_unsetup(void); int sev_cpu_init(struct svm_cpu_data *sd); int sev_dev_get_attr(u32 group, u64 attr, u64 *val); extern unsigned int max_sev_asid; -void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code= ); +void sev_handle_rmp_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code) + __must_hold_shared(&vcpu->kvm->srcu); int sev_gmem_make_private(struct kvm *kvm, gfn_t gfn, kvm_pfn_t pfn, kvm_p= fn_t nr_pages); void sev_gmem_make_shared(kvm_pfn_t pfn, kvm_pfn_t nr_pages); void sev_gmem_invalidate_range(struct kvm *kvm, struct kvm_gfn_range *rang= e); diff --git a/arch/x86/kvm/vmx/hyperv.h b/arch/x86/kvm/vmx/hyperv.h index 11a339009781..a0e126d2a5b4 100644 --- a/arch/x86/kvm/vmx/hyperv.h +++ b/arch/x86/kvm/vmx/hyperv.h @@ -52,14 +52,16 @@ static inline bool guest_cpu_cap_has_evmcs(struct kvm_v= cpu *vcpu) to_vmx(vcpu)->nested.enlightened_vmcs_enabled; } =20 -u64 nested_get_evmptr(struct kvm_vcpu *vcpu); +u64 nested_get_evmptr(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu); uint16_t nested_get_evmcs_version(struct kvm_vcpu *vcpu); int nested_enable_evmcs(struct kvm_vcpu *vcpu, uint16_t *vmcs_version); void nested_evmcs_filter_control_msr(struct kvm_vcpu *vcpu, u32 msr_index,= u64 *pdata); int nested_evmcs_check_controls(struct vmcs12 *vmcs12); bool nested_evmcs_l2_tlb_flush_enabled(struct kvm_vcpu *vcpu); -void vmx_hv_inject_synthetic_vmexit_post_tlb_flush(struct kvm_vcpu *vcpu); +void vmx_hv_inject_synthetic_vmexit_post_tlb_flush(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu); #else static inline bool evmptr_is_valid(u64 evmptr) { diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 4c52ab8d0786..c2a0c259ceea 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -156,6 +156,7 @@ static fastpath_t vt_vcpu_run(struct kvm_vcpu *vcpu, u6= 4 run_flags) =20 static int vt_handle_exit(struct kvm_vcpu *vcpu, enum exit_fastpath_completion fastpath) + __must_hold_shared(&vcpu->kvm->srcu) { if (is_td_vcpu(vcpu)) return tdx_handle_exit(vcpu, fastpath); @@ -174,6 +175,7 @@ static bool vt_unhandleable_emulation_required(struct k= vm_vcpu *vcpu) } =20 static int vt_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) + __must_hold_shared(&vcpu->kvm->srcu) { if (unlikely(is_td_vcpu(vcpu))) return tdx_set_msr(vcpu, msr_info); @@ -233,6 +235,7 @@ static int vt_smi_allowed(struct kvm_vcpu *vcpu, bool f= or_injection) } =20 static int vt_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram) + __must_hold_shared(&vcpu->kvm->srcu) { if (KVM_BUG_ON(is_td_vcpu(vcpu), vcpu->kvm)) return 0; @@ -241,6 +244,7 @@ static int vt_enter_smm(struct kvm_vcpu *vcpu, union kv= m_smram *smram) } =20 static int vt_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smra= m) + __must_hold_shared(&vcpu->kvm->srcu) { if (KVM_BUG_ON(is_td_vcpu(vcpu), vcpu->kvm)) return 0; @@ -722,6 +726,7 @@ static void vt_update_cr8_intercept(struct kvm_vcpu *vc= pu, int tpr, int irr) } =20 static void vt_set_apic_access_page_addr(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { if (is_td_vcpu(vcpu)) return; diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 151873407abd..506e206bad5d 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -250,6 +250,7 @@ static inline void nested_release_evmcs(struct kvm_vcpu= *vcpu) } =20 static bool nested_evmcs_handle_vmclear(struct kvm_vcpu *vcpu, gpa_t vmptr) + __must_hold_shared(&vcpu->kvm->srcu) { #ifdef CONFIG_KVM_HYPERV struct vcpu_vmx *vmx =3D to_vmx(vcpu); @@ -386,7 +387,8 @@ static void free_nested(struct kvm_vcpu *vcpu) void nested_vmx_free_vcpu(struct kvm_vcpu *vcpu) { vcpu_load(vcpu); - vmx_leave_nested(vcpu); + /* VM is being destroyed; vcpu->kvm->srcu is not held. */ + context_unsafe(vmx_leave_nested(vcpu)); vcpu_put(vcpu); } =20 @@ -421,6 +423,7 @@ static void nested_ept_invalidate_addr(struct kvm_vcpu = *vcpu, gpa_t eptp, static void nested_ept_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fault, bool from_hardware) + __must_hold_shared(&vcpu->kvm->srcu) { struct vmcs12 *vmcs12 =3D get_vmcs12(vcpu); struct vcpu_vmx *vmx =3D to_vmx(vcpu); @@ -587,6 +590,7 @@ static int nested_vmx_check_msr_bitmap_controls(struct = kvm_vcpu *vcpu, =20 static int nested_vmx_check_tpr_shadow_controls(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12) + __must_hold_shared(&vcpu->kvm->srcu) { gpa_t vtpr_gpa =3D vmcs12->virtual_apic_page_addr + APIC_TASKPRI; u32 vtpr; @@ -850,6 +854,7 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct= kvm_vcpu *vcpu, =20 static void nested_cache_shadow_vmcs12(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); struct gfn_to_hva_cache *ghc =3D &vmx->nested.shadow_vmcs12_cache; @@ -869,6 +874,7 @@ static void nested_cache_shadow_vmcs12(struct kvm_vcpu = *vcpu, =20 static void nested_flush_cached_shadow_vmcs12(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); struct gfn_to_hva_cache *ghc =3D &vmx->nested.shadow_vmcs12_cache; @@ -1095,6 +1101,7 @@ static int nested_vmx_store_msr_check(struct kvm_vcpu= *vcpu, * for a capacity violation. */ static u32 nested_vmx_load_msr(struct kvm_vcpu *vcpu, u64 gpa, u32 count) + __must_hold_shared(&vcpu->kvm->srcu) { u32 i; struct vmx_msr_entry e; @@ -1159,6 +1166,7 @@ static bool nested_vmx_get_vmexit_msr_value(struct kv= m_vcpu *vcpu, =20 static bool read_and_check_msr_entry(struct kvm_vcpu *vcpu, u64 gpa, int i, struct vmx_msr_entry *e) + __must_hold_shared(&vcpu->kvm->srcu) { if (kvm_vcpu_read_guest(vcpu, gpa + i * sizeof(*e), @@ -1178,6 +1186,7 @@ static bool read_and_check_msr_entry(struct kvm_vcpu = *vcpu, u64 gpa, int i, } =20 static int nested_vmx_store_msr(struct kvm_vcpu *vcpu, u64 gpa, u32 count) + __must_hold_shared(&vcpu->kvm->srcu) { u64 data; u32 i; @@ -1208,6 +1217,7 @@ static int nested_vmx_store_msr(struct kvm_vcpu *vcpu= , u64 gpa, u32 count) } =20 static bool nested_msr_store_list_has_msr(struct kvm_vcpu *vcpu, u32 msr_i= ndex) + __must_hold_shared(&vcpu->kvm->srcu) { struct vmcs12 *vmcs12 =3D get_vmcs12(vcpu); u32 count =3D vmcs12->vm_exit_msr_store_count; @@ -1234,6 +1244,7 @@ static bool nested_msr_store_list_has_msr(struct kvm_= vcpu *vcpu, u32 msr_index) static int nested_vmx_load_cr3(struct kvm_vcpu *vcpu, unsigned long cr3, bool nested_ept, bool reload_pdptrs, enum vm_entry_failure_code *entry_failure_code) + __must_hold_shared(&vcpu->kvm->srcu) { if (CC(!kvm_vcpu_is_legal_cr3(vcpu, cr3))) { *entry_failure_code =3D ENTRY_FAIL_DEFAULT; @@ -2183,6 +2194,7 @@ static void copy_vmcs12_to_enlightened(struct vcpu_vm= x *vmx) */ static enum nested_evmptrld_status nested_vmx_handle_enlightened_vmptrld( struct kvm_vcpu *vcpu, bool from_launch) + __must_hold_shared(&vcpu->kvm->srcu) { #ifdef CONFIG_KVM_HYPERV struct vcpu_vmx *vmx =3D to_vmx(vcpu); @@ -2622,6 +2634,7 @@ static void vmcs_write_cet_state(struct kvm_vcpu *vcp= u, u64 s_cet, } =20 static void prepare_vmcs02_rare(struct vcpu_vmx *vmx, struct vmcs12 *vmcs1= 2) + __must_hold_shared(&vmx->vcpu.kvm->srcu) { struct hv_enlightened_vmcs *hv_evmcs =3D nested_vmx_evmcs(vmx); =20 @@ -2760,11 +2773,14 @@ static void prepare_vmcs02_rare(struct vcpu_vmx *vm= x, struct vmcs12 *vmcs12) static int prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, bool from_vmentry, enum vm_entry_failure_code *entry_failure_code) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); struct hv_enlightened_vmcs *evmcs =3D nested_vmx_evmcs(vmx); bool load_guest_pdptrs_vmcs12 =3D false; =20 + lockdep_assert_held(&vmx->vcpu.kvm->srcu); /* vmx->vcpu.kvm =3D=3D vcpu->= kvm */ + if (vmx->nested.dirty_vmcs12 || nested_vmx_is_evmptr12_valid(vmx)) { prepare_vmcs02_rare(vmx, vmcs12); vmx->nested.dirty_vmcs12 =3D false; @@ -2968,6 +2984,7 @@ static bool nested_vmx_check_eptp(struct kvm_vcpu *vc= pu, u64 new_eptp) */ static int nested_check_vm_execution_controls(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); =20 @@ -3126,6 +3143,7 @@ static int nested_check_vm_entry_controls(struct kvm_= vcpu *vcpu, =20 static int nested_vmx_check_controls(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12) + __must_hold_shared(&vcpu->kvm->srcu) { if (nested_check_vm_execution_controls(vcpu, vmcs12) || nested_check_vm_exit_controls(vcpu, vmcs12) || @@ -3267,6 +3285,7 @@ static int nested_vmx_check_host_state(struct kvm_vcp= u *vcpu, =20 static int nested_vmx_check_vmcs_link_ptr(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); struct gfn_to_hva_cache *ghc =3D &vmx->nested.shadow_vmcs12_cache; @@ -3311,6 +3330,7 @@ static int nested_check_guest_non_reg_state(struct vm= cs12 *vmcs12) static int nested_vmx_check_guest_state(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, enum vm_entry_failure_code *entry_failure_code) + __must_hold_shared(&vcpu->kvm->srcu) { bool ia32e =3D !!(vmcs12->vm_entry_controls & VM_ENTRY_IA32E_MODE); =20 @@ -3411,6 +3431,7 @@ static int nested_vmx_check_guest_state(struct kvm_vc= pu *vcpu, =20 #ifdef CONFIG_KVM_HYPERV static bool nested_get_evmcs_page(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); =20 @@ -3440,6 +3461,7 @@ static bool nested_get_evmcs_page(struct kvm_vcpu *vc= pu) #endif =20 static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vmcs12 *vmcs12 =3D get_vmcs12(vcpu); struct vcpu_vmx *vmx =3D to_vmx(vcpu); @@ -3528,6 +3550,7 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu *= vcpu) } =20 static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { #ifdef CONFIG_KVM_HYPERV /* @@ -3555,6 +3578,7 @@ static bool vmx_get_nested_state_pages(struct kvm_vcp= u *vcpu) } =20 static int nested_vmx_write_pml_buffer(struct kvm_vcpu *vcpu, gpa_t gpa) + __must_hold_shared(&vcpu->kvm->srcu) { struct vmcs12 *vmcs12; struct vcpu_vmx *vmx =3D to_vmx(vcpu); @@ -3790,6 +3814,7 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mo= de(struct kvm_vcpu *vcpu, * for running an L2 nested guest. */ static int nested_vmx_run(struct kvm_vcpu *vcpu, bool launch) + __must_hold_shared(&vcpu->kvm->srcu) { struct vmcs12 *vmcs12; enum nvmx_vmentry_status status; @@ -4027,6 +4052,7 @@ static void vmcs12_save_pending_event(struct kvm_vcpu= *vcpu, } =20 static int vmx_complete_nested_posted_interrupt(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); int max_irr; @@ -4070,6 +4096,7 @@ static int vmx_complete_nested_posted_interrupt(struc= t kvm_vcpu *vcpu) } =20 static void nested_vmx_inject_exception_vmexit(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct kvm_queued_exception *ex =3D &vcpu->arch.exception_vmexit; u32 intr_info =3D ex->vector | INTR_INFO_VALID_MASK; @@ -4303,6 +4330,7 @@ static bool vmx_has_nested_events(struct kvm_vcpu *vc= pu, bool for_injection) * priority over external interrupts and lower priority events. */ static int vmx_check_nested_events(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct kvm_lapic *apic =3D vcpu->arch.apic; struct vcpu_vmx *vmx =3D to_vmx(vcpu); @@ -4744,6 +4772,7 @@ static void sync_vmcs02_to_vmcs12(struct kvm_vcpu *vc= pu, struct vmcs12 *vmcs12) static void prepare_vmcs12(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, u32 vm_exit_reason, u32 exit_intr_info, unsigned long exit_qualification, u32 exit_insn_len) + __must_hold_shared(&vcpu->kvm->srcu) { /* update exit information fields: */ vmcs12->vm_exit_reason =3D vm_exit_reason; @@ -4799,6 +4828,7 @@ static void prepare_vmcs12(struct kvm_vcpu *vcpu, str= uct vmcs12 *vmcs12, */ static void load_vmcs12_host_state(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12) + __must_hold_shared(&vcpu->kvm->srcu) { enum vm_entry_failure_code ignored; struct kvm_segment seg; @@ -4958,6 +4988,7 @@ static inline u64 nested_vmx_get_vmcs01_guest_efer(st= ruct vcpu_vmx *vmx) } =20 static void nested_vmx_restore_host_state(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vmcs12 *vmcs12 =3D get_vmcs12(vcpu); struct vcpu_vmx *vmx =3D to_vmx(vcpu); @@ -5230,6 +5261,7 @@ void __nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 v= m_exit_reason, } =20 static void nested_vmx_triple_fault(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { kvm_clear_request(KVM_REQ_TRIPLE_FAULT, vcpu); nested_vmx_vmexit(vcpu, EXIT_REASON_TRIPLE_FAULT, 0, 0); @@ -5368,6 +5400,7 @@ int get_vmx_mem_address(struct kvm_vcpu *vcpu, unsign= ed long exit_qualification, =20 static int nested_vmx_get_vmptr(struct kvm_vcpu *vcpu, gpa_t *vmpointer, int *ret) + __must_hold_shared(&vcpu->kvm->srcu) { gva_t gva; struct x86_exception e; @@ -5475,6 +5508,7 @@ static int enter_vmx_operation(struct kvm_vcpu *vcpu) =20 /* Emulate the VMXON instruction. */ static int handle_vmxon(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { int ret; gpa_t vmptr; @@ -5566,6 +5600,7 @@ static int handle_vmxon(struct kvm_vcpu *vcpu) } =20 static inline void nested_release_vmcs12(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); =20 @@ -5608,6 +5643,7 @@ static int handle_vmxoff(struct kvm_vcpu *vcpu) =20 /* Emulate the VMCLEAR instruction */ static int handle_vmclear(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); u32 zero =3D 0; @@ -5650,18 +5686,20 @@ static int handle_vmclear(struct kvm_vcpu *vcpu) =20 /* Emulate the VMLAUNCH instruction */ static int handle_vmlaunch(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { return nested_vmx_run(vcpu, true); } =20 /* Emulate the VMRESUME instruction */ static int handle_vmresume(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { - return nested_vmx_run(vcpu, false); } =20 static int handle_vmread(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vmcs12 *vmcs12 =3D is_guest_mode(vcpu) ? get_shadow_vmcs12(vcpu) : get_vmcs12(vcpu); @@ -5768,6 +5806,7 @@ static bool is_shadow_field_ro(unsigned long field) } =20 static int handle_vmwrite(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vmcs12 *vmcs12 =3D is_guest_mode(vcpu) ? get_shadow_vmcs12(vcpu) : get_vmcs12(vcpu); @@ -5889,6 +5928,7 @@ static void set_current_vmptr(struct vcpu_vmx *vmx, g= pa_t vmptr) =20 /* Emulate the VMPTRLD instruction */ static int handle_vmptrld(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); gpa_t vmptr; @@ -5959,6 +5999,7 @@ static int handle_vmptrld(struct kvm_vcpu *vcpu) =20 /* Emulate the VMPTRST instruction */ static int handle_vmptrst(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { unsigned long exit_qual =3D vmx_get_exit_qual(vcpu); u32 instr_info =3D vmcs_read32(VMX_INSTRUCTION_INFO); @@ -5987,6 +6028,7 @@ static int handle_vmptrst(struct kvm_vcpu *vcpu) =20 /* Emulate the INVEPT instruction */ static int handle_invept(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); u32 vmx_instruction_info, types; @@ -6067,6 +6109,7 @@ static int handle_invept(struct kvm_vcpu *vcpu) } =20 static int handle_invvpid(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); u32 vmx_instruction_info; @@ -6159,6 +6202,7 @@ static int handle_invvpid(struct kvm_vcpu *vcpu) =20 static int nested_vmx_eptp_switching(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12) + __must_hold_shared(&vcpu->kvm->srcu) { u32 index =3D kvm_ecx_read(vcpu); u64 new_eptp; @@ -6191,6 +6235,7 @@ static int nested_vmx_eptp_switching(struct kvm_vcpu = *vcpu, } =20 static int handle_vmfunc(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); struct vmcs12 *vmcs12; @@ -6280,6 +6325,7 @@ bool nested_vmx_check_io_bitmaps(struct kvm_vcpu *vcp= u, unsigned int port, =20 static bool nested_vmx_exit_handled_io(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12) + __must_hold_shared(&vcpu->kvm->srcu) { unsigned long exit_qualification; unsigned short port; @@ -6305,6 +6351,7 @@ static bool nested_vmx_exit_handled_io(struct kvm_vcp= u *vcpu, static bool nested_vmx_exit_handled_msr(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, union vmx_exit_reason exit_reason) + __must_hold_shared(&vcpu->kvm->srcu) { u32 msr_index; gpa_t bitmap; @@ -6434,6 +6481,7 @@ static bool nested_vmx_exit_handled_encls(struct kvm_= vcpu *vcpu, =20 static bool nested_vmx_exit_handled_vmcs_access(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12, gpa_t bitmap) + __must_hold_shared(&vcpu->kvm->srcu) { u32 vmx_instruction_info; unsigned long field; @@ -6561,6 +6609,7 @@ static bool nested_vmx_l0_wants_exit(struct kvm_vcpu = *vcpu, */ static bool nested_vmx_l1_wants_exit(struct kvm_vcpu *vcpu, union vmx_exit_reason exit_reason) + __must_hold_shared(&vcpu->kvm->srcu) { struct vmcs12 *vmcs12 =3D get_vmcs12(vcpu); u32 intr_info; @@ -6908,6 +6957,7 @@ int nested_vmx_check_restored_vmcs12(struct kvm_vcpu = *vcpu) static int vmx_set_nested_state(struct kvm_vcpu *vcpu, struct kvm_nested_state __user *user_kvm_nested_state, struct kvm_nested_state *kvm_state) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); struct vmcs12 *vmcs12; diff --git a/arch/x86/kvm/vmx/nested.h b/arch/x86/kvm/vmx/nested.h index c6de848bd9ce..3b8b6579c104 100644 --- a/arch/x86/kvm/vmx/nested.h +++ b/arch/x86/kvm/vmx/nested.h @@ -17,23 +17,29 @@ enum nvmx_vmentry_status { NVMX_VMENTRY_KVM_INTERNAL_ERROR,/* KVM internal error */ }; =20 -void vmx_leave_nested(struct kvm_vcpu *vcpu); +void vmx_leave_nested(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu); void nested_vmx_setup_ctls_msrs(struct vmcs_config *vmcs_conf, u32 ept_cap= s); void nested_vmx_hardware_unsetup(void); __init int nested_vmx_hardware_setup(int (*exit_handlers[])(struct kvm_vcp= u *)); void nested_vmx_set_vmcs_shadowing_bitmap(void); -int nested_vmx_check_restored_vmcs12(struct kvm_vcpu *vcpu); +int nested_vmx_check_restored_vmcs12(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu); void nested_vmx_free_vcpu(struct kvm_vcpu *vcpu); enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *v= cpu, - bool from_vmentry); -bool nested_vmx_reflect_vmexit(struct kvm_vcpu *vcpu); + bool from_vmentry) + __must_hold_shared(&vcpu->kvm->srcu); +bool nested_vmx_reflect_vmexit(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu); void __nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_reason, u32 exit_intr_info, unsigned long exit_qualification, - u32 exit_insn_len); + u32 exit_insn_len) + __must_hold_shared(&vcpu->kvm->srcu); =20 static inline void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 vm_exit_re= ason, u32 exit_intr_info, unsigned long exit_qualification) + __must_hold_shared(&vcpu->kvm->srcu) { u32 exit_insn_len; =20 @@ -53,7 +59,8 @@ int vmx_get_vmx_msr(struct nested_vmx_msrs *msrs, u32 msr= _index, u64 *pdata); int get_vmx_mem_address(struct kvm_vcpu *vcpu, unsigned long exit_qualific= ation, u32 vmx_instruction_info, bool wr, int len, gva_t *ret); bool nested_vmx_check_io_bitmaps(struct kvm_vcpu *vcpu, unsigned int port, - int size); + int size) + __must_hold_shared(&vcpu->kvm->srcu); =20 static inline struct vmcs12 *get_vmcs12(struct kvm_vcpu *vcpu) { diff --git a/arch/x86/kvm/vmx/sgx.c b/arch/x86/kvm/vmx/sgx.c index 771c75a58343..588cc8ae1f8b 100644 --- a/arch/x86/kvm/vmx/sgx.c +++ b/arch/x86/kvm/vmx/sgx.c @@ -91,6 +91,7 @@ static int sgx_gva_to_gpa(struct kvm_vcpu *vcpu, gva_t gv= a, bool write, } =20 static int sgx_gpa_to_hva(struct kvm_vcpu *vcpu, gpa_t gpa, unsigned long = *hva) + __must_hold_shared(&vcpu->kvm->srcu) { *hva =3D kvm_vcpu_gfn_to_hva(vcpu, PFN_DOWN(gpa)); if (kvm_is_error_hva(*hva)) { @@ -216,6 +217,7 @@ static int __handle_encls_ecreate(struct kvm_vcpu *vcpu, } =20 static int handle_encls_ecreate(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { gva_t pageinfo_gva, secs_gva; gva_t metadata_gva, contents_gva; @@ -296,6 +298,7 @@ static int handle_encls_ecreate(struct kvm_vcpu *vcpu) } =20 static int handle_encls_einit(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { unsigned long sig_hva, secs_hva, token_hva, rflags; struct vcpu_vmx *vmx =3D to_vmx(vcpu); diff --git a/arch/x86/kvm/vmx/sgx.h b/arch/x86/kvm/vmx/sgx.h index a400888b376d..08b1d5e90aa8 100644 --- a/arch/x86/kvm/vmx/sgx.h +++ b/arch/x86/kvm/vmx/sgx.h @@ -10,7 +10,8 @@ #ifdef CONFIG_X86_SGX_KVM extern bool __read_mostly enable_sgx; =20 -int handle_encls(struct kvm_vcpu *vcpu); +int handle_encls(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu); =20 void setup_default_sgx_lepubkeyhash(void); void vcpu_setup_sgx_lepubkeyhash(struct kvm_vcpu *vcpu); diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index b272c20586a7..7cf53c5dc712 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1155,6 +1155,7 @@ static int complete_hypercall_exit(struct kvm_vcpu *v= cpu) } =20 static int tdx_emulate_vmcall(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { kvm_rax_write_raw(vcpu, to_tdx(vcpu)->vp_enter_args.r10); kvm_rbx_write_raw(vcpu, to_tdx(vcpu)->vp_enter_args.r11); @@ -1408,6 +1409,7 @@ static int tdx_complete_mmio_read(struct kvm_vcpu *vc= pu) =20 static inline int tdx_mmio_write(struct kvm_vcpu *vcpu, gpa_t gpa, int siz= e, unsigned long val) + __must_hold_shared(&vcpu->kvm->srcu) { if (!kvm_io_bus_write(vcpu, KVM_FAST_MMIO_BUS, gpa, 0, NULL)) { trace_kvm_fast_mmio(gpa); @@ -1422,6 +1424,7 @@ static inline int tdx_mmio_write(struct kvm_vcpu *vcp= u, gpa_t gpa, int size, } =20 static inline int tdx_mmio_read(struct kvm_vcpu *vcpu, gpa_t gpa, int size) + __must_hold_shared(&vcpu->kvm->srcu) { unsigned long val; =20 @@ -1434,6 +1437,7 @@ static inline int tdx_mmio_read(struct kvm_vcpu *vcpu= , gpa_t gpa, int size) } =20 static int tdx_emulate_mmio(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_tdx *tdx =3D to_tdx(vcpu); int size, write, r; @@ -1912,6 +1916,7 @@ static inline bool tdx_is_sept_violation_unexpected_p= ending(struct kvm_vcpu *vcp } =20 static int tdx_handle_ept_violation(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { unsigned long exit_qual; gpa_t gpa =3D to_tdx(vcpu)->exit_gpa; @@ -2723,6 +2728,7 @@ static int tdx_read_cpuid(struct kvm_vcpu *vcpu, u32 = leaf, u32 sub_leaf, typedef void *tdx_vm_state_guard_t; =20 static tdx_vm_state_guard_t tdx_acquire_vm_state_locks(struct kvm *kvm) + __context_unsafe(/* multi-lock acquisition */) { int r; =20 @@ -2750,6 +2756,7 @@ static tdx_vm_state_guard_t tdx_acquire_vm_state_lock= s(struct kvm *kvm) } =20 static void tdx_release_vm_state_locks(struct kvm *kvm) + __context_unsafe(/* multi-lock release */) { mutex_unlock(&kvm->slots_lock); kvm_unlock_all_vcpus(kvm); @@ -3213,6 +3220,7 @@ static int tdx_gmem_post_populate(struct kvm *kvm, gf= n_t gfn, kvm_pfn_t pfn, } =20 static int tdx_vcpu_init_mem_region(struct kvm_vcpu *vcpu, struct kvm_tdx_= cmd *cmd) + __must_hold(&vcpu->kvm->slots_lock) { struct vcpu_tdx *tdx =3D to_tdx(vcpu); struct kvm *kvm =3D vcpu->kvm; @@ -3292,6 +3300,7 @@ int tdx_vcpu_unlocked_ioctl(struct kvm_vcpu *vcpu, vo= id __user *argp) CLASS(tdx_vm_state_guard, guard)(kvm); if (IS_ERR(guard)) return PTR_ERR(guard); + lockdep_assert_held(&kvm->slots_lock); =20 if (!is_hkid_assigned(kvm_tdx) || kvm_tdx->state =3D=3D TD_STATE_RUNNABLE) return -EINVAL; diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 612ab07d4100..ddf5c05955e1 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -5429,6 +5429,7 @@ static int vmx_handle_page_fault(struct kvm_vcpu *vcp= u, u32 error_code) } =20 static int handle_exception_nmi(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); struct kvm_run *kvm_run =3D vcpu->run; @@ -5589,6 +5590,7 @@ static int handle_triple_fault(struct kvm_vcpu *vcpu) } =20 static int handle_io(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { unsigned long exit_qualification; int size, in, string; @@ -5621,6 +5623,7 @@ void vmx_patch_hypercall(struct kvm_vcpu *vcpu, unsig= ned char *hypercall) =20 /* called to set cr0 as appropriate for a mov-to-cr0 exit. */ static int handle_set_cr0(struct kvm_vcpu *vcpu, unsigned long val) + __must_hold_shared(&vcpu->kvm->srcu) { if (is_guest_mode(vcpu)) { struct vmcs12 *vmcs12 =3D get_vmcs12(vcpu); @@ -5647,6 +5650,7 @@ static int handle_set_cr0(struct kvm_vcpu *vcpu, unsi= gned long val) } =20 static int handle_set_cr4(struct kvm_vcpu *vcpu, unsigned long val) + __must_hold_shared(&vcpu->kvm->srcu) { if (is_guest_mode(vcpu)) { struct vmcs12 *vmcs12 =3D get_vmcs12(vcpu); @@ -5676,6 +5680,7 @@ static int handle_desc(struct kvm_vcpu *vcpu) } =20 static int handle_cr(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { unsigned long exit_qualification, val; int cr; @@ -6003,6 +6008,7 @@ static int handle_ept_violation(struct kvm_vcpu *vcpu) } =20 static int handle_ept_misconfig(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { gpa_t gpa; =20 @@ -6140,6 +6146,7 @@ static int handle_monitor_trap(struct kvm_vcpu *vcpu) } =20 static int handle_invpcid(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { u32 vmx_instruction_info; unsigned long type; @@ -6427,6 +6434,7 @@ static void vmx_destroy_pml_buffer(struct vcpu_vmx *v= mx) } =20 static void vmx_flush_pml_buffer(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); u16 pml_idx, pml_tail_index; @@ -6466,6 +6474,7 @@ static void vmx_flush_pml_buffer(struct kvm_vcpu *vcp= u) } =20 static void nested_vmx_mark_all_vmcs12_pages_dirty(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); =20 @@ -6706,6 +6715,7 @@ void dump_vmcs(struct kvm_vcpu *vcpu) * assistance. */ static int __vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpa= th) + __must_hold_shared(&vcpu->kvm->srcu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); union vmx_exit_reason exit_reason =3D vmx_get_exit_reason(vcpu); @@ -8144,6 +8154,7 @@ static __init void vmx_set_cpu_caps(void) static bool vmx_is_io_intercepted(struct kvm_vcpu *vcpu, struct x86_instruction_info *info, unsigned long *exit_qualification) + __must_hold_shared(&vcpu->kvm->srcu) { struct vmcs12 *vmcs12 =3D get_vmcs12(vcpu); unsigned short port; diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index 054fd14bb2e1..f92ae9e60211 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -26,16 +26,20 @@ void vmx_vcpu_free(struct kvm_vcpu *vcpu); void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event); void vmx_vcpu_load(struct kvm_vcpu *vcpu, int cpu); void vmx_vcpu_put(struct kvm_vcpu *vcpu); -int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath); +int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath) + __must_hold_shared(&vcpu->kvm->srcu); void vmx_handle_exit_irqoff(struct kvm_vcpu *vcpu); int vmx_skip_emulated_instruction(struct kvm_vcpu *vcpu); void vmx_update_emulated_instruction(struct kvm_vcpu *vcpu); bool vmx_unhandleable_emulation_required(struct kvm_vcpu *vcpu); -int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info); +int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) + __must_hold_shared(&vcpu->kvm->srcu); #ifdef CONFIG_KVM_SMM int vmx_smi_allowed(struct kvm_vcpu *vcpu, bool for_injection); -int vmx_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram); -int vmx_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smram); +int vmx_enter_smm(struct kvm_vcpu *vcpu, union kvm_smram *smram) + __must_hold_shared(&vcpu->kvm->srcu); +int vmx_leave_smm(struct kvm_vcpu *vcpu, const union kvm_smram *smram) + __must_hold_shared(&vcpu->kvm->srcu); void vmx_enable_smi_window(struct kvm_vcpu *vcpu); #endif int vmx_check_emulate_instruction(struct kvm_vcpu *vcpu, int emul_type, @@ -43,7 +47,8 @@ int vmx_check_emulate_instruction(struct kvm_vcpu *vcpu, = int emul_type, int vmx_check_intercept(struct kvm_vcpu *vcpu, struct x86_instruction_info *info, enum x86_intercept_stage stage, - struct x86_exception *exception); + struct x86_exception *exception) + __must_hold_shared(&vcpu->kvm->srcu); bool vmx_apic_init_signal_blocked(struct kvm_vcpu *vcpu); void vmx_migrate_timers(struct kvm_vcpu *vcpu); void vmx_set_virtual_apic_mode(struct kvm_vcpu *vcpu); @@ -98,7 +103,8 @@ void vmx_set_nmi_mask(struct kvm_vcpu *vcpu, bool masked= ); void vmx_enable_nmi_window(struct kvm_vcpu *vcpu); void vmx_enable_irq_window(struct kvm_vcpu *vcpu); void vmx_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr); -void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu); +void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu); void vmx_refresh_apicv_exec_ctrl(struct kvm_vcpu *vcpu); void vmx_load_eoi_exitmap(struct kvm_vcpu *vcpu, u64 *eoi_exit_bitmap); int vmx_set_tss_addr(struct kvm *kvm, unsigned int addr); @@ -142,7 +148,8 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_= flags); void tdx_prepare_switch_to_guest(struct kvm_vcpu *vcpu); void tdx_vcpu_put(struct kvm_vcpu *vcpu); int tdx_handle_exit(struct kvm_vcpu *vcpu, - enum exit_fastpath_completion fastpath); + enum exit_fastpath_completion fastpath) + __must_hold_shared(&vcpu->kvm->srcu); =20 void tdx_deliver_interrupt(struct kvm_lapic *apic, int delivery_mode, int trig_mode, int vector); @@ -151,7 +158,8 @@ void tdx_get_exit_info(struct kvm_vcpu *vcpu, u32 *reas= on, u64 *info1, u64 *info2, u32 *intr_info, u32 *error_code); bool tdx_has_emulated_msr(u32 index); int tdx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr); -int tdx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr); +int tdx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) + __must_hold_shared(&vcpu->kvm->srcu); =20 int tdx_vcpu_ioctl(struct kvm_vcpu *vcpu, void __user *argp); int tdx_vcpu_unlocked_ioctl(struct kvm_vcpu *vcpu, void __user *argp); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 79468ddfe473..9893705d0dfa 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -321,7 +321,8 @@ static struct kmem_cache *kvm_alloc_emulator_cache(void) size - useroffset, NULL); } =20 -static int emulator_fix_hypercall(struct x86_emulate_ctxt *ctxt); +static int emulator_fix_hypercall(struct x86_emulate_ctxt *ctxt) + __must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu); =20 /* * Handle a fault on a hardware virtualization (VMX or SVM) instruction. @@ -1076,6 +1077,7 @@ static inline bool gtod_is_based_on_tsc(int mode) #endif =20 static void kvm_track_tsc_matching(struct kvm_vcpu *vcpu, bool new_generat= ion) + __must_hold(&vcpu->kvm->arch.tsc_write_lock) { #ifdef CONFIG_X86_64 struct kvm_arch *ka =3D &vcpu->kvm->arch; @@ -1604,12 +1606,14 @@ static void kvm_make_mclock_inprogress_request(stru= ct kvm *kvm) } =20 static void __kvm_start_pvclock_update(struct kvm *kvm) + __acquires(&kvm->arch.tsc_write_lock) { raw_spin_lock_irq(&kvm->arch.tsc_write_lock); write_seqcount_begin(&kvm->arch.pvclock_sc); } =20 static void kvm_start_pvclock_update(struct kvm *kvm) + __acquires(&kvm->arch.tsc_write_lock) { kvm_make_mclock_inprogress_request(kvm); =20 @@ -1618,6 +1622,7 @@ static void kvm_start_pvclock_update(struct kvm *kvm) } =20 static void kvm_end_pvclock_update(struct kvm *kvm) + __releases(&kvm->arch.tsc_write_lock) { struct kvm_arch *ka =3D &kvm->arch; struct kvm_vcpu *vcpu; @@ -1716,11 +1721,14 @@ static void kvm_setup_guest_pvclock(struct pvclock_= vcpu_time_info *ref_hv_clock, struct kvm_vcpu *vcpu, struct gfn_to_pfn_cache *gpc, unsigned int offset) + __must_hold_shared(&vcpu->kvm->srcu) { struct pvclock_vcpu_time_info *guest_hv_clock; struct pvclock_vcpu_time_info hv_clock; unsigned long flags; =20 + lockdep_assert_held(&gpc->kvm->srcu); /* gpc->kvm =3D=3D vcpu->kvm */ + memcpy(&hv_clock, ref_hv_clock, sizeof(hv_clock)); =20 read_lock_irqsave(&gpc->lock, flags); @@ -2046,6 +2054,7 @@ void kvm_service_local_tlb_flush_requests(struct kvm_= vcpu *vcpu) EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_service_local_tlb_flush_requests); =20 static void record_steal_time(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct gfn_to_hva_cache *ghc =3D &vcpu->arch.st.cache; struct kvm_steal_time __user *st; @@ -2613,6 +2622,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cp= u) } =20 static void kvm_steal_time_set_preempted(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { struct gfn_to_hva_cache *ghc =3D &vcpu->arch.st.cache; struct kvm_steal_time __user *st; @@ -4739,6 +4749,7 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned int= ioctl, unsigned long arg) =20 static int vcpu_mmio_write(struct kvm_vcpu *vcpu, gpa_t addr, int len, void *__v) + __must_hold_shared(&vcpu->kvm->srcu) { const void *v =3D __v; int handled =3D 0; @@ -4762,6 +4773,7 @@ static int vcpu_mmio_write(struct kvm_vcpu *vcpu, gpa= _t addr, int len, } =20 static int vcpu_mmio_read(struct kvm_vcpu *vcpu, gpa_t addr, int len, void= *v) + __must_hold_shared(&vcpu->kvm->srcu) { int handled =3D 0; int n; @@ -4819,6 +4831,7 @@ gpa_t kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu= , gva_t gva, static int kvm_read_guest_virt_helper(gva_t addr, void *val, unsigned int = bytes, struct kvm_vcpu *vcpu, u64 access, struct x86_exception *exception) + __must_hold_shared(&vcpu->kvm->srcu) { struct kvm_pagewalk *gva_walk =3D &vcpu->arch.gva_walk; void *data =3D val; @@ -4851,6 +4864,7 @@ static int kvm_read_guest_virt_helper(gva_t addr, voi= d *val, unsigned int bytes, static int kvm_fetch_guest_virt(struct x86_emulate_ctxt *ctxt, gva_t addr, void *val, unsigned int bytes, struct x86_exception *exception) + __must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu) { struct kvm_vcpu *vcpu =3D emul_to_vcpu(ctxt); struct kvm_pagewalk *gva_walk =3D &vcpu->arch.gva_walk; @@ -4896,6 +4910,7 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_read_guest_virt); static int emulator_read_std(struct x86_emulate_ctxt *ctxt, gva_t addr, void *val, unsigned int bytes, struct x86_exception *exception, bool system) + __must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu) { struct kvm_vcpu *vcpu =3D emul_to_vcpu(ctxt); u64 access =3D 0; @@ -4911,6 +4926,7 @@ static int emulator_read_std(struct x86_emulate_ctxt = *ctxt, static int kvm_write_guest_virt_helper(gva_t addr, void *val, unsigned int= bytes, struct kvm_vcpu *vcpu, u64 access, struct x86_exception *exception) + __must_hold_shared(&vcpu->kvm->srcu) { struct kvm_pagewalk *gva_walk =3D &vcpu->arch.gva_walk; void *data =3D val; @@ -4941,6 +4957,7 @@ static int kvm_write_guest_virt_helper(gva_t addr, vo= id *val, unsigned int bytes static int emulator_write_std(struct x86_emulate_ctxt *ctxt, gva_t addr, v= oid *val, unsigned int bytes, struct x86_exception *exception, bool system) + __must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu) { struct kvm_vcpu *vcpu =3D emul_to_vcpu(ctxt); u64 access =3D PFERR_WRITE_MASK; @@ -5001,6 +5018,7 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(handle_ud); =20 static int vcpu_is_mmio_gpa(struct kvm_vcpu *vcpu, unsigned long gva, gpa_t gpa, bool write) + __must_hold_shared(&vcpu->kvm->srcu) { /* For APIC access vmexit */ if ((gpa & PAGE_MASK) =3D=3D APIC_DEFAULT_PHYS_BASE) @@ -5017,6 +5035,7 @@ static int vcpu_is_mmio_gpa(struct kvm_vcpu *vcpu, un= signed long gva, static int vcpu_mmio_gva_to_gpa(struct kvm_vcpu *vcpu, unsigned long gva, gpa_t *gpa, struct x86_exception *exception, bool write) + __must_hold_shared(&vcpu->kvm->srcu) { struct kvm_pagewalk *gva_walk =3D &vcpu->arch.gva_walk; u64 access =3D ((kvm_x86_call(get_cpl)(vcpu) =3D=3D 3) ? PFERR_USER_MASK = : 0) @@ -5054,12 +5073,14 @@ struct read_write_emulator_ops { =20 static int emulator_read_guest(struct kvm_vcpu *vcpu, gpa_t gpa, void *val, int bytes) + __must_hold_shared(&vcpu->kvm->srcu) { return !kvm_vcpu_read_guest(vcpu, gpa, val, bytes); } =20 static int emulator_write_guest(struct kvm_vcpu *vcpu, gpa_t gpa, void *val, int bytes) + __must_hold_shared(&vcpu->kvm->srcu) { int ret; =20 @@ -5075,6 +5096,7 @@ static int emulator_read_write_onepage(unsigned long = addr, void *val, struct x86_exception *exception, struct kvm_vcpu *vcpu, const struct read_write_emulator_ops *ops) + __must_hold_shared(&vcpu->kvm->srcu) { gpa_t gpa; int handled, ret; @@ -5150,6 +5172,7 @@ static int emulator_read_write(struct x86_emulate_ctx= t *ctxt, void *val, unsigned int bytes, struct x86_exception *exception, const struct read_write_emulator_ops *ops) + __must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu) { struct kvm_vcpu *vcpu =3D emul_to_vcpu(ctxt); int rc; @@ -5229,6 +5252,7 @@ static int emulator_read_emulated(struct x86_emulate_= ctxt *ctxt, void *val, unsigned int bytes, struct x86_exception *exception) + __must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu) { static const struct read_write_emulator_ops ops =3D { .read_write_guest =3D emulator_read_guest, @@ -5244,6 +5268,7 @@ static int emulator_write_emulated(struct x86_emulate= _ctxt *ctxt, const void *val, unsigned int bytes, struct x86_exception *exception) + __must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu) { static const struct read_write_emulator_ops ops =3D { .read_write_guest =3D emulator_write_guest, @@ -5263,6 +5288,7 @@ static int emulator_cmpxchg_emulated(struct x86_emula= te_ctxt *ctxt, const void *new, unsigned int bytes, struct x86_exception *exception) + __must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu) { struct kvm_vcpu *vcpu =3D emul_to_vcpu(ctxt); u64 page_line_mask; @@ -5343,6 +5369,7 @@ static int emulator_cmpxchg_emulated(struct x86_emula= te_ctxt *ctxt, static int emulator_pio_in_out(struct kvm_vcpu *vcpu, int size, unsigned short port, void *data, unsigned int count, bool in) + __must_hold_shared(&vcpu->kvm->srcu) { unsigned i; int r; @@ -5393,6 +5420,7 @@ static int emulator_pio_in_out(struct kvm_vcpu *vcpu,= int size, =20 static int emulator_pio_in(struct kvm_vcpu *vcpu, int size, unsigned short port, void *val, unsigned int count) + __must_hold_shared(&vcpu->kvm->srcu) { int r =3D emulator_pio_in_out(vcpu, size, port, val, count, true); if (r) @@ -5413,6 +5441,7 @@ static void complete_emulator_pio_in(struct kvm_vcpu = *vcpu, void *val) static int emulator_pio_in_emulated(struct x86_emulate_ctxt *ctxt, int size, unsigned short port, void *val, unsigned int count) + __must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu) { struct kvm_vcpu *vcpu =3D emul_to_vcpu(ctxt); if (vcpu->arch.pio.count) { @@ -5433,6 +5462,7 @@ static int emulator_pio_in_emulated(struct x86_emulat= e_ctxt *ctxt, static int emulator_pio_out(struct kvm_vcpu *vcpu, int size, unsigned short port, const void *val, unsigned int count) + __must_hold_shared(&vcpu->kvm->srcu) { trace_kvm_pio(KVM_PIO_OUT, port, size, count, val); return emulator_pio_in_out(vcpu, size, port, (void *)val, count, false); @@ -5441,6 +5471,7 @@ static int emulator_pio_out(struct kvm_vcpu *vcpu, in= t size, static int emulator_pio_out_emulated(struct x86_emulate_ctxt *ctxt, int size, unsigned short port, const void *val, unsigned int count) + __must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu) { return emulator_pio_out(emul_to_vcpu(ctxt), size, port, val, count); } @@ -5533,6 +5564,7 @@ static unsigned long emulator_get_cr(struct x86_emula= te_ctxt *ctxt, int cr) } =20 static int emulator_set_cr(struct x86_emulate_ctxt *ctxt, int cr, ulong va= l) + __must_hold_shared(&emul_to_vcpu(ctxt)->kvm->srcu) { struct kvm_vcpu *vcpu =3D emul_to_vcpu(ctxt); int res =3D 0; @@ -6578,6 +6610,7 @@ static int complete_fast_pio_out(struct kvm_vcpu *vcp= u) =20 static int kvm_fast_pio_out(struct kvm_vcpu *vcpu, int size, unsigned short port) + __must_hold_shared(&vcpu->kvm->srcu) { unsigned long val =3D kvm_rax_read_raw(vcpu); int ret =3D emulator_pio_out(vcpu, size, port, &val, 1); @@ -6625,6 +6658,7 @@ static int complete_fast_pio_in(struct kvm_vcpu *vcpu) =20 static int kvm_fast_pio_in(struct kvm_vcpu *vcpu, int size, unsigned short port) + __must_hold_shared(&vcpu->kvm->srcu) { unsigned long val; int ret; @@ -7192,6 +7226,7 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_x86_vendor_exit); #ifdef CONFIG_X86_64 static int kvm_pv_clock_pairing(struct kvm_vcpu *vcpu, gpa_t paddr, unsigned long clock_type) + __must_hold_shared(&vcpu->kvm->srcu) { struct kvm_clock_pairing clock_pairing; struct timespec64 ts; @@ -7571,6 +7606,7 @@ static void kvm_inject_exception(struct kvm_vcpu *vcp= u) */ static int kvm_check_and_inject_events(struct kvm_vcpu *vcpu, bool *req_immediate_exit) + __must_hold_shared(&vcpu->kvm->srcu) { bool can_inject; int r; @@ -7868,6 +7904,7 @@ void __kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu) EXPORT_SYMBOL_FOR_KVM_INTERNAL(__kvm_vcpu_update_apicv); =20 static void kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { if (!lapic_in_kernel(vcpu)) return; @@ -8049,6 +8086,7 @@ static void kvm_vcpu_reload_apic_access_page(struct k= vm_vcpu *vcpu) * userspace. */ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { int r; bool req_int_win =3D @@ -8561,6 +8599,7 @@ int kvm_arch_vcpu_runnable(struct kvm_vcpu *vcpu) =20 /* Called within kvm->srcu read side. */ static inline int vcpu_block(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { bool hv_timer; =20 @@ -8629,6 +8668,7 @@ static inline int vcpu_block(struct kvm_vcpu *vcpu) =20 /* Called within kvm->srcu read side. */ static int vcpu_run(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { int r; =20 @@ -9940,10 +9980,13 @@ void __user * __x86_set_memory_region(struct kvm *k= vm, int id, gpa_t gpa, { int i, r; unsigned long hva, old_npages; - struct kvm_memslots *slots =3D kvm_memslots(kvm); + struct kvm_memslots *slots; struct kvm_memory_slot *slot; =20 lockdep_assert_held(&kvm->slots_lock); + __assume_shared_ctx_lock(&kvm->srcu); /* update-side lock is held */ + + slots =3D kvm_memslots(kvm); =20 if (WARN_ON(id >=3D KVM_MEM_SLOTS_NUM)) return ERR_PTR_USR(-EINVAL); @@ -10452,6 +10495,7 @@ static void kvm_del_async_pf_gfn(struct kvm_vcpu *v= cpu, gfn_t gfn) } =20 static inline int apf_put_user_notpresent(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { u32 reason =3D KVM_PV_REASON_PAGE_NOT_PRESENT; =20 @@ -10460,6 +10504,7 @@ static inline int apf_put_user_notpresent(struct kv= m_vcpu *vcpu) } =20 static inline int apf_put_user_ready(struct kvm_vcpu *vcpu, u32 token) + __must_hold_shared(&vcpu->kvm->srcu) { unsigned int offset =3D offsetof(struct kvm_vcpu_pv_apf_data, token); =20 @@ -10468,6 +10513,7 @@ static inline int apf_put_user_ready(struct kvm_vcp= u *vcpu, u32 token) } =20 static inline bool apf_pageready_slot_free(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { unsigned int offset =3D offsetof(struct kvm_vcpu_pv_apf_data, token); u32 val; @@ -10892,9 +10938,11 @@ static void advance_sev_es_emulated_pio(struct kvm= _vcpu *vcpu, unsigned count, i } =20 static int kvm_sev_es_outs(struct kvm_vcpu *vcpu, unsigned int size, - unsigned int port); + unsigned int port) + __must_hold_shared(&vcpu->kvm->srcu); =20 static int complete_sev_es_emulated_outs(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { int size =3D vcpu->arch.pio.size; int port =3D vcpu->arch.pio.port; @@ -10928,9 +10976,11 @@ static int kvm_sev_es_outs(struct kvm_vcpu *vcpu, = unsigned int size, } =20 static int kvm_sev_es_ins(struct kvm_vcpu *vcpu, unsigned int size, - unsigned int port); + unsigned int port) + __must_hold_shared(&vcpu->kvm->srcu); =20 static int complete_sev_es_emulated_ins(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { unsigned count =3D vcpu->arch.pio.count; int size =3D vcpu->arch.pio.size; diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 0f5919b092e4..b97b23e2b262 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -249,6 +249,7 @@ static inline bool is_noncanonical_invlpg_address(u64 l= a, struct kvm_vcpu *vcpu) =20 static inline void vcpu_cache_mmio_info(struct kvm_vcpu *vcpu, gva_t gva, gfn_t gfn, unsigned access) + __must_hold_shared(&vcpu->kvm->srcu) { u64 gen =3D kvm_memslots(vcpu->kvm)->generation; =20 @@ -266,6 +267,7 @@ static inline void vcpu_cache_mmio_info(struct kvm_vcpu= *vcpu, } =20 static inline bool vcpu_match_mmio_gen(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu) { return vcpu->arch.mmio_gen =3D=3D kvm_memslots(vcpu->kvm)->generation; } @@ -285,6 +287,7 @@ static inline void vcpu_clear_mmio_info(struct kvm_vcpu= *vcpu, gva_t gva) } =20 static inline bool vcpu_match_mmio_gva(struct kvm_vcpu *vcpu, unsigned lon= g gva) + __must_hold_shared(&vcpu->kvm->srcu) { if (vcpu_match_mmio_gen(vcpu) && vcpu->arch.mmio_gva && vcpu->arch.mmio_gva =3D=3D (gva & PAGE_MASK)) @@ -294,6 +297,7 @@ static inline bool vcpu_match_mmio_gva(struct kvm_vcpu = *vcpu, unsigned long gva) } =20 static inline bool vcpu_match_mmio_gpa(struct kvm_vcpu *vcpu, gpa_t gpa) + __must_hold_shared(&vcpu->kvm->srcu) { if (vcpu_match_mmio_gen(vcpu) && vcpu->arch.mmio_gfn && vcpu->arch.mmio_gfn =3D=3D gpa >> PAGE_SHIFT) @@ -328,7 +332,8 @@ void kvm_inject_realmode_interrupt(struct kvm_vcpu *vcp= u, int irq, int inc_eip); u64 get_kvmclock_ns(struct kvm *kvm); uint64_t kvm_get_wall_clock_epoch(struct kvm *kvm); bool kvm_get_monotonic_and_clockread(s64 *kernel_ns, u64 *tsc_timestamp); -int kvm_guest_time_update(struct kvm_vcpu *v); +int kvm_guest_time_update(struct kvm_vcpu *v) + __must_hold_shared(&v->kvm->srcu); =20 void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value); u64 kvm_scale_tsc(u64 tsc, u64 ratio); @@ -355,13 +360,16 @@ static inline void adjust_tsc_offset_host(struct kvm_= vcpu *vcpu, s64 adjustment) =20 int kvm_read_guest_virt(struct kvm_vcpu *vcpu, gva_t addr, void *val, unsigned int bytes, - struct x86_exception *exception); + struct x86_exception *exception) + __must_hold_shared(&vcpu->kvm->srcu); =20 int kvm_write_guest_virt_system(struct kvm_vcpu *vcpu, gva_t addr, void *val, unsigned int bytes, - struct x86_exception *exception); + struct x86_exception *exception) + __must_hold_shared(&vcpu->kvm->srcu); =20 -int handle_ud(struct kvm_vcpu *vcpu); +int handle_ud(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu); =20 void kvm_deliver_exception_payload(struct kvm_vcpu *vcpu, struct kvm_queued_exception *ex); @@ -473,7 +481,8 @@ int kvm_emulate_mwait(struct kvm_vcpu *vcpu); int kvm_handle_invalid_op(struct kvm_vcpu *vcpu); int kvm_emulate_monitor(struct kvm_vcpu *vcpu); =20 -int kvm_fast_pio(struct kvm_vcpu *vcpu, int size, unsigned short port, int= in); +int kvm_fast_pio(struct kvm_vcpu *vcpu, int size, unsigned short port, int= in) + __must_hold_shared(&vcpu->kvm->srcu); int kvm_emulate_cpuid(struct kvm_vcpu *vcpu); int kvm_emulate_halt(struct kvm_vcpu *vcpu); int kvm_emulate_halt_noskip(struct kvm_vcpu *vcpu); @@ -526,7 +535,8 @@ void kvm_inject_nmi(struct kvm_vcpu *vcpu); int kvm_get_nr_pending_nmis(struct kvm_vcpu *vcpu); =20 void __user *__x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa, - u32 size); + u32 size) + __must_hold(&kvm->slots_lock); int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages); =20 bool kvm_vcpu_is_reset_bsp(struct kvm_vcpu *vcpu); @@ -840,13 +850,16 @@ static inline void kvm_machine_check(void) int kvm_handle_memory_failure(struct kvm_vcpu *vcpu, int r, struct x86_exception *e); void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid); -int kvm_handle_invpcid(struct kvm_vcpu *vcpu, unsigned long type, gva_t gv= a); +int kvm_handle_invpcid(struct kvm_vcpu *vcpu, unsigned long type, gva_t gv= a) + __must_hold_shared(&vcpu->kvm->srcu); =20 int kvm_sev_es_mmio(struct kvm_vcpu *vcpu, bool is_write, gpa_t gpa, - unsigned int bytes, void *data); + unsigned int bytes, void *data) + __must_hold_shared(&vcpu->kvm->srcu); int kvm_sev_es_string_io(struct kvm_vcpu *vcpu, unsigned int size, unsigned int port, void *data, unsigned int count, - int in); + int in) + __must_hold_shared(&vcpu->kvm->srcu); =20 static inline void __kvm_prepare_emulated_mmio_exit(struct kvm_vcpu *vcpu, gpa_t gpa, unsigned int len, @@ -886,7 +899,8 @@ static inline bool user_exit_on_hypercall(struct kvm *k= vm, unsigned long hc_nr) } =20 int ____kvm_emulate_hypercall(struct kvm_vcpu *vcpu, int cpl, - int (*complete_hypercall)(struct kvm_vcpu *)); + int (*complete_hypercall)(struct kvm_vcpu *)) + __must_hold_shared(&vcpu->kvm->srcu); =20 #define __kvm_emulate_hypercall(_vcpu, cpl, complete_hypercall) \ ({ \ @@ -898,6 +912,7 @@ int ____kvm_emulate_hypercall(struct kvm_vcpu *vcpu, in= t cpl, __ret; \ }) =20 -int kvm_emulate_hypercall(struct kvm_vcpu *vcpu); +int kvm_emulate_hypercall(struct kvm_vcpu *vcpu) + __must_hold_shared(&vcpu->kvm->srcu); =20 #endif --=20 2.55.0.1003.g10538fe699-goog From nobody Fri Sep 25 16:02:05 2026 Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E45052B1E1 for ; Thu, 10 Sep 2026 16:24:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.72 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057451; cv=none; b=ATHbmmkcexv07cJOxRG4HpMvdmpxQNK168fTbpcllnmgc51dJWcW4uUQfUV6G+NUWavQEunp1mI3Dodfzx4XIbyaQKOslFdt6mz6JeeVyLpigIi7BB6w4rbHS4Dojqt4XzjUPSfLgi4nh4YKk1vcrw5d3PIJ+JTnvKQo0/h31pE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057451; c=relaxed/simple; bh=AHvgtIL5obyEkQeMJj9TEFvDUyBEEuMsjvgCq4Ntp70=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=hivY6NK1hMAYwVByehCRF0GaQW4455H+UUwzFuqO1DZI0uLg3ciHs6GkCIXlEC7HHqb8+Il9DZ5j8tigi//RIWs++yTIefZgh+b94ZlWyIsH84dtC7Z707LesIPXa28kVdD8wEiCxc+dDQJikj/1fUISBkoDEFrhSj9YFsoaRj0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--elver.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Z1fA3ekL; arc=none smtp.client-ip=209.85.128.72 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--elver.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Z1fA3ekL" Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-4955e865174so16545e9.3 for ; Thu, 10 Sep 2026 09:24:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1789057445; x=1789662245; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=3WjqNYFOMqEwxR4wUCekRhwd9RX45J94sJ1enbo8buA=; b=Z1fA3ekLfjqyfjhKTE48Je3loqRDWXZcI308y5qa77KWwYaBmgbCxuADX2bqjm4q3r gTsuCQwhev5huXwKsNtyPFTTXxz5Ta+I4ZXSSBm6bbrccnyEharkhf8Kst05NmMacIIu 5Q0dX/WtgQ9WcvNOg+MRawOcXkCDWDCGc7pqRkKw/BBOSV4EJoPttDz+gd5Bp2CWM6mT spe6S1/VgOhZirQc8u4lckEX6IuJR0u2YZbnQw2uoZdoUiCyPzcoBpFhipeTnytXEnnK lobL9ChhPIIVYwRMi8FqCq8UISzk8ckaGlvZUxVv3CicfFYDiIpBZDaPzBHxsaqZ9BOQ d/AQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789057445; x=1789662245; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=3WjqNYFOMqEwxR4wUCekRhwd9RX45J94sJ1enbo8buA=; b=Ra0Uv7rHC8IrpEAT9Gno0vDjt8JabRt8z6mDh+6v3znckU+PzUB8f+qirZghVZgli6 wZoOPqpAtreAtuy+rPaOlhCXe7H370InMyRhcczK66noSQ2/CAyBDtqGhzIijMij7IEl 93TpGSo+AMrG18ADPFLu7QHqf0YqNsNAzuEyd9hDAThzu2FcXUX7uvtKyws4+x4tBOmy 7t+kP8MLdQIpC6syFFkxAWA+kTnQRvqX1/REqeIxzMXM8H/PEdisSrxuvyz1WMX/40Ur DhcPNDKVB3Jcm7V5Csyqa3gEo8ETEFVO1kMI6XfElMFX2PNg1CZrAIJhjyETr82oYGNC FdrA== X-Forwarded-Encrypted: i=1; AKwUvBwhTx5chrLXfNNPKyPmezlZ7GqEiTSOmzEeA6A02uBFLyR1WQqDj4HEMZX+8Ra1WQVE5JiEB/7g7JA69lA=@vger.kernel.org X-Gm-Message-State: AFuF++k211rLXs3Qa/dwD+xg35MIsQLCRI8Avq0t04eoCLwzWPH0r23i UQk0huGR1tvUHeGjVcJk51ZPZqYH8lXoS3AvWF7YCMIia2CpzUkM7M+le6JpIDQJZEn0Iil4IAh XbQ== X-Received: from wmbb26.prod.google.com ([2002:a05:600c:589a:b0:49e:5ffd:edcc]) (user=elver job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:5493:b0:49b:8f18:714a with SMTP id 5b1f17b1804b1-49cf824a6c5mr440314875e9.12.1789057444629; Thu, 10 Sep 2026 09:24:04 -0700 (PDT) Date: Thu, 10 Sep 2026 16:21:41 +0000 In-Reply-To: <20260910162343.4092060-1-elver@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260910162343.4092060-1-elver@google.com> X-Mailer: git-send-email 2.55.0.1003.g10538fe699-goog Message-ID: <20260910162343.4092060-9-elver@google.com> Subject: [PATCH RFC 08/10] KVM: Add guarded_by to members in struct kvm From: Marco Elver To: elver@google.com Cc: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Vitaly Kuznetsov , Kiryl Shutsemau , Rick Edgecombe , David Hildenbrand , kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add __guarded_by annotations to a subset of members in struct kvm where the protecting locks reside in the same struct scope. Members protected by locks across nested or parent struct boundaries are omitted due to C lexical scoping limitations. No functional change intended. Signed-off-by: Marco Elver --- include/linux/kvm_host.h | 12 ++++++------ virt/kvm/eventfd.c | 4 ++-- virt/kvm/kvm_main.c | 3 +++ 3 files changed, 11 insertions(+), 8 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 12f241304228..36edf0a59107 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -771,7 +771,7 @@ struct kvm { /* The two memslot sets - active and inactive (per address space) */ struct kvm_memslots __memslots[KVM_MAX_NR_ADDRESS_SPACES][2]; /* The current active memslot set for each address space */ - struct kvm_memslots __rcu *memslots[KVM_MAX_NR_ADDRESS_SPACES]; + struct kvm_memslots __rcu *memslots[KVM_MAX_NR_ADDRESS_SPACES] __guarded_= by(&srcu, &slots_lock, &slots_arch_lock); struct xarray vcpu_array; DECLARE_BITMAP(vcpu_ids, KVM_MAX_VCPU_IDS); /* @@ -787,7 +787,7 @@ struct kvm { =20 /* For management / invalidation of gfn_to_pfn_caches */ spinlock_t gpc_lock; - struct list_head gpc_list; + struct list_head gpc_list __guarded_by(&gpc_lock); =20 /* * created_vcpus is protected by kvm->lock, and is incremented @@ -801,11 +801,11 @@ struct kvm { int last_boosted_vcpu; struct list_head vm_list; struct mutex lock; - struct kvm_io_bus __rcu *buses[KVM_NR_BUSES]; + struct kvm_io_bus __rcu *buses[KVM_NR_BUSES] __guarded_by(&srcu, &slots_l= ock); #ifdef CONFIG_HAVE_KVM_IRQCHIP struct { spinlock_t lock; - struct list_head items; + struct list_head items __guarded_by(&lock); /* resampler_list update side is protected by resampler_lock. */ struct list_head resampler_list; struct mutex resampler_lock; @@ -826,9 +826,9 @@ struct kvm { /* * Update side is protected by irq_lock. */ - struct kvm_irq_routing_table __rcu *irq_routing; + struct kvm_irq_routing_table __rcu *irq_routing __guarded_by(&irq_srcu, &= irq_lock); =20 - struct hlist_head irq_ack_notifier_list; + struct hlist_head irq_ack_notifier_list __guarded_by(&irq_srcu, &irq_lock= ); #endif =20 struct mmu_notifier mmu_notifier; diff --git a/virt/kvm/eventfd.c b/virt/kvm/eventfd.c index 6701390336f6..6051fec5ec90 100644 --- a/virt/kvm/eventfd.c +++ b/virt/kvm/eventfd.c @@ -1046,8 +1046,8 @@ void kvm_eventfd_init(struct kvm *kvm) { #ifdef CONFIG_HAVE_KVM_IRQCHIP - spin_lock_init(&kvm->irqfds.lock); - INIT_LIST_HEAD(&kvm->irqfds.items); + scoped_guard(spinlock_init, &kvm->irqfds.lock) + INIT_LIST_HEAD(&kvm->irqfds.items); INIT_LIST_HEAD(&kvm->irqfds.resampler_list); mutex_init(&kvm->irqfds.resampler_lock); #endif diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 2f22d5439d39..871acc3aebbd 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -1111,6 +1111,7 @@ void __weak kvm_arch_create_vm_debugfs(struct kvm *kv= m) /* Called only on cleanup and destruction paths when there are no users. */ static inline struct kvm_io_bus *kvm_get_bus_for_destruction(struct kvm *k= vm, enum kvm_bus idx) + __context_unsafe(/* destruction */) { return rcu_dereference_protected(kvm->buses[idx], !refcount_read(&kvm->users_count)); @@ -1120,6 +1121,7 @@ static int kvm_enable_virtualization(void); static void kvm_disable_virtualization(void); =20 static struct kvm *kvm_create_vm(unsigned long type, const char *fdname) + __context_unsafe(/* constructor */) { struct kvm *kvm =3D kvm_arch_alloc_vm(); struct kvm_memslots *slots; @@ -1276,6 +1278,7 @@ static void kvm_destroy_devices(struct kvm *kvm) } =20 static void kvm_destroy_vm(struct kvm *kvm) + __context_unsafe(/* destructor */) { int i; struct mm_struct *mm =3D kvm->mm; --=20 2.55.0.1003.g10538fe699-goog From nobody Fri Sep 25 16:02:05 2026 Received: from mail-ej1-f71.google.com (mail-ej1-f71.google.com [209.85.218.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 704714E13E7 for ; Thu, 10 Sep 2026 16:24:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.71 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057453; cv=none; b=PXENa2zprjY9nPoL1syP71MUusZLnV5l4NX5kVIKpHzr4iFXcNOmSBRp/ligTpLjJmaTV/QLEfBfuJMivT0N0OnwbgUgWOBL5B8nF3c8bry2mAuuQ5QE8tLB1xoTyNFIW1EoqHTn/Eo0L3bWnSBH2opYhrnv8vC+7c9hJTNkfic= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057453; c=relaxed/simple; bh=Uw18xgi0BxdfHnCsIhuDM+SwgIYJ0SP8g/ovgWkILJA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=dQ5umLAd/Qk+qCvJ5vNZDr9uzWOxet/SyUfJhxvaxw9jQEb3zExaZyEx63gmyJnzBx3tz1SWPlMM0tfqnrd/QBrNbOnm8wWHCHfK5YpPWOx8ZRD9jmMRxJ6ydNF7kRBG60LjpOEahaFSLKM1/PFy4abKDlNy24Wp16QBa5Ipn8I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--elver.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=q7TAiWc8; arc=none smtp.client-ip=209.85.218.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--elver.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="q7TAiWc8" Received: by mail-ej1-f71.google.com with SMTP id a640c23a62f3a-c253185765dso794505466b.0 for ; Thu, 10 Sep 2026 09:24:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1789057446; x=1789662246; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=KnZriOrx76oWc7IbFKP8WTxfBR0BHG1CwkYJSO6Ivsw=; b=q7TAiWc8wCSCe26DpL8+PXoZ6NhrB/sxGstZdjN4DZ4OgqGfMRBRDTuYXsbmD5ETCV 0uQHtb1IuQycvP4jRYZrOQIMGJHwg1Yrr3A0bUMuB2rXiHaItDTPchvAbTPfy/D8UsBS q3Sej2ZQXqbcxDPp3S3D3Bh5WEVFerAlyn++tJET6deY23IxFb523leFG69VHZoHVQ6k Wjs0wZEv/fAnM+ngBoomLa9LG2eq4isiwwA+x03XAWtNc4PR3uV3H/DE7XrE/5+u254n wfZmxZUiC2d9Yg4v23GLMtVUR91BAQY/oGr/lPGiPyYiPLgzeeaNicnqwHDoeqxgrhV2 Loeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789057446; x=1789662246; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=KnZriOrx76oWc7IbFKP8WTxfBR0BHG1CwkYJSO6Ivsw=; b=D5omDqs/0lxrE9P+0ixGeGu6UYUDKxNbnsVma5Tv6va4FJvPylIP66BZuEREgx4jXH 0+tg6Y3Oo7DAkdL68cWSf4ub1SsBoKmcwKuG1BeulKoLUAUwvodlH5rCuuesz7fCEWUn uOnTH4T8ceAsSrBxhCOEpEahUh1WCAxJSRoLKp2Q0gIdLWYwxTY185DsJ7x1YybM8/zA hoPteCOhlu6T8n49e/x0L3dE4b/AknuFsZrg+r3g6fcapeEHeHSVW1E5Zcw5oNibl6lV wshV16WFPtwUvE/E0vZwxHqYebDZLIWnTLZ8T1/9M5WiiCiX7M8Mow2jZ92TRGLR2W14 wtBw== X-Forwarded-Encrypted: i=1; AKwUvBysRnRI6NkvAK2sDoLlTjPLyxe9Bwohovli+YPixHco5/HBZrkqMZo5jZEEyD1BkdEp2I5m3WSVIiuwrx0=@vger.kernel.org X-Gm-Message-State: AFuF++l+ZS4gI86Z2Embhp12vb8R5Oh10o/QpHQQ3BazEvm9CgpnIBRv W1O2Qy5p50lv4AW9QR/l/zc4ducYqKRiMxQnygwylaFFvAhEuPtsj3SWcL9PYFHaFQsCKZuzP4b YOg== X-Received: from ejcey15.prod.google.com ([2002:a17:907:b8f:b0:c25:1e06:d698]) (user=elver job=prod-delivery.src-stubby-dispatcher) by 2002:a17:907:9709:b0:c1c:4e36:eec6 with SMTP id a640c23a62f3a-c260ca22ce1mr1766102666b.18.1789057445690; Thu, 10 Sep 2026 09:24:05 -0700 (PDT) Date: Thu, 10 Sep 2026 16:21:42 +0000 In-Reply-To: <20260910162343.4092060-1-elver@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260910162343.4092060-1-elver@google.com> X-Mailer: git-send-email 2.55.0.1003.g10538fe699-goog Message-ID: <20260910162343.4092060-10-elver@google.com> Subject: [PATCH RFC 09/10] KVM: x86: Add guarded_by annotations for kvm_arch, kvm_hv, and ioapic From: Marco Elver To: elver@google.com Cc: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Vitaly Kuznetsov , Kiryl Shutsemau , Rick Edgecombe , David Hildenbrand , kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add __guarded_by annotations to a subset of fields across x86 state (struct kvm_hv, struct kvm_arch, and struct kvm_ioapic) where the protecting locks reside in the same struct scope. Mark deliberate lockless updates with data_race(). No functional change intended. Signed-off-by: Marco Elver --- arch/x86/include/asm/kvm_host.h | 32 ++++++++++++++++---------------- arch/x86/kvm/ioapic.c | 4 ++-- arch/x86/kvm/ioapic.h | 16 ++++++++-------- arch/x86/kvm/x86.c | 8 ++++---- 4 files changed, 30 insertions(+), 30 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 683bb8bf43a9..a0d2d6c08e47 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1073,20 +1073,20 @@ struct kvm_hv { struct mutex hv_lock; u64 hv_guest_os_id; u64 hv_hypercall; - u64 hv_tsc_page; + u64 hv_tsc_page __guarded_by(&hv_lock); enum hv_tsc_page_status hv_tsc_page_status; =20 /* Hyper-v based guest crash (NT kernel bugcheck) parameters */ - u64 hv_crash_param[HV_X64_MSR_CRASH_PARAMS]; - u64 hv_crash_ctl; + u64 hv_crash_param[HV_X64_MSR_CRASH_PARAMS] __guarded_by(&hv_lock); + u64 hv_crash_ctl __guarded_by(&hv_lock); =20 struct ms_hyperv_tsc_page tsc_ref; =20 struct idr conn_to_evt; =20 - u64 hv_reenlightenment_control; - u64 hv_tsc_emulation_control; - u64 hv_tsc_emulation_status; + u64 hv_reenlightenment_control __guarded_by(&hv_lock); + u64 hv_tsc_emulation_control __guarded_by(&hv_lock); + u64 hv_tsc_emulation_status __guarded_by(&hv_lock); u64 hv_invtsc_control; =20 /* How many vCPUs have VP index !=3D vCPU index */ @@ -1232,15 +1232,15 @@ struct kvm_arch { * preemption-disabled region, so it must be a raw spinlock. */ raw_spinlock_t tsc_write_lock; - u64 last_tsc_nsec; - u64 last_tsc_write; - u32 last_tsc_khz; - u64 last_tsc_offset; - u64 cur_tsc_nsec; - u64 cur_tsc_write; - u64 cur_tsc_offset; - u64 cur_tsc_generation; - int nr_vcpus_matched_tsc; + u64 last_tsc_nsec __guarded_by(&tsc_write_lock); + u64 last_tsc_write __guarded_by(&tsc_write_lock); + u32 last_tsc_khz __guarded_by(&tsc_write_lock); + u64 last_tsc_offset __guarded_by(&tsc_write_lock); + u64 cur_tsc_nsec __guarded_by(&tsc_write_lock); + u64 cur_tsc_write __guarded_by(&tsc_write_lock); + u64 cur_tsc_offset __guarded_by(&tsc_write_lock); + u64 cur_tsc_generation __guarded_by(&tsc_write_lock); + int nr_vcpus_matched_tsc __guarded_by(&tsc_write_lock); =20 u32 default_tsc_khz; bool user_set_tsc; @@ -1370,7 +1370,7 @@ struct kvm_arch { #endif =20 #if IS_ENABLED(CONFIG_HYPERV) - hpa_t hv_root_tdp; + hpa_t hv_root_tdp __guarded_by(&hv_root_tdp_lock); spinlock_t hv_root_tdp_lock; struct hv_partition_assist_pg *hv_pa_pg; #endif diff --git a/arch/x86/kvm/ioapic.c b/arch/x86/kvm/ioapic.c index d6865e557abe..7affe2584036 100644 --- a/arch/x86/kvm/ioapic.c +++ b/arch/x86/kvm/ioapic.c @@ -739,11 +739,11 @@ int kvm_ioapic_init(struct kvm *kvm) ioapic =3D kzalloc_obj(struct kvm_ioapic, GFP_KERNEL_ACCOUNT); if (!ioapic) return -ENOMEM; - spin_lock_init(&ioapic->lock); INIT_DELAYED_WORK(&ioapic->eoi_inject, kvm_ioapic_eoi_inject_work); INIT_HLIST_HEAD(&ioapic->mask_notifier_list); kvm->arch.vioapic =3D ioapic; - kvm_ioapic_reset(ioapic); + scoped_guard(spinlock_init, &ioapic->lock) + kvm_ioapic_reset(ioapic); kvm_iodevice_init(&ioapic->dev, &ioapic_mmio_ops); ioapic->kvm =3D kvm; mutex_lock(&kvm->slots_lock); diff --git a/arch/x86/kvm/ioapic.h b/arch/x86/kvm/ioapic.h index 81b576513116..1f87396c0a79 100644 --- a/arch/x86/kvm/ioapic.h +++ b/arch/x86/kvm/ioapic.h @@ -70,19 +70,19 @@ union kvm_ioapic_redirect_entry { =20 struct kvm_ioapic { u64 base_address; - u32 ioregsel; - u32 id; - u32 irr; + u32 ioregsel __guarded_by(&lock); + u32 id __guarded_by(&lock); + u32 irr __guarded_by(&lock); u32 pad; - union kvm_ioapic_redirect_entry redirtbl[IOAPIC_NUM_PINS]; - unsigned long irq_states[IOAPIC_NUM_PINS]; + union kvm_ioapic_redirect_entry redirtbl[IOAPIC_NUM_PINS] __guarded_by(&l= ock); + unsigned long irq_states[IOAPIC_NUM_PINS] __guarded_by(&lock); struct kvm_io_device dev; struct kvm *kvm; spinlock_t lock; - struct rtc_status rtc_status; + struct rtc_status rtc_status __guarded_by(&lock); struct delayed_work eoi_inject; - u32 irq_eoi[IOAPIC_NUM_PINS]; - u32 irr_delivered; + u32 irq_eoi[IOAPIC_NUM_PINS] __guarded_by(&lock); + u32 irr_delivered __guarded_by(&lock); =20 /* reads protected by irq_srcu, writes by irq_lock */ struct hlist_head mask_notifier_list; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 9893705d0dfa..5d4b2c7aa9b8 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -9813,8 +9813,8 @@ int kvm_arch_enable_virtualization_cpu(void) * you may have some problem. Solving this issue is * left as an exercise to the reader. */ - kvm->arch.last_tsc_nsec =3D 0; - kvm->arch.last_tsc_write =3D 0; + data_race(kvm->arch.last_tsc_nsec =3D 0); + data_race(kvm->arch.last_tsc_write =3D 0); } =20 } @@ -9927,8 +9927,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long t= ype) kvm->arch.enable_pmu =3D enable_pmu && !kvm->arch.has_protected_pmu; =20 #if IS_ENABLED(CONFIG_HYPERV) - spin_lock_init(&kvm->arch.hv_root_tdp_lock); - kvm->arch.hv_root_tdp =3D INVALID_PAGE; + scoped_guard(spinlock_init, &kvm->arch.hv_root_tdp_lock) + kvm->arch.hv_root_tdp =3D INVALID_PAGE; #endif =20 kvm_apicv_init(kvm); --=20 2.55.0.1003.g10538fe699-goog From nobody Fri Sep 25 16:02:05 2026 Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 31A50380FFD for ; Thu, 10 Sep 2026 16:24:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.71 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057456; cv=none; b=nC4XWMOyqFfPZsYd4cOnpcisNm2mHzCGz8Mk2sFlzdkjgAJM+Qai2+/bh+KTHs5pb0Zs4+luPHp63vm/90LP9R0hE9ghEFvY12Rt+Ar3fQ3stv/VNfmZhQNPG+vKb/mT35p68Q/1Z+2wXaJmWBL5O0ccdIpq+rXI1j16gCBHPyw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789057456; c=relaxed/simple; bh=pxrnsvzW8MKgeFG0MarEtyI0HnlG7oA7pNmC3DpRc9U=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=eQo91VoXxtcEvrW+IGEW2lEODfrq/eWO334wa/KdGnRpipw/QH+0qL8pNuPOc+gs7bUK18MWBQx3w9UtpL5okJrMw28RPOhi0qG0CMYh3q2Nie7UjRbWKuhYCz9Y/1MufMNql6BtNDY/pxDJt7u5rjr6zaFBkxJz6/ukaam1/QI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--elver.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=oIKkq3Jp; arc=none smtp.client-ip=209.85.128.71 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--elver.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="oIKkq3Jp" Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-49ccfad90f1so137305e9.0 for ; Thu, 10 Sep 2026 09:24:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1789057449; x=1789662249; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=eHI1zbWWi1BOEuhxInEo9eXwbPIfP+RyCmucdS/YFCg=; b=oIKkq3Jp0ry6K0DsHLDkf84OH4pYLX+kknjykm17cdBxO2SIuUe2jmB5SlSaX5JfGc Fv7YeiWWQHr7bAuUuUDKoGDyhCyXF1fnjC0UZuUhWBFo55H67c+ZSw/1bs9/1druy6X1 Ea2D8yXPWlcWeXKt5f5wfE7y3+ZJyf0Y6iDiK5RlH8j3CrPVtmyeV63rKYSAqfOPq+Nh qIIzE7eMAALtaxVmKsNph7p5KSpJWMX4t7wIjeinimv4OsBHI9OAipvGYbHlz6CzOY1e SjlojdwUctVwIOu3meiAMyXM+tQbe5xLSlmHFTmtm+941L6YybKtm+Ni8WwqO6B1iuHp azNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1789057449; x=1789662249; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=eHI1zbWWi1BOEuhxInEo9eXwbPIfP+RyCmucdS/YFCg=; b=sz4sHieLuuozc4HY+KQEdfm4KvQ8Z6MjJSbErhQF66vrJLuuajgj9xQ2Cq381RrVot runWGWN/fcg3yotzv2qonGkFwCmETPxMOwFXY8E8yEaKEjMRuKcuoigq2jfiadKJdXWx 0c7Qt3a4di5KosQ8+AqdIkDVo4o58lPh/rt5dsqwzvbjEjTJKR4Sy+fLafQzfM5JyBA8 ejaxorJI6VXNj4zMEmMddvu4LsDZMnAEhopOekRIOWLaK1VyEViJpAx+WoLZDqOZNd2/ b6igTWQTLX0CLu8YEBbbJDRBFdJWX6f+6S7sTfEFSxaPJqdaLZcmVx9LXiIimaQNwj1z 0S9A== X-Forwarded-Encrypted: i=1; AKwUvBwsCBTj5ULQb+1cfRheRU+IJnsz5ld4ilcNKJ5otPNSow9xrQW4SVvlbtOa2YhdE6f/SuBzh/+mVdWoq8s=@vger.kernel.org X-Gm-Message-State: AFuF++lZibL+/cZ6JKpm5KV0CJQii0excqbObZfL3co1NqNr7tR4NzWt B9ApLLx7L7UMwC9NgLIjuQL/YXnVDAJIsZVEgAZeAiYp6qmMTAtXwDvfXdzUO3xn5NtdPml5E3k p3Q== X-Received: from wmhj22.prod.google.com ([2002:a05:600c:3016:b0:49d:1474:3527]) (user=elver job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:1c29:b0:49c:fc6c:be03 with SMTP id 5b1f17b1804b1-49cfc6cc06cmr410272355e9.26.1789057448808; Thu, 10 Sep 2026 09:24:08 -0700 (PDT) Date: Thu, 10 Sep 2026 16:21:43 +0000 In-Reply-To: <20260910162343.4092060-1-elver@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260910162343.4092060-1-elver@google.com> X-Mailer: git-send-email 2.55.0.1003.g10538fe699-goog Message-ID: <20260910162343.4092060-11-elver@google.com> Subject: [PATCH RFC 10/10] KVM: x86: Enable CONTEXT_ANALYSIS with opt-outs From: Marco Elver To: elver@google.com Cc: Sean Christopherson , Paolo Bonzini , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H. Peter Anvin" , Vitaly Kuznetsov , Kiryl Shutsemau , Rick Edgecombe , David Hildenbrand , kvm@vger.kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Enable Clang context analysis (-Wthread-safety) for x86 KVM. Because virt/kvm/ objects are built directly into the x86 KVM module, this also enables checking for common KVM sources. Temporarily opt out sources that require more invasive annotations: mmu/mmu.c, mmu/tdp_mmu.c, i8259.c, and xen.c. No functional change intended. Signed-off-by: Marco Elver --- arch/x86/kvm/Makefile | 8 ++++++++ scripts/context-analysis-suppression.txt | 1 + 2 files changed, 9 insertions(+) diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile index 0474604ab8a1..a93b54b00ae2 100644 --- a/arch/x86/kvm/Makefile +++ b/arch/x86/kvm/Makefile @@ -1,5 +1,13 @@ # SPDX-License-Identifier: GPL-2.0 =20 +CONTEXT_ANALYSIS :=3D y + +# Opt out x86 objects pending context annotations: +CONTEXT_ANALYSIS_mmu/mmu.o :=3D n +CONTEXT_ANALYSIS_mmu/tdp_mmu.o :=3D n +CONTEXT_ANALYSIS_i8259.o :=3D n +CONTEXT_ANALYSIS_xen.o :=3D n + ccflags-y +=3D -I $(srctree)/arch/x86/kvm ccflags-$(CONFIG_KVM_WERROR) +=3D -Werror =20 diff --git a/scripts/context-analysis-suppression.txt b/scripts/context-ana= lysis-suppression.txt index 1c51b6153f08..666c4ca54708 100644 --- a/scripts/context-analysis-suppression.txt +++ b/scripts/context-analysis-suppression.txt @@ -17,6 +17,7 @@ src:*include/net/* src:*include/linux/bit_spinlock.h=3Demit src:*include/linux/cleanup.h=3Demit src:*include/linux/kref.h=3Demit +src:*include/linux/kvm_host.h=3Demit src:*include/linux/list*.h=3Demit src:*include/linux/local_lock*.h=3Demit src:*include/linux/lockdep.h=3Demit --=20 2.55.0.1003.g10538fe699-goog