From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.50.34 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844072; b=0mvPJ4oiyXa23EEUVogA921T3CveBck2HaIiLzaCmjOJY5n8fLEdd1Lxb8wcrar1XkQZ 3J72mzjUIeXPSLtJbbdCUE/aKYlyY+bt1HNOrXpafLgfnmIzgkzC0l3dbRxcW8zdwLoTC +HD3JbJ+gjPxJabwHq3SVaYT4JMaU0n0qM/frUDrhWkSOemtRA+vjiq5hG1EMWzG/GB4f uaJSCndAR+Yyv/Q8OG39y08DO/IdOpRRofAqxBcVCLu8lwrH0L6zTzaDxVyPNaBGQxAWH ej+p9N4BBtHLp8WuQcqIz6/IhQxxdDcSYtn3olqj2OuEdm7VgKNv/v0lm9iC2p2Xd0Q== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844072; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=KSzhuJP/1dug5WU6DX/hxQAI8dWQXM9qoEPSMrGRfhc=; b=TU69fsDzpjNqUE9uAAu3no/J7fTWDVrkWkM/SLs0pt6qi+MFdzCrzU9Me+peDkVChipN 2SnqbG4xbWZ6vU3QoV/ujWVEFEay+CyLzFzjU+q042wSdu2tmt+rDBLSfO43kU+UfqbdZ 2e6fRqTKVOGjV6HD6fNI1ETC6C0C5m56FLneCkLi6RBzu+kkExpoidShnThS1rwlYWpPO O/pS6Ue1UldgRI0wNubpQ4FdCZl79kWT4z/grsqzLQ0sdS9IHhRCOyouBcD7PT/uj168N HcHEaMHCzX6dnljXNQ7Z1LY0qrljZbZ2iwv/Cgi6fa/mfrdt/PmpctWr3WXeLApnBbg== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=casper.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=sSLM+wEZ; arc=none smtp.remote-ip=90.155.50.34 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAB75606A338; Thu, 5 Nov 2026 02:01:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org DAB75606A338 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=casper.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=sSLM+wEZ DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=KSzhuJP/1dug5WU6DX/hxQAI8dWQXM9qoEPSMrGRfhc=; b=sSLM+wEZ0RAGrshJ0WnfiNFrWB 0PW7Qwl4biKt2CKvniyPtUPM/+KKY9PJarIrJJAl+cmqTwUrVJTfjvnengZfBWJENxvpBWGDn/gY/ pecfLTM7auTFS1/xave/ZteSyGl/spAUl9WzVckvvarfbdD63e7HRnMzmNOzkP7iXmV4CLZl8urHt DHQMmB3EjcA6NbwiJ6rf9dKzCF7QVX8tOEFYMrUFTwV7HkAXj0UY3KETf0gvd6qswDsAw6lHHo9dt vuYvNawUWAdXlNbz2kj8DlKULI0mfEdULeoJTVwrCCUpCCbG/uKEnuN3exUJOpaPtdERcZ2pyHuxp n+9w8nRA==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by casper.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmn-000000037j9-0FHW; Thu, 05 Nov 2026 02:01:05 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmm-0000000AWD4-0WQw; Thu, 05 Nov 2026 02:01:04 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 01/20] KVM: x86/xen: Fill the hypercall page with a single guest write Date: Thu, 5 Nov 2026 02:00:43 +0000 Message-ID: <20261105020104.2507083-2-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Content-Type: text/plain; charset="utf-8" From: David Woodhouse Writing the hypercall page as 128 separate 32-byte guest writes pushes 128 entries into the dirty ring for the same gfn. The ring's headroom above the soft limit is KVM_DIRTY_RING_RSVD_ENTRIES (64) plus kvm_cpu_dirty_log_size(), so when the CPU assists with dirty logging (PML adds 512) a single WRMSR fits; without that assist (e.g. VMX with PML disabled) one guest WRMSR can overflow it: a guest-reachable WARN_ON_ONCE() in kvm_dirty_ring_push(), and an unharvested entry overwritten, losing a page from dirty tracking during live migration. Build the page in a buffer and write it with one call. Fixes: fb04a1eddb1a ("KVM: X86: Implement ring-based dirty memory tracking") Signed-off-by: David Woodhouse --- arch/x86/kvm/xen.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c index 6aa00f746cba..5875d10ae3a9 100644 --- a/arch/x86/kvm/xen.c +++ b/arch/x86/kvm/xen.c @@ -1345,7 +1345,8 @@ int kvm_xen_write_hypercall_page(struct kvm_vcpu *vcp= u, u64 data) */ if (kvm_xen_hypercall_enabled(kvm)) { u8 instructions[32]; - int i; + u8 *page; + int ret, i; =20 if (page_num) return 1; @@ -1362,13 +1363,20 @@ int kvm_xen_write_hypercall_page(struct kvm_vcpu *v= cpu, u64 data) /* int3 to pad */ memset(instructions + 9, 0xcc, sizeof(instructions) - 9); =20 + page =3D kmalloc(PAGE_SIZE, GFP_KERNEL); + if (!page) + return -ENOMEM; + for (i =3D 0; i < PAGE_SIZE / sizeof(instructions); i++) { *(u32 *)&instructions[1] =3D i; - if (kvm_vcpu_write_guest(vcpu, - page_addr + (i * sizeof(instructions)), - instructions, sizeof(instructions))) - return 1; + memcpy(page + (i * sizeof(instructions)), instructions, + sizeof(instructions)); } + + ret =3D kvm_vcpu_write_guest(vcpu, page_addr, page, PAGE_SIZE); + kfree(page); + if (ret) + return 1; } else { /* * Note, truncation is a non-issue as 'lm' is guaranteed to be --=20 2.43.0 From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.92.199 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844079; b=mhOb4NHx4iABPO5FegwziGEYQPjbpPaiMbg9KqOdVbJVjTy+dzWgsErLXTcQrSEhvkcT BOEf5FLp9wIuTTKX9DOYMcaky3lXvvi/ONP7DODndKbCRc9oZ+2jEs8wIcIVNydhEB1Tl qXhb0ptHAZyb/mssUbF2gkvMGrF83+RB1qww8mZCHlgtWI47n70Zs5oDlKnwB2r1oRp9M j3SrxCJ57sqgI5aZ7ROeeAT+IDjkp9yvIFTH/KVTLSejxgESVFcTtKnrKup9y0bBJ+E9a FW4lurIwy4IWrQybTVkJUUZS26md/17+Z3ICrYYU8ONUmjwH8kyAQFt4uH5aBTaUwAw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844079; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Type:Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=ihM6W4dC3NBcXSxQRanAZyoU5ddBab3XKFn/HuD51Wc=; b=jp14PpYFU5BUhB1rvgL3WvZoYDe2yzZU1/ZA8rVabUMuB+EKefEZLQh5sGhow3Y1agIE zfaM+GpJm6oqbSpGj2r5MQP0LGwpdBpjAigLZ+yd3cxs4+M7Nao5m4vdbst1LGaXnqWyH iHZd5KgVx/lvixGSHn6HdpbTHLrSeK8E6l4V7cBb2MkHHURWdCOJiW6P8GckqQmE+1i7r BpFm+Rw7JyCXM7l/rbi1Xs237wXRkYLEXYEqZl28FMyIRoI7MPpf7Ti7LZC4xNmrZCLXf +ecR+B2mrPbNbQwY2Jjre1DBw/kkuVFZ0boeAaIZbjxZRULwpGpHutqUDp24Xp6siXQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=desiato.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=desiato.20200630 header.b=GQmbxoGn; arc=none smtp.remote-ip=90.155.92.199 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F8D5606EE99; Thu, 5 Nov 2026 02:01:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 3F8D5606EE99 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=desiato.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=desiato.20200630 header.b=GQmbxoGn DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description; bh=ihM6W4dC3NBcXSxQRanAZyoU5ddBab3XKFn/HuD51Wc=; b=GQmbxoGnz87qdbqTY6OXzQHbYj I4kFK9Kdgbo09j349zxtFiYaP6kwlwpp6ZqTcIOhbhPwIoBcG/X5Ghe0ngQNSbFSynEktxRg1ebzz FbTbmXwtuyt2pshjrp6YUqClUSde9rqEHfogjfaRdzy3xWe5vkVpOqfFMdianJnxsBLJIIb6b7zhu LIZmPR7ZN9YNdhXMw/KTqxO34uCvBoFhQsooHC0rtz5s7Xw6FN2Z4AhOd2OB7OYqPWVZg45oir7Fd oLJJ/8hLKJbcTLKb+PiHCYP8cRhBGZVOXWS14ZJD0avSmM5NKG7soPeLGMmwBKhO7Q7jAQeTBIhcI 7O8PPKFQ==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by desiato.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmn-00000002ROM-26z6; Thu, 05 Nov 2026 02:01:05 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmm-0000000AWD7-0ouU; Thu, 05 Nov 2026 02:01:04 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 02/20] KVM: x86/xen: Add KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE Date: Thu, 5 Nov 2026 02:00:44 +0000 Message-ID: <20261105020104.2507083-3-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by desiato.infradead.org. See http://www.infradead.org/rpr.html From: David Woodhouse Commit 3617c0ee7dec ("KVM: x86/xen: Only write Xen hypercall page for guest writes to MSR") blocked host-initiated writes from triggering the Xen hypercall page setup, to fix an SRCU usage violation when the hypercall MSR index collides with a real MSR written during vCPU reset. However, some VMMs legitimately need to trigger hypercall page setup from host context. For example, a VMM may intercept the guest's MSR write to track an epoch (for kexec/crash recovery), and then replay the write as a host-initiated KVM_SET_MSRS to populate the hypercall page. The host_initiated check breaks this use case. Add KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE as a vCPU attribute which invokes the write explicitly. The work moves into a helper called with xen_lock held =E2=80=94 which the attribute handler already hol= ds, and which now covers the whole operation, serializing the writes to guest memory =E2=80=94 in the shape the original posting's review asked for: handled in the attribute case-statement, with no locking games of its own. The helper returns raw errnos for each caller to map onto its own contract: the attribute returns them to userspace as they are, while the guest WRMSR raises #GP for anything but -ENOMEM. That includes a fault reading the VMM-supplied blob, which previously escaped to userspace as -EFAULT: by that point it is indistinguishable from the guest's own bad page_addr, and #GP serves them both. The helper also reverts the latched long_mode when the shared_info re-init fails =E2=80=94 on both entry points, including the guest WRMSR path =E2=80=94 where the old code left it latched, letting a retry of the same write skip the re-init and "succeed" with the wallclock still published at the offset for the old bitness. The Fixes: tag documents which commit regressed the use case; since this patch adds new UAPI (an attribute and a capability bit), it is not suitable for backporting to stable kernels. Fixes: 3617c0ee7dec ("KVM: x86/xen: Only write Xen hypercall page for guest= writes to MSR") Signed-off-by: David Woodhouse --- Documentation/virt/kvm/api.rst | 22 +++- arch/x86/include/uapi/asm/kvm.h | 3 + arch/x86/kvm/x86.c | 3 +- arch/x86/kvm/xen.c | 73 ++++++++--- .../selftests/kvm/x86/xen_vmcall_test.c | 118 ++++++++++++++++++ 5 files changed, 203 insertions(+), 16 deletions(-) diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index e95c9cda085d..70818f1353c3 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -5848,6 +5848,23 @@ KVM_XEN_VCPU_ATTR_TYPE_UPCALL_VECTOR vector configured with HVM_PARAM_CALLBACK_IRQ. It is disabled by setting the vector to zero. =20 +KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE + This attribute is available when the KVM_CAP_XEN_HVM ioctl indicates + support for KVM_XEN_HVM_CONFIG_WRITE_HYPERCALL_PAGE. It populates the + Xen hypercall page at the guest physical address given in ``gpa``, + just as a guest write to the hypercall MSR would. This is for VMMs + which intercept the guest's MSR write (e.g. to track an epoch for + kexec/crash recovery) and then need to perform the write themselves: + host-initiated writes through KVM_SET_MSRS do not populate the page. + As for the guest's own write, the vCPU's current long mode is latched + as the VM-wide Xen long_mode, and the shared_info wallclock is + re-published if that changed. The value is interpreted exactly as the + guest's MSR write would be, not as a plain address: when Xen hypercall + interception is enabled, any non-zero bits below the page size make + the write fail with -EINVAL; without interception, those bits select + which page of the VMM-provided hypercall blob is written, and a page + number beyond the blob fails with -EINVAL. + =20 4.129 KVM_XEN_VCPU_GET_ATTR --------------------------- @@ -5861,7 +5878,8 @@ KVM_XEN_VCPU_ATTR_TYPE_UPCALL_VECTOR Allows Xen vCPU attributes to be read. For the structure and types, see KVM_XEN_VCPU_SET_ATTR above. =20 -The KVM_XEN_VCPU_ATTR_TYPE_RUNSTATE_ADJUST type may not be used +The KVM_XEN_VCPU_ATTR_TYPE_RUNSTATE_ADJUST and +KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE types may not be used with the KVM_XEN_VCPU_GET_ATTR ioctl. =20 4.130 KVM_ARM_MTE_COPY_TAGS @@ -9509,6 +9527,8 @@ PVHVM guests. Valid flags are:: #define KVM_XEN_HVM_CONFIG_EVTCHN_SEND (1 << 5) #define KVM_XEN_HVM_CONFIG_RUNSTATE_UPDATE_FLAG (1 << 6) #define KVM_XEN_HVM_CONFIG_PVCLOCK_TSC_UNSTABLE (1 << 7) + #define KVM_XEN_HVM_CONFIG_SHARED_INFO_HVA (1 << 8) + #define KVM_XEN_HVM_CONFIG_WRITE_HYPERCALL_PAGE (1 << 9) =20 The KVM_XEN_HVM_CONFIG_HYPERCALL_MSR flag indicates that the KVM_XEN_HVM_C= ONFIG ioctl is available, for the guest to set its hypercall page. diff --git a/arch/x86/include/uapi/asm/kvm.h b/arch/x86/include/uapi/asm/kv= m.h index 00ce265391e1..e53a26370e50 100644 --- a/arch/x86/include/uapi/asm/kvm.h +++ b/arch/x86/include/uapi/asm/kvm.h @@ -596,6 +596,7 @@ struct kvm_x86_mce { #define KVM_XEN_HVM_CONFIG_RUNSTATE_UPDATE_FLAG (1 << 6) #define KVM_XEN_HVM_CONFIG_PVCLOCK_TSC_UNSTABLE (1 << 7) #define KVM_XEN_HVM_CONFIG_SHARED_INFO_HVA (1 << 8) +#define KVM_XEN_HVM_CONFIG_WRITE_HYPERCALL_PAGE (1 << 9) =20 #define KVM_XEN_MSR_MIN_INDEX 0x40000000u #define KVM_XEN_MSR_MAX_INDEX 0x4fffffffu @@ -704,6 +705,8 @@ struct kvm_xen_vcpu_attr { #define KVM_XEN_VCPU_ATTR_TYPE_UPCALL_VECTOR 0x8 /* Available with KVM_CAP_XEN_HVM / KVM_XEN_HVM_CONFIG_SHARED_INFO_HVA */ #define KVM_XEN_VCPU_ATTR_TYPE_VCPU_INFO_HVA 0x9 +/* Available with KVM_CAP_XEN_HVM / KVM_XEN_HVM_CONFIG_WRITE_HYPERCALL_PAG= E */ +#define KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE 0xa =20 /* Secure Encrypted Virtualization command */ enum sev_cmd_id { diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 32563a91fba9..6a1f0404f029 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -2315,7 +2315,8 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, lon= g ext) KVM_XEN_HVM_CONFIG_EVTCHN_2LEVEL | KVM_XEN_HVM_CONFIG_EVTCHN_SEND | KVM_XEN_HVM_CONFIG_PVCLOCK_TSC_UNSTABLE | - KVM_XEN_HVM_CONFIG_SHARED_INFO_HVA; + KVM_XEN_HVM_CONFIG_SHARED_INFO_HVA | + KVM_XEN_HVM_CONFIG_WRITE_HYPERCALL_PAGE; if (sched_info_on()) r |=3D KVM_XEN_HVM_CONFIG_RUNSTATE | KVM_XEN_HVM_CONFIG_RUNSTATE_UPDATE_FLAG; diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c index 5875d10ae3a9..2718da1320e0 100644 --- a/arch/x86/kvm/xen.c +++ b/arch/x86/kvm/xen.c @@ -33,6 +33,7 @@ static int kvm_xen_set_evtchn(struct kvm_xen_evtchn *xe, struct kvm *kvm); static int kvm_xen_setattr_evtchn(struct kvm *kvm, struct kvm_xen_hvm_attr= *data); static bool kvm_xen_hcall_evtchn_send(struct kvm_vcpu *vcpu, u64 param, u6= 4 *r); +static int __kvm_xen_write_hypercall_page(struct kvm_vcpu *vcpu, u64 data); =20 DEFINE_STATIC_KEY_DEFERRED_FALSE(kvm_xen_enabled, HZ); =20 @@ -1177,6 +1178,15 @@ int kvm_xen_vcpu_set_attr(struct kvm_vcpu *vcpu, str= uct kvm_xen_vcpu_attr *data) } break; =20 + case KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE: + /* KVM cannot read a protected guest's bitness to latch it. */ + if (vcpu->arch.guest_state_protected) { + r =3D -EINVAL; + break; + } + r =3D __kvm_xen_write_hypercall_page(vcpu, data->u.gpa); + break; + default: break; } @@ -1312,30 +1322,42 @@ int kvm_xen_vcpu_get_attr(struct kvm_vcpu *vcpu, st= ruct kvm_xen_vcpu_attr *data) return r; } =20 -int kvm_xen_write_hypercall_page(struct kvm_vcpu *vcpu, u64 data) +/* + * Latch the guest's bitness, re-publish the wallclock in the right + * place for it if needed, and fill the hypercall page. Called with + * xen_lock held, which serializes the writes to guest memory, and + * within a kvm->srcu read section. + * + * Returns raw errnos for each caller to map onto its own contract: + * -EINVAL for an out-of-range page_num; -EFAULT for a fault writing + * the page, whether through the guest's page_addr or from the VMM's + * blob address; -ENOMEM from the page allocations. + */ +static int __kvm_xen_write_hypercall_page(struct kvm_vcpu *vcpu, u64 data) { struct kvm *kvm =3D vcpu->kvm; u32 page_num =3D data & ~PAGE_MASK; u64 page_addr =3D data & PAGE_MASK; bool lm =3D is_long_mode(vcpu); - int r =3D 0; =20 - mutex_lock(&kvm->arch.xen.xen_lock); + lockdep_assert_held(&kvm->arch.xen.xen_lock); + if (kvm->arch.xen.long_mode !=3D lm) { kvm->arch.xen.long_mode =3D lm; =20 /* * Re-initialize shared_info to put the wallclock in the - * correct place. + * correct place. On failure, revert the latched long_mode: + * leaving it latched would make a retry of the same write + * skip this block and "succeed" with the wallclock still + * published at the offset for the old bitness. */ if (kvm->arch.xen.shinfo_cache.active && - kvm_xen_shared_info_init(kvm)) - r =3D 1; + kvm_xen_shared_info_init(kvm)) { + kvm->arch.xen.long_mode =3D !lm; + return -EFAULT; + } } - mutex_unlock(&kvm->arch.xen.xen_lock); - - if (r) - return r; =20 /* * If Xen hypercall intercept is enabled, fill the hypercall @@ -1349,7 +1371,7 @@ int kvm_xen_write_hypercall_page(struct kvm_vcpu *vcp= u, u64 data) int ret, i; =20 if (page_num) - return 1; + return -EINVAL; =20 /* mov imm32, %eax */ instructions[0] =3D 0xb8; @@ -1376,7 +1398,7 @@ int kvm_xen_write_hypercall_page(struct kvm_vcpu *vcp= u, u64 data) ret =3D kvm_vcpu_write_guest(vcpu, page_addr, page, PAGE_SIZE); kfree(page); if (ret) - return 1; + return ret; } else { /* * Note, truncation is a non-issue as 'lm' is guaranteed to be @@ -1390,7 +1412,7 @@ int kvm_xen_write_hypercall_page(struct kvm_vcpu *vcp= u, u64 data) int ret; =20 if (page_num >=3D blob_size) - return 1; + return -EINVAL; =20 blob_addr +=3D page_num * PAGE_SIZE; =20 @@ -1401,11 +1423,34 @@ int kvm_xen_write_hypercall_page(struct kvm_vcpu *v= cpu, u64 data) ret =3D kvm_vcpu_write_guest(vcpu, page_addr, page, PAGE_SIZE); kfree(page); if (ret) - return 1; + return ret; } return 0; } =20 +/* The guest WRMSR entry point. */ +int kvm_xen_write_hypercall_page(struct kvm_vcpu *vcpu, u64 data) +{ + struct kvm *kvm =3D vcpu->kvm; + int r; + + mutex_lock(&kvm->arch.xen.xen_lock); + r =3D __kvm_xen_write_hypercall_page(vcpu, data); + mutex_unlock(&kvm->arch.xen.xen_lock); + + /* + * Any failure to populate the page raises #GP; only -ENOMEM is + * signalled upwards. The one deliberate change: a fault reading + * the VMM-supplied blob previously escaped to userspace as + * -EFAULT, but by this point it is indistinguishable from the + * guest's own bad page_addr, and #GP serves them both. + */ + if (r && r !=3D -ENOMEM) + return 1; + + return r; +} + int kvm_xen_hvm_config(struct kvm *kvm, struct kvm_xen_hvm_config *xhc) { /* Only some feature flags need to be *enabled* by userspace */ diff --git a/tools/testing/selftests/kvm/x86/xen_vmcall_test.c b/tools/test= ing/selftests/kvm/x86/xen_vmcall_test.c index 2585087cdf5c..85d836ddbdb8 100644 --- a/tools/testing/selftests/kvm/x86/xen_vmcall_test.c +++ b/tools/testing/selftests/kvm/x86/xen_vmcall_test.c @@ -12,6 +12,8 @@ #include "processor.h" #include "hyperv.h" =20 +#include + #define HCALL_REGION_GPA 0xc0000000ULL #define HCALL_REGION_SLOT 10 =20 @@ -26,6 +28,10 @@ #define HVCALL_SIGNAL_EVENT 0x005d #define HV_STATUS_INVALID_ALIGNMENT 4 =20 +enum { + TEST_WRITE_HYPERCALL_PAGE =3D 1, +}; + static void guest_code(void) { unsigned long rax =3D INPUTVALUE; @@ -76,18 +82,74 @@ static void guest_code(void) "r"(r8)); GUEST_ASSERT(rax =3D=3D HV_STATUS_INVALID_ALIGNMENT); =20 + /* + * Test KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE: ask userspace + * to set up MSR filtering, then write the MSR. The WRMSR will exit + * to userspace (not populate the page). Userspace verifies the page + * is empty, uses the attr to populate it, then resumes us. + */ + GUEST_SYNC(TEST_WRITE_HYPERCALL_PAGE); + + __asm__ __volatile__("wrmsr" : : "c" (XEN_HYPERCALL_MSR), + "a" (HCALL_REGION_GPA & 0xffffffff), + "d" (HCALL_REGION_GPA >> 32)); + + /* Userspace populated the page via the attr =E2=80=94 verify it works */ + rax =3D INPUTVALUE; + rdi =3D ARGVALUE(1); + rsi =3D ARGVALUE(2); + rdx =3D ARGVALUE(3); + r10 =3D ARGVALUE(4); + r8 =3D ARGVALUE(5); + r9 =3D ARGVALUE(6); + __asm__ __volatile__("call *%1" : "=3Da"(rax) : + "r"(HCALL_REGION_GPA + INPUTVALUE * 32), + "a"(rax), "D"(rdi), "S"(rsi), "d"(rdx), + "r"(r10), "r"(r8), "r"(r9)); + GUEST_ASSERT(rax =3D=3D RETVALUE); + GUEST_DONE(); } =20 +static void setup_msr_filter(struct kvm_vm *vm) +{ + uint64_t deny_bits =3D 0; + struct kvm_msr_filter filter =3D { + .flags =3D KVM_MSR_FILTER_DEFAULT_ALLOW, + .ranges =3D { + { + .flags =3D KVM_MSR_FILTER_WRITE, + .nmsrs =3D 1, + .base =3D XEN_HYPERCALL_MSR, + .bitmap =3D (uint8_t *)&deny_bits, + }, + }, + }; + + vm_ioctl(vm, KVM_X86_SET_MSR_FILTER, &filter); +} + int main(int argc, char *argv[]) { unsigned int xen_caps; struct kvm_vcpu *vcpu; struct kvm_vm *vm; + bool msr_filter_ready =3D false; + bool attr_tested =3D false; =20 xen_caps =3D kvm_check_cap(KVM_CAP_XEN_HVM); TEST_REQUIRE(xen_caps & KVM_XEN_HVM_CONFIG_INTERCEPT_HCALL); =20 + /* + * Without these, skip only the WRITE_HYPERCALL_PAGE leg: the MSR + * filter is never installed, so the guest's second WRMSR fills + * the page through KVM as the first did, and the rest of the + * test still runs on older kernels. + */ + bool have_whp =3D (xen_caps & KVM_XEN_HVM_CONFIG_WRITE_HYPERCALL_PAGE) && + kvm_check_cap(KVM_CAP_X86_USER_SPACE_MSR) && + kvm_check_cap(KVM_CAP_X86_MSR_FILTER); + vm =3D vm_create_with_one_vcpu(&vcpu, guest_code); vcpu_set_hv_cpuid(vcpu); =20 @@ -123,6 +185,42 @@ int main(int argc, char *argv[]) continue; } =20 + if (run->exit_reason =3D=3D KVM_EXIT_X86_WRMSR) { + /* MSR filter caught the Xen hypercall MSR write */ + TEST_ASSERT(msr_filter_ready, + "Unexpected WRMSR exit before filter setup"); + TEST_ASSERT_EQ(run->msr.index, XEN_HYPERCALL_MSR); + + /* + * The MSR filter should have prevented KVM from + * populating the page. Verify it's empty. + */ + uint8_t *hcall_page =3D addr_gpa2hva(vm, HCALL_REGION_GPA); + + TEST_ASSERT_EQ(hcall_page[0], 0); + + /* + * Now use the attr to populate the page, as a + * VMM would after intercepting the MSR write. + */ + struct kvm_xen_vcpu_attr attr =3D { + .type =3D KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE, + .u.gpa =3D HCALL_REGION_GPA, + }; + vcpu_ioctl(vcpu, KVM_XEN_VCPU_SET_ATTR, &attr); + + /* Verify the page is now populated */ + TEST_ASSERT_EQ(hcall_page[0], 0xb8); + TEST_ASSERT_EQ(*(uint32_t *)&hcall_page[1], 0); + TEST_ASSERT_EQ(hcall_page[INPUTVALUE * 32], 0xb8); + TEST_ASSERT_EQ(*(uint32_t *)&hcall_page[INPUTVALUE * 32 + 1], + INPUTVALUE); + attr_tested =3D true; + + run->msr.error =3D 0; + continue; + } + TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO); =20 switch (get_ucall(vcpu, &uc)) { @@ -130,6 +228,21 @@ int main(int argc, char *argv[]) REPORT_GUEST_ASSERT(uc); /* NOT REACHED */ case UCALL_SYNC: + TEST_ASSERT_EQ(uc.args[1], TEST_WRITE_HYPERCALL_PAGE); + + if (!have_whp) + break; + + /* + * Guest is about to write the Xen MSR. Clear the + * hypercall page, install an MSR filter to intercept + * the write, and enable userspace MSR exits. + */ + memset(addr_gpa2hva(vm, HCALL_REGION_GPA), 0, PAGE_SIZE); + vm_enable_cap(vm, KVM_CAP_X86_USER_SPACE_MSR, + KVM_MSR_EXIT_REASON_FILTER); + setup_msr_filter(vm); + msr_filter_ready =3D true; break; case UCALL_DONE: goto done; @@ -138,6 +251,11 @@ int main(int argc, char *argv[]) } } done: + if (have_whp) + TEST_ASSERT(attr_tested, + "WRITE_HYPERCALL_PAGE attr was never exercised"); + else + pr_info("Skipped KVM_XEN_VCPU_ATTR_TYPE_WRITE_HYPERCALL_PAGE\n"); kvm_vm_free(vm); return 0; } --=20 2.43.0 From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.50.34 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844072; b=ju1EVJ3aFW0P6yVRj7yYKT3zeDWipaBS7dNiX2kv7Uq8E/3EBTRwEwtqcJXrkIqD0BR5 F2st1cP0/eCqbSwjStHJe4n39cJmAL81bSzVp+POv7VXITkym2HGgDvYP6fZn9Td6vHUU 05aQYmVZevqBzMEBQ4n6eLZARl5SqkKS3DqdgXivOAgPYoMINj3fmFer6qMGRwhC7w74z Kb4cUMjwu8OllIrf8SgZCQ1DYCsZHdigjbKvnZnMmk5Xo2JCHm9FQ57ROgWJWUILN8lAg lD/tbWZQ0hasWZMHK7nK9AtxOXc/ZnJRF78Udc0X1bxIIW95ldqSurY/nqw6xtQ0giw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844072; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Type:Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=5NTMmASLeB2L/VhizvxNP+opz3b0eVIoDdUhideV8+0=; b=Ae9CB01L9lU7OBU5rEYrHRtWIaWoYGm0o/O6ozmkZsU3Kt/t2Fs0fOsdzpnvuyl9Gfd7 QyjK6HLWz16X+5r39WkgjiqwFT3ml1Zb/dMrJ4rcavSZu7CF8X1j92ECOJRDpNFFCI6j3 35abMwHoiu4ST4tpwoL/1TSORlti4xLc1MuWWngHbJYbm6RKgE+wf+3bQkReaqcQpA8vN enfaFoEWMCpkyUA7YZaOS8korVa8e1xgLAe/ouvqeuEvxrVAoC2uardvQHck2N4O4ATgA 7mmVbxLE8r2M85a0G03qR735qpoemx6hVB6gH08T9aQwqhDKFu8uuCfN14rs9Bhurzw== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=casper.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=glkXx2Rv; arc=none smtp.remote-ip=90.155.50.34 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DCB07606A33B; Thu, 5 Nov 2026 02:01:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org DCB07606A33B Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=casper.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=glkXx2Rv DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description; bh=5NTMmASLeB2L/VhizvxNP+opz3b0eVIoDdUhideV8+0=; b=glkXx2Rv8ZK4f/LZtNxR3J2Mxl YyjCguWA4Nhxpxlw3694qYHDyRqXMo89aEmyefHMJoXGQdS3DE6WO+SCHKkX/5+8R4ZU9QI3uO6lg Bc02QgRdQ9b+Hsm45zALUwhiWau9ScCoUlV3d7gmgBGDmV6aR1eUfNpg71igNIyaTP25IcXej+0MK Wc78I4Rc5o65fa31dEE4UBocG8I9ruE3Iiu0b/G3AJJcTME0UVfmx6oFmHSH6Pa+Y0aWmCC7HhXWC A8/bynSWpLWVV9hLLY4R1nymJVupSFlvvjWbhzjWgIxF6J9Hru59YtQbwCNl+hHB2eEbwxLiuqrG5 9RApx1oQ==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by casper.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmn-000000037j6-09Cw; Thu, 05 Nov 2026 02:01:05 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmm-0000000AWDA-0wz8; Thu, 05 Nov 2026 02:01:04 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 03/20] KVM: x86/xen: Extract delivery of event to vCPU into a separate helper Date: Thu, 5 Nov 2026 02:00:45 +0000 Message-ID: <20261105020104.2507083-4-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html From: Sean Christopherson Move the marking of an event as pending in the target vCPU's vcpu_info into a separate helper, __kvm_xen_set_evtchn_fast(), invoked after the shared_info processing is complete and the shared_info cache's read lock has been dropped. This makes both halves easier to read. Signed-off-by: Sean Christopherson [dwmw2: rebased onto the -EAGAIN series, and reworded. Sean's series first converted the GPC locks to read_trylock(), which this series does not carry, so the helper takes the plain read lock and the original's trylock failure path does not apply. The caller's kvm->srcu section extends across the helper call, since kvm_gpc_check() on the vcpu_info cache consults the memslot generation and the irqfd path enters holding only irq_srcu; in Sean's series that was covered by a guard(srcu) spanning the whole function. The original said "no functional change intended", but here the lock scopes do change: kvm_xen_check_poller() and the upcall vector injection now run outside the GPC read locks (and with interrupts restored), deliberately, so that the vCPU kick and the APIC map walk are not nested inside them.] Signed-off-by: David Woodhouse --- arch/x86/kvm/xen.c | 158 ++++++++++++++++++++++++++++----------------- 1 file changed, 97 insertions(+), 61 deletions(-) diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c index 2718da1320e0..4517eeb24909 100644 --- a/arch/x86/kvm/xen.c +++ b/arch/x86/kvm/xen.c @@ -1884,6 +1884,89 @@ static void kvm_xen_check_poller(struct kvm_vcpu *vc= pu, int port) } } =20 +static void __kvm_xen_set_evtchn_fast(struct kvm_vcpu *vcpu, int port_word= _bit, + bool has_64bit_shinfo) +{ + struct gfn_to_pfn_cache *gpc =3D &vcpu->arch.xen.vcpu_info_cache; + unsigned long vi_pending_sel_ofs; + bool inject_upcall =3D false; + bool kick_vcpu =3D false; + unsigned long flags; + bool old; + + vi_pending_sel_ofs =3D has_64bit_shinfo ? + offsetof(struct vcpu_info, evtchn_pending_sel) : + offsetof(struct compat_vcpu_info, evtchn_pending_sel); + + /* Now switch to the vCPU's vcpu_info to set the index and pending_sel */ + read_lock_irqsave(&gpc->lock, flags); + if (!kvm_gpc_check(gpc, sizeof(struct vcpu_info))) { + /* + * Could not access the vcpu_info. Set the bit in-kernel and + * prod the vCPU to deliver it for itself. Note that an MSI is + * *not* an acceptable substitute here even if the vCPU has an + * upcall vector: the pending bit only exists in + * evtchn_pending_sel so far, and the vCPU has to be forced + * out of the guest so that kvm_xen_inject_pending_events() + * can refresh the cache and propagate it to the guest. + */ + if (!test_and_set_bit(port_word_bit, &vcpu->arch.xen.evtchn_pending_sel)) + kick_vcpu =3D true; + goto out_unlock; + } + inject_upcall =3D true; + + /* + * Explicitly use a 32-bit btsl instead of test_and_set_bit(), which + * would use btsq on x86-64. The vcpu_info is guest-controlled and only + * required to be 32-bit aligned, so a 64-bit access could generate a + * split-lock #AC. + * + * Note, this does not apply to the test_and_set_bit() on pending_bits + * in the caller: that is in the per-VM shared_info, which is page + * aligned, so the access is guaranteed to be 64-bit aligned. + */ + old =3D GEN_BINARY_RMWcc(LOCK_PREFIX "btsl", + *(u32 *)(gpc->khva + vi_pending_sel_ofs), + c, "Ir", port_word_bit); + if (!old) { + struct vcpu_info *vi =3D gpc->khva; + + /* No need for compat handling */ + BUILD_BUG_ON(offsetof(struct vcpu_info, evtchn_upcall_pending) !=3D + offsetof(struct compat_vcpu_info, evtchn_upcall_pending)); + + WRITE_ONCE(vi->evtchn_upcall_pending, 1); + kick_vcpu =3D true; + } + +out_unlock: + read_unlock_irqrestore(&gpc->lock, flags); + + /* + * Deliver the upcall or kick the vCPU only after dropping the GPC + * read lock. Both paths end up in kvm_vcpu_kick(), and the MSI + * delivery also walks the APIC map and takes APIC locks; none of + * that wants to be nested inside the GPC read lock, which a + * concurrent invalidation may be waiting to take for write. + */ + if (!kick_vcpu) + return; + + /* + * For the per-vCPU lapic vector, deliver it as MSI =E2=80=94 but only if= the + * vcpu_info was actually updated above. If it wasn't, the vCPU must + * be kicked instead (see above). + */ + if (inject_upcall && vcpu->arch.xen.upcall_vector) { + kvm_xen_inject_vcpu_vector(vcpu); + return; + } + + kvm_make_request(KVM_REQ_UNBLOCK, vcpu); + kvm_vcpu_kick(vcpu); +} + /* * The return value from this function is propagated to kvm_set_irq() API, * so it returns: @@ -1898,11 +1981,10 @@ int kvm_xen_set_evtchn_fast(struct kvm_xen_evtchn *= xe, struct kvm *kvm) { struct gfn_to_pfn_cache *gpc =3D &kvm->arch.xen.shinfo_cache; bool has_64bit_shinfo =3D kvm_xen_has_64bit_shinfo(kvm); - unsigned long *pending_bits, *mask_bits, vi_pending_sel_ofs; + unsigned long *pending_bits, *mask_bits; struct kvm_vcpu *vcpu; unsigned long flags; int port_word_bit; - bool kick_vcpu =3D false; int vcpu_idx, idx, rc; =20 vcpu_idx =3D READ_ONCE(xe->vcpu_idx); @@ -1924,23 +2006,19 @@ int kvm_xen_set_evtchn_fast(struct kvm_xen_evtchn *= xe, struct kvm *kvm) =20 read_lock_irqsave(&gpc->lock, flags); if (!kvm_gpc_check(gpc, PAGE_SIZE)) - goto out_rcu; + goto out_unlock; =20 if (has_64bit_shinfo) { struct shared_info *shinfo =3D gpc->khva; pending_bits =3D (unsigned long *)&shinfo->evtchn_pending; mask_bits =3D (unsigned long *)&shinfo->evtchn_mask; port_word_bit =3D xe->port / 64; - - vi_pending_sel_ofs =3D offsetof(struct vcpu_info, evtchn_pending_sel); } else { struct compat_shared_info *shinfo =3D gpc->khva; pending_bits =3D (unsigned long *)&shinfo->evtchn_pending; mask_bits =3D (unsigned long *)&shinfo->evtchn_mask; port_word_bit =3D xe->port / 32; =20 - vi_pending_sel_ofs =3D offsetof(struct compat_vcpu_info, evtchn_pending_= sel); - /* test_and_set_bit() needs 64-bit alignment, but that's OK */ BUILD_BUG_ON(offsetof(struct compat_shared_info, evtchn_pending) & 7); } @@ -1956,66 +2034,24 @@ int kvm_xen_set_evtchn_fast(struct kvm_xen_evtchn *= xe, struct kvm *kvm) rc =3D 0; /* It was already raised */ } else if (test_bit(xe->port, mask_bits)) { rc =3D -ENOTCONN; /* Masked */ - kvm_xen_check_poller(vcpu, xe->port); } else { - bool old; - rc =3D 1; /* Delivered to the bitmap in shared_info. */ - /* Now switch to the vCPU's vcpu_info to set the index and pending_sel */ - read_unlock_irqrestore(&gpc->lock, flags); - gpc =3D &vcpu->arch.xen.vcpu_info_cache; - - read_lock_irqsave(&gpc->lock, flags); - if (!kvm_gpc_check(gpc, sizeof(struct vcpu_info))) { - /* - * Could not access the vcpu_info. Set the bit in-kernel - * and prod the vCPU to deliver it for itself. - */ - if (!test_and_set_bit(port_word_bit, &vcpu->arch.xen.evtchn_pending_sel= )) - kick_vcpu =3D true; - goto out_rcu; - } - - /* - * Explicitly use a 32-bit btsl instead of test_and_set_bit(), - * which would use btsq on x86-64. The vcpu_info is guest- - * controlled and only required to be 32-bit aligned, so a - * 64-bit access could generate a split-lock #AC. - * - * Note, this does not apply to the test_and_set_bit() on - * pending_bits above: that is in the per-VM shared_info, which - * is page aligned, so the access is guaranteed to be 64-bit - * aligned. - */ - old =3D GEN_BINARY_RMWcc(LOCK_PREFIX "btsl", - *(u32 *)(gpc->khva + vi_pending_sel_ofs), - c, "Ir", port_word_bit); - if (!old) { - struct vcpu_info *vi =3D gpc->khva; - - /* No need for compat handling */ - BUILD_BUG_ON(offsetof(struct vcpu_info, evtchn_upcall_pending) !=3D - offsetof(struct compat_vcpu_info, evtchn_upcall_pending)); - - WRITE_ONCE(vi->evtchn_upcall_pending, 1); - kick_vcpu =3D true; - } - - /* For the per-vCPU lapic vector, deliver it as MSI. */ - if (kick_vcpu && vcpu->arch.xen.upcall_vector) { - kvm_xen_inject_vcpu_vector(vcpu); - kick_vcpu =3D false; - } } =20 - out_rcu: + out_unlock: read_unlock_irqrestore(&gpc->lock, flags); - srcu_read_unlock(&kvm->srcu, idx); =20 - if (kick_vcpu) { - kvm_make_request(KVM_REQ_UNBLOCK, vcpu); - kvm_vcpu_kick(vcpu); - } + /* + * Both of these can kick the vCPU, so keep them outside the GPC + * read lock; an invalidation may be waiting to take it for write. + */ + if (rc =3D=3D -ENOTCONN) + kvm_xen_check_poller(vcpu, xe->port); + else if (rc =3D=3D 1) + __kvm_xen_set_evtchn_fast(vcpu, port_word_bit, + has_64bit_shinfo); + + srcu_read_unlock(&kvm->srcu, idx); =20 return rc; } --=20 2.43.0 From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.50.34 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844071; b=UFNWXQLcAg+x291Riv7cIRnnj3ZWaoWSYwLedA3Q9h1oG6Zbg7HZFl7KtACvREjHRcp8 YaevPrRa4rpBNdT6VcKbnFmccI0Uxx65g5ohqvsPgG40mWtyOXVB1TikEgE69G4wrTSTD rHJbXdCwgvLCR5dpk4l3mLlX0C6fAjHiNbBD8cHS2JoU4AT+QY1xlo249vgVf4ceWfJF2 hXnToAayukj6PLwDnL+vBZ65QWU//9aBQFi1rTwi115Ea6g/kSKuHVLeh2VHdoDj0BvTs dJutxYtGwkfzk07bq9m0gxSJ1iRnrfRJrnPsBpMbYfH8KFnIMsdHMWzjMGptH40hYqA== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844071; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=EjC93C9IT1HB5U8uFJ3DkGgCAi5EPZ8cZ/bI0C1PcNQ=; b=q+k2Z3VgsrBtW0vec+jeU/QOam1fgtA27RH8aNxSdXztX2EJkItz2dQaVLwuJg/I52g4 PoyKXEoNlrmlW5MHi+Cj1fEl0kWSNbFT5S1FeRkvMUnaPnvfMFgeaCGssZdQtkqzOxi8j 6Aprhi2/71dkPUFYg6wbZ71cQDCXWaCL5a7l2SSLsiUgX20e73NZFtYvI/DCsulZug75o zCVTXfga3xX4bLBuPr7vSkJvsWMxcJR75LLIsNBNpqZiasJ1WIsiD74RT/Ai3xCTibeXV +iJVTZXQXtmEivgbgOSJKwigrDwR/MCRO8t+/nUTdkCTAi5JMfPaLDhrbRNqih1u4RA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=casper.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=KZEVV+Cu; arc=none smtp.remote-ip=90.155.50.34 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAAE3606A335; Thu, 5 Nov 2026 02:01:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org DAAE3606A335 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=casper.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=KZEVV+Cu DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=EjC93C9IT1HB5U8uFJ3DkGgCAi5EPZ8cZ/bI0C1PcNQ=; b=KZEVV+Cusbxv+cqJ1kbyuEa7Xp Az5YvzTCVTn9BkjUQ7VoLal2X0iJkw80mTjX0LIKlBJtOMc6AFj5zUxXoY3upW8Y44SaabIXm0NT0 gMFVL5VU6OwVFfCc8BjLvtJ8NedNhW/ullEBc3evR79R3CpbEijH9bn0OUb0na0yElKw3Mf3upGBk HT6tNjFHToAEN7pEX52g2+Akf+UGqrtNXYXd3Mz2aAdy0hmTBOshjxqKlq6/TkHb8vwl+/Z/IIcAA aVfCQ2V9x73SR9sYDQ79umK5i4H/e4pCwDw5/TkyFevg+8n+okm/x/oFvt3MXcC2HPXn0K1I4Ik57 rhdArMew==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by casper.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmn-000000037j8-0FsG; Thu, 05 Nov 2026 02:01:05 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmm-0000000AWDD-150I; Thu, 05 Nov 2026 02:01:04 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 04/20] KVM: x86/xen: Explicitly tag "shared info" page as never being dirty tracked Date: Thu, 5 Nov 2026 02:00:46 +0000 Message-ID: <20261105020104.2507083-5-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Content-Type: text/plain; charset="utf-8" From: Sean Christopherson Explicitly mark the Xen shared info page as never being dirty tracked so that higher-level gpc APIs can be added to automatically take care of things like dirty tracking, without reintroducing the bug fixed by commit 55749769fe60 ("KVM: x86: Fix wall clock writes in Xen shared_info not to mark page dirty"). And because the code _looks_ buggy. No functional change intended. Signed-off-by: Sean Christopherson Signed-off-by: David Woodhouse --- arch/x86/kvm/xen.c | 2 +- include/linux/kvm_host.h | 15 +++++++++++---- include/linux/kvm_types.h | 1 + virt/kvm/pfncache.c | 4 +++- 4 files changed, 16 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c index 4517eeb24909..d3773416cd44 100644 --- a/arch/x86/kvm/xen.c +++ b/arch/x86/kvm/xen.c @@ -2485,7 +2485,7 @@ void kvm_xen_init_vm(struct kvm *kvm) { mutex_init(&kvm->arch.xen.xen_lock); xa_init(&kvm->arch.xen.evtchn_ports); - kvm_gpc_init(&kvm->arch.xen.shinfo_cache, kvm); + __kvm_gpc_init(&kvm->arch.xen.shinfo_cache, kvm, true); } =20 void kvm_xen_destroy_vm(struct kvm *kvm) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index e7fc94e2910a..cadd0318400e 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1459,16 +1459,23 @@ int kvm_vcpu_write_guest(struct kvm_vcpu *vcpu, gpa= _t gpa, const void *data, unsigned long len); =20 /** - * kvm_gpc_init - initialize gfn_to_pfn_cache. + * __kvm_gpc_init - initialize gfn_to_pfn_cache. * * @gpc: struct gfn_to_pfn_cache object. * @kvm: pointer to kvm instance. + * @never_dirty: the cache never needs to mark its page dirty in its mem= slot. * * This sets up a gfn_to_pfn_cache by initializing locks and assigning the * immutable attributes. Note, the cache must be zero-allocated (or zeroe= d by - * the caller before init). + * the caller before init). Most callers want the kvm_gpc_init() wrapper. */ -void kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm); +void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm, + bool never_dirty); + +static inline void kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *= kvm) +{ + __kvm_gpc_init(gpc, kvm, false); +} =20 /** * kvm_gpc_activate - prepare a cached kernel mapping and HPA for a given = guest @@ -1979,7 +1986,7 @@ static inline void kvm_gpc_mark_dirty_in_slot(struct = gfn_to_pfn_cache *gpc) { lockdep_assert_held(&gpc->lock); =20 - if (!gpc->memslot) + if (!gpc->memslot || gpc->never_dirty) return; =20 mark_page_dirty_in_slot(gpc->kvm, gpc->memslot, gpa_to_gfn(gpc->gpa)); diff --git a/include/linux/kvm_types.h b/include/linux/kvm_types.h index a568d8e6f4e8..e850adc3f47e 100644 --- a/include/linux/kvm_types.h +++ b/include/linux/kvm_types.h @@ -94,6 +94,7 @@ struct gfn_to_pfn_cache { kvm_pfn_t pfn; bool active; bool valid; + bool never_dirty; }; =20 #ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c index 3659686b97c2..17a30da00d40 100644 --- a/virt/kvm/pfncache.c +++ b/virt/kvm/pfncache.c @@ -382,7 +382,8 @@ int kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, unsig= ned long len) return __kvm_gpc_refresh(gpc, gpc->gpa, uhva); } =20 -void kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm) +void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm, + bool never_dirty) { rwlock_init(&gpc->lock); mutex_init(&gpc->refresh_lock); @@ -392,6 +393,7 @@ void kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct = kvm *kvm) gpc->gpa =3D INVALID_GPA; gpc->uhva =3D KVM_HVA_ERR_BAD; gpc->active =3D gpc->valid =3D false; + gpc->never_dirty =3D never_dirty; } =20 static int __kvm_gpc_activate(struct gfn_to_pfn_cache *gpc, gpa_t gpa, uns= igned long uhva, --=20 2.43.0 From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.50.34 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844071; b=4jZdBQja2CCufCPPAaFUTQoiEtfy8/nqw9NGayDxk2MOGcTtsZuDdJxSflQ+LDNXVtjq S9Ob5uhbK65UsFIDtcWR6HT28o01rWIbJCx4qDdWOtIe0bf/AGA7he6eBu334qpiGptLQ g0d0ztmZ5UPRfKXB74aPNQ6n10e9otjbdJpExSshGkBaBb8hBQO77r7B867bYXFDx9lJz abmjhNVQhcz3AWU1KTV+WtGs2CW1d55wJ9E3rKyoWAI7bicuHxTT9fAiw20fGt5Goq6DR s4ZTLo8ncUZMekmemDkc95/hKQEa5IWEO3eXZnfn7KmaciSH7WxfWQw6+35Hae9fRSQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844071; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=xWajFXWJuXrJAKbtDPmQO5pr3uDz4D2E84mtmYB7ov4=; b=gXU05PTlXrsj37KTehux6QjDp+Ee5N1YIqdQhY6aWMBcAHTm+6F51Dqk7DUsjPKrRtD7 /KnkDSwfHJNTVUUM5m9wH2JLBScAdPN3HVWkB6O7ux2sFOyO/i9wpRGakz31Yu6IfDaiD TYAwwhNoslHSSTz+NEMY/HhyZelqTcESMwlTt6YZXde1uRHSGFA8TWV5m4/wLA/GDq1iW gYgaWA2O8Ug6KfB27AeyRKu1wfyeOAzK5dEzJFj+MXbYaDPVYceyRLM3H5alTR1bBJuaA eDS93RgJXvEsT0Y4uBfjXnAYP0VRyHH248QdjQMYUc9RvlLAlbLsiFVpX5EYJE6nRQQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=casper.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=CgaDSHDY; arc=none smtp.remote-ip=90.155.50.34 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAB4A606A337; Thu, 5 Nov 2026 02:01:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org DAB4A606A337 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=casper.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=CgaDSHDY DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=xWajFXWJuXrJAKbtDPmQO5pr3uDz4D2E84mtmYB7ov4=; b=CgaDSHDYP24zDUkZZWqry3n+XW Ls4wgdyfS5KdZRdyHotvJlWydY5wyGPFx+/mA5yY2cEZJLSRlJIRQGZSBZ6/Dy1kY7/E27Z6TiiWv UPOezUssoxV5mGUXGBfomkcvTDpnCspYIcEKZoYbOSA2j3xXCQZfd5CKGZw2mHr0yuE6E5TkRblRI 5GGi9rMgYbiMv77gmysNS8FRowdkf6WQnYexSzPNbWql5VO8lcvPfMgF78nfXaNFQHEaOwKhPqfh2 5ZghbuuxdzH+GHAjHbuXA1OJDaBEArYsrTuKzD1pFDjyXTaZfCU0FZN5TBajF0+uPNchZi47YBi9I B/rGPOsw==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by casper.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmn-000000037j7-0HhA; Thu, 05 Nov 2026 02:01:05 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmm-0000000AWDG-1DI6; Thu, 05 Nov 2026 02:01:04 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 05/20] KVM: x86/xen: Don't dirty track "vCPU info" page Date: Thu, 5 Nov 2026 02:00:47 +0000 Message-ID: <20261105020104.2507083-6-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Content-Type: text/plain; charset="utf-8" From: Sean Christopherson Give the Xen per-vCPU info page the same treatment as the per-VM shared info page, and never mark it dirty, as KVM clearly relies on userspace to assume the page is always dirty. While the page is marked dirty on writes via kvm_xen_inject_pending_events(), it's not marked dirty when written by __kvm_xen_set_evtchn_fast(). Furthermore, as was the case with the shared info page, writes in the event channel fastpath may be done without an active vCPU, e.g. when called via timer callback or irqfd injection. I.e. attempting to fix the fastpath would run afoul of the same issue that was fixed by commit 55749769fe60 ("KVM: x86: Fix wall clock writes in Xen shared_info not to mark page dirty"). This also covers the pvclock writes kvm_guest_time_update() makes through the same cache; userspace assuming the page is always dirty covers those equally. Signed-off-by: Sean Christopherson [dwmw2: also update the KVM_GET_DIRTY_LOG note in api.rst, which named only the shared_info page] Signed-off-by: David Woodhouse --- Documentation/virt/kvm/api.rst | 5 +++-- arch/x86/kvm/xen.c | 3 +-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index 70818f1353c3..6f0500f7ffe2 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -385,8 +385,9 @@ The bits in the dirty bitmap are cleared before the ioc= tl returns, unless KVM_CAP_MANUAL_DIRTY_LOG_PROTECT2 is enabled. For more information, see the description of the capability. =20 -Note that the Xen shared_info page, if configured, shall always be assumed -to be dirty. KVM will not explicitly mark it such. +Note that the Xen shared_info page and vcpu_info area, if configured, +shall always be assumed to be dirty. KVM will not explicitly mark them +such. =20 =20 4.10 KVM_RUN diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c index d3773416cd44..e54988522b8b 100644 --- a/arch/x86/kvm/xen.c +++ b/arch/x86/kvm/xen.c @@ -706,7 +706,6 @@ void kvm_xen_inject_pending_events(struct kvm_vcpu *v) WRITE_ONCE(vi->evtchn_upcall_pending, 1); } =20 - kvm_gpc_mark_dirty_in_slot(gpc); read_unlock_irqrestore(&gpc->lock, flags); =20 /* For the per-vCPU lapic vector, deliver it as MSI. */ @@ -2464,7 +2463,7 @@ void kvm_xen_init_vcpu(struct kvm_vcpu *vcpu) =20 kvm_gpc_init(&vcpu->arch.xen.runstate_cache, vcpu->kvm); kvm_gpc_init(&vcpu->arch.xen.runstate2_cache, vcpu->kvm); - kvm_gpc_init(&vcpu->arch.xen.vcpu_info_cache, vcpu->kvm); + __kvm_gpc_init(&vcpu->arch.xen.vcpu_info_cache, vcpu->kvm, true); kvm_gpc_init(&vcpu->arch.xen.vcpu_time_info_cache, vcpu->kvm); } =20 --=20 2.43.0 From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.50.34 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844072; b=pkvjtUBHzuAKm9eA+0tLb6dJr+cp/oYWz6moJhY7QfHZA5u4x0F3uh46Y1ZJnYh1kqH4 XDXVskJR5/10gSxuUI0idEMMqpZNjXcLoXu2VbDjDmsm9uAc6o9jWjKMMm6yrSnBuwyQm D7F8s4qaCPqMDpok93i6N/kreTPT3ZNCFk+4n405I3IYTPG4BJhHWsl4faYPLrBO+DPjW a8lPgkcc3PtXZ4kwizoF6GGra/TzA/UtqjYxWk0nX6tGa5ZJcc9nvjeMydamY932uN/8Q UOynvJ6wEEXqjAxNrznG3+LS5xP6XkUzDbJM2uyHnw9tfETLeELCuDc69NhrYZVhxLg== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844072; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Type:Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=lXeVdKZgplIsnHrxxLZrBVIeJZ4mQkNiuMtdZoSEMdo=; b=Pm/vdXk9kodH/2OJSAsykU2zJs3VBuCDtqhjNV1YJlqFAmJbJc2nCuu0gePzIy/aOn2M JAAHDkGa2qPmO2QK8jjxmnTj6RygPLVHfZJTOQkijUxOpxSZbFfrcBN2DJYmRUlv86F98 fnZxSKv7y0Kv5Bqe/Axovqv3IInHKdWMO8LgVjNbQ08AAun+GAXcIm9Ioi5LPyRw9bE8Y 6m4wF1QisBkkdWi+PNDqUPJndAMBZT2DoLmkCPZedptoS3fKScKwqKhK+H7fFW26XmOpT TUhqADDSGUL0wvH1jJ2OC7LnbUs+dB5rSnywb82QcqWJSd5+iflhcfOZFl6iJfDy13A== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=casper.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=Q0mTKClI; arc=none smtp.remote-ip=90.155.50.34 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAA5E606A332; Thu, 5 Nov 2026 02:01:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org DAA5E606A332 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=casper.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=Q0mTKClI DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description; bh=lXeVdKZgplIsnHrxxLZrBVIeJZ4mQkNiuMtdZoSEMdo=; b=Q0mTKClI9VooOYT679tqFrQtl6 xuCH0mZbWbK+FGnQpTilLo0MkCIpY8grbwbYQEqNuCnyjvdA8+Opdm7YOWLuyX96rDEz6WHzB0JK/ XaYo0OtsN1p2LxySiL65rwFkuU5lvRSJfdYSbSwgBZ70DANcbpyhNn5R9wZAznZ6+pWWKn4Zv5xKE 8if70cDKkbabaiNilOaf6GyvgNyXmGzDBwuDNWdxY48W07eIuvmiZNvQGWjD4FJz7vN+RWMZImJ/A kykuL6uVDvg8KGxzjzg5WdX31h8n0QVDI0HlaMtdw9Yj0pr2NsDt8Kj4gnEOPpX3gpo3LxZlzHxsf me7f1aOA==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by casper.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmn-000000037j3-0IyH; Thu, 05 Nov 2026 02:01:05 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmm-0000000AWDJ-1Lfa; Thu, 05 Nov 2026 02:01:04 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 06/20] KVM: x86: Request the guest TLB flush from record_steal_time() Date: Thu, 5 Nov 2026 02:00:48 +0000 Message-ID: <20261105020104.2507083-7-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html From: David Woodhouse record_steal_time() performs a TLB flush on the guest's behalf when the guest sets KVM_VCPU_FLUSH_TLB. With shadow paging, kvm_vcpu_flush_tlb_guest() ends up in mmu_sync_children(), which can reschedule via cond_resched_rwlock_write(). That is why the flush currently sits between a user_access_end() and a fresh user_access_begin(): it cannot be done while the guest page is mapped for access. Raise KVM_REQ_TLB_FLUSH_GUEST instead of flushing inline, and move the processing of KVM_REQ_STEAL_UPDATE ahead of where the TLB flush requests are serviced in vcpu_enter_guest() so that the flush still happens before the vCPU enters the guest. There is no requirement for it to complete any earlier than that. kvm_make_request() on the vCPU which is running is nothing more than a barrier and a set_bit(), so unlike the flush itself it can be done from any context =E2=80=94 which lets the surrounding access be simplified later. Note that the xchg() must stay where it is: it atomically consumes any KVM_VCPU_FLUSH_TLB which the guest has set, as required by commit b043138246a4 ("x86/KVM: Make sure KVM_VCPU_FLUSH_TLB flag is not missed"). Reported-by: Sashiko AI review Closes: https://lore.kernel.org/all/20260530061932.7849D1F00893@smtp.kernel= .org Suggested-by: Sean Christopherson Signed-off-by: David Woodhouse --- arch/x86/kvm/x86.c | 26 +++++++++++++++++++++++--- 1 file changed, 23 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 6a1f0404f029..3c26c0eb9865 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -2117,8 +2117,22 @@ static void record_steal_time(struct kvm_vcpu *vcpu) =20 trace_kvm_pv_tlb_flush(vcpu->vcpu_id, st_preempted & KVM_VCPU_FLUSH_TLB); + /* + * Request the flush instead of performing it inline. With + * shadow paging kvm_vcpu_flush_tlb_guest() can reschedule + * (via mmu_sync_children()), so it could not be called while + * the user access above was open. Requesting is just a bit + * set on the running vCPU, and the request is serviced + * before the vCPU enters the guest; nothing requires the + * flush to have completed any earlier than that. + * + * Note the xchg above must remain as-is: it atomically + * consumes any KVM_VCPU_FLUSH_TLB the guest set, as required + * by commit b043138246a4 ("x86/KVM: Make sure + * KVM_VCPU_FLUSH_TLB flag is not missed"). + */ if (st_preempted & KVM_VCPU_FLUSH_TLB) - kvm_vcpu_flush_tlb_guest(vcpu); + kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); =20 if (!user_access_begin(st, sizeof(*st))) goto dirty; @@ -8168,6 +8182,14 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) if (unlikely(r)) goto out; } + /* + * Process the steal time update before the TLB flush requests + * are serviced below; record_steal_time() requests + * KVM_REQ_TLB_FLUSH_GUEST on the guest's behalf rather than + * performing the flush itself. + */ + if (kvm_check_request(KVM_REQ_STEAL_UPDATE, vcpu)) + record_steal_time(vcpu); if (kvm_check_request(KVM_REQ_MMU_SYNC, vcpu)) kvm_mmu_sync_roots(vcpu); if (kvm_check_request(KVM_REQ_LOAD_MMU_PGD, vcpu)) @@ -8217,8 +8239,6 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) r =3D 1; goto out; } - if (kvm_check_request(KVM_REQ_STEAL_UPDATE, vcpu)) - record_steal_time(vcpu); if (kvm_check_request(KVM_REQ_PMU, vcpu)) kvm_pmu_handle_event(vcpu); if (kvm_check_request(KVM_REQ_PMI, vcpu)) --=20 2.43.0 From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.50.34 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844071; b=O1th+aQp6OT3P5PBXZwiXxArlJtrblsv2xLmRjvCBglwxVeupfl4Jdhcg+vN7GX45nS4 6AhKhWjohIbxqfoUIqtth1LvxaQbMpGof9CUAS/DOnLDy4EtTXnAQoSE9NQKZXCBc6qLm xYDsKz8sHwzmLjGFdNBl+2ZOZr9J/6wSSaZ5WHXnZLAl+kGPQQnKCLFYsvLs6nWcOKFrw 9Vum+rvht35TR/RmFdo8xORBNRXLeFhC0J+zuDEAHQSgHy/7Jpa0FMJDexGJ/d0dN7dja tDW+Q9KjdyvbLB7W7kof7H7FOMVPnF3N16cfCxQp6OeQLXnsLyPNksZMtetgX0dvnOQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844071; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Type:Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=lwhWK78aKBCZA+QaUNKsEkehxG8l623JQyS+IL7qpjA=; b=LpwjvyUcE3Cp5CW5F29tbKJyEc3HwdarOvqykCg1MqUYi+k1wR4gMQR7o8IzKaIqpBzo M4t8Iawm5WEDSmrs0PpavDVOpDsqgyEWgvYkdCJ8RTTOGwYNl07V6uRo4e5uIu3FuPg9t XgdXDrz1Hob32zWhAlPSD9Py1Giw/bovvEwKLXUoRuGsyjgJhb9zLcmvgNP8s+XlBjWzR Zssm/wQR1eVwPF3nbFpg9x8R7sVa7DWKgd29s/Vf1j44n8Ifl10hJt9P8srT2JlXSoJIA /IHAeRvamdaER5a8r6VZZjrh36F83a0fdfaRmMyKKJ2Ejjd75ARRGP1kd3LSz9FLwYw== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=casper.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=n8WkuTIU; arc=none smtp.remote-ip=90.155.50.34 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA9CB606A32E; Thu, 5 Nov 2026 02:01:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org DA9CB606A32E Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=casper.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=n8WkuTIU DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description; bh=lwhWK78aKBCZA+QaUNKsEkehxG8l623JQyS+IL7qpjA=; b=n8WkuTIUJBAEQLjaPqryS5ggCl y26l0X14vqxw718ZEtCq79SsSiA3OnMAVSMEjK1rF0ZVRMizKHjmNfBVE34V2CDiw6ty3ekC/1coo Puv8PqQzow2JBcn/eyqjUN/e6nt0/lZXDkjoTvECAqi7kmuXsB0tMZE6CEtfcnybTDYTYAgOmIqs2 PL7l3sokUGAUH7FjWGiLeMtD0nxKN/c/hU7QlyX+31PQYkih3kaogc7KUI81DGy1+dRPpYkhl6y/z iyMwF/EB2yZ5PT5xKjFakTTUeamZR9CLG3g7vma6bqzUzaQATSq8T8nxnnknhgYvJWDNBaJhC20cw qMpOOAvA==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by casper.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmn-000000037j5-0GGC; Thu, 05 Nov 2026 02:01:05 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmm-0000000AWDM-1U12; Thu, 05 Nov 2026 02:01:04 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 07/20] KVM: selftests: Actually deactivate the HVA-based shared_info cache in xen_shinfo_test Date: Thu, 5 Nov 2026 02:00:49 +0000 Message-ID: <20261105020104.2507083-8-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html From: David Woodhouse The "deactivate by HVA" leg of juggle_shinfo_state() uses attribute type KVM_XEN_ATTR_TYPE_SHARED_INFO with .u.shared_info.hva =3D 0. The hva field shares a union with gfn, so this activates the shared_info cache at guest GFN 0 =E2=80=94 zero is not KVM_XEN_INVALID_GFN =E2=80=94 in= stead of deactivating it. The juggle thread therefore cycles through an unintended fourth state, with three consequences: - Every "deactivation" splats wallclock data over guest frame 0. - The HVA-based deactivation path is never tested at all. - In the TEST_LOCKING_POLL_RACE phase, a SCHEDOP_poll which samples the shared_info in this state finds evtchn_pending[0] bit 0 clear (the real shared_info page has it set; frame 0 does not) and blocks in kvm_vcpu_halt() =E2=80=94 the only state in which that poll can block, as a deactivated cache fails kvm_gpc_check() and reports the port pending. If the timer event is then dropped, nothing ever wakes the guest and the test hangs. Use KVM_XEN_ATTR_TYPE_SHARED_INFO_HVA, where hva =3D 0 deactivates. Fixes: 9397b5334af1 ("KVM: selftests: map Xen's shared_info page using HVA = rather than GFN") Reported-by: sashiko-bot@kernel.org Closes: https://lore.kernel.org/all/sashiko-outbox-177931@kernel.org Signed-off-by: David Woodhouse --- tools/testing/selftests/kvm/x86/xen_shinfo_test.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/testing/selftests/kvm/x86/xen_shinfo_test.c b/tools/test= ing/selftests/kvm/x86/xen_shinfo_test.c index aaa414c677ad..fb3c834c7cb7 100644 --- a/tools/testing/selftests/kvm/x86/xen_shinfo_test.c +++ b/tools/testing/selftests/kvm/x86/xen_shinfo_test.c @@ -410,7 +410,7 @@ static void *juggle_shinfo_state(void *arg) }; =20 struct kvm_xen_hvm_attr cache_deactivate_hva =3D { - .type =3D KVM_XEN_ATTR_TYPE_SHARED_INFO, + .type =3D KVM_XEN_ATTR_TYPE_SHARED_INFO_HVA, .u.shared_info.hva =3D 0 }; =20 --=20 2.43.0 From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.50.34 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844072; b=zwqx1HHKjiQmp9Y4LUkENxr+NZ7WUMMojLTnyKiBNxVDkuC4n0YxXw7oui0GkJoBFhSe IIvNatZVKeMOpqVcarTeLNpYvgYYycmL2UxkZ7RaX97r3uzENJgF6X48KFJ7Wc83dOsxd 2B60urUHP27U3eg6p9bB2L5E6LJ2QDXhrk30NRIQ0GPxF1++yceQmQevFs+6zBxgtpXyv /ce2HW/IY4TEQDXp5RsAT7wRfUlT0pAPTxz6lhr7s8jVkQ49X4qz651UBA8WRj51owxhQ mcWkO+g7XcqWWo5RGhKr3LfiK9DS2xiGxJi4h105t58FbzV/QyUjsuMVtC7a3jX7UPg== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844072; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Type:Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=UgyLs6o4Orjp7r9TrZvgcRHUHKMCRka330m3aCKWdEQ=; b=V0O40xGlelVl2O7tBdXxrewmAAxbKCBJ1zqCBvG3gsjsVuNDuQKqcYqOIcei5rj9lc1O kcEelcjwdVdxq5O0HSlrdd0NUZ1uTltVrNKPuXssGeMR/8tCNI4rhqMTPpA1pW10ztMCG gh0VT+0zgi2bb5bW4U/LNM1fC+pgGKZcfVAwWLR4CA16suEFRl/Ed3PBXKvstTa/dbHEt rk3GQKpNSagz3AFfch6c32iGYeewqz+GJwQ5UL93g5PdlclsCFvzuO4qvusxLtgNi0jVz ydZes2V0CE/jzA72SvQlZpVP+DUhLmEM14uwqDTG+WXzeEkLUhmrK9AEiJv3Bp34rfQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=casper.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=MujwxWKs; arc=none smtp.remote-ip=90.155.50.34 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAB18606A336; Thu, 5 Nov 2026 02:01:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org DAB18606A336 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=casper.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=MujwxWKs DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description; bh=UgyLs6o4Orjp7r9TrZvgcRHUHKMCRka330m3aCKWdEQ=; b=MujwxWKsiDg/pCHAc4bLWCV5Cg 7QHM7AcNwnwbTY80H3ZNP4o2CQ0jIkAmUbdvSl6Ml2XqU3IE947Ow+x5FsJreKPvBEoVdd80bfyXZ 5ChIZKpFWO5KujoGUe5MEDIWV40DaDY6V2DcQgdyqeZLI2nvYqrz8JXEm3UoIH0f/5bE3wN0xnX3K q4IOiBPV/Uz0tLXn9farqxTLb/uh+AptumD61Zb5+UGWS+VhgK0IdvPJne/TV0O96Bmk9OAa6IOcl ijX97daC6NziEYUSVQsUA78budms2uHb2dLCScJ54mU5MllRA1mfJRrL29TqZtf92qGI637kp/mVc 4oA/ephw==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by casper.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmn-000000037jA-0D4N; Thu, 05 Nov 2026 02:01:05 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmm-0000000AWDP-1ey4; Thu, 05 Nov 2026 02:01:04 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 08/20] KVM: pfncache: Use atomic SRCU for readers instead of a rwlock Date: Thu, 5 Nov 2026 02:00:50 +0000 Message-ID: <20261105020104.2507083-9-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html From: David Woodhouse Replace the per-cache rwlock with a dedicated per-VM atomic SRCU domain (kvm->gpc_srcu) for the read side. Readers now run under srcu_read_lock_atomic() alone: preemption disabled and non-blocking enforced, which every GPC read-side section already satisfied as a former rwlock critical section. It works in any context (including hardirq and sched-out paths) and never fails or spins, eliminating the read_trylock() contortions in the atomic paths, the double-lock dance with the lockdep subclass hack in the runstate update, and the PREEMPT_RT problems inherent to taking an rwlock in those contexts. In particular, kvm_xen_set_evtchn_fast() is called from hardirq context (timer callback, kvm_arch_set_irq_inatomic()), where taking gpc->lock is a sleeping-lock-in-atomic-context bug on PREEMPT_RT. The mutators' reader drains use synchronize_srcu_atomic(), which spins for the bounded readers instead of sleeping: no path waiting out GPC readers ever schedules. That satisfies the OOM-reaper non-blocking requirement in the invalidation walk, whose wait must happen even on unblockable ranges =E2=80=94 GPC readers never allocate, take mmap_lock, or sleep, so waiting them out cannot deadlock against reclaim. SRCU rather than plain RCU so that wait covers only this VM's GPC readers, not a grace period across every pCPU in the system; the read-side cost is a this_cpu_inc() and a barrier. Without gpc->lock, the refresh can no longer rely on holding it to make its final invalidation check and its publish atomic with respect to the mmu_notifier walk, and can no longer treat "GPC_VALID already clear" as "no readers remain". Three state bits in a single atomic word replace it: GPC_BECOMING_VALID lets the walk veto an in-flight refresh's pending publish (the publish is a cmpxchg which fails if the bit was cleared, closing the window between the refresh's last check and its publish); GPC_INVALIDATING records that a grace period for the previously published mapping is still owed: whoever clears GPC_VALID leaves the marker standing until their grace-period wait completes, so that a later mutator =E2=80=94 or a later mmu_notifier walk, whose return is what lets the primary MMU free the page =E2=80=94 finding GPC_VALID already clear still performs a wait of its own rather than trusting one it cannot see finish. The comments in pfncache.c and kvm_types.h carry the full protocol. As before, a refresh which resolves to the same uHVA in the same memslot leaves the cache valid and undisturbed, skipping the grace period: routine host memory management (page migration, NUMA balancing, KSM) usually re-backs the same uHVA, and a grace period in the reader's refresh path for each such event would add seconds to a guest boot. Not suitable for stable backport despite the Fixes tag (it depends on the atomic SRCU reader flavour, new in this cycle); the tag documents where the PREEMPT_RT sleeping-lock problem was introduced. Fixes: 14243b387137 ("KVM: x86/xen: Add KVM_IRQ_ROUTING_XEN_EVTCHN and even= t channel delivery") Reported-by: syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com Closes: https://syzkaller.appspot.com/bug?extid=3D208f7f3e5f59c11aeb90 Reported-by: syzbot+919877893c9d28162dc2@syzkaller.appspotmail.com Closes: https://syzkaller.appspot.com/bug?extid=3D919877893c9d28162dc2 Tested-by: syzbot+919877893c9d28162dc2@syzkaller.appspotmail.com Signed-off-by: David Woodhouse --- arch/x86/kvm/x86.c | 10 +- arch/x86/kvm/xen.c | 146 ++++++------ include/linux/kvm_host.h | 26 ++- include/linux/kvm_types.h | 37 ++- include/linux/srcu.h | 7 + virt/kvm/kvm_main.c | 9 + virt/kvm/pfncache.c | 481 ++++++++++++++++++++++++++++++-------- 7 files changed, 528 insertions(+), 188 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 3c26c0eb9865..f5f3459ed4a5 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -1747,18 +1747,18 @@ static void kvm_setup_guest_pvclock(struct pvclock_= vcpu_time_info *ref_hv_clock, { struct pvclock_vcpu_time_info *guest_hv_clock; struct pvclock_vcpu_time_info hv_clock; - unsigned long flags; + int idx; =20 memcpy(&hv_clock, ref_hv_clock, sizeof(hv_clock)); =20 - read_lock_irqsave(&gpc->lock, flags); + idx =3D srcu_read_lock_atomic(&vcpu->kvm->gpc_srcu); while (!kvm_gpc_check(gpc, offset + sizeof(*guest_hv_clock))) { - read_unlock_irqrestore(&gpc->lock, flags); + srcu_read_unlock_atomic(&vcpu->kvm->gpc_srcu, idx); =20 if (kvm_gpc_refresh(gpc, offset + sizeof(*guest_hv_clock))) return; =20 - read_lock_irqsave(&gpc->lock, flags); + idx =3D srcu_read_lock_atomic(&vcpu->kvm->gpc_srcu); } =20 guest_hv_clock =3D (void *)(gpc->khva + offset); @@ -1783,7 +1783,7 @@ static void kvm_setup_guest_pvclock(struct pvclock_vc= pu_time_info *ref_hv_clock, guest_hv_clock->version =3D ++hv_clock.version; =20 kvm_gpc_mark_dirty_in_slot(gpc); - read_unlock_irqrestore(&gpc->lock, flags); + srcu_read_unlock_atomic(&vcpu->kvm->gpc_srcu, idx); =20 trace_kvm_pvclock_update(vcpu->vcpu_id, &hv_clock); } diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c index e54988522b8b..13e287bc17bb 100644 --- a/arch/x86/kvm/xen.c +++ b/arch/x86/kvm/xen.c @@ -46,16 +46,17 @@ static int kvm_xen_shared_info_init(struct kvm *kvm) u64 wall_nsec; int ret =3D 0; int idx =3D srcu_read_lock(&kvm->srcu); + int gpc_idx; =20 - read_lock_irq(&gpc->lock); + gpc_idx =3D srcu_read_lock_atomic(&kvm->gpc_srcu); while (!kvm_gpc_check(gpc, PAGE_SIZE)) { - read_unlock_irq(&gpc->lock); + srcu_read_unlock_atomic(&kvm->gpc_srcu, gpc_idx); =20 ret =3D kvm_gpc_refresh(gpc, PAGE_SIZE); if (ret) goto out; =20 - read_lock_irq(&gpc->lock); + gpc_idx =3D srcu_read_lock_atomic(&kvm->gpc_srcu); } =20 /* @@ -98,7 +99,7 @@ static int kvm_xen_shared_info_init(struct kvm *kvm) smp_wmb(); =20 wc->version =3D wc_version + 1; - read_unlock_irq(&gpc->lock); + srcu_read_unlock_atomic(&kvm->gpc_srcu, gpc_idx); =20 out: srcu_read_unlock(&kvm->srcu, idx); @@ -155,22 +156,21 @@ static int xen_get_guest_pvclock(struct kvm_vcpu *vcp= u, struct gfn_to_pfn_cache *gpc, unsigned int offset) { - unsigned long flags; - int r; + int idx, r; =20 - read_lock_irqsave(&gpc->lock, flags); + idx =3D srcu_read_lock_atomic(&vcpu->kvm->gpc_srcu); while (!kvm_gpc_check(gpc, offset + sizeof(*hv_clock))) { - read_unlock_irqrestore(&gpc->lock, flags); + srcu_read_unlock_atomic(&vcpu->kvm->gpc_srcu, idx); =20 r =3D kvm_gpc_refresh(gpc, offset + sizeof(*hv_clock)); if (r) return r; =20 - read_lock_irqsave(&gpc->lock, flags); + idx =3D srcu_read_lock_atomic(&vcpu->kvm->gpc_srcu); } =20 memcpy(hv_clock, gpc->khva + offset, sizeof(*hv_clock)); - read_unlock_irqrestore(&gpc->lock, flags); + srcu_read_unlock_atomic(&vcpu->kvm->gpc_srcu, idx); =20 /* * Sanity check TSC shift+multiplier to verify the guest's view of time @@ -325,12 +325,12 @@ static void kvm_xen_update_runstate_guest(struct kvm_= vcpu *v, bool atomic) struct gfn_to_pfn_cache *gpc2 =3D &vx->runstate2_cache; size_t user_len, user_len1, user_len2; struct vcpu_runstate_info rs; - unsigned long flags; size_t times_ofs; uint8_t *update_bit =3D NULL; uint64_t entry_time; uint64_t *rs_times; int *rs_state; + int idx; =20 /* * The only difference between 32-bit and 64-bit versions of the @@ -417,20 +417,12 @@ static void kvm_xen_update_runstate_guest(struct kvm_= vcpu *v, bool atomic) =20 retry: /* - * Attempt to obtain the GPC lock on *both* (if there are two) - * gfn_to_pfn caches that cover the region. + * Check *both* (if there are two) gfn_to_pfn caches that cover + * the region, under a single SRCU read-side critical section. */ - if (atomic) { - local_irq_save(flags); - if (!read_trylock(&gpc1->lock)) { - local_irq_restore(flags); - return; - } - } else { - read_lock_irqsave(&gpc1->lock, flags); - } + idx =3D srcu_read_lock_atomic(&v->kvm->gpc_srcu); while (!kvm_gpc_check(gpc1, user_len1)) { - read_unlock_irqrestore(&gpc1->lock, flags); + srcu_read_unlock_atomic(&v->kvm->gpc_srcu, idx); =20 /* When invoked from kvm_sched_out() we cannot sleep */ if (atomic) @@ -439,7 +431,7 @@ static void kvm_xen_update_runstate_guest(struct kvm_vc= pu *v, bool atomic) if (kvm_gpc_refresh(gpc1, user_len1)) return; =20 - read_lock_irqsave(&gpc1->lock, flags); + idx =3D srcu_read_lock_atomic(&v->kvm->gpc_srcu); } =20 if (likely(!user_len2)) { @@ -459,24 +451,11 @@ static void kvm_xen_update_runstate_guest(struct kvm_= vcpu *v, bool atomic) } else { /* * The guest's runstate_info is split across two pages and we - * need to hold and validate both GPCs simultaneously. We can - * declare a lock ordering GPC1 > GPC2 because nothing else - * takes them more than one at a time. Set a subclass on the - * gpc1 lock to make lockdep shut up about it. + * need to validate both GPCs simultaneously. They are both + * covered by the single SRCU read-side critical section above. */ - lock_set_subclass(&gpc1->lock.dep_map, 1, _THIS_IP_); - if (atomic) { - if (!read_trylock(&gpc2->lock)) { - read_unlock_irqrestore(&gpc1->lock, flags); - return; - } - } else { - read_lock(&gpc2->lock); - } - if (!kvm_gpc_check(gpc2, user_len2)) { - read_unlock(&gpc2->lock); - read_unlock_irqrestore(&gpc1->lock, flags); + srcu_read_unlock_atomic(&v->kvm->gpc_srcu, idx); =20 /* When invoked from kvm_sched_out() we cannot sleep */ if (atomic) @@ -575,13 +554,11 @@ static void kvm_xen_update_runstate_guest(struct kvm_= vcpu *v, bool atomic) smp_wmb(); } =20 - if (user_len2) { + if (user_len2) kvm_gpc_mark_dirty_in_slot(gpc2); - read_unlock(&gpc2->lock); - } =20 kvm_gpc_mark_dirty_in_slot(gpc1); - read_unlock_irqrestore(&gpc1->lock, flags); + srcu_read_unlock_atomic(&v->kvm->gpc_srcu, idx); } =20 void kvm_xen_update_runstate(struct kvm_vcpu *v, int state) @@ -656,7 +633,7 @@ void kvm_xen_inject_pending_events(struct kvm_vcpu *v) { unsigned long evtchn_pending_sel =3D READ_ONCE(v->arch.xen.evtchn_pending= _sel); struct gfn_to_pfn_cache *gpc =3D &v->arch.xen.vcpu_info_cache; - unsigned long flags; + int idx; =20 if (!evtchn_pending_sel) return; @@ -666,14 +643,14 @@ void kvm_xen_inject_pending_events(struct kvm_vcpu *v) * does anyway. Page it in and retry the instruction. We're just a * little more honest about it. */ - read_lock_irqsave(&gpc->lock, flags); + idx =3D srcu_read_lock_atomic(&v->kvm->gpc_srcu); while (!kvm_gpc_check(gpc, sizeof(struct vcpu_info))) { - read_unlock_irqrestore(&gpc->lock, flags); + srcu_read_unlock_atomic(&v->kvm->gpc_srcu, idx); =20 if (kvm_gpc_refresh(gpc, sizeof(struct vcpu_info))) return; =20 - read_lock_irqsave(&gpc->lock, flags); + idx =3D srcu_read_lock_atomic(&v->kvm->gpc_srcu); } =20 /* Now gpc->khva is a valid kernel address for the vcpu_info */ @@ -706,7 +683,7 @@ void kvm_xen_inject_pending_events(struct kvm_vcpu *v) WRITE_ONCE(vi->evtchn_upcall_pending, 1); } =20 - read_unlock_irqrestore(&gpc->lock, flags); + srcu_read_unlock_atomic(&v->kvm->gpc_srcu, idx); =20 /* For the per-vCPU lapic vector, deliver it as MSI. */ if (v->arch.xen.upcall_vector) @@ -716,8 +693,8 @@ void kvm_xen_inject_pending_events(struct kvm_vcpu *v) int __kvm_xen_has_interrupt(struct kvm_vcpu *v) { struct gfn_to_pfn_cache *gpc =3D &v->arch.xen.vcpu_info_cache; - unsigned long flags; u8 rc =3D 0; + int idx; =20 /* * If the global upcall vector (HVMIRQ_callback_vector) is set and @@ -742,9 +719,9 @@ int __kvm_xen_has_interrupt(struct kvm_vcpu *v) */ guard(srcu)(&v->kvm->srcu); =20 - read_lock_irqsave(&gpc->lock, flags); + idx =3D srcu_read_lock_atomic(&v->kvm->gpc_srcu); while (!kvm_gpc_check(gpc, sizeof(struct vcpu_info))) { - read_unlock_irqrestore(&gpc->lock, flags); + srcu_read_unlock_atomic(&v->kvm->gpc_srcu, idx); =20 /* * This function gets called from kvm_vcpu_block() after setting the @@ -764,11 +741,11 @@ int __kvm_xen_has_interrupt(struct kvm_vcpu *v) */ return 0; } - read_lock_irqsave(&gpc->lock, flags); + idx =3D srcu_read_lock_atomic(&v->kvm->gpc_srcu); } =20 rc =3D ((struct vcpu_info *)gpc->khva)->evtchn_upcall_pending; - read_unlock_irqrestore(&gpc->lock, flags); + srcu_read_unlock_atomic(&v->kvm->gpc_srcu, idx); return rc; } =20 @@ -803,6 +780,13 @@ int kvm_xen_hvm_set_attr(struct kvm *kvm, struct kvm_x= en_hvm_attr *data) =20 mutex_lock(&kvm->arch.xen.xen_lock); =20 + /* + * The cache's own readers hold kvm->gpc_srcu, but activating + * it resolves the gfn through kvm_memslots(), which is + * protected by kvm->srcu. (Holding kvm->srcu across the + * deactivation is also fine: its grace-period wait is on the + * gpc_srcu domain, not this one.) + */ idx =3D srcu_read_lock(&kvm->srcu); =20 if (data->type =3D=3D KVM_XEN_ATTR_TYPE_SHARED_INFO) { @@ -937,6 +921,14 @@ int kvm_xen_vcpu_set_attr(struct kvm_vcpu *vcpu, struc= t kvm_xen_vcpu_attr *data) int idx, r =3D -ENOENT; =20 mutex_lock(&vcpu->kvm->arch.xen.xen_lock); + + /* + * Activating the caches below resolves gfns through kvm_memslots(), + * and the runstate cases read it via kvm_gpc_check(); both are + * protected by kvm->srcu (the caches' readers hold kvm->gpc_srcu, + * which does not cover the memslots, and whose grace-period waits + * are safe to perform with kvm->srcu held). + */ idx =3D srcu_read_lock(&vcpu->kvm->srcu); =20 switch (data->type) { @@ -1532,14 +1524,13 @@ static bool wait_pending_event(struct kvm_vcpu *vcp= u, bool has_64bit_shinfo, struct kvm *kvm =3D vcpu->kvm; struct gfn_to_pfn_cache *gpc =3D &kvm->arch.xen.shinfo_cache; unsigned long *pending_bits; - unsigned long flags; bool ret =3D true; - int idx, i; + int idx, gpc_idx, i; =20 idx =3D srcu_read_lock(&kvm->srcu); - read_lock_irqsave(&gpc->lock, flags); + gpc_idx =3D srcu_read_lock_atomic(&kvm->gpc_srcu); if (!kvm_gpc_check(gpc, PAGE_SIZE)) - goto out_rcu; + goto out_gpc_srcu; =20 ret =3D false; if (has_64bit_shinfo) { @@ -1557,8 +1548,8 @@ static bool wait_pending_event(struct kvm_vcpu *vcpu,= bool has_64bit_shinfo, } } =20 - out_rcu: - read_unlock_irqrestore(&gpc->lock, flags); + out_gpc_srcu: + srcu_read_unlock_atomic(&kvm->gpc_srcu, gpc_idx); srcu_read_unlock(&kvm->srcu, idx); =20 return ret; @@ -1883,6 +1874,15 @@ static void kvm_xen_check_poller(struct kvm_vcpu *vc= pu, int port) } } =20 +/* + * Called with kvm->srcu held, as kvm_gpc_check() consults the memslots. + * + * The caller passes its own has_64bit_shinfo snapshot, from which it also + * derived port_word_bit; re-reading the mode here could race with a + * concurrent mode flip and pair a 64-bit bit index with the compat + * evtchn_pending_sel offset (or vice versa), writing outside the intended + * field within the guest's vcpu_info. + */ static void __kvm_xen_set_evtchn_fast(struct kvm_vcpu *vcpu, int port_word= _bit, bool has_64bit_shinfo) { @@ -1890,15 +1890,15 @@ static void __kvm_xen_set_evtchn_fast(struct kvm_vc= pu *vcpu, int port_word_bit, unsigned long vi_pending_sel_ofs; bool inject_upcall =3D false; bool kick_vcpu =3D false; - unsigned long flags; bool old; + int idx; =20 vi_pending_sel_ofs =3D has_64bit_shinfo ? offsetof(struct vcpu_info, evtchn_pending_sel) : offsetof(struct compat_vcpu_info, evtchn_pending_sel); =20 /* Now switch to the vCPU's vcpu_info to set the index and pending_sel */ - read_lock_irqsave(&gpc->lock, flags); + idx =3D srcu_read_lock_atomic(&vcpu->kvm->gpc_srcu); if (!kvm_gpc_check(gpc, sizeof(struct vcpu_info))) { /* * Could not access the vcpu_info. Set the bit in-kernel and @@ -1940,14 +1940,17 @@ static void __kvm_xen_set_evtchn_fast(struct kvm_vc= pu *vcpu, int port_word_bit, } =20 out_unlock: - read_unlock_irqrestore(&gpc->lock, flags); + srcu_read_unlock_atomic(&vcpu->kvm->gpc_srcu, idx); =20 /* * Deliver the upcall or kick the vCPU only after dropping the GPC * read lock. Both paths end up in kvm_vcpu_kick(), and the MSI * delivery also walks the APIC map and takes APIC locks; none of - * that wants to be nested inside the GPC read lock, which a - * concurrent invalidation may be waiting to take for write. + * that wants to be nested inside the GPC read-side critical + * section, which must be no longer than the accesses to gpc->khva + * above. Invalidation waits for a grace period, so holding the + * read lock across the kick would extend how long a memory + * invalidation is blocked. */ if (!kick_vcpu) return; @@ -1982,9 +1985,8 @@ int kvm_xen_set_evtchn_fast(struct kvm_xen_evtchn *xe= , struct kvm *kvm) bool has_64bit_shinfo =3D kvm_xen_has_64bit_shinfo(kvm); unsigned long *pending_bits, *mask_bits; struct kvm_vcpu *vcpu; - unsigned long flags; int port_word_bit; - int vcpu_idx, idx, rc; + int vcpu_idx, idx, gpc_idx, rc; =20 vcpu_idx =3D READ_ONCE(xe->vcpu_idx); if (vcpu_idx >=3D 0) @@ -2003,9 +2005,9 @@ int kvm_xen_set_evtchn_fast(struct kvm_xen_evtchn *xe= , struct kvm *kvm) =20 idx =3D srcu_read_lock(&kvm->srcu); =20 - read_lock_irqsave(&gpc->lock, flags); + gpc_idx =3D srcu_read_lock_atomic(&kvm->gpc_srcu); if (!kvm_gpc_check(gpc, PAGE_SIZE)) - goto out_unlock; + goto out_gpc_srcu; =20 if (has_64bit_shinfo) { struct shared_info *shinfo =3D gpc->khva; @@ -2037,12 +2039,12 @@ int kvm_xen_set_evtchn_fast(struct kvm_xen_evtchn *= xe, struct kvm *kvm) rc =3D 1; /* Delivered to the bitmap in shared_info. */ } =20 - out_unlock: - read_unlock_irqrestore(&gpc->lock, flags); + out_gpc_srcu: + srcu_read_unlock_atomic(&kvm->gpc_srcu, gpc_idx); =20 /* * Both of these can kick the vCPU, so keep them outside the GPC - * read lock; an invalidation may be waiting to take it for write. + * read-side critical section; invalidation waits for a grace period. */ if (rc =3D=3D -ENOTCONN) kvm_xen_check_poller(vcpu, xe->port); diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index cadd0318400e..3ed0b8c9c102 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -798,6 +798,16 @@ struct kvm { /* For management / invalidation of gfn_to_pfn_caches */ spinlock_t gpc_lock; struct list_head gpc_list; + /* + * Protects the contents of active gfn_to_pfn_caches. Readers hold it + * across kvm_gpc_check() and any subsequent access to gpc->khva; the + * invalidation and refresh paths clear GPC_VALID in gpc->state and + * then wait for a grace period before mutating anything a reader + * might be looking at. + * A dedicated domain (rather than kvm->srcu) keeps those waits from + * being lengthened by unrelated memslot readers. + */ + struct srcu_struct gpc_srcu; =20 /* * created_vcpus is protected by kvm->lock, and is incremented @@ -1520,12 +1530,9 @@ int kvm_gpc_activate_hva(struct gfn_to_pfn_cache *gp= c, unsigned long hva, unsign * @return: %true if the cache is still valid and the address matches. * %false if the cache is not valid. * - * Callers outside IN_GUEST_MODE context should hold a read lock on @gpc->= lock - * while calling this function, and then continue to hold the lock until t= he - * access is complete. - * - * Callers in IN_GUEST_MODE may do so without locking, although they should - * still hold a read lock on kvm->scru for the memslot checks. + * Callers must hold a read lock on @gpc->kvm->gpc_srcu across this functi= on + * and any subsequent access to the target page, and must hold a read lock= on + * kvm->srcu for the memslot checks. */ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigned long len); =20 @@ -1542,8 +1549,8 @@ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsi= gned long len); * This will attempt to refresh a gfn_to_pfn_cache. Note that a successful * return from this function does not mean the page can be immediately * accessed because it may have raced with an invalidation. Callers must - * still lock and check the cache status, as this function does not return - * with the lock still held to permit access. + * still check the cache status under a kvm->gpc_srcu read lock, via + * kvm_gpc_check(), before accessing the target page. */ int kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, unsigned long len); =20 @@ -1984,7 +1991,8 @@ static inline bool kvm_is_gpa_in_memslot(struct kvm *= kvm, gpa_t gpa) =20 static inline void kvm_gpc_mark_dirty_in_slot(struct gfn_to_pfn_cache *gpc) { - lockdep_assert_held(&gpc->lock); + RCU_LOCKDEP_WARN(!srcu_read_lock_held(&gpc->kvm->gpc_srcu), + "kvm_gpc_mark_dirty_in_slot() without kvm->gpc_srcu read lock"); =20 if (!gpc->memslot || gpc->never_dirty) return; diff --git a/include/linux/kvm_types.h b/include/linux/kvm_types.h index e850adc3f47e..22393f97266a 100644 --- a/include/linux/kvm_types.h +++ b/include/linux/kvm_types.h @@ -88,15 +88,48 @@ struct gfn_to_pfn_cache { struct kvm_memory_slot *memslot; struct kvm *kvm; struct list_head list; - rwlock_t lock; struct mutex refresh_lock; void *khva; kvm_pfn_t pfn; bool active; - bool valid; + /* + * GPC_VALID, GPC_BECOMING_VALID and GPC_INVALIDATING live in a + * single atomic word so that a refresh can publish + * (BECOMING_VALID =3D> VALID) with a single conditional RMW, an + * invalidation can veto that publish by clearing both of those bits + * (setting INVALIDATING in the same operation if VALID was set, to + * record that its grace period is still owed), and a mutator can + * consume the whole state with one xchg to learn whether it must + * wait for readers before touching anything they see. See the + * comments in pfncache.c. + * + * 'active' deliberately stays separate: the invalidation and + * mutation paths operate on the word without having to care about + * racing with deactivation, which is ordered by refresh_lock + * instead. + */ + atomic_t state; bool never_dirty; }; =20 +/* The cache is valid: pfn and khva may be used by readers. */ +#define GPC_VALID 0x1 +/* + * A refresh is in flight and intends to publish. Set before the uhva is + * resolved, so an invalidation of that uhva can veto the publish even + * though the cache is not yet valid. + */ +#define GPC_BECOMING_VALID 0x2 +/* + * GPC_VALID has been cleared but the grace period for readers of the + * previously published mapping has not yet completed. Nothing may mutate + * reader-visible fields (pfn, khva, uhva, gpa, memslot) until whoever + * next needs to do so has waited it out. Set by whichever party clears + * GPC_VALID; cleared after a synchronize_srcu() which began after the + * VALID clear. + */ +#define GPC_INVALIDATING 0x4 + #ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE /* * Memory caches are used to preallocate memory ahead of various MMU flows, diff --git a/include/linux/srcu.h b/include/linux/srcu.h index 1a8a465a5650..f7a6bbcabc4e 100644 --- a/include/linux/srcu.h +++ b/include/linux/srcu.h @@ -706,6 +706,13 @@ DEFINE_LOCK_GUARD_1(srcu, struct srcu_struct, DECLARE_LOCK_GUARD_1_ATTRS(srcu, __acquires_shared(_T), __releases_shared(= *(struct srcu_struct **)_T)) #define class_srcu_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(srcu, _T) =20 +DEFINE_LOCK_GUARD_1(srcu_atomic, struct srcu_struct, + _T->idx =3D srcu_read_lock_atomic(_T->lock), + srcu_read_unlock_atomic(_T->lock, _T->idx), + int idx) +DECLARE_LOCK_GUARD_1_ATTRS(srcu_atomic, __acquires_shared(_T), __releases_= shared(*(struct srcu_struct **)_T)) +#define class_srcu_atomic_constructor(_T) WITH_LOCK_GUARD_1_ATTRS(srcu_ato= mic, _T) + DEFINE_LOCK_GUARD_1(srcu_fast, struct srcu_struct, _T->scp =3D srcu_read_lock_fast(_T->lock), srcu_read_unlock_fast(_T->lock, _T->scp), diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index cf4cb586f140..a3b4ca1ae88d 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -1160,6 +1160,8 @@ static struct kvm *kvm_create_vm(unsigned long type, = const char *fdname) goto out_err_no_srcu; if (init_srcu_struct(&kvm->irq_srcu)) goto out_err_no_irq_srcu; + if (init_srcu_struct_atomic(&kvm->gpc_srcu)) + goto out_err_no_gpc_srcu; =20 r =3D kvm_init_irq_routing(kvm); if (r) @@ -1240,6 +1242,8 @@ static struct kvm *kvm_create_vm(unsigned long type, = const char *fdname) kfree(kvm_get_bus_for_destruction(kvm, i)); kvm_free_irq_routing(kvm); out_err_no_irq_routing: + cleanup_srcu_struct(&kvm->gpc_srcu); +out_err_no_gpc_srcu: cleanup_srcu_struct(&kvm->irq_srcu); out_err_no_irq_srcu: cleanup_srcu_struct(&kvm->srcu); @@ -1316,6 +1320,11 @@ static void kvm_destroy_vm(struct kvm *kvm) kvm_free_memslots(kvm, &kvm->__memslots[i][0]); kvm_free_memslots(kvm, &kvm->__memslots[i][1]); } + /* + * All gfn_to_pfn_caches have been deactivated by kvm_arch_destroy_vm() + * above, so there can be no readers or pending invalidations left. + */ + cleanup_srcu_struct(&kvm->gpc_srcu); cleanup_srcu_struct(&kvm->irq_srcu); srcu_barrier(&kvm->srcu); cleanup_srcu_struct(&kvm->srcu); diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c index 17a30da00d40..1c98b7bdbf09 100644 --- a/virt/kvm/pfncache.c +++ b/virt/kvm/pfncache.c @@ -26,35 +26,149 @@ void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm= , unsigned long start, unsigned long end) { struct gfn_to_pfn_cache *gpc; + bool cleared =3D false; =20 spin_lock(&kvm->gpc_lock); + + /* + * Order the caller's earlier mn_active_invalidate_count increment + * (and gpc_invalidate_seq bump, for range_end) before the state + * reads below. This is a store->load ordering, which neither the + * unlock/lock the increment travelled through (RCtso) nor the + * acquire on the state read provides generically. It pairs with + * smp_mb__after_atomic() in hva_to_pfn_retry(): whichever side's + * full barrier executes second guarantees that either this walk + * observes GPC_BECOMING_VALID and vetoes the pending publish, or + * the refresh observes the elevated count / bumped seq and + * retries =E2=80=94 closing the store-buffering window in which each + * side misses the other and a stale pfn gets published. (On x86 + * both sides' locked RMW operations already provide this; generic + * code must not rely on that.) + */ + smp_mb__after_spinlock(); + list_for_each_entry(gpc, &kvm->gpc_list, list) { - read_lock_irq(&gpc->lock); + /* + * A cache covers only a single page, keyed by its uhva, so + * there is no length to consider. + * + * The acquire pairs with the release-publish in + * hva_to_pfn_retry() (for GPC_VALID) and with the + * smp_mb__before_atomic() before the GPC_BECOMING_VALID + * announce (for that bit): whichever is seen, the uhva read + * below is guaranteed to see the value stored before it. A + * stale uhva paired with a fresh state bit could otherwise + * cause a cache whose new uhva is in the invalidated range + * to be skipped. + * + * uhva is read once, with READ_ONCE(), and only after the + * state (the acquire provides that ordering): it is written + * (with WRITE_ONCE()) by __kvm_gpc_refresh() without holding + * gpc_lock. Note the refresh only mutates uhva while the + * state word is 0 =E2=80=94 after its xchg, before its announce =E2=80= =94 + * so this walk cannot observe a half-updated cache: if + * either state bit is visible, the uhva is current. + * + * Keying on GPC_BECOMING_VALID as well as GPC_VALID means a + * cache whose refresh is in flight is *not* skipped: its + * uhva is already known at that point, and clearing the bit + * vetoes the pending publish. Testing only GPC_VALID would + * leave the window between the refresh's last invalidation + * check and its publish unprotected. (For an invalidation + * which begins before that check, this veto is belt and + * braces: the mn_active_invalidate_count / gpc_invalidate_seq + * check makes the refresh retry anyway. The window after + * that check has only the veto.) + * + * Keying on GPC_INVALIDATING too means a cache whose + * VALID-clearing party is still mid-grace-period is not + * skipped either: see the comment at the 'cleared' update + * below. + */ + unsigned long gpc_uhva; + int old, new; =20 - /* Only a single page so no need to care about length */ - if (gpc->valid && !is_error_noslot_pfn(gpc->pfn) && - gpc->uhva >=3D start && gpc->uhva < end) { - read_unlock_irq(&gpc->lock); + old =3D atomic_read_acquire(&gpc->state); + if (!(old & (GPC_VALID | GPC_BECOMING_VALID | GPC_INVALIDATING))) + continue; =20 + gpc_uhva =3D READ_ONCE(gpc->uhva); + if (gpc_uhva >=3D start && gpc_uhva < end) { /* - * There is a small window here where the cache could - * be modified, and invalidation would no longer be - * necessary. Hence check again whether invalidation - * is still necessary once the write lock has been - * acquired. + * Clear VALID and BECOMING_VALID; if VALID was set, + * also set INVALIDATING so that anyone else who wants + * to mutate reader-visible fields before our grace + * period below completes knows to wait for one of + * their own. (A cache which was merely becoming + * valid has no readers: the veto alone suffices.) + * + * 'old' is seeded from the read above; if the state + * changed since (e.g. the refresh published), the + * cmpxchg fails, reloads it, and goes around. */ + do { + new =3D old & ~(GPC_VALID | GPC_BECOMING_VALID); + if (old & GPC_VALID) + new |=3D GPC_INVALIDATING; + } while (!atomic_try_cmpxchg(&gpc->state, &old, new)); =20 - write_lock_irq(&gpc->lock); - if (gpc->valid && !is_error_noslot_pfn(gpc->pfn) && - gpc->uhva >=3D start && gpc->uhva < end) - gpc->valid =3D false; - write_unlock_irq(&gpc->lock); - continue; + /* + * Wait below not only when this walk itself cleared + * GPC_VALID, but also when GPC_INVALIDATING alone + * was found: the party which cleared GPC_VALID (a + * concurrent walk, or a refresh/deactivation mid + * grace-period =E2=80=94 they keep the marker up until their + * wait completes) may still be inside its own + * synchronize, with readers outstanding on the old + * mapping. Those readers are covered by *its* grace + * period, which this walk's caller does not wait + * for; it must begin (and finish) one of its own + * before the caller frees the page. + */ + if (old & (GPC_VALID | GPC_INVALIDATING)) + cleared =3D true; } - - read_unlock_irq(&gpc->lock); } spin_unlock(&kvm->gpc_lock); + + /* + * Readers may still be using the old mapping, having sampled + * GPC_VALID before it was cleared. Wait for them all to drain + * before the caller proceeds to zap the page tables; like a TLB + * shootdown, this guarantees no access via the stale mapping + * once the invalidation completes. + * + * This wait is safe even on unblockable ranges (the OOM reaper): + * the constraint there is not "no sleeping" but "no blocking on + * anything which may itself depend on memory allocation to make + * progress" (see the reasoning in commit 312364f3534c and its + * discussion). A GPC read-side critical section has no such + * dependency: readers never allocate, never take mmap_lock, and + * never sleep. + * + * The atomic variant: this stalls progress of the primary MMU + * operation which triggered the invalidation, and the readers being + * drained are bounded, short, non-sleeping sections, so the wait + * spins (preemption disabled) rather than sleeping =E2=80=94 and never + * queues a callback, so it depends on no allocation at all. That + * satisfies even the strictest reading of the OOM-reaper + * requirement; it is also the only wait primitive permitted on a + * domain whose readers are the atomic flavor. + */ + if (cleared) + synchronize_srcu_atomic(&kvm->gpc_srcu); + + /* + * Note the GPC_INVALIDATING markers set above are deliberately NOT + * cleared here: this walk dropped gpc_lock before waiting, so it + * cannot tell whether the marker it would clear is still the one + * it set (consider a republish-and-reinvalidate cycle completing + * entirely within this synchronize_srcu()). Leaving it standing is + * merely conservative: a later walk over the same uhva pays a + * redundant grace period, and the next mutation of the cache (a + * refresh or deactivation, which performs its own wait) retires + * it. Readers ignore the bit. + */ } =20 static bool kvm_gpc_is_valid_len(gpa_t gpa, unsigned long uhva, @@ -74,14 +188,31 @@ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsig= ned long len) { struct kvm_memslots *slots =3D kvm_memslots(gpc->kvm); =20 - if (!gpc->active) + RCU_LOCKDEP_WARN(!srcu_read_lock_held(&gpc->kvm->gpc_srcu), + "kvm_gpc_check() without kvm->gpc_srcu read lock"); + + /* + * Check valid *first*. The acquire pairs with the release-publish + * in hva_to_pfn_retry(), so every field read below =E2=80=94 and any use + * of gpc->khva by the caller =E2=80=94 is guaranteed to be from the + * published generation, not a stale value reordered from before + * the publish. The fields are then stable for the remainder of + * the RCU read-side critical section, because every mutator + * clears valid and waits a full grace period before changing + * anything. + */ + if (!(atomic_read_acquire(&gpc->state) & GPC_VALID)) + return false; + + if (!READ_ONCE(gpc->active)) return false; =20 /* * If the page was cached from a memslot, make sure the memslots have * not been re-configured. */ - if (!kvm_is_error_gpa(gpc->gpa) && gpc->generation !=3D slots->generation) + if (!kvm_is_error_gpa(gpc->gpa) && + READ_ONCE(gpc->generation) !=3D slots->generation) return false; =20 if (kvm_is_error_hva(gpc->uhva)) @@ -90,9 +221,6 @@ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsigne= d long len) if (!kvm_gpc_is_valid_len(gpc->gpa, gpc->uhva, len)) return false; =20 - if (!gpc->valid) - return false; - return true; } =20 @@ -134,8 +262,8 @@ static inline bool mmu_notifier_retry_cache(struct kvm = *kvm, unsigned long gpc_s * is elevated. * * Note, it does not matter that mn_active_invalidate_count - * is not protected by gpc->lock. It is guaranteed to - * be elevated before the mmu_notifier acquires gpc->lock, and + * is not protected by any lock the refresher holds. It is + * guaranteed to be elevated before the mmu_notifier walk, and * isn't dropped until after gpc_invalidate_seq is updated. */ if (kvm->mn_active_invalidate_count) @@ -156,10 +284,13 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_c= ache *gpc) { /* Note, the new page offset may be different than the old! */ void *old_khva =3D (void *)PAGE_ALIGN_DOWN((uintptr_t)gpc->khva); + void *old_khva_exact =3D gpc->khva; + kvm_pfn_t old_pfn =3D gpc->pfn; kvm_pfn_t new_pfn =3D KVM_PFN_ERR_FAULT; void *new_khva =3D NULL; unsigned long gpc_seq; struct page *page; + int state; =20 struct kvm_follow_pfn kfp =3D { .slot =3D gpc->memslot, @@ -171,21 +302,42 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_c= ache *gpc) =20 lockdep_assert_held(&gpc->refresh_lock); =20 - lockdep_assert_held_write(&gpc->lock); - - /* - * Invalidate the cache prior to dropping gpc->lock, the gpa=3D>uhva - * assets have already been updated and so a concurrent check() from a - * different task may not fail the gpa/uhva/generation checks. - */ - gpc->valid =3D false; + WARN_ON_ONCE(atomic_read(&gpc->state) & GPC_VALID); =20 do { + /* + * Announce the intent to publish before the uhva is resolved + * and before any lock is dropped. From here on an + * invalidation of this uhva will clear the bit, and the + * cmpxchg below will then fail and send us around again. + * This is what makes the check-and-publish atomic with + * respect to the mmu_notifier walk without holding a lock + * across it. + * + * The barrier orders the caller's gpc->uhva store before the + * bit becoming visible (atomic_or() alone is unordered, and + * x86's lock-or being a full barrier must not be relied on + * in generic code). It pairs with the atomic_read_acquire() + * in gfn_to_pfn_cache_invalidate_start(): a walker which + * observes the bit observes the uhva this refresh is + * resolving, and so aims its veto correctly. + */ + smp_mb__before_atomic(); + atomic_or(GPC_BECOMING_VALID, &gpc->state); + + /* + * Order the announce store before the gpc_invalidate_seq + * and mn_active_invalidate_count reads which follow + * (store->load; atomic_or() alone is unordered). Pairs + * with smp_mb__after_spinlock() in + * gfn_to_pfn_cache_invalidate_start(); see the comment + * there for the store-buffering window this closes. + */ + smp_mb__after_atomic(); + gpc_seq =3D gpc->kvm->gpc_invalidate_seq; smp_rmb(); =20 - write_unlock_irq(&gpc->lock); - /* * If the previous iteration "failed" due to an mmu_notifier * event, release the pfn and unmap the kernel virtual address @@ -213,9 +365,15 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_ca= che *gpc) /* * Obtain a new kernel mapping if KVM itself will access the * pfn. Note, kmap() and memremap() can both sleep, so this - * too must be done outside of gpc->lock! + * can sleep, which is fine: this path holds no spinning locks. */ - if (new_pfn =3D=3D gpc->pfn) + /* + * Compare against the *published* pfn captured at entry, via + * a local: gpc->pfn must not be consulted mid-loop. (A + * previous iteration's vetoed publish attempt must not be + * able to leak its doomed pfn into this comparison.) + */ + if (new_pfn =3D=3D old_pfn) new_khva =3D old_khva; else new_khva =3D gpc_map(new_pfn); @@ -225,18 +383,39 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_c= ache *gpc) goto out_error; } =20 - write_lock_irq(&gpc->lock); + if (mmu_notifier_retry_cache(gpc->kvm, gpc_seq)) + continue; + + gpc->pfn =3D new_pfn; + gpc->khva =3D new_khva + offset_in_page(gpc->uhva); =20 /* - * Other tasks must wait for _this_ refresh to complete before - * attempting to refresh. + * Publish: BECOMING_VALID =3D> VALID, but only if the bit is + * still set. If an invalidation of this uhva landed at any + * point since it was set =E2=80=94 including in the window after the + * retry check just above =E2=80=94 the bit is gone, the cmpxchg fails + * and we go around again. The release ordering pairs with the + * atomic_read_acquire() in kvm_gpc_check() so the pfn/khva + * stores are visible before GPC_VALID is. (The stores must + * precede the VALID publish; a reader which observes VALID + * must observe the fields it covers.) */ - WARN_ON_ONCE(gpc->valid); - } while (mmu_notifier_retry_cache(gpc->kvm, gpc_seq)); + state =3D GPC_BECOMING_VALID; + if (atomic_try_cmpxchg_release(&gpc->state, &state, GPC_VALID)) + break; =20 - gpc->valid =3D true; - gpc->pfn =3D new_pfn; - gpc->khva =3D new_khva + offset_in_page(gpc->uhva); + /* + * Vetoed: an invalidation cleared GPC_BECOMING_VALID after + * the retry check above. No reader ever saw the values just + * stored (VALID was never set for them), but the loop's own + * cleanup and aliasing logic must not see them either =E2=80=94 + * restore the published values. This runs under refresh_lock + * with the cache unpublished, so it is single-writer plain + * stores. + */ + gpc->pfn =3D old_pfn; + gpc->khva =3D old_khva_exact; + } while (1); =20 /* * Put the reference to the _new_ page. The page is now tracked by the @@ -248,8 +427,19 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_ca= che *gpc) return 0; =20 out_error: - write_lock_irq(&gpc->lock); - + /* + * Withdraw the announcement: no publish is coming. Leaving the bit + * set would be harmless =E2=80=94 an invalidation walk would merely veto= a + * publish which was never going to happen, and the next refresh or + * deactivation starts by xchg'ing the whole state word to zero =E2=80=94 + * but clearing it here keeps the invariant that GPC_BECOMING_VALID + * means a refresh is between announce and publish. Only VALID can + * have been set here by anyone else, and only by our own publish + * (which didn't happen), and INVALIDATING only by whoever clears + * VALID; the sole concurrent writer is an invalidation walk also + * clearing this bit, so a plain andnot cannot lose an update. + */ + atomic_andnot(GPC_BECOMING_VALID, &gpc->state); return -EFAULT; } =20 @@ -259,7 +449,7 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *g= pc, gpa_t gpa, unsigned l bool unmap_old =3D false; unsigned long old_uhva; kvm_pfn_t old_pfn; - bool hva_change =3D false; + bool must_drain; void *old_khva; int ret; =20 @@ -269,13 +459,103 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache= *gpc, gpa_t gpa, unsigned l =20 lockdep_assert_held(&gpc->refresh_lock); =20 - write_lock_irq(&gpc->lock); + if (!gpc->active) + return -EINVAL; =20 - if (!gpc->active) { - ret =3D -EINVAL; - goto out_unlock; + /* + * Resolve the target uHVA (and memslot, for a GPA-based cache) before + * deciding whether anything needs to be invalidated at all. + * + * If the cache is still valid and this refresh resolves to exactly the + * same uHVA, in the same memslot, for the same GPA, then nothing which + * a reader can observe is changing: the gPA =3D> uHVA translation is + * unchanged, and GPC_VALID being set is precisely the assertion that + * the second stage (uHVA =3D> PFN, and the kernel mapping of it) is + * still good. No mapping is retired, so there is nothing for a grace + * period to wait for. At most the memslot generation needs updating, + * which no reader consults except via kvm_gpc_check() itself. + * + * All of the following must hold to take this shortcut: + * + * - GPC_VALID: the PFN and its kernel mapping are still good. + * - the resolved uHVA is unchanged. kvm_gpc_check() validates + * 'offset + len <=3D PAGE_SIZE' from gpc->uhva/gpa on the reader's + * behalf, and the reader then accesses gpc->khva for 'len' bytes + * with no recheck; moving the offset under a live reader would let + * it run off the end of the page. + * - the memslot is unchanged. kvm_gpc_mark_dirty_in_slot() uses + * gpc->memslot, so replacing it under a reader could mark the wrong + * slot dirty and thus lose a dirty page for live migration. + * - the gPA is unchanged, as it provides the gfn for dirty tracking. + */ + if (atomic_read(&gpc->state) & GPC_VALID) { + struct kvm_memory_slot *new_slot =3D gpc->memslot; + unsigned long new_uhva =3D KVM_HVA_ERR_BAD; + u64 new_generation =3D gpc->generation; + + if (kvm_is_error_gpa(gpa)) { + new_uhva =3D uhva; + } else { + struct kvm_memslots *slots =3D kvm_memslots(gpc->kvm); + gfn_t gfn =3D gpa_to_gfn(gpa); + + new_generation =3D slots->generation; + new_slot =3D __gfn_to_memslot(slots, gfn); + if (new_slot) + new_uhva =3D gfn_to_hva_memslot(new_slot, gfn) + + offset_in_page(gpa); + } + + if (!kvm_is_error_hva(new_uhva) && new_uhva =3D=3D gpc->uhva && + new_slot =3D=3D gpc->memslot && gpa =3D=3D gpc->gpa) { + /* + * Nothing to invalidate. A concurrent reader may be + * using the cache right now and can safely continue + * to do so; only the memslot generation, which is + * read solely by kvm_gpc_check(), may need updating. + */ + if (new_generation !=3D gpc->generation) + WRITE_ONCE(gpc->generation, new_generation); + + return 0; + } } =20 + /* + * Take the cache invalid and wait for all current readers to + * drain before mutating anything they might be looking at. Once + * the grace period has elapsed, this task (serialized by + * refresh_lock) owns all the cache fields exclusively: readers + * check valid (with an acquire load) inside their SRCU read-side + * critical sections and back off. This mirrors what a TLB + * shootdown does for the hardware page tables. + * + * The clear and the test are one operation on purpose, and the wait + * triggers on GPC_INVALIDATING as well as GPC_VALID: an mmu_notifier + * invalidation which cleared GPC_VALID a moment ago is not + * necessarily finished =E2=80=94 its grace period may still be running, = with + * readers which sampled GPC_VALID before the clear still + * dereferencing gpc->khva. GPC_INVALIDATING is set by whoever clears + * GPC_VALID precisely so that this path knows those readers may + * exist. Our own grace-period wait below necessarily begins after + * that clear, so waiting here covers them; only then may + * reader-visible fields (uhva, gpa, memslot, and later pfn/khva) be + * mutated or the old mapping torn down. + * + * The xchg leaves GPC_INVALIDATING *standing* (rather than clearing + * the word outright) for the duration of the wait, for the same + * reason it is tested: an mmu_notifier walk which arrives while + * this grace period is still running must not read a zero state, + * skip the cache, and let its caller free the page under readers + * only this wait covers. The marker comes down only once the drain + * is complete and the readers it advertised cannot exist. + */ + must_drain =3D atomic_xchg(&gpc->state, GPC_INVALIDATING) & + (GPC_VALID | GPC_INVALIDATING); + if (must_drain) + synchronize_srcu_atomic(&gpc->kvm->gpc_srcu); + atomic_set(&gpc->state, 0); + old_pfn =3D gpc->pfn; old_khva =3D (void *)PAGE_ALIGN_DOWN((uintptr_t)gpc->khva); old_uhva =3D PAGE_ALIGN_DOWN(gpc->uhva); @@ -285,10 +565,14 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache = *gpc, gpa_t gpa, unsigned l =20 gpc->gpa =3D INVALID_GPA; gpc->memslot =3D NULL; - gpc->uhva =3D PAGE_ALIGN_DOWN(uhva); - - if (gpc->uhva !=3D old_uhva) - hva_change =3D true; + /* + * uhva stores are WRITE_ONCE(): the invalidation walks read + * it without holding refresh_lock. This mutation window is + * safe because the state word is 0 here (cleared by the + * xchg above, announced again only in hva_to_pfn_retry() + * after these stores), so no walk will match this cache. + */ + WRITE_ONCE(gpc->uhva, PAGE_ALIGN_DOWN(uhva)); } else { struct kvm_memslots *slots =3D kvm_memslots(gpc->kvm); =20 @@ -301,43 +585,31 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache = *gpc, gpa_t gpa, unsigned l gpc->gpa =3D gpa; gpc->generation =3D slots->generation; gpc->memslot =3D __gfn_to_memslot(slots, gfn); - gpc->uhva =3D gfn_to_hva_memslot(gpc->memslot, gfn); + WRITE_ONCE(gpc->uhva, gfn_to_hva_memslot(gpc->memslot, gfn)); =20 if (kvm_is_error_hva(gpc->uhva)) { ret =3D -EFAULT; goto out; } - - /* - * Even if the GPA and/or the memslot generation changed, the - * HVA may still be the same. - */ - if (gpc->uhva !=3D old_uhva) - hva_change =3D true; } else { - gpc->uhva =3D old_uhva; + WRITE_ONCE(gpc->uhva, old_uhva); } } =20 /* Note: the offset must be correct before calling hva_to_pfn_retry() */ - gpc->uhva +=3D page_offset; + WRITE_ONCE(gpc->uhva, gpc->uhva + page_offset); =20 /* - * If the userspace HVA changed or the PFN was already invalid, - * drop the lock and do the HVA to PFN lookup again. + * Always redo the HVA to PFN lookup: an invalidation of our uhva + * may have raced with (or followed) the valid-clearing above, in + * which case the notifier walk skipped this cache (valid was + * already false) and the old pfn may already be stale. The + * gpc_invalidate_seq check in hva_to_pfn_retry() is what detects + * that race, so the lookup path is the only safe way to publish. + * If the mapping is in fact unchanged, hva_to_pfn_retry() reuses + * the existing kernel mapping for the same pfn. */ - if (!gpc->valid || hva_change) { - ret =3D hva_to_pfn_retry(gpc); - } else { - /* - * If the HVA=E2=86=92PFN mapping was already valid, don't unmap it. - * But do update gpc->khva because the offset within the page - * may have changed. - */ - gpc->khva =3D old_khva + page_offset; - ret =3D 0; - goto out_unlock; - } + ret =3D hva_to_pfn_retry(gpc); =20 out: /* @@ -346,17 +618,12 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache = *gpc, gpa_t gpa, unsigned l * valid, leave it as is. */ if (ret) { - gpc->valid =3D false; + WARN_ON_ONCE(atomic_read(&gpc->state) & GPC_VALID); gpc->pfn =3D KVM_PFN_ERR_FAULT; gpc->khva =3D NULL; } =20 - /* Detect a pfn change before dropping the lock! */ unmap_old =3D (old_pfn !=3D gpc->pfn); - -out_unlock: - write_unlock_irq(&gpc->lock); - if (unmap_old) gpc_unmap(old_pfn, old_khva); =20 @@ -385,14 +652,14 @@ int kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, uns= igned long len) void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm, bool never_dirty) { - rwlock_init(&gpc->lock); mutex_init(&gpc->refresh_lock); =20 gpc->kvm =3D kvm; gpc->pfn =3D KVM_PFN_ERR_FAULT; gpc->gpa =3D INVALID_GPA; gpc->uhva =3D KVM_HVA_ERR_BAD; - gpc->active =3D gpc->valid =3D false; + gpc->active =3D false; + atomic_set(&gpc->state, 0); gpc->never_dirty =3D never_dirty; } =20 @@ -407,7 +674,7 @@ static int __kvm_gpc_activate(struct gfn_to_pfn_cache *= gpc, gpa_t gpa, unsigned guard(mutex)(&gpc->refresh_lock); =20 if (!gpc->active) { - if (KVM_BUG_ON(gpc->valid, kvm)) + if (KVM_BUG_ON(atomic_read(&gpc->state) & GPC_VALID, kvm)) return -EIO; =20 spin_lock(&kvm->gpc_lock); @@ -417,11 +684,11 @@ static int __kvm_gpc_activate(struct gfn_to_pfn_cache= *gpc, gpa_t gpa, unsigned /* * Activate the cache after adding it to the list, a concurrent * refresh must not establish a mapping until the cache is - * reachable by mmu_notifier events. + * reachable by mmu_notifier events. (Refreshes are serialized + * by refresh_lock, which we hold; the store ordering matters + * only against the notifier walk, which holds gpc_lock.) */ - write_lock_irq(&gpc->lock); - gpc->active =3D true; - write_unlock_irq(&gpc->lock); + WRITE_ONCE(gpc->active, true); } return __kvm_gpc_refresh(gpc, gpa, uhva); } @@ -456,13 +723,28 @@ void kvm_gpc_deactivate(struct gfn_to_pfn_cache *gpc) =20 if (gpc->active) { /* - * Deactivate the cache before removing it from the list, KVM - * must stall mmu_notifier events until all users go away, i.e. - * until gpc->lock is dropped and refresh is guaranteed to fail. + * Mark the cache inactive and invalid, and wait for all + * current readers to drain, before tearing down the mapping + * they may have been using. Refreshes are excluded by + * refresh_lock, which we hold. + */ + WRITE_ONCE(gpc->active, false); + + /* + * As in __kvm_gpc_refresh(): clear and test in one + * operation, waiting on GPC_INVALIDATING as well as + * GPC_VALID. An invalidation which cleared GPC_VALID first + * may still be inside its own synchronize_srcu() with + * readers outstanding =E2=80=94 it left GPC_INVALIDATING set to say + * so =E2=80=94 and the unmap below must not run under those readers. + * And, also as there, leave GPC_INVALIDATING standing until + * the drain completes, so that an mmu_notifier walk arriving + * mid-wait does not read a zero state and skip the cache. */ - write_lock_irq(&gpc->lock); - gpc->active =3D false; - gpc->valid =3D false; + if (atomic_xchg(&gpc->state, GPC_INVALIDATING) & + (GPC_VALID | GPC_INVALIDATING)) + synchronize_srcu_atomic(&kvm->gpc_srcu); + atomic_set(&gpc->state, 0); =20 /* * Leave the GPA =3D> uHVA cache intact, it's protected by the @@ -475,7 +757,6 @@ void kvm_gpc_deactivate(struct gfn_to_pfn_cache *gpc) =20 old_pfn =3D gpc->pfn; gpc->pfn =3D KVM_PFN_ERR_FAULT; - write_unlock_irq(&gpc->lock); =20 spin_lock(&kvm->gpc_lock); list_del(&gpc->list); --=20 2.43.0 From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.50.34 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844071; b=O3tLJPt4lAg3qbJDwctoQ3Py9Su/r/E+lwsUEVJ75T43YeiANWfGr9YtPo983id603rd 8/LGsyppiQy42lbj6z0+g0m8PtU2o+Q0OWt59MMc4xYfy16hvvohmbjWW39joruswlb72 Cxd7oQSW58D+BXJqNdTuOg1Z6gCd+D3cLzv9BwB6QvHRt1jBFRe1aQGdl/ToDSduQGg7k pgZeIv0bKBw5zfIMcrPneRbKN8bR/6qMDJ1NslVtsL9tE5vyvOGTqPwDI3hyoyP/vCsA4 ocVksfhlJc/NPDOkBojusIIpkQa1MDa67DUH1vyi5g5Tk80PYW73lKaCYbhMM9Qdatw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844071; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Type:Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=8OScpE0FBH4jzhyM8NQfWUyjdAAqoPLioveXmLb2kcA=; b=t2OCp+pzaAfXLmX3Jdpbsi4KnDvYIXREho3Y2fAnAi54XOeEuU6tH0vlJnGg4z1RqN3X Pv6Mxh4wHWiXbMATWoZaoG5myIAxuALPWCJpsRQFp9+n819RImQHCO1Yp9HldcnlnuXVA oPbX7nni+zAHtiInAmAHqohKq2ebZyakgSg9EOIYV+qq18o5jfDKRppz+o1vmperjWcoG QOCvf0eGz0lTj2ya+W8hEh6QWUvgTdmt+KHgrWXXJZiBgp3trwzJYSFGLsGyERm6jk4xj p8auZNSRLaQgAKkMCPrwHENmuoepE1L1/nsin1in1nYcalQtig4k/XYMP0LiieVhIIQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=casper.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=pdGLH5Os; arc=none smtp.remote-ip=90.155.50.34 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAAAA606A334; Thu, 5 Nov 2026 02:01:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org DAAAA606A334 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=casper.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=pdGLH5Os DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description; bh=8OScpE0FBH4jzhyM8NQfWUyjdAAqoPLioveXmLb2kcA=; b=pdGLH5OsdZr8EtyKIJmt5IQH7C HMAP0xrE9eMwFFAslFyxq52n7DdB0E65EfRvLuAPgDE7AZ8V99ljYb9ygdzUze1U92MQb4DIn00Fv g57J0EjspdfGGvZ60oGWTIqEdK3IyrKcZYyW2JxPfpSypMIFtEScrsZSZR8TP+x0m4WmrcCR28c4H YUkVVN8HM2jBMYF88K73U/GDjdav1dpqrYqwDWnmf7vR1bKcvq0ZYNa80JBshkz1OCf3UTMLgMtOR kEawzuAu3mHxoEKInA6AfDPTLvlJkAtx8R+afzLFvkmHO4PZxJ+fKY3cIBhEngveKkcfDNHUY2KIs B2jIzEeg==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by casper.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmn-000000037jB-0EFY; Thu, 05 Nov 2026 02:01:05 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmm-0000000AWDS-1oI4; Thu, 05 Nov 2026 02:01:04 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 09/20] KVM: x86: Use gfn_to_pfn_cache for steal time / preempted status Date: Thu, 5 Nov 2026 02:00:51 +0000 Message-ID: <20261105020104.2507083-10-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html From: Carsten Stollmaier This largely reverts commit 7e2175ebd695 ("KVM: x86: Fix recording of guest steal time / preempted status"), which dropped the use of the gfn_to_pfn_cache because it was not integrated with the MMU notifiers at the time. That shortcoming has long since been addressed, making the GPC work correctly for this use case. Aside from cleaning up the last open-coded assembler access to user addresses and associated explicit asm exception fixups, moving back to the now-functional GPC also resolves an issue with contention on the mmap_lock with userfaultfd. The contention issue is as follows: On vcpu_run, before entering the guest, the update of the steal time information causes a page-fault if the page is not present. In our scenario, this gets handled by do_user_addr_fault() and successively handle_userfault() because the region is registered to that. Since handle_userfault() uses TASK_INTERRUPTIBLE, it is interruptible by signals. But do_user_addr_fault() then busy-retries if the pending signal is non-fatal, which leads to heavy contention of the mmap_lock. By restoring the use of GPC for accessing the guest steal time, the contention is avoided and refreshing the GPC happens when the vCPU is next scheduled. Since the gfn_to_pfn_cache gives a kernel mapping rather than a userspace HVA, accesses are now plain C instead of unsafe_put_user() et al. READ_ONCE()/WRITE_ONCE() and an smp_wmb() before the final version increment replace the ordering the old inline assembly provided implicitly. kvm_steal_time_set_preempted(), called from the scheduler path, just takes the atomic SRCU read lock (unconditionally safe there) and bails if the cache is not currently valid: setting the preempted flag is best-effort anyway =E2=80=94 the old code used copy_to_user_nofault(), which could equally silently fail =E2=80=94 and kvm_arch_vcpu_load() raises KVM_REQ_STEAL_UPDATE, so the cache is refreshed from task context before the vCPU next enters the guest. MSR disable (kvm_steal_time_reset()) leaves last_steal alone: it is the baseline against the vCPU task's monotonic run_delay, and zeroing it would dump the task's lifetime run_delay into the guest's steal counter when the MSR is next enabled (e.g. after INIT-SIPI or kexec). [dwmw2: Port to the SRCU-protected GPC. The original was written against a gpc->lock rwlock and needed a read_trylock() dance to be safe from the scheduler path on PREEMPT_RT; the atomic SRCU read lock is unconditionally safe there, so all of that goes away.] Signed-off-by: Carsten Stollmaier Signed-off-by: Sean Christopherson Co-developed-by: David Woodhouse Signed-off-by: David Woodhouse --- arch/x86/include/asm/kvm_host.h | 2 +- arch/x86/kvm/msrs.c | 7 +- arch/x86/kvm/x86.c | 140 ++++++++++++++++---------------- 3 files changed, 76 insertions(+), 73 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 1ee4b127004d..4b0c18c77771 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -872,7 +872,7 @@ struct kvm_vcpu_arch { u8 preempted; u64 msr_val; u64 last_steal; - struct gfn_to_hva_cache cache; + struct gfn_to_pfn_cache cache; } st; =20 u64 l1_tsc_offset; diff --git a/arch/x86/kvm/msrs.c b/arch/x86/kvm/msrs.c index e91650af44c2..23407f8c441d 100644 --- a/arch/x86/kvm/msrs.c +++ b/arch/x86/kvm/msrs.c @@ -1728,8 +1728,13 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct= msr_data *msr_info) =20 vcpu->arch.st.msr_val =3D data; =20 - if (!(data & KVM_MSR_ENABLED)) + if (!(data & KVM_MSR_ENABLED)) { + kvm_gpc_deactivate(&vcpu->arch.st.cache); break; + } + + kvm_gpc_activate(&vcpu->arch.st.cache, data & ~KVM_MSR_ENABLED, + sizeof(struct kvm_steal_time)); =20 kvm_make_request(KVM_REQ_STEAL_UPDATE, vcpu); =20 diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index f5f3459ed4a5..321f29789f9f 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -2058,12 +2058,11 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_service_local_tl= b_flush_requests); =20 static void record_steal_time(struct kvm_vcpu *vcpu) { - struct gfn_to_hva_cache *ghc =3D &vcpu->arch.st.cache; - struct kvm_steal_time __user *st; - struct kvm_memslots *slots; - gpa_t gpa =3D vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS; + struct gfn_to_pfn_cache *gpc =3D &vcpu->arch.st.cache; + struct kvm_steal_time *st; u64 steal; u32 version; + int idx; =20 if (kvm_xen_msr_enabled(vcpu->kvm)) { kvm_xen_runstate_set_running(vcpu); @@ -2076,55 +2075,41 @@ static void record_steal_time(struct kvm_vcpu *vcpu) if (WARN_ON_ONCE(current->mm !=3D vcpu->kvm->mm)) return; =20 - slots =3D kvm_memslots(vcpu->kvm); + /* We rely on the fact that it fits in a single page. */ + BUILD_BUG_ON((sizeof(*st) - 1) & KVM_STEAL_VALID_BITS); =20 - if (unlikely(slots->generation !=3D ghc->generation || - gpa !=3D ghc->gpa || - kvm_is_error_hva(ghc->hva) || !ghc->memslot)) { - /* We rely on the fact that it fits in a single page. */ - BUILD_BUG_ON((sizeof(*st) - 1) & KVM_STEAL_VALID_BITS); + idx =3D srcu_read_lock_atomic(&vcpu->kvm->gpc_srcu); + while (!kvm_gpc_check(gpc, sizeof(*st))) { + srcu_read_unlock_atomic(&vcpu->kvm->gpc_srcu, idx); =20 - if (kvm_gfn_to_hva_cache_init(vcpu->kvm, ghc, gpa, sizeof(*st)) || - kvm_is_error_hva(ghc->hva) || !ghc->memslot) + if (kvm_gpc_refresh(gpc, sizeof(*st))) return; + + idx =3D srcu_read_lock_atomic(&vcpu->kvm->gpc_srcu); } =20 - st =3D (struct kvm_steal_time __user *)ghc->hva; + st =3D gpc->khva; + /* * Doing a TLB flush here, on the guest's behalf, can avoid * expensive IPIs. */ if (guest_pv_has(vcpu, KVM_FEATURE_PV_TLB_FLUSH)) { - u8 st_preempted =3D 0; - int err =3D -EFAULT; - - if (!user_access_begin(st, sizeof(*st))) - return; - - asm volatile("1: xchgb %0, %2\n" - "xor %1, %1\n" - "2:\n" - _ASM_EXTABLE_UA(1b, 2b) - : "+q" (st_preempted), - "+&r" (err), - "+m" (st->preempted)); - if (err) - goto out; - - user_access_end(); + u8 st_preempted =3D xchg(&st->preempted, 0); =20 vcpu->arch.st.preempted =3D 0; =20 trace_kvm_pv_tlb_flush(vcpu->vcpu_id, st_preempted & KVM_VCPU_FLUSH_TLB); /* - * Request the flush instead of performing it inline. With - * shadow paging kvm_vcpu_flush_tlb_guest() can reschedule - * (via mmu_sync_children()), so it could not be called while - * the user access above was open. Requesting is just a bit - * set on the running vCPU, and the request is serviced - * before the vCPU enters the guest; nothing requires the - * flush to have completed any earlier than that. + * Request the flush instead of performing it inline. This + * runs inside an atomic gpc_srcu read-side section, with + * preemption disabled, and with shadow paging + * kvm_vcpu_flush_tlb_guest() can reschedule (via + * mmu_sync_children()). Requesting is just a bit set on the + * running vCPU, and the request is serviced before the vCPU + * enters the guest; nothing requires the flush to have + * completed any earlier than that. * * Note the xchg above must remain as-is: it atomically * consumes any KVM_VCPU_FLUSH_TLB the guest set, as required @@ -2133,39 +2118,33 @@ static void record_steal_time(struct kvm_vcpu *vcpu) */ if (st_preempted & KVM_VCPU_FLUSH_TLB) kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); - - if (!user_access_begin(st, sizeof(*st))) - goto dirty; } else { - if (!user_access_begin(st, sizeof(*st))) - return; - - unsafe_put_user(0, &st->preempted, out); + WRITE_ONCE(st->preempted, 0); vcpu->arch.st.preempted =3D 0; } =20 - unsafe_get_user(version, &st->version, out); + version =3D READ_ONCE(st->version); if (version & 1) version +=3D 1; /* first time write, random junk */ =20 version +=3D 1; - unsafe_put_user(version, &st->version, out); + WRITE_ONCE(st->version, version); =20 smp_wmb(); =20 - unsafe_get_user(steal, &st->steal, out); + steal =3D READ_ONCE(st->steal); steal +=3D current->sched_info.run_delay - vcpu->arch.st.last_steal; vcpu->arch.st.last_steal =3D current->sched_info.run_delay; - unsafe_put_user(steal, &st->steal, out); + WRITE_ONCE(st->steal, steal); + + smp_wmb(); =20 version +=3D 1; - unsafe_put_user(version, &st->version, out); + WRITE_ONCE(st->version, version); =20 - out: - user_access_end(); - dirty: - mark_page_dirty_in_slot(vcpu->kvm, ghc->memslot, gpa_to_gfn(ghc->gpa)); + kvm_gpc_mark_dirty_in_slot(gpc); + srcu_read_unlock_atomic(&vcpu->kvm->gpc_srcu, idx); } =20 static inline bool kvm_can_mwait_in_guest(void) @@ -2640,11 +2619,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int c= pu) =20 static void kvm_steal_time_set_preempted(struct kvm_vcpu *vcpu) { - struct gfn_to_hva_cache *ghc =3D &vcpu->arch.st.cache; - struct kvm_steal_time __user *st; - struct kvm_memslots *slots; - static const u8 preempted =3D KVM_VCPU_PREEMPTED; - gpa_t gpa =3D vcpu->arch.st.msr_val & KVM_STEAL_VALID_BITS; + struct gfn_to_pfn_cache *gpc =3D &vcpu->arch.st.cache; + struct kvm_steal_time *st; =20 /* * The vCPU can be marked preempted if and only if the VM-Exit was on @@ -2669,20 +2645,37 @@ static void kvm_steal_time_set_preempted(struct kvm= _vcpu *vcpu) if (unlikely(current->mm !=3D vcpu->kvm->mm)) return; =20 - slots =3D kvm_memslots(vcpu->kvm); - - if (unlikely(slots->generation !=3D ghc->generation || - gpa !=3D ghc->gpa || - kvm_is_error_hva(ghc->hva) || !ghc->memslot)) - return; - - st =3D (struct kvm_steal_time __user *)ghc->hva; - BUILD_BUG_ON(sizeof(st->preempted) !=3D sizeof(preempted)); - - if (!copy_to_user_nofault(&st->preempted, &preempted, sizeof(preempted))) + /* + * This is called from the scheduler path (via kvm_sched_out), so + * the cache cannot be refreshed here. Setting 'preempted' is + * best-effort anyway; the old HVA-based code used + * copy_to_user_nofault(), which could also silently fail. If the + * cache is invalid, it will be refreshed from task context by + * record_steal_time() before the vCPU next enters the guest, since + * kvm_arch_vcpu_load() unconditionally raises KVM_REQ_STEAL_UPDATE. + */ + guard(srcu_atomic)(&vcpu->kvm->gpc_srcu); + if (kvm_gpc_check(gpc, sizeof(*st))) { + st =3D gpc->khva; + WRITE_ONCE(st->preempted, KVM_VCPU_PREEMPTED); vcpu->arch.st.preempted =3D KVM_VCPU_PREEMPTED; =20 - mark_page_dirty_in_slot(vcpu->kvm, ghc->memslot, gpa_to_gfn(ghc->gpa)); + kvm_gpc_mark_dirty_in_slot(gpc); + } +} + +static void kvm_steal_time_reset(struct kvm_vcpu *vcpu) +{ + kvm_gpc_deactivate(&vcpu->arch.st.cache); + vcpu->arch.st.preempted =3D 0; + vcpu->arch.st.msr_val =3D 0; + /* + * Leave last_steal alone: it is the baseline against the vCPU + * task's cumulative sched_info.run_delay, which is never reset. + * Zeroing it here would cause the first record_steal_time() after + * a vCPU reset (INIT-SIPI, kexec) to account the task's entire + * lifetime run_delay as steal time in one jump. + */ } =20 void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) @@ -9460,6 +9453,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) =20 kvm_gpc_init(&vcpu->arch.pv_time, vcpu->kvm); =20 + kvm_gpc_init(&vcpu->arch.st.cache, vcpu->kvm); + if (!irqchip_in_kernel(vcpu->kvm) || kvm_vcpu_is_reset_bsp(vcpu)) kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE); else @@ -9569,6 +9564,8 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu) kvm_clear_async_pf_completion_queue(vcpu); kvm_mmu_unload(vcpu); =20 + kvm_steal_time_reset(vcpu); + kvmclock_reset(vcpu); =20 for_each_possible_cpu(cpu) @@ -9689,7 +9686,8 @@ void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_= event) kvm_make_request(KVM_REQ_EVENT, vcpu); vcpu->arch.apf.msr_en_val =3D 0; vcpu->arch.apf.msr_int_val =3D 0; - vcpu->arch.st.msr_val =3D 0; + + kvm_steal_time_reset(vcpu); =20 kvmclock_reset(vcpu); =20 --=20 2.43.0 From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.50.34 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844071; b=MLcAUoq4evCwdaQTobRXy+I3v5HJSqAE1TJ7+No0SGn60+MLd3x5/NV+ZjojRwk5hfVt b3azyYugzCvdmOPkt5Kk4BD5vz/gAWIfZfHouQUoY6zR/1jY+dInQPXZC7dR+nfe/K7yy 95UWSe4iXPUhz/C2j0GMbnUgXi86OYPe+YUGFgW7qWw0AC/jVbJf4HeGdhUUc9wuHVvPf 4spQ3eI4a2XGbdwrX7uWy1NgwLZ4FZ6kmIyBjY8gFonuB9/we0O88hrPFNTCWwXqsbOsE 0EfPgBwoa939Ji4Lrg0dyalHU6N5Y7PsaZZH4xQ0pTG+oodCiN7D2ziPPgnpo0TbifA== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844071; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Type:Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=cS36VXsZObmcF5KZcGY9YdwDsYCRTEkKDtw4SlNBJeo=; b=UCNgVze4k22cl0iQWTi/CMDXVDmWBkw7W/NK/7t01aY5op7YVvfLjr5bVj8ZYmstjc1B bGcnXvZP9U7uy3uOwIxi/QKV+2FL7nGgqR0pihSyDN/RJLRlbGx6SPUkY0IUhc/P3jwBn XHxAss0KT0GhUxZidgPtCFuWvBo1h9RF3u46XyPO3WSrXPaU0dMBJ+6+tl0q/gnV11y/X u10kN+yqeay1MPrDm/hQwGlvBlTxNk+EkrDgA+Nor1FSQGi5J+Wl7vwFXOl1GRMtIgIwc DMD4NTmk+pRakgeDi3CEUkDf+6/W9bmzkIX0Ylw8d30Nql2ue40B2LPX15vMyZ5ZKmg== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=casper.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=ZJgZKNV0; arc=none smtp.remote-ip=90.155.50.34 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAB8A606A339; Thu, 5 Nov 2026 02:01:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org DAB8A606A339 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=casper.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=ZJgZKNV0 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description; bh=cS36VXsZObmcF5KZcGY9YdwDsYCRTEkKDtw4SlNBJeo=; b=ZJgZKNV0eN8AdXxFvUuk90tiD8 vE1BPAG6D/aNhtTw2m3t/hRNH5aPI3/7NCLRhexAECEu009aXZwg18CNDrXNj2axRztLy5ZIexufa 59g/7PUKYEZYh3i/SXuHx77nZl0yiK1EcW3/4rWV5W0rj0JF7IBDA61WmwRm7rvGHnz3dBiBD50q+ q6N18WSHGIv1OUplmLQGr+pNckiHWs4CHpJ0Cqvj+JRfZkjwUpTTiFlZvPciTCJYp4c9SkEhanV08 zy6YuGEJUt0d5wBDGHv1fpIbuQQsXN2kOEAEAssSBDI8cZI0txnS0LPmdKuYBaRqvjazi+eMSGZPw o+qTyGRA==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by casper.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmn-000000037jC-0E92; Thu, 05 Nov 2026 02:01:06 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmm-0000000AWDV-1xzt; Thu, 05 Nov 2026 02:01:04 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 10/20] KVM: pfncache: Add guest-mode pinning (GUEST_USES_PFN successor) Date: Thu, 5 Nov 2026 02:00:52 +0000 Message-ID: <20261105020104.2507083-11-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html From: David Woodhouse Reinstate support for caches whose pfn a vCPU uses directly from guest mode (stuffed into hardware control structures such as vmcs02 fields), addressing the reasons the original KVM_GUEST_USES_PFN was removed as incomplete/broken=C2=B9=C2=B2: - The kick now comes with a guarantee. The original KVM_REQ_OUTSIDE_GUEST_MODE kick did "absolutely nothing to guarantee KVM refreshes the cache before re-entering the guest"=C2=B9. Now, a vCPU pins with a cmpxchg which succeeds only while GPC_VALID holds, the invalidation walk reads the pin in the same atomic word it uses to veto in-flight publishes, and re-pinning requires a refresh which cannot complete until invalidate_range_end(): a forced-out vCPU cannot re-enter with the stale mapping. - The service request is per-cache, given at init=C2=B3 (kvm_gpc_init_for_vcpu()), so e.g. nVMX can request KVM_REQ_GET_NESTED_STATE_PAGES rather than KVM guessing what a kicked vCPU must do before re-entering. - No special-casing of non-blockable invalidations. The original stripped KVM_REQUEST_WAIT for the OOM reaper, assuming a reaped task's vCPUs are stopped; that is false for process_mrelease() and opened the very use-after-free the kick exists to prevent=E2=81=B4. The wait is unconditionally safe: kvm_make_vcpus_request_mask() spin-waits for IPI acks with preemption disabled, never sleeping. - Pin lifecycle has single-writer discipline: only the pinning vCPU sets or clears its pin, except that a mutator's xchg of the state word may consume it, inheriting the kick-and-wait duty. A stale pin costs at most a spurious request bit. No users yet; the nVMX conversion=E2=81=B5 builds on this. =C2=B9 https://lore.kernel.org/all/ZBEEQtmtNPaEqU1i@google.com =C2=B2 https://lore.kernel.org/all/ZQiR8IpqOZrOpzHC@google.com =C2=B3 https://lore.kernel.org/all/agJnqh7ACEkB-ftx@google.com =E2=81=B4 per Jann Horn's analysis in =C2=B2 =E2=81=B5 https://lore.kernel.org/all/20260102142429.896101-1-griffoul@gmai= l.com Signed-off-by: David Woodhouse --- arch/x86/kvm/xen.c | 4 +- include/linux/kvm_host.h | 64 +++++++++++++- include/linux/kvm_types.h | 28 ++++-- virt/kvm/pfncache.c | 181 ++++++++++++++++++++++++++++++++++++-- 4 files changed, 259 insertions(+), 18 deletions(-) diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c index 13e287bc17bb..132f0c379537 100644 --- a/arch/x86/kvm/xen.c +++ b/arch/x86/kvm/xen.c @@ -2465,7 +2465,7 @@ void kvm_xen_init_vcpu(struct kvm_vcpu *vcpu) =20 kvm_gpc_init(&vcpu->arch.xen.runstate_cache, vcpu->kvm); kvm_gpc_init(&vcpu->arch.xen.runstate2_cache, vcpu->kvm); - __kvm_gpc_init(&vcpu->arch.xen.vcpu_info_cache, vcpu->kvm, true); + __kvm_gpc_init(&vcpu->arch.xen.vcpu_info_cache, vcpu->kvm, true, NULL, 0); kvm_gpc_init(&vcpu->arch.xen.vcpu_time_info_cache, vcpu->kvm); } =20 @@ -2486,7 +2486,7 @@ void kvm_xen_init_vm(struct kvm *kvm) { mutex_init(&kvm->arch.xen.xen_lock); xa_init(&kvm->arch.xen.evtchn_ports); - __kvm_gpc_init(&kvm->arch.xen.shinfo_cache, kvm, true); + __kvm_gpc_init(&kvm->arch.xen.shinfo_cache, kvm, true, NULL, 0); } =20 void kvm_xen_destroy_vm(struct kvm *kvm) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 3ed0b8c9c102..008012d74d58 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1474,19 +1474,79 @@ int kvm_vcpu_write_guest(struct kvm_vcpu *vcpu, gpa= _t gpa, const void *data, * @gpc: struct gfn_to_pfn_cache object. * @kvm: pointer to kvm instance. * @never_dirty: the cache never needs to mark its page dirty in its mem= slot. + * @vcpu: the vCPU which may pin this cache's pfn for direct use from + * guest mode, or NULL for a cache which is never pinned. + * @vcpu_req: service request posted to @vcpu when a pinned cache is + * invalidated; must be non-zero when @vcpu is given (request + * 0 is a real request, so it cannot mean "none"). Most + * callers want the kvm_gpc_init_for_vcpu() wrapper. * * This sets up a gfn_to_pfn_cache by initializing locks and assigning the * immutable attributes. Note, the cache must be zero-allocated (or zeroe= d by * the caller before init). Most callers want the kvm_gpc_init() wrapper. */ void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm, - bool never_dirty); + bool never_dirty, struct kvm_vcpu *vcpu, u32 vcpu_req); =20 static inline void kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *= kvm) { - __kvm_gpc_init(gpc, kvm, false); + __kvm_gpc_init(gpc, kvm, false, NULL, 0); } =20 +/** + * kvm_gpc_init_for_vcpu - initialize a cache whose pfn a vCPU may pin for + * direct use from guest mode. + * + * @gpc: struct gfn_to_pfn_cache object. + * @vcpu: the vCPU which will pin and directly access this cache. + * @req: service request to post to @vcpu when the cache is invalidated + * while pinned. The invalidation separately forces @vcpu out of + * guest mode (KVM_REQ_OUTSIDE_GUEST_MODE, with WAIT) and does not + * complete until it has left; @req says what the vCPU must then do + * before re-entering. Its handler must re-establish the mapping + * (via a refresh, which cannot complete until the invalidation + * ends) and re-pin before the pfn may be used from guest mode + * again, or exit to userspace. + */ +static inline void kvm_gpc_init_for_vcpu(struct gfn_to_pfn_cache *gpc, + struct kvm_vcpu *vcpu, u32 req) +{ + __kvm_gpc_init(gpc, vcpu->kvm, false, vcpu, req); +} + +/** + * kvm_gpc_pin_for_guest - pin a valid cache's pfn for use from guest mode. + * + * @gpc: struct gfn_to_pfn_cache object (initialized with + * kvm_gpc_init_for_vcpu()). + * + * Marks the cache as being used directly by the vCPU in guest mode, so + * that an invalidation will kick the vCPU (with the request given at init + * time) and wait for it to be out of guest mode before completing. + * Succeeds only while the cache is valid: on success the pfn is + * guaranteed to remain mapped and unreclaimed until the pin is released + * or the vCPU is kicked. Must be called by @gpc->vcpu itself. + * + * Returns false if the cache is not currently valid; the caller must + * refresh and retry, or bail. Also returns false, permanently, for a + * cache initialized without a vCPU: a retry loop must not treat pin + * failure as transient unless the cache was initialized with + * kvm_gpc_init_for_vcpu(). + */ +bool kvm_gpc_pin_for_guest(struct gfn_to_pfn_cache *gpc); + +/** + * kvm_gpc_unpin_for_guest - release a guest-mode pin. + * + * @gpc: struct gfn_to_pfn_cache object. + * + * Must be called by @gpc->vcpu itself, after it has left guest mode (or + * before entering it), when the pfn is no longer stuffed into any + * hardware control structure. Idempotent: a pin already consumed by a + * mutator's teardown is harmless to release again. + */ +void kvm_gpc_unpin_for_guest(struct gfn_to_pfn_cache *gpc); + /** * kvm_gpc_activate - prepare a cached kernel mapping and HPA for a given = guest * physical address. diff --git a/include/linux/kvm_types.h b/include/linux/kvm_types.h index 22393f97266a..bdee01f22a2d 100644 --- a/include/linux/kvm_types.h +++ b/include/linux/kvm_types.h @@ -87,21 +87,34 @@ struct gfn_to_pfn_cache { unsigned long uhva; struct kvm_memory_slot *memslot; struct kvm *kvm; + /* + * A cache initialized with kvm_gpc_init_for_vcpu() may be pinned for + * direct use from guest mode (its pfn stuffed into hardware control + * structures such as vmcs02 fields) by setting GPC_GUEST_USING while + * GPC_VALID holds. When such a cache is invalidated, @vcpu is kicked + * out of guest mode with @vcpu_req, whose handler must re-pin (via a + * refresh which cannot complete until the invalidation ends) or exit. + */ + struct kvm_vcpu *vcpu; + u32 vcpu_req; struct list_head list; struct mutex refresh_lock; void *khva; kvm_pfn_t pfn; bool active; /* - * GPC_VALID, GPC_BECOMING_VALID and GPC_INVALIDATING live in a - * single atomic word so that a refresh can publish - * (BECOMING_VALID =3D> VALID) with a single conditional RMW, an + * GPC_VALID, GPC_BECOMING_VALID, GPC_INVALIDATING and + * GPC_GUEST_USING live in a single atomic word so that a refresh can + * publish (BECOMING_VALID =3D> VALID) with a single conditional RMW, an * invalidation can veto that publish by clearing both of those bits * (setting INVALIDATING in the same operation if VALID was set, to - * record that its grace period is still owed), and a mutator can - * consume the whole state with one xchg to learn whether it must - * wait for readers before touching anything they see. See the - * comments in pfncache.c. + * record that its grace period is still owed) and observe in the same + * read whether a vCPU has the pfn pinned for guest-mode use + * (GUEST_USING, set by the vCPU with a cmpxchg only while VALID + * holds, cleared only by that vCPU or by a mutator's xchg), and a + * mutator can consume the whole state with one xchg to learn whether + * it must wait for readers and/or kick the pinning vCPU before + * touching anything they see. See the comments in pfncache.c. * * 'active' deliberately stays separate: the invalidation and * mutation paths operate on the word without having to care about @@ -129,6 +142,7 @@ struct gfn_to_pfn_cache { * VALID clear. */ #define GPC_INVALIDATING 0x4 +#define GPC_GUEST_USING 0x8 =20 #ifdef KVM_ARCH_NR_OBJS_PER_MEMORY_CACHE /* diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c index 1c98b7bdbf09..5cb30525e577 100644 --- a/virt/kvm/pfncache.c +++ b/virt/kvm/pfncache.c @@ -25,7 +25,9 @@ void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm, unsigned long star= t, unsigned long end) { + DECLARE_BITMAP(vcpu_bitmap, KVM_MAX_VCPUS); struct gfn_to_pfn_cache *gpc; + bool evict_vcpus =3D false; bool cleared =3D false; =20 spin_lock(&kvm->gpc_lock); @@ -127,10 +129,60 @@ void gfn_to_pfn_cache_invalidate_start(struct kvm *kv= m, unsigned long start, */ if (old & (GPC_VALID | GPC_INVALIDATING)) cleared =3D true; + + /* + * If a vCPU has the pfn pinned for direct use from + * guest mode, it must be forced out of guest mode + * before the invalidation may complete. The pin bit + * was set with a cmpxchg only while GPC_VALID held, + * so this same read of 'old' cannot miss it. It is + * NOT cleared here: only the vCPU itself (on request + * service, before re-pinning) or a mutator's xchg + * (which inherits the kick duty) may clear it. A + * stale pin costs at most a spurious request bit: + * kvm_make_vcpus_request_mask() only IPIs and waits + * for targets actually in guest mode. + */ + if (old & GPC_GUEST_USING) { + if (!evict_vcpus) { + evict_vcpus =3D true; + bitmap_zero(vcpu_bitmap, KVM_MAX_VCPUS); + } + __set_bit(gpc->vcpu->vcpu_idx, vcpu_bitmap); + /* + * Set this cache's own service request (what + * the vCPU must do before re-entering); the + * force-out below is common to all of them. + */ + __kvm_make_request(gpc->vcpu_req, gpc->vcpu); + } } } spin_unlock(&kvm->gpc_lock); =20 + /* + * Force pinned vCPUs out of guest mode before waiting for the + * kernel readers below and before the caller zaps the page tables. + * KVM_REQ_OUTSIDE_GUEST_MODE carries KVM_REQUEST_WAIT, so this + * does not return until the vCPUs have left guest mode; they cannot + * re-enter with the stale mapping, because re-pinning requires + * GPC_VALID, which requires a refresh, which cannot complete until + * invalidate_range_end(). + * + * This is safe on unblockable (OOM reaper) ranges too, with no + * special-casing: kvm_make_vcpus_request_mask() runs with + * preemption disabled and spin-waits for IPI acks; it never sleeps. + * Do not be tempted to skip the wait for !blockable ranges on the + * assumption that a reaped task's vCPUs are already stopped: that + * is false for process_mrelease(), which only requires + * SIGNAL_GROUP_EXIT, and a vCPU still in guest mode after this + * walk returns is exactly the use-after-free this kick exists to + * prevent. + */ + if (evict_vcpus) + kvm_make_vcpus_request_mask(kvm, KVM_REQ_OUTSIDE_GUEST_MODE, + vcpu_bitmap); + /* * Readers may still be using the old mapping, having sampled * GPC_VALID before it was cleared. Wait for them all to drain @@ -224,6 +276,107 @@ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsi= gned long len) return true; } =20 +/* + * A mutator's xchg of the state word to zero may have consumed a live + * GPC_GUEST_USING pin. Clearing that bit obliges the clearer to do what + * the invalidation walk would have done: post the cache's service + * request and force the vCPU out of guest mode, waiting until it has + * left, before mutating or tearing down what the guest may be using. + * + * The service request is posted unconditionally: consuming a pin means + * whatever physical address the vCPU had latched (e.g. into a hardware + * control structure) is stale, and only the request handler makes that + * good. That holds even when the pinning vCPU is the caller itself =E2=80= =94 + * a vCPU refreshing its own pinned cache may re-map at a different pfn, + * and without the request nothing would rewrite the stale address + * before the next guest entry. Only the waited force-out is skipped in + * that case: a vCPU running a refresh is by definition not in guest + * mode. (The vCPU's own unpin is the one silent path, in + * kvm_gpc_unpin_for_guest(), because unpinning asserts the address is + * no longer latched anywhere.) + */ +static void gpc_kick_pinned_vcpu(struct gfn_to_pfn_cache *gpc, int old_sta= te) +{ + DECLARE_BITMAP(vcpu_bitmap, KVM_MAX_VCPUS); + + if (!(old_state & GPC_GUEST_USING)) + return; + if (WARN_ON_ONCE(!gpc->vcpu)) + return; + + __kvm_make_request(gpc->vcpu_req, gpc->vcpu); + + if (kvm_get_running_vcpu() =3D=3D gpc->vcpu) + return; + + /* + * As in the invalidation walk: force the vCPU out of guest mode and + * wait until it has left (KVM_REQ_OUTSIDE_GUEST_MODE carries + * KVM_REQUEST_WAIT). A plain kvm_vcpu_kick() would not wait. + */ + bitmap_zero(vcpu_bitmap, KVM_MAX_VCPUS); + __set_bit(gpc->vcpu->vcpu_idx, vcpu_bitmap); + kvm_make_vcpus_request_mask(gpc->kvm, KVM_REQ_OUTSIDE_GUEST_MODE, + vcpu_bitmap); +} + +bool kvm_gpc_pin_for_guest(struct gfn_to_pfn_cache *gpc) +{ + int old, new; + + /* + * Fail closed on a cache initialized without a vCPU: proceeding + * would arm the gpc->vcpu->vcpu_idx dereference in the + * invalidation walk, crashing inside the mmu_notifier rather + * than in the misusing caller. + */ + if (WARN_ON_ONCE(!gpc->vcpu)) + return false; + WARN_ON_ONCE(kvm_get_running_vcpu() !=3D gpc->vcpu); + + /* + * Set GPC_GUEST_USING only while GPC_VALID holds: the pin then + * cannot race with an invalidation, because the invalidation walk + * reads the whole state word in one atomic operation =E2=80=94 either it + * sees VALID (and this pin, if set) and kicks; or the publish which + * set VALID has not happened and the walk's veto of + * GPC_BECOMING_VALID prevents it. The acquire pairs with the + * release-publish so the pinned pfn/khva are the published ones. + */ + old =3D atomic_read_acquire(&gpc->state); + do { + if (!(old & GPC_VALID)) + return false; + new =3D old | GPC_GUEST_USING; + } while (!atomic_try_cmpxchg(&gpc->state, &old, new)); + + return true; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gpc_pin_for_guest); + +void kvm_gpc_unpin_for_guest(struct gfn_to_pfn_cache *gpc) +{ + /* + * Only the pinning vCPU itself may unpin =E2=80=94 except on teardown + * paths (vCPU destruction), which run with no running-vCPU + * context but with the vCPU necessarily out of guest mode. + */ + WARN_ON_ONCE(gpc->vcpu && kvm_get_running_vcpu() && + kvm_get_running_vcpu() !=3D gpc->vcpu); + + /* + * The vCPU's last access to the pfn must be visible before the pin + * is seen clear by an invalidator deciding it need not kick + * (atomic_andnot() alone is unordered). In practice the VM-exit + * which preceded this call already serialized the guest's accesses, + * but the API should not depend on its callers' exit paths. A pin + * already consumed by a mutator's xchg makes this a harmless no-op. + */ + smp_mb__before_atomic(); + atomic_andnot(GPC_GUEST_USING, &gpc->state); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gpc_unpin_for_guest); + static void *gpc_map(kvm_pfn_t pfn) { if (pfn_valid(pfn)) @@ -449,7 +602,7 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *g= pc, gpa_t gpa, unsigned l bool unmap_old =3D false; unsigned long old_uhva; kvm_pfn_t old_pfn; - bool must_drain; + int old_state; void *old_khva; int ret; =20 @@ -550,9 +703,9 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *g= pc, gpa_t gpa, unsigned l * only this wait covers. The marker comes down only once the drain * is complete and the readers it advertised cannot exist. */ - must_drain =3D atomic_xchg(&gpc->state, GPC_INVALIDATING) & - (GPC_VALID | GPC_INVALIDATING); - if (must_drain) + old_state =3D atomic_xchg(&gpc->state, GPC_INVALIDATING); + gpc_kick_pinned_vcpu(gpc, old_state); + if (old_state & (GPC_VALID | GPC_INVALIDATING)) synchronize_srcu_atomic(&gpc->kvm->gpc_srcu); atomic_set(&gpc->state, 0); =20 @@ -650,7 +803,7 @@ int kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, unsig= ned long len) } =20 void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm, - bool never_dirty) + bool never_dirty, struct kvm_vcpu *vcpu, u32 vcpu_req) { mutex_init(&gpc->refresh_lock); =20 @@ -661,6 +814,18 @@ void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, stru= ct kvm *kvm, gpc->active =3D false; atomic_set(&gpc->state, 0); gpc->never_dirty =3D never_dirty; + + /* + * A vCPU-owned cache must come with a service request for the + * invalidation walk to post; request 0 is a real request + * (KVM_REQ_TLB_FLUSH on x86), so a missing one cannot be allowed + * to alias it. Refuse the vcpu rather than arm that footgun =E2=80=94 + * the pin API then fails closed on !gpc->vcpu. + */ + if (WARN_ON_ONCE(vcpu && !vcpu_req)) + vcpu =3D NULL; + gpc->vcpu =3D vcpu; + gpc->vcpu_req =3D vcpu_req; } =20 static int __kvm_gpc_activate(struct gfn_to_pfn_cache *gpc, gpa_t gpa, uns= igned long uhva, @@ -718,6 +883,7 @@ void kvm_gpc_deactivate(struct gfn_to_pfn_cache *gpc) struct kvm *kvm =3D gpc->kvm; kvm_pfn_t old_pfn; void *old_khva; + int old_state; =20 guard(mutex)(&gpc->refresh_lock); =20 @@ -741,8 +907,9 @@ void kvm_gpc_deactivate(struct gfn_to_pfn_cache *gpc) * the drain completes, so that an mmu_notifier walk arriving * mid-wait does not read a zero state and skip the cache. */ - if (atomic_xchg(&gpc->state, GPC_INVALIDATING) & - (GPC_VALID | GPC_INVALIDATING)) + old_state =3D atomic_xchg(&gpc->state, GPC_INVALIDATING); + gpc_kick_pinned_vcpu(gpc, old_state); + if (old_state & (GPC_VALID | GPC_INVALIDATING)) synchronize_srcu_atomic(&kvm->gpc_srcu); atomic_set(&gpc->state, 0); =20 --=20 2.43.0 From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.50.34 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844072; b=a+Fg0TBnMzyJUC1/Xleui5t+Lusgbq8geRrFY6/sYu8n+bDxHLFKPS/Hy+sBDYcfivcz eMeE0o/+CbrpAvjNcTls84eo7OXKAC3bvsOmbWZ8cwfBiVh06gJJLS36EVWFmb+cHI4Cg 9qvNAMTEwDWgLBjcIGXuuNnhCtOMfYwY/qfJocN/j/HVGSpGODbYnwu8DHvBa2pl2lG4Q C/1WeMKjQ0CP2wt6i236nCmRsUc0IlVDkfW0cVhqvsxWMHyEqYrXYLpST3wNvvegoRj+a rLF/y+VDWxCVG0XbMpsmbrBXt65DuJshsgyxExlYlLUIZzV8LJVCEDRed1TqdP/Ye/Q== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844072; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=YL+lBqDUJNOy/hv9jUl5JyMaYSXCWE0E46LaudV422w=; b=M/PRonBBF6hK198TSEnqaZySW2DmfLRMCOLqLjqDwtL5DeDB+nMoAbLHYNPmJ/hyqH16 lAiKL5TJ87FuKN2IxACtCvwRMvtrmy/Pb0qDSqMXw0QVUlbK42aMvkclUXF44Z2tqLGUa j4IsPlJMg36wxMTyxxqsidw++3z8Omq/e2N12SbSxxrGO5/9kFrD6XvoI075sC2/9sjGF WySuKoWsQxbqkVBzKaLf8RPtUmFStDgqfXOgJDdt3B6ktbUJaN915DsPR6GMEANCeiEA0 3aNhRqhx1aDQL5y1KeeeXOBNCAEaRMT/VSL9FW5lgCdTmkvgZjszS4dDKReHqg1s1iw== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=casper.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=LHX+gf4e; arc=none smtp.remote-ip=90.155.50.34 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DBCCD606A33A; Thu, 5 Nov 2026 02:01:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org DBCCD606A33A Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=casper.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=LHX+gf4e DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=YL+lBqDUJNOy/hv9jUl5JyMaYSXCWE0E46LaudV422w=; b=LHX+gf4esAQAwNNl+6sG/OPhfw R0AS7Ypu1vuUXonqAi82K6sEU8nEQTohPmpglaMnfgujKbxMYtiZDwfnMc5RgxzYx9BXYeQiDAMZm KWqJ0hLZCM7yO6mVf4/S1eBxkDQN8JLP9eLte98q8UKOmgmNqgFKAG4DyeC8mQi2hq/bNhNak04yX xCAuPMY17KxtGBhqDlsyhA+eE5+JvKMghaoZ74/3CkRfTebGLS1uC/q109pOeaVTFbL2433a9qPFZ ICHIJMOOlMylejLTkfMikIFaOfAmdTYrWZYSCrWJc+nbP5tagApWfND7A6iCHsnUn6i7xhsLhDESJ 6Ebd5+9w==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by casper.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmn-000000037jD-0EW9; Thu, 05 Nov 2026 02:01:05 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmm-0000000AWDi-25oS; Thu, 05 Nov 2026 02:01:04 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 11/20] KVM: nVMX: Implement cache for L1 MSR bitmap Date: Thu, 5 Nov 2026 02:00:53 +0000 Message-ID: <20261105020104.2507083-12-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Content-Type: text/plain; charset="utf-8" From: Fred Griffoul Introduce a gfn_to_pfn_cache to optimize L1 MSR bitmap access by replacing map/unmap operations. This optimization reduces overhead during L2 VM-entry where nested_vmx_prepare_msr_bitmap() merges L1's MSR intercepts with L0's requirements. The current implementation maps and unmaps the page on every bitmap merge, which is a significant performance impact with unmanaged guest memory (where the map is a memremap/memunmap cycle). The cache is initialized once at vCPU creation and deactivated when VMX operation ends; re-entering VMX operation reactivates it through the normal check/activate path. Readers run under kvm->gpc_srcu, following the check/activate/retry pattern; KVM only ever reads the L1 bitmap, so the cache is mapped read-only (which L1 may legitimately demand for its bitmap) and never marked dirty. This exports the core pfncache API to kvm-intel.ko, which is its first modular user. [dwmw2: Port from the rwlock-protected pfncache to the SRCU reader protocol, rebase over the kvm_vcpu_map_local_readonly CLASS() conversion, introduce the lock/unlock helpers in x86.h as kvm_gpc_lock_page()/kvm_gpc_unlock() for later nSVM use, map the cache read-only, init at vCPU creation.] Reported-by: sashiko-bot@kernel.org Closes: https://lore.kernel.org/all/20260920215613.1CC0C1F000FF@smtp.kernel= .org Signed-off-by: Fred Griffoul Co-developed-by: David Woodhouse Signed-off-by: David Woodhouse --- arch/x86/kvm/vmx/nested.c | 30 +++++++++++++++++++++++++---- arch/x86/kvm/vmx/nested.h | 1 + arch/x86/kvm/vmx/vmx.c | 2 ++ arch/x86/kvm/vmx/vmx.h | 2 ++ arch/x86/kvm/x86.h | 33 ++++++++++++++++++++++++++++++++ arch/x86/kvm/xen.c | 4 ++-- include/linux/kvm_host.h | 40 +++++++++++++++++++++++++++------------ include/linux/kvm_types.h | 12 ++++++++++++ virt/kvm/pfncache.c | 37 +++++++++++++++++++++++++++++++----- 9 files changed, 138 insertions(+), 23 deletions(-) diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 516ba03bf608..bb0baa79c28c 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -370,6 +370,8 @@ static void free_nested(struct kvm_vcpu *vcpu) kfree(vmx->nested.cached_shadow_vmcs12); vmx->nested.cached_shadow_vmcs12 =3D NULL; =20 + kvm_gpc_deactivate(&vmx->nested.msr_bitmap_cache); + nested_put_vmcs12_pages(vcpu); =20 kvm_mmu_free_roots(vcpu->kvm, &vcpu->arch.guest_mmu, KVM_MMU_ROOTS_ALL); @@ -390,6 +392,24 @@ void nested_vmx_free_vcpu(struct kvm_vcpu *vcpu) vcpu_put(vcpu); } =20 +/* + * Initialize the nested gfn_to_pfn_caches, once per vCPU. Unlike the + * allocations in enter_vmx_operation(), the caches persist across + * VMXOFF/VMXON cycles: free_nested() only deactivates them, and reuse + * goes through kvm_gpc_activate() without re-init. + */ +void nested_vmx_init_caches(struct kvm_vcpu *vcpu) +{ + struct vcpu_vmx *vmx =3D to_vmx(vcpu); + + /* + * KVM only ever reads the L1 MSR bitmap, which L1 may keep in + * memory that its VMM maps read-only: map the cache read-only + * (which also means never marking it dirty). + */ + __kvm_gpc_init(&vmx->nested.msr_bitmap_cache, vcpu->kvm, false, true, NUL= L, 0); +} + #define EPTP_PA_MASK GENMASK_ULL(51, 12) =20 static bool nested_ept_root_matches(hpa_t root_hpa, u64 root_eptp, u64 ept= p) @@ -734,7 +754,7 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct= kvm_vcpu *vcpu, struct vmcs12 *vmcs12) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); - int msr; + int msr, idx; unsigned long *msr_bitmap_l1; unsigned long *msr_bitmap_l0 =3D vmx->nested.vmcs02.msr_bitmap; =20 @@ -759,11 +779,11 @@ static inline bool nested_vmx_prepare_msr_bitmap(stru= ct kvm_vcpu *vcpu, return true; } =20 - CLASS(kvm_vcpu_map_local_readonly, m)(vcpu, gpa_to_gfn(vmcs12->msr_bitmap= )); - if (m.ret) + idx =3D kvm_gpc_lock_page(&vmx->nested.msr_bitmap_cache, vmcs12->msr_bitm= ap); + if (idx < 0) return false; =20 - msr_bitmap_l1 =3D (unsigned long *)m.map.hva; + msr_bitmap_l1 =3D (unsigned long *)vmx->nested.msr_bitmap_cache.khva; =20 /* * To keep the control flow simple, pay eight 8-byte writes (sixteen @@ -843,6 +863,8 @@ static inline bool nested_vmx_prepare_msr_bitmap(struct= kvm_vcpu *vcpu, =20 nested_vmx_merge_pmu_msr_bitmaps(vcpu, msr_bitmap_l1, msr_bitmap_l0); =20 + kvm_gpc_unlock(&vmx->nested.msr_bitmap_cache, idx); + vmx->nested.force_msr_bitmap_recalc =3D false; =20 return true; diff --git a/arch/x86/kvm/vmx/nested.h b/arch/x86/kvm/vmx/nested.h index c6de848bd9ce..e27299b9f30b 100644 --- a/arch/x86/kvm/vmx/nested.h +++ b/arch/x86/kvm/vmx/nested.h @@ -24,6 +24,7 @@ __init int nested_vmx_hardware_setup(int (*exit_handlers[= ])(struct kvm_vcpu *)); void nested_vmx_set_vmcs_shadowing_bitmap(void); int nested_vmx_check_restored_vmcs12(struct kvm_vcpu *vcpu); void nested_vmx_free_vcpu(struct kvm_vcpu *vcpu); +void nested_vmx_init_caches(struct kvm_vcpu *vcpu); enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *v= cpu, bool from_vmentry); bool nested_vmx_reflect_vmexit(struct kvm_vcpu *vcpu); diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 82eb891f684c..590095bcb665 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7613,6 +7613,8 @@ int vmx_vcpu_create(struct kvm_vcpu *vcpu) WRITE_ONCE(to_kvm_vmx(vcpu->kvm)->pid_table[vcpu->vcpu_id], __pa(&vmx->vt.pi_desc) | PID_TABLE_ENTRY_VALID); =20 + nested_vmx_init_caches(vcpu); + return 0; =20 free_vmcs: diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index 74cd50f29fa4..c6827368c84a 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -146,6 +146,8 @@ struct nested_vmx { * Guest pages referred to in the vmcs02 with host-physical * pointers, so we must keep them pinned while L2 runs. */ + struct gfn_to_pfn_cache msr_bitmap_cache; + struct kvm_host_map apic_access_page_map; struct kvm_host_map virtual_apic_map; struct kvm_host_map pi_desc_map; diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 436e2fb1396c..c48f3aeb4d64 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -14,6 +14,39 @@ =20 #define KVM_MAX_MCE_BANKS 32 =20 +/* + * Map a single page of L1 guest memory at @gpa into a gfn_to_pfn_cache + * and enter an SRCU read-side critical section protecting the mapping. + * Returns the SRCU index (>=3D 0) to pass to kvm_gpc_unlock(), or a + * negative error code. While locked, the cache's khva and pfn are + * guaranteed to remain valid: an invalidation cannot complete until the + * reader drains. + */ +static inline int kvm_gpc_lock_page(struct gfn_to_pfn_cache *gpc, gpa_t gp= a) +{ + int idx, err; + + if (!PAGE_ALIGNED(gpa)) + return -EINVAL; +retry: + idx =3D srcu_read_lock_atomic(&gpc->kvm->gpc_srcu); + if (!kvm_gpc_check(gpc, PAGE_SIZE) || gpc->gpa !=3D gpa) { + srcu_read_unlock_atomic(&gpc->kvm->gpc_srcu, idx); + err =3D kvm_gpc_activate(gpc, gpa, PAGE_SIZE); + if (err) + return err; + + goto retry; + } + + return idx; +} + +static inline void kvm_gpc_unlock(struct gfn_to_pfn_cache *gpc, int idx) +{ + srcu_read_unlock_atomic(&gpc->kvm->gpc_srcu, idx); +} + int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops); void kvm_x86_vendor_exit(void); =20 diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c index 132f0c379537..ecfeeea57c66 100644 --- a/arch/x86/kvm/xen.c +++ b/arch/x86/kvm/xen.c @@ -2465,7 +2465,7 @@ void kvm_xen_init_vcpu(struct kvm_vcpu *vcpu) =20 kvm_gpc_init(&vcpu->arch.xen.runstate_cache, vcpu->kvm); kvm_gpc_init(&vcpu->arch.xen.runstate2_cache, vcpu->kvm); - __kvm_gpc_init(&vcpu->arch.xen.vcpu_info_cache, vcpu->kvm, true, NULL, 0); + __kvm_gpc_init(&vcpu->arch.xen.vcpu_info_cache, vcpu->kvm, true, false, N= ULL, 0); kvm_gpc_init(&vcpu->arch.xen.vcpu_time_info_cache, vcpu->kvm); } =20 @@ -2486,7 +2486,7 @@ void kvm_xen_init_vm(struct kvm *kvm) { mutex_init(&kvm->arch.xen.xen_lock); xa_init(&kvm->arch.xen.evtchn_ports); - __kvm_gpc_init(&kvm->arch.xen.shinfo_cache, kvm, true, NULL, 0); + __kvm_gpc_init(&kvm->arch.xen.shinfo_cache, kvm, true, false, NULL, 0); } =20 void kvm_xen_destroy_vm(struct kvm *kvm) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 008012d74d58..c604b7bb0b4c 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -1471,26 +1471,35 @@ int kvm_vcpu_write_guest(struct kvm_vcpu *vcpu, gpa= _t gpa, const void *data, /** * __kvm_gpc_init - initialize gfn_to_pfn_cache. * - * @gpc: struct gfn_to_pfn_cache object. - * @kvm: pointer to kvm instance. - * @never_dirty: the cache never needs to mark its page dirty in its mem= slot. - * @vcpu: the vCPU which may pin this cache's pfn for direct use from - * guest mode, or NULL for a cache which is never pinned. - * @vcpu_req: service request posted to @vcpu when a pinned cache is - * invalidated; must be non-zero when @vcpu is given (request - * 0 is a real request, so it cannot mean "none"). Most - * callers want the kvm_gpc_init_for_vcpu() wrapper. + * @gpc: struct gfn_to_pfn_cache object. + * @kvm: pointer to kvm instance. + * @never_dirty: KVM never marks the cached page dirty in its memslot; + * userspace must assume the page is always dirty (as for + * the Xen shared_info page, written on every event channel + * delivery). kvm_gpc_mark_dirty_in_slot() becomes a no-op. + * @readonly: map without FOLL_WRITE; KVM must never write through + * @gpc->khva. For guest pages KVM only reads, which the + * guest may place in memory its VMM maps read-only. + * Implies @never_dirty. + * @vcpu: the vCPU which may pin this cache's pfn for direct use + * from guest mode, or NULL. Callers use the + * kvm_gpc_init() / kvm_gpc_init_for_vcpu() wrappers + * rather than passing this directly. + * @vcpu_req: service request posted to @vcpu when the cache is + * invalidated while pinned; required (non-zero) when + * @vcpu is given. * * This sets up a gfn_to_pfn_cache by initializing locks and assigning the * immutable attributes. Note, the cache must be zero-allocated (or zeroe= d by * the caller before init). Most callers want the kvm_gpc_init() wrapper. */ void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm, - bool never_dirty, struct kvm_vcpu *vcpu, u32 vcpu_req); + bool never_dirty, bool readonly, + struct kvm_vcpu *vcpu, u32 vcpu_req); =20 static inline void kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *= kvm) { - __kvm_gpc_init(gpc, kvm, false, NULL, 0); + __kvm_gpc_init(gpc, kvm, false, false, NULL, 0); } =20 /** @@ -1511,7 +1520,7 @@ static inline void kvm_gpc_init(struct gfn_to_pfn_cac= he *gpc, struct kvm *kvm) static inline void kvm_gpc_init_for_vcpu(struct gfn_to_pfn_cache *gpc, struct kvm_vcpu *vcpu, u32 req) { - __kvm_gpc_init(gpc, vcpu->kvm, false, vcpu, req); + __kvm_gpc_init(gpc, vcpu->kvm, false, false, vcpu, req); } =20 /** @@ -2054,6 +2063,13 @@ static inline void kvm_gpc_mark_dirty_in_slot(struct= gfn_to_pfn_cache *gpc) RCU_LOCKDEP_WARN(!srcu_read_lock_held(&gpc->kvm->gpc_srcu), "kvm_gpc_mark_dirty_in_slot() without kvm->gpc_srcu read lock"); =20 + /* + * Nobody may write through (and hence dirty) a read-only cache: + * its mapping was created without FOLL_WRITE and may lack write + * permission entirely. + */ + WARN_ON_ONCE(gpc->readonly); + if (!gpc->memslot || gpc->never_dirty) return; =20 diff --git a/include/linux/kvm_types.h b/include/linux/kvm_types.h index bdee01f22a2d..193dd5b64c2a 100644 --- a/include/linux/kvm_types.h +++ b/include/linux/kvm_types.h @@ -122,7 +122,19 @@ struct gfn_to_pfn_cache { * instead. */ atomic_t state; + /* + * KVM never marks the cached page dirty in its memslot; userspace + * must assume the page is always dirty (as for the Xen shared_info + * page, written on every event channel delivery). + */ bool never_dirty; + /* + * The cache is mapped without FOLL_WRITE and KVM must never write + * through khva: for guest pages KVM only reads (e.g. the L1 MSR + * bitmap), which the guest may legitimately place in memory its VMM + * maps read-only. Implies never_dirty (there is nothing to mark). + */ + bool readonly; }; =20 /* The cache is valid: pfn and khva may be used by readers. */ diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c index 5cb30525e577..42adfdccd6fe 100644 --- a/virt/kvm/pfncache.c +++ b/virt/kvm/pfncache.c @@ -223,6 +223,20 @@ void gfn_to_pfn_cache_invalidate_start(struct kvm *kvm= , unsigned long start, */ } =20 +/* + * As gfn_to_hva_memslot(), but tolerate a KVM_MEM_READONLY slot for a + * cache which promises never to write through its mapping. + */ +static unsigned long gpc_gfn_to_hva_memslot(struct gfn_to_pfn_cache *gpc, + struct kvm_memory_slot *slot, + gfn_t gfn) +{ + if (gpc->readonly) + return gfn_to_hva_memslot_prot(slot, gfn, NULL); + + return gfn_to_hva_memslot(slot, gfn); +} + static bool kvm_gpc_is_valid_len(gpa_t gpa, unsigned long uhva, unsigned long len) { @@ -275,6 +289,7 @@ bool kvm_gpc_check(struct gfn_to_pfn_cache *gpc, unsign= ed long len) =20 return true; } +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gpc_check); =20 /* * A mutator's xchg of the state word to zero may have consumed a live @@ -448,7 +463,7 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cac= he *gpc) struct kvm_follow_pfn kfp =3D { .slot =3D gpc->memslot, .gfn =3D gpa_to_gfn(gpc->gpa), - .flags =3D FOLL_WRITE, + .flags =3D gpc->readonly ? 0 : FOLL_WRITE, .hva =3D gpc->uhva, .refcounted_page =3D &page, }; @@ -655,7 +670,7 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *g= pc, gpa_t gpa, unsigned l new_generation =3D slots->generation; new_slot =3D __gfn_to_memslot(slots, gfn); if (new_slot) - new_uhva =3D gfn_to_hva_memslot(new_slot, gfn) + + new_uhva =3D gpc_gfn_to_hva_memslot(gpc, new_slot, gfn) + offset_in_page(gpa); } =20 @@ -738,7 +753,8 @@ static int __kvm_gpc_refresh(struct gfn_to_pfn_cache *g= pc, gpa_t gpa, unsigned l gpc->gpa =3D gpa; gpc->generation =3D slots->generation; gpc->memslot =3D __gfn_to_memslot(slots, gfn); - WRITE_ONCE(gpc->uhva, gfn_to_hva_memslot(gpc->memslot, gfn)); + WRITE_ONCE(gpc->uhva, + gpc_gfn_to_hva_memslot(gpc, gpc->memslot, gfn)); =20 if (kvm_is_error_hva(gpc->uhva)) { ret =3D -EFAULT; @@ -801,9 +817,11 @@ int kvm_gpc_refresh(struct gfn_to_pfn_cache *gpc, unsi= gned long len) =20 return __kvm_gpc_refresh(gpc, gpc->gpa, uhva); } +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gpc_refresh); =20 void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struct kvm *kvm, - bool never_dirty, struct kvm_vcpu *vcpu, u32 vcpu_req) + bool never_dirty, bool readonly, + struct kvm_vcpu *vcpu, u32 vcpu_req) { mutex_init(&gpc->refresh_lock); =20 @@ -813,7 +831,13 @@ void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, stru= ct kvm *kvm, gpc->uhva =3D KVM_HVA_ERR_BAD; gpc->active =3D false; atomic_set(&gpc->state, 0); - gpc->never_dirty =3D never_dirty; + /* + * A read-only cache maps without FOLL_WRITE, so writing through + * gpc->khva would hit a mapping which may lack write permission; + * there is never anything to mark dirty. + */ + gpc->never_dirty =3D never_dirty || readonly; + gpc->readonly =3D readonly; =20 /* * A vCPU-owned cache must come with a service request for the @@ -827,6 +851,7 @@ void __kvm_gpc_init(struct gfn_to_pfn_cache *gpc, struc= t kvm *kvm, gpc->vcpu =3D vcpu; gpc->vcpu_req =3D vcpu_req; } +EXPORT_SYMBOL_FOR_KVM_INTERNAL(__kvm_gpc_init); =20 static int __kvm_gpc_activate(struct gfn_to_pfn_cache *gpc, gpa_t gpa, uns= igned long uhva, unsigned long len) @@ -869,6 +894,7 @@ int kvm_gpc_activate(struct gfn_to_pfn_cache *gpc, gpa_= t gpa, unsigned long len) =20 return __kvm_gpc_activate(gpc, gpa, KVM_HVA_ERR_BAD, len); } +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gpc_activate); =20 int kvm_gpc_activate_hva(struct gfn_to_pfn_cache *gpc, unsigned long uhva,= unsigned long len) { @@ -932,3 +958,4 @@ void kvm_gpc_deactivate(struct gfn_to_pfn_cache *gpc) gpc_unmap(old_pfn, old_khva); } } +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_gpc_deactivate); --=20 2.43.0 From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.92.199 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844079; b=oLs6MuR/2OxNsjWjwpJNGFrRhN4X0DQpwJAs/AWrQ7NktV05u6KCFIgLRMxydholwN00 m/qkHR7oMuVx8FacUzsUuXYya9FUeE5NSoulKGBBS+5BU2ePvDQjhxpd3+OuHlPhDGMKb 695Ap3FePjL1DjxldzksAk29tU8060/5VNx9qQe0EmUIopBHWE8dsuZFDPctAL+Xr65Lk oxSk9fVH1P9VmZnlFo9Q/sDJF2OPaBXe0bmGUGSoyNIuRmF4iE9/g+X+7EOWUG7kPNWca TE2DK6lGiMH8cAKjN3a7+pbYvZ2PdWnn+rlb2UzOfwl/XViEPEib6ysqM7QIdnEg3pg== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844079; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Type:Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=lq/Kk5OeAuXHYB6wOIZIFiKrOrr7XjZr/0fAEOHzrvs=; b=iIU0FPazy7C8f9eLs2YWZ/DI3kcKRuzxrKYET1OaaNXaVgJ3c5/d26oA8ws5dXC2HWDU boCwCR/SBLai97ox2n9Fp0Vbtxs58O89h23pYRl9YA60IN9tcgG7rRTB/AIS0ixGq0hur 1ySaurl2AfsKOU1SdFLtbCidvU5etNKOqG12XOJGDOr77l6+MTQarGo510T5Yyp2dJQ2B YzgvkBvC4WYHZeMuE6Wu53qTi1IKDmwJJkq5WcIN0RkoSnT/9/0p/6qzSu+EsfWuCkT3N 6lVY+LEPF2pBuYS0673GJUlARCwrz9gTmkf1RSxXhUzdMYoOiKYS2bvi6VoORhNM1QA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=desiato.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=desiato.20200630 header.b=O89uQik+; arc=none smtp.remote-ip=90.155.92.199 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3F8B2606EE98; Thu, 5 Nov 2026 02:01:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 3F8B2606EE98 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=desiato.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=desiato.20200630 header.b=O89uQik+ DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description; bh=lq/Kk5OeAuXHYB6wOIZIFiKrOrr7XjZr/0fAEOHzrvs=; b=O89uQik+whhvPMgG8J+xQHJBrN 9XpIKGBPlO7HfZuA1emoSdJmLmCKTL39GTpJ59I8rMQSVOEDwpLFGVwSLaUkF1nZkXeCY+/s+SfcW D2QFsFLR+6LAD+gEtcQzbAMjvp6xoPnXiqsdewFEIYWfoaWUr0ITVsH+yF47RTHzPvvn6ZvPjBDI3 TxJR5RIKorPu3jPB34BspRwIf3472huijbKnEFeHHcHZj4w5Z1IvOa1hpU8ox8i8v6W/FIab07JjH ZPEVSyYSv03NTP8b9cZthlBIWMCkCAb1gcT93PNJEzlDBgBZrhZufqINdBP+r9/jtogu/TQdHzq8j S1PAWfDg==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by desiato.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmn-00000002ROO-25Pd; Thu, 05 Nov 2026 02:01:05 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmm-0000000AWDm-2Fyj; Thu, 05 Nov 2026 02:01:04 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 12/20] KVM: nVMX: Use pinned pfncache for L1 APIC virtualization pages Date: Thu, 5 Nov 2026 02:00:54 +0000 Message-ID: <20261105020104.2507083-13-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by desiato.infradead.org. See http://www.infradead.org/rpr.html From: Fred Griffoul Replace kvm_host_map usage with gfn_to_pfn_cache for the L1 pages whose physical addresses are handed to the CPU in vmcs02 for direct use while running L2: the APIC-access page, the virtual-APIC page, and the posted interrupt descriptor. This eliminates the map/unmap (with unmanaged memory: memremap/memunmap) cycle on every L2 VM-entry/exit. The caches are initialized once at vCPU creation with kvm_gpc_init_for_vcpu(), naming KVM_REQ_GET_NESTED_STATE_PAGES as the service request, and pinned when written into vmcs02 by nested_get_vmcs12_pages(). An invalidation of a pinned cache posts that request, forces the vCPU out of guest mode and waits; the handler re-runs nested_get_vmcs12_pages(). Pins are dropped at nested VM-exit (including the vmentry_fail_vmexit path, which does not pass through __nested_vmx_vmexit()). No validity check is added to the guest-entry path: a stale physical address latched in vmcs02 always implies a pending request. Every path which consumes a pin posts the request, and the handler cannot lose it: any failure of vmx_get_nested_state_pages() re-posts it before exiting to userspace, so a persistently failing configuration bounces to userspace rather than ever entering L2 stale. This closes the historical KVM_GUEST_USES_PFN gaps=C2=B9 without polling cache validity on every entry. Resolution failure distinguishes three cases. A slot mid-update (the pfncache reports -EFAULT; finding the KVM_MEMSLOT_INVALID placeholder still installed identifies the failure as transient) re-posts the request and continues, retrying until the update commits or reverts. At entry time (VMLAUNCH/VMRESUME, or resume from KVM_SET_NESTED_STATE, tracked in nested.resume_pending =E2=80=94 which dies with the request at nested VM-exit, lest a later lazy request inherit entry-time semantics) a genuinely unbackable page is a misconfiguration: KVM_EXIT_INTERNAL_ERROR, as ever. On lazy revalidation it means userspace yanked the backing from under a running L2: for the APIC-access page and the posted interrupt descriptor, clear the corresponding vmcs02 control instead of killing the VM, so L2's accesses become ordinary EPT violations handled as MMIO; the successful re-resolution path sets the control back, so a later update re-backing the gfn heals the degradation without waiting for a nested VM-entry. The virtual-APIC page allows no such degradation (the TPR shadow is live, and the entry-time INVALID_GPA fallback would deliver a VMfailValid to an L1 whose VMLAUNCH retired long ago), and an unbacked gfn cannot be an error either: a routine DELETE-then-recreate reshuffle has a committed gap in which the gfn is genuinely absent, and the lazy revalidation runs inside that gap precisely because the DELETE's commit posted the request. So a vAPIC loss while L2 runs is ridden out like the in-flight window: the pending request keeps L2 un-enterable until a later update re-backs the gfn, and a userspace which never does leaves its vCPU retrying in KVM_RUN, as for RET_PF_RETRY. The PAE PDPTR reload runs only at entry-time resolution: PDPTRs latch at MOV CR3 and VM-entry, so a lazy revalidation must not re-derive them. Host-side (khva) readers do not need the pin. vmx_complete_nested_posted_interrupt() resolves both caches before entering either one's atomic read section, and vmx_has_nested_events(), which runs under set_current_state() in the block loop where a refresh must not sleep, uses a non-sleeping try-lock and reports "event pending" for an unusable cache, posting KVM_REQ_GET_NESTED_STATE_PAGES itself (a bare memslot-generation bump fails the check without anything else posting it): the bounce through vcpu_run re-establishes the cache and re-evaluates for real, and a failed activation deactivates the cache so an unbackable page cannot livelock that conservative answer. The try-lock takes kvm->srcu across its memslot-generation check; callers in the vcpu_block() loop run with the vCPU's srcu lock dropped. Dirty marking moves from the unmap path to gfn-based marking on every L2 exit (__vmx_handle_exit() already did that, making unmap-time marking redundant); the one software writer, __kvm_apic_update_irr(), marks at the write site. =C2=B9 https://lore.kernel.org/all/ZBEEQtmtNPaEqU1i@google.com Signed-off-by: Fred Griffoul Co-developed-by: David Woodhouse Signed-off-by: David Woodhouse --- arch/x86/kvm/vmx/nested.c | 483 ++++++++++++++++++++++++++++++++++---- arch/x86/kvm/vmx/vmx.c | 11 +- arch/x86/kvm/vmx/vmx.h | 21 +- 3 files changed, 458 insertions(+), 57 deletions(-) diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index bb0baa79c28c..23f1fff97917 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -320,14 +320,140 @@ static void vmx_switch_vmcs(struct kvm_vcpu *vcpu, s= truct loaded_vmcs *vmcs) kvm_reset_dirty_registers(vcpu); } =20 +/* + * Map a page of L1 memory, pin it for direct use by the guest (i.e. by + * the CPU while running the L2 guest, via a physical address in vmcs02), + * and return the host physical address to write into vmcs02. The pin + * guarantees that if the mapping is invalidated, this vCPU is forced out + * of guest mode and KVM_REQ_GET_NESTED_STATE_PAGES is posted so that the + * address is re-established (or the vCPU exits to userspace) before the + * next entry to guest mode. + */ +static int nested_gpc_hpa(struct gfn_to_pfn_cache *gpc, gpa_t gpa, hpa_t *= hpa) +{ + int idx; + + do { + idx =3D kvm_gpc_lock_page(gpc, gpa); + if (idx =3D=3D -EFAULT) { + struct kvm_memory_slot *slot; + + /* + * An unbackable gfn and a memslot mid-update (a + * DELETE or MOVE keeps a KVM_MEMSLOT_INVALID + * placeholder installed until it commits) both fail + * uHVA resolution with -EFAULT. Distinguish them: + * finding the placeholder in the active memslots + * (kvm->srcu is held by every caller) means the + * failure is transient, and the callers' -EAGAIN + * handling =E2=80=94 re-post the request, don't enter L2 =E2=80=94 + * is what must ride it out; only a truly absent + * slot means the page is unbacked. + */ + slot =3D gfn_to_memslot(gpc->kvm, gpa_to_gfn(gpa)); + if (slot && (slot->flags & KVM_MEMSLOT_INVALID)) + return -EAGAIN; + } + if (idx < 0) + return idx; + + /* + * The pin can fail only if an invalidation cleared the + * valid bit after kvm_gpc_lock_page() checked it; go back + * around to refresh (which will not complete until the + * invalidation is over) and try again. + */ + if (kvm_gpc_pin_for_guest(gpc)) + break; + + kvm_gpc_unlock(gpc, idx); + } while (1); + + *hpa =3D pfn_to_hpa(gpc->pfn); + kvm_gpc_unlock(gpc, idx); + return 0; +} + +/* + * Try to lock an already-configured cache for host-side (khva) access, + * without sleeping. Returns -ENOENT if the cache is not active (e.g. + * posted interrupts not configured for this L2), -EWOULDBLOCK if it is + * active but currently invalidated (only a sleeping refresh would make + * it usable). Does not pin; the mapping is stable only until + * kvm_gpc_unlock(). + */ +static int nested_gpc_try_lock_if_active(struct gfn_to_pfn_cache *gpc) +{ + int idx, srcu_idx; + bool usable; + + /* + * The caches are initialized at vCPU creation, so gpc->kvm is + * always valid; !active covers a cache never activated (or since + * deactivated) for this L2. Deactivation only happens on this + * vCPU (under vcpu->mutex), so the check cannot race with the + * lock which follows it. + */ + if (!gpc->active) + return -ENOENT; + + idx =3D srcu_read_lock_atomic(&gpc->kvm->gpc_srcu); + if (!gpc->active) { + srcu_read_unlock_atomic(&gpc->kvm->gpc_srcu, idx); + return -ENOENT; + } + + /* + * kvm_gpc_check() consults kvm_memslots() (for the generation of + * a GPA-based cache), which is protected by kvm->srcu =E2=80=94 and not + * every caller holds that: kvm_arch_vcpu_runnable() runs from the + * vcpu_block() loop with the vCPU's srcu lock deliberately + * dropped. Take it across the check; the khva the caller then + * reads is protected by gpc_srcu, not kvm->srcu. + */ + srcu_idx =3D srcu_read_lock(&gpc->kvm->srcu); + usable =3D kvm_gpc_check(gpc, PAGE_SIZE); + srcu_read_unlock(&gpc->kvm->srcu, srcu_idx); + + if (!usable) { + srcu_read_unlock_atomic(&gpc->kvm->gpc_srcu, idx); + return -EWOULDBLOCK; + } + + return idx; +} + +/* + * As above, but refresh the cache if it has been invalidated. May sleep; + * callers must be in a sleepable context (in particular, NOT under + * set_current_state() in a block loop, and not in a VM-exit fastpath). + */ +static int nested_gpc_lock_if_active(struct gfn_to_pfn_cache *gpc) +{ + int idx, err; + + while ((idx =3D nested_gpc_try_lock_if_active(gpc)) =3D=3D -EWOULDBLOCK) { + err =3D kvm_gpc_refresh(gpc, PAGE_SIZE); + if (err) + return err; + } + + return idx; +} + +static struct pi_desc *nested_pi_desc(struct vcpu_vmx *vmx) +{ + return (struct pi_desc *)((u8 *)vmx->nested.pi_desc_cache.khva + + vmx->nested.pi_desc_offset); +} + static void nested_put_vmcs12_pages(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); =20 - kvm_vcpu_unmap(vcpu, &vmx->nested.apic_access_page_map); - kvm_vcpu_unmap(vcpu, &vmx->nested.virtual_apic_map); - kvm_vcpu_unmap(vcpu, &vmx->nested.pi_desc_map); - vmx->nested.pi_desc =3D NULL; + kvm_gpc_unpin_for_guest(&vmx->nested.apic_access_page_cache); + kvm_gpc_unpin_for_guest(&vmx->nested.virtual_apic_cache); + kvm_gpc_unpin_for_guest(&vmx->nested.pi_desc_cache); } =20 /* @@ -346,6 +472,7 @@ static void free_nested(struct kvm_vcpu *vcpu) return; =20 kvm_clear_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu); + vmx->nested.resume_pending =3D false; =20 vmx->nested.vmxon =3D false; vmx->nested.smm.vmxon =3D false; @@ -370,10 +497,13 @@ static void free_nested(struct kvm_vcpu *vcpu) kfree(vmx->nested.cached_shadow_vmcs12); vmx->nested.cached_shadow_vmcs12 =3D NULL; =20 - kvm_gpc_deactivate(&vmx->nested.msr_bitmap_cache); - nested_put_vmcs12_pages(vcpu); =20 + kvm_gpc_deactivate(&vmx->nested.pi_desc_cache); + kvm_gpc_deactivate(&vmx->nested.virtual_apic_cache); + kvm_gpc_deactivate(&vmx->nested.apic_access_page_cache); + kvm_gpc_deactivate(&vmx->nested.msr_bitmap_cache); + kvm_mmu_free_roots(vcpu->kvm, &vcpu->arch.guest_mmu, KVM_MMU_ROOTS_ALL); =20 nested_release_evmcs(vcpu); @@ -408,6 +538,20 @@ void nested_vmx_init_caches(struct kvm_vcpu *vcpu) * (which also means never marking it dirty). */ __kvm_gpc_init(&vmx->nested.msr_bitmap_cache, vcpu->kvm, false, true, NUL= L, 0); + + /* + * The pfns of these three caches are handed to the CPU in vmcs02 + * fields, for direct use while running the L2 guest. Pinning them + * for guest use makes an invalidation force this vCPU out of + * guest mode and post KVM_REQ_GET_NESTED_STATE_PAGES, whose + * handler re-establishes them before the next entry. + */ + kvm_gpc_init_for_vcpu(&vmx->nested.apic_access_page_cache, vcpu, + KVM_REQ_GET_NESTED_STATE_PAGES); + kvm_gpc_init_for_vcpu(&vmx->nested.virtual_apic_cache, vcpu, + KVM_REQ_GET_NESTED_STATE_PAGES); + kvm_gpc_init_for_vcpu(&vmx->nested.pi_desc_cache, vcpu, + KVM_REQ_GET_NESTED_STATE_PAGES); } =20 #define EPTP_PA_MASK GENMASK_ULL(51, 12) @@ -3467,41 +3611,117 @@ static bool nested_get_evmcs_page(struct kvm_vcpu = *vcpu) } #endif =20 -static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu) +static bool nested_get_vmcs12_pages(struct kvm_vcpu *vcpu, bool lazy) { struct vmcs12 *vmcs12 =3D get_vmcs12(vcpu); struct vcpu_vmx *vmx =3D to_vmx(vcpu); - struct kvm_host_map *map; + hpa_t hpa; =20 - if (!vcpu->arch.pdptrs_from_userspace && + if (!lazy && !vcpu->arch.pdptrs_from_userspace && !nested_cpu_has_ept(vmcs12) && is_pae_paging(vcpu)) { /* * Reload the guest's PDPTRs since after a migration * the guest CR3 might be restored prior to setting the nested - * state which can lead to a load of wrong PDPTRs. + * state which can lead to a load of wrong PDPTRs. Only at + * entry-time resolution: PDPTRs latch at MOV CR3 and + * VM-entry, so a lazy revalidation (a memslot update while + * L2 runs) must not re-derive them at a moment the + * architecture does not allow. */ - if (CC(!load_pdptrs(vcpu, vcpu->arch.cr3))) + if (CC(!load_pdptrs(vcpu, vcpu->arch.cr3))) { + /* + * Report the unusable PDPT rather than return to + * userspace with a stale exit_reason. + */ + vcpu->run->exit_reason =3D KVM_EXIT_INTERNAL_ERROR; + vcpu->run->internal.suberror =3D + KVM_INTERNAL_ERROR_EMULATION; + vcpu->run->internal.ndata =3D 0; return false; + } } =20 =20 if (nested_cpu_has2(vmcs12, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES)) { - map =3D &vmx->nested.apic_access_page_map; - - if (!kvm_vcpu_map(vcpu, gpa_to_gfn(vmcs12->apic_access_addr), map)) { - vmcs_write64(APIC_ACCESS_ADDR, pfn_to_hpa(map->pfn)); - } else { + int err =3D nested_gpc_hpa(&vmx->nested.apic_access_page_cache, + vmcs12->apic_access_addr, &hpa); + if (!err) { + vmcs_write64(APIC_ACCESS_ADDR, hpa); + /* + * Undo any earlier lazy degradation: once the page + * is backed again the vmcs02 control must match + * vmcs12, and prepare_vmcs02_early() only runs at + * nested VM-entry, which a vCPU parked in L2 may + * not reach for a long time. + */ + secondary_exec_controls_setbit(vmx, + SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES); + } else if (err =3D=3D -EAGAIN) { + kvm_gpc_unpin_for_guest(&vmx->nested.apic_access_page_cache); + kvm_gpc_deactivate(&vmx->nested.apic_access_page_cache); + goto retry; + } else if (!lazy) { + /* + * Entry-time resolution (VMLAUNCH/VMRESUME, or resume + * from KVM_SET_NESTED_STATE): an unbackable + * APIC-access page is a misconfiguration to report, + * not a transient to ride out. + */ pr_debug_ratelimited("%s: no backing for APIC-access address in vmcs12\= n", __func__); + vcpu->run->exit_reason =3D KVM_EXIT_INTERNAL_ERROR; + vcpu->run->internal.suberror =3D + KVM_INTERNAL_ERROR_EMULATION; + vcpu->run->internal.ndata =3D 0; + kvm_gpc_unpin_for_guest(&vmx->nested.apic_access_page_cache); + kvm_gpc_deactivate(&vmx->nested.apic_access_page_cache); return false; + } else { + /* + * Lazy revalidation (a memslot update or invalidation + * while L2 runs) found the gfn unbacked: userspace + * yanked the backing from under a running L2. Turn + * off the APIC-access match for vmcs02 instead of + * killing the VM; L2's accesses to the page then take + * ordinary EPT violations on an unbacked gfn and are + * handled (or reported) as MMIO, exactly as a + * non-nested guest touching the same gpa would be. + * prepare_vmcs02_early() recomputes the control from + * vmcs12 on the next nested entry, and a later + * memslot update which re-backs the gfn re-posts the + * request and re-latches the page, so the + * degradation heals itself. + */ + kvm_gpc_unpin_for_guest(&vmx->nested.apic_access_page_cache); + kvm_gpc_deactivate(&vmx->nested.apic_access_page_cache); + secondary_exec_controls_clearbit(vmx, + SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES); } } =20 if (nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW)) { - map =3D &vmx->nested.virtual_apic_map; + int vapic_err =3D nested_gpc_hpa(&vmx->nested.virtual_apic_cache, + vmcs12->virtual_apic_page_addr, &hpa); + + /* + * On failure, deactivate the cache: an active-but-invalid + * cache would read as "might have an event" to the + * non-sleeping check in vmx_has_nested_events(), causing + * spurious wakeups for as long as the page remains + * unbackable. + */ + if (vapic_err) { + kvm_gpc_unpin_for_guest(&vmx->nested.virtual_apic_cache); + kvm_gpc_deactivate(&vmx->nested.virtual_apic_cache); + } =20 - if (!kvm_vcpu_map(vcpu, gpa_to_gfn(vmcs12->virtual_apic_page_addr), map)= ) { - vmcs_write64(VIRTUAL_APIC_PAGE_ADDR, pfn_to_hpa(map->pfn)); + if (vapic_err =3D=3D -EAGAIN) + goto retry; + + if (!vapic_err) { + vmcs_write64(VIRTUAL_APIC_PAGE_ADDR, hpa); + /* As for the APIC-access page: undo a lazy degrade. */ + exec_controls_setbit(vmx, CPU_BASED_TPR_SHADOW); } else if (nested_cpu_has(vmcs12, CPU_BASED_CR8_LOAD_EXITING) && nested_cpu_has(vmcs12, CPU_BASED_CR8_STORE_EXITING) && !nested_cpu_has2(vmcs12, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES)) { @@ -3514,6 +3734,27 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu = *vcpu) * only possibility we have. */ exec_controls_clearbit(vmx, CPU_BASED_TPR_SHADOW); + } else if (lazy) { + /* + * Unlike the two sibling pages, a vAPIC loss cannot + * be degraded around: the TPR shadow is live. Nor is + * the entry-time INVALID_GPA fallback usable here =E2=80=94 + * the resulting hardware VM-entry failure would be + * delivered as VMfailValid to an L1 whose VMLAUNCH + * retired long ago, rewinding L2. Nor can an unbacked + * gfn be reported as an error: a routine userspace + * DELETE-then-recreate reshuffle has a committed gap + * in which the gfn is genuinely absent, and this + * revalidation runs inside it precisely because the + * DELETE's commit posted the request. Ride it out + * exactly like the in-flight window: keep the + * request pending so L2 cannot be entered, and let + * the recreate (whose commit posts again) supply the + * page. A userspace which never re-backs the gfn + * leaves the vCPU retrying in KVM_RUN indefinitely =E2=80=94 + * as for RET_PF_RETRY, that is its own doing. + */ + goto retry; } else { /* * Write an illegal value to VIRTUAL_APIC_PAGE_ADDR to @@ -3524,14 +3765,15 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu= *vcpu) } =20 if (nested_cpu_has_posted_intr(vmcs12)) { - map =3D &vmx->nested.pi_desc_map; - - if (!kvm_vcpu_map(vcpu, gpa_to_gfn(vmcs12->posted_intr_desc_addr), map))= { - vmx->nested.pi_desc =3D - (struct pi_desc *)(((void *)map->hva) + - offset_in_page(vmcs12->posted_intr_desc_addr)); + int pi_err =3D nested_gpc_hpa(&vmx->nested.pi_desc_cache, + vmcs12->posted_intr_desc_addr & PAGE_MASK, &hpa); + if (!pi_err) { + vmx->nested.pi_desc_offset =3D + offset_in_page(vmcs12->posted_intr_desc_addr); vmcs_write64(POSTED_INTR_DESC_ADDR, - pfn_to_hpa(map->pfn) + offset_in_page(vmcs12->posted_intr_desc_ad= dr)); + hpa + offset_in_page(vmcs12->posted_intr_desc_addr)); + /* As for the APIC-access page: undo a lazy degrade. */ + pin_controls_setbit(vmx, PIN_BASED_POSTED_INTR); } else { /* * Defer the KVM_INTERNAL_EXIT until KVM tries to @@ -3539,7 +3781,10 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu = *vcpu) * descriptor. (Note that KVM may do this when it * should not, per the architectural specification.) */ - vmx->nested.pi_desc =3D NULL; + kvm_gpc_unpin_for_guest(&vmx->nested.pi_desc_cache); + kvm_gpc_deactivate(&vmx->nested.pi_desc_cache); + if (pi_err =3D=3D -EAGAIN) + goto retry; pin_controls_clearbit(vmx, PIN_BASED_POSTED_INTR); } } @@ -3549,9 +3794,27 @@ static bool nested_get_vmcs12_pages(struct kvm_vcpu = *vcpu) exec_controls_clearbit(vmx, CPU_BASED_USE_MSR_BITMAPS); =20 return true; + +retry: + /* + * The gfn's memslot is being moved or deleted: present but + * flagged invalid, so the lookup failure is transient and the + * gfn must not (yet) be treated as unbacked. Re-post the + * request and report success: the pending request prevents + * any VM entry (vcpu_enter_guest() bails and reprocesses + * requests), so L2 cannot run with the partially-updated + * vmcs02, and the vCPU retries here until the memslot update + * completes or reverts. This mirrors RET_PF_RETRY for + * ordinary guest faults on an invalid slot; as there, a + * userspace which never completes the update leaves the vCPU + * retrying indefinitely, with signals and request processing + * still serviced. + */ + kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu); + return true; } =20 -static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu) +static bool __vmx_get_nested_state_pages(struct kvm_vcpu *vcpu) { #ifdef CONFIG_KVM_HYPERV /* @@ -3567,9 +3830,39 @@ static bool vmx_get_nested_state_pages(struct kvm_vc= pu *vcpu) } #endif =20 - if (is_guest_mode(vcpu) && !nested_get_vmcs12_pages(vcpu)) + if (is_guest_mode(vcpu) && + !nested_get_vmcs12_pages(vcpu, !to_vmx(vcpu)->nested.resume_pending)) return false; =20 + /* + * Success consumes the resume classification (a -EAGAIN retry + * "succeeds" too: the resume did all it could against a memslot + * mid-update, and the re-posted request's consumption is a fresh + * revalidation like any other). + */ + to_vmx(vcpu)->nested.resume_pending =3D false; + return true; +} + +static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu) +{ + if (!__vmx_get_nested_state_pages(vcpu)) { + /* + * The caller (vcpu_enter_guest) consumed + * KVM_REQ_GET_NESTED_STATE_PAGES before calling in, and a + * failure here may leave a previously latched =E2=80=94 and, after + * an invalidation, stale and unreferenced =E2=80=94 physical + * address in vmcs02. Re-post the request so that no path + * can re-enter L2 without retrying this function: the + * invariant is that a stale latched address always implies + * a pending request. If the failure persists, the vCPU + * bounces to userspace on every KVM_RUN rather than ever + * entering L2, which is the invariant doing its job. + */ + kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu); + return false; + } + return true; } =20 @@ -3698,7 +3991,7 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mo= de(struct kvm_vcpu *vcpu, prepare_vmcs02_early(vmx, &vmx->vmcs01, vmcs12); =20 if (from_vmentry) { - if (unlikely(!nested_get_vmcs12_pages(vcpu))) { + if (unlikely(!nested_get_vmcs12_pages(vcpu, false))) { vmx_switch_vmcs(vcpu, &vmx->vmcs01); return NVMX_VMENTRY_KVM_INTERNAL_ERROR; } @@ -3736,6 +4029,7 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mo= de(struct kvm_vcpu *vcpu, * to nested_get_vmcs12_pages before the next VM-entry. The MSRs * have already been set at vmentry time and should not be reset. */ + vmx->nested.resume_pending =3D true; kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu); } =20 @@ -3792,13 +4086,19 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_= mode(struct kvm_vcpu *vcpu, vmentry_fail_vmexit: vmx_switch_vmcs(vcpu, &vmx->vmcs01); =20 + /* + * Release any guest-mode pins taken by nested_get_vmcs12_pages() + * during this failed entry: this path returns to L1 without going + * through __nested_vmx_vmexit(), which is where they are normally + * dropped. (Idempotent if a later vmexit drops them again.) + */ + nested_put_vmcs12_pages(vcpu); + if (!from_vmentry) { nested_ept_uninit_mmu_context(vcpu); return NVMX_VMENTRY_VMEXIT; } =20 - nested_put_vmcs12_pages(vcpu); - load_vmcs12_host_state(vcpu, vmcs12); vmcs12->vm_exit_reason =3D exit_reason.full; if (enable_shadow_vmcs || nested_vmx_is_evmptr12_valid(vmx)) @@ -4054,6 +4354,8 @@ static void vmcs12_save_pending_event(struct kvm_vcpu= *vcpu, static int vmx_complete_nested_posted_interrupt(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); + int pi_idx, vapic_idx; + struct pi_desc *pi_desc; int max_irr; void *vapic_page; u16 status; @@ -4061,22 +4363,58 @@ static int vmx_complete_nested_posted_interrupt(str= uct kvm_vcpu *vcpu) if (!vmx->nested.pi_pending) return 0; =20 - if (!vmx->nested.pi_desc) + /* + * Resolve both caches before touching the descriptor: taking one + * cache's lock inside the other's would sleep in refresh (and + * wait a grace period on the gpc_srcu domain this task already + * reads) with preemption disabled. The vAPIC lock may go unused + * (no pending vector), and its absence only matters once a + * vector is found, so a failure is remembered, not acted on. + */ +retry: + vapic_idx =3D nested_gpc_lock_if_active(&vmx->nested.virtual_apic_cache); + + pi_idx =3D nested_gpc_try_lock_if_active(&vmx->nested.pi_desc_cache); + if (pi_idx =3D=3D -EWOULDBLOCK) { + if (vapic_idx >=3D 0) + kvm_gpc_unlock(&vmx->nested.virtual_apic_cache, + vapic_idx); + if (kvm_gpc_refresh(&vmx->nested.pi_desc_cache, PAGE_SIZE)) + goto mmio_needed; + goto retry; + } + if (pi_idx < 0) { + if (vapic_idx >=3D 0) + kvm_gpc_unlock(&vmx->nested.virtual_apic_cache, + vapic_idx); goto mmio_needed; + } =20 + pi_desc =3D nested_pi_desc(vmx); vmx->nested.pi_pending =3D false; =20 - if (!pi_test_and_clear_on(vmx->nested.pi_desc)) + if (!pi_test_and_clear_on(pi_desc)) { + kvm_gpc_unlock(&vmx->nested.pi_desc_cache, pi_idx); + if (vapic_idx >=3D 0) + kvm_gpc_unlock(&vmx->nested.virtual_apic_cache, + vapic_idx); return 0; + } =20 - max_irr =3D pi_find_highest_vector(vmx->nested.pi_desc); + max_irr =3D pi_find_highest_vector(pi_desc); if (max_irr > 0) { - vapic_page =3D vmx->nested.virtual_apic_map.hva; - if (!vapic_page) + if (vapic_idx < 0) { + /* The ON bit is already cleared: mark the write. */ + kvm_gpc_mark_dirty_in_slot(&vmx->nested.pi_desc_cache); + kvm_gpc_unlock(&vmx->nested.pi_desc_cache, pi_idx); goto mmio_needed; + } + vapic_page =3D vmx->nested.virtual_apic_cache.khva; + + __kvm_apic_update_irr(pi_desc->pir, vapic_page, &max_irr); + + kvm_gpc_mark_dirty_in_slot(&vmx->nested.virtual_apic_cache); =20 - __kvm_apic_update_irr(vmx->nested.pi_desc->pir, - vapic_page, &max_irr); status =3D vmcs_read16(GUEST_INTR_STATUS); if ((u8)max_irr > ((u8)status & 0xff)) { status &=3D ~0xff; @@ -4085,8 +4423,10 @@ static int vmx_complete_nested_posted_interrupt(stru= ct kvm_vcpu *vcpu) } } =20 - kvm_vcpu_map_mark_dirty(vcpu, &vmx->nested.virtual_apic_map); - kvm_vcpu_map_mark_dirty(vcpu, &vmx->nested.pi_desc_map); + if (vapic_idx >=3D 0) + kvm_gpc_unlock(&vmx->nested.virtual_apic_cache, vapic_idx); + kvm_gpc_mark_dirty_in_slot(&vmx->nested.pi_desc_cache); + kvm_gpc_unlock(&vmx->nested.pi_desc_cache, pi_idx); return 0; =20 mmio_needed: @@ -4205,8 +4545,9 @@ static bool nested_vmx_preemption_timer_pending(struc= t kvm_vcpu *vcpu) static bool vmx_has_nested_events(struct kvm_vcpu *vcpu, bool for_injectio= n) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); - void *vapic =3D vmx->nested.virtual_apic_map.hva; - int max_irr, vppr; + int max_irr, vppr, idx; + struct pi_desc *pi_desc; + void *vapic; =20 if (nested_vmx_preemption_timer_pending(vcpu) || vmx->nested.mtf_pending) @@ -4225,20 +4566,57 @@ static bool vmx_has_nested_events(struct kvm_vcpu *= vcpu, bool for_injection) __vmx_interrupt_blocked(vcpu)) return false; =20 - if (!vapic) + /* + * This is called from non-sleeping contexts (notably the + * kvm_vcpu_block() loop, under set_current_state()), so it must + * not refresh an unusable cache. Claim a pending event instead, + * and post KVM_REQ_GET_NESTED_STATE_PAGES: the bounce through + * vcpu_run services the request in a sleepable context, + * re-establishing the cache, and re-evaluates events for real. + * The request must be posted here rather than assumed pending =E2=80=94 + * an invalidation posts one, but a cache can also fail the check + * on a bare memslot-generation bump (e.g. a dirty-logging toggle, + * KVM_MR_FLAGS_ONLY), where nothing else does, and without it + * the woken vCPU would re-enter, HLT, and spin back here. A + * spurious wakeup is safe; sleeping here is not, and returning + * false could miss a wakeup. + */ + idx =3D nested_gpc_try_lock_if_active(&vmx->nested.virtual_apic_cache); + if (idx =3D=3D -EWOULDBLOCK) { + kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu); + return true; + } + if (idx < 0) return false; =20 + vapic =3D vmx->nested.virtual_apic_cache.khva; vppr =3D *((u32 *)(vapic + APIC_PROCPRI)); =20 + kvm_gpc_unlock(&vmx->nested.virtual_apic_cache, idx); + max_irr =3D vmx_get_rvi(); if ((max_irr & 0xf0) > (vppr & 0xf0)) return true; =20 - if (vmx->nested.pi_pending && vmx->nested.pi_desc && - pi_test_on(vmx->nested.pi_desc)) { - max_irr =3D pi_find_highest_vector(vmx->nested.pi_desc); - if (max_irr > 0 && (max_irr & 0xf0) > (vppr & 0xf0)) + if (vmx->nested.pi_pending) { + idx =3D nested_gpc_try_lock_if_active(&vmx->nested.pi_desc_cache); + if (idx =3D=3D -EWOULDBLOCK) { + kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu); return true; + } + if (idx < 0) + return false; + + pi_desc =3D nested_pi_desc(vmx); + if (pi_test_on(pi_desc)) { + max_irr =3D pi_find_highest_vector(pi_desc); + if (max_irr > 0 && (max_irr & 0xf0) > (vppr & 0xf0)) { + kvm_gpc_unlock(&vmx->nested.pi_desc_cache, idx); + return true; + } + } + + kvm_gpc_unlock(&vmx->nested.pi_desc_cache, idx); } =20 return false; @@ -5121,6 +5499,15 @@ void __nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 = vm_exit_reason, #endif } =20 + /* + * The resume classification dies with the request: a stale + * resume_pending would give the next request (e.g. posted by an + * invalidation after L1 re-enters L2) entry-time semantics, + * re-deriving PDPTRs mid-L2 and treating a transiently + * unbackable page as a terminal error. + */ + vmx->nested.resume_pending =3D false; + /* Service pending TLB flush requests for L2 before switching to L1. */ kvm_service_local_tlb_flush_requests(vcpu); =20 diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 590095bcb665..bb766380276e 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6428,11 +6428,14 @@ static void vmx_flush_pml_buffer(struct kvm_vcpu *v= cpu) =20 static void nested_vmx_mark_all_vmcs12_pages_dirty(struct kvm_vcpu *vcpu) { - struct vcpu_vmx *vmx =3D to_vmx(vcpu); + struct vmcs12 *vmcs12 =3D get_vmcs12(vcpu); =20 - kvm_vcpu_map_mark_dirty(vcpu, &vmx->nested.apic_access_page_map); - kvm_vcpu_map_mark_dirty(vcpu, &vmx->nested.virtual_apic_map); - kvm_vcpu_map_mark_dirty(vcpu, &vmx->nested.pi_desc_map); + if (nested_cpu_has2(vmcs12, SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES)) + kvm_vcpu_mark_page_dirty(vcpu, gpa_to_gfn(vmcs12->apic_access_addr)); + if (nested_cpu_has(vmcs12, CPU_BASED_TPR_SHADOW)) + kvm_vcpu_mark_page_dirty(vcpu, gpa_to_gfn(vmcs12->virtual_apic_page_addr= )); + if (nested_cpu_has_posted_intr(vmcs12)) + kvm_vcpu_mark_page_dirty(vcpu, gpa_to_gfn(vmcs12->posted_intr_desc_addr)= ); } =20 static void vmx_dump_sel(char *name, uint32_t sel) diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index c6827368c84a..7d8bced38ca0 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -73,6 +73,12 @@ struct pt_desc { struct nested_vmx { /* Has the level1 guest done vmxon? */ bool vmxon; + /* + * The pending KVM_REQ_GET_NESTED_STATE_PAGES was posted by + * KVM_SET_NESTED_STATE: an unbackable page is then a userspace + * error to report, not a transient state to ride out. + */ + bool resume_pending; gpa_t vmxon_ptr; bool pml_full; =20 @@ -146,13 +152,18 @@ struct nested_vmx { * Guest pages referred to in the vmcs02 with host-physical * pointers, so we must keep them pinned while L2 runs. */ - struct gfn_to_pfn_cache msr_bitmap_cache; + struct gfn_to_pfn_cache apic_access_page_cache; + struct gfn_to_pfn_cache virtual_apic_cache; + struct gfn_to_pfn_cache pi_desc_cache; =20 - struct kvm_host_map apic_access_page_map; - struct kvm_host_map virtual_apic_map; - struct kvm_host_map pi_desc_map; + /* + * L1's MSR bitmap, read by KVM when merging into the bitmap the + * CPU actually uses (vmcs02 points at the merged copy). Never + * pinned, never written through. + */ + struct gfn_to_pfn_cache msr_bitmap_cache; =20 - struct pi_desc *pi_desc; + u64 pi_desc_offset; bool pi_pending; u16 posted_intr_nv; =20 --=20 2.43.0 From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.50.34 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844071; b=V7u3W/wzl+JPyaf/kLbnXqqpxyKJnaWOHIuS3ea3IFf7u3AdnE2tmUWsdzdf3A3p7sz2 HAYx/RLiqzRTdygiZQyF8cD0lvF08FCgP702HRjRzHARj5ClfDSI0SJIvG2IwMg2xstBC oXg96HZ4Ti56Sktd5m/+Zl4xFTL251LRNb1IyFRrajigVp14FexZeAVKIBhv7eL47ERdr jcoV3BXe+Wvgi0GTkD1YKRz33Tu/gafbpXr1AkvggSe7dg/QLxcGfyuH7y1pPDijmENTE sDBLjPXuZ0Ug1uChjZmM0R3qLzkiBZlc0deugk5FfCHd8ly0IAxXjpPvr862+UkSUPA== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844071; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Type:Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=mjNES8BIaoIyrAOmFi0HcJENXzorOo9BmJYMd+hdINg=; b=LE8lFB5sfWzIfTbvdFukdwJkoHUz7ugy2/iTItKQ8ZmKMAMSZuoKT5cwGT/F8kYuAOrw DMGObcU3rhJJLqFQWdWl/WGcJV3g2wehKVUEHj/COHKrwvQ0qW5yh6fxx8+hKojaaGyfP baxGLBIM49SEH/R6rnnZhunqbzpcOEqK7imznQ01Y/CCgBXIxSnwEU4EAUhEMkgRcDeaw 5Ku9RA1jwdNbPxnEJgMdByAwmfSEutbVFvfLgZGzWZBWNbXMTWZ+R2h1P6ckNcTGClMVJ GpH6MnUbPz2swkk4Kh6T9FfxxkHW2meGDUo2eB6T7uhyTguOyxPVDQihopSVLlF2xuw== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=casper.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=Y+4ZLfxu; arc=none smtp.remote-ip=90.155.50.34 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAA3A606A32F; Thu, 5 Nov 2026 02:01:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org DAA3A606A32F Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=casper.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=Y+4ZLfxu DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description; bh=mjNES8BIaoIyrAOmFi0HcJENXzorOo9BmJYMd+hdINg=; b=Y+4ZLfxuIGwYr8Uuoj4oxXYUxI 2oJ32ETKK9nkI2LUJkr/qCpbOZ6N38Q/VT7lXm9jthEy2qTdRxsGo0+jNuA5e9MjpQdyyECD1hwVr Mh/5PBP1h9o+Sj6hZwUGFghzge3H8qjmvMz+H5Rt9myJGs0Bv6pDIRmeeQtlqbyqTuj9thTui/dMH NThFTvPHocdsHhSKoxEdpkYXa8NQj22AuxzZHD9SQEVAdXYsW94n2mox8K0UQAcFMPZDmbeSWeUHN oUjKDK/22Pc+dB081JaC6QQZj0iiviUUz0Yo7zxM+akDVn1MDCyNA0MEj/gx0lO1++5nvqaR7xGVQ aEYmqtXA==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by casper.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmn-000000037jH-12gf; Thu, 05 Nov 2026 02:01:05 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmm-0000000AWDq-2wrs; Thu, 05 Nov 2026 02:01:04 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 13/20] KVM: x86: Post KVM_REQ_GET_NESTED_STATE_PAGES on memslot updates Date: Thu, 5 Nov 2026 02:00:55 +0000 Message-ID: <20261105020104.2507083-14-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html From: David Woodhouse The physical addresses which nested page setup latches into hardware control structures (vmcs02's APIC-access, virtual-APIC and posted interrupt descriptor addresses) are derived from gPA=E2=86=92uHVA translations which a memslot update can change. Software readers of the underlying gfn_to_pfn_caches catch that lazily, via the memslot generation check at their next use =E2=80=94 but the CPU's use of a latched address from guest mode is continuous and checks nothing. A vCPU running L2 across a memslot move keeps using the old translation until a nested VM-exit happens to re-run nested page setup. (A staleness problem, not a lifetime one: freeing the page is the mmu_notifier's business. The replaced kvm_host_map code had the same staleness with no remedy at all.) Post the request from the memslot update itself =E2=80=94 the single point where the generation changes, and a slow path by definition =E2=80=94 rather than checking generations in the VM-entry path forever. The request is what survives racing with a concurrent VM-entry (a bare kick landing before vcpu->mode is set would be lost), and a vCPU which resolved its pages against the old memslots but has not yet entered guest mode is invisible to any is_guest_mode() filter, so it is posted unconditionally to every vCPU. TD vCPUs share vmx_nested_ops but are not struct vcpu_vmx; the VMX handler bails for them up front. The posting site is kvm_arch_commit_memory_region(), once per user-visible update with the final memslots in place =E2=80=94 not kvm_arch_memslots_updated(), which also runs when a DELETE or MOVE installs its transient KVM_MEMSLOT_INVALID placeholder, where the handler could only retry, and skipping FLAGS_ONLY updates, which cannot change any translation. kvm_arch_memslots_updated() keeps a bare kick for the steal_time cache's check-at-use sites. The handlers distinguish the request's origin, extending VMX's nested.resume_pending to nSVM. A resume into L2 from saved state (KVM_SET_NESTED_STATE after migration, or RSM) re-derives PAE PDPTRs and re-runs the MSRPM merge; a failure there is a userspace error (or, in a narrow window, a concurrent memslot update's transient KVM_MEMSLOT_INVALID placeholder), reported as KVM_EXIT_INTERNAL_ERROR with the request re-posted and resume_pending kept, so a further KVM_RUN bounces back out rather than entering L2 with the restore work undone =E2=80=94 and so a userspace which retries recovers once a transient failure clears. A memslot-posted request re-latches the vmcs02 ad= dresses on VMX and is a no-op on SVM, which latches no L1-owned physical address in hardware (the merged MSRPM and IOPM are kernel pages); neither vendor re-derives PDPTRs for it, as those latch only at MOV CR3 and VM-entry. The classification dies with the request: nested VM-exit clears resume_pending on both vendors, so a request posted after L1 re-enters L2 cannot inherit entry-time semantics. Accordingly, downgrade the WARN in svm_get_nested_state_pages(): a request with nothing to do is now expected. The memslot-move mode of the vmx_apic_update_test selftest, added later in this series, exercises this path. Signed-off-by: David Woodhouse --- arch/x86/kvm/svm/nested.c | 45 ++++++++++++++++++++++++++++++++++----- arch/x86/kvm/svm/svm.h | 12 +++++++++++ arch/x86/kvm/vmx/nested.c | 12 +++++++++-- arch/x86/kvm/x86.c | 44 +++++++++++++++++++++++++++++++++++++- 4 files changed, 105 insertions(+), 8 deletions(-) diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index af0ef678433d..ab187abc11b5 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -1078,8 +1078,10 @@ int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 = vmcb12_gpa, bool from_vmrun) if (ret) return ret; =20 - if (!from_vmrun) + if (!from_vmrun) { + svm->nested.resume_pending =3D true; kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu); + } =20 svm_set_gif(svm, true); =20 @@ -1340,6 +1342,7 @@ void nested_svm_vmexit(struct vcpu_svm *svm) kvm_warn_on_nested_run_pending(vcpu); =20 kvm_clear_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu); + svm->nested.resume_pending =3D false; =20 /* in case we halted in L2 */ kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE); @@ -1568,6 +1571,7 @@ void svm_leave_nested(struct kvm_vcpu *vcpu) } =20 kvm_clear_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu); + svm->nested.resume_pending =3D false; } =20 static int nested_svm_exit_handled_msr(struct vcpu_svm *svm) @@ -2113,6 +2117,7 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu, if (kvm_vcpu_apicv_active(vcpu)) kvm_make_request(KVM_REQ_APICV_UPDATE, vcpu); =20 + svm->nested.resume_pending =3D true; kvm_make_request(KVM_REQ_GET_NESTED_STATE_PAGES, vcpu); out_free: kfree(save); @@ -2123,7 +2128,21 @@ static int svm_set_nested_state(struct kvm_vcpu *vcp= u, =20 static bool svm_get_nested_state_pages(struct kvm_vcpu *vcpu) { - if (WARN_ON(!is_guest_mode(vcpu))) + struct vcpu_svm *svm =3D to_svm(vcpu); + + /* + * Memslot updates post this request to every vCPU (to make any + * vCPU which has guest pages latched in hardware re-resolve them + * against the new memslots), so it can arrive with nothing to + * do: outside guest mode, or (unlike VMX) even with L2 + * running, because SVM latches no L1-owned physical address in + * hardware (the merged MSRPM and the IOPM are kernel pages). + * The restore work below (PDPTR re-derivation, MSRPM merge) + * runs only when the request came from a migration restore; + * running it for a memslot update would re-latch a live L2's + * PDPTRs at a moment the architecture does not allow. + */ + if (!is_guest_mode(vcpu) || !svm->nested.resume_pending) return true; =20 if (is_pae_paging(vcpu)) { @@ -2139,16 +2158,32 @@ static bool svm_get_nested_state_pages(struct kvm_v= cpu *vcpu) kvm_register_mark_for_reload(vcpu, VCPU_REG_PDPTR); else if (!vcpu->arch.pdptrs_from_userspace && CC(!load_pdptrs(vcpu, vcpu->arch.cr3))) - return false; + goto internal_error; } =20 if (!nested_svm_merge_msrpm(vcpu)) - return false; + goto internal_error; =20 if (kvm_hv_verify_vp_assist(vcpu)) - return false; + goto internal_error; =20 + /* Success consumes the resume classification. */ + svm->nested.resume_pending =3D false; return true; + +internal_error: + /* + * Restored state names an unusable page. (In a narrow window, a + * concurrent memslot update's KVM_MEMSLOT_INVALID placeholder + * lands here too; a userspace which retries KVM_RUN recovers + * once the update commits.) The caller fills in the + * KVM_EXIT_INTERNAL_ERROR exit and re-posts the request; what is + * SVM's to keep is resume_pending, still set, so that no + * subsequent KVM_RUN can enter L2 with the restore work undone: + * the vCPU bounces back here (and to userspace) until the + * failure is resolved or the vCPU state is replaced. + */ + return false; } =20 static gpa_t svm_translate_nested_gpa(struct kvm_vcpu *vcpu, gpa_t gpa, diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index 6d0242966dc8..1b8f3f2057d9 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -236,6 +236,18 @@ struct svm_nested_state { =20 bool initialized; =20 + /* + * The pending KVM_REQ_GET_NESTED_STATE_PAGES was posted by a + * migration restore (KVM_SET_NESTED_STATE or RSM back into L2), + * where PDPTRs must be re-derived and the MSRPM merge re-run + * before re-entering L2, and a failure is a userspace error to + * report. Unlike VMX, SVM latches no L1-owned physical address + * in hardware (the merged MSRPM and the IOPM are kernel pages), + * so a request posted by a memslot update finds nothing to + * re-latch and the handler no-ops on !resume_pending. + */ + bool resume_pending; + /* * Indicates whether MSR bitmap for L2 needs to be rebuilt due to * changes in MSR bitmap for L1 or switching to a different L2. Note, diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 23f1fff97917..6c8004009bcd 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -3846,6 +3846,14 @@ static bool __vmx_get_nested_state_pages(struct kvm_= vcpu *vcpu) =20 static bool vmx_get_nested_state_pages(struct kvm_vcpu *vcpu) { + /* + * Memslot updates post this request to every vCPU, including TD + * vCPUs, which are not struct vcpu_vmx and have no nested state + * to re-resolve. + */ + if (is_td_vcpu(vcpu)) + return true; + if (!__vmx_get_nested_state_pages(vcpu)) { /* * The caller (vcpu_enter_guest) consumed @@ -5501,8 +5509,8 @@ void __nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 v= m_exit_reason, =20 /* * The resume classification dies with the request: a stale - * resume_pending would give the next request (e.g. posted by an - * invalidation after L1 re-enters L2) entry-time semantics, + * resume_pending would give the next request (e.g. posted by a + * memslot update after L1 re-enters L2) entry-time semantics, * re-deriving PDPTRs mid-L2 and treating a transiently * unbackable page as a terminal error. */ diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 321f29789f9f..892c6f81d704 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -10282,7 +10282,16 @@ void kvm_arch_memslots_updated(struct kvm *kvm, u6= 4 gen) */ kvm_mmu_invalidate_mmio_sptes(kvm, gen); =20 - /* Force re-initialization of steal_time cache */ + /* + * Force re-initialization of the steal_time cache: its + * check-at-use sites only see the new generation once the vCPU + * goes around its run loop. This runs on both memslot swaps of a + * DELETE/MOVE; the re-resolution of latched nested-state pages + * is posted from kvm_arch_commit_memory_region() instead, once + * per update and only when the final memslots are in place, so + * that its handler never chases an in-flight update's + * KVM_MEMSLOT_INVALID placeholder. + */ kvm_for_each_vcpu(i, vcpu, kvm) kvm_vcpu_kick(vcpu); } @@ -10452,6 +10461,39 @@ void kvm_arch_commit_memory_region(struct kvm *kvm, const struct kvm_memory_slot *new, enum kvm_mr_change change) { + /* + * Force re-resolution of any nested-state pages whose physical + * addresses a vCPU has latched in hardware control structures + * (e.g. vmcs02) from a gfn_to_pfn_cache. Software readers of + * such caches catch the generation bump lazily, via + * kvm_gpc_check() at their next use; the CPU's use from guest + * mode is continuous and checks nothing, so the vCPU must be + * told to re-resolve and re-latch before it next enters the + * guest. The request is the artifact that survives racing with + * a concurrent VM-entry (a bare kick landing before vcpu->mode + * is set would be lost); its handler re-runs nested page setup, + * whose gPA lookups then see the new generation and refresh. + * + * This is posted from commit, not from kvm_arch_memslots_updated(), + * so it runs once per user-visible update with the final memslots + * in place: a request serviced during a DELETE/MOVE's + * invalidation phase would find the KVM_MEMSLOT_INVALID + * placeholder and could only retry. A FLAGS_ONLY update (e.g. a + * dirty-log toggle) cannot change any translation =E2=80=94 base_gfn and + * userspace_addr are immutable =E2=80=94 so it posts nothing. + * + * The kick is deliberately fire-and-forget (no KVM_REQUEST_WAIT): + * a vCPU still in L2 may keep using the old translation until it + * services the request, which matches the staleness semantics the + * replaced kvm_host_map code had, unboundedly. Lifetime is not at + * stake =E2=80=94 the underlying page is freed only via the mmu_notifier, + * whose invalidation forces pinned vCPUs out of guest mode and + * waits. Making the ioctl wait here would buy only a tighter + * staleness bound, at a vCPU-roundtrip cost on every update. + */ + if (change !=3D KVM_MR_FLAGS_ONLY) + kvm_make_all_cpus_request(kvm, KVM_REQ_GET_NESTED_STATE_PAGES); + if (change =3D=3D KVM_MR_DELETE) kvm_page_track_delete_slot(kvm, old); =20 --=20 2.43.0 From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.92.199 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844079; b=I4UB3g2jdB7LkkVSyOUBQc9A6IRVFJ9TQjZpFwhsGj0fmcy0RTx8ui4JUECEOgDkYvOQ lOpDcW1bIEb48x2Ap9RhY9lfG4LdTfdrPUColSJ+hOro53gA6RywIylURa4gOoyAucJVM M369bSsgJfq082DT/gXGAZ6iQN9a+NjgW+omAdof0P/gbq6r72fs7EQBK/o1Yp4FPusna M0WA36YUxDyKxAFEXRm0JH+kt+ZNrVogZ6Itt+WhVAZBEkGzfODiJ2ffAw+p9VBjNDtTi m+8JM0pGYzS6uMUWpv7+Dl9owMSVsNq1N9Gcyq/JIsMN9O2SSqBLk493EaeAoHO+IQA== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844079; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=MrykgqblsgkTBBhZT90jYeBND9vd1pcgeJZVOE7ReYk=; b=nMx6o1clV6Eu4rSRLqzOLH7XDyfVORJsoWlnW9MMpZ1k5w4XnO3SrMEK/m3rmX2r8wW+ XFdd2w/gUiE1U+OnpnnMyVnB6YzxSMgsBrVHbeDkBK9q6vd9i/dgVwcY5aqw4X/79CJbv mHY+Hkc/ThKz6yqxGGG/iBnZVRuVUs2iWZqwKjdYLU0/Cd/srpOHWxHMmbAgUcZ3XE+SR jojc220tw64CNzdu2ec2VqCtOiIULm2RnHPeGdUjQhDbY+Mp4ndAt0eEwbmlkBql8jR3V TEZRaofQvgzI5YepMLzB7O5yWOYnL0tkTqIQ/zWbTDhUsgu0Or9G0PsEGJ5cLrWsgZA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=desiato.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=desiato.20200630 header.b=E89PU2Po; arc=none smtp.remote-ip=90.155.92.199 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0EE16606EEA7; Thu, 5 Nov 2026 02:01:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 0EE16606EEA7 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=desiato.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=desiato.20200630 header.b=E89PU2Po DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=MrykgqblsgkTBBhZT90jYeBND9vd1pcgeJZVOE7ReYk=; b=E89PU2Po5NQYe57K5iPKPyxkd6 EIFv08iZkTq/YiSNs1ncnEO5K7DjkjDC3IC3NUTXGYeevXC0fQe4r9thR/oKu5IlQI0WlB9Qya8Te c7HrdPPF7zjoVGaa1EdF2FXEIHGmKdWODQsWFXfDNJTHa9r7nGsIGk7ZzsJEd0OxHB05SGUOKuNY2 aO4pJCl+2mJm5TNBhuBw3aGVvpsgp8T6pvG6ZV3zOsLyZd35My2L1XELY+FzEO+30p1n9YNHMqTGp IuqlM18MEWRI7Dxy7omuWMw4q7MP3LDMfJDqlmEq/N1IFs6dMk4FApecH+BxctBp15fmddZD8tWGx BKPpjO/A==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by desiato.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmo-00000002ROP-25S4; Thu, 05 Nov 2026 02:01:08 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmm-0000000AWDu-3GYz; Thu, 05 Nov 2026 02:01:04 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 14/20] KVM: selftests: Add nested VMX APIC cache invalidation test Date: Thu, 5 Nov 2026 02:00:56 +0000 Message-ID: <20261105020104.2507083-15-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by desiato.infradead.org. See http://www.infradead.org/rpr.html Content-Type: text/plain; charset="utf-8" From: Fred Griffoul Add a selftest for the nested VMX APIC page caches: an L2 guest writes the APIC-access page while a second thread repeatedly invalidates the underlying mapping, either with madvise(MADV_DONTNEED) (MMU notifier path) or by moving the memslot away and back (memslot update path), so the GPA under test is repeatedly unbacked and re-latched. The test asserts that L2 keeps running across the invalidations with no guest-visible errors, that at least one invalidation actually ran concurrently, and that the access only ever bounces to userspace as MMIO in the memslot-move variant (while the slot is moved away the GPA is unbacked; madvise never unbacks the GPA, so an MMIO exit there means KVM lost the mapping). Signed-off-by: Fred Griffoul [dwmw2: Adapt to selftest API drift since the v4 posting: prepare_vmcs() no longer takes an L2 stack (4c396f19de5d), vm_vaddr_t is now gva_t, and the prepare_eptp()/nested_map*() EPT helpers were replaced by vm_enable_tdp()/tdp_map()/ tdp_identity_map_default_memslots().] Signed-off-by: David Woodhouse Assisted-by: LLM --- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../selftests/kvm/x86/vmx_apic_update_test.c | 323 ++++++++++++++++++ 2 files changed, 324 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86/vmx_apic_update_test.c diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selft= ests/kvm/Makefile.kvm index 17a4256e9081..dcfe815f4a02 100644 --- a/tools/testing/selftests/kvm/Makefile.kvm +++ b/tools/testing/selftests/kvm/Makefile.kvm @@ -161,6 +161,7 @@ TEST_GEN_PROGS_x86 +=3D x86/triple_fault_event_test TEST_GEN_PROGS_x86 +=3D x86/recalc_apic_map_test TEST_GEN_PROGS_x86 +=3D x86/aperfmperf_test TEST_GEN_PROGS_x86 +=3D x86/guest_memfd_conversions_test +TEST_GEN_PROGS_x86 +=3D x86/vmx_apic_update_test TEST_GEN_PROGS_x86 +=3D access_tracking_perf_test TEST_GEN_PROGS_x86 +=3D coalesced_io_test TEST_GEN_PROGS_x86 +=3D dirty_log_perf_test diff --git a/tools/testing/selftests/kvm/x86/vmx_apic_update_test.c b/tools= /testing/selftests/kvm/x86/vmx_apic_update_test.c new file mode 100644 index 000000000000..357e10c88cd0 --- /dev/null +++ b/tools/testing/selftests/kvm/x86/vmx_apic_update_test.c @@ -0,0 +1,323 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * vmx_apic_update_test + * + * Copyright (C) 2025, Amazon.com, Inc. or its affiliates. All Rights Rese= rved. + * + * Test L2 guest APIC access page writes with concurrent MMU + * notification and memslot move updates. + */ +#include +#include +#include "test_util.h" +#include "kvm_util.h" +#include "processor.h" +#include "vmx.h" + +#define VAPIC_GPA 0xc0000000 +#define VAPIC_SLOT 1 + + +#define L2_DELAY (100) + +static void l2_guest_code(void) +{ + uint32_t *vapic_addr =3D (uint32_t *) (VAPIC_GPA + 0x80); + + /* Unroll the loop to avoid any compiler side effect */ + + WRITE_ONCE(*vapic_addr, 1 << 0); + udelay(msecs_to_usecs(L2_DELAY)); + + WRITE_ONCE(*vapic_addr, 1 << 1); + udelay(msecs_to_usecs(L2_DELAY)); + + WRITE_ONCE(*vapic_addr, 1 << 2); + udelay(msecs_to_usecs(L2_DELAY)); + + WRITE_ONCE(*vapic_addr, 1 << 3); + udelay(msecs_to_usecs(L2_DELAY)); + + WRITE_ONCE(*vapic_addr, 1 << 4); + udelay(msecs_to_usecs(L2_DELAY)); + + WRITE_ONCE(*vapic_addr, 1 << 5); + udelay(msecs_to_usecs(L2_DELAY)); + + WRITE_ONCE(*vapic_addr, 1 << 6); + udelay(msecs_to_usecs(L2_DELAY)); + + WRITE_ONCE(*vapic_addr, 0); + udelay(msecs_to_usecs(L2_DELAY)); + + /* Exit to L1 */ + vmcall(); +} + +static void l1_guest_code(struct vmx_pages *vmx_pages) +{ + uint32_t control, exit_reason; + + prepare_for_vmx_operation(vmx_pages); + load_vmcs(vmx_pages); + prepare_vmcs(vmx_pages, l2_guest_code); + + /* Enable APIC access */ + control =3D vmread(CPU_BASED_VM_EXEC_CONTROL); + control |=3D CPU_BASED_ACTIVATE_SECONDARY_CONTROLS; + vmwrite(CPU_BASED_VM_EXEC_CONTROL, control); + control =3D vmread(SECONDARY_VM_EXEC_CONTROL); + control |=3D SECONDARY_EXEC_VIRTUALIZE_APIC_ACCESSES; + vmwrite(SECONDARY_VM_EXEC_CONTROL, control); + vmwrite(APIC_ACCESS_ADDR, VAPIC_GPA); + + GUEST_SYNC1(0); + vmlaunch(); +again: + exit_reason =3D vmread(VM_EXIT_REASON); + if (exit_reason =3D=3D EXIT_REASON_APIC_ACCESS) { + uint64_t guest_rip =3D vmread(GUEST_RIP); + uint64_t instr_len =3D vmread(VM_EXIT_INSTRUCTION_LEN); + + vmwrite(GUEST_RIP, guest_rip + instr_len); + vmresume(); + goto again; + } + + GUEST_SYNC1(exit_reason); + GUEST_ASSERT(exit_reason =3D=3D EXIT_REASON_VMCALL); + GUEST_DONE(); +} + +static const char *progname; +static int update_period_ms =3D L2_DELAY / 4; + +struct update_control { + pthread_mutex_t mutex; + pthread_cond_t start_cond; + struct kvm_vm *vm; + bool running; + bool started; + int updates; +}; + +static void wait_for_start_signal(struct update_control *ctrl) +{ + pthread_mutex_lock(&ctrl->mutex); + while (!ctrl->started) + pthread_cond_wait(&ctrl->start_cond, &ctrl->mutex); + + pthread_mutex_unlock(&ctrl->mutex); + printf("%s: starting update\n", progname); +} + +static bool is_running(struct update_control *ctrl) +{ + return READ_ONCE(ctrl->running); +} + +static void set_running(struct update_control *ctrl, bool running) +{ + WRITE_ONCE(ctrl->running, running); +} + +static void signal_thread_start(struct update_control *ctrl) +{ + pthread_mutex_lock(&ctrl->mutex); + if (!ctrl->started) { + ctrl->started =3D true; + pthread_cond_signal(&ctrl->start_cond); + } + pthread_mutex_unlock(&ctrl->mutex); +} + +static void *update_madvise(void *arg) +{ + struct update_control *ctrl =3D arg; + void *hva; + + wait_for_start_signal(ctrl); + + hva =3D addr_gpa2hva(ctrl->vm, VAPIC_GPA); + memset(hva, 0x45, ctrl->vm->page_size); + + while (is_running(ctrl)) { + usleep(update_period_ms * 1000); + madvise(hva, ctrl->vm->page_size, MADV_DONTNEED); + ctrl->updates++; + } + + return NULL; +} + +static void *update_move_memslot(void *arg) +{ + struct update_control *ctrl =3D arg; + uint64_t gpa =3D VAPIC_GPA; + + wait_for_start_signal(ctrl); + + /* + * Alternate the slot between VAPIC_GPA and an alternate address, + * so that the GPA under test is repeatedly unbacked (L2's APIC + * access bounces to userspace as MMIO) and re-backed (KVM must + * re-latch the page). Moving strictly away would change + * vCPU-visible state only once. + */ + while (is_running(ctrl)) { + usleep(update_period_ms * 1000); + gpa =3D (gpa =3D=3D VAPIC_GPA) ? VAPIC_GPA + 0x10000 : VAPIC_GPA; + vm_mem_region_move(ctrl->vm, VAPIC_SLOT, gpa); + ctrl->updates++; + } + + return NULL; +} + +static void run(void * (*update)(void *), const char *name, bool mmio_ok) +{ + struct kvm_vm *vm; + struct kvm_vcpu *vcpu; + struct update_control ctrl; + struct ucall uc; + gva_t vmx_pages_gva; + pthread_t update_thread; + bool done =3D false; + + vm =3D vm_create_with_one_vcpu(&vcpu, l1_guest_code); + + /* Set up nested EPT */ + vm_enable_tdp(vm); + + /* Allocate VMX pages */ + vcpu_alloc_vmx(vm, &vmx_pages_gva); + + /* Allocate memory and create VAPIC memslot */ + vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, VAPIC_GPA, + VAPIC_SLOT, 1, 0); + + /* Allocate guest page table */ + virt_map(vm, VAPIC_GPA, VAPIC_GPA, 1); + + /* Identity-map everything, including the VAPIC page, in the EPT */ + tdp_identity_map_default_memslots(vm); + tdp_map(vm, VAPIC_GPA, VAPIC_GPA, vm->page_size); + + vcpu_args_set(vcpu, 1, vmx_pages_gva); + + pthread_mutex_init(&ctrl.mutex, NULL); + pthread_cond_init(&ctrl.start_cond, NULL); + ctrl.vm =3D vm; + ctrl.running =3D true; + ctrl.started =3D false; + ctrl.updates =3D 0; + + kvm_pthread_create(&update_thread, NULL, update, &ctrl); + + printf("%s: running %s (tsc_khz %lu)\n", progname, name, guest_tsc_khz); + + while (!done) { + vcpu_run(vcpu); + + switch (vcpu->run->exit_reason) { + case KVM_EXIT_IO: + switch (get_ucall(vcpu, &uc)) { + case UCALL_SYNC: + printf("%s: sync(%ld)\n", progname, uc.args[0]); + if (uc.args[0] =3D=3D 0) + signal_thread_start(&ctrl); + break; + case UCALL_ABORT: + REPORT_GUEST_ASSERT(uc); + /* NOT REACHED */ + case UCALL_DONE: + done =3D true; + break; + default: + TEST_ASSERT(false, "Unknown ucall %lu", uc.cmd); + } + break; + case KVM_EXIT_MMIO: + /* + * While the slot is moved away, the GPA under test + * is unbacked and the access legitimately bounces + * to userspace as MMIO (until the updater moves it + * back). madvise() never unbacks the GPA, so there + * an MMIO exit means KVM lost the mapping: fail. + */ + TEST_ASSERT(mmio_ok, + "%s: unexpected APIC MMIO access at 0x%llx", + progname, vcpu->run->mmio.phys_addr); + printf + ("%s: APIC MMIO access at 0x%llx (memslot move effect)\n", + progname, vcpu->run->mmio.phys_addr); + break; + default: + TEST_FAIL("%s: Unexpected exit reason: %d (flags 0x%x)", + progname, + vcpu->run->exit_reason, vcpu->run->flags); + } + } + + set_running(&ctrl, false); + if (!ctrl.started) + signal_thread_start(&ctrl); + kvm_pthread_join(update_thread, NULL); + printf("%s: completed with %d updates\n", progname, ctrl.updates); + TEST_ASSERT(ctrl.updates > 0, + "%s: %s performed no updates; nothing was tested", + progname, name); + + pthread_mutex_destroy(&ctrl.mutex); + pthread_cond_destroy(&ctrl.start_cond); + kvm_vm_free(vm); +} + +int main(int argc, char *argv[]) +{ + int opt_madvise =3D 0; + int opt_memslot_move =3D 0; + + TEST_REQUIRE(kvm_cpu_has(X86_FEATURE_VMX)); + TEST_REQUIRE(kvm_cpu_has_ept()); + + if (argc =3D=3D 1) { + opt_madvise =3D 1; + opt_memslot_move =3D 1; + } else { + int opt; + + while ((opt =3D getopt(argc, argv, "amp:")) !=3D -1) { + switch (opt) { + case 'a': + opt_madvise =3D 1; + break; + case 'm': + opt_memslot_move =3D 1; + break; + case 'p': + update_period_ms =3D atoi(optarg); + TEST_ASSERT(update_period_ms > 0 && + update_period_ms < INT_MAX / 1000, + "Update period must be a positive number of milliseconds, not '%s= '", + optarg); + break; + default: + exit(1); + } + } + } + + TEST_ASSERT(opt_madvise + || opt_memslot_move, "No update test configured"); + + progname =3D argv[0]; + + if (opt_madvise) + run(update_madvise, "madvise", false); + + if (opt_memslot_move) + run(update_move_memslot, "move memslot", true); + + return 0; +} --=20 2.43.0 From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.50.34 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844071; b=BnE4h1c7z1FCznGQPXdOSRFhW8LoMXpoKuCZzONIkvOXutCWlPAMwtpxcHFhC0zLHEG0 1P7fBXC7WFJNR1Ud0fZnbLcCru/R5qb+SVZb7FIfr99VuYcX3NOKuAlyh6Bu0PYs/8zfb mkYaXk1JAR/rSzJnKOrrDN1ydiKGNSIDPSJeJtUFz864nu2BVmsIg5XWpCrDhgmO5TanE 4YNd4iRTGUrj8GsAiGyltF95TfiSZIAq9+V05j+8E4fEFl7/X484g4FFOYbUgWf+I5foY mWe2UJY77qc0kjDJHMuGR6v0rrd+r67qF7y1+Jo4kUHAlQ8HtavRqApmWwUbrqzBCPQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844071; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Type:Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=SyJB48g5d3dD51rFyA0ineUHrgUKJTcQUMy2B9WJ/Ig=; b=2NkmevKPQNdZmNsZIXQqLTT0wtaPirmw7GX+u0OdRAfAmyFHZZCF5uplzIAKvXdiOxk3 RFO2uc9q/Mal3HA4Nn6AL3LBHQoom0ogjOx7xntgO5yMq/7seBEJ7k8MXiHClUyOdAg1Q da2F0YfQdXYELapapD0aWlBlQMdBd+Zpcms8VDhwxCYaGGYtMteZKnv+tRn6kI57PeF/b 4MwH2qnYpDo6s57zOBHxTpmHo0IZsUOcGwKLobL9eJoxPKJc3A7wCfpRd5FkTevjllfmD mv0CQoGruJgtB+eCm2PkHeACcXz0HiiXwBsTtRFEPRNdAEuTvBVT5FUc8F0o0DLcBdw== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=casper.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=q+UEGOur; arc=none smtp.remote-ip=90.155.50.34 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA99B606A32D; Thu, 5 Nov 2026 02:01:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org DA99B606A32D Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=casper.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=q+UEGOur DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description; bh=SyJB48g5d3dD51rFyA0ineUHrgUKJTcQUMy2B9WJ/Ig=; b=q+UEGOuravXo/K331KrtFCNxgR yGeICDKLCUqm4pZ4e7wxbFkTioNil0ej75YKEOWaR7o9JGSUiKSSw/uYJ02Gh5tf/sf59LLkzrdps CS/0Kv3SBSE0zvu+omRqtwUiUha7fHrVv7SEDHCUPXIZfPbBJ9VAuR+kyNlpy/4E5IDZnPH33qCeq rHOfz42T/bP9iPF9QLjw2Jf2Qnwp9m9G/EQs2ehl9LfbKvrOhe5kMmH2BhaIkcqVaSoF1U94dHX2/ /bs/rlx/iWVWaPr1zB18CDFWK6LSCVM1jXhL26m8ATaMahHBHzYDmnwigehz2o0/5577bD6jj1vHW SBbQIFaA==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by casper.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmn-000000037jJ-1XPR; Thu, 05 Nov 2026 02:01:05 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmm-0000000AWDy-3aRK; Thu, 05 Nov 2026 02:01:04 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 15/20] KVM: nSVM: Use a gfn_to_pfn_cache for the vmcb12 page Date: Thu, 5 Nov 2026 02:00:57 +0000 Message-ID: <20261105020104.2507083-16-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html From: David Woodhouse Replace the transient map/unmap of the vmcb12 page on every nested VMRUN (control/save copy-in) and nested VM-exit (state copy-back) with a gfn_to_pfn_cache keyed on vmcb12_gpa. With unmanaged guest memory the transient map is a memremap/memunmap cycle on every L1<->L2 transition; the cache persists the kernel mapping for as long as the gPA and its translation are unchanged. An L1 which multiplexes several L2s on one vCPU re-keys the cache on each switch, paying a refresh; the common single-L2 case pays nothing. No pinning is needed, unlike the nVMX APIC pages: SVM never hands an L1-owned physical address to the CPU in vmcb02 (the merged MSRPM and the IOPM are kernel-owned pages), so the vmcb12 page is only ever accessed by KVM itself, under the SRCU read lock, with the usual check/refresh protocol. Dirty marking moves from unconditional-at-unmap to the actual write paths: the SVM_EXIT_ERR reflection in the VMRUN consistency-check failure path, and the copy-back at nested VM-exit. The SMM enter/leave and VMLOAD/VMSAVE emulation paths keep the transient map: they are cold, and VMLOAD/VMSAVE takes an arbitrary gPA from RAX which would thrash a single-slot cache. svm_leave_smm() now validates the vmcb12 GPA it reads from SMRAM, as VMRUN does for RAX; the copy-back through the cache would otherwise fail on an unaligned address at the eventual nested VM-exit. The refused RSM still ends in TRIPLE_FAULT, but at the corrupt-SMRAM instruction itself, before any L2 state is committed, as for the other SMRAM consistency checks here. Signed-off-by: David Woodhouse --- arch/x86/kvm/svm/nested.c | 27 ++++++++++++++++++++------- arch/x86/kvm/svm/svm.c | 15 +++++++++++++++ arch/x86/kvm/svm/svm.h | 3 +++ 3 files changed, 38 insertions(+), 7 deletions(-) diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index ab187abc11b5..32854c8b2bbc 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -1098,12 +1098,13 @@ static int nested_svm_copy_vmcb12_to_cache(struct k= vm_vcpu *vcpu, u64 vmcb12_gpa struct vcpu_svm *svm =3D to_svm(vcpu); struct vmcb *vmcb12; int r =3D 0; + int idx; =20 - CLASS(kvm_vcpu_map_local, m)(vcpu, gpa_to_gfn(vmcb12_gpa)); - if (m.ret) + idx =3D kvm_gpc_lock_page(&svm->nested.vmcb12_cache, vmcb12_gpa); + if (idx < 0) return -EFAULT; =20 - vmcb12 =3D m.map.hva; + vmcb12 =3D svm->nested.vmcb12_cache.khva; nested_copy_vmcb_control_to_cache(svm, &vmcb12->control); nested_copy_vmcb_save_to_cache(svm, &vmcb12->save); =20 @@ -1114,9 +1115,11 @@ static int nested_svm_copy_vmcb12_to_cache(struct kv= m_vcpu *vcpu, u64 vmcb12_gpa vmcb12->control.event_inj =3D 0; vmcb12->control.event_inj_err =3D 0; svm_set_gif(svm, false); + kvm_gpc_mark_dirty_in_slot(&svm->nested.vmcb12_cache); r =3D -EINVAL; } =20 + kvm_gpc_unlock(&svm->nested.vmcb12_cache, idx); return r; } =20 @@ -1261,12 +1264,14 @@ static int nested_svm_vmexit_update_vmcb12(struct k= vm_vcpu *vcpu) struct vcpu_svm *svm =3D to_svm(vcpu); struct vmcb *vmcb02 =3D svm->nested.vmcb02.ptr; struct vmcb *vmcb12; + int idx; =20 - CLASS(kvm_vcpu_map_local, m)(vcpu, gpa_to_gfn(svm->nested.vmcb12_gpa)); - if (m.ret) - return m.ret; + idx =3D kvm_gpc_lock_page(&svm->nested.vmcb12_cache, + svm->nested.vmcb12_gpa); + if (idx < 0) + return idx; =20 - vmcb12 =3D m.map.hva; + vmcb12 =3D svm->nested.vmcb12_cache.khva; =20 vmcb12->save.es =3D vmcb02->save.es; vmcb12->save.cs =3D vmcb02->save.cs; @@ -1321,6 +1326,9 @@ static int nested_svm_vmexit_update_vmcb12(struct kvm= _vcpu *vcpu) vmcb12->control.exit_int_info_err, KVM_ISA_SVM); =20 + kvm_gpc_mark_dirty_in_slot(&svm->nested.vmcb12_cache); + kvm_gpc_unlock(&svm->nested.vmcb12_cache, idx); + return 0; } =20 @@ -1506,6 +1514,9 @@ int svm_allocate_nested(struct vcpu_svm *svm) goto err_free_vmcb02; =20 svm->nested.vmcb02.cpu =3D -1; + + kvm_gpc_init(&svm->nested.vmcb12_cache, svm->vcpu.kvm); + svm->nested.initialized =3D true; return 0; =20 @@ -1525,6 +1536,8 @@ void svm_free_nested(struct vcpu_svm *svm) svm_vcpu_free_msrpm(svm->nested.msrpm); svm->nested.msrpm =3D NULL; =20 + kvm_gpc_deactivate(&svm->nested.vmcb12_cache); + __free_page(virt_to_page(svm->nested.vmcb02.ptr)); svm->nested.vmcb02.ptr =3D NULL; =20 diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 136069dfcd44..b78727cca9b1 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -5206,6 +5206,21 @@ static int svm_leave_smm(struct kvm_vcpu *vcpu, cons= t union kvm_smram *smram) if (!(smram64->efer & EFER_SVME)) return 1; =20 + /* + * The vmcb12 GPA read back from SMRAM feeds the vmcb12 cache at + * the eventual nested VM-exit, whose copy-back would fail on an + * unaligned or illegal address. Refusing the RSM here ends in + * TRIPLE_FAULT just the same (em_rsm() shoots a refused RSM), + * but it fails before any L2 state has been committed, at the + * instruction whose SMRAM contents are corrupt, matching how + * this function already treats other inconsistent SMRAM state =E2=80=94 + * rather than at an unrelated nested VM-exit later. VMRUN + * (#GP) and KVM_SET_NESTED_STATE (-EINVAL) police the same + * field on their own entry points. + */ + if (!page_address_valid(vcpu, smram64->svm_guest_vmcb_gpa)) + return 1; + CLASS(kvm_vcpu_map_local, m)(vcpu, gpa_to_gfn(smram64->svm_guest_vmcb_gpa= )); if (m.ret) return 1; diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index 1b8f3f2057d9..fe964a961305 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -219,6 +219,9 @@ struct svm_nested_state { u64 last_vmcb12_gpa; u64 last_bus_lock_rip; =20 + /* Cached mapping of the vmcb12 page, keyed on vmcb12_gpa */ + struct gfn_to_pfn_cache vmcb12_cache; + /* * The MSR permissions map used for vmcb02, which is the merge result * of vmcb01 and vmcb12 --=20 2.43.0 From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.92.199 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844078; b=JcD33Nu3MBPakFti0NY7njx3TjcDG1EkbqkgqXhKPk1g42mXM+IJ7iEe1SMbfvfzCuyG fz9Nztpqn3rdWXYMZ7cPJkVKPrTo5Fu+DXWUe3ajWdCnOl29wyrnKEjF9TY++x+vW8Njc ngk2bha3w/S/mACIWoxUJbiGY2VHZNJQ8YGo6mvIThFEKWNcAXzkDQ6vclHFNQqSESj9+ n+ti0qcj92/t1mRBlXhSfMCtMn+nKs1qcJgiBY1afn+Qb+PMy8V84/ktn02HoA753NlzT 9yVT4/jkRnDB88hSa31fMCD/I5xsJFLsqALt2yaOfeJElaeweMEnzcz2J3NMErEWNgQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844078; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=tlil6pgzu2xPgb8UCS8ovNT3btJoY4ercWr+hPOutBs=; b=qZbSLHLMjiqHJEX7SbNB08ReGbBuPhkhGsu/JW/PHUoUusOR+1M8F9mQVrNV8WGiQcgl jR89/3cWp5D1Qid5aPOp/d8K9trh7eTybjm3E7adMPQUlau0hAawkkKslHDeXudFfUq6n dVLDRX/D60pyEVcCiehG9o+7zulG6LAzj+/5GIIRXAcehrr8FCaHaXc7+z6Gc/5hyWb84 TGkXP43N4xjUWM93QAb3AfEIqTv3hHYL4ANZ/Tlw65Q1pHUM2/zlHhtLiUIXvwhhFYqny HkVDmFPOqigfl2lLefrNNLDAvJwDj8frvVbGKs003vPvFnWl273oghGIF/i7cA2wbqw== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=desiato.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=desiato.20200630 header.b=QoSxerRj; arc=none smtp.remote-ip=90.155.92.199 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0EE41606EEA8; Thu, 5 Nov 2026 02:01:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 0EE41606EEA8 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=desiato.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=desiato.20200630 header.b=QoSxerRj DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=tlil6pgzu2xPgb8UCS8ovNT3btJoY4ercWr+hPOutBs=; b=QoSxerRjbEEU1udbAgchHwSKwt FomsXBiFW5t2aoAJeE2TdtAtOcC/DfRns6DbNX+NeVxphZThpnL6hmODqQGX9WMeJCtAltXLLNc6q /Xmjfrm3jclX4VMiKz5h5Xd7iCCTuoiF8RaeVu7Wu+oOKh8MTz//nHfywsXys6nOPUMXWPyYSG224 mpqmZoMFefEcchf3fpLDG+I+XIo+E+uO9M2ZzViyLclSlYGtG8K8cxV1vKb0gL0lWcbB2xmO0HWJK 8O4DICb+DKJXqpYvviVwOAkCGP2Ip3oO08aOZfhgaG0n3rE7/fqNbZ++6UNMr4H7tZYtQ75opwIn4 iPmkqnKA==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by desiato.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmo-00000002ROQ-265G; Thu, 05 Nov 2026 02:01:08 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmm-0000000AWE2-3lIT; Thu, 05 Nov 2026 02:01:04 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 16/20] KVM: nSVM: Cache L1's MSR permissions map pages Date: Thu, 5 Nov 2026 02:00:58 +0000 Message-ID: <20261105020104.2507083-17-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by desiato.infradead.org. See http://www.infradead.org/rpr.html Content-Type: text/plain; charset="utf-8" From: David Woodhouse Reads of L1's MSRPM go through kvm_vcpu_read_guest() on two paths: the vmcb02 MSRPM merge on nested VMRUN (~10 reads per entry), and the per-intercept check on every trapped L2 MSR access. With unmanaged guest memory each read is a memremap/memunmap cycle. Cache the two pages of L1's MSRPM in a pair of gfn_to_pfn_caches, keyed on msrpm_base_pa (re-keying automatically when L1 switches to a different L2 with a different bitmap). Reads preserve the existing failure semantics: an unreadable bitmap means NESTED_EXIT_DONE (reflect to L1) on the intercept path and a failed VMRUN on the merge path. An 8-byte merge read is naturally aligned and a one-byte intercept read cannot cross a page, so each access hits exactly one cached page; a WARN in the accessor enforces that invariant. Signed-off-by: David Woodhouse --- arch/x86/kvm/svm/nested.c | 57 ++++++++++++++++++++++++++++++++------- arch/x86/kvm/svm/svm.h | 7 +++++ 2 files changed, 54 insertions(+), 10 deletions(-) diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 32854c8b2bbc..4e22cd7a38e1 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -288,10 +288,35 @@ int __init nested_svm_init_msrpm_merge_offsets(void) } =20 /* - * Merge L0's (KVM) and L1's (Nested VMCB) MSR permission bitmaps. The fun= ction - * is optimized in that it only merges the parts where KVM MSR permission = bitmap - * may contain zero bits. + * Read from L1's MSR permissions map through the per-page caches. + * @offset is a byte offset into the (two-page) MSRPM; the read must not + * cross a page boundary. msrpm_base_pa is page-aligned by construction: + * __nested_copy_vmcb_control_to_cache() masks it with PAGE_MASK (hardware + * ignores bits 11:0), and nested_vmcb_check_controls() bounds-checks it + * via nested_svm_check_bitmap_pa(). */ +static int nested_svm_read_l1_msrpm(struct vcpu_svm *svm, unsigned long of= fset, + void *val, unsigned int len) +{ + gpa_t base =3D svm->nested.ctl.msrpm_base_pa; + struct gfn_to_pfn_cache *gpc; + int idx; + + if (WARN_ON_ONCE(offset >=3D MSRPM_SIZE || + offset_in_page(offset) + len > PAGE_SIZE)) + return -EINVAL; + + gpc =3D &svm->nested.msrpm12_cache[offset >> PAGE_SHIFT]; + + idx =3D kvm_gpc_lock_page(gpc, base + (offset & PAGE_MASK)); + if (idx < 0) + return idx; + + memcpy(val, gpc->khva + offset_in_page(offset), len); + kvm_gpc_unlock(gpc, idx); + return 0; +} + static bool nested_svm_merge_msrpm(struct kvm_vcpu *vcpu) { struct vcpu_svm *svm =3D to_svm(vcpu); @@ -324,11 +349,9 @@ static bool nested_svm_merge_msrpm(struct kvm_vcpu *vc= pu) for (i =3D 0; i < nested_svm_nr_msrpm_merge_offsets; i++) { const int p =3D nested_svm_msrpm_merge_offsets[i]; nsvm_msrpm_merge_t l1_val; - gpa_t gpa; - - gpa =3D svm->nested.ctl.msrpm_base_pa + (p * sizeof(l1_val)); =20 - if (kvm_vcpu_read_guest(vcpu, gpa, &l1_val, sizeof(l1_val))) + if (nested_svm_read_l1_msrpm(svm, p * sizeof(l1_val), + &l1_val, sizeof(l1_val))) return false; =20 msrpm02[p] =3D msrpm01[p] | l1_val; @@ -1499,6 +1522,7 @@ static void nested_svm_triple_fault(struct kvm_vcpu *= vcpu) int svm_allocate_nested(struct vcpu_svm *svm) { struct page *vmcb02_page; + int i; =20 if (svm->nested.initialized) return 0; @@ -1517,6 +1541,16 @@ int svm_allocate_nested(struct vcpu_svm *svm) =20 kvm_gpc_init(&svm->nested.vmcb12_cache, svm->vcpu.kvm); =20 + /* + * KVM only ever reads the MSR permissions map, which L1 may keep in + * memory that its VMM maps read-only: map the cache read-only + * (which also means never marking it dirty), matching the read + * semantics of the kvm_vcpu_read_guest() calls it replaces. + */ + for (i =3D 0; i < ARRAY_SIZE(svm->nested.msrpm12_cache); i++) + __kvm_gpc_init(&svm->nested.msrpm12_cache[i], svm->vcpu.kvm, + false, true, NULL, 0); + svm->nested.initialized =3D true; return 0; =20 @@ -1527,6 +1561,8 @@ int svm_allocate_nested(struct vcpu_svm *svm) =20 void svm_free_nested(struct vcpu_svm *svm) { + int i; + if (!svm->nested.initialized) return; =20 @@ -1537,6 +1573,8 @@ void svm_free_nested(struct vcpu_svm *svm) svm->nested.msrpm =3D NULL; =20 kvm_gpc_deactivate(&svm->nested.vmcb12_cache); + for (i =3D 0; i < ARRAY_SIZE(svm->nested.msrpm12_cache); i++) + kvm_gpc_deactivate(&svm->nested.msrpm12_cache[i]); =20 __free_page(virt_to_page(svm->nested.vmcb02.ptr)); svm->nested.vmcb02.ptr =3D NULL; @@ -1589,7 +1627,6 @@ void svm_leave_nested(struct kvm_vcpu *vcpu) =20 static int nested_svm_exit_handled_msr(struct vcpu_svm *svm) { - gpa_t base =3D svm->nested.ctl.msrpm_base_pa; int write, bit_nr; u8 value, mask; u32 msr; @@ -1604,8 +1641,8 @@ static int nested_svm_exit_handled_msr(struct vcpu_sv= m *svm) if (bit_nr < 0) return NESTED_EXIT_DONE; =20 - if (kvm_vcpu_read_guest(&svm->vcpu, base + bit_nr / BITS_PER_BYTE, - &value, sizeof(value))) + if (nested_svm_read_l1_msrpm(svm, bit_nr / BITS_PER_BYTE, + &value, sizeof(value))) return NESTED_EXIT_DONE; =20 mask =3D BIT(write) << (bit_nr & (BITS_PER_BYTE - 1)); diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index fe964a961305..8a5dff5a9ee3 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -222,6 +222,13 @@ struct svm_nested_state { /* Cached mapping of the vmcb12 page, keyed on vmcb12_gpa */ struct gfn_to_pfn_cache vmcb12_cache; =20 + /* + * Cached mappings of the two pages of L1's MSR permissions map, + * keyed on ctl.msrpm_base_pa, for the vmcb02 merge and the + * per-intercept checks. + */ + struct gfn_to_pfn_cache msrpm12_cache[MSRPM_SIZE / PAGE_SIZE]; + /* * The MSR permissions map used for vmcb02, which is the merge result * of vmcb01 and vmcb12 --=20 2.43.0 From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.92.199 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844079; b=y8ElK9gPtWJmvTj+MsbhO9l738ikOpUTBySGMoKkT+8X5RatNY4G7V9JaTxZSwzOoS+n gBUxTB6QvIaunUcGarVktcihag2UdaMqR/gZutBpQAR+g7m/oZUV+5uRfIdVeDFXiTsoe 172oadm8Av5/kYVJbokcxyoet6qdT3rQ7ZVQFacX0zZXEwJ6lYSCsTihTqTQTzHrj3A+m COpIzd8tAPm9NH/ASJ5Deh1pngat5r77AvMXiPBiNrsVrhenVYlsvuB2iivOGfUoAhap7 NSR8Q0glhAZqXU0pZ42Psp1kVplEyi+W3lD/sbNIJj4bkea9cAkOxel3YwA/0drA6rw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844079; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Type:Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=HHbMXrntQTOJ5IPdESBOGZVApcYK5UI5lu9oMqvobYE=; b=XuZbXpuUTXvPaiCY86dXynbJdTRqSr/Wyh01zFaVjrynEXt5gybKPfsO+NL9eLY+wnrt r/lMFqOhwkDgzuNpOSpU30nMnnecbGWOvEtrwbkY8X+dOnA2rHOXm+qwEPfnoOLyuFMN8 ReFdnusBOXfj0I1NXfLlTIehxyOegcUnMX1r0Q/LD+jDICaKSFZNHjp5F11ZUMjo0JZgP VruAjkJumwhYhyhY1i8ZHmebDKJU1enkA4rrc0IK0n92L7d76k/GkpiU+4UtvQuc081N2 1+ugoYmMw4g4jvHV7Sq2U0wV0PRysBkjrU0186NS2ngSf3DLDpwaJxbQeh6Bab3XkJQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=desiato.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=desiato.20200630 header.b=Ae+tqrD0; arc=none smtp.remote-ip=90.155.92.199 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0EC99606EEA6; Thu, 5 Nov 2026 02:01:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 0EC99606EEA6 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=desiato.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=desiato.20200630 header.b=Ae+tqrD0 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description; bh=HHbMXrntQTOJ5IPdESBOGZVApcYK5UI5lu9oMqvobYE=; b=Ae+tqrD0A5cIWp9q1MmrdrlOhm x0DHa/HcBOUb8qpxiTZNQGso0gaFp60cS1T4G+k/gU+stshu3i38AotQfBfdAfTVoIaZt0GKHNtOv 1xIrt0/A/Qf433sSC8WDNBQT5CQu86Qs6LujeOtqo2kml1k9vlrsI1Rg8cJ4epb3o0SVHtLo10tDb Bka/PNm22PqSLTkTcgJD7DwCbNWBvP1QbHQqVFnV/PQuKz76uTOKf+3XHGmyNLeMcSPI5oWqY05ar DAuaXhf8/SNlQ9InzAkjGLZ4o4PRxsduxy4SN91W/GhAxZ7O3Ywe6d3KPGYt9qFII5v9FQYyj21e0 vIZZreKg==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by desiato.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmo-00000002ROR-265C; Thu, 05 Nov 2026 02:01:08 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmm-0000000AWE6-3u0A; Thu, 05 Nov 2026 02:01:04 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 17/20] KVM: nSVM: Cache L1's IO permissions map pages Date: Thu, 5 Nov 2026 02:00:59 +0000 Message-ID: <20261105020104.2507083-18-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by desiato.infradead.org. See http://www.infradead.org/rpr.html From: David Woodhouse The per-intercept IOPM check on every trapped L2 IO instruction reads L1's IO permissions map with kvm_vcpu_read_guest() =E2=80=94 a memremap/memunmap cycle per access with unmanaged guest memory. Cache the three IOPM pages in gfn_to_pfn_caches keyed on iopm_base_pa, as for the MSR permissions map. Unlike the MSRPM accesses, an IOPM read is not naturally aligned: a two-byte read for a port range straddling a page boundary (e.g. ports 0x7ff8-0x7fff at bytes 0xfff-0x1000) spans two pages, so read byte by byte, each from its own page's cache. Failure semantics are preserved: an unreadable IOPM reflects the intercept to L1 (NESTED_EXIT_DONE). Signed-off-by: David Woodhouse --- arch/x86/kvm/svm/nested.c | 71 +++++++++++++++++++++++++++++---------- arch/x86/kvm/svm/svm.c | 2 ++ arch/x86/kvm/svm/svm.h | 7 ++++ 3 files changed, 63 insertions(+), 17 deletions(-) diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 4e22cd7a38e1..a682b7446c3e 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -1519,10 +1519,36 @@ static void nested_svm_triple_fault(struct kvm_vcpu= *vcpu) nested_svm_simple_vmexit(to_svm(vcpu), SVM_EXIT_SHUTDOWN); } =20 +/* + * Initialize the nested gfn_to_pfn_caches, once per vCPU. Unlike the + * allocations in svm_allocate_nested(), the caches persist across + * EFER.SVME toggles: svm_free_nested() only deactivates them, and + * reuse goes through kvm_gpc_activate() without re-init. + */ +void svm_init_nested_caches(struct vcpu_svm *svm) +{ + int i; + + kvm_gpc_init(&svm->nested.vmcb12_cache, svm->vcpu.kvm); + + /* + * KVM only ever reads the MSR and IO permissions maps, which L1 may + * keep in memory that its VMM maps read-only: map the caches + * read-only (which also means never marking them dirty), matching + * the read semantics of the kvm_vcpu_read_guest() calls they + * replace. + */ + for (i =3D 0; i < ARRAY_SIZE(svm->nested.msrpm12_cache); i++) + __kvm_gpc_init(&svm->nested.msrpm12_cache[i], svm->vcpu.kvm, + false, true, NULL, 0); + for (i =3D 0; i < ARRAY_SIZE(svm->nested.iopm12_cache); i++) + __kvm_gpc_init(&svm->nested.iopm12_cache[i], svm->vcpu.kvm, + false, true, NULL, 0); +} + int svm_allocate_nested(struct vcpu_svm *svm) { struct page *vmcb02_page; - int i; =20 if (svm->nested.initialized) return 0; @@ -1539,18 +1565,6 @@ int svm_allocate_nested(struct vcpu_svm *svm) =20 svm->nested.vmcb02.cpu =3D -1; =20 - kvm_gpc_init(&svm->nested.vmcb12_cache, svm->vcpu.kvm); - - /* - * KVM only ever reads the MSR permissions map, which L1 may keep in - * memory that its VMM maps read-only: map the cache read-only - * (which also means never marking it dirty), matching the read - * semantics of the kvm_vcpu_read_guest() calls it replaces. - */ - for (i =3D 0; i < ARRAY_SIZE(svm->nested.msrpm12_cache); i++) - __kvm_gpc_init(&svm->nested.msrpm12_cache[i], svm->vcpu.kvm, - false, true, NULL, 0); - svm->nested.initialized =3D true; return 0; =20 @@ -1575,6 +1589,8 @@ void svm_free_nested(struct vcpu_svm *svm) kvm_gpc_deactivate(&svm->nested.vmcb12_cache); for (i =3D 0; i < ARRAY_SIZE(svm->nested.msrpm12_cache); i++) kvm_gpc_deactivate(&svm->nested.msrpm12_cache[i]); + for (i =3D 0; i < ARRAY_SIZE(svm->nested.iopm12_cache); i++) + kvm_gpc_deactivate(&svm->nested.iopm12_cache[i]); =20 __free_page(virt_to_page(svm->nested.vmcb02.ptr)); svm->nested.vmcb02.ptr =3D NULL; @@ -1654,7 +1670,7 @@ static int nested_svm_intercept_ioio(struct vcpu_svm = *svm) unsigned port, size, iopm_len; u16 val, mask; u8 start_bit; - u64 gpa; + int i; =20 if (!(vmcb12_is_intercept(&svm->nested.ctl, INTERCEPT_IOIO_PROT))) return NESTED_EXIT_HOST; @@ -1662,14 +1678,35 @@ static int nested_svm_intercept_ioio(struct vcpu_sv= m *svm) port =3D svm->vmcb->control.exit_info_1 >> 16; size =3D (svm->vmcb->control.exit_info_1 & SVM_IOIO_SIZE_MASK) >> SVM_IOIO_SIZE_SHIFT; - gpa =3D svm->nested.ctl.iopm_base_pa + (port / 8); start_bit =3D port % 8; iopm_len =3D (start_bit + size > 8) ? 2 : 1; mask =3D (0xf >> (4 - size)) << start_bit; val =3D 0; =20 - if (kvm_vcpu_read_guest(&svm->vcpu, gpa, &val, iopm_len)) - return NESTED_EXIT_DONE; + /* + * Read byte by byte: a two-byte read at the end of a page (e.g. + * ports 0x7ff8-0x7fff spanning bytes 0xfff-0x1000) crosses into + * the next page of the IOPM, i.e. the next cache. + */ + for (i =3D 0; i < iopm_len; i++) { + unsigned long offset =3D port / 8 + i; + struct gfn_to_pfn_cache *gpc; + gpa_t gpa; + int idx; + + if (WARN_ON_ONCE(offset >=3D IOPM_SIZE)) + return NESTED_EXIT_DONE; + + gpc =3D &svm->nested.iopm12_cache[offset >> PAGE_SHIFT]; + gpa =3D svm->nested.ctl.iopm_base_pa + (offset & PAGE_MASK); + + idx =3D kvm_gpc_lock_page(gpc, gpa); + if (idx < 0) + return NESTED_EXIT_DONE; + + val |=3D *(u8 *)(gpc->khva + offset_in_page(offset)) << (i * 8); + kvm_gpc_unlock(gpc, idx); + } =20 return (val & mask) ? NESTED_EXIT_DONE : NESTED_EXIT_HOST; } diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index b78727cca9b1..f601e0650c36 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -1382,6 +1382,8 @@ static int svm_vcpu_create(struct kvm_vcpu *vcpu) svm->vmcb01.pa =3D __sme_set(page_to_pfn(vmcb01_page) << PAGE_SHIFT); svm_switch_vmcb(svm, &svm->vmcb01); =20 + svm_init_nested_caches(svm); + svm->guest_state_loaded =3D false; =20 return 0; diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index 8a5dff5a9ee3..256eefb27a83 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -229,6 +229,12 @@ struct svm_nested_state { */ struct gfn_to_pfn_cache msrpm12_cache[MSRPM_SIZE / PAGE_SIZE]; =20 + /* + * Cached mappings of the three pages of L1's IO permissions map, + * keyed on ctl.iopm_base_pa, for the per-intercept checks. + */ + struct gfn_to_pfn_cache iopm12_cache[IOPM_SIZE / PAGE_SIZE]; + /* * The MSR permissions map used for vmcb02, which is the merge result * of vmcb01 and vmcb12 @@ -904,6 +910,7 @@ int __init nested_svm_init_msrpm_merge_offsets(void); =20 int enter_svm_guest_mode(struct kvm_vcpu *vcpu, u64 vmcb_gpa, bool from_vm= run); void svm_leave_nested(struct kvm_vcpu *vcpu); +void svm_init_nested_caches(struct vcpu_svm *svm); void svm_free_nested(struct vcpu_svm *svm); int svm_allocate_nested(struct vcpu_svm *svm); int nested_svm_vmrun(struct kvm_vcpu *vcpu); --=20 2.43.0 From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.50.34 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844072; b=3qDyKL4AYhGASmvEBdEZPI3k9aTJ8QqdNhXV8TRBthyuKCSCJkR1AP8baZ/iRXIFE4aA hEoNMkhL6kaFUqPw92ixfu6GeS0g++thsyBkU4FFdJ6mO/01bCbmFdLwKvOSXbPr6iEnU B3qbMs9wZWmW+2tRWWEPaBt9U/Ru0koIjv5+h1kQzzax2cJhlWC8FR/pYZJnK2/U/Qu+H cg/GJg2Hl3KFxVZZ0gQFVBBoAo9ukphZSPbutRu7cYTpU5RD7/ollyOlyz8qWoQQwgZNF 1uEYPM7EOvEETy3JaPT3ihO2HO+6dUPSHLWzV4ACYE1bPZ/1ItC03sqfIq5gdyp/vqQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844072; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=ddwtYMIgIqYLX7L/896a+WHQ4S2b0Us6ZQUDCLv9brk=; b=uodMTTcF2XhoSte7/dCp9x1EkJIEWloiEdulk2M+GePN5/INJHYLv73YaXDLKCUoPY/8 4L42iqv+6rkGuvA/kSm/HQqGA1JG2Mc3hbYvpf+fWCHDQtgIH4hsch+SeIKADfS4RO93j Gs0CxNugAuVtr+tUT2beit8sdnvUxGeOyMpALVylNai4mBrj+S7Lvpu5Zk8Yp6g/fYQns aboXHh77VIDlaY7UBtCh2PTOD+2OFi1JKO+p3JwwuH9ujPe4B3hq0rvriOFYWZ8LY4aC5 P+HUW/35RvCrXy80bSWfeojccvtU45qxA0O3cKIOnya4stQYpylZlvl5cOuGETbE7+w== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=casper.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=ldzMubhh; arc=none smtp.remote-ip=90.155.50.34 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAA8D606A333; Thu, 5 Nov 2026 02:01:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org DAA8D606A333 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=casper.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=ldzMubhh DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=ddwtYMIgIqYLX7L/896a+WHQ4S2b0Us6ZQUDCLv9brk=; b=ldzMubhhw0uNbJyP6Luvoz+z/6 4rCLI8wrG8+4Cj8ACR1erqujTCctAjYU2yXzHNdeCNINWAyoeIB6a0g5hh3ek39ZM7a75EnY322AG sK8evPJu3ZKHflFM+gewOUPFXU65mJvn//HcYCbREolUjSi5Qt93c5f0JiBpS2z5TzypAjUiOWQqu qoc3xzvcU4Dd+y6VwCWk9T2wl7dnqHfBHJssZioPoC4fDUPckO78PMFrGZn/mj4q6vH3h8C9A57rX fRV5CzVjCgD2kg6uGqI4sFQR1U+8Tdmbc+9fLz8z4SmwEvnbPbjCxhp+aMPhcEGXNtZwh0cnhdy0w kw1mxWpA==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by casper.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmn-000000037jT-2028; Thu, 05 Nov 2026 02:01:05 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmm-0000000AWEA-480s; Thu, 05 Nov 2026 02:01:04 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 18/20] KVM: selftests: Add nested transition benchmark Date: Thu, 5 Nov 2026 02:01:00 +0000 Message-ID: <20261105020104.2507083-19-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html Content-Type: text/plain; charset="utf-8" From: David Woodhouse Measure L1<->L2 transition costs from L1's viewpoint, in TSC cycles: a reflected-exit phase (cpuid in L2, forwarded to L1: the full emulated vmexit->vmentry round trip, exercising the vmcb12/vmcs12 access paths), and on SVM an MSR-intercept phase (rdmsr in L2 with INTERCEPT_MSR_PROT: exercises the per-intercept lookup in L1's MSR permissions map). Reports min/avg/p50/p99/max over 10k iterations. Works on both VMX and SVM. Signed-off-by: David Woodhouse Assisted-by: LLM --- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../kvm/x86/nested_transition_bench.c | 210 ++++++++++++++++++ 2 files changed, 211 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86/nested_transition_bench= .c diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selft= ests/kvm/Makefile.kvm index dcfe815f4a02..65639e54b554 100644 --- a/tools/testing/selftests/kvm/Makefile.kvm +++ b/tools/testing/selftests/kvm/Makefile.kvm @@ -162,6 +162,7 @@ TEST_GEN_PROGS_x86 +=3D x86/recalc_apic_map_test TEST_GEN_PROGS_x86 +=3D x86/aperfmperf_test TEST_GEN_PROGS_x86 +=3D x86/guest_memfd_conversions_test TEST_GEN_PROGS_x86 +=3D x86/vmx_apic_update_test +TEST_GEN_PROGS_x86 +=3D x86/nested_transition_bench TEST_GEN_PROGS_x86 +=3D access_tracking_perf_test TEST_GEN_PROGS_x86 +=3D coalesced_io_test TEST_GEN_PROGS_x86 +=3D dirty_log_perf_test diff --git a/tools/testing/selftests/kvm/x86/nested_transition_bench.c b/to= ols/testing/selftests/kvm/x86/nested_transition_bench.c new file mode 100644 index 000000000000..ea91ffcc4e0e --- /dev/null +++ b/tools/testing/selftests/kvm/x86/nested_transition_bench.c @@ -0,0 +1,210 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * nested_transition_bench + * + * Measure nested virtualization transition costs: + * 1. L2 exit reflected to L1 (cpuid): full emulated vmexit->vmentry + * round trip, exercising the vmcb12/vmcs12 access paths. + * 2. L2 MSR access with MSR intercepts enabled: exercises the + * per-intercept L1 permissions-bitmap lookup on SVM, and the + * bitmap merge path when force_msr_bitmap_recalc is set. + * + * Reports TSC cycles per iteration (min/avg/p99) from L1's viewpoint. + */ +#include +#include + +#include "test_util.h" +#include "kvm_util.h" +#include "processor.h" +#include "vmx.h" +#include "svm_util.h" + +/* + * Keep the iteration count modest: every timed iteration includes a + * ucall round trip to userspace to report its sample, which on a + * debug (KASAN) kernel costs milliseconds. 10k samples still give a + * stable p99. + */ +#define N_WARMUP 1000 +#define N_ITERS 10000 + + +static uint64_t results[N_ITERS]; + +/* L2: exit to L1 in a tight loop. */ +static void l2_guest_code(void) +{ + u32 eax, ebx, ecx, edx; + + for (;;) + cpuid(0, &eax, &ebx, &ecx, &edx); +} + +static void l2_guest_code_msr(void) +{ + for (;;) + rdmsr(MSR_IA32_TSC_DEADLINE); +} + +/* SVM L1: run L2, timing VMRUN -> #VMEXIT round trips. */ +static void l1_svm_code(struct svm_test_data *svm) +{ + struct vmcb *vmcb =3D svm->vmcb; + uint64_t t0, t1; + int i; + + generic_svm_setup(svm, l2_guest_code); + /* + * Unlike VMX, where CPUID unconditionally exits, SVM only + * intercepts CPUID if asked; without it L0 emulates L2's cpuid + * and resumes L2 directly, and run_guest() never returns. + */ + vmcb->control.intercept |=3D BIT_ULL(INTERCEPT_CPUID) | + BIT_ULL(INTERCEPT_MSR_PROT); + + /* + * Set the read-intercept bit for MSR_IA32_TSC_DEADLINE (0x6e0, + * MSRPM range 0: bit 2*msr, i.e. byte 0x1b8 bit 0) in L1's MSRPM, + * so that L2's rdmsr in phase 2 reflects to L1. L0's per-exit + * lookup in this bitmap is the path being measured. + */ + ((u8 *)svm->msr)[(MSR_IA32_TSC_DEADLINE & 0x1fff) * 2 / 8] |=3D + BIT((MSR_IA32_TSC_DEADLINE & 0x1fff) * 2 % 8); + + for (i =3D 0; i < N_WARMUP; i++) { + run_guest(vmcb, svm->vmcb_gpa); + GUEST_ASSERT(vmcb->control.exit_code =3D=3D SVM_EXIT_CPUID); + vmcb->save.rip +=3D 2; /* cpuid */ + } + + for (i =3D 0; i < N_ITERS; i++) { + t0 =3D rdtsc(); + run_guest(vmcb, svm->vmcb_gpa); + t1 =3D rdtsc(); + GUEST_ASSERT(vmcb->control.exit_code =3D=3D SVM_EXIT_CPUID); + vmcb->save.rip +=3D 2; + GUEST_SYNC1(t1 - t0); + } + + /* Phase 2: MSR intercept path */ + vmcb->save.rip =3D (u64)l2_guest_code_msr; + for (i =3D 0; i < N_WARMUP; i++) { + run_guest(vmcb, svm->vmcb_gpa); + GUEST_ASSERT(vmcb->control.exit_code =3D=3D SVM_EXIT_MSR); + vmcb->save.rip +=3D 2; /* rdmsr */ + } + + for (i =3D 0; i < N_ITERS; i++) { + t0 =3D rdtsc(); + run_guest(vmcb, svm->vmcb_gpa); + t1 =3D rdtsc(); + GUEST_ASSERT(vmcb->control.exit_code =3D=3D SVM_EXIT_MSR); + vmcb->save.rip +=3D 2; /* rdmsr */ + GUEST_SYNC1(t1 - t0); + } + + GUEST_DONE(); +} + +/* VMX L1: run L2, timing VMRESUME -> vmexit round trips. */ +static void l1_vmx_code(struct vmx_pages *vmx) +{ + uint64_t t0, t1; + int i; + + prepare_for_vmx_operation(vmx); + load_vmcs(vmx); + prepare_vmcs(vmx, l2_guest_code); + + vmlaunch(); + /* first exit is here; loop resumes */ + for (i =3D 0; i < N_WARMUP; i++) { + GUEST_ASSERT(vmread(VM_EXIT_REASON) =3D=3D EXIT_REASON_CPUID); + vmwrite(GUEST_RIP, vmread(GUEST_RIP) + + vmread(VM_EXIT_INSTRUCTION_LEN)); + vmresume(); + } + + for (i =3D 0; i < N_ITERS; i++) { + GUEST_ASSERT(vmread(VM_EXIT_REASON) =3D=3D EXIT_REASON_CPUID); + vmwrite(GUEST_RIP, vmread(GUEST_RIP) + + vmread(VM_EXIT_INSTRUCTION_LEN)); + t0 =3D rdtsc(); + vmresume(); + t1 =3D rdtsc(); + GUEST_SYNC1(t1 - t0); + } + + GUEST_DONE(); +} + +static int cmp_u64(const void *a, const void *b) +{ + uint64_t x =3D *(const uint64_t *)a, y =3D *(const uint64_t *)b; + + return x < y ? -1 : x > y; +} + +static void report(const char *phase, int n) +{ + uint64_t sum =3D 0; + int i; + + qsort(results, n, sizeof(results[0]), cmp_u64); + for (i =3D 0; i < n; i++) + sum +=3D results[i]; + + printf("%-24s n=3D%d min=3D%" PRIu64 " avg=3D%" PRIu64 " p50=3D%" PRIu64 + " p99=3D%" PRIu64 " max=3D%" PRIu64 " cycles\n", + phase, n, results[0], sum / n, results[n / 2], + results[(int)(n * 0.99)], results[n - 1]); +} + +int main(int argc, char *argv[]) +{ + struct kvm_vcpu *vcpu; + struct kvm_vm *vm; + struct ucall uc; + gva_t nested_gva =3D 0; + bool is_svm =3D kvm_cpu_has(X86_FEATURE_SVM); + int n =3D 0, phase =3D 0; + bool done =3D false; + + TEST_REQUIRE(is_svm || kvm_cpu_has(X86_FEATURE_VMX)); + + vm =3D vm_create_with_one_vcpu(&vcpu, is_svm ? (void *)l1_svm_code + : (void *)l1_vmx_code); + if (is_svm) + vcpu_alloc_svm(vm, &nested_gva); + else + vcpu_alloc_vmx(vm, &nested_gva); + vcpu_args_set(vcpu, 1, nested_gva); + + while (!done) { + vcpu_run(vcpu); + switch (get_ucall(vcpu, &uc)) { + case UCALL_SYNC: + results[n++] =3D uc.args[0]; + if (n =3D=3D N_ITERS) { + report(phase =3D=3D 0 ? "reflected-exit" : + "msr-intercept", n); + n =3D 0; + phase++; + } + break; + case UCALL_ABORT: + REPORT_GUEST_ASSERT(uc); + case UCALL_DONE: + if (n) + report("partial", n); + done =3D true; + break; + default: + TEST_FAIL("Unknown ucall %lu", uc.cmd); + } + } + + kvm_vm_free(vm); + return 0; +} --=20 2.43.0 From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.50.34 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844073; b=XBqs+kaQEOJ9Dus7moelKRa2pPpj1dVo0d5PR+boNYGjRPXbstSRt+2vBgn8bXguE7uk RbYdukguhqFOYntrTgE7WTphpekSfUdHmlE1APk0Uy51OFPydnFJEQc+u/B2Rfakw1EqT dF/GDZB2EqrnDIyomAAjHN5gxhmFgpn0ry5N7/zTF6nvlAAI9Qf/C3WoEO4A7/95Ya5g9 8AzjexV8QsP8wc3AQALZv7GhdseDgXbU0jTwhgBpr9au2Os1gzY9J1tBmD5eRtkBXH+bW 4Pumpi9JHn+JlgcCwP7bNUFtN+a1aTewvfE5TLX9bJoYhxrZC8W5yii9UDuHljDpV2w== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844073; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Type:Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=pphF2Gry51KkjraDv3DbKbxipU/YNYO4YsKlt2HijlE=; b=AvpTNI+NSMReB7CQ46Mq2bOPnHtTeidxkB2gpB1jKgnPUWQgLc2dUB+7wvqUWLeeIsJr c+NaTopr8HBpNEVLrIAWzM4kOV9wrlDbLVvqxW9oVkEAGZFx3HLkj6pvq1PrG4cUyg43/ Rdi/EBtQK1S/ec2tqfPB31LUxF7fK/ylwi6AKMlKp71SKnePIse/7YF1WTJKHEo/vejPV psYKYOwBPzXc0KVGU/lggbhE84J77M1AEJCMNYBRVXTrCnOQAgI8Rqm9x93PJRQY8GVhC y8kEjYburZKDTLKyblKnFa1M5+tCg95XFnYSBn4E0ugRv7UuH7wxl/cFAHzSxfUvAww== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=casper.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=W0697dEo; arc=none smtp.remote-ip=90.155.50.34 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B73C0606EE8A; Thu, 5 Nov 2026 02:01:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org B73C0606EE8A Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=casper.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=casper.20170209 header.b=W0697dEo DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description; bh=pphF2Gry51KkjraDv3DbKbxipU/YNYO4YsKlt2HijlE=; b=W0697dEoJqJFNkTbdno7tSxg8G ryaiNpDrSQQ35mK6NawV27thPKFt0wfyyzkBPAsPnhLMa2+Eupz6srh4Ox0LFNxPCbpsiB78Ac83t 1LjqqOktxFG7baANhZHISm+eT5QS9ADz5a9+rlEcE8Ijp8Ik664gJ3UZ1NqW2anAl29T2YrA7zHCM fWPgveNZhaN9uj8iDSgjwRQU6Dr9381q5jYKSUBPRG5H6k9YqsbkJQI8eFhiF0+h0E5HyzEVCN4eG X5RLEagfvhmb07LoZa0QIPu+BlMtf6GxLPaxqMBC+OK70ln83PM0NoheGP6U0PXyG8DcwCGbFs8IY fAcvXPWQ==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by casper.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmn-000000037jU-292l; Thu, 05 Nov 2026 02:01:05 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmn-0000000AWEE-056f; Thu, 05 Nov 2026 02:01:05 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 19/20] KVM: selftests: Add Xen evtchn vs memslot-update race test Date: Thu, 5 Nov 2026 02:01:01 +0000 Message-ID: <20261105020104.2507083-20-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html From: David Woodhouse Race a guest EVTCHNOP_send loop against a host thread which deletes and recreates the shared_info memslot. The delivery path runs in the vCPU's exit-handling context, under vcpu_run's kvm->srcu read section; if it ever waited there for the memslot update to complete, it would deadlock against kvm_swap_active_memslots() waiting out that very read section in synchronize_srcu(), and this test livelocks within seconds. The deadlock is the one claim the test pins: a send which races the update legitimately fails (the hypercall bounces to userspace, which completes it with an error), so the test makes no assertion about delivery counts beyond "the churn did not starve delivery entirely". Signed-off-by: David Woodhouse --- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../kvm/x86/xen_evtchn_memslot_race_test.c | 232 ++++++++++++++++++ 2 files changed, 233 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86/xen_evtchn_memslot_race= _test.c diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selft= ests/kvm/Makefile.kvm index 65639e54b554..c17408f8d61f 100644 --- a/tools/testing/selftests/kvm/Makefile.kvm +++ b/tools/testing/selftests/kvm/Makefile.kvm @@ -149,6 +149,7 @@ TEST_GEN_PROGS_x86 +=3D x86/xss_msr_test TEST_GEN_PROGS_x86 +=3D x86/debug_regs TEST_GEN_PROGS_x86 +=3D x86/tsc_msrs_test TEST_GEN_PROGS_x86 +=3D x86/vmx_pmu_caps_test +TEST_GEN_PROGS_x86 +=3D x86/xen_evtchn_memslot_race_test TEST_GEN_PROGS_x86 +=3D x86/xen_shinfo_test TEST_GEN_PROGS_x86 +=3D x86/xen_vmcall_test TEST_GEN_PROGS_x86 +=3D x86/sev_dbg_test diff --git a/tools/testing/selftests/kvm/x86/xen_evtchn_memslot_race_test.c= b/tools/testing/selftests/kvm/x86/xen_evtchn_memslot_race_test.c new file mode 100644 index 000000000000..c9448df52a6e --- /dev/null +++ b/tools/testing/selftests/kvm/x86/xen_evtchn_memslot_race_test.c @@ -0,0 +1,232 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * xen_evtchn_memslot_race_test + * + * A guest EVTCHNOP_send whose delivery races a DELETE/re-CREATE of the + * memslot holding the shared_info page. The delivery path runs in the + * vCPU's exit-handling context, under vcpu_run's kvm->srcu read + * section; if it waits there for the memslot update to complete, it + * deadlocks against kvm_swap_active_memslots()'s + * synchronize_srcu_expedited() waiting out that very read section. + * Regression test for that deadlock: KVM must leave the hypercall + * uncompleted so the guest re-executes it, rather than wait in exit + * context. + * + * The guest sends events in a tight loop; a host thread deletes and + * recreates the shared_info slot in a tight loop. On a broken host + * this livelocks within a second or two (the test then dies by + * timeout); on a fixed host the guest completes all N + * iterations (sends which hit the no-slot gap between DELETE and + * re-CREATE legitimately fail; most deliver). + */ +#include +#include + +#include "test_util.h" +#include "kvm_util.h" +#include "processor.h" + +#include + +#define SHINFO_REGION_GPA 0xc0000000ULL +#define SHINFO_REGION_SLOT 10 + +#define EVTCHN_PORT 15 +#define TEST_ITERATIONS 4000 + +#define EVTCHNOP_send 4 +#define __HYPERVISOR_event_channel_op 32 + +#define XEN_HYPERCALL_MSR 0x40000000 + +#define EVTCHNSTAT_ipi 5 + +struct evtchn_send { + uint32_t port; +}; + +static void guest_code(void) +{ + struct evtchn_send s =3D { .port =3D EVTCHN_PORT }; + int delivered =3D 0; + int i; + + for (i =3D 0; i < TEST_ITERATIONS; i++) { + /* + * The delivery may legitimately fail while the shared_info + * slot is mid-churn (between DELETE and re-CREATE there is + * no slot at all, which is -EFAULT, not a transient). The + * failure mode under test is the vCPU never coming back at + * all (deadlock -> timeout), not the hypercall rc. + */ + if (!__xen_hypercall(__HYPERVISOR_event_channel_op, + EVTCHNOP_send, &s)) + delivered++; + GUEST_SYNC(i); + } + + /* + * The churn must not have starved delivery entirely. Every + * iteration exits to userspace for the GUEST_SYNC, so the slot + * spends real wall time present between churn cycles; empirically + * a few hundred of the sends deliver. + */ + GUEST_ASSERT(delivered); + + /* + * Report the delivered count. The value is always at least + * TEST_ITERATIONS + 1, so the host can distinguish it from the + * in-loop syncs, whose values are below TEST_ITERATIONS. + */ + GUEST_SYNC(TEST_ITERATIONS + delivered); + + GUEST_DONE(); +} + +static atomic_bool churn_stop; + +static void *memslot_churn(void *arg) +{ + struct kvm_vm *vm =3D arg; + + while (!atomic_load(&churn_stop)) { + vm_mem_region_delete(vm, SHINFO_REGION_SLOT); + vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, + SHINFO_REGION_GPA, + SHINFO_REGION_SLOT, 2, 0); + } + return NULL; +} + +int main(int argc, char *argv[]) +{ + struct kvm_vcpu *vcpu; + struct kvm_vm *vm; + pthread_t thread; + struct ucall uc; + int bounced =3D 0; + int done =3D 0; + + int xen_caps =3D kvm_check_cap(KVM_CAP_XEN_HVM); + + TEST_REQUIRE(xen_caps & KVM_XEN_HVM_CONFIG_SHARED_INFO); + TEST_REQUIRE(xen_caps & KVM_XEN_HVM_CONFIG_EVTCHN_SEND); + + vm =3D vm_create_with_one_vcpu(&vcpu, guest_code); + + vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, + SHINFO_REGION_GPA, SHINFO_REGION_SLOT, 2, 0); + + struct kvm_xen_hvm_config hvmc =3D { + .flags =3D KVM_XEN_HVM_CONFIG_INTERCEPT_HCALL | + KVM_XEN_HVM_CONFIG_EVTCHN_SEND, + .msr =3D XEN_HYPERCALL_MSR, + }; + vm_ioctl(vm, KVM_XEN_HVM_CONFIG, &hvmc); + + struct kvm_xen_hvm_attr lm =3D { + .type =3D KVM_XEN_ATTR_TYPE_LONG_MODE, + .u.long_mode =3D 1, + }; + vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &lm); + + struct kvm_xen_hvm_attr ha =3D { + .type =3D KVM_XEN_ATTR_TYPE_SHARED_INFO, + .u.shared_info.gfn =3D SHINFO_REGION_GPA / PAGE_SIZE, + }; + vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &ha); + + struct kvm_xen_vcpu_attr vi =3D { + .type =3D KVM_XEN_VCPU_ATTR_TYPE_VCPU_INFO, + .u.gpa =3D SHINFO_REGION_GPA + 0x40, + }; + vcpu_ioctl(vcpu, KVM_XEN_VCPU_SET_ATTR, &vi); + + /* Loop the port back to this vCPU as an IPI. */ + struct kvm_xen_hvm_attr evt =3D { + .type =3D KVM_XEN_ATTR_TYPE_EVTCHN, + .u.evtchn.send_port =3D EVTCHN_PORT, + .u.evtchn.type =3D EVTCHNSTAT_ipi, + .u.evtchn.flags =3D 0, + .u.evtchn.deliver.port.port =3D EVTCHN_PORT, + .u.evtchn.deliver.port.vcpu =3D vcpu->id, + .u.evtchn.deliver.port.priority =3D KVM_IRQ_ROUTING_XEN_EVTCHN_PRIO_2LEV= EL, + }; + vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &evt); + + kvm_pthread_create(&thread, NULL, memslot_churn, vm); + + /* + * Self-guard: on a kernel with the deadlock under test, the vCPU + * never returns from KVM_RUN. Die with SIGALRM rather than hang + * when run standalone. Deliberately no SIGALRM handler: in the + * deadlock, no thread ever returns to userspace to run one, and a + * handled signal is no longer fatal, so only the default + * disposition can kill the process (exit by SIGALRM is the + * failure signature). One budget for the whole run =E2=80=94 a healthy + * kernel finishes in seconds, and a per-iteration bound would + * impose a KVM_RUN latency claim this test doesn't make. + * (On a CONFIG_DEBUG_LOCK_ALLOC kernel which has regressed, the + * wait degrades to a transient failure instead of deadlocking, + * and this test cannot see that; the WARN it fires can.) + */ + alarm(90); + + while (!done) { + vcpu_run(vcpu); + + /* + * In the window where the shared_info slot is fully deleted + * (between the churn thread's DELETE and re-CREATE), the + * delivery fails with -EFAULT and the hypercall bounces to + * userspace as KVM_EXIT_XEN. Complete it with an error + * result and keep going; the failure mode under test is + * the vCPU never coming back (deadlock -> timeout), not + * the hypercall rc. + */ + if (vcpu->run->exit_reason =3D=3D KVM_EXIT_XEN && + vcpu->run->xen.type =3D=3D KVM_EXIT_XEN_HCALL) { + TEST_ASSERT(vcpu->run->xen.u.hcall.input =3D=3D + __HYPERVISOR_event_channel_op && + vcpu->run->xen.u.hcall.params[0] =3D=3D + EVTCHNOP_send, + "Unexpected hypercall bounce: input %llu params[0] %llu", + vcpu->run->xen.u.hcall.input, + vcpu->run->xen.u.hcall.params[0]); + vcpu->run->xen.u.hcall.result =3D -EFAULT; + bounced++; + continue; + } + + TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO); + + switch (get_ucall(vcpu, &uc)) { + case UCALL_SYNC: + /* + * The final sync reports how many of the sends were + * delivered: purely a diagnostic for how much of the + * churn the guest actually raced with. + */ + if (uc.args[1] >=3D TEST_ITERATIONS) + pr_info("%lu of %u sends delivered, %d bounced\n", + uc.args[1] - TEST_ITERATIONS, + TEST_ITERATIONS, bounced); + break; + case UCALL_DONE: + done =3D 1; + break; + case UCALL_ABORT: + REPORT_GUEST_ASSERT(uc); + /* NOT REACHED */ + default: + TEST_FAIL("Unexpected ucall %lu", uc.cmd); + } + } + + atomic_store(&churn_stop, true); + kvm_pthread_join(thread, NULL); + alarm(0); + + kvm_vm_free(vm); + return 0; +} --=20 2.43.0 From nobody Thu Nov 5 05:12:28 2026 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=90.155.92.199 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793844078; b=Ab/CvqPQoJKQY+TfIIbE9fXMchy8tzTl6GA36bqv2f9rnce+v5ugacyFPRJcMmFvHh08 JhXq2JrAAHRCxp+hv51ZiEx7x4bNx3LjFzfAuZKRIG2UDXxmKAsVAAggkwdkq9IYn9kKm I7MAek+1NR66jaEAqpqGI9+AlpFOe0wa4/yT/LdR26DnftVCzw2IJWkU0W4iAq0zHGL/A +68ZqURYV4oOhYZEk6aSa2Uzgf6o6LAREq020KpNFK2fZgWwQDnSkZ8MRrTHDl62UxOGw Oi+1L2o3b37DByuA8En0T+SxrZBFiILmM4tuzIwa8ZPStkBGExioS4VkyaPsf9u2JSw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793844078; h=DMARC-Filter:DKIM-Signature:Received:Received:From:To:Cc:Subject: Date:Message-ID:X-Mailer:In-Reply-To:References:MIME-Version: Content-Type:Content-Transfer-Encoding:Sender:X-SRS-Rewrite; bh=5928Fm1r4SKQS+jH4OMcVcprvnhzKwbhu15jXKH4aPQ=; b=42glvD8ktPV2R4qacoM8o5mqYcQnIDSkm1MoOI8L/0ow2N0nX5hFeFoG72x4diJe5/Ed L0rwy9JCKZt0TJPyf2SfgXmyidFKj22o3ekZkLwr5a9Zn9roo/Jb0ScSeL+cUkXIxIIto 9iYPsjMqm4jst0eYP0DxrDbWBX3OOgZ4xvhs9pCwHeiUncGw2Y8W74OoTMPmvDqAi8vtw EjW0oiAMwJ9fQ8JlaDo4cVxwnM/PIi6qVPCLFUTrDT0YGwHRUfIGJfLN3FU/KUpo50fGw J/BMPujc1Ld2sf6K+oXP46q+W/Z7wPRmwgKw9tezLL4C0acDk1dKFQAsGJ5YrEv8SJA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass header.from=infradead.org; spf=none smtp.mailfrom=desiato.srs.infradead.org; dkim=pass header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=desiato.20200630 header.b=c+cJAlqC; arc=none smtp.remote-ip=90.155.92.199 Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0EE68606EEA9; Thu, 5 Nov 2026 02:01:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.2 smtp.subspace.kernel.org 0EE68606EEA9 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=infradead.org Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=desiato.srs.infradead.org Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key, unprotected) header.d=infradead.org header.i=@infradead.org header.a=rsa-sha256 header.s=desiato.20200630 header.b=c+cJAlqC DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description; bh=5928Fm1r4SKQS+jH4OMcVcprvnhzKwbhu15jXKH4aPQ=; b=c+cJAlqCYvwVUBT4ZnFxngvmEg j06UhJBCENC+EWoZ8uxTBe1BBkm/K3xpWMIPcGenCA3LmH+U9Q/Dg5bWoQ54giJNHszlHgrZ/nmot dhX1Ys8wR7p0RlzClAALbcdELVnNJ+n8ApVB4ktMzDm28gJUocHY6GxrbkEN8ILJRpbL4N90IN/wv KEfe9NuUJTyFIeG5ezpKPKjEKxQNKWJlzfO9vpntoZ27Nlk2dRBasRMfGfzsrAVa0oU66QBYPA3ou HkHH3OBDWOEl+/9HKxOSoRtrVciGSUmi9AVPYqFy0schWzdIpdFTNN1CCTuLrEv1Jf2heYs55ZGDh Yd1RUHlQ==; Received: from [2001:8b0:10b:1::425] (helo=i7.infradead.org) by desiato.infradead.org with esmtpsa (Exim 4.100.1 #2 (Red Hat Linux)) id 1xOmmo-00000002ROS-24dv; Thu, 05 Nov 2026 02:01:08 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.99.5 #2 (Red Hat Linux)) id 1xOmmn-0000000AWEJ-0E2y; Thu, 05 Nov 2026 02:01:05 +0000 From: David Woodhouse To: kvm@vger.kernel.org Cc: linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, rcu@vger.kernel.org, linux-rt-devel@lists.linux.dev, Paolo Bonzini , Sean Christopherson , Paul Durrant , Vitaly Kuznetsov , Fred Griffoul , "Paul E . McKenney" , Kunwu Chan , Kunwu Chan , syzbot+208f7f3e5f59c11aeb90@syzkaller.appspotmail.com, David Woodhouse Subject: [PATCH v2 20/20] KVM: selftests: Add Xen evtchn vs mmu_notifier race test for HVA caches Date: Thu, 5 Nov 2026 02:01:02 +0000 Message-ID: <20261105020104.2507083-21-dwmw2@infradead.org> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20261105020104.2507083-1-dwmw2@infradead.org> References: <20261105020104.2507083-1-dwmw2@infradead.org> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Sender: David Woodhouse X-SRS-Rewrite: SMTP reverse-path rewritten from by desiato.infradead.org. See http://www.infradead.org/rpr.html From: David Woodhouse Race a guest EVTCHNOP_send loop against a host thread hammering MADV_DONTNEED on the userspace page backing HVA-based shared_info and vcpu_info caches. Every madvise zaps the PTE and invalidates both gfn_to_pfn_caches; every delivery must refresh through the window (hva_to_pfn_retry() retries until the invalidation has finished, then faults the page back in) and must succeed. The caches are HVA-based, so no memslot semantics excuse a drop: unlike xen_evtchn_memslot_race_test, which only pins the no-deadlock claim, this test asserts every one of the 20000 sends delivers, and any hypercall bounce to userspace fails the test. This exercises the pfncache invalidation protocol itself =E2=80=94 the invalidate_start walk, the reader draining for a cache mid-teardown, and the handshake between a refresh publishing a new mapping and a concurrent invalidation vetoing it =E2=80=94 which is where a lifetime bug in the cache would live; on a debug kernel one is expected to surface as a splat rather than a dropped event. Signed-off-by: David Woodhouse --- tools/testing/selftests/kvm/Makefile.kvm | 1 + .../kvm/x86/xen_evtchn_hva_notify_race_test.c | 188 ++++++++++++++++++ 2 files changed, 189 insertions(+) create mode 100644 tools/testing/selftests/kvm/x86/xen_evtchn_hva_notify_r= ace_test.c diff --git a/tools/testing/selftests/kvm/Makefile.kvm b/tools/testing/selft= ests/kvm/Makefile.kvm index c17408f8d61f..19dc0744f25f 100644 --- a/tools/testing/selftests/kvm/Makefile.kvm +++ b/tools/testing/selftests/kvm/Makefile.kvm @@ -150,6 +150,7 @@ TEST_GEN_PROGS_x86 +=3D x86/debug_regs TEST_GEN_PROGS_x86 +=3D x86/tsc_msrs_test TEST_GEN_PROGS_x86 +=3D x86/vmx_pmu_caps_test TEST_GEN_PROGS_x86 +=3D x86/xen_evtchn_memslot_race_test +TEST_GEN_PROGS_x86 +=3D x86/xen_evtchn_hva_notify_race_test TEST_GEN_PROGS_x86 +=3D x86/xen_shinfo_test TEST_GEN_PROGS_x86 +=3D x86/xen_vmcall_test TEST_GEN_PROGS_x86 +=3D x86/sev_dbg_test diff --git a/tools/testing/selftests/kvm/x86/xen_evtchn_hva_notify_race_tes= t.c b/tools/testing/selftests/kvm/x86/xen_evtchn_hva_notify_race_test.c new file mode 100644 index 000000000000..c105435f6357 --- /dev/null +++ b/tools/testing/selftests/kvm/x86/xen_evtchn_hva_notify_race_test.c @@ -0,0 +1,188 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * xen_evtchn_hva_notify_race_test + * + * A guest EVTCHNOP_send loop racing continuous mmu_notifier + * invalidation of the HVA-based shared_info and vcpu_info caches: a + * host thread hammers MADV_DONTNEED on the userspace page backing + * both, zapping the PTE and invalidating the gfn_to_pfn_caches on + * every pass. Delivery must refresh through each invalidation window + * (hva_to_pfn_retry() retries until the notifier has finished, then + * faults the page back in) and must never fail: the caches are + * HVA-based, so no memslot semantics excuse a drop. + * + * This is the stress for the pfncache invalidation protocol itself =E2=80= =94 + * the invalidate_start walk, the GPC_INVALIDATING grace-period + * draining, and the announce/veto handshake between a refresh's + * publish and a concurrent invalidation. Unlike the sibling + * xen_evtchn_memslot_race_test, every send is asserted to succeed; + * any -EFAULT bounce to userspace, or any guest-visible failure, is + * a test failure, and a lifetime bug is expected to show up as a + * crash or splat on a debug kernel. + */ +#include +#include +#include + +#include "test_util.h" +#include "kvm_util.h" +#include "processor.h" + +#include + +#define EVTCHN_PORT 15 +#define TEST_ITERATIONS 5000 + +#define EVTCHNOP_send 4 +#define __HYPERVISOR_event_channel_op 32 + +#define XEN_HYPERCALL_MSR 0x40000000 + +#define EVTCHNSTAT_ipi 5 + +struct evtchn_send { + uint32_t port; +}; + +static void guest_code(void) +{ + struct evtchn_send s =3D { .port =3D EVTCHN_PORT }; + uint64_t rc; + int i; + + for (i =3D 0; i < TEST_ITERATIONS; i++) { + rc =3D __xen_hypercall(__HYPERVISOR_event_channel_op, + EVTCHNOP_send, &s); + __GUEST_ASSERT(!rc, + "EVTCHNOP_send failed with %ld on iteration %d", + (long)rc, i); + GUEST_SYNC(i); + } + + GUEST_DONE(); +} + +static atomic_bool churn_stop; + +static void *backing_churn(void *arg) +{ + void *backing =3D arg; + + while (!atomic_load(&churn_stop)) + madvise(backing, PAGE_SIZE, MADV_DONTNEED); + + return NULL; +} + +int main(int argc, char *argv[]) +{ + struct kvm_vcpu *vcpu; + struct kvm_vm *vm; + pthread_t thread; + struct ucall uc; + void *backing; + int done =3D 0; + + int xen_caps =3D kvm_check_cap(KVM_CAP_XEN_HVM); + + TEST_REQUIRE(xen_caps & KVM_XEN_HVM_CONFIG_SHARED_INFO_HVA); + TEST_REQUIRE(xen_caps & KVM_XEN_HVM_CONFIG_EVTCHN_SEND); + + vm =3D vm_create_with_one_vcpu(&vcpu, guest_code); + + /* + * The shared_info page and the vcpu_info within it live in plain + * anonymous VMM memory, not in any memslot: nothing but the + * mmu_notifier churn below can perturb the caches. + */ + backing =3D mmap(NULL, PAGE_SIZE, PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS, -1, 0); + TEST_ASSERT(backing !=3D MAP_FAILED, "mmap of backing page failed"); + + struct kvm_xen_hvm_config hvmc =3D { + .flags =3D KVM_XEN_HVM_CONFIG_INTERCEPT_HCALL | + KVM_XEN_HVM_CONFIG_EVTCHN_SEND, + .msr =3D XEN_HYPERCALL_MSR, + }; + vm_ioctl(vm, KVM_XEN_HVM_CONFIG, &hvmc); + + struct kvm_xen_hvm_attr lm =3D { + .type =3D KVM_XEN_ATTR_TYPE_LONG_MODE, + .u.long_mode =3D 1, + }; + vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &lm); + + struct kvm_xen_hvm_attr ha =3D { + .type =3D KVM_XEN_ATTR_TYPE_SHARED_INFO_HVA, + .u.shared_info.hva =3D (unsigned long)backing, + }; + vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &ha); + + struct kvm_xen_vcpu_attr vi =3D { + .type =3D KVM_XEN_VCPU_ATTR_TYPE_VCPU_INFO_HVA, + .u.hva =3D (unsigned long)backing + 0x40, + }; + vcpu_ioctl(vcpu, KVM_XEN_VCPU_SET_ATTR, &vi); + + /* Loop the port back to this vCPU as an IPI. */ + struct kvm_xen_hvm_attr evt =3D { + .type =3D KVM_XEN_ATTR_TYPE_EVTCHN, + .u.evtchn.send_port =3D EVTCHN_PORT, + .u.evtchn.type =3D EVTCHNSTAT_ipi, + .u.evtchn.flags =3D 0, + .u.evtchn.deliver.port.port =3D EVTCHN_PORT, + .u.evtchn.deliver.port.vcpu =3D vcpu->id, + .u.evtchn.deliver.port.priority =3D KVM_IRQ_ROUTING_XEN_EVTCHN_PRIO_2LEV= EL, + }; + vm_ioctl(vm, KVM_XEN_HVM_SET_ATTR, &evt); + + kvm_pthread_create(&thread, NULL, backing_churn, backing); + + /* + * Self-guard against a kernel which livelocks a refresh under + * continuous invalidation. Deliberately no SIGALRM handler, as in + * xen_evtchn_memslot_race_test: only the default disposition can + * kill a process whose vCPU thread never returns to userspace. + * Empirically ~100-170 iterations/sec on a c6a.metal host: nearly + * every send pays a full refresh (fault the page back in and remap + * it), so budget generously; the healthy runtime is ~30-50s. + */ + alarm(240); + + while (!done) { + vcpu_run(vcpu); + + /* + * An HVA-based cache has no memslot to be mid-update: + * there is no legitimate reason for delivery to fail, so + * unlike the memslot race test, a hypercall bouncing to + * userspace is a straight failure. + */ + TEST_ASSERT(vcpu->run->exit_reason !=3D KVM_EXIT_XEN, + "EVTCHNOP_send bounced to userspace: delivery failed"); + + TEST_ASSERT_KVM_EXIT_REASON(vcpu, KVM_EXIT_IO); + + switch (get_ucall(vcpu, &uc)) { + case UCALL_SYNC: + if (uc.args[1] && !(uc.args[1] % 1000)) + pr_info("progress: %lu iterations\n", uc.args[1]); + break; + case UCALL_DONE: + done =3D 1; + break; + case UCALL_ABORT: + REPORT_GUEST_ASSERT(uc); + /* NOT REACHED */ + default: + TEST_FAIL("Unexpected ucall %lu", uc.cmd); + } + } + + atomic_store(&churn_stop, true); + kvm_pthread_join(thread, NULL); + alarm(0); + + kvm_vm_free(vm); + return 0; +} --=20 2.43.0