From nobody Fri Apr 17 00:17:57 2026 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 15B1E251795 for ; Wed, 25 Feb 2026 01:20:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982456; cv=none; b=FyXCYCzTt/S/h1OjQmqxIo+9rg2agnIdfbiLUGvN4KvG/ccDk/wSOYiDV87t4eQco0q47WFLcw6HG7PL8uc6W0jfGY/PKkNBgfk5uvyFIJKX+R3mFBeu8BH20luMgIxMKC01Jd5HY2kbmWo9ai9UMN3XUgCpfEC4PfZwCVK2YOM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982456; c=relaxed/simple; bh=JQvjxr/hWlUh39Ucu5ME+KW8AVQFVGTcKHfOZ9vH9Vs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=IlWMaJjwM1aP6oV2+3XX+Ya8ompgbzehjiSCHhI7e+17W/yLPxxxliVABJ2i6uVsJ/O0FE4DPDf/TfPGfKM5YNa1Xgi5yCM8ZGJ1zqmwyTxKYmCSBmOsYpMiMUd2pnsN89Cp35FnAsUXJHJ07oaP83RI3bmANaQy4JRUKB29SvU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WjkuWGFG; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WjkuWGFG" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-35842aa350fso32790765a91.0 for ; Tue, 24 Feb 2026 17:20:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1771982453; x=1772587253; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=AbvWdZpYsKEAzpHKAosx3SnBmphxJwW/5cCaeIJUZho=; b=WjkuWGFG8vtXhPII1eZvJ06P7Kd+eTHns2AtHWp9ukdOLzfwcDJNCXASqaq0917B7G 9Skkr7yQeS6EljUglDSjt10o7eagm4XIaj927ZRqF231yXAiUWEIbuZ9cSuhf1olWbUL NWDkrR8V17UOoUAHfLLr0NYrXgzQSX2Fo/yh1ahwHeTsL+KQ72GJCDQcR1iG2IJTdPY3 6ZY2+OahVJVuGkuZtLpg2BCKv4As8vx+gogaF//auLCoqkyEe2n5pIkdUkHdLYYWECQP bcirZYOLPEE5yg0iTgqui39n0AiyuvhVA2KZRAKGFSxU4RCB2XrpcLWO9+BuOdjlG6Pd VKKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771982453; x=1772587253; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=AbvWdZpYsKEAzpHKAosx3SnBmphxJwW/5cCaeIJUZho=; b=LEvNqsBCMf2W6ToYrBMgpur8UoimmwbSf072s57yF2M3QutOi1/i4gVTkVAdhiJSX6 OWgBwYkjX/cmEeHXK3e6UFG5BKDAA16biT561qTsdDHopssRNH8G3JayB93qO7WM0Yg/ cYminP66e3PLShjSuRWRcW9REd6XdE4KmpKKsw8zfa4Ej+4LIurHYKpKhzw4z7xRMeCy 6yS0Xk3cKfpZRxx31FY+Ndoccyf6p8jU1l/gJ95zDJ2zpzAkpwKPwHT4o7i87o5NPKwW 43JPWCipkTM5EWq1igCDSbmL7fPLVq4hTEv6cYpgDPaBk0ImoEPOzO9CT66uizcmvxVL agKw== X-Forwarded-Encrypted: i=1; AJvYcCUPp6MJOKpsReZY5AL8jRzdIe4ETKpj7lEVy7B6lsXs0jDIuoer3eYcEmPp52480jJe9ri/DwjY8HsyFCU=@vger.kernel.org X-Gm-Message-State: AOJu0YyX4ShiHJ1VGLXtPmKoPaGqscqfggGHpiavZiprGvbNLbSYiprb x/mEFPK5cusCDAZ/RsF1pOOfLq1cXScBm/DaGXO8VvbgMgrnV4v9nAaaHOyPKWHGAgNunRPZkoF SmAdPug== X-Received: from pjzl20.prod.google.com ([2002:a17:90b:794:b0:356:1f53:fad5]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:3941:b0:356:282e:7eb7 with SMTP id 98e67ed59e1d1-358ae810f45mr11868323a91.12.1771982453280; Tue, 24 Feb 2026 17:20:53 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 24 Feb 2026 17:20:36 -0800 In-Reply-To: <20260225012049.920665-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260225012049.920665-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.414.gf7e9f6c205-goog Message-ID: <20260225012049.920665-2-seanjc@google.com> Subject: [PATCH 01/14] KVM: x86: Use scratch field in MMIO fragment to hold small write values From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, Yashu Zhang , Rick Edgecombe , Binbin Wu , Xiaoyao Li , Tom Lendacky , Michael Roth Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When exiting to userspace to service an emulated MMIO write, copy the to-be-written value to a scratch field in the MMIO fragment if the size of the data payload is 8 bytes or less, i.e. can fit in a single chunk, instead of pointing the fragment directly at the source value. This fixes a class of use-after-free bugs that occur when the emulator initiates a write using an on-stack, local variable as the source, the write splits a page boundary, *and* both pages are MMIO pages. Because KVM's ABI only allows for physically contiguous MMIO requests, accesses that split MMIO pages are separated into two fragments, and are sent to userspace one at a time. When KVM attempts to complete userspace MMIO in response to KVM_RUN after the first fragment, KVM will detect the second fragment and generate a second userspace exit, and reference the on-stack variable. The issue is most visible if the second KVM_RUN is performed by a separate task, in which case the stack of the initiating task can show up as truly freed data. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D BUG: KASAN: use-after-free in complete_emulated_mmio+0x305/0x420 Read of size 1 at addr ffff888009c378d1 by task syz-executor417/984 CPU: 1 PID: 984 Comm: syz-executor417 Not tainted 5.10.0-182.0.0.95.h2627= .eulerosv2r13.x86_64 #3 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.15.0-0-= g2dd4b9b3f840-prebuilt.qemu.org 04/01/2014 Call Trace: dump_stack+0xbe/0xfd print_address_description.constprop.0+0x19/0x170 __kasan_report.cold+0x6c/0x84 kasan_report+0x3a/0x50 check_memory_region+0xfd/0x1f0 memcpy+0x20/0x60 complete_emulated_mmio+0x305/0x420 kvm_arch_vcpu_ioctl_run+0x63f/0x6d0 kvm_vcpu_ioctl+0x413/0xb20 __se_sys_ioctl+0x111/0x160 do_syscall_64+0x30/0x40 entry_SYSCALL_64_after_hwframe+0x67/0xd1 RIP: 0033:0x42477d Code: <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b0 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007faa8e6890e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 00000000004d7338 RCX: 000000000042477d RDX: 0000000000000000 RSI: 000000000000ae80 RDI: 0000000000000005 RBP: 00000000004d7330 R08: 00007fff28d546df R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 00000000004d733c R13: 0000000000000000 R14: 000000000040a200 R15: 00007fff28d54720 The buggy address belongs to the page: page:0000000029f6a428 refcount:0 mapcount:0 mapping:0000000000000000 inde= x:0x0 pfn:0x9c37 flags: 0xfffffc0000000(node=3D0|zone=3D1|lastcpupid=3D0x1fffff) raw: 000fffffc0000000 0000000000000000 ffffea0000270dc8 0000000000000000 raw: 0000000000000000 0000000000000000 00000000ffffffff 0000000000000000 = page dumped because: kasan: bad access detected Memory state around the buggy address: ffff888009c37780: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ffff888009c37800: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff >ffff888009c37880: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ^ ffff888009c37900: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ffff888009c37980: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D The bug can also be reproduced with a targeted KVM-Unit-Test by hacking KVM to fill a large on-stack variable in complete_emulated_mmio(), i.e. by overwrite the data value with garbage. Limit the use of the scratch fields to 8-byte or smaller accesses, and to just writes, as larger accesses and reads are not affected thanks to implementation details in the emulator, but add a sanity check to ensure those details don't change in the future. Specifically, KVM never uses on-stack variables for accesses larger that 8 bytes, e.g. uses an operand in the emulator context, and *all* reads are buffered through the mem_read cache. Note! Using the scratch field for reads is not only unnecessary, it's also extremely difficult to handle correctly. As above, KVM buffers all reads through the mem_read cache, and heavily relies on that behavior when re-emulating the instruction after a userspace MMIO read exit. If a read splits a page, the first page is NOT an MMIO page, and the second page IS an MMIO page, then the MMIO fragment needs to point at _just_ the second chunk of the destination, i.e. its position in the mem_read cache. Taking the "obvious" approach of copying the fragment value into the destination when re-emulating the instruction would clobber the first chunk of the destination, i.e. would clobber the data that was read from guest memory. Fixes: f78146b0f923 ("KVM: Fix page-crossing MMIO") Suggested-by: Yashu Zhang Reported-by: Yashu Zhang Closes: https://lore.kernel.org/all/369eaaa2b3c1425c85e8477066391bc7@huawei= .com Cc: stable@vger.kernel.org Signed-off-by: Sean Christopherson Tested-by: Rick Edgecombe Tested-by: Tom Lendacky --- arch/x86/kvm/x86.c | 14 +++++++++++++- include/linux/kvm_host.h | 3 ++- 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index db3f393192d9..ff3a6f86973f 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -8226,7 +8226,13 @@ static int emulator_read_write_onepage(unsigned long= addr, void *val, WARN_ON(vcpu->mmio_nr_fragments >=3D KVM_MAX_MMIO_FRAGMENTS); frag =3D &vcpu->mmio_fragments[vcpu->mmio_nr_fragments++]; frag->gpa =3D gpa; - frag->data =3D val; + if (write && bytes <=3D 8u) { + frag->val =3D 0; + frag->data =3D &frag->val; + memcpy(&frag->val, val, bytes); + } else { + frag->data =3D val; + } frag->len =3D bytes; return X86EMUL_CONTINUE; } @@ -8241,6 +8247,9 @@ static int emulator_read_write(struct x86_emulate_ctx= t *ctxt, gpa_t gpa; int rc; =20 + if (WARN_ON_ONCE((bytes > 8u || !ops->write) && object_is_on_stack(val))) + return X86EMUL_UNHANDLEABLE; + if (ops->read_write_prepare && ops->read_write_prepare(vcpu, val, bytes)) return X86EMUL_CONTINUE; @@ -11847,6 +11856,9 @@ static int complete_emulated_mmio(struct kvm_vcpu *= vcpu) frag++; vcpu->mmio_cur_fragment++; } else { + if (WARN_ON_ONCE(frag->data =3D=3D &frag->val)) + return -EIO; + /* Go forward to the next mmio piece. */ frag->data +=3D len; frag->gpa +=3D len; diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 2c7d76262898..0bb2a34fb93d 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -320,7 +320,8 @@ static inline bool kvm_vcpu_can_poll(ktime_t cur, ktime= _t stop) struct kvm_mmio_fragment { gpa_t gpa; void *data; - unsigned len; + u64 val; + unsigned int len; }; =20 struct kvm_vcpu { --=20 2.53.0.414.gf7e9f6c205-goog From nobody Fri Apr 17 00:17:57 2026 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F34A3239099 for ; Wed, 25 Feb 2026 01:20:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982457; cv=none; b=EQBY3y501ZXYtdcuj3yfeCbinJLAePSVBEP32juHsufDfhZx3Uv/mg5qi93YnJdpLwOYA5PxkMaogFvWttvaC9+h64k2Qqca1qqCErnce5JzsAqsqObJuCrcDXGVQpEls42nHjShQIsWnINCsBRUKgl6pU+OroAaDOWzH3abypY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982457; c=relaxed/simple; bh=ueMQ7lNZG1mNPcnmXsV6s6mz7PApwRZnhIghYU/HyV0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=XnjNouQmJ0NCC97IblVgblR8yPTHM7Zxl2NvanVL9iCuhQtrRq35JugkZS2zkx2EaDpJc8jESN7FYg6TmE0/Wbsu2LKcNtsYb4myBUgF9U5VNpM3GXQ3Lca/Jm4xWqusLhpF2a9sSO2H8l/JykPhZXr9FzmZZV0Mgt5++1PD3E0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=aPq45hIx; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="aPq45hIx" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-358e425c261so1340444a91.3 for ; Tue, 24 Feb 2026 17:20:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1771982455; x=1772587255; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=ufrjZKfuPvWxhnZ9cZ27jbL/GsnZnbCcINCZtRWeedI=; b=aPq45hIxscwrfC7Ky7Z8jMtyJKr5SaJQy3n40ZhvIlYO+rclSRHgHqXGmeVmqDJsF6 ykFP/pOZryLxPVnwzwF83YiwN66B1PlKna93alqio6gPdxL5S/mvqHD7DvMi5wLtIDv3 m5Ui5Bq/Sl1BVvbsiibu7fYlEHhsJYv/trbh99GqBf1SW1W2gzfXlCSDtmQkgeEskpZr dEcU/wuWeNLaOpCZfARW+uXtJiSXtuZ7IUdAk/8Ykym+DBTYJJeVabZnD1jpNgfESzM9 WoLurWulEbw8yiHZmxUL1Or7umWANhtRxkk6RAwT7S2uFm08dxEUwAJBSzBTP80RxGRL GMrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771982455; x=1772587255; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ufrjZKfuPvWxhnZ9cZ27jbL/GsnZnbCcINCZtRWeedI=; b=uKUCp4PBCP+aebmFokfN952nnG7M45FkxoDYhKM3NgTAbrDb9fGGAl39A3i9HbSnH5 U0hvq95JteNCHi0WtG1Gte776iMJdZdaYygowkrMrnOseRPf74SxNqzEZMNTBBC5d453 K78/1mS+fulKyBUXb3tMN5+3vEDPvgOahG6gm1hATSCud0od6qckSb4v9QSJOMgq6DOl osTnDK+KyB70srvwcE3e544+3OgwzWvQCSloDHVrZBAI37dXI4nfHHaFurD55h6O37lL 0y10YizYrj+av+KUO1/6iP5L+/Jzlnud+ORppauKF8O/7gZVj202oO+QQRi0mh5hYdcS 4O9Q== X-Forwarded-Encrypted: i=1; AJvYcCVaxlQOigYrE6yUnc4PzHVVIDVqLJB/Y6UEmYd24G/SEoEG2qM26ne+gZQSmdpnnds4YMqTCPXd5ed9DxI=@vger.kernel.org X-Gm-Message-State: AOJu0Yxj4h+y9oWPjctvXinCwAZ2PN2MFPytqf+pf6N/C6qvi5A7BvpF MD8S1EtBnwpiiK8AIYMXL8fPqlbAcbXdZYTMeqzG0r7nb0urRk/yQT1RZ/fwBOhIjvEpbfcU/QL up7ffxA== X-Received: from pjyr11.prod.google.com ([2002:a17:90a:e18b:b0:355:76b7:bf7]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90a:d887:b0:359:120f:d3aa with SMTP id 98e67ed59e1d1-359120fd628mr174902a91.14.1771982455286; Tue, 24 Feb 2026 17:20:55 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 24 Feb 2026 17:20:37 -0800 In-Reply-To: <20260225012049.920665-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260225012049.920665-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.414.gf7e9f6c205-goog Message-ID: <20260225012049.920665-3-seanjc@google.com> Subject: [PATCH 02/14] KVM: x86: Open code handling of completed MMIO reads in emulator_read_write() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, Yashu Zhang , Rick Edgecombe , Binbin Wu , Xiaoyao Li , Tom Lendacky , Michael Roth Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Open code the handling of completed MMIO reads instead of using an ops hook, as burying the logic behind a (likely RETPOLINE'd) indirect call, and with an unintuitive name, makes relatively straightforward code hard to comprehend. Opportunistically add comments to explain the dependencies between the emulator's mem_read cache and the MMIO read completion logic, as it's very easy to overlook the cache's role in getting the read data into the correct destination. No functional change intended. Signed-off-by: Sean Christopherson Tested-by: Rick Edgecombe Tested-by: Tom Lendacky --- arch/x86/kvm/emulate.c | 13 +++++++++++++ arch/x86/kvm/x86.c | 33 ++++++++++++++++----------------- 2 files changed, 29 insertions(+), 17 deletions(-) diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c index c8e292e9a24d..70850e591350 100644 --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -1297,12 +1297,25 @@ static int read_emulated(struct x86_emulate_ctxt *c= txt, int rc; struct read_cache *mc =3D &ctxt->mem_read; =20 + /* + * If the read gets a cache hit, simply copy the value from the cache. + * A "hit" here means that there is unused data in the cache, i.e. when + * re-emulating an instruction to complete a userspace exit, KVM relies + * on "no decode" to ensure the instruction is re-emulated in the same + * sequence, so that multiple reads are fulfilled in the correct order. + */ if (mc->pos < mc->end) goto read_cached; =20 if (KVM_EMULATOR_BUG_ON((mc->end + size) >=3D sizeof(mc->data), ctxt)) return X86EMUL_UNHANDLEABLE; =20 + /* + * Route all reads to the cache. This allows @dest to be an on-stack + * variable without triggering use-after-free if KVM needs to exit to + * userspace to handle an MMIO read (the MMIO fragment will point at + * the current location in the cache). + */ rc =3D ctxt->ops->read_emulated(ctxt, addr, mc->data + mc->end, size, &ctxt->exception); if (rc !=3D X86EMUL_CONTINUE) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index ff3a6f86973f..8b1f02cc8196 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -8109,8 +8109,6 @@ int emulator_write_phys(struct kvm_vcpu *vcpu, gpa_t = gpa, } =20 struct read_write_emulator_ops { - int (*read_write_prepare)(struct kvm_vcpu *vcpu, void *val, - int bytes); int (*read_write_emulate)(struct kvm_vcpu *vcpu, gpa_t gpa, void *val, int bytes); int (*read_write_mmio)(struct kvm_vcpu *vcpu, gpa_t gpa, @@ -8120,18 +8118,6 @@ struct read_write_emulator_ops { bool write; }; =20 -static int read_prepare(struct kvm_vcpu *vcpu, void *val, int bytes) -{ - if (vcpu->mmio_read_completed) { - trace_kvm_mmio(KVM_TRACE_MMIO_READ, bytes, - vcpu->mmio_fragments[0].gpa, val); - vcpu->mmio_read_completed =3D 0; - return 1; - } - - return 0; -} - static int read_emulate(struct kvm_vcpu *vcpu, gpa_t gpa, void *val, int bytes) { @@ -8167,7 +8153,6 @@ static int write_exit_mmio(struct kvm_vcpu *vcpu, gpa= _t gpa, } =20 static const struct read_write_emulator_ops read_emultor =3D { - .read_write_prepare =3D read_prepare, .read_write_emulate =3D read_emulate, .read_write_mmio =3D vcpu_mmio_read, .read_write_exit_mmio =3D read_exit_mmio, @@ -8250,9 +8235,23 @@ static int emulator_read_write(struct x86_emulate_ct= xt *ctxt, if (WARN_ON_ONCE((bytes > 8u || !ops->write) && object_is_on_stack(val))) return X86EMUL_UNHANDLEABLE; =20 - if (ops->read_write_prepare && - ops->read_write_prepare(vcpu, val, bytes)) + /* + * If the read was already completed via a userspace MMIO exit, there's + * nothing left to do except trace the MMIO read. When completing MMIO + * reads, KVM re-emulates the instruction to propagate the value into + * the correct destination, e.g. into the correct register, but the + * value itself has already been copied to the read cache. + * + * Note! This is *tightly* coupled to read_emulated() satisfying reads + * from the emulator's mem_read cache, so that the MMIO fragment data + * is copied to the correct chunk of the correct operand. + */ + if (!ops->write && vcpu->mmio_read_completed) { + trace_kvm_mmio(KVM_TRACE_MMIO_READ, bytes, + vcpu->mmio_fragments[0].gpa, val); + vcpu->mmio_read_completed =3D 0; return X86EMUL_CONTINUE; + } =20 vcpu->mmio_nr_fragments =3D 0; =20 --=20 2.53.0.414.gf7e9f6c205-goog From nobody Fri Apr 17 00:17:57 2026 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DD7262641FC for ; Wed, 25 Feb 2026 01:20:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982459; cv=none; b=VxIFI1T+eqq5uIc1YdNFib0e0pCv86mNYxViS7SbRQKpylnUtiKbmGZjrg3N7GVa/hSksIK20BRGqrh3Cat4dcVp9An7/sCPNWphh5oMCjoiMg0gVeoTERXE4Dg6agVpISDZyxnfaZ6Tl+QUVrHrRovvO67WpeN50rmuh9KMoMI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982459; c=relaxed/simple; bh=NEXZh6xUltoNADNu/zge14qIqr3iJw1QwwcpMaiHnHs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=OHm8W0FDskhYGrKn8Z4GMa937AV4HFNbQAduuGqh01FCT1sF9T5n/YVBZ4f3R22KxDVn73w/W5+bMSNHeqWbx5nOBmwA64DMvndaOT9tvgydrRRckcllHfNK19ht5/wYqYGVaw6/WhzPGjFisXbUEnBb3pu+i+PrK5i0tIQVW6g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=1idEPnTE; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="1idEPnTE" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2add182f21eso2045265ad.1 for ; Tue, 24 Feb 2026 17:20:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1771982457; x=1772587257; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=47Wsj1dtKii3bX7J+O5CZp3t4E2O0GQAfD/mQfvdoX0=; b=1idEPnTE7gESp2YjGpbht+zPS2l02rprNBsFR8SmusKZDBW5lbNcSDmSnR43NdW0Mg kNmP1qIi+VMc9ZOwo7iwnVlVsod4nIcAbOczmPs1ykvxqAsDIkCTKGDxyJXTIK3fZu6l DPfGimbgr7o9oPDKuHBMbFqkLrq6NL2GoRURRY2ZVuWhJVC/3uvk9zMXucptswTCrHZ9 2vZLaeBxnosd0xTP0I2e+h3oDDLooqZ8Rk6RCPV7jGZxVpF9zgLbbl4LX7NER5FTS1ai iWnw08x1D/e3q6z/ts5fk4o7aZj6EwB0psRKD1Rv1q1UWdVoL69WYUSjBLS0TX5bryPC 6+eg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771982457; x=1772587257; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=47Wsj1dtKii3bX7J+O5CZp3t4E2O0GQAfD/mQfvdoX0=; b=QxR6eBvb+nTBdzmHm4+v0lY9YyM3tVHQRcZ2dv/C1xRX/zS0SY8T8orwFXxbZqUeGI BIMBQa8cy+ult4CG5cmmaoDEjQBP+hgkDXIMwYKozfkL+e9XB8Rmc9coAyx5ARSGBhPr AJhWn9kV9aqZBQA+NCpj8uL1ZdLNrdiGfGv4mThuNlzfBNHWV6gU3zRxTRSfShaKLRnO pW6J0DpIKd72RJ3Qergg0bEMJW3x/JvnWZtmLY7lsfZaOKqcNh2/m2zxbeFRCGVS1foW 5PKlmsWl/EwWlhRKaH6B2p5YKil8aiSq6PToy15UsYm5k3/EBsqJa6+VgRgf9JM2vPva TQxA== X-Forwarded-Encrypted: i=1; AJvYcCWGztzPZd5smZ25D6O+xJDN6HdBAU7eeP7SE+vzIs3efxLzsUpOPRvHDgDjnjV6Im2uTgTheQ9K+UnyhLY=@vger.kernel.org X-Gm-Message-State: AOJu0Yw4hPfCyW8n5gxLtJXEJU2hY9mhdE5X+bkIJUEnBwBdiJvKYwGK kXK4var+dTSQ0ExmFzO/5czs6I84YB+806Wgx3KRpwoxmjPE2wvqBapH+kpIzGNGCzK7w2CTFCG vr0l7cA== X-Received: from pjvh24.prod.google.com ([2002:a17:90a:db98:b0:359:c21:c138]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:2352:b0:2a0:9d16:5fb4 with SMTP id d9443c01a7336-2add11e8c96mr6365125ad.18.1771982457153; Tue, 24 Feb 2026 17:20:57 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 24 Feb 2026 17:20:38 -0800 In-Reply-To: <20260225012049.920665-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260225012049.920665-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.414.gf7e9f6c205-goog Message-ID: <20260225012049.920665-4-seanjc@google.com> Subject: [PATCH 03/14] KVM: x86: Trace unsatisfied MMIO reads on a per-page basis From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, Yashu Zhang , Rick Edgecombe , Binbin Wu , Xiaoyao Li , Tom Lendacky , Michael Roth Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Invoke the "unsatisfied MMIO reads" when KVM first detects that a particular access "chunk" requires an exit to userspace instead of tracing the entire access at the time KVM initiates the exit to userspace. I.e. precisely trace the first and/or second fragments of a page split instead of tracing the entire access, as the GPA could be wrong on a page split case. Leave the completion tracepoint alone, at least for now, as fixing the completion path would incur significantly complexity to track exactly which fragment(s) of the overall access actually triggered MMIO, but add a comment that the tracing for completed reads in is technically wrong. Signed-off-by: Sean Christopherson Tested-by: Rick Edgecombe Tested-by: Tom Lendacky --- arch/x86/kvm/x86.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 8b1f02cc8196..a74ae3a81076 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -7808,6 +7808,9 @@ static int vcpu_mmio_read(struct kvm_vcpu *vcpu, gpa_= t addr, int len, void *v) v +=3D n; } while (len); =20 + if (len) + trace_kvm_mmio(KVM_TRACE_MMIO_READ_UNSATISFIED, len, addr, NULL); + return handled; } =20 @@ -8139,7 +8142,6 @@ static int write_mmio(struct kvm_vcpu *vcpu, gpa_t gp= a, int bytes, void *val) static int read_exit_mmio(struct kvm_vcpu *vcpu, gpa_t gpa, void *val, int bytes) { - trace_kvm_mmio(KVM_TRACE_MMIO_READ_UNSATISFIED, bytes, gpa, NULL); return X86EMUL_IO_NEEDED; } =20 @@ -8247,6 +8249,11 @@ static int emulator_read_write(struct x86_emulate_ct= xt *ctxt, * is copied to the correct chunk of the correct operand. */ if (!ops->write && vcpu->mmio_read_completed) { + /* + * For simplicity, trace the entire MMIO read in one shot, even + * though the GPA might be incorrect if there are two fragments + * that aren't contiguous in the GPA space. + */ trace_kvm_mmio(KVM_TRACE_MMIO_READ, bytes, vcpu->mmio_fragments[0].gpa, val); vcpu->mmio_read_completed =3D 0; --=20 2.53.0.414.gf7e9f6c205-goog From nobody Fri Apr 17 00:17:57 2026 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D3B6A257431 for ; Wed, 25 Feb 2026 01:21:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982462; cv=none; b=Yeq2sYt+sAq7I8IgWxMoHtFqGZn0jZpLnIWg+h40d7fJ2i46q6w1Z+VcJ0oGQsClU9WVHIjrvGhrTmR3CaRSdr9nyWz1wabA/tXtTaToppKVICb8a/czV9Pi48/4NVxELjzZKrobjKMJDRPWi6f2leGMFDV/R9yfKq/tQ1JdfbY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982462; c=relaxed/simple; bh=5WioRAn/exL/M/dDLMs+BGBP7zMGvVlgJ+hWWegWOB8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=lbFD4jW5mSd1XJZmeiSEZguUlCXf2z5//4wtUC5fOJuG71ZwkJElsVBxVEjpZ6FgxW+7xlxZBZYaZnVCuGR1KMDMmcIheXvGoRUy82MsLlugsbij26TzxOTb3AcN01JB0CRI/xIwHXDL0ZY3kdmhT/74wQIf2VufDQxbBkxJ/pg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=md6QiK/0; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="md6QiK/0" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-c6e18f1cb86so25185631a12.0 for ; Tue, 24 Feb 2026 17:21:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1771982460; x=1772587260; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=dv3TZbd1nBU/AEQJbeAhZb6gpTw6vQIp0AVXQNpKYls=; b=md6QiK/0BuVkI1hqEPStqXIqHSYp4VhzieptPk+8zWvamoimyR/caDCl3rbe25EPV+ Q4hqe/lkdD0svXec9h0QUbspkYjoqHfjVbjVuG6cJ1EXSiCq1KKBDSCHiF1vrgd4mymY ik6p5CEL/T+AdIs6J9fji0KRXU95Rsou1xZDodQDclANnK0PG2eDt1bT0vtXwQQZkU3P zsxHxHRRD3JVYl8AiZjwS9okXpi6UFWEyu3sAX/CxKXHpedljNcjjm+I0FoXVbPdchIb oYVezqLrlhxyAHMAeD2MRu3eyDalwBWhf91S+APbQqA6OLuLnWxRDxHlqkgsbm/k+HHg zufw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771982460; x=1772587260; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=dv3TZbd1nBU/AEQJbeAhZb6gpTw6vQIp0AVXQNpKYls=; b=X01qEtLtdd4DnvqJCGEXuDqVuekHRVVzOpDpjrxE7chr+T699WUBN9QgpM+O1sRFbN KjSn09YJcxEDNOZhaTBB3kRuzo4fuXWazr8Ok95DAI+gC46XPVWP089Uke17FrvuUOCh b3mqRJUQk8AiYk7a8LCR3BrmiI4S7aHppqM0X42B//O/ziLpu8OY7h22mo0eqhOjMhtm omXvdUWTyB3ZuKCdA/XiEzndnPTyq8xrdMwavMCTEM3+fjUiM76V6mNgQEINo/AsD7e7 yKWFVapmC17uXCx4q2Eak2m23C7xN1f9BkB1NDcvUn2vNHEYZ1MS5s4tyyrEFQKzoynx WhdQ== X-Forwarded-Encrypted: i=1; AJvYcCX4SE3/vxzhdSjUy0hvm/s+irnDJEN41KCJe1skZNMvxzeXU475bvaJOjEjHyQljE3RdiwahSd/WgWBbdM=@vger.kernel.org X-Gm-Message-State: AOJu0Yz/fdU6CTbRJ5bovbfoPkncwH9d27ONyJXQM8NpId8Eqj6Lid+l 0286q165mN/lSAT9ecluSh9dRqedPRknzCzNeqhjBj7xdrJUMhnHMpACrfvPQ87ioV6HuopppND OMXVIzw== X-Received: from pgbbq17.prod.google.com ([2002:a05:6a02:451:b0:c6e:288a:3ab9]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:496:b0:394:f4fd:b751 with SMTP id adf61e73a8af0-39545fc60c5mr11022872637.51.1771982460005; Tue, 24 Feb 2026 17:21:00 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 24 Feb 2026 17:20:39 -0800 In-Reply-To: <20260225012049.920665-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260225012049.920665-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.414.gf7e9f6c205-goog Message-ID: <20260225012049.920665-5-seanjc@google.com> Subject: [PATCH 04/14] KVM: x86: Use local MMIO fragment variable to clean up emulator_read_write() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, Yashu Zhang , Rick Edgecombe , Binbin Wu , Xiaoyao Li , Tom Lendacky , Michael Roth Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Grab the MMIO fragment used by emulator_read_write() to initiate an exit to userspace in a local variable to make the code easier to read. No functional change intended. Signed-off-by: Sean Christopherson Tested-by: Rick Edgecombe Tested-by: Tom Lendacky --- arch/x86/kvm/x86.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index a74ae3a81076..7cbd6f7d8578 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -8231,7 +8231,7 @@ static int emulator_read_write(struct x86_emulate_ctx= t *ctxt, const struct read_write_emulator_ops *ops) { struct kvm_vcpu *vcpu =3D emul_to_vcpu(ctxt); - gpa_t gpa; + struct kvm_mmio_fragment *frag; int rc; =20 if (WARN_ON_ONCE((bytes > 8u || !ops->write) && object_is_on_stack(val))) @@ -8287,17 +8287,16 @@ static int emulator_read_write(struct x86_emulate_c= txt *ctxt, if (!vcpu->mmio_nr_fragments) return X86EMUL_CONTINUE; =20 - gpa =3D vcpu->mmio_fragments[0].gpa; - vcpu->mmio_needed =3D 1; vcpu->mmio_cur_fragment =3D 0; =20 - vcpu->run->mmio.len =3D min(8u, vcpu->mmio_fragments[0].len); + frag =3D &vcpu->mmio_fragments[0]; + vcpu->run->mmio.len =3D min(8u, frag->len); vcpu->run->mmio.is_write =3D vcpu->mmio_is_write =3D ops->write; vcpu->run->exit_reason =3D KVM_EXIT_MMIO; - vcpu->run->mmio.phys_addr =3D gpa; + vcpu->run->mmio.phys_addr =3D frag->gpa; =20 - return ops->read_write_exit_mmio(vcpu, gpa, val, bytes); + return ops->read_write_exit_mmio(vcpu, frag->gpa, val, bytes); } =20 static int emulator_read_emulated(struct x86_emulate_ctxt *ctxt, --=20 2.53.0.414.gf7e9f6c205-goog From nobody Fri Apr 17 00:17:57 2026 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 731D82609C5 for ; Wed, 25 Feb 2026 01:21:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982465; cv=none; b=XgOBjX9pOF5wGlLQJfbPC8t+1VbKO1majdzbYfGiCI4n4t4r1bqUHAAZMLeuvwi/BGiNb4vrgWc+GX1zOGZFOLfnH+OVwmtd41D86c9ZRlBbZP39VpBCpsEYOa+pCDr0KUbIXS89sAiu32x/eMSea/oJHYz2rrJxedbLAP075Gg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982465; c=relaxed/simple; bh=iyxIPblZQF7VigUXYeAExOzxkERFEm+SJWV+/cinaYk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=creTHSWMOf9xf1tq/fMzz61OcGkai+2pn375b2WZLpJTlyVjRvWWSbo/yADwHGibFrf34CM9vOVtYInvl7akaKEgC2Mz+xpcDRSwFpl1EoOodGzSufplQL8g27wfgB2DewpUjixgk0iOC4m2gs3NoMgRvEuCIWQGZl4EYxsbL7Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=zYMeJgCP; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="zYMeJgCP" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-b630753cc38so34290126a12.1 for ; Tue, 24 Feb 2026 17:21:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1771982462; x=1772587262; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=iAM51bp4Q7vtLVTuwPEvx5LguTW6LfJ4+fu/BmQleqk=; b=zYMeJgCPC8e8ZZrN9bYnKZA0FtR5BhIDDC+9MUmF7IxvUihatePxZLes3O+KkV+QI1 byVnyhqlR1NKdBMY4HvbViqcDPi3pYV2HQlknMs8A20llalvf4RfQBMPpZ2ho1pw3nzc zWw5xtEniGJDVvfOYPO0tC5JhvurbxIJzFYM9oGuXEC5Qe+mQzflIqBUXerC4mbQfH8r iPtEo3MOHj1ZP130rUz1MeZi8EWiPnegbdfp2/T3Xeva4xToXCUp1Y+OEwEyq0QAWge4 C3S+mSoks4j6FFHwJLVl/L4gMO2WZEl4Ns6q27SNR/ztA0ZLNfL/GRJyE2LZ8oY8BPUd o0QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771982462; x=1772587262; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=iAM51bp4Q7vtLVTuwPEvx5LguTW6LfJ4+fu/BmQleqk=; b=e+3C/4GwFpUtUOtrNpDOlAAhROeNTdhFnsrKuCWsFwnn5WfSIBTFU5PakAPhWWtHev ccVmLDuXgzSpaPOn8071a5J5v/1hBFNMUJQ7tY0oLhoskcD4lBSOLyAFhyVlqjgYSPIt ZfU3jQhhTl36EsGEsujLhv7rlo6lvUp7vJ9I9nR8Tg/syYXHlPcHYD3b7jshzjvZznif A0X0ViOHGSOe3UTOKGviug88gVlcjywy68+FUW40TWWj1+c4VzZipLQ14igiRjYaMYkk yCN7Gy1UBqPtpwgq06btcC02QAXLCCwCHaEosQP4MTLrr2WE7O5KiyyPErEHLxl0gp1c Ncaw== X-Forwarded-Encrypted: i=1; AJvYcCWS9CFDkcQbfSVLqhQWxCmBULXnWhHclo7Y/WMs8XgptWcCX1VFODodjnrOBP3+Ww+R08D0o20JpzF19I0=@vger.kernel.org X-Gm-Message-State: AOJu0YyOTVQxc19RY+9L3bmAN7JkWJWt6P3HPWffprScilJJo+4BH9ij q3vdianqtneWRftqsH7wj2w3vw49BJs0pJh9PAswYfc92Q2Sa8E1Z55tXHSv9dcVAshVL6svX3c lNIrnLA== X-Received: from pgla15.prod.google.com ([2002:a63:b4f:0:b0:c07:89a0:6746]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:4c0e:b0:393:7575:a8d5 with SMTP id adf61e73a8af0-39545ed0443mr12719399637.22.1771982461746; Tue, 24 Feb 2026 17:21:01 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 24 Feb 2026 17:20:40 -0800 In-Reply-To: <20260225012049.920665-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260225012049.920665-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.414.gf7e9f6c205-goog Message-ID: <20260225012049.920665-6-seanjc@google.com> Subject: [PATCH 05/14] KVM: x86: Open code read vs. write userspace MMIO exits in emulator_read_write() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, Yashu Zhang , Rick Edgecombe , Binbin Wu , Xiaoyao Li , Tom Lendacky , Michael Roth Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Open code the differences in read vs. write userspace MMIO exits instead of burying three lines of code behind indirect callbacks, as splitting the logic makes it extremely hard to track that KVM's handling of reads vs. write is _significantly_ different. Add a comment to explain why the semantics are different, and how on earth an MMIO write ends up triggering an exit to userspace. No functional change intended. Signed-off-by: Sean Christopherson Tested-by: Rick Edgecombe Tested-by: Tom Lendacky --- arch/x86/kvm/x86.c | 33 +++++++++++++-------------------- 1 file changed, 13 insertions(+), 20 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 7cbd6f7d8578..5fde5bb010e7 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -8116,8 +8116,6 @@ struct read_write_emulator_ops { void *val, int bytes); int (*read_write_mmio)(struct kvm_vcpu *vcpu, gpa_t gpa, int bytes, void *val); - int (*read_write_exit_mmio)(struct kvm_vcpu *vcpu, gpa_t gpa, - void *val, int bytes); bool write; }; =20 @@ -8139,31 +8137,14 @@ static int write_mmio(struct kvm_vcpu *vcpu, gpa_t = gpa, int bytes, void *val) return vcpu_mmio_write(vcpu, gpa, bytes, val); } =20 -static int read_exit_mmio(struct kvm_vcpu *vcpu, gpa_t gpa, - void *val, int bytes) -{ - return X86EMUL_IO_NEEDED; -} - -static int write_exit_mmio(struct kvm_vcpu *vcpu, gpa_t gpa, - void *val, int bytes) -{ - struct kvm_mmio_fragment *frag =3D &vcpu->mmio_fragments[0]; - - memcpy(vcpu->run->mmio.data, frag->data, min(8u, frag->len)); - return X86EMUL_CONTINUE; -} - static const struct read_write_emulator_ops read_emultor =3D { .read_write_emulate =3D read_emulate, .read_write_mmio =3D vcpu_mmio_read, - .read_write_exit_mmio =3D read_exit_mmio, }; =20 static const struct read_write_emulator_ops write_emultor =3D { .read_write_emulate =3D write_emulate, .read_write_mmio =3D write_mmio, - .read_write_exit_mmio =3D write_exit_mmio, .write =3D true, }; =20 @@ -8296,7 +8277,19 @@ static int emulator_read_write(struct x86_emulate_ct= xt *ctxt, vcpu->run->exit_reason =3D KVM_EXIT_MMIO; vcpu->run->mmio.phys_addr =3D frag->gpa; =20 - return ops->read_write_exit_mmio(vcpu, frag->gpa, val, bytes); + /* + * For MMIO reads, stop emulating and immediately exit to userspace, as + * KVM needs the value to correctly emulate the instruction. For MMIO + * writes, continue emulating as the write to MMIO is a side effect for + * all intents and purposes. KVM will still exit to userspace, but + * after completing emulation (see the check on vcpu->mmio_needed in + * x86_emulate_instruction()). + */ + if (!ops->write) + return X86EMUL_IO_NEEDED; + + memcpy(vcpu->run->mmio.data, frag->data, min(8u, frag->len)); + return X86EMUL_CONTINUE; } =20 static int emulator_read_emulated(struct x86_emulate_ctxt *ctxt, --=20 2.53.0.414.gf7e9f6c205-goog From nobody Fri Apr 17 00:17:57 2026 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7209D256C84 for ; Wed, 25 Feb 2026 01:21:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982467; cv=none; b=ZZ2SOlMqPECZ40dck/Hk7rPej4/ylP3KF8WVM2HC/39S/FC4iz4JI7U7DleTw+6KRyOP12EaWopCayn7/b9+fINxyusmmO3iIPr1370fcgz4LpKg64eQ+JhKYf4ZIPy/ag455x81VKKxnc7SrcucoTnEvC3GgtLl1Ibr/Rv21Lk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982467; c=relaxed/simple; bh=7lRp2eCkoo+DRHWUNEDaUBTQXTHdWU47sEkYGqsesug=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=I96ol5D5DESaec3+Jj5FXZ85yE3KsSqgcXE1S+Dhr4j1ILNmO7dPu1ceLdE/KzdTaFLIHlnSuexKByL7l+UVK2btFTvBr7iurneweU0DiAcKp5qbhQ8gXjCS9nhGyk0gyxqkz+15SFSwavVNYqZebdnVC0ZLdn1WeB3OaG/HYIU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=t7HwgQyb; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="t7HwgQyb" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-3568090851aso39448817a91.1 for ; Tue, 24 Feb 2026 17:21:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1771982464; x=1772587264; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=gV7BZ60qmf3JAsxc6wDH0RQvnfJYSY1NcM7wseiHg1Y=; b=t7HwgQybqbbZUi30gFfENo1IvEUYBHKRNSWdodWfVKUDcAjRnPpqUKGyWoVAF53xDV i/tlYY+Z3bbE5sJQCfA2tOHRJJQ3ORd3NIUFhXHqrDVR/31dYplwHkzAWAPxnPe4qpnr g3sIEo+e5ICEsCxAeLZOIRXoYwM8s39Vya5zgW5THDZUZs5voTCkcr6AyVZmLTzkyDDa q01JqmMTaCNh2i2tNaLGe0rpLIlbC4/ygIwy6pMd+nDeL9miw2YMRdKKqlE+U4VRkU6W wy5PPXHD7g6HhSMSD4IFApFfQGbzhAjn3kej138uJ3WtRRIKu1MtvY4k3c+BGcS0JARQ OIdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771982464; x=1772587264; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=gV7BZ60qmf3JAsxc6wDH0RQvnfJYSY1NcM7wseiHg1Y=; b=bDV1+JdLTKq2AgatVrdu23H+3NyG27dfDMAICrrdEy+GEB6wr/hAaqDAF9G4SQ6VF6 wFE/fgha3vMbtGjJPpsu7vW9AMGycD4S308DOYiqcIEPD6moJ5paMIJHvn6J2l9F2s1Z zNdFhuH3YuoGlerXkVEwZRLghsSxppZdtqFCEshESJTM4r9XuC8nh9V0SKBGZkXytnRi YjWtS1+9YWkSjLP5xw6ZltOxoPv+KvEZBWvypXlONQV4gcK3sGYqN1XbOhK7lD1dkl7u /mmzOqOf/obchHAGTfyYMUViEp7P5iA8KK94Pp66Y1LqKzoU0ZUPCU9CerVTF0v1lYi1 dq5g== X-Forwarded-Encrypted: i=1; AJvYcCXZUruDcNdm/q7Rkk/JH8cAG0OcqsxZHbTXv3s52cjWVlHajdPu5fqR3MDAa92Wagc1mTN6OiFh018P/KI=@vger.kernel.org X-Gm-Message-State: AOJu0Yy1uPMItcC1i8D+1qJf2hFHk25brfR3dVctYL63FEvHrd36zXln R+s4gFKa4OUEIG5alMmg70CCw56goHHZKpYsjSoiz/u9P89crsSV/3RxbrIFGQj1V0/QB5mo9P/ 6gUqwNA== X-Received: from pjal2.prod.google.com ([2002:a17:90a:1502:b0:34a:bebf:c162]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:5285:b0:354:c629:efaf with SMTP id 98e67ed59e1d1-3590f26515dmr508907a91.35.1771982463612; Tue, 24 Feb 2026 17:21:03 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 24 Feb 2026 17:20:41 -0800 In-Reply-To: <20260225012049.920665-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260225012049.920665-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.414.gf7e9f6c205-goog Message-ID: <20260225012049.920665-7-seanjc@google.com> Subject: [PATCH 06/14] KVM: x86: Move MMIO write tracing into vcpu_mmio_write() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, Yashu Zhang , Rick Edgecombe , Binbin Wu , Xiaoyao Li , Tom Lendacky , Michael Roth Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move the invocation of MMIO write tracepoint into vcpu_mmio_write() and drop its largely-useless wrapper to cull pointless code and to make the code symmetrical with respect to vcpu_mmio_read(). No functional change intended. Signed-off-by: Sean Christopherson Tested-by: Rick Edgecombe Tested-by: Tom Lendacky --- arch/x86/kvm/x86.c | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 5fde5bb010e7..7abd6f93c386 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -7769,11 +7769,14 @@ static void kvm_init_msr_lists(void) } =20 static int vcpu_mmio_write(struct kvm_vcpu *vcpu, gpa_t addr, int len, - const void *v) + void *__v) { + const void *v =3D __v; int handled =3D 0; int n; =20 + trace_kvm_mmio(KVM_TRACE_MMIO_WRITE, len, addr, __v); + do { n =3D min(len, 8); if (!(lapic_in_kernel(vcpu) && @@ -8131,12 +8134,6 @@ static int write_emulate(struct kvm_vcpu *vcpu, gpa_= t gpa, return emulator_write_phys(vcpu, gpa, val, bytes); } =20 -static int write_mmio(struct kvm_vcpu *vcpu, gpa_t gpa, int bytes, void *v= al) -{ - trace_kvm_mmio(KVM_TRACE_MMIO_WRITE, bytes, gpa, val); - return vcpu_mmio_write(vcpu, gpa, bytes, val); -} - static const struct read_write_emulator_ops read_emultor =3D { .read_write_emulate =3D read_emulate, .read_write_mmio =3D vcpu_mmio_read, @@ -8144,7 +8141,7 @@ static const struct read_write_emulator_ops read_emul= tor =3D { =20 static const struct read_write_emulator_ops write_emultor =3D { .read_write_emulate =3D write_emulate, - .read_write_mmio =3D write_mmio, + .read_write_mmio =3D vcpu_mmio_write, .write =3D true, }; =20 --=20 2.53.0.414.gf7e9f6c205-goog From nobody Fri Apr 17 00:17:57 2026 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CA20F257431 for ; Wed, 25 Feb 2026 01:21:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982467; cv=none; b=fc3zRyYt3GDgDtBX9Sf0VgL+pZhX4DCY/+GlF2ijQFuZfT/EHx6dF47m9GCG63ADFhL1DSy1w1QPb2LTNVaD72/6PFuBPedjVb+15q3tqDxlYlp7dbznjwM6lEEGKkJHJedEuUA0smensQ5z5HJAAFfGAEl64OFEvFs47B0VQFc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982467; c=relaxed/simple; bh=YugX2YLyYIqesMyLVhe1vuRLpba0vyw2iEWUNwc9aeU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=H1qqqiwk3cgrEIKvciFJZdzFRUfVA1pDtkZOeE9aBTL3MILL0gh/IKjTl/V3icZQk+N15P06bu5VHHrxfskzgcfICzEQrd7Gf/Po7sGtHq4ta9Wj3u3JoPjQ3nzg3q/Itl7/rxboGOGDMqW7dKXOb3O++SIq53Ui5kCcJ2/SzAg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=EghX4vDZ; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="EghX4vDZ" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-3562370038dso5388247a91.3 for ; Tue, 24 Feb 2026 17:21:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1771982466; x=1772587266; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=U5hmNv02IBvGPnO8Neh+XMn/il3YSxAkvHfsbK9B4L0=; b=EghX4vDZyn+mqoAu5lR8XTM56GUhSs0EV/EXjz0SQCYmIUvAvFNu7Nx68/ESPD82MF djiQIsBNz7FcqVrApJEPDV7VYUAtaTIbRUSQoLeBgfAa64Nf1QmbLTwfLY4vf+eu+Rnw NBDt3P86dI0YRcwahUHq8VWcpX2O18SK7I6G8bW5rtHQUlY9MU/hSJ/IIVfP2+5HuSeO gJgaUBVJQrPtrqthmd7sPO4kAyKK7Mitc3ECvwiTJhPmlZYQC5Q08H6xUIOSeCEeJ94O 4b5yXvYKVnA4ox2n7ja8OuhpTLxjX44CcbQg4Smco5jW5QQJ9VPjiX7tOYxcZm4WCLv3 L1Bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771982466; x=1772587266; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=U5hmNv02IBvGPnO8Neh+XMn/il3YSxAkvHfsbK9B4L0=; b=QE/s/7605FOkFBlolyc6erXyACJ4C7fuRMLLj3ml6Dd1izbeoIwopbTcfW4jgrbxFL gv35czwsBb0GfXqxSPhHcYgpT106mNWm6yXcSsRGS+Q0zy9LBshhZHUWWj18oOWkd73K 4Hk8EEPD9LMzdt051IU8nHsTZYJn1uw5eQhvJ0k2EtX0VhJd0ZiFzO6QHmXTRiaA6DhE /68qu2PlG7JHouWTqNN41I4vuL0eXIMjw1qvB+cvMMv8OWJ5j5NBf9SZ5SlIxA7t7rGY UG80BU0I+bYdoDYSV1j6OgT5P0x2Bk/55CRgCUdwLVrpSw048fZN0gCFWz8CwxUHa0GW T2BA== X-Forwarded-Encrypted: i=1; AJvYcCX3gvPglOL03llPSgO0iJKP8LwNHX3DVEHcSWOakJ8RLn5aBJr7ejTX37JdGRBmjJCZYvLrpxGZa3uFh8o=@vger.kernel.org X-Gm-Message-State: AOJu0Yy5k9lG2mff9yhPVo1m6u9o7KUt+6D5IOB6XAXZVeM4AwgDwQCB 0V6/Rna+01JBxoXatscySaluhKou90uzkWILt/ZcxMZL/rfQRPNwp20lQQ4rzgIbPaXNEXyDeX5 mOyf89A== X-Received: from pjee13.prod.google.com ([2002:a17:90b:578d:b0:358:f01f:25f3]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:248f:b0:359:79e:39bb with SMTP id 98e67ed59e1d1-359079e3faemr996889a91.24.1771982465502; Tue, 24 Feb 2026 17:21:05 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 24 Feb 2026 17:20:42 -0800 In-Reply-To: <20260225012049.920665-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260225012049.920665-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.414.gf7e9f6c205-goog Message-ID: <20260225012049.920665-8-seanjc@google.com> Subject: [PATCH 07/14] KVM: x86: Harden SEV-ES MMIO against on-stack use-after-free From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, Yashu Zhang , Rick Edgecombe , Binbin Wu , Xiaoyao Li , Tom Lendacky , Michael Roth Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add a sanity check to ensure KVM doesn't use an on-stack variable when handling an MMIO request for an SEV-ES guest. The source/destination for SEV-ES MMIO should _always_ be the #VMGEXIT scratch area. Opportunistically update the comment in the completion side of things to clarify that frag->data doesn't need to be copied anywhere, and the VMEGEXIT is trap-like (the current comment doesn't clarify *how* RIP is advanced). Signed-off-by: Sean Christopherson Tested-by: Rick Edgecombe Tested-by: Tom Lendacky --- arch/x86/kvm/x86.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 7abd6f93c386..2db0bf738d2d 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -14273,8 +14273,10 @@ static int complete_sev_es_emulated_mmio(struct kv= m_vcpu *vcpu) if (vcpu->mmio_cur_fragment >=3D vcpu->mmio_nr_fragments) { vcpu->mmio_needed =3D 0; =20 - // VMG change, at this point, we're always done - // RIP has already been advanced + /* + * All done, as frag->data always points at the GHCB scratch + * area and VMGEXIT is trap-like (RIP is advanced by hardware). + */ return 1; } =20 @@ -14297,7 +14299,7 @@ int kvm_sev_es_mmio_write(struct kvm_vcpu *vcpu, gp= a_t gpa, unsigned int bytes, int handled; struct kvm_mmio_fragment *frag; =20 - if (!data) + if (!data || WARN_ON_ONCE(object_is_on_stack(data))) return -EINVAL; =20 handled =3D write_emultor.read_write_mmio(vcpu, gpa, bytes, data); @@ -14336,7 +14338,7 @@ int kvm_sev_es_mmio_read(struct kvm_vcpu *vcpu, gpa= _t gpa, unsigned int bytes, int handled; struct kvm_mmio_fragment *frag; =20 - if (!data) + if (!data || WARN_ON_ONCE(object_is_on_stack(data))) return -EINVAL; =20 handled =3D read_emultor.read_write_mmio(vcpu, gpa, bytes, data); --=20 2.53.0.414.gf7e9f6c205-goog From nobody Fri Apr 17 00:17:57 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D32A3262FE7 for ; Wed, 25 Feb 2026 01:21:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982473; cv=none; b=IDhe+OfOu77BAJO0zSQlM3/lrirEtTLhCxDjZvz6XCwS+wjb0srO3ESf57ZzCosPTM2R6rWzUdyqQWuIlP12Wd8h6vj3tXzL0OE0ocLi0LBzqI5J0AkEXbKIpFR8UM3o3/1AjGqw0uvqr7nw6GR86iA4sppoCb2m6//tuR7rSwI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982473; c=relaxed/simple; bh=4zmfeUoFw0++ZD2R7XqZF7I3p6nxfaKrCAs/KOcytN0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=DWn5dmKD6XkKpk7bU+X+PqtHFUctmjeeogPxha53nTMcv29VJ+y57qp44HVtr/FYq5Ss0pK5o9yE8vVG2S8hJW297O1eGLsLzc/j6+JnwjjJnl7oS/5LCfpx9tJDtFPdZvXvxTo+B6j7JbK72VqW6TyJR8r0Jq7E2QpT7qX2g+U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=O9TYJ9n7; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="O9TYJ9n7" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2adc527eaf5so2765625ad.0 for ; Tue, 24 Feb 2026 17:21:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1771982468; x=1772587268; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=LNNapYowtT0YYRygm4WC7Vz/wQDCwBu2kBJka11VtD0=; b=O9TYJ9n7RHCsH7UGDZAJ5Ib0hBi5YppZ+tPlu+EIN8rIlN/bHFlPMI800sm+DdXgNT +JdJoPqVdHJqfMzI4wYuYRvdlAvvmnN430z0T/Xm2pRQlhN4w0IbzJhDc0ioRlx6GZLk BP9vnTrDNA0xI2ID53Q9laWvviaLXd3hYZ/JahYYgGEAwSwqkQjZZQ1Cu4Qk90u1/3AH EHiBV6HNyNvwl3lXBEr9QloNaRoqC0CrgdYHcp0eDC30W4r1TKU+kBtvNEGUE4R6wF5i 3yt50VCrGJ3gfbZwKNGYQxgzRrOg+BCjYCaunx3UJlL1bpsYjgzkHOO3ZmRJzX1j7k1L nkGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771982468; x=1772587268; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=LNNapYowtT0YYRygm4WC7Vz/wQDCwBu2kBJka11VtD0=; b=MMpLvPf1YSzNtNLf31NyaKhkfVLG2rcprUf0yA00jm7zEtGTj3cP7YnbB5M3anT+Ga VEYoAqxt3YXzIgZImgUI88pgX4TD2YOC4vQV1wytrr8d7dbCZB6cOUy19DYI3AIwMefD opXeumFfBWpxUKGcvz/ib5gDBjaF4X1y9gfvpRLRJI1Gsv712yigQ7yONOJjUULGjUSG WSNR2upGRx8aQEWuC/BTveLTC9qrcvp2LM5NTpLDgpdESZd9eJvZWpMAidDsZTfEG6o1 CuoEdGISRUzuSBEI7Z91mSE0Zr1LhVgY8to33nqhusBNE2JuNDJ9fDwCksn3ghVdwftf pKCA== X-Forwarded-Encrypted: i=1; AJvYcCXV29duibYbqDWqsRzzgn5DiFxetgZ+uIe37+u8/Cfsd9x0Z7QtXfYndzlttrZrX04nisRKhFs3SJuca7Q=@vger.kernel.org X-Gm-Message-State: AOJu0YzhTngW19zW6gNDoe1QUy7xIL8xscXjCxrsV17xw78adaV8HCnQ vQJ1JfHtS2yuIvUeQSTAIZFN5Pm1vc4Q8o5v92gxj1alcl/yyoyjphd6e8TiF6+vfsBIyA/IWgt fcgsPwQ== X-Received: from pjbnh18.prod.google.com ([2002:a17:90b:3652:b0:358:dfd8:3150]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:1a8f:b0:2aa:f2ce:5ac5 with SMTP id d9443c01a7336-2ad744e0d19mr151791755ad.32.1771982468138; Tue, 24 Feb 2026 17:21:08 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 24 Feb 2026 17:20:43 -0800 In-Reply-To: <20260225012049.920665-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260225012049.920665-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.414.gf7e9f6c205-goog Message-ID: <20260225012049.920665-9-seanjc@google.com> Subject: [PATCH 08/14] KVM: x86: Dedup kvm_sev_es_mmio_{read,write}() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, Yashu Zhang , Rick Edgecombe , Binbin Wu , Xiaoyao Li , Tom Lendacky , Michael Roth Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Dedup the SEV-ES emulated MMIO code by using the read vs. write emulator ops to handle the few differences between reads and writes. Opportunistically tweak the comment about fragments to call out that KVM should verify that userspace can actually handle MMIO requests that cross page boundaries. Unlike emulated MMIO, the request is made in the GPA space, not the GVA space, i.e. emulation across page boundaries can work generically, at least in theory. No functional change intended. Signed-off-by: Sean Christopherson Tested-by: Rick Edgecombe Tested-by: Tom Lendacky --- arch/x86/kvm/x86.c | 110 +++++++++++++++++++-------------------------- 1 file changed, 45 insertions(+), 65 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 2db0bf738d2d..f93f0f8961af 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -14293,80 +14293,60 @@ static int complete_sev_es_emulated_mmio(struct k= vm_vcpu *vcpu) return 0; } =20 +static int kvm_sev_es_do_mmio(struct kvm_vcpu *vcpu, gpa_t gpa, + unsigned int bytes, void *data, + const struct read_write_emulator_ops *ops) +{ + struct kvm_mmio_fragment *frag; + int handled; + + if (!data || WARN_ON_ONCE(object_is_on_stack(data))) + return -EINVAL; + + handled =3D ops->read_write_mmio(vcpu, gpa, bytes, data); + if (handled =3D=3D bytes) + return 1; + + bytes -=3D handled; + gpa +=3D handled; + data +=3D handled; + + /* + * TODO: Determine whether or not userspace plays nice with MMIO + * requests that split a page boundary. + */ + frag =3D vcpu->mmio_fragments; + vcpu->mmio_nr_fragments =3D 1; + frag->len =3D bytes; + frag->gpa =3D gpa; + frag->data =3D data; + + vcpu->mmio_needed =3D 1; + vcpu->mmio_cur_fragment =3D 0; + + vcpu->run->mmio.phys_addr =3D gpa; + vcpu->run->mmio.len =3D min(8u, frag->len); + vcpu->run->mmio.is_write =3D ops->write; + if (ops->write) + memcpy(vcpu->run->mmio.data, frag->data, min(8u, frag->len)); + vcpu->run->exit_reason =3D KVM_EXIT_MMIO; + + vcpu->arch.complete_userspace_io =3D complete_sev_es_emulated_mmio; + + return 0; +} + int kvm_sev_es_mmio_write(struct kvm_vcpu *vcpu, gpa_t gpa, unsigned int b= ytes, void *data) { - int handled; - struct kvm_mmio_fragment *frag; - - if (!data || WARN_ON_ONCE(object_is_on_stack(data))) - return -EINVAL; - - handled =3D write_emultor.read_write_mmio(vcpu, gpa, bytes, data); - if (handled =3D=3D bytes) - return 1; - - bytes -=3D handled; - gpa +=3D handled; - data +=3D handled; - - /*TODO: Check if need to increment number of frags */ - frag =3D vcpu->mmio_fragments; - vcpu->mmio_nr_fragments =3D 1; - frag->len =3D bytes; - frag->gpa =3D gpa; - frag->data =3D data; - - vcpu->mmio_needed =3D 1; - vcpu->mmio_cur_fragment =3D 0; - - vcpu->run->mmio.phys_addr =3D gpa; - vcpu->run->mmio.len =3D min(8u, frag->len); - vcpu->run->mmio.is_write =3D 1; - memcpy(vcpu->run->mmio.data, frag->data, min(8u, frag->len)); - vcpu->run->exit_reason =3D KVM_EXIT_MMIO; - - vcpu->arch.complete_userspace_io =3D complete_sev_es_emulated_mmio; - - return 0; + return kvm_sev_es_do_mmio(vcpu, gpa, bytes, data, &write_emultor); } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_sev_es_mmio_write); =20 int kvm_sev_es_mmio_read(struct kvm_vcpu *vcpu, gpa_t gpa, unsigned int by= tes, void *data) { - int handled; - struct kvm_mmio_fragment *frag; - - if (!data || WARN_ON_ONCE(object_is_on_stack(data))) - return -EINVAL; - - handled =3D read_emultor.read_write_mmio(vcpu, gpa, bytes, data); - if (handled =3D=3D bytes) - return 1; - - bytes -=3D handled; - gpa +=3D handled; - data +=3D handled; - - /*TODO: Check if need to increment number of frags */ - frag =3D vcpu->mmio_fragments; - vcpu->mmio_nr_fragments =3D 1; - frag->len =3D bytes; - frag->gpa =3D gpa; - frag->data =3D data; - - vcpu->mmio_needed =3D 1; - vcpu->mmio_cur_fragment =3D 0; - - vcpu->run->mmio.phys_addr =3D gpa; - vcpu->run->mmio.len =3D min(8u, frag->len); - vcpu->run->mmio.is_write =3D 0; - vcpu->run->exit_reason =3D KVM_EXIT_MMIO; - - vcpu->arch.complete_userspace_io =3D complete_sev_es_emulated_mmio; - - return 0; + return kvm_sev_es_do_mmio(vcpu, gpa, bytes, data, &read_emultor); } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_sev_es_mmio_read); =20 --=20 2.53.0.414.gf7e9f6c205-goog From nobody Fri Apr 17 00:17:57 2026 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 630FE278E5D for ; Wed, 25 Feb 2026 01:21:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982471; cv=none; b=GHzAX9xJkMbiAUZG5qHxwIKMtxVoSk9z2pk8LxpfWiVrf8UKYY+/SQWVpqh5Rk8aQ+vXbZgRNhmjPpLdEk8Bwaw5mcRkowNULgf+3LKwfaX+V/ZiFCREpocB3wCvwCO8lVqg3Q9kNhmlT3VVXb5avQGm+tTj9hI9bs24rsxCCKw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982471; c=relaxed/simple; bh=jhFdYJ/k2G4ufjPQSuQbjjN1qZ0Psh4hxK61qelisac=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Timk2gvTfrpDG1wRlsxNDwrJD6YhlhPdWUX49i4KP92ZfV+tnWAiEEhtmStb5+hVItX5UVJmNl1wPo7ls1fXYneI6spMaBgUlJwXmuocvYx8vAGDgC4WyVejCZuvN1XTKbiVY2nyb5O+TkcRsKWOJRG69R1n/CWBDFA16rC22Ec= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=xC4R8j28; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="xC4R8j28" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-358f42fad0bso2947703a91.0 for ; Tue, 24 Feb 2026 17:21:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1771982470; x=1772587270; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=sb25WqFS5MOa5y8whWmlEi7jKLk6DH13/xEv6hI9zy0=; b=xC4R8j28BYATpbd4dNrttLzukZgp4xeTD460YuiSAiIKVOUbvqmHHn4vIV7upcAM3N 3vLWzOGvlCuLOydh3/n6icAZpWu0d+bhvyXe/9B25tkpqlh5wTIEbNaq94VXlknwzU5V +16gLG0hng1RzWTVnTTuGZX1pTkpwtqHlYK5VjtMMiV/yL3YlfSHyWShcn1Bo3mySqpW B21CixIfnMBbaKJRA3vtWVmkRedDCvXPURcYICVHa/PG6nE7pGS3q6gzS3mDHdu5ryY4 Rq31VrO+2n0XZREPbJmGjhHJzTj0myd7T+booGaY0cB5/tv9oNV1ksSE+PKs8uVPY3Ew rkrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771982470; x=1772587270; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=sb25WqFS5MOa5y8whWmlEi7jKLk6DH13/xEv6hI9zy0=; b=ZYeAh/RoJ7MdwrPspx8k3+ZNSMyz2HS+ZhWkx9EzjmarK5dSWnmynuwc7j2DEmIvpB y1iQD60Fz2PGpLcjVpMHT1HH/H98SLnfNXT/OGqKrn4fDHWH5fwN0jIZQyZJjk0SpGLe UorPsstJ4NA+dUwFDzILHRhXVtSvEeMeuk5Aj3Cow3jyyQW7s4zS04VhbNB/nPRl6e8g DoPI1OhbEEkhM5Zy/Xyfg2LS61v9Rm03hdInw8zxfPWiniE9sMcd0eoMrXc61Un9484r tIlOdhVoy6TIWkFXbAEbodmuThJ0cBaJ4iDeZySeH33B/GlwdYei2ntRC+kT2dchqRTz 2ngg== X-Forwarded-Encrypted: i=1; AJvYcCX7g15UNW57957mbFH0i+3TSPqSXvv4A+OvsYE2lGD4OYe9+6WygTDT7u4qpd94nNB2DJqoYUnlWaLZJLI=@vger.kernel.org X-Gm-Message-State: AOJu0YxeU0eElk/eWquy6AsW4t8iJ3384WyPYMSo9dSe3flVECBhAS4W wiyrTLXtZ5AURsSg9tHHdHTzD5gq/R6mdFtTyqW3978IXjfhX/vD4YsRS1JErzUjB3vFkQGEH/b eKbwFWw== X-Received: from pjz13.prod.google.com ([2002:a17:90b:56cd:b0:354:c1db:b113]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:1f8e:b0:356:2c7b:c013 with SMTP id 98e67ed59e1d1-3590f205431mr539675a91.29.1771982469774; Tue, 24 Feb 2026 17:21:09 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 24 Feb 2026 17:20:44 -0800 In-Reply-To: <20260225012049.920665-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260225012049.920665-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.414.gf7e9f6c205-goog Message-ID: <20260225012049.920665-10-seanjc@google.com> Subject: [PATCH 09/14] KVM: x86: Consolidate SEV-ES MMIO emulation into a single public API From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, Yashu Zhang , Rick Edgecombe , Binbin Wu , Xiaoyao Li , Tom Lendacky , Michael Roth Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Dedup kvm_sev_es_mmio_{read,write}() into a single API, as the "cost" of plumbing in a boolean is largely negligible since KVM pulls out a boolean for ops->write anyways, and consolidating the APIs will allow for additional cleanups. No functional change intended. Signed-off-by: Sean Christopherson Tested-by: Rick Edgecombe Tested-by: Tom Lendacky --- arch/x86/kvm/svm/sev.c | 20 ++++++-------------- arch/x86/kvm/x86.c | 29 +++++++++-------------------- arch/x86/kvm/x86.h | 6 ++---- 3 files changed, 17 insertions(+), 38 deletions(-) diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index ea515cf41168..723f4452302a 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -4434,25 +4434,17 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu) =20 switch (control->exit_code) { case SVM_VMGEXIT_MMIO_READ: - ret =3D setup_vmgexit_scratch(svm, true, control->exit_info_2); - if (ret) - break; + case SVM_VMGEXIT_MMIO_WRITE: { + bool is_write =3D control->exit_code =3D=3D SVM_VMGEXIT_MMIO_WRITE; =20 - ret =3D kvm_sev_es_mmio_read(vcpu, - control->exit_info_1, - control->exit_info_2, - svm->sev_es.ghcb_sa); - break; - case SVM_VMGEXIT_MMIO_WRITE: - ret =3D setup_vmgexit_scratch(svm, false, control->exit_info_2); + ret =3D setup_vmgexit_scratch(svm, !is_write, control->exit_info_2); if (ret) break; =20 - ret =3D kvm_sev_es_mmio_write(vcpu, - control->exit_info_1, - control->exit_info_2, - svm->sev_es.ghcb_sa); + ret =3D kvm_sev_es_mmio(vcpu, is_write, control->exit_info_1, + control->exit_info_2, svm->sev_es.ghcb_sa); break; + } case SVM_VMGEXIT_NMI_COMPLETE: ++vcpu->stat.nmi_window_exits; svm->nmi_masked =3D false; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index f93f0f8961af..022e953906f7 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -14293,9 +14293,8 @@ static int complete_sev_es_emulated_mmio(struct kvm= _vcpu *vcpu) return 0; } =20 -static int kvm_sev_es_do_mmio(struct kvm_vcpu *vcpu, gpa_t gpa, - unsigned int bytes, void *data, - const struct read_write_emulator_ops *ops) +int kvm_sev_es_mmio(struct kvm_vcpu *vcpu, bool is_write, gpa_t gpa, + unsigned int bytes, void *data) { struct kvm_mmio_fragment *frag; int handled; @@ -14303,7 +14302,10 @@ static int kvm_sev_es_do_mmio(struct kvm_vcpu *vcp= u, gpa_t gpa, if (!data || WARN_ON_ONCE(object_is_on_stack(data))) return -EINVAL; =20 - handled =3D ops->read_write_mmio(vcpu, gpa, bytes, data); + if (is_write) + handled =3D vcpu_mmio_write(vcpu, gpa, bytes, data); + else + handled =3D vcpu_mmio_read(vcpu, gpa, bytes, data); if (handled =3D=3D bytes) return 1; =20 @@ -14326,8 +14328,8 @@ static int kvm_sev_es_do_mmio(struct kvm_vcpu *vcpu= , gpa_t gpa, =20 vcpu->run->mmio.phys_addr =3D gpa; vcpu->run->mmio.len =3D min(8u, frag->len); - vcpu->run->mmio.is_write =3D ops->write; - if (ops->write) + vcpu->run->mmio.is_write =3D is_write; + if (is_write) memcpy(vcpu->run->mmio.data, frag->data, min(8u, frag->len)); vcpu->run->exit_reason =3D KVM_EXIT_MMIO; =20 @@ -14335,20 +14337,7 @@ static int kvm_sev_es_do_mmio(struct kvm_vcpu *vcp= u, gpa_t gpa, =20 return 0; } - -int kvm_sev_es_mmio_write(struct kvm_vcpu *vcpu, gpa_t gpa, unsigned int b= ytes, - void *data) -{ - return kvm_sev_es_do_mmio(vcpu, gpa, bytes, data, &write_emultor); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_sev_es_mmio_write); - -int kvm_sev_es_mmio_read(struct kvm_vcpu *vcpu, gpa_t gpa, unsigned int by= tes, - void *data) -{ - return kvm_sev_es_do_mmio(vcpu, gpa, bytes, data, &read_emultor); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_sev_es_mmio_read); +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_sev_es_mmio); =20 static void advance_sev_es_emulated_pio(struct kvm_vcpu *vcpu, unsigned co= unt, int size) { diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 94d4f07aaaa0..1d0f0edd31b3 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -712,10 +712,8 @@ static inline bool __kvm_is_valid_cr4(struct kvm_vcpu = *vcpu, unsigned long cr4) __reserved_bits; \ }) =20 -int kvm_sev_es_mmio_write(struct kvm_vcpu *vcpu, gpa_t src, unsigned int b= ytes, - void *dst); -int kvm_sev_es_mmio_read(struct kvm_vcpu *vcpu, gpa_t src, unsigned int by= tes, - void *dst); +int kvm_sev_es_mmio(struct kvm_vcpu *vcpu, bool is_write, gpa_t gpa, + unsigned int bytes, void *data); int kvm_sev_es_string_io(struct kvm_vcpu *vcpu, unsigned int size, unsigned int port, void *data, unsigned int count, int in); --=20 2.53.0.414.gf7e9f6c205-goog From nobody Fri Apr 17 00:17:57 2026 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 15F4E26D4F7 for ; Wed, 25 Feb 2026 01:21:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982473; cv=none; b=Xuz+M5sfJ5tEgfAN6VDvw8mLWkV7qgs2+7+Exd8X0PFlR/KsYmoT2PhD+8T4InhxYRxcdDJhQJpA/pF7uFIC4W6s34dUloIFHyFrXTN4s10Suog1OU8H6XuzrOU3wEmJ8XBNnaeK9B2BfJpUyslk3Qb8tHk8IqQ5E2XgKX4zofw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982473; c=relaxed/simple; bh=8BpbiNXkNmG68PauqKn1vI4bfavvNUcZL8H/4h/xCDU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=HTMgMstxsVyR3ntimn1Rm0CxRHyvlNsJZpcsH5oKt5+fFhpnOFWTy37C3rDFVFXUKAjbvtSveVCleaj79CULkF9oU8RDrpezUHqm7kfUgrkQ7aTj7io0Wmc+8SH3wTzlNNysg2eCe/+D2dqU2Xp7I/NKK+mTp82tOxoWLK3FQkA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=E/S28+Gi; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="E/S28+Gi" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-3568090851aso39449396a91.1 for ; Tue, 24 Feb 2026 17:21:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1771982471; x=1772587271; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=NC+PweffU72Wb4W/8tySGqQ5rYKiIvlwoGA5HRqPPw0=; b=E/S28+GiKw/h0AgddShLg2kxTtGNWU+SqqePcjI7qhDLWptlTcGOycmmLV0LjriD65 4MHsIEiJ6vAWwM025BjKmgcX1lsGolh4fdfHW30dmZuuRe/1rqQU9LFnAca3NRZh5Go0 cRpXF1t9nSE15ZoJ4sBVoq6UyqjUTthcGR79kv0xT2Y0dPTP1DTpI6FCB5Q+cxMCf2vS zQnC7leasPNY3hCTzDYRspDm1rO6ns07SqlAf4WCAiAvV6XUk+P7KCx/GWGb2rsMthpy +xE0VWCp68HzvW4RWhFgCy40Vy9vpOGx9Ik6D5kUgG8wsMIhZnS5dyGI/8Jf9/1VKtvc 4ZHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771982471; x=1772587271; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=NC+PweffU72Wb4W/8tySGqQ5rYKiIvlwoGA5HRqPPw0=; b=fCmfIajgWkuXN6dDZxfwODfwmxwnhXCyRpIUUXZppEpkco6UCKpx0sK4YNOO+4i3QL uhXysc4nVS+1NTI+iXctgGX2ABAv2RG5KcXB1Bk3i0U5oSJulk2RAJbaL6VPZt1L3zPJ 2VOMRAQdEIAzEFnVy0GCKMmuSlUmPG4T2K7fidYIzf4/vTm9JpmjUU81RbYsgtjvRMqP ioBDXdh0cQyUH/ZJ8hSLhGmuYWxEkPlHnRjNGU+GYd8P6f0g0c+Qk8DBaiJwBC+Epsxe OZo9rbkuYfemb6yzhkvpRhnrGQg72bKqnOlg7qWDPPsW2XD9Dajp0O/U4q255Z8XfWaA YkFw== X-Forwarded-Encrypted: i=1; AJvYcCXE92gokrYOYUibF7EAsy2HFj3+A0GwA3B1PcclqlS0dcczjx/4ofiYHYZbfVNW6QuG9iArqZlZYODrJ9Q=@vger.kernel.org X-Gm-Message-State: AOJu0YwQWz22QsNKdAwM+PPTPBUrUOMr/Xmddkx5Gf8Tp/xiSvHz7yWZ HUFA84WmkTqokN4jeY2sMKh78cK5WH2mVd/SUSHFegADquLs6GfuvJUx12VYIPIknaij0BGv1Sr qF+ckMg== X-Received: from pjbev14.prod.google.com ([2002:a17:90a:eace:b0:356:3104:ed7]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:4ec5:b0:343:7714:4ca8 with SMTP id 98e67ed59e1d1-3590f08c0e8mr548523a91.15.1771982471453; Tue, 24 Feb 2026 17:21:11 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 24 Feb 2026 17:20:45 -0800 In-Reply-To: <20260225012049.920665-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260225012049.920665-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.414.gf7e9f6c205-goog Message-ID: <20260225012049.920665-11-seanjc@google.com> Subject: [PATCH 10/14] KVM: x86: Bury emulator read/write ops in emulator_{read,write}_emulated() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, Yashu Zhang , Rick Edgecombe , Binbin Wu , Xiaoyao Li , Tom Lendacky , Michael Roth Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that SEV-ES invokes vcpu_mmio_{read,write}() directly, bury the read vs. write emulator ops in the dedicated emulator callbacks so that they are colocated with their usage, and to make it harder for non-emulator code to use hooks that are intended for the emulator. Opportunistically rename the structures to get rid of the long-standing "emultor" typo. No functional change intended. Signed-off-by: Sean Christopherson Tested-by: Rick Edgecombe Tested-by: Tom Lendacky --- arch/x86/kvm/x86.c | 29 ++++++++++++++--------------- 1 file changed, 14 insertions(+), 15 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 022e953906f7..5fb5259c208f 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -8134,17 +8134,6 @@ static int write_emulate(struct kvm_vcpu *vcpu, gpa_= t gpa, return emulator_write_phys(vcpu, gpa, val, bytes); } =20 -static const struct read_write_emulator_ops read_emultor =3D { - .read_write_emulate =3D read_emulate, - .read_write_mmio =3D vcpu_mmio_read, -}; - -static const struct read_write_emulator_ops write_emultor =3D { - .read_write_emulate =3D write_emulate, - .read_write_mmio =3D vcpu_mmio_write, - .write =3D true, -}; - static int emulator_read_write_onepage(unsigned long addr, void *val, unsigned int bytes, struct x86_exception *exception, @@ -8295,8 +8284,13 @@ static int emulator_read_emulated(struct x86_emulate= _ctxt *ctxt, unsigned int bytes, struct x86_exception *exception) { - return emulator_read_write(ctxt, addr, val, bytes, - exception, &read_emultor); + static const struct read_write_emulator_ops ops =3D { + .read_write_emulate =3D read_emulate, + .read_write_mmio =3D vcpu_mmio_read, + .write =3D false, + }; + + return emulator_read_write(ctxt, addr, val, bytes, exception, &ops); } =20 static int emulator_write_emulated(struct x86_emulate_ctxt *ctxt, @@ -8305,8 +8299,13 @@ static int emulator_write_emulated(struct x86_emulat= e_ctxt *ctxt, unsigned int bytes, struct x86_exception *exception) { - return emulator_read_write(ctxt, addr, (void *)val, bytes, - exception, &write_emultor); + static const struct read_write_emulator_ops ops =3D { + .read_write_emulate =3D write_emulate, + .read_write_mmio =3D vcpu_mmio_write, + .write =3D true, + }; + + return emulator_read_write(ctxt, addr, (void *)val, bytes, exception, &op= s); } =20 #define emulator_try_cmpxchg_user(t, ptr, old, new) \ --=20 2.53.0.414.gf7e9f6c205-goog From nobody Fri Apr 17 00:17:57 2026 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E66F4281376 for ; Wed, 25 Feb 2026 01:21:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982477; cv=none; b=AYiCPhrUwUJWo7nH2Bykzd+u78URgflsESvvnJ9Oe7kj4sLVoTN4jaJbZcStIzuewDEbLYfEMt3Eg+CdKUhBzXx536FDBurHWPle7Ohap3p8yAJsR3v/oHxrbVE/zLy/5mpNfwoJEP/HjrHlhTHur/MA/7HiggN2ftU6Iaj1R2Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982477; c=relaxed/simple; bh=XAkvXmxYSWcnAm2ngYtj/+yAtyCpKD+kGWhH+fNj3zE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=dTY7DFC2sPo3GzhDpodN3xze03WBrXZqzbxOn7VWTNHqDu7MWPksTah5OAiBHXXG+OFgXjlZnnQdEi+C8UJ9SaTsEc1dbmXVKFg4lGn+WkV51vnB1B9uIL4ExTiqR1Fm2WekD5nBnFQnMCT2GXnD6hqyfQNFxQb6g856wQzQpjw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=txXm5jUT; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="txXm5jUT" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-358f42fad0bso2947870a91.0 for ; Tue, 24 Feb 2026 17:21:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1771982473; x=1772587273; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=cC3kOG4PMTKHbBJlg/C79tGxZ75SzIpZ8/Nyw/YcEt4=; b=txXm5jUTtjG2qaNQibAT1ILeyH3rvkgFc38bzZWG+lkFX9V1r8ZGBOu47LlMZLzgvz 5vlSmFhKkw9UyDkDMXeG8ppHMEEnez/Ue5Buau4uDtNL1XFR5rheE5oFPV+ng1v1yuiH Aa44i75Xd4SGFNMPlcWyvFhEg2wEQmatg+KiQRqJWtuiktbpJrNbjjTrjPyKFFniXH8n nzTmrYYPQOp8ESklKI9HC9VylQSnJoOgC4LxaEDpCAPi8f8KrdLOp2Z/K6aqcs1Gy8XL lo7ki9o648kzXuqxVnaKrm9Z9pXjeHrrOhvo27sY+4LgJTFBCuz2Sqm93gKPb01cHwij FQsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771982473; x=1772587273; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=cC3kOG4PMTKHbBJlg/C79tGxZ75SzIpZ8/Nyw/YcEt4=; b=j9/Ddbv3QmjXcUW4BSs6gWhMZEFszoTx6OwTRJKF85Up3jEiEx8eZOnPiTp4pB+Nil kpPdNQkOFxqTZx67puWmQttiOtRumXh+G950Dq02N554t0fEj0lMPmya6gg+AkM/d2ZR su/z9efpB/Hq4S9Qi1sZDPnAVs6cNPxI15oBNtwSExOtkAolENLGudNL1S9qHUxs6oiw I+AeG04ks7FtpMW+BpyGvX5AShw/qLm7OWWtb/IjypQ6hO9F4VjXa4L4YEmcxNeQNl5L XCtuIJZZiPuTVTs1xPtGNDvI4+Pjhm3KJQGmtqKnl/6QM5rCpll4b054uo+Os4Z2YaIg 8hkw== X-Forwarded-Encrypted: i=1; AJvYcCVDaTeOyQEiUdX5pUC23rvpjQBXa5NGHrraxMcjITct/5yCmIT8KoBADmFWwRK6OBlb4XyFMhRutDjBd9M=@vger.kernel.org X-Gm-Message-State: AOJu0Yyp+H2qaHzCQmphKOEYMZTZ+Lh/EkfJaOApKg64vwPAFWrbh+I4 uelM+YRTgFCXAdbTvkPlD3afjZTNaAAwRduL0At4q3lkt7VlyknMoWOvc+ghmkhmpq6Mi+s9Ir0 WdMM69g== X-Received: from pjre10.prod.google.com ([2002:a17:90a:b38a:b0:359:41:e301]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:4ec5:b0:343:7714:4ca8 with SMTP id 98e67ed59e1d1-3590f08c0e8mr548564a91.15.1771982473219; Tue, 24 Feb 2026 17:21:13 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 24 Feb 2026 17:20:46 -0800 In-Reply-To: <20260225012049.920665-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260225012049.920665-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.414.gf7e9f6c205-goog Message-ID: <20260225012049.920665-12-seanjc@google.com> Subject: [PATCH 11/14] KVM: x86: Fold emulator_write_phys() into write_emulate() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, Yashu Zhang , Rick Edgecombe , Binbin Wu , Xiaoyao Li , Tom Lendacky , Michael Roth Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Fold emulator_write_phys() into write_emulate() to drop a superfluous wrapper, and to provide more symmetry between the read and write paths. No functional change intended. Signed-off-by: Sean Christopherson Tested-by: Rick Edgecombe Tested-by: Tom Lendacky --- arch/x86/include/asm/kvm_host.h | 3 --- arch/x86/kvm/x86.c | 20 +++++++------------- 2 files changed, 7 insertions(+), 16 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index ff07c45e3c73..aa030fbd669d 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2097,9 +2097,6 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_sta= rt, gfn_t gfn_end); =20 int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3); =20 -int emulator_write_phys(struct kvm_vcpu *vcpu, gpa_t gpa, - const void *val, int bytes); - extern bool tdp_enabled; =20 u64 vcpu_tsc_khz(struct kvm_vcpu *vcpu); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 5fb5259c208f..5a3ba161db7b 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -8102,18 +8102,6 @@ static int vcpu_mmio_gva_to_gpa(struct kvm_vcpu *vcp= u, unsigned long gva, return vcpu_is_mmio_gpa(vcpu, gva, *gpa, write); } =20 -int emulator_write_phys(struct kvm_vcpu *vcpu, gpa_t gpa, - const void *val, int bytes) -{ - int ret; - - ret =3D kvm_vcpu_write_guest(vcpu, gpa, val, bytes); - if (ret < 0) - return 0; - kvm_page_track_write(vcpu, gpa, val, bytes); - return 1; -} - struct read_write_emulator_ops { int (*read_write_emulate)(struct kvm_vcpu *vcpu, gpa_t gpa, void *val, int bytes); @@ -8131,7 +8119,13 @@ static int read_emulate(struct kvm_vcpu *vcpu, gpa_t= gpa, static int write_emulate(struct kvm_vcpu *vcpu, gpa_t gpa, void *val, int bytes) { - return emulator_write_phys(vcpu, gpa, val, bytes); + int ret; + + ret =3D kvm_vcpu_write_guest(vcpu, gpa, val, bytes); + if (ret < 0) + return 0; + kvm_page_track_write(vcpu, gpa, val, bytes); + return 1; } =20 static int emulator_read_write_onepage(unsigned long addr, void *val, --=20 2.53.0.414.gf7e9f6c205-goog From nobody Fri Apr 17 00:17:57 2026 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 795CB26E708 for ; Wed, 25 Feb 2026 01:21:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982477; cv=none; b=AchLQLNKn7QRbzXDwiAG12EDybOQkM1esmdGsiXdgQhB0a+UhgGTgj8QAT5MrvlX5sBDoaXeC7zyhqo6u/6Gl5NMzzSYvjGsUNs2QYQe2alWuFz8Agep3UEAoFAWwg3xH6HaWwXLWJwdK+b/mNuS7NgDpXU6XbNjFQVhkj6eKJg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982477; c=relaxed/simple; bh=85i39OWQhVtkn+KmWvGw+H0S5qjkbaqht3+QjSVkoLI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=aWVCXuPwuLUWBR6i58TxqHFWB1mNNfkhq1s4QQ2XrwISQTTIibTf6uvDPrkqRa2rJdwoufHx236jp7dVakCT998m2urmPi3r3gcczmL9E/h0Pc5vEtADQgohcEXwNFYvjwLsdu2ADIKvLXX+bLX1VZnRPDtaU8w5O9LQUlwQjo4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WxnGNGx9; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WxnGNGx9" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-358dc09b43eso1361279a91.2 for ; Tue, 24 Feb 2026 17:21:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1771982475; x=1772587275; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=PEtXjeylvLkO6uPQdY2VXz1BpnUVZoay6R5c8Z2Fy64=; b=WxnGNGx9MHeq6Vnm9AblAYlxLtsfYWdaxUYa51xS2cIYozLVBka5m3nTHIYUOwfeMF e0e6LBhGb0UHaY1JDiWeCSDeox/awWImdp49hc9f//17RYa97DWlCPAq0hRw+R8U5q5X 9lt6KCyq4G68ZLtNKoQC0yXclx9CF7Hnya2VrZj1EB0N0lI0M1LngyV9wnH1YkSbb0W5 2osq3hozdRJAgImHkNE6NXHyEOqR6dYZVXZK3HRJAhzacHEDsAurB9+lnxJuOTZLr7Z0 gcq+J/PVrMKqZqYoQzh5yRQ6zkXVaKDC/4K6xc4k9Pg89EFa8dEZEhU2KTwBvPVVXzzI daBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771982475; x=1772587275; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=PEtXjeylvLkO6uPQdY2VXz1BpnUVZoay6R5c8Z2Fy64=; b=KY3+kkBNtHU2tvU52pD6Z1FS6koMj0URFLAgo3oaV2V6O8YCYCJDO1nW6b2Y3KEKC0 el0VZ7Wx0SCmwBwYUnFlyX5pZpouLrgVbRLn2GFadSd4gnHmKYYT3KNbaWBZ2ylxco1v xz4jK6Sc1SpqdLA/OGaWWVoxqiRP18U62YqGlBzieWvJ/b2XkyegtTltuF52N0cKpkFO wrt2giuj+vnD8zKvzJvmfbtPp4p9MUpeDNv8Rqrm2XmjvvdQFNvgybtcbEk1fqrdDYKr Ah5emKhDaUERaAs9O8vpWsV2kEysggTiRHdtPmTXlDVyLn1YqFe/Xff+prs0xjlbYnt2 /IGg== X-Forwarded-Encrypted: i=1; AJvYcCVRMbNb7EE/LImsi4olyemDhRGOm2ZmRmnH1V4QtiufFnVWzfX4c+jZi6ykGwCkm6StrKFtYCbFR+EdErM=@vger.kernel.org X-Gm-Message-State: AOJu0YzDR85WoHhlwEzEwD0IhSkHMCs7rChnrHho0yTzZbgCADxMc7lq BW1cB2v0JCSJziSkppbZiN78758zo5YK74TobPipccHtC0t4HespY3sY6a55ZLm83WBU9FixdMc TxrSdCA== X-Received: from pgbdw17.prod.google.com ([2002:a05:6a02:4491:b0:c6e:2e2c:d4a2]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:cf85:b0:395:9d89:db69 with SMTP id adf61e73a8af0-3959f270843mr632443637.7.1771982474849; Tue, 24 Feb 2026 17:21:14 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 24 Feb 2026 17:20:47 -0800 In-Reply-To: <20260225012049.920665-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260225012049.920665-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.414.gf7e9f6c205-goog Message-ID: <20260225012049.920665-13-seanjc@google.com> Subject: [PATCH 12/14] KVM: x86: Rename .read_write_emulate() to .read_write_guest() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, Yashu Zhang , Rick Edgecombe , Binbin Wu , Xiaoyao Li , Tom Lendacky , Michael Roth Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Rename the ops and helpers to read/write guest memory to clarify that they do exactly that, i.e. aren't generic emulation flows and don't do anything related to emulated MMIO. Opportunistically add comments to explain the flow, e.g. it's not exactly obvious that KVM deliberately treats "failed" accesses to guest memory as emulated MMIO. No functional change intended. Signed-off-by: Sean Christopherson Tested-by: Rick Edgecombe Tested-by: Tom Lendacky --- arch/x86/kvm/x86.c | 38 ++++++++++++++++++++++++++++---------- 1 file changed, 28 insertions(+), 10 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 5a3ba161db7b..f3e2ec7e1828 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -8103,21 +8103,21 @@ static int vcpu_mmio_gva_to_gpa(struct kvm_vcpu *vc= pu, unsigned long gva, } =20 struct read_write_emulator_ops { - int (*read_write_emulate)(struct kvm_vcpu *vcpu, gpa_t gpa, - void *val, int bytes); + int (*read_write_guest)(struct kvm_vcpu *vcpu, gpa_t gpa, + void *val, int bytes); int (*read_write_mmio)(struct kvm_vcpu *vcpu, gpa_t gpa, int bytes, void *val); bool write; }; =20 -static int read_emulate(struct kvm_vcpu *vcpu, gpa_t gpa, - void *val, int bytes) +static int emulator_read_guest(struct kvm_vcpu *vcpu, gpa_t gpa, + void *val, int bytes) { return !kvm_vcpu_read_guest(vcpu, gpa, val, bytes); } =20 -static int write_emulate(struct kvm_vcpu *vcpu, gpa_t gpa, - void *val, int bytes) +static int emulator_write_guest(struct kvm_vcpu *vcpu, gpa_t gpa, + void *val, int bytes) { int ret; =20 @@ -8157,11 +8157,22 @@ static int emulator_read_write_onepage(unsigned lon= g addr, void *val, return X86EMUL_PROPAGATE_FAULT; } =20 - if (!ret && ops->read_write_emulate(vcpu, gpa, val, bytes)) + /* + * If the memory is not _known_ to be emulated MMIO, attempt to access + * guest memory. If accessing guest memory fails, e.g. because there's + * no memslot, then handle the access as MMIO. Note, treating the + * access as emulated MMIO is technically wrong if there is a memslot, + * i.e. if accessing host user memory failed, but this has been KVM's + * historical ABI for decades. + */ + if (!ret && ops->read_write_guest(vcpu, gpa, val, bytes)) return X86EMUL_CONTINUE; =20 /* - * Is this MMIO handled locally? + * Attempt to handle emulated MMIO within the kernel, e.g. for accesses + * to an in-kernel local or I/O APIC, or to an ioeventfd range attached + * to MMIO bus. If the access isn't fully resolved, insert an MMIO + * fragment with the relevant details. */ handled =3D ops->read_write_mmio(vcpu, gpa, bytes, val); if (handled =3D=3D bytes) @@ -8182,6 +8193,13 @@ static int emulator_read_write_onepage(unsigned long= addr, void *val, frag->data =3D val; } frag->len =3D bytes; + + /* + * Continue emulating, even though KVM needs to (eventually) do an MMIO + * exit to userspace. If the access splits multiple pages, then KVM + * needs to exit to userspace only after emulating both parts of the + * access. + */ return X86EMUL_CONTINUE; } =20 @@ -8279,7 +8297,7 @@ static int emulator_read_emulated(struct x86_emulate_= ctxt *ctxt, struct x86_exception *exception) { static const struct read_write_emulator_ops ops =3D { - .read_write_emulate =3D read_emulate, + .read_write_guest =3D emulator_read_guest, .read_write_mmio =3D vcpu_mmio_read, .write =3D false, }; @@ -8294,7 +8312,7 @@ static int emulator_write_emulated(struct x86_emulate= _ctxt *ctxt, struct x86_exception *exception) { static const struct read_write_emulator_ops ops =3D { - .read_write_emulate =3D write_emulate, + .read_write_guest =3D emulator_write_guest, .read_write_mmio =3D vcpu_mmio_write, .write =3D true, }; --=20 2.53.0.414.gf7e9f6c205-goog From nobody Fri Apr 17 00:17:57 2026 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9CBDA2652B7 for ; Wed, 25 Feb 2026 01:21:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982478; cv=none; b=d50Nfve4MNKmvE+GpBLOHeHidzCtHLwogQtSyY19SuX77Jz5uMmO31r6FbEdduwvXcCpopJCA6ztF5pmIsy1TWic3lfGLN+a607R58l/ohmv4cwtGnVoWGBhQnaBGSR8gqhxqoWPDZo0+k5zHGVxsX2ZOfca44k9VFRtWo+NG8s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982478; c=relaxed/simple; bh=Qy31+uRAyXc552YjYjFbiMoMOXbW3e9UP2BZgteKHMs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=QSxjpmL9A+jWNrmy0LiesOTaXrIYEVFIZoIOCiEQj5FrbwQUMaMCyiXsnhR5hMoDnCauOMIYLhRAPE9o1aK1MchTf5V46JJdYpCc//ebw3PV8pGC/uvE4cH4gDUMrO5KJinTjSqJ4EMGI8q36hM+AvfhvtJ7huXA6Ls048zBsQM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=zH1okmbq; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="zH1okmbq" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-358f058973fso906327a91.1 for ; Tue, 24 Feb 2026 17:21:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1771982477; x=1772587277; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=LC9/uqgNjMKR3Pk2VQH/jaz/lG7goMkRYlbLKdqdUyY=; b=zH1okmbqk7N039NOHaa7iZu7VCsU59eWPClkOHV62+tC+WvWTXksNXYyxMfvikYiVK N0UnKPFYdwmB4C+G408v4rAQJj/xXpC9XBFDs1CCbNcC4m/WFfWCpWpYlC6DrDANKXF+ UE1FqGWpX3Sel3Ta8mk7Y3LqtndiPn/fM1NOPqW54l9woHTnu9ZS7q9/hHykGeyWXbwc AutQdKJRNvw4h4/bQ5YaX2vQQgOlRUlMnc7YMzVvL3jWSs5NqjIxAivCDKoytSWb+6EW LUah5xsfGVcTR+jEQ6F3wB1jQBNFzRY/bDC1mW3yCHOpdAew93EKY+XL4PJ9waPCiVo9 NUOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771982477; x=1772587277; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=LC9/uqgNjMKR3Pk2VQH/jaz/lG7goMkRYlbLKdqdUyY=; b=lQnWPvF+hR9Xmg6PWEp5yeVSes9SkNmW64Kw5J1NB2p6a5K2Ul4pqF3pVFF/OC8xAl 0d5fus3a7baEjAlzZ0dtEHr2tzO8T4kbDY7mg8XNo2dkVgcB8YUdjzNsstwXcQuDEjI3 jIogRuHkZptk2TUAyp3rzfgs4nWSMXaa7nOvJGbKJHq+Zo2bp+mVlIPyksOUH+zvLUSp r383ArqLm5+x3XHpiStD/GHTNZprgT8Ye6ow5EEfFjJjKMYBrShzyXFBiRBACDqrQ45p JIpk1Xyq+qiU4o4KTdbjsCoTzfB7OS101zp1RL8b7QK+vkitm7WM+l4DR0LIaj/nxPXi MZIg== X-Forwarded-Encrypted: i=1; AJvYcCXPtamxe+ZX0sf3IP0vN+xMLyHaxO6Faz13AOALPYC2Cnk2IXMwj4Z3mmArtBhptS75hrt5RBk8FF8Lw3I=@vger.kernel.org X-Gm-Message-State: AOJu0YyrPqdAHK1de+sNj39fFbzwv6h5a/A8baw1AFmrjoSN7wRodIe+ EkxtHI258px09KN8V/9goAZm2W/PV/LNLViZqxoZkd2sNELZxrK8vsVwOqTGxiVBcdZ58dfAg1g hpxwQ7Q== X-Received: from pgbcz10.prod.google.com ([2002:a05:6a02:230a:b0:c6d:cb1b:286e]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:6a04:b0:394:8455:d1a9 with SMTP id adf61e73a8af0-39545e5081dmr10146147637.2.1771982476785; Tue, 24 Feb 2026 17:21:16 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 24 Feb 2026 17:20:48 -0800 In-Reply-To: <20260225012049.920665-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260225012049.920665-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.414.gf7e9f6c205-goog Message-ID: <20260225012049.920665-14-seanjc@google.com> Subject: [PATCH 13/14] KVM: x86: Don't panic the kernel if completing userspace I/O / MMIO goes sideways From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, Yashu Zhang , Rick Edgecombe , Binbin Wu , Xiaoyao Li , Tom Lendacky , Michael Roth Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Kill the VM instead of the host kernel if KVM botches I/O and/or MMIO handling. There is zero danger to the host or guest, i.e. panicking the host isn't remotely justified. Signed-off-by: Sean Christopherson Tested-by: Rick Edgecombe Tested-by: Tom Lendacky --- arch/x86/kvm/x86.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index f3e2ec7e1828..5376b370b4db 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -9710,7 +9710,8 @@ static int complete_fast_pio_in(struct kvm_vcpu *vcpu) unsigned long val; =20 /* We should only ever be called with arch.pio.count equal to 1 */ - BUG_ON(vcpu->arch.pio.count !=3D 1); + if (KVM_BUG_ON(vcpu->arch.pio.count !=3D 1, vcpu->kvm)) + return -EIO; =20 if (unlikely(!kvm_is_linear_rip(vcpu, vcpu->arch.cui_linear_rip))) { vcpu->arch.pio.count =3D 0; @@ -11820,7 +11821,8 @@ static inline int complete_emulated_io(struct kvm_v= cpu *vcpu) =20 static int complete_emulated_pio(struct kvm_vcpu *vcpu) { - BUG_ON(!vcpu->arch.pio.count); + if (KVM_BUG_ON(!vcpu->arch.pio.count, vcpu->kvm)) + return -EIO; =20 return complete_emulated_io(vcpu); } @@ -11849,7 +11851,8 @@ static int complete_emulated_mmio(struct kvm_vcpu *= vcpu) struct kvm_mmio_fragment *frag; unsigned len; =20 - BUG_ON(!vcpu->mmio_needed); + if (KVM_BUG_ON(!vcpu->mmio_needed, vcpu->kvm)) + return -EIO; =20 /* Complete previous fragment */ frag =3D &vcpu->mmio_fragments[vcpu->mmio_cur_fragment]; @@ -14262,7 +14265,8 @@ static int complete_sev_es_emulated_mmio(struct kvm= _vcpu *vcpu) struct kvm_mmio_fragment *frag; unsigned int len; =20 - BUG_ON(!vcpu->mmio_needed); + if (KVM_BUG_ON(!vcpu->mmio_needed, vcpu->kvm)) + return -EIO; =20 /* Complete previous fragment */ frag =3D &vcpu->mmio_fragments[vcpu->mmio_cur_fragment]; --=20 2.53.0.414.gf7e9f6c205-goog From nobody Fri Apr 17 00:17:57 2026 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A7E9D28EA56 for ; Wed, 25 Feb 2026 01:21:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982482; cv=none; b=m/i5ML29Le4/I25d6eV+B6KVbXL2nlpYmXTOStDd+vAe3n6khSvBoK+XnDQ1EnedUGjfKtiusBDVaX5x7JrtJLHiLs9FMJnHy53j+9mGIy7K6fQGo6pGpJy5wi56Cfg6xv+IP6VShnRm9sm5Rf5q7IYsFckmtimMVf0nVM6+VgQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1771982482; c=relaxed/simple; bh=IuDnIRUgFkFMGAUuxCS338MB/VHIWV/Tt1I66OqunQE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=F14CCMPcxXeRecUddx818kVLHMqSHScJ+8RvhgG0oHWTtVXfgMTEMugU2AK7Zmh+WeQQ28965IvgN/hCS5LDg+SPN4xTDhMAavqNXSKgejuCblBEddZYDqrZm2YGK9NTjwYDQevc0U4QQpiwwei8OO3idWwytHE1P9ClJ+oYyGA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=BH2WDDCB; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="BH2WDDCB" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-c65d08b623aso263800a12.0 for ; Tue, 24 Feb 2026 17:21:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1771982479; x=1772587279; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=06Kr2IGpW9wlaU49L7FaC2EfLlqxR0kfrYYtwtYSIqY=; b=BH2WDDCBdSTPYuywWHLI7FH7LB7c4yiB9eBtYCCMOsx8Y/Ue5MmmbBiwulwW6eTTzk MczA0kws+5MaxIGhEH/eifdoy13BY4330sIM5kgfyGfaodGuxnGJwlgeyrlg6e9NUzdv O8gH3rtGgDv4oCZZNq5mviSvkookc11ZvE2018R3eLA5+uPOeCCq13ucga57dzLFcgLz ehOLVBHrlnnsNYI813oQcLnvottbzM1nrZlMzTO8SKpfCJEZtVCGeyGoMHA/H8u63WM7 GtZDD01Gy2yk8nQg9BXRmu2XfPDgStsWJ5kh7QtNgiLEc7YJg/QOrySmKqYYH2O7DRCy EtGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1771982479; x=1772587279; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=06Kr2IGpW9wlaU49L7FaC2EfLlqxR0kfrYYtwtYSIqY=; b=ne5lnftkuUKRber3YgbKBaBcCGipD5qZKu8ZfR0hghrLW1q/pbEBV1YOVPcNTTbgaO VLCAPbuWQ+KAm99UOiPgQp4Qj0KPjZcsgF93cad0iYKZIIy+LPffb8J0+zlb5+IVxKSN 0tS9hgkZxvywu0y79tSFM+DUjzgRoPqv6XJYWBhCY8343+h5Cn16kLhgkzciTVXsDPnn bBMtUBYzC3CxY8D6vC09mBdvtIluTQxZ2E8RoKtlaU49+Tv+ggG0lc1/7b7VjLy8PeBy CknvB5CgWgoPhDGwP3Sf4Sksg1S2X4XPSPYQUUrVv0Mjjmu7pWMJjqzySGclXhb7GcuD jBIA== X-Forwarded-Encrypted: i=1; AJvYcCXUnvtzXui6agWS6PU/Vl2GpZMN72Ahxnm9wrTCCPd1SidKeQiKkhLdOj1/BDHDvHSp41KXMRuKeIlny9I=@vger.kernel.org X-Gm-Message-State: AOJu0YxGfQ669kChbKcwLNnu+jHvZS+kABlmvRM/uId6VXuvH7wsP2gM vDFUSMGDtlIDHl2jr8GFaqQ509J24dp5gJxoybf6bg09tax1o7Ezyhp7cX5BN6LYEqbmMbkwt3Q PGBiX5Q== X-Received: from pgeh5.prod.google.com ([2002:a05:6a02:53c5:b0:c6e:8a5c:a772]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:7d9b:b0:37c:b74c:d8c7 with SMTP id adf61e73a8af0-3959ac989c4mr1507787637.22.1771982478801; Tue, 24 Feb 2026 17:21:18 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 24 Feb 2026 17:20:49 -0800 In-Reply-To: <20260225012049.920665-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260225012049.920665-1-seanjc@google.com> X-Mailer: git-send-email 2.53.0.414.gf7e9f6c205-goog Message-ID: <20260225012049.920665-15-seanjc@google.com> Subject: [PATCH 14/14] KVM: x86: Add helpers to prepare kvm_run for userspace MMIO exit From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Kiryl Shutsemau Cc: kvm@vger.kernel.org, x86@kernel.org, linux-coco@lists.linux.dev, linux-kernel@vger.kernel.org, Yashu Zhang , Rick Edgecombe , Binbin Wu , Xiaoyao Li , Tom Lendacky , Michael Roth Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add helpers to fill kvm_run for userspace MMIO exits to deduplicate a variety of code, and to allow for a cleaner return path in emulator_read_write(). No functional change intended. Cc: Rick Edgecombe Cc: Binbin Wu Cc: Xiaoyao Li Cc: Tom Lendacky Cc: Michael Roth Signed-off-by: Sean Christopherson Tested-by: Rick Edgecombe Tested-by: Tom Lendacky --- arch/x86/kvm/vmx/tdx.c | 14 ++++---------- arch/x86/kvm/x86.c | 42 ++++++++---------------------------------- arch/x86/kvm/x86.h | 24 ++++++++++++++++++++++++ 3 files changed, 36 insertions(+), 44 deletions(-) diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 5df9d32d2058..a813c502336c 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1467,17 +1467,11 @@ static int tdx_emulate_mmio(struct kvm_vcpu *vcpu) =20 /* Request the device emulation to userspace device model. */ vcpu->mmio_is_write =3D write; - if (!write) + + __kvm_prepare_emulated_mmio_exit(vcpu, gpa, size, &val, write); + + if (!write) { vcpu->arch.complete_userspace_io =3D tdx_complete_mmio_read; - - vcpu->run->mmio.phys_addr =3D gpa; - vcpu->run->mmio.len =3D size; - vcpu->run->mmio.is_write =3D write; - vcpu->run->exit_reason =3D KVM_EXIT_MMIO; - - if (write) { - memcpy(vcpu->run->mmio.data, &val, size); - } else { vcpu->mmio_fragments[0].gpa =3D gpa; vcpu->mmio_fragments[0].len =3D size; trace_kvm_mmio(KVM_TRACE_MMIO_READ_UNSATISFIED, size, gpa, NULL); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 5376b370b4db..889a9098403c 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -8210,7 +8210,6 @@ static int emulator_read_write(struct x86_emulate_ctx= t *ctxt, const struct read_write_emulator_ops *ops) { struct kvm_vcpu *vcpu =3D emul_to_vcpu(ctxt); - struct kvm_mmio_fragment *frag; int rc; =20 if (WARN_ON_ONCE((bytes > 8u || !ops->write) && object_is_on_stack(val))) @@ -8268,12 +8267,9 @@ static int emulator_read_write(struct x86_emulate_ct= xt *ctxt, =20 vcpu->mmio_needed =3D 1; vcpu->mmio_cur_fragment =3D 0; + vcpu->mmio_is_write =3D ops->write; =20 - frag =3D &vcpu->mmio_fragments[0]; - vcpu->run->mmio.len =3D min(8u, frag->len); - vcpu->run->mmio.is_write =3D vcpu->mmio_is_write =3D ops->write; - vcpu->run->exit_reason =3D KVM_EXIT_MMIO; - vcpu->run->mmio.phys_addr =3D frag->gpa; + kvm_prepare_emulated_mmio_exit(vcpu, &vcpu->mmio_fragments[0]); =20 /* * For MMIO reads, stop emulating and immediately exit to userspace, as @@ -8283,11 +8279,7 @@ static int emulator_read_write(struct x86_emulate_ct= xt *ctxt, * after completing emulation (see the check on vcpu->mmio_needed in * x86_emulate_instruction()). */ - if (!ops->write) - return X86EMUL_IO_NEEDED; - - memcpy(vcpu->run->mmio.data, frag->data, min(8u, frag->len)); - return X86EMUL_CONTINUE; + return ops->write ? X86EMUL_CONTINUE : X86EMUL_IO_NEEDED; } =20 static int emulator_read_emulated(struct x86_emulate_ctxt *ctxt, @@ -11884,12 +11876,7 @@ static int complete_emulated_mmio(struct kvm_vcpu = *vcpu) return complete_emulated_io(vcpu); } =20 - run->exit_reason =3D KVM_EXIT_MMIO; - run->mmio.phys_addr =3D frag->gpa; - if (vcpu->mmio_is_write) - memcpy(run->mmio.data, frag->data, min(8u, frag->len)); - run->mmio.len =3D min(8u, frag->len); - run->mmio.is_write =3D vcpu->mmio_is_write; + kvm_prepare_emulated_mmio_exit(vcpu, frag); vcpu->arch.complete_userspace_io =3D complete_emulated_mmio; return 0; } @@ -14296,15 +14283,8 @@ static int complete_sev_es_emulated_mmio(struct kv= m_vcpu *vcpu) } =20 // More MMIO is needed - run->mmio.phys_addr =3D frag->gpa; - run->mmio.len =3D min(8u, frag->len); - run->mmio.is_write =3D vcpu->mmio_is_write; - if (run->mmio.is_write) - memcpy(run->mmio.data, frag->data, min(8u, frag->len)); - run->exit_reason =3D KVM_EXIT_MMIO; - + kvm_prepare_emulated_mmio_exit(vcpu, frag); vcpu->arch.complete_userspace_io =3D complete_sev_es_emulated_mmio; - return 0; } =20 @@ -14333,23 +14313,17 @@ int kvm_sev_es_mmio(struct kvm_vcpu *vcpu, bool i= s_write, gpa_t gpa, * requests that split a page boundary. */ frag =3D vcpu->mmio_fragments; - vcpu->mmio_nr_fragments =3D 1; frag->len =3D bytes; frag->gpa =3D gpa; frag->data =3D data; =20 vcpu->mmio_needed =3D 1; vcpu->mmio_cur_fragment =3D 0; + vcpu->mmio_nr_fragments =3D 1; + vcpu->mmio_is_write =3D is_write; =20 - vcpu->run->mmio.phys_addr =3D gpa; - vcpu->run->mmio.len =3D min(8u, frag->len); - vcpu->run->mmio.is_write =3D is_write; - if (is_write) - memcpy(vcpu->run->mmio.data, frag->data, min(8u, frag->len)); - vcpu->run->exit_reason =3D KVM_EXIT_MMIO; - + kvm_prepare_emulated_mmio_exit(vcpu, frag); vcpu->arch.complete_userspace_io =3D complete_sev_es_emulated_mmio; - return 0; } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_sev_es_mmio); diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 1d0f0edd31b3..d66f1c53d2b5 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -718,6 +718,30 @@ int kvm_sev_es_string_io(struct kvm_vcpu *vcpu, unsign= ed int size, unsigned int port, void *data, unsigned int count, int in); =20 +static inline void __kvm_prepare_emulated_mmio_exit(struct kvm_vcpu *vcpu, + gpa_t gpa, unsigned int len, + const void *data, + bool is_write) +{ + struct kvm_run *run =3D vcpu->run; + + run->mmio.len =3D min(8u, len); + run->mmio.is_write =3D is_write; + run->exit_reason =3D KVM_EXIT_MMIO; + run->mmio.phys_addr =3D gpa; + if (is_write) + memcpy(run->mmio.data, data, min(8u, len)); +} + +static inline void kvm_prepare_emulated_mmio_exit(struct kvm_vcpu *vcpu, + struct kvm_mmio_fragment *frag) +{ + WARN_ON_ONCE(!vcpu->mmio_needed || !vcpu->mmio_nr_fragments); + + __kvm_prepare_emulated_mmio_exit(vcpu, frag->gpa, frag->len, frag->data, + vcpu->mmio_is_write); +} + static inline bool user_exit_on_hypercall(struct kvm *kvm, unsigned long h= c_nr) { return kvm->arch.hypercall_exit_enabled & BIT(hc_nr); --=20 2.53.0.414.gf7e9f6c205-goog