From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3C5F3381AE2 for ; Fri, 29 May 2026 22:22:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093366; cv=none; b=pRrACxQVwgbasz0Y1bk0MD3TOICXEgylJHklFJ31boaBdkyWN/bJggvjORU2JNqoQDXTeaImi2NoAfm6+6pk1ZZfShKoKox3//BLxZh5lXd0m988WDeUQCHFprJdpPpq6jTJfZ86K1U3UxLpVapmNqWa6opuOmP1j6jxUebXCJQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093366; c=relaxed/simple; bh=HVJcLc1+iQzcxGBbY8rv4XaVVOVl53XEoJsD6aE5Kn0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=pzC3H0BB1FK2e7jv4r5qKZuf0kkwwjuVR4YPMCywH2SlbFWKyg/wB7HFyKxlNtQ+x/TIyx2oPKHdsOCRoWlLKH5q0g3BE2zoWwMzhiNo3XdFcTxa7jqS+k8raTKGvR/6SBQemcKOZHqa4K5k+cUyKYjjijiyrLC4SU2P3M/6yeI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=iKzM47U+; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="iKzM47U+" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2bd6cc53fd6so149733305ad.3 for ; Fri, 29 May 2026 15:22:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093364; x=1780698164; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=PzVLZm7rmlBKzWSViLi5m09WnJImm2miEV9qMFfztnU=; b=iKzM47U+iSl180sQHYVr1wKo4CPWCQ/Yj/s/9adVUcQ5ahi+qYqy21LrNss7p76Kvu 96xSOcf2xnO4j7h49+8wSC30hvCsS+ygQ0YjPE2ias3WNhGoYw9rX2ytvDa9RjcWOtI3 juUDcfqQYcq+WFTxHAIdDNjP+5ZRmKXhlHKKUo9s1FyrdD30jKKYDcQM5F0NKG/JYVke lkyhu6j96M9FLX2OEUf9hVWqgIUixgNF+IIGoq/N7f1NFu15FKu7MgVYxNrq8Lfkkn3I +t15RHSxvPiM17DWKuofD1UuiDY/zhEOmwTdEx0ql4I7fzfz2c/C5sm1AoJoz8Tz5NMg s7vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093364; x=1780698164; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=PzVLZm7rmlBKzWSViLi5m09WnJImm2miEV9qMFfztnU=; b=WppQr8D5cKD1RXtQ4oB52rGJzKeHbd25mWQKA9yv4INFA7lptf5Hy0IyH1yH/eo8B4 SdYEn+xT7yXmrJQS3+MOIPnceU3CqNzsbBl502own23MCdgcRPfkPpWnmFSdAwbjvYKD DyLl8TCdIb/jiYac9fWTyXWXgE/AdE1o/v1BmZT2Br3bu+roMcyXlAJ1HtAZfNjOTH1P ZRsxxJBqrZ6WgLnIZJ4/qT9Gjh3gOEyJhsk+cGpha1RGUpwr7EqM5cWJporOxj+af3Q7 cdz5oMGqXySIU9Gy3mPbCPOiVwU3W3ZTiMFMpfBuTVmUla/cJnoZ3FvkGDbhZaMacJDb xyEg== X-Forwarded-Encrypted: i=1; AFNElJ/3o7R6BI3V/Umw4D/Wita/mP0hfJRfwQVX6cB/iXKfnTR/bftCHrfQSolVp+qFUKD5p8Qu8JHzdJj/T8w=@vger.kernel.org X-Gm-Message-State: AOJu0YwdkcXp+2t8s6yVw6ultynlbD82oOZt21lL3RhdVAGiNt+NwShX toHi7JRZ9eUiy5Q+gNCdUFxw5b+Q39DtJrvX0BFx5BfnYBwOQn52gmHSVZSmoRYunQ1ziOz+kPt MMs54YQ== X-Received: from plmc1.prod.google.com ([2002:a17:903:e81:b0:2bd:859d:8959]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:1b64:b0:2bf:e5c:d90b with SMTP id d9443c01a7336-2bf36858d29mr19511515ad.32.1780093364326; Fri, 29 May 2026 15:22:44 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:21:44 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-2-seanjc@google.com> Subject: [PATCH v3 01/40] KVM: SVM: Truncate INVLPGA address in compatibility mode From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Check for full 64-bit mode, not just long mode, when truncating the virtual address as part of INVLPGA emulation. Compatibility mode doesn't support 64-bit addressing. Note, the FIXME still applies, e.g. if the guest deliberately targeted EAX while in 64-bit via an address size override. That flaw isn't worth fixing as it would require decoding the code stream, which would open an entirely different can of worms, and in practice no sane guest would shove garbage into RAX[63:32] and execute INVLPGA. Note #2, VMSAVE, VMLOAD, and VMRUN all suffer from the same architectural flaw of not providing the full linear address in a VMCB exit information field, because, quoting the APM verbatim: the linear address is available directly from the guest rAX register (VMSAVE, VMLOAD, and VMRUN take a physical address, but their behavior with respect to rAX is otherwise identical). Fixes: bc9eff67fc35 ("KVM: SVM: Use default rAX size for INVLPGA emulation") Reviewed-by: Yosry Ahmed Reviewed-by: Binbin Wu Signed-off-by: Sean Christopherson --- arch/x86/kvm/svm/svm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 717af5c4d057..7d8a433b5c5e 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -2416,7 +2416,7 @@ static int invlpga_interception(struct kvm_vcpu *vcpu) return 1; =20 /* FIXME: Handle an address size prefix. */ - if (!is_long_mode(vcpu)) + if (!is_64_bit_mode(vcpu)) gva =3D (u32)gva; =20 trace_kvm_invlpga(to_svm(vcpu)->vmcb->save.rip, asid, gva); --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B74E3BC668 for ; Fri, 29 May 2026 22:22:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093367; cv=none; b=lXEk9XEcJHSXUsIofoDxNrnEQ8CWM7CYsDqMTQ5EXBNY8TgmMNI3Gw1QRoAlZncNWZGzczRsiwL1QIckOzUChn9fj2mgTKGEtkpyV5vQCEoCkJPTCradiCGMLAYRta78owwtnTiC/pCUOb0Aw1l2s60PRpdxDs76G22nl+2ElWo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093367; c=relaxed/simple; bh=CdEPrCw2bOZnD6mQAwYnCo+zNToy/znjkjkLqqXPq+0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=b/lT5gdPkEIoEkfXiRSzAKzaEBy/yxbOV2h5EeTPfRdDFDYwvY73Zk6caP9rULpDIS/Nrzxf5WvpHmhpPgygAaYpoUR5RFmMednXirmwxmthvdJgoQo7E9s3hg87gWvyXbS+e6KO8GTkB0fdO8kWHPsQ3A55cfiIwEyzSyF+1wA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=A3nFK79q; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="A3nFK79q" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-36ba98cc003so1282595a91.1 for ; Fri, 29 May 2026 15:22:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093365; x=1780698165; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=8I5HJekqNdLlzk9wWpRps9qAFoWEmeMnW4XpaBDLerI=; b=A3nFK79qpr5i2829Ll+p6ndqC8iAb8z2OnTP3h+gg1//PWgZ/0C6hAt2cW3bObed1Q 07ZUJkkJZOWXQ4u3claoMPIDJDQOiziIu4tN/g6+7R6kxPUJ64uuKdWa/0yL4veOR+o+ lmawTYnKoD2LuRBtSWJW1vifbnvHjRI5df7i3KONVf9s634br4b/gm/YkhPgiGVrwBli qMVLVgCc1LXII62EK2hyWQFN7PujOXF87woqEuk5I8D4nI6O17n9mGot3riDZ8qRJyDA YfOIIfn0niJrUi77b/GmpccUF5keHthiRAG9yRAWB2GBYy4ntu2zFW1d281xgHxYXnKW mm0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093365; x=1780698165; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=8I5HJekqNdLlzk9wWpRps9qAFoWEmeMnW4XpaBDLerI=; b=OxREFX1Zqzro6TJoJMVOm89CvM6Spjp8xG1PT8CH2JEdRdbULK3T1uwRPYGa15nmbk 8FpQ1j0g1GSGpkl+psLPeo28aUoVjGW9rUq/Nvxbfp/FH6I/jqiUBzG/Zr5zuJ5iawDH 3uFnI6Kze8+FVLts7ZI7wo2RAEcM2ETveP13rbRL65Nd9p1UYSDfOAUZe/uJmf9AO6ju 712BhOi552r9VtTYXbJupxj1zs36h4zS5mdAlRtGCdk70NmlwSeyEnLSm/OCn7rOIjMN 3VQ/mCuUdN8G5ucAl50w+DXA1zA1BcbYkZ7ZAv+0DKFVh/0ZCsb4IRmNHq0tByUrOav1 bi7g== X-Forwarded-Encrypted: i=1; AFNElJ//Sb+zRzMqEAnsqV8wvzKwK8nXfwpWezpb8RgF5Z37X1ufF1TDeRhFrUOeatdYbsl7YJ6Jc7GMmq8mFwU=@vger.kernel.org X-Gm-Message-State: AOJu0Yx1Xe5yvmIQP3XFpXMJn8l5At/Y0FvMoAN8Xsm/DhRm8CVEbK+i 3IRhT5+jY721vEcY3/qZM8XTQJmUn8R9OyygEgpCWdqVnbjJljsAx9a0NHb2rP6IW3mFtF74162 dv5MFpw== X-Received: from pgam4.prod.google.com ([2002:a05:6a02:2b44:b0:c85:85c9:de8c]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:50cc:b0:369:b9db:b885 with SMTP id 98e67ed59e1d1-36c501db446mr1012982a91.15.1780093365369; Fri, 29 May 2026 15:22:45 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:21:45 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-3-seanjc@google.com> Subject: [PATCH v3 02/40] KVM: x86/xen: Bug the VM if 32-bit KVM observes a 64-bit mode hypercall From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Bug the VM if 32-bit KVM attempts to handle a 64-bit hypercall, primarily so that a future change to set "input" in mode-specific code doesn't trigger a false positive warn=3D>error: arch/x86/kvm/xen.c:1687:6: error: variable 'input' is used uninitialized whenever 'if' condition is false [-Werr= or,-Wsometimes-uninitialized] 1687 | if (!longmode) { | ^~~~~~~~~ arch/x86/kvm/xen.c:1708:31: note: uninitialized use occurs here 1708 | trace_kvm_xen_hypercall(cpl, input, params[0], params[1],= params[2], | ^~~~~ x86/kvm/xen.c:1687:2: note: remove the 'if' if its condition is always tr= ue 1687 | if (!longmode) { | ^~~~~~~~~~~~~~ arch/x86/kvm/xen.c:1677:11: note: initialize the variable 'input' to sile= nce this warning 1677 | u64 input, params[6], r =3D -ENOSYS; | ^ 1 error generated. Note, params[] also has the same flaw, but -Wsometimes-uninitialized doesn't seem to be enforced for arrays, presumably because it's difficult to avoid false positives on specific entries. Reviewed-by: Binbin Wu Signed-off-by: Sean Christopherson --- arch/x86/kvm/xen.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c index 91fd3673c09a..6d9be74bb673 100644 --- a/arch/x86/kvm/xen.c +++ b/arch/x86/kvm/xen.c @@ -1694,16 +1694,19 @@ int kvm_xen_hypercall(struct kvm_vcpu *vcpu) params[4] =3D (u32)kvm_rdi_read(vcpu); params[5] =3D (u32)kvm_rbp_read(vcpu); } -#ifdef CONFIG_X86_64 else { +#ifdef CONFIG_X86_64 params[0] =3D (u64)kvm_rdi_read(vcpu); params[1] =3D (u64)kvm_rsi_read(vcpu); params[2] =3D (u64)kvm_rdx_read(vcpu); params[3] =3D (u64)kvm_r10_read(vcpu); params[4] =3D (u64)kvm_r8_read(vcpu); params[5] =3D (u64)kvm_r9_read(vcpu); - } +#else + KVM_BUG_ON(1, vcpu->kvm); + return -EIO; #endif + } cpl =3D kvm_x86_call(get_cpl)(vcpu); trace_kvm_xen_hypercall(cpl, input, params[0], params[1], params[2], params[3], params[4], params[5]); --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 933ED3C1977 for ; Fri, 29 May 2026 22:22:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093368; cv=none; b=rpmDRLsxOkUyTdSRRK7JLmqRSLCJ+FlocmN9/jjqYm0LDBjtACEuOWKJpF4QssVW2dGfeWbw3dhuMe6u4l0FGLokOKTACDRW0lQ+Tek2mwSGsvRFEoNZjlYQjhpuzFe4PG6fH9uLP1gYWQARa/EgZoINApt93SZggQuKPmcyQn4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093368; c=relaxed/simple; bh=kHQdd89SpJpJbZYHLaSOdch/rT72KJmxA2Z0ED5ycMo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=RsJqUCuAk4PRKZEKU3gSWX8NBsJiVgAAatrkluBGV3jAxErQVYqjIexm5xZeXbP2hQfIKCGWSG3aT8wiz5ywRSDq1ySQH8wH0wWQkqGlomaeU7ib8E93WUxNjxodKRTuq7Tw8PpLxCHORAhsFaMiSjoyoVQO2zEKtGqjYbtORtk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=R0fdLgfW; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="R0fdLgfW" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2bd04e4fe3dso118358895ad.3 for ; Fri, 29 May 2026 15:22:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093367; x=1780698167; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=6RQHD5fTjvOKMw+g+RyJJ/KDM3+OIdVSezR/Zl8r0J0=; b=R0fdLgfWOBdT5OtgTCeGyrxtPWrxLUuFjAwfa0kr+Eb2S4i4+773kFvlOhqmaj+5qK 9nMES+jz8PLxHT6KeNAJe4zLdqx9dlp+No6FyrsBPzN/VGSa1mcBTdvjV6+Aj5SjODN+ j7T+wM2thm3X4t4hXFb64Lj1yrdyOIX9hVmNmvD/vuQpLlo65vJL1S5k9WquVB3svXIS 2LueXMuIl17/FQufuifoUtx6NPEXC1GJ5jWJ7AA7DR1XZCwAjLYdUeidt+q/J83jYTLU fw5mmTeF0AmPsuMSvrMM+M//yCCBMfng8A6emhfRRh2SeZCp9v1uktin8797j7y0ljIb gObA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093367; x=1780698167; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=6RQHD5fTjvOKMw+g+RyJJ/KDM3+OIdVSezR/Zl8r0J0=; b=HcKdyWyi5QY9v3kxfzzAGgMCt1KUg0MeZX12T4HheKA2xdQePGm8gKQvQLgfp0JldB l/VAjI8814b6VUSBlDBS2p4jFvr27wBG8bz95INwUIX+hbGnr8LcmUV+hEdNUhKVDxRv B1yqDzwMDBDrb3+9DKp0K4IKvrJoJxXDdS2ox5ZOHyMZGn6OXSlU1MrEJUQ2FA4UbJn1 o4vrzxlkeK4tz2zqEz2T0rPw9H3IGeysoi37FycLblrraKREOKzSOI/LaOI/hYN7+Irk NFoHhiPoPZddPV8hFW81rWnxybQ7A/O8Eb6NX9jIIwTJdsA+AIkrUgPgP7UVXDGI76HL +LFw== X-Forwarded-Encrypted: i=1; AFNElJ/SNym0KpJcaBbr0+dhp+yYEv5lOHocA7Q0jYD8ZM3pkQORJnUf3bw5vagLL0d56IUThJ8eQmIdyOEfb/I=@vger.kernel.org X-Gm-Message-State: AOJu0YxKNcfW34JKDnKZXMKhBtP7A2aDc2Z/rQOpaw+4jAlkzSrpmDx+ GigThwTC5/jA6hZZfN61WyfgRFqINS8lhqRe93AyFdojWsr6vFpQjiomQ8HbczOOkKw/Wn1Eemw cCmGEvw== X-Received: from pgnp18.prod.google.com ([2002:a63:7f52:0:b0:c80:dc0a:9f7f]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:20c3:b0:3aa:ec28:edc6 with SMTP id adf61e73a8af0-3b428248dcbmr1208955637.39.1780093366486; Fri, 29 May 2026 15:22:46 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:21:46 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-4-seanjc@google.com> Subject: [PATCH v3 03/40] KVM: x86/xen: Don't truncate RAX when handling hypercall from protected guest From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Don't truncate RAX when handling a Xen hypercall for a guest with protected state, as KVM's ABI is to assume the guest is in 64-bit for such cases (the guest leaving garbage in 63:32 after a transition to 32-bit mode is far less likely than 63:32 being necessary to complete the hypercall). Fixes: b5aead0064f3 ("KVM: x86: Assume a 64-bit hypercall for guests with p= rotected state") Reviewed-by: David Woodhouse Signed-off-by: Sean Christopherson --- arch/x86/kvm/xen.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c index 6d9be74bb673..895095dc684e 100644 --- a/arch/x86/kvm/xen.c +++ b/arch/x86/kvm/xen.c @@ -1678,15 +1678,14 @@ int kvm_xen_hypercall(struct kvm_vcpu *vcpu) bool handled =3D false; u8 cpl; =20 - input =3D (u64)kvm_register_read(vcpu, VCPU_REGS_RAX); - /* Hyper-V hypercalls get bit 31 set in EAX */ - if ((input & 0x80000000) && + if ((kvm_rax_read(vcpu) & 0x80000000) && kvm_hv_hypercall_enabled(vcpu)) return kvm_hv_hypercall(vcpu); =20 longmode =3D is_64_bit_hypercall(vcpu); if (!longmode) { + input =3D (u32)kvm_rax_read(vcpu); params[0] =3D (u32)kvm_rbx_read(vcpu); params[1] =3D (u32)kvm_rcx_read(vcpu); params[2] =3D (u32)kvm_rdx_read(vcpu); @@ -1696,6 +1695,7 @@ int kvm_xen_hypercall(struct kvm_vcpu *vcpu) } else { #ifdef CONFIG_X86_64 + input =3D (u64)kvm_rax_read(vcpu); params[0] =3D (u64)kvm_rdi_read(vcpu); params[1] =3D (u64)kvm_rsi_read(vcpu); params[2] =3D (u64)kvm_rdx_read(vcpu); --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D23833E0C5C for ; Fri, 29 May 2026 22:22:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093376; cv=none; b=KLSnjndVu7ZK+Q4fseIzyP9UxkfUrXrGu4KYVOQrs8TCq+M3EjhETJDiOJWCiwWPv/v2FcGt3uo4l9ZZPqxXfOy9uaE3pw+Tk0cCgRI6586A33aYQO8l9Q4sd9xPmVYe65wFPYgX196dqyNPvcm+2Qh9T4ulFup8tZPsXMcHOWI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093376; c=relaxed/simple; bh=rNEMOfQP0lISV4GXkkh7r4YHYgmX95gg+RO88qDWtLA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=qbfuR7/1IvvykrCW9kVrH4vmIPcyyb1VipofhBnIb2yILtguoZx+LjzaeYdO0O2nDFckTd6sy/DfCTvCFnCeh2Ok/vFl6FPGa01fD0r80ZEPtYpFldKVPglhjkmrX/9UZTkMw+c63pdpB3jg2AOzK8d0/ufTxfSh2/LUg21g5uA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=X9FoXxLR; arc=none smtp.client-ip=209.85.210.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="X9FoXxLR" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-8422aa07fe2so58596b3a.2 for ; Fri, 29 May 2026 15:22:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093368; x=1780698168; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=IFUIQ/grqF4H0Xyudg/Qm5uUCCNXYoJ9yfo9zM/sbkk=; b=X9FoXxLRCfoRy11dmMlzJPQzdEcFkvbPH4ZyXOxHrh4rD0UqB7nI33XDYdQKetSF2+ Hn4oTGhlbu4I9Lb04v7ivE2rDtvUTJEK+cihxP17PmT8putSG5JOeS38AztDHoVwQUyw Kle1YmsYH75pldbcqkacPJBMYze15F9YbdVh5wG4onY2CMhSWQHHEN/9Zv/SL6NTW/n6 bTB8GoMgZ3F4T9ImySSQPJXOvYgl2StdOOmGQnPYu8C74xZunZmM2EOFRJLEl0TajVex poI7u9ofCxukYtw9qKhPfgWab8NXqE2IY8JhpHT661RcI0w36njfuXdzPojN0S8ZeqC0 TDgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093368; x=1780698168; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=IFUIQ/grqF4H0Xyudg/Qm5uUCCNXYoJ9yfo9zM/sbkk=; b=G2sFqs+i/y+W8UHOYmT0vnCjnrK7JZlQ+WH/kgYyI6fo5jp2SBbb4m4haJTki/bnLU lu2diKW3zT9fl2z20SfhMHJuKBlgpIKSd/s9dy388RXU/xb5FxFvH1NR9X9PgdqGf5S7 FD8IDA+e1q4eAXfk0hed3vsJD7QIqZNGGD8mJU0fH2Uzw4oqWKLRKbgHfEbIWYxP3Yg0 5xDC/lr0JYH60saj8wy4PB12768377UAHtKcTSTYD4OiAC50bAulwMrlNhEGkthMJ8qG o0ictqALb7q4ACL9BYJqfHnyY6OjKkDrBj28Z0dvmzyA/fYQXwKJicuX+qriUILB9EU2 JnOQ== X-Forwarded-Encrypted: i=1; AFNElJ+8zuhSwqo3B8o8RyStrl9DXbwVAUdFz8Cou66nauYHKi7LE/GgPuV1aG0qY+tqDxAqngygYHo6Obc/Q4I=@vger.kernel.org X-Gm-Message-State: AOJu0YzbnFLsQEbXWNPcUZgzSqSjuqKBPedAKM9VtkmahCVYmPYxsHid Wzzphnv3QjGUYmVpP+jrv4pEm8pwB0aGP0ic58I3tyaqkFiQsK3oVxhCvI8AvMcNP71LoHjfXk5 FLUBpeg== X-Received: from pfbk22.prod.google.com ([2002:a05:6a00:b016:b0:842:1f96:2713]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:27a0:b0:833:2398:cde2 with SMTP id d2e1a72fcca58-8422570a361mr1184893b3a.43.1780093367744; Fri, 29 May 2026 15:22:47 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:21:47 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-5-seanjc@google.com> Subject: [PATCH v3 04/40] KVM: VMX: Read 32-bit GPR values for ENCLS instructions outside of 64-bit mode From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When getting register values for ENCLS emulation, use kvm_register_read() instead of kvm__read() so that bits 63:32 of the register are dropped if the guest is in 32-bit mode. Note, the misleading/surprising behavior of kvm__read() being "raw" variants under the hood will be addressed once all non-benign bugs are fixed. Fixes: 70210c044b4e ("KVM: VMX: Add SGX ENCLS[ECREATE] handler to enforce C= PUID restrictions") Fixes: b6f084ca5538 ("KVM: VMX: Add ENCLS[EINIT] handler to support SGX Lau= nch Control (LC)") Acked-by: Kai Huang Reviewed-by: Binbin Wu Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/sgx.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/vmx/sgx.c b/arch/x86/kvm/vmx/sgx.c index df1d0cf76947..4c61fc33f764 100644 --- a/arch/x86/kvm/vmx/sgx.c +++ b/arch/x86/kvm/vmx/sgx.c @@ -225,8 +225,8 @@ static int handle_encls_ecreate(struct kvm_vcpu *vcpu) struct x86_exception ex; int r; =20 - if (sgx_get_encls_gva(vcpu, kvm_rbx_read(vcpu), 32, 32, &pageinfo_gva) || - sgx_get_encls_gva(vcpu, kvm_rcx_read(vcpu), 4096, 4096, &secs_gva)) + if (sgx_get_encls_gva(vcpu, kvm_register_read(vcpu, VCPU_REGS_RBX), 32, 3= 2, &pageinfo_gva) || + sgx_get_encls_gva(vcpu, kvm_register_read(vcpu, VCPU_REGS_RCX), 4096,= 4096, &secs_gva)) return 1; =20 /* @@ -302,9 +302,9 @@ static int handle_encls_einit(struct kvm_vcpu *vcpu) gpa_t sig_gpa, secs_gpa, token_gpa; int ret, trapnr; =20 - if (sgx_get_encls_gva(vcpu, kvm_rbx_read(vcpu), 1808, 4096, &sig_gva) || - sgx_get_encls_gva(vcpu, kvm_rcx_read(vcpu), 4096, 4096, &secs_gva) || - sgx_get_encls_gva(vcpu, kvm_rdx_read(vcpu), 304, 512, &token_gva)) + if (sgx_get_encls_gva(vcpu, kvm_register_read(vcpu, VCPU_REGS_RBX), 1808,= 4096, &sig_gva) || + sgx_get_encls_gva(vcpu, kvm_register_read(vcpu, VCPU_REGS_RCX), 4096,= 4096, &secs_gva) || + sgx_get_encls_gva(vcpu, kvm_register_read(vcpu, VCPU_REGS_RDX), 304, = 512, &token_gva)) return 1; =20 /* --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC6B83C140F for ; Fri, 29 May 2026 22:22:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093376; cv=none; b=HBEjXQ3ULn8J9aVyzPHsymXYQycVetTtEipc1Gq6joB8pwmBBcrXFVo/HKYv6cOW8PWgrAbuqgbSARqAWteNt1vnj8mZxRBd0AyRhkXSfZKjj+aVtxbM3+GgyjjPSjD2oDRLBoyY6IZRokRFaYEP/oJ/+xK45xcV/CuRFhnDYvE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093376; c=relaxed/simple; bh=lwn3MA9KM8thDveYybS6qHk91YxwQNF6NdtAA1Sy97c=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=if2QrO7SLzkwQzRAI5FV3lz2b2PTs/urTst4KvJuOtYxUXzDA68U0r3JqKiDXEQ7Kx2D0aJcHP7fg0bJfKkhz+uzIC5jLPZQj0d7MCnLrbrbmam71ZitBQNeZGIO4pC7Sp7OTbJYZtaO5NLsNrqxgjxh6a4pHFSZXetrQIDyBzs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=bULHbIyK; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="bULHbIyK" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2befec3fd8fso18467945ad.3 for ; Fri, 29 May 2026 15:22:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093369; x=1780698169; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=oUZ7pkbs+pCLPe3p8XbPZerSxi7aaVAC0nyW5XYMnDo=; b=bULHbIyK3h9GW/tzmxS3b/8SiDJa+O3vJo7ExMatUIj6icP4f+8KdmVfeJdu0saBqq KohALdQZjiCCs5fteLJt7a/80QwqZSKrW/HKxjiQDfOVX1JidN7mh81cpafb58A2Hwhq cy7qcWiUAbdfS5dxKxROzjvb2Jqc/qkcDU2lwBewBTVj5TOzJiHzCDQtE8ry3G8t7NZ0 wTn4IuGfcefqRf1l7PlF0WdG2WgjQ38Xoh/6HQIb/5rMTKFfQ8PUBVEPAaWsx2PvsUOx wI/gF7yuP/P0h1gnJh5JMsBPE+gx06Yi2TirSYTJOoO9X0MNi78qjFcxaooc6OL6bLBP L6YA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093369; x=1780698169; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=oUZ7pkbs+pCLPe3p8XbPZerSxi7aaVAC0nyW5XYMnDo=; b=F1QosCMLvzyyE0IemW7MzXYcAaJImmJgYz8I0DvrbHrGX5S0U+WejEomRBv1Z7AO1d tSKjIPEnbtZgVgCAY+6fwOG6e1SPVqllY1Ux47iS5O2jp1fZcXfgnFwwCv+GoSa/PP0r eYV/SNPJaqpKrMzvOq1cvO5WQqQF4zM4AiSjlCmBSEn/VOaO13PMrr97bzEYVEeYK1d2 ph+IRDbT2aqoz82KXTZUdIfXMOjBJ8ykEZepfh4M2+nAHdaehknLrn4RCm3XOOlk3b4t L7coC60USueCRyEXSu5Q3r7QBvjmSurQ4VtPeRj4kODK0SYE/HG8PL0mvxKsqyldef8c xVUQ== X-Forwarded-Encrypted: i=1; AFNElJ/kBfCe8NiHF9zN6sz7NHgTgfMbvshRcS/RP3MnRoR2PiNhDwCgDtatquzeIy2yupBDr680CuiD7vVUCZ8=@vger.kernel.org X-Gm-Message-State: AOJu0Yw0Ph703duqiRbOQfVo+8ief/xpcv7q8FToWe8A6iXR7tgIAnpp pzQIDM9AWwgFRsSSWUDnzVMUd+LtFJXBqEPWIiXA7fDwGpiJpSMhMuLNsAyffzIxM/MVU/6Cw8o 1sk4dBw== X-Received: from plbmn14.prod.google.com ([2002:a17:903:a4e:b0:2bf:b2c:fa1d]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:1cb:b0:2b0:6e6a:8504 with SMTP id d9443c01a7336-2bf36845c9amr19204545ad.27.1780093368939; Fri, 29 May 2026 15:22:48 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:21:48 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-6-seanjc@google.com> Subject: [PATCH v3 05/40] KVM: x86: Trace hypercall register *after* truncating values for 32-bit From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" When tracing hypercalls, invoke the tracepoint *after* truncating the register values for 32-bit guests so as not to record unused garbage (in the extremely unlikely scenario that the guest left garbage in a register after transitioning from 64-bit mode to 32-bit mode). Fixes: 229456fc34b1 ("KVM: convert custom marker based tracing to event tra= ces") Reviewed-by: Yosry Ahmed Reviewed-by: Binbin Wu Signed-off-by: Sean Christopherson --- arch/x86/kvm/x86.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index e6f1dd84f22d..634a6d0197f6 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -10429,8 +10429,6 @@ int ____kvm_emulate_hypercall(struct kvm_vcpu *vcpu= , int cpl, =20 ++vcpu->stat.hypercalls; =20 - trace_kvm_hypercall(nr, a0, a1, a2, a3); - if (!op_64_bit) { nr &=3D 0xFFFFFFFF; a0 &=3D 0xFFFFFFFF; @@ -10439,6 +10437,8 @@ int ____kvm_emulate_hypercall(struct kvm_vcpu *vcpu= , int cpl, a3 &=3D 0xFFFFFFFF; } =20 + trace_kvm_hypercall(nr, a0, a1, a2, a3); + if (cpl) { ret =3D -KVM_EPERM; goto out; --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 064531A6801 for ; Fri, 29 May 2026 22:22:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093377; cv=none; b=RMTKuL5tB1gxVQxghv6wtANrN6N/lID+aLE0102/sjUfSaYgOwNZUY9v11DJiTXEBcY4rGgyAsAN4BAxGMLwTkABDWTpp/skv1Rl76CybzOsY5tW8sTUIm2alkpGCyzTSXnFj1X3a2U1AOLUromwcQ2E+H4CBIh/NFaaXRqvuUw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093377; c=relaxed/simple; bh=qorWr+Wuyaa1/wm5b2Bbu4vbtL+yom4vUJcmXV5SO2k=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=FyuB3n7JzJGE1ABTV8pxAg/7FCV0wsvTWTIZWTOUkOUoFKmi3TPs5VsVNx/Cbc4Cy9LR0y2sfaWlFDkdNvGrCnx2XdyXqfai6wGL9697yYvmSSNuJxXRlJiDqVvxJbbiZ/krT5D7WdW9sQIgAwsVhmIn3rZyeuEwUnyxhy182g4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=s6Azm6q6; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="s6Azm6q6" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-c82ac893940so7300452a12.1 for ; Fri, 29 May 2026 15:22:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093370; x=1780698170; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=R6G4651Su27Kldq1bd8liUGC3loQ5pH7nVS2++qmd+w=; b=s6Azm6q6hxsxbWLYouE2YomLASFoK3XgkqAwVFCwAL1dkVakHVo0qwJV0jg14nlTey cYuWS1V9x3u5AOp/b7oi/GP8IGeagAfcRLKADbN+uuS03OAml7Rwvhlavbj/uica+gXD ZqM8QUVUF+x6y7YIJ/klzGOCPNDnNSBWVVvXz0CA/hpkDJb1jkzdJxrZxE7D9TRRYFO0 KWUmjHKP9E2Fk2a7xr6D8fot33sBavrHwbuFa19xPfb0fohR9Lz/RgiMqYwWrkTA9ZFF calPiLm5C4JcTvpkuGwv8ApPEinikigl+/hogcNR8ZOAYyZLrpjUhYiAgNFI6OIpWMdv 6tFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093370; x=1780698170; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=R6G4651Su27Kldq1bd8liUGC3loQ5pH7nVS2++qmd+w=; b=RlnfEp27qsi99QDZMK5JnNkPf3aLqASB5LhofUIAx2B5wS3ISD7CY8asXFcY4aGjlR w/ItGoOOBKLMyS5e5p29rcAL7zYdZOSDU2P+TWpevM2VFGh8+I1nA5GK+7/NpQ/3mBOd 9pgQYgL2aqmbAOXXUrogpHCiWc5mb/+nXOVd7AtdApvY0JzJzP+ceA8AZgdJM0ekcmsP rifV9eudoGY3WsRCzGfQHQT9oBTNBwNtm1WQF8o6nyZ3uZxJLgE/nfd08HPX0KVDCXiP Ym1dtQs4wsYc6U0t/7ifthK3KO/1ZsH1GINGRvJGYEB+tVMa3FqjFUA2C5XkBC8GISgA Obfg== X-Forwarded-Encrypted: i=1; AFNElJ+qi2SoiCaEhFMXxXpaIbdfQpsi2Hvd+6j67RfiCUAikQXr/teWI8sMJpcuRxk6SQag0Y2ot74yRwXB3Uk=@vger.kernel.org X-Gm-Message-State: AOJu0YwDJ6Il4MPvgkrT90P0Hc4GhGY6h8BUs/jRrdkCFhee93tCcJzD W6ORf34ywCjFI7oNkChJfMN3xlMSn/3ceMseGBoDQd92j7/CJZVXeI5QFKw9UOsZ5s/GZik4wf9 6JwSyTQ== X-Received: from pgjw1.prod.google.com ([2002:a63:fb41:0:b0:c79:788d:5b72]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:ad98:b0:3a2:e0d3:37d2 with SMTP id adf61e73a8af0-3b427f97d94mr1274269637.35.1780093370067; Fri, 29 May 2026 15:22:50 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:21:49 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-7-seanjc@google.com> Subject: [PATCH v3 06/40] KVM: x86: Rename kvm_cache_regs.h => regs.h From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Rename kvm_cache_regs.h to simply regs.h, as the "cache" nomenclature is already a lie (the file deals with state/registers that aren't cached per se), and so that more code/functionality can be landed in the header without making it a truly horrible misnomer. Deliberately drop the kvm_ prefix/namespace to align with other "local" headers, and to further differentiate regs.h from the public/global arch/x86/include/asm/kvm_vcpu_regs.h, which sadly needs to stay in asm/ so that the number of registers can be referenced by kvm_vcpu_arch. No functional change intended. Reviewed-by: Yosry Ahmed Reviewed-by: Binbin Wu Signed-off-by: Sean Christopherson --- arch/x86/kvm/emulate.c | 2 +- arch/x86/kvm/lapic.c | 2 +- arch/x86/kvm/mmu.h | 2 +- arch/x86/kvm/mmu/mmu.c | 2 +- arch/x86/kvm/{kvm_cache_regs.h =3D> regs.h} | 4 ++-- arch/x86/kvm/smm.c | 2 +- arch/x86/kvm/svm/svm.c | 2 +- arch/x86/kvm/svm/svm.h | 2 +- arch/x86/kvm/vmx/nested.h | 2 +- arch/x86/kvm/vmx/sgx.c | 2 +- arch/x86/kvm/vmx/vmx.c | 2 +- arch/x86/kvm/vmx/vmx.h | 2 +- arch/x86/kvm/x86.c | 2 +- arch/x86/kvm/x86.h | 2 +- 14 files changed, 15 insertions(+), 15 deletions(-) rename arch/x86/kvm/{kvm_cache_regs.h =3D> regs.h} (99%) diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c index 585a8ceab220..b566ab5c7515 100644 --- a/arch/x86/kvm/emulate.c +++ b/arch/x86/kvm/emulate.c @@ -20,7 +20,7 @@ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt =20 #include -#include "kvm_cache_regs.h" +#include "regs.h" #include "kvm_emulate.h" #include #include diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index 4e34f75e705d..15777869b292 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -37,7 +37,7 @@ #include #include #include -#include "kvm_cache_regs.h" +#include "regs.h" #include "irq.h" #include "ioapic.h" #include "trace.h" diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index ddf4e467c071..e1bb663ebbd5 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -3,7 +3,7 @@ #define __KVM_X86_MMU_H =20 #include -#include "kvm_cache_regs.h" +#include "regs.h" #include "x86.h" #include "cpuid.h" =20 diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index c87c26bf4149..b8f2edf2cfeb 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -22,7 +22,7 @@ #include "mmu_internal.h" #include "tdp_mmu.h" #include "x86.h" -#include "kvm_cache_regs.h" +#include "regs.h" #include "smm.h" #include "kvm_emulate.h" #include "page_track.h" diff --git a/arch/x86/kvm/kvm_cache_regs.h b/arch/x86/kvm/regs.h similarity index 99% rename from arch/x86/kvm/kvm_cache_regs.h rename to arch/x86/kvm/regs.h index 2ae492ad6412..4440f3992fce 100644 --- a/arch/x86/kvm/kvm_cache_regs.h +++ b/arch/x86/kvm/regs.h @@ -1,6 +1,6 @@ /* SPDX-License-Identifier: GPL-2.0 */ -#ifndef ASM_KVM_CACHE_REGS_H -#define ASM_KVM_CACHE_REGS_H +#ifndef ARCH_X86_KVM_REGS_H +#define ARCH_X86_KVM_REGS_H =20 #include =20 diff --git a/arch/x86/kvm/smm.c b/arch/x86/kvm/smm.c index f623c5986119..a446487bdd5c 100644 --- a/arch/x86/kvm/smm.c +++ b/arch/x86/kvm/smm.c @@ -3,7 +3,7 @@ =20 #include #include "x86.h" -#include "kvm_cache_regs.h" +#include "regs.h" #include "kvm_emulate.h" #include "smm.h" #include "cpuid.h" diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 7d8a433b5c5e..d4bfda10f1df 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -4,7 +4,7 @@ =20 #include "irq.h" #include "mmu.h" -#include "kvm_cache_regs.h" +#include "regs.h" #include "x86.h" #include "smm.h" #include "cpuid.h" diff --git a/arch/x86/kvm/svm/svm.h b/arch/x86/kvm/svm/svm.h index 87c6b105deef..cbc716885398 100644 --- a/arch/x86/kvm/svm/svm.h +++ b/arch/x86/kvm/svm/svm.h @@ -23,7 +23,7 @@ #include =20 #include "cpuid.h" -#include "kvm_cache_regs.h" +#include "regs.h" #include "x86.h" #include "pmu.h" =20 diff --git a/arch/x86/kvm/vmx/nested.h b/arch/x86/kvm/vmx/nested.h index 213a448104af..6d6cd5904ddf 100644 --- a/arch/x86/kvm/vmx/nested.h +++ b/arch/x86/kvm/vmx/nested.h @@ -2,7 +2,7 @@ #ifndef __KVM_X86_VMX_NESTED_H #define __KVM_X86_VMX_NESTED_H =20 -#include "kvm_cache_regs.h" +#include "regs.h" #include "hyperv.h" #include "vmcs12.h" #include "vmx.h" diff --git a/arch/x86/kvm/vmx/sgx.c b/arch/x86/kvm/vmx/sgx.c index 4c61fc33f764..66c315554b46 100644 --- a/arch/x86/kvm/vmx/sgx.c +++ b/arch/x86/kvm/vmx/sgx.c @@ -6,7 +6,7 @@ #include =20 #include "x86.h" -#include "kvm_cache_regs.h" +#include "regs.h" #include "nested.h" #include "sgx.h" #include "vmx.h" diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index cbc2034d7924..bb19f6df921b 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -59,7 +59,7 @@ #include "hyperv.h" #include "kvm_onhyperv.h" #include "irq.h" -#include "kvm_cache_regs.h" +#include "regs.h" #include "lapic.h" #include "mmu.h" #include "nested.h" diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index daedf663c0a9..de9de0d2016c 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -10,7 +10,7 @@ #include =20 #include "capabilities.h" -#include "../kvm_cache_regs.h" +#include "../regs.h" #include "pmu_intel.h" #include "vmcs.h" #include "vmx_ops.h" diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 634a6d0197f6..4fe2fb3ceba6 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -23,7 +23,7 @@ #include "mmu.h" #include "i8254.h" #include "tss.h" -#include "kvm_cache_regs.h" +#include "regs.h" #include "kvm_emulate.h" #include "mmu/page_track.h" #include "x86.h" diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index a49424f9c968..a6b2be462e6d 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -6,7 +6,7 @@ #include #include #include -#include "kvm_cache_regs.h" +#include "regs.h" #include "kvm_emulate.h" #include "cpuid.h" =20 --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 063E93C1977 for ; Fri, 29 May 2026 22:22:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093378; cv=none; b=ForaAEV7Z9Pfy2v1bkr6zldogG7LJSR+lQyq2oB4XXRvyFwQU73lKwSl/NsKxC45vZ6RPCpWzmB8ucEUqclU+WaPmAIo37HT9J2O2iSuBNENJUfKjJvFUL8qvIHt6wwqcfJswxuu6GXS+T+NBDww0rv3utWGRbFukqaMpsRtelk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093378; c=relaxed/simple; bh=I4JrKYYUi9Z/UJsEexbtKy+IMRYAO5db/1mDENYIFq0=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=XoReHJgjjIT2r4+haJeV/0ZOK87Un9x/cz/LjfPbaw6Pbi+J4lw0vl3NZIZcLN+pEdosaPbntrbD5e9i9G2F3O6nDSMphfgj9GFcHm4Dq/F9yyEOPMlHbc17E2aOXPGsDyvYSA9+pta+mtNc8TkPfjhTT760qcZsP6QqNIzHbnk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=WJULEdPA; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="WJULEdPA" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-c8581f7723aso274523a12.0 for ; Fri, 29 May 2026 15:22:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093371; x=1780698171; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=JiUeYThTwHb2vUrssGyx+Ryc7M1Q55eYse5ayLPWdBA=; b=WJULEdPA/rC83ClaQ5A17GqrFzMWe98NmkexurxeaAUzWgqUjMAdbEJ0hVRsZQV+Nv U1E79HoZdlm+xZ08OmSfCY3z00TyYb1CsJN4s+H5Ve8SjfmbaWzMpi19u/Kgej9AkFL2 ZpYCJPpPtfcX0mvdV6sz0klsCDWtvI1Vjs/ClRXDHbmy8HKdm4T+aFg9hETtGX19I5Ph me3V/K3n+HgPi1Aq8ys1GviWEpI0Y9Swc8xR0d1+MK1LCPqYq0TBiIwuCzQZQ0pJlDcU n+d1E5K1ZDK4AA2cGBFsrbEextFZrps9PZTcR50Lq9c+xzLokIRjx05Xx9FaUguzwjSj c7IQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093371; x=1780698171; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=JiUeYThTwHb2vUrssGyx+Ryc7M1Q55eYse5ayLPWdBA=; b=ocswSJmQb+3H18QsFuK7u88lmdlcIL3HwBe24CGLrmV2QU/3DtHSU4tVzNxHroy0Fk kgR0362OLtCs9A/49C7aAsyYDO4U3fR0t0qlQ66QaVCgsCmA0RqSHIQQxgvk96VdYNAN P2TFbf58mNnhic/YOmCm2F1BXVqSH4XyoG/jxo0jk4ewxRv+AXO29dN6CV4pJPkz8dIz uIu2FRnHuaquzcRDd0MgsUf0NhEaXei8RA6ZMA9/GMop2f/1PqGkpj+O2giqa5slNpqQ Qn2PPP6WIJq5j+pa7lvj56zesdImlAQJgX3SNeP9y7bgvZRJj/KOe3Hck/S0+fo0FC/V XhiQ== X-Forwarded-Encrypted: i=1; AFNElJ8stykuUYxQztXcBLWh1En2epXqWMiYRIeDWG8WuXJA9UKLHQZPG9tYZLvB/jY79+771XFPXe6JaGNsU/c=@vger.kernel.org X-Gm-Message-State: AOJu0Yx8Tt9MRaNgCJUgLdXj6gWhLEeAj15S1ftdXC3Dk0cRpoSwlWwu jQgThU+a2ukPuVZc4yz0RdiTk+QtQk10aM/2DFkKn1quaczxYVuElpyWvmNGSVC1fA+wlDebhdZ QKlQnHw== X-Received: from pgbcs5.prod.google.com ([2002:a05:6a02:4185:b0:c82:3948:7e01]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:ce4e:b0:3a2:d79c:416c with SMTP id adf61e73a8af0-3b427f97563mr1248571637.32.1780093371151; Fri, 29 May 2026 15:22:51 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:21:50 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-8-seanjc@google.com> Subject: [PATCH v3 07/40] KVM: x86: Move inlined GPR, CR, and DR helpers from x86.h to regs.h From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move inlined General Purpose Register, Control Register, and Debug Register helpers from x86.h to the aptly named regs.h, to help trim down x86.h (and x86.c in the future). Move *very* select EFER functionality as well, but leave behind the bulk of EFER handling and all other MSR handling. There is more than enough MSR code to carve out msrs.{c,h} in the future. Give is_long_bit_mode() special treatment as it's more along the lines of a CR4 bit check, but just happens to be accessed through an MSR interface. And more importantly, because giving regs.h access to is_long_bit_mode() greatly simplifies dependency chains. No functional change intended. Reviewed-by: Yosry Ahmed Reviewed-by: Binbin Wu Signed-off-by: Sean Christopherson --- arch/x86/kvm/regs.h | 123 ++++++++++++++++++++++++++++++++++++++++++-- arch/x86/kvm/x86.h | 117 ----------------------------------------- 2 files changed, 120 insertions(+), 120 deletions(-) diff --git a/arch/x86/kvm/regs.h b/arch/x86/kvm/regs.h index 4440f3992fce..62cc9deea226 100644 --- a/arch/x86/kvm/regs.h +++ b/arch/x86/kvm/regs.h @@ -16,6 +16,37 @@ =20 static_assert(!(KVM_POSSIBLE_CR0_GUEST_BITS & X86_CR0_PDPTR_BITS)); =20 +static inline bool is_long_mode(struct kvm_vcpu *vcpu) +{ +#ifdef CONFIG_X86_64 + return !!(vcpu->arch.efer & EFER_LMA); +#else + return false; +#endif +} + +static inline bool is_64_bit_mode(struct kvm_vcpu *vcpu) +{ + int cs_db, cs_l; + + WARN_ON_ONCE(vcpu->arch.guest_state_protected); + + if (!is_long_mode(vcpu)) + return false; + kvm_x86_call(get_cs_db_l_bits)(vcpu, &cs_db, &cs_l); + return cs_l; +} + +static inline bool is_64_bit_hypercall(struct kvm_vcpu *vcpu) +{ + /* + * If running with protected guest state, the CS register is not + * accessible. The hypercall register values will have had to been + * provided in 64-bit mode, so assume the guest is in 64-bit. + */ + return vcpu->arch.guest_state_protected || is_64_bit_mode(vcpu); +} + #define BUILD_KVM_GPR_ACCESSORS(lname, uname) \ static __always_inline unsigned long kvm_##lname##_read(struct kvm_vcpu *v= cpu)\ { \ @@ -143,6 +174,13 @@ static inline unsigned long kvm_register_read_raw(stru= ct kvm_vcpu *vcpu, int reg return vcpu->arch.regs[reg]; } =20 +static inline unsigned long kvm_register_read(struct kvm_vcpu *vcpu, int r= eg) +{ + unsigned long val =3D kvm_register_read_raw(vcpu, reg); + + return is_64_bit_mode(vcpu) ? val : (u32)val; +} + static inline void kvm_register_write_raw(struct kvm_vcpu *vcpu, int reg, unsigned long val) { @@ -153,6 +191,14 @@ static inline void kvm_register_write_raw(struct kvm_v= cpu *vcpu, int reg, kvm_register_mark_dirty(vcpu, reg); } =20 +static inline void kvm_register_write(struct kvm_vcpu *vcpu, + int reg, unsigned long val) +{ + if (!is_64_bit_mode(vcpu)) + val =3D (u32)val; + return kvm_register_write_raw(vcpu, reg, val); +} + static inline unsigned long kvm_rip_read(struct kvm_vcpu *vcpu) { if (!kvm_register_is_available(vcpu, VCPU_REG_RIP)) @@ -177,6 +223,12 @@ static inline void kvm_rsp_write(struct kvm_vcpu *vcpu= , unsigned long val) kvm_register_write_raw(vcpu, VCPU_REGS_RSP, val); } =20 +static inline u64 kvm_read_edx_eax(struct kvm_vcpu *vcpu) +{ + return (kvm_rax_read(vcpu) & -1u) + | ((u64)(kvm_rdx_read(vcpu) & -1u) << 32); +} + static inline u64 kvm_pdptr_read(struct kvm_vcpu *vcpu, int index) { might_sleep(); /* on svm */ @@ -243,10 +295,75 @@ static inline ulong kvm_read_cr4(struct kvm_vcpu *vcp= u) return kvm_read_cr4_bits(vcpu, ~0UL); } =20 -static inline u64 kvm_read_edx_eax(struct kvm_vcpu *vcpu) +static inline bool __kvm_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long= cr4) { - return (kvm_rax_read(vcpu) & -1u) - | ((u64)(kvm_rdx_read(vcpu) & -1u) << 32); + return !(cr4 & vcpu->arch.cr4_guest_rsvd_bits); +} + +#define __cr4_reserved_bits(__cpu_has, __c) \ +({ \ + u64 __reserved_bits =3D CR4_RESERVED_BITS; \ + \ + if (!__cpu_has(__c, X86_FEATURE_XSAVE)) \ + __reserved_bits |=3D X86_CR4_OSXSAVE; \ + if (!__cpu_has(__c, X86_FEATURE_SMEP)) \ + __reserved_bits |=3D X86_CR4_SMEP; \ + if (!__cpu_has(__c, X86_FEATURE_SMAP)) \ + __reserved_bits |=3D X86_CR4_SMAP; \ + if (!__cpu_has(__c, X86_FEATURE_FSGSBASE)) \ + __reserved_bits |=3D X86_CR4_FSGSBASE; \ + if (!__cpu_has(__c, X86_FEATURE_PKU)) \ + __reserved_bits |=3D X86_CR4_PKE; \ + if (!__cpu_has(__c, X86_FEATURE_LA57)) \ + __reserved_bits |=3D X86_CR4_LA57; \ + if (!__cpu_has(__c, X86_FEATURE_UMIP)) \ + __reserved_bits |=3D X86_CR4_UMIP; \ + if (!__cpu_has(__c, X86_FEATURE_VMX)) \ + __reserved_bits |=3D X86_CR4_VMXE; \ + if (!__cpu_has(__c, X86_FEATURE_PCID)) \ + __reserved_bits |=3D X86_CR4_PCIDE; \ + if (!__cpu_has(__c, X86_FEATURE_LAM)) \ + __reserved_bits |=3D X86_CR4_LAM_SUP; \ + if (!__cpu_has(__c, X86_FEATURE_SHSTK) && \ + !__cpu_has(__c, X86_FEATURE_IBT)) \ + __reserved_bits |=3D X86_CR4_CET; \ + __reserved_bits; \ +}) + +static inline bool is_protmode(struct kvm_vcpu *vcpu) +{ + return kvm_is_cr0_bit_set(vcpu, X86_CR0_PE); +} + +static inline bool is_pae(struct kvm_vcpu *vcpu) +{ + return kvm_is_cr4_bit_set(vcpu, X86_CR4_PAE); +} + +static inline bool is_pse(struct kvm_vcpu *vcpu) +{ + return kvm_is_cr4_bit_set(vcpu, X86_CR4_PSE); +} + +static inline bool is_paging(struct kvm_vcpu *vcpu) +{ + return likely(kvm_is_cr0_bit_set(vcpu, X86_CR0_PG)); +} + +static inline bool is_pae_paging(struct kvm_vcpu *vcpu) +{ + return !is_long_mode(vcpu) && is_pae(vcpu) && is_paging(vcpu); +} + +static inline bool kvm_dr7_valid(u64 data) +{ + /* Bits [63:32] are reserved */ + return !(data >> 32); +} +static inline bool kvm_dr6_valid(u64 data) +{ + /* Bits [63:32] are reserved */ + return !(data >> 32); } =20 static inline void enter_guest_mode(struct kvm_vcpu *vcpu) diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index a6b2be462e6d..3845b10020c9 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -243,42 +243,6 @@ static inline bool kvm_exception_is_soft(unsigned int = nr) return (nr =3D=3D BP_VECTOR) || (nr =3D=3D OF_VECTOR); } =20 -static inline bool is_protmode(struct kvm_vcpu *vcpu) -{ - return kvm_is_cr0_bit_set(vcpu, X86_CR0_PE); -} - -static inline bool is_long_mode(struct kvm_vcpu *vcpu) -{ -#ifdef CONFIG_X86_64 - return !!(vcpu->arch.efer & EFER_LMA); -#else - return false; -#endif -} - -static inline bool is_64_bit_mode(struct kvm_vcpu *vcpu) -{ - int cs_db, cs_l; - - WARN_ON_ONCE(vcpu->arch.guest_state_protected); - - if (!is_long_mode(vcpu)) - return false; - kvm_x86_call(get_cs_db_l_bits)(vcpu, &cs_db, &cs_l); - return cs_l; -} - -static inline bool is_64_bit_hypercall(struct kvm_vcpu *vcpu) -{ - /* - * If running with protected guest state, the CS register is not - * accessible. The hypercall register values will have had to been - * provided in 64-bit mode, so assume the guest is in 64-bit. - */ - return vcpu->arch.guest_state_protected || is_64_bit_mode(vcpu); -} - static inline bool x86_exception_has_error_code(unsigned int vector) { static u32 exception_has_error_code =3D BIT(DF_VECTOR) | BIT(TS_VECTOR) | @@ -293,26 +257,6 @@ static inline bool mmu_is_nested(struct kvm_vcpu *vcpu) return vcpu->arch.walk_mmu =3D=3D &vcpu->arch.nested_mmu; } =20 -static inline bool is_pae(struct kvm_vcpu *vcpu) -{ - return kvm_is_cr4_bit_set(vcpu, X86_CR4_PAE); -} - -static inline bool is_pse(struct kvm_vcpu *vcpu) -{ - return kvm_is_cr4_bit_set(vcpu, X86_CR4_PSE); -} - -static inline bool is_paging(struct kvm_vcpu *vcpu) -{ - return likely(kvm_is_cr0_bit_set(vcpu, X86_CR0_PG)); -} - -static inline bool is_pae_paging(struct kvm_vcpu *vcpu) -{ - return !is_long_mode(vcpu) && is_pae(vcpu) && is_paging(vcpu); -} - static inline u8 vcpu_virt_addr_bits(struct kvm_vcpu *vcpu) { return kvm_is_cr4_bit_set(vcpu, X86_CR4_LA57) ? 57 : 48; @@ -421,21 +365,6 @@ static inline bool vcpu_match_mmio_gpa(struct kvm_vcpu= *vcpu, gpa_t gpa) return false; } =20 -static inline unsigned long kvm_register_read(struct kvm_vcpu *vcpu, int r= eg) -{ - unsigned long val =3D kvm_register_read_raw(vcpu, reg); - - return is_64_bit_mode(vcpu) ? val : (u32)val; -} - -static inline void kvm_register_write(struct kvm_vcpu *vcpu, - int reg, unsigned long val) -{ - if (!is_64_bit_mode(vcpu)) - val =3D (u32)val; - return kvm_register_write_raw(vcpu, reg, val); -} - static inline bool kvm_check_has_quirk(struct kvm *kvm, u64 quirk) { return !(kvm->arch.disabled_quirks & quirk); @@ -627,17 +556,6 @@ static inline bool kvm_pat_valid(u64 data) return (data | ((data & 0x0202020202020202ull) << 1)) =3D=3D data; } =20 -static inline bool kvm_dr7_valid(u64 data) -{ - /* Bits [63:32] are reserved */ - return !(data >> 32); -} -static inline bool kvm_dr6_valid(u64 data) -{ - /* Bits [63:32] are reserved */ - return !(data >> 32); -} - /* * Trigger machine check on the host. We assume all the MSRs are already s= et up * by the CPU and that we still run on the same CPU as the MCE occurred on. @@ -684,41 +602,6 @@ enum kvm_msr_access { #define KVM_MSR_RET_UNSUPPORTED 2 #define KVM_MSR_RET_FILTERED 3 =20 -static inline bool __kvm_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long= cr4) -{ - return !(cr4 & vcpu->arch.cr4_guest_rsvd_bits); -} - -#define __cr4_reserved_bits(__cpu_has, __c) \ -({ \ - u64 __reserved_bits =3D CR4_RESERVED_BITS; \ - \ - if (!__cpu_has(__c, X86_FEATURE_XSAVE)) \ - __reserved_bits |=3D X86_CR4_OSXSAVE; \ - if (!__cpu_has(__c, X86_FEATURE_SMEP)) \ - __reserved_bits |=3D X86_CR4_SMEP; \ - if (!__cpu_has(__c, X86_FEATURE_SMAP)) \ - __reserved_bits |=3D X86_CR4_SMAP; \ - if (!__cpu_has(__c, X86_FEATURE_FSGSBASE)) \ - __reserved_bits |=3D X86_CR4_FSGSBASE; \ - if (!__cpu_has(__c, X86_FEATURE_PKU)) \ - __reserved_bits |=3D X86_CR4_PKE; \ - if (!__cpu_has(__c, X86_FEATURE_LA57)) \ - __reserved_bits |=3D X86_CR4_LA57; \ - if (!__cpu_has(__c, X86_FEATURE_UMIP)) \ - __reserved_bits |=3D X86_CR4_UMIP; \ - if (!__cpu_has(__c, X86_FEATURE_VMX)) \ - __reserved_bits |=3D X86_CR4_VMXE; \ - if (!__cpu_has(__c, X86_FEATURE_PCID)) \ - __reserved_bits |=3D X86_CR4_PCIDE; \ - if (!__cpu_has(__c, X86_FEATURE_LAM)) \ - __reserved_bits |=3D X86_CR4_LAM_SUP; \ - if (!__cpu_has(__c, X86_FEATURE_SHSTK) && \ - !__cpu_has(__c, X86_FEATURE_IBT)) \ - __reserved_bits |=3D X86_CR4_CET; \ - __reserved_bits; \ -}) - int kvm_sev_es_mmio(struct kvm_vcpu *vcpu, bool is_write, gpa_t gpa, unsigned int bytes, void *data); int kvm_sev_es_string_io(struct kvm_vcpu *vcpu, unsigned int size, --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 540CD3B5847 for ; Fri, 29 May 2026 22:22:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093380; cv=none; b=dvcfFmADCMD0eOsyVnnKp5w9dfy1irHbDjp0TOpRL5hicD5PFpaQCqqh5dGjSisAYt47a1KZdWFWiCsm/vqI9rAb2nok1MGSIzL4/HV7CnlX+ZampkMv49iscCHoAm4S36Bct8nQuItZQ5DoK7h4MBuJum9ljnXT7JkjPiT2EoA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093380; c=relaxed/simple; bh=xjAIQteLMqHfY21rM/s6bwjc4Au8udfDNArzvMlig2w=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=NwKO4LKJ65v2X9GqqUm5unBNtzmlnfUKkytaHj9ux7g0lF97WYAXxaXLHJvHiQPWEr9lNcbx+LDiBTjpvS99pDg2McU6kkG9bWA+HeFp3izOGMd4PqCOoufZMu/cAZw6CAdoRCQauO8VNP7I6pGRipU4PwC2J/f7rIcyvTdZeJY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Jt0uklbc; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Jt0uklbc" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2b99eb06178so72285635ad.2 for ; Fri, 29 May 2026 15:22:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093372; x=1780698172; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=zSnMZ07GxQ9ezxjo5/6a7rsfOhY3fI+SftZS6P79xyU=; b=Jt0uklbcEveatJ2XP5ibFIs9zucBdoXZniGSCqHC88sz/qC5u4MnkmvtyfibidTaeL AI1uy5AeK9nczEdA6J12WZZu8KR0NABceZkOB0Y7hzQPgRx2USb3xkMERkc3XU8mSsOa 2ti1qzHOCFVUmmVddTaIEQMhZ2SrXIShm1pPsDWXlV8SFGICUKiDS8KFSs5tfNIjJWv5 j94DK4YF2fBzMWeQHyudtAP2Lx98j1SxsA2Hwpoh66dGxHWczuFNaz8bbX5HEodHgPJI YcoIndmOB3udWthkdjD07d/c+3yFDiNi/402UFD/8/9H5MTCuy1rp1rmBTHckoJnRrW4 m37g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093372; x=1780698172; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=zSnMZ07GxQ9ezxjo5/6a7rsfOhY3fI+SftZS6P79xyU=; b=lzHnV0mr331YPKD+/XwvGVPQ9WXVWuq8YUT1LNEg2c3R98elA3W1mbmMYud1nAM+wf /Rqj+fTAIxlyT+OlbC/1HQWBu71r/iWtLj70nF7RrWleYcPJ0dbp/7DquoI8k/fSsmOv go0VRDrn8f5lhpE2uvAyE1s395zTj+PdC7FesZ2D7oKyx5dMwgPXmxlHGVzRrVNSrjq3 nSEpAJnGvHan2ny2Owik6HnAI97DusI6a7NqCFs8Kwf+mrtaZeaKChkbNsRXrpnMuLuY v0XxijiFfrgjyVMBUtnSYKBdleO23iCzh2SAuhY9Yo1bvuxyY0t5vXmVLT6wAMos81sX Sutw== X-Forwarded-Encrypted: i=1; AFNElJ9IhXvfCjXtOYsVAw39yJUeCDJ9EQGoVPe46WQpiPdYItwNRhXP0gCp1NzgVa0BkreXZ/mAuzICOCcypTk=@vger.kernel.org X-Gm-Message-State: AOJu0Yw+Uy/6JdVxFnrxJkDV9HoD6pBfH5+DEZY5Nc3HLkLZiM01jE9z SA8bCsIzbt6iHWnJQVh2CqsNMnicuzc8tbG3/9KNhKtkr0xeqCLK2NLlEbwlwZSF+Z6x1LH+9yK nPciucg== X-Received: from pldy13.prod.google.com ([2002:a17:902:cacd:b0:2b2:a70b:98ea]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:ebcd:b0:2be:1b3e:3a1f with SMTP id d9443c01a7336-2bf36779d38mr21033785ad.2.1780093372306; Fri, 29 May 2026 15:22:52 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:21:51 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-9-seanjc@google.com> Subject: [PATCH v3 08/40] KVM: x86: Add mode-aware versions of kvm__{read,write}() helpers From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Make kvm__{read,write}() mode-aware (where the value is truncated to 32 bits if the vCPU isn't in 64-bit mode), and convert all the intentional "raw" accesses to kvm__{read,write}_raw() versions. To avoid confusion and bikeshedding over whether or not explicit 32-bit accesses should use the "raw" or mode-aware variants, add and use "e" versions, e.g. for things like RDMSR, WRMSR, and CPUID, where the instruction uses only bits 31:0, regardless of mode. No functional change intended (all use of "e" versions is for cases where the value is already truncated due to bouncing through a u32). Cc: Binbin Wu Cc: Kai Huang Reviewed-by: Binbin Wu Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang --- arch/x86/kvm/cpuid.c | 12 ++-- arch/x86/kvm/hyperv.c | 21 +++---- arch/x86/kvm/hyperv.h | 4 +- arch/x86/kvm/regs.h | 88 +++++++++++++++++---------- arch/x86/kvm/svm/nested.c | 6 +- arch/x86/kvm/svm/svm.c | 13 ++-- arch/x86/kvm/vmx/nested.c | 8 +-- arch/x86/kvm/vmx/sgx.c | 4 +- arch/x86/kvm/vmx/tdx.c | 18 +++--- arch/x86/kvm/x86.c | 121 +++++++++++++++++++------------------- arch/x86/kvm/xen.c | 32 +++++----- 11 files changed, 173 insertions(+), 154 deletions(-) diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index 8e5340dd2621..fd3b02575cd0 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -2166,13 +2166,13 @@ int kvm_emulate_cpuid(struct kvm_vcpu *vcpu) return 1; } =20 - eax =3D kvm_rax_read(vcpu); - ecx =3D kvm_rcx_read(vcpu); + eax =3D kvm_eax_read(vcpu); + ecx =3D kvm_ecx_read(vcpu); kvm_cpuid(vcpu, &eax, &ebx, &ecx, &edx, false); - kvm_rax_write(vcpu, eax); - kvm_rbx_write(vcpu, ebx); - kvm_rcx_write(vcpu, ecx); - kvm_rdx_write(vcpu, edx); + kvm_eax_write(vcpu, eax); + kvm_ebx_write(vcpu, ebx); + kvm_ecx_write(vcpu, ecx); + kvm_edx_write(vcpu, edx); return kvm_skip_emulated_instruction(vcpu); } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_cpuid); diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c index 015c6947b462..3551af9a9453 100644 --- a/arch/x86/kvm/hyperv.c +++ b/arch/x86/kvm/hyperv.c @@ -2377,10 +2377,10 @@ static void kvm_hv_hypercall_set_result(struct kvm_= vcpu *vcpu, u64 result) =20 longmode =3D is_64_bit_hypercall(vcpu); if (longmode) - kvm_rax_write(vcpu, result); + kvm_rax_write_raw(vcpu, result); else { - kvm_rdx_write(vcpu, result >> 32); - kvm_rax_write(vcpu, result & 0xffffffff); + kvm_edx_write(vcpu, result >> 32); + kvm_eax_write(vcpu, result); } } =20 @@ -2544,18 +2544,15 @@ int kvm_hv_hypercall(struct kvm_vcpu *vcpu) =20 #ifdef CONFIG_X86_64 if (is_64_bit_hypercall(vcpu)) { - hc.param =3D kvm_rcx_read(vcpu); - hc.ingpa =3D kvm_rdx_read(vcpu); - hc.outgpa =3D kvm_r8_read(vcpu); + hc.param =3D kvm_rcx_read_raw(vcpu); + hc.ingpa =3D kvm_rdx_read_raw(vcpu); + hc.outgpa =3D kvm_r8_read_raw(vcpu); } else #endif { - hc.param =3D ((u64)kvm_rdx_read(vcpu) << 32) | - (kvm_rax_read(vcpu) & 0xffffffff); - hc.ingpa =3D ((u64)kvm_rbx_read(vcpu) << 32) | - (kvm_rcx_read(vcpu) & 0xffffffff); - hc.outgpa =3D ((u64)kvm_rdi_read(vcpu) << 32) | - (kvm_rsi_read(vcpu) & 0xffffffff); + hc.param =3D ((u64)kvm_edx_read(vcpu) << 32) | kvm_eax_read(vcpu); + hc.ingpa =3D ((u64)kvm_ebx_read(vcpu) << 32) | kvm_ecx_read(vcpu); + hc.outgpa =3D ((u64)kvm_edi_read(vcpu) << 32) | kvm_esi_read(vcpu); } =20 hc.code =3D hc.param & 0xffff; diff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h index 6301f79fcbae..65e89ed65349 100644 --- a/arch/x86/kvm/hyperv.h +++ b/arch/x86/kvm/hyperv.h @@ -232,8 +232,8 @@ static inline bool kvm_hv_is_tlb_flush_hcall(struct kvm= _vcpu *vcpu) if (!hv_vcpu) return false; =20 - code =3D is_64_bit_hypercall(vcpu) ? kvm_rcx_read(vcpu) : - kvm_rax_read(vcpu); + code =3D is_64_bit_hypercall(vcpu) ? kvm_rcx_read_raw(vcpu) : + kvm_eax_read(vcpu); =20 return (code =3D=3D HVCALL_FLUSH_VIRTUAL_ADDRESS_SPACE || code =3D=3D HVCALL_FLUSH_VIRTUAL_ADDRESS_LIST || diff --git a/arch/x86/kvm/regs.h b/arch/x86/kvm/regs.h index 62cc9deea226..12db5039aace 100644 --- a/arch/x86/kvm/regs.h +++ b/arch/x86/kvm/regs.h @@ -47,32 +47,61 @@ static inline bool is_64_bit_hypercall(struct kvm_vcpu = *vcpu) return vcpu->arch.guest_state_protected || is_64_bit_mode(vcpu); } =20 -#define BUILD_KVM_GPR_ACCESSORS(lname, uname) \ -static __always_inline unsigned long kvm_##lname##_read(struct kvm_vcpu *v= cpu)\ -{ \ - return vcpu->arch.regs[VCPU_REGS_##uname]; \ -} \ -static __always_inline void kvm_##lname##_write(struct kvm_vcpu *vcpu, = \ - unsigned long val) \ -{ \ - vcpu->arch.regs[VCPU_REGS_##uname] =3D val; \ +static __always_inline unsigned long kvm_reg_mode_mask(struct kvm_vcpu *vc= pu) +{ +#ifdef CONFIG_X86_64 + return is_64_bit_mode(vcpu) ? GENMASK(63, 0) : GENMASK(31, 0); +#else + return GENMASK(31, 0); +#endif +} + +#define __BUILD_KVM_GPR_ACCESSORS(lname, uname) \ +static __always_inline unsigned long kvm_##lname##_read(struct kvm_vcpu *v= cpu) \ +{ \ + return vcpu->arch.regs[VCPU_REGS_##uname] & kvm_reg_mode_mask(vcpu); \ +} \ +static __always_inline void kvm_##lname##_write(struct kvm_vcpu *vcpu, \ + unsigned long val) \ +{ \ + vcpu->arch.regs[VCPU_REGS_##uname] =3D val & kvm_reg_mode_mask(vcpu); \ +} \ +static __always_inline unsigned long kvm_##lname##_read_raw(struct kvm_vcp= u *vcpu) \ +{ \ + return vcpu->arch.regs[VCPU_REGS_##uname]; \ +} \ +static __always_inline void kvm_##lname##_write_raw(struct kvm_vcpu *vcpu,= \ + unsigned long val) \ +{ \ + vcpu->arch.regs[VCPU_REGS_##uname] =3D val; \ } -BUILD_KVM_GPR_ACCESSORS(rax, RAX) -BUILD_KVM_GPR_ACCESSORS(rbx, RBX) -BUILD_KVM_GPR_ACCESSORS(rcx, RCX) -BUILD_KVM_GPR_ACCESSORS(rdx, RDX) -BUILD_KVM_GPR_ACCESSORS(rbp, RBP) -BUILD_KVM_GPR_ACCESSORS(rsi, RSI) -BUILD_KVM_GPR_ACCESSORS(rdi, RDI) +#define BUILD_KVM_GPR_ACCESSORS(lname, uname) \ +static __always_inline u32 kvm_e##lname##_read(struct kvm_vcpu *vcpu) \ +{ \ + return vcpu->arch.regs[VCPU_REGS_##uname]; \ +} \ +static __always_inline void kvm_e##lname##_write(struct kvm_vcpu *vcpu, u3= 2 val) \ +{ \ + vcpu->arch.regs[VCPU_REGS_##uname] =3D val; \ +} \ +__BUILD_KVM_GPR_ACCESSORS(r##lname, uname) + +BUILD_KVM_GPR_ACCESSORS(ax, RAX) +BUILD_KVM_GPR_ACCESSORS(bx, RBX) +BUILD_KVM_GPR_ACCESSORS(cx, RCX) +BUILD_KVM_GPR_ACCESSORS(dx, RDX) +BUILD_KVM_GPR_ACCESSORS(bp, RBP) +BUILD_KVM_GPR_ACCESSORS(si, RSI) +BUILD_KVM_GPR_ACCESSORS(di, RDI) #ifdef CONFIG_X86_64 -BUILD_KVM_GPR_ACCESSORS(r8, R8) -BUILD_KVM_GPR_ACCESSORS(r9, R9) -BUILD_KVM_GPR_ACCESSORS(r10, R10) -BUILD_KVM_GPR_ACCESSORS(r11, R11) -BUILD_KVM_GPR_ACCESSORS(r12, R12) -BUILD_KVM_GPR_ACCESSORS(r13, R13) -BUILD_KVM_GPR_ACCESSORS(r14, R14) -BUILD_KVM_GPR_ACCESSORS(r15, R15) +__BUILD_KVM_GPR_ACCESSORS(r8, R8) +__BUILD_KVM_GPR_ACCESSORS(r9, R9) +__BUILD_KVM_GPR_ACCESSORS(r10, R10) +__BUILD_KVM_GPR_ACCESSORS(r11, R11) +__BUILD_KVM_GPR_ACCESSORS(r12, R12) +__BUILD_KVM_GPR_ACCESSORS(r13, R13) +__BUILD_KVM_GPR_ACCESSORS(r14, R14) +__BUILD_KVM_GPR_ACCESSORS(r15, R15) #endif =20 /* @@ -176,9 +205,7 @@ static inline unsigned long kvm_register_read_raw(struc= t kvm_vcpu *vcpu, int reg =20 static inline unsigned long kvm_register_read(struct kvm_vcpu *vcpu, int r= eg) { - unsigned long val =3D kvm_register_read_raw(vcpu, reg); - - return is_64_bit_mode(vcpu) ? val : (u32)val; + return kvm_register_read_raw(vcpu, reg) & kvm_reg_mode_mask(vcpu); } =20 static inline void kvm_register_write_raw(struct kvm_vcpu *vcpu, int reg, @@ -194,9 +221,7 @@ static inline void kvm_register_write_raw(struct kvm_vc= pu *vcpu, int reg, static inline void kvm_register_write(struct kvm_vcpu *vcpu, int reg, unsigned long val) { - if (!is_64_bit_mode(vcpu)) - val =3D (u32)val; - return kvm_register_write_raw(vcpu, reg, val); + return kvm_register_write_raw(vcpu, reg, val & kvm_reg_mode_mask(vcpu)); } =20 static inline unsigned long kvm_rip_read(struct kvm_vcpu *vcpu) @@ -225,8 +250,7 @@ static inline void kvm_rsp_write(struct kvm_vcpu *vcpu,= unsigned long val) =20 static inline u64 kvm_read_edx_eax(struct kvm_vcpu *vcpu) { - return (kvm_rax_read(vcpu) & -1u) - | ((u64)(kvm_rdx_read(vcpu) & -1u) << 32); + return kvm_eax_read(vcpu) | (u64)(kvm_edx_read(vcpu)) << 32; } =20 static inline u64 kvm_pdptr_read(struct kvm_vcpu *vcpu, int index) diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 7ad4b4fb7a1c..d817dbb350d6 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -793,7 +793,7 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm = *svm) =20 svm->vcpu.arch.cr2 =3D save->cr2; =20 - kvm_rax_write(vcpu, save->rax); + kvm_rax_write_raw(vcpu, save->rax); kvm_rsp_write(vcpu, save->rsp); kvm_rip_write(vcpu, save->rip); =20 @@ -1272,7 +1272,7 @@ static int nested_svm_vmexit_update_vmcb12(struct kvm= _vcpu *vcpu) vmcb12->save.rflags =3D kvm_get_rflags(vcpu); vmcb12->save.rip =3D kvm_rip_read(vcpu); vmcb12->save.rsp =3D kvm_rsp_read(vcpu); - vmcb12->save.rax =3D kvm_rax_read(vcpu); + vmcb12->save.rax =3D kvm_rax_read_raw(vcpu); vmcb12->save.dr7 =3D vmcb02->save.dr7; vmcb12->save.dr6 =3D svm->vcpu.arch.dr6; vmcb12->save.cpl =3D vmcb02->save.cpl; @@ -1424,7 +1424,7 @@ void nested_svm_vmexit(struct vcpu_svm *svm) svm_set_efer(vcpu, vmcb01->save.efer); svm_set_cr0(vcpu, vmcb01->save.cr0 | X86_CR0_PE); svm_set_cr4(vcpu, vmcb01->save.cr4); - kvm_rax_write(vcpu, vmcb01->save.rax); + kvm_rax_write_raw(vcpu, vmcb01->save.rax); kvm_rsp_write(vcpu, vmcb01->save.rsp); kvm_rip_write(vcpu, vmcb01->save.rip); =20 diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index d4bfda10f1df..8402e94ac094 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -2409,15 +2409,12 @@ static int clgi_interception(struct kvm_vcpu *vcpu) =20 static int invlpga_interception(struct kvm_vcpu *vcpu) { - gva_t gva =3D kvm_rax_read(vcpu); - u32 asid =3D kvm_rcx_read(vcpu); - - if (nested_svm_check_permissions(vcpu)) - return 1; - /* FIXME: Handle an address size prefix. */ - if (!is_64_bit_mode(vcpu)) - gva =3D (u32)gva; + gva_t gva =3D kvm_rax_read(vcpu); + u32 asid =3D kvm_ecx_read(vcpu); + + if (nested_svm_check_permissions(vcpu)) + return 1; =20 trace_kvm_invlpga(to_svm(vcpu)->vmcb->save.rip, asid, gva); =20 diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 30dcabc899a2..b2c851cc7d5c 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -6165,7 +6165,7 @@ static int handle_invvpid(struct kvm_vcpu *vcpu) static int nested_vmx_eptp_switching(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12) { - u32 index =3D kvm_rcx_read(vcpu); + u32 index =3D kvm_ecx_read(vcpu); u64 new_eptp; =20 if (WARN_ON_ONCE(!nested_cpu_has_ept(vmcs12))) @@ -6199,7 +6199,7 @@ static int handle_vmfunc(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); struct vmcs12 *vmcs12; - u32 function =3D kvm_rax_read(vcpu); + u32 function =3D kvm_eax_read(vcpu); =20 /* * VMFUNC should never execute cleanly while L1 is active; KVM supports @@ -6321,7 +6321,7 @@ static bool nested_vmx_exit_handled_msr(struct kvm_vc= pu *vcpu, exit_reason.basic =3D=3D EXIT_REASON_MSR_WRITE_IMM) msr_index =3D vmx_get_exit_qual(vcpu); else - msr_index =3D kvm_rcx_read(vcpu); + msr_index =3D kvm_ecx_read(vcpu); =20 /* * The MSR_BITMAP page is divided into four 1024-byte bitmaps, @@ -6431,7 +6431,7 @@ static bool nested_vmx_exit_handled_encls(struct kvm_= vcpu *vcpu, !nested_cpu_has2(vmcs12, SECONDARY_EXEC_ENCLS_EXITING)) return false; =20 - encls_leaf =3D kvm_rax_read(vcpu); + encls_leaf =3D kvm_eax_read(vcpu); if (encls_leaf > 62) encls_leaf =3D 63; return vmcs12->encls_exiting_bitmap & BIT_ULL(encls_leaf); diff --git a/arch/x86/kvm/vmx/sgx.c b/arch/x86/kvm/vmx/sgx.c index 66c315554b46..2f5a1c58f3c5 100644 --- a/arch/x86/kvm/vmx/sgx.c +++ b/arch/x86/kvm/vmx/sgx.c @@ -352,7 +352,7 @@ static int handle_encls_einit(struct kvm_vcpu *vcpu) rflags &=3D ~X86_EFLAGS_ZF; vmx_set_rflags(vcpu, rflags); =20 - kvm_rax_write(vcpu, ret); + kvm_eax_write(vcpu, ret); return kvm_skip_emulated_instruction(vcpu); } =20 @@ -380,7 +380,7 @@ static inline bool sgx_enabled_in_guest_bios(struct kvm= _vcpu *vcpu) =20 int handle_encls(struct kvm_vcpu *vcpu) { - u32 leaf =3D (u32)kvm_rax_read(vcpu); + u32 leaf =3D kvm_eax_read(vcpu); =20 if (!enable_sgx || !guest_cpu_cap_has(vcpu, X86_FEATURE_SGX) || !guest_cpu_cap_has(vcpu, X86_FEATURE_SGX1)) { diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 77e3d1bb24cb..ffe9d0db58c5 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -1169,11 +1169,11 @@ static int complete_hypercall_exit(struct kvm_vcpu = *vcpu) =20 static int tdx_emulate_vmcall(struct kvm_vcpu *vcpu) { - kvm_rax_write(vcpu, to_tdx(vcpu)->vp_enter_args.r10); - kvm_rbx_write(vcpu, to_tdx(vcpu)->vp_enter_args.r11); - kvm_rcx_write(vcpu, to_tdx(vcpu)->vp_enter_args.r12); - kvm_rdx_write(vcpu, to_tdx(vcpu)->vp_enter_args.r13); - kvm_rsi_write(vcpu, to_tdx(vcpu)->vp_enter_args.r14); + kvm_rax_write_raw(vcpu, to_tdx(vcpu)->vp_enter_args.r10); + kvm_rbx_write_raw(vcpu, to_tdx(vcpu)->vp_enter_args.r11); + kvm_rcx_write_raw(vcpu, to_tdx(vcpu)->vp_enter_args.r12); + kvm_rdx_write_raw(vcpu, to_tdx(vcpu)->vp_enter_args.r13); + kvm_rsi_write_raw(vcpu, to_tdx(vcpu)->vp_enter_args.r14); =20 return __kvm_emulate_hypercall(vcpu, 0, complete_hypercall_exit); } @@ -2107,12 +2107,12 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath= _t fastpath) case EXIT_REASON_IO_INSTRUCTION: return tdx_emulate_io(vcpu); case EXIT_REASON_MSR_READ: - kvm_rcx_write(vcpu, tdx->vp_enter_args.r12); + kvm_ecx_write(vcpu, tdx->vp_enter_args.r12); return kvm_emulate_rdmsr(vcpu); case EXIT_REASON_MSR_WRITE: - kvm_rcx_write(vcpu, tdx->vp_enter_args.r12); - kvm_rax_write(vcpu, tdx->vp_enter_args.r13 & -1u); - kvm_rdx_write(vcpu, tdx->vp_enter_args.r13 >> 32); + kvm_ecx_write(vcpu, tdx->vp_enter_args.r12); + kvm_eax_write(vcpu, tdx->vp_enter_args.r13); + kvm_edx_write(vcpu, tdx->vp_enter_args.r13 >> 32); return kvm_emulate_wrmsr(vcpu); case EXIT_REASON_EPT_MISCONFIG: return tdx_emulate_mmio(vcpu); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 4fe2fb3ceba6..a102269a7af9 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -1300,7 +1300,7 @@ int kvm_emulate_xsetbv(struct kvm_vcpu *vcpu) { /* Note, #UD due to CR4.OSXSAVE=3D0 has priority over the intercept. */ if (kvm_x86_call(get_cpl)(vcpu) !=3D 0 || - __kvm_set_xcr(vcpu, kvm_rcx_read(vcpu), kvm_read_edx_eax(vcpu))) { + __kvm_set_xcr(vcpu, kvm_ecx_read(vcpu), kvm_read_edx_eax(vcpu))) { kvm_inject_gp(vcpu, 0); return 1; } @@ -1597,7 +1597,7 @@ static unsigned long kvm_get_effective_dr7(struct kvm= _vcpu *vcpu) =20 int kvm_emulate_rdpmc(struct kvm_vcpu *vcpu) { - u32 pmc =3D kvm_rcx_read(vcpu); + u32 pmc =3D kvm_ecx_read(vcpu); u64 data; =20 if (kvm_pmu_rdpmc(vcpu, pmc, &data)) { @@ -1605,8 +1605,8 @@ int kvm_emulate_rdpmc(struct kvm_vcpu *vcpu) return 1; } =20 - kvm_rax_write(vcpu, (u32)data); - kvm_rdx_write(vcpu, data >> 32); + kvm_eax_write(vcpu, data); + kvm_edx_write(vcpu, data >> 32); return kvm_skip_emulated_instruction(vcpu); } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_rdpmc); @@ -2053,8 +2053,8 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_msr_write); static void complete_userspace_rdmsr(struct kvm_vcpu *vcpu) { if (!vcpu->run->msr.error) { - kvm_rax_write(vcpu, (u32)vcpu->run->msr.data); - kvm_rdx_write(vcpu, vcpu->run->msr.data >> 32); + kvm_eax_write(vcpu, vcpu->run->msr.data); + kvm_edx_write(vcpu, vcpu->run->msr.data >> 32); } } =20 @@ -2135,8 +2135,8 @@ static int __kvm_emulate_rdmsr(struct kvm_vcpu *vcpu,= u32 msr, int reg, trace_kvm_msr_read(msr, data); =20 if (reg < 0) { - kvm_rax_write(vcpu, data & -1u); - kvm_rdx_write(vcpu, (data >> 32) & -1u); + kvm_eax_write(vcpu, data); + kvm_edx_write(vcpu, data >> 32); } else { kvm_register_write(vcpu, reg, data); } @@ -2153,7 +2153,7 @@ static int __kvm_emulate_rdmsr(struct kvm_vcpu *vcpu,= u32 msr, int reg, =20 int kvm_emulate_rdmsr(struct kvm_vcpu *vcpu) { - return __kvm_emulate_rdmsr(vcpu, kvm_rcx_read(vcpu), -1, + return __kvm_emulate_rdmsr(vcpu, kvm_ecx_read(vcpu), -1, complete_fast_rdmsr); } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_rdmsr); @@ -2189,7 +2189,7 @@ static int __kvm_emulate_wrmsr(struct kvm_vcpu *vcpu,= u32 msr, u64 data) =20 int kvm_emulate_wrmsr(struct kvm_vcpu *vcpu) { - return __kvm_emulate_wrmsr(vcpu, kvm_rcx_read(vcpu), + return __kvm_emulate_wrmsr(vcpu, kvm_ecx_read(vcpu), kvm_read_edx_eax(vcpu)); } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_wrmsr); @@ -2299,7 +2299,7 @@ static fastpath_t __handle_fastpath_wrmsr(struct kvm_= vcpu *vcpu, u32 msr, u64 da =20 fastpath_t handle_fastpath_wrmsr(struct kvm_vcpu *vcpu) { - return __handle_fastpath_wrmsr(vcpu, kvm_rcx_read(vcpu), + return __handle_fastpath_wrmsr(vcpu, kvm_ecx_read(vcpu), kvm_read_edx_eax(vcpu)); } EXPORT_SYMBOL_FOR_KVM_INTERNAL(handle_fastpath_wrmsr); @@ -9690,7 +9690,7 @@ static int complete_fast_pio_out(struct kvm_vcpu *vcp= u) static int kvm_fast_pio_out(struct kvm_vcpu *vcpu, int size, unsigned short port) { - unsigned long val =3D kvm_rax_read(vcpu); + unsigned long val =3D kvm_rax_read_raw(vcpu); int ret =3D emulator_pio_out(vcpu, size, port, &val, 1); =20 if (ret) @@ -9726,10 +9726,10 @@ static int complete_fast_pio_in(struct kvm_vcpu *vc= pu) } =20 /* For size less than 4 we merge, else we zero extend */ - val =3D (vcpu->arch.pio.size < 4) ? kvm_rax_read(vcpu) : 0; + val =3D (vcpu->arch.pio.size < 4) ? kvm_rax_read_raw(vcpu) : 0; =20 complete_emulator_pio_in(vcpu, &val); - kvm_rax_write(vcpu, val); + kvm_rax_write_raw(vcpu, val); =20 return kvm_skip_emulated_instruction(vcpu); } @@ -9741,11 +9741,11 @@ static int kvm_fast_pio_in(struct kvm_vcpu *vcpu, i= nt size, int ret; =20 /* For size less than 4 we merge, else we zero extend */ - val =3D (size < 4) ? kvm_rax_read(vcpu) : 0; + val =3D (size < 4) ? kvm_rax_read_raw(vcpu) : 0; =20 ret =3D emulator_pio_in(vcpu, size, port, &val, 1); if (ret) { - kvm_rax_write(vcpu, val); + kvm_rax_write_raw(vcpu, val); return ret; } =20 @@ -10412,29 +10412,30 @@ static int complete_hypercall_exit(struct kvm_vcp= u *vcpu) =20 if (!is_64_bit_hypercall(vcpu)) ret =3D (u32)ret; - kvm_rax_write(vcpu, ret); + kvm_rax_write_raw(vcpu, ret); return kvm_skip_emulated_instruction(vcpu); } =20 int ____kvm_emulate_hypercall(struct kvm_vcpu *vcpu, int cpl, int (*complete_hypercall)(struct kvm_vcpu *)) { - unsigned long ret; - unsigned long nr =3D kvm_rax_read(vcpu); - unsigned long a0 =3D kvm_rbx_read(vcpu); - unsigned long a1 =3D kvm_rcx_read(vcpu); - unsigned long a2 =3D kvm_rdx_read(vcpu); - unsigned long a3 =3D kvm_rsi_read(vcpu); int op_64_bit =3D is_64_bit_hypercall(vcpu); + unsigned long ret, nr, a0, a1, a2, a3; =20 ++vcpu->stat.hypercalls; =20 - if (!op_64_bit) { - nr &=3D 0xFFFFFFFF; - a0 &=3D 0xFFFFFFFF; - a1 &=3D 0xFFFFFFFF; - a2 &=3D 0xFFFFFFFF; - a3 &=3D 0xFFFFFFFF; + if (op_64_bit) { + nr =3D kvm_rax_read_raw(vcpu); + a0 =3D kvm_rbx_read_raw(vcpu); + a1 =3D kvm_rcx_read_raw(vcpu); + a2 =3D kvm_rdx_read_raw(vcpu); + a3 =3D kvm_rsi_read_raw(vcpu); + } else { + nr =3D kvm_eax_read(vcpu); + a0 =3D kvm_ebx_read(vcpu); + a1 =3D kvm_ecx_read(vcpu); + a2 =3D kvm_edx_read(vcpu); + a3 =3D kvm_esi_read(vcpu); } =20 trace_kvm_hypercall(nr, a0, a1, a2, a3); @@ -12132,23 +12133,23 @@ static void __get_regs(struct kvm_vcpu *vcpu, str= uct kvm_regs *regs) emulator_writeback_register_cache(vcpu->arch.emulate_ctxt); vcpu->arch.emulate_regs_need_sync_to_vcpu =3D false; } - regs->rax =3D kvm_rax_read(vcpu); - regs->rbx =3D kvm_rbx_read(vcpu); - regs->rcx =3D kvm_rcx_read(vcpu); - regs->rdx =3D kvm_rdx_read(vcpu); - regs->rsi =3D kvm_rsi_read(vcpu); - regs->rdi =3D kvm_rdi_read(vcpu); + regs->rax =3D kvm_rax_read_raw(vcpu); + regs->rbx =3D kvm_rbx_read_raw(vcpu); + regs->rcx =3D kvm_rcx_read_raw(vcpu); + regs->rdx =3D kvm_rdx_read_raw(vcpu); + regs->rsi =3D kvm_rsi_read_raw(vcpu); + regs->rdi =3D kvm_rdi_read_raw(vcpu); regs->rsp =3D kvm_rsp_read(vcpu); - regs->rbp =3D kvm_rbp_read(vcpu); + regs->rbp =3D kvm_rbp_read_raw(vcpu); #ifdef CONFIG_X86_64 - regs->r8 =3D kvm_r8_read(vcpu); - regs->r9 =3D kvm_r9_read(vcpu); - regs->r10 =3D kvm_r10_read(vcpu); - regs->r11 =3D kvm_r11_read(vcpu); - regs->r12 =3D kvm_r12_read(vcpu); - regs->r13 =3D kvm_r13_read(vcpu); - regs->r14 =3D kvm_r14_read(vcpu); - regs->r15 =3D kvm_r15_read(vcpu); + regs->r8 =3D kvm_r8_read_raw(vcpu); + regs->r9 =3D kvm_r9_read_raw(vcpu); + regs->r10 =3D kvm_r10_read_raw(vcpu); + regs->r11 =3D kvm_r11_read_raw(vcpu); + regs->r12 =3D kvm_r12_read_raw(vcpu); + regs->r13 =3D kvm_r13_read_raw(vcpu); + regs->r14 =3D kvm_r14_read_raw(vcpu); + regs->r15 =3D kvm_r15_read_raw(vcpu); #endif =20 regs->rip =3D kvm_rip_read(vcpu); @@ -12172,23 +12173,23 @@ static void __set_regs(struct kvm_vcpu *vcpu, str= uct kvm_regs *regs) vcpu->arch.emulate_regs_need_sync_from_vcpu =3D true; vcpu->arch.emulate_regs_need_sync_to_vcpu =3D false; =20 - kvm_rax_write(vcpu, regs->rax); - kvm_rbx_write(vcpu, regs->rbx); - kvm_rcx_write(vcpu, regs->rcx); - kvm_rdx_write(vcpu, regs->rdx); - kvm_rsi_write(vcpu, regs->rsi); - kvm_rdi_write(vcpu, regs->rdi); + kvm_rax_write_raw(vcpu, regs->rax); + kvm_rbx_write_raw(vcpu, regs->rbx); + kvm_rcx_write_raw(vcpu, regs->rcx); + kvm_rdx_write_raw(vcpu, regs->rdx); + kvm_rsi_write_raw(vcpu, regs->rsi); + kvm_rdi_write_raw(vcpu, regs->rdi); kvm_rsp_write(vcpu, regs->rsp); - kvm_rbp_write(vcpu, regs->rbp); + kvm_rbp_write_raw(vcpu, regs->rbp); #ifdef CONFIG_X86_64 - kvm_r8_write(vcpu, regs->r8); - kvm_r9_write(vcpu, regs->r9); - kvm_r10_write(vcpu, regs->r10); - kvm_r11_write(vcpu, regs->r11); - kvm_r12_write(vcpu, regs->r12); - kvm_r13_write(vcpu, regs->r13); - kvm_r14_write(vcpu, regs->r14); - kvm_r15_write(vcpu, regs->r15); + kvm_r8_write_raw(vcpu, regs->r8); + kvm_r9_write_raw(vcpu, regs->r9); + kvm_r10_write_raw(vcpu, regs->r10); + kvm_r11_write_raw(vcpu, regs->r11); + kvm_r12_write_raw(vcpu, regs->r12); + kvm_r13_write_raw(vcpu, regs->r13); + kvm_r14_write_raw(vcpu, regs->r14); + kvm_r15_write_raw(vcpu, regs->r15); #endif =20 kvm_rip_write(vcpu, regs->rip); @@ -13091,7 +13092,7 @@ void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool ini= t_event) * on RESET. But, go through the motions in case that's ever remedied. */ cpuid_0x1 =3D kvm_find_cpuid_entry(vcpu, 1); - kvm_rdx_write(vcpu, cpuid_0x1 ? cpuid_0x1->eax : 0x600); + kvm_edx_write(vcpu, cpuid_0x1 ? cpuid_0x1->eax : 0x600); =20 kvm_x86_call(vcpu_reset)(vcpu, init_event); =20 diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c index 895095dc684e..694b31c1fcc9 100644 --- a/arch/x86/kvm/xen.c +++ b/arch/x86/kvm/xen.c @@ -1408,7 +1408,7 @@ int kvm_xen_hvm_config(struct kvm *kvm, struct kvm_xe= n_hvm_config *xhc) =20 static int kvm_xen_hypercall_set_result(struct kvm_vcpu *vcpu, u64 result) { - kvm_rax_write(vcpu, result); + kvm_rax_write_raw(vcpu, result); return kvm_skip_emulated_instruction(vcpu); } =20 @@ -1679,29 +1679,29 @@ int kvm_xen_hypercall(struct kvm_vcpu *vcpu) u8 cpl; =20 /* Hyper-V hypercalls get bit 31 set in EAX */ - if ((kvm_rax_read(vcpu) & 0x80000000) && + if ((kvm_rax_read_raw(vcpu) & 0x80000000) && kvm_hv_hypercall_enabled(vcpu)) return kvm_hv_hypercall(vcpu); =20 longmode =3D is_64_bit_hypercall(vcpu); if (!longmode) { - input =3D (u32)kvm_rax_read(vcpu); - params[0] =3D (u32)kvm_rbx_read(vcpu); - params[1] =3D (u32)kvm_rcx_read(vcpu); - params[2] =3D (u32)kvm_rdx_read(vcpu); - params[3] =3D (u32)kvm_rsi_read(vcpu); - params[4] =3D (u32)kvm_rdi_read(vcpu); - params[5] =3D (u32)kvm_rbp_read(vcpu); + input =3D kvm_eax_read(vcpu); + params[0] =3D kvm_ebx_read(vcpu); + params[1] =3D kvm_ecx_read(vcpu); + params[2] =3D kvm_edx_read(vcpu); + params[3] =3D kvm_esi_read(vcpu); + params[4] =3D kvm_edi_read(vcpu); + params[5] =3D kvm_ebp_read(vcpu); } else { #ifdef CONFIG_X86_64 - input =3D (u64)kvm_rax_read(vcpu); - params[0] =3D (u64)kvm_rdi_read(vcpu); - params[1] =3D (u64)kvm_rsi_read(vcpu); - params[2] =3D (u64)kvm_rdx_read(vcpu); - params[3] =3D (u64)kvm_r10_read(vcpu); - params[4] =3D (u64)kvm_r8_read(vcpu); - params[5] =3D (u64)kvm_r9_read(vcpu); + input =3D (u64)kvm_rax_read_raw(vcpu); + params[0] =3D (u64)kvm_rdi_read_raw(vcpu); + params[1] =3D (u64)kvm_rsi_read_raw(vcpu); + params[2] =3D (u64)kvm_rdx_read_raw(vcpu); + params[3] =3D (u64)kvm_r10_read_raw(vcpu); + params[4] =3D (u64)kvm_r8_read_raw(vcpu); + params[5] =3D (u64)kvm_r9_read_raw(vcpu); #else KVM_BUG_ON(1, vcpu->kvm); return -EIO; --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52B78376475 for ; Fri, 29 May 2026 22:22:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093377; cv=none; b=cf0j+YjUatxPNxeqmW/0JckkYwyQDeinbDEyQNEP4ITQZ5LtY8oTSIK9m03H6+c9VBlP225HAQ5wtj/ncEzIVVgHBFKY1/PldQT6A6oi9J0LxelOc8t0cQiyG6s7lCYoXG01+sCnUI80tQVJ0a3ni+w5Pli37ageFlII/rrDDf4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093377; c=relaxed/simple; bh=7mIOAYDO5oSpz1HswBSG6LYqEatOLHW5qOqOaTlM01Q=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=MyAaaU3+InLlXVwVWLW5HAOBXdJXNsVsrdTemRECVOp5yodE3J3yH0fARu2kLN5HAticsPi5trSC5lfbjnAROv1il6+NuPrQFHZ09ezTug27Tj3j0FrNr08ETDVKFalsFQCT3xcZpS2TBX0sgdtLAAxuiszj1PvCqfGzVlR9C10= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=KQE68mFt; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="KQE68mFt" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-36bc54005a7so1341293a91.0 for ; Fri, 29 May 2026 15:22:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093373; x=1780698173; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=jKofG2Hvtgv/l66zJ72qbyioy0cgxVopkyT0Bv1ocwE=; b=KQE68mFtJfvxiBmycb+hSiJsRsK8mD4hZH3AZZFQKxAerQrgXcSM946oMyN3HSCyYK byYvUxeVVmezKDecLfbhxe5RfYcoiVwbXmUYs73v8mD4ifKtuQaXQlqo0DKhJ7743P48 V7/3khwfJ6jlGj7GpKUngvpnYyrgi3Kz24x+I+j8Re7r54ztKM8jY0Kh1o4aiQlAn/xY u3FrLovgXSvKuxNqfbVPTmKW/VoTKsIJMBz+IgnWd5BH3w1N2tYxtYkaZMndWpBGahp4 d6FCT6qFJgwx1BuBIVh/HvAGy+f4NsE1BbeKCGLQju4j2peOV6f6VzWRoUVxQ2hE1cxd vLWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093373; x=1780698173; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=jKofG2Hvtgv/l66zJ72qbyioy0cgxVopkyT0Bv1ocwE=; b=B9a3/FX/YOU1JhCcxYMQ4ozcZmncy+0YLHAw+pRo8TyQi1K8Jr4lVpM6AQXogkezNc YMoq1+zPxgfqkYVRSYodWwN37j7xqoTQaW4bHoOYrdrZ3ZjS5qGzOlzlbRoz/m3UBUCq 2utglDA1OURP7oGO0VVWvyWTi4WRGzYi6lx5xpn1d8wxfoKfPRiQSnSOTZFDaVNv+oUj oT23qHALoFNSFezyTDwwMk7KND7KSfKEeiuBlhKJlp1bZTbwHTtdIZu7PXnI9/jMtm8o ZVKJrPAtMVkGSiPnZLrHFJc/KIs+IJlpt3WJc7zcAtLy9COaS6Inu0eGGdSG0GaldIO8 gIdQ== X-Forwarded-Encrypted: i=1; AFNElJ8CGUZd/0mnRzh4zRHAcCylkb43eRQyVsgFRWMzhDAB2hBk6efk19slA2xV1HHgsnKCeR1u3YMmjmtljEM=@vger.kernel.org X-Gm-Message-State: AOJu0YwbR0bUegr01wWGsJLgjdCF3qZFdmJ7nEF4aGOtFoX9Woxh83bW PtImQ0vovc89ju87kj7+j/mgV4Im6rr5icmb+0gY6IA0+QzyNnwjM8XC68kYr0w7PbKDVPAX9bx sV08OxA== X-Received: from pgig5.prod.google.com ([2002:a63:f405:0:b0:c80:2399:151]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:6017:b0:39f:a42:9251 with SMTP id adf61e73a8af0-3b427f2af2cmr1231557637.22.1780093373357; Fri, 29 May 2026 15:22:53 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:21:52 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-10-seanjc@google.com> Subject: [PATCH v3 09/40] KVM: x86: Drop non-raw kvm__write() helpers From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Drop the non-raw, mode-aware kvm__write() helpers as there is no usage in KVM, and in all likelihood there will never be usage in KVM as use of hardcoded registers in instructions is uncommon, and *modifying* hardcoded registers is practically unheard of. While there are a few instructions that modify registers in mode-aware ways, e.g. REP string and some ENCLS varieties, the odds of KVM needing to emulate such instructions (outside of the fully emulator) are vanishingly small. Drop kvm__write() to prevent incorrect usage; _if_ a new instruction comes along that needs to modify a hardcoded register, this can be reverted. No functional change intended. Reviewed-by: Binbin Wu Signed-off-by: Sean Christopherson --- arch/x86/kvm/regs.h | 5 ----- 1 file changed, 5 deletions(-) diff --git a/arch/x86/kvm/regs.h b/arch/x86/kvm/regs.h index 12db5039aace..f22b3a8cd483 100644 --- a/arch/x86/kvm/regs.h +++ b/arch/x86/kvm/regs.h @@ -61,11 +61,6 @@ static __always_inline unsigned long kvm_##lname##_read(= struct kvm_vcpu *vcpu) { \ return vcpu->arch.regs[VCPU_REGS_##uname] & kvm_reg_mode_mask(vcpu); \ } \ -static __always_inline void kvm_##lname##_write(struct kvm_vcpu *vcpu, \ - unsigned long val) \ -{ \ - vcpu->arch.regs[VCPU_REGS_##uname] =3D val & kvm_reg_mode_mask(vcpu); \ -} \ static __always_inline unsigned long kvm_##lname##_read_raw(struct kvm_vcp= u *vcpu) \ { \ return vcpu->arch.regs[VCPU_REGS_##uname]; \ --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 894903BCD0F for ; Fri, 29 May 2026 22:22:55 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093378; cv=none; b=E8Nnv4iW1cMtEjoevV7kB9UXI//5UwYwSIX+NH4HYB/HP3Uf3vT1nDYjnR03XG39hAEhxVa6SEiN2g7Hhpn8oechrN8TkPwn5qFDRqHd1zpQZfwnLAhV+/1tQg8CI4onfOh4MCGNe15/Gp5dch842xJFsj0yT90qDPR52P/BHhE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093378; c=relaxed/simple; bh=yAMlD+Ut9Mr5Vv3GBz5a6xiUa7kPP+TY0iPeK2IZP3s=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=cXbBU/rM8GoyTgwVxzrIUQ4Lgt1QiJv/fho77iiF0CGJhGcUU7sdjsGwaIIHWtuTTE0KZiVwGmq4oS+1+2zyBix90E6qZlIp+OBzXqL9UKlu9iBXjHf5HHMmZztoocoW7KcHA1M/y27ZEx+jrm2JqWKvtdwYI6z/vJasVBVNP/E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=qXci73IX; arc=none smtp.client-ip=209.85.210.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="qXci73IX" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-83eefe4867eso7271406b3a.1 for ; Fri, 29 May 2026 15:22:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093375; x=1780698175; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=TO/Qd/fwKscqv3OevgPA+uIhx1kFfn/8CSAsDvSeZD0=; b=qXci73IX+54/z3/rLmqRUQeSmg/ZPdxhomI9rOPgHzmDl0B1/SWainOJPYYkhUJ5Zs 3qhv8sLCmI0G5yiiIW/PunEysWjG0/JarVyfkVgplxOhl9vP4+h4qhD5pxs8YwZYv8Kf Fcbc73vnjASbwriUEkz45swcgql4bp4CB3MNI9hxPP96MrnCDLUPzjRgA0UsYMW8CQ+o 5DHZhtd8MUEow7KrI8c6dFJ7itOkgbavIWoKxNmG3bdqfct9TGsJS2n4FCIW318QpdsU T2RzZyq6QQ0CGzMJhY0WNjufSE8RUCrbwmdgQdTuJdpTcbAyn096tDqlC9cNMKtL8NRL 8o0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093375; x=1780698175; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=TO/Qd/fwKscqv3OevgPA+uIhx1kFfn/8CSAsDvSeZD0=; b=FOLOe+pbZfCf3GfJ4dAyz/l4PMkVcwZ/zHw67MLjX1KF1SdcfvpVBIZfIl9Ep8vjm8 4cgtU78BtKELiNMmBvIBQ/3NHh07UVSzUjcUm75oT3obYCvjl7BBZM4LG5ls3H3F0VKD DCM4G+WLtbaQ7zO2iDq1tqF/J4ibRvcxl2DyX0vXFTSMKNyoDcgiThGfPhxSDxDp7cyN MaatJXaOhhvYIO+QHtZX6/KkzoQlLh7mBn90LM7N7BxuvJzUkcgbC8TPEMpAKTrzHb2S otfvRTAbKBszeF0HOTy3CJnqP6YcWf77dU/G/lTAJDZQYExOAHmoBWEn36sLd/O8jL2w zsoA== X-Forwarded-Encrypted: i=1; AFNElJ+YOdD8Xh22jPDdcaTbTXI+oyqLasKja6j0b3GxYm8QAAW89kRamNIqGH7ohlvaPOWwlcA8LXcHgQCn7pw=@vger.kernel.org X-Gm-Message-State: AOJu0Yzi4bATqhFfG55+lkmFWjcvM1i0cjhDXIa+HlygE0almHzfvZle vFA7tkXWYbiWzGAhuLYy0wGFVg+/g7JSHajhk0pHqkzaiapCeRf1o2fU4FDaKt5wpovQoigjXVT Q+s/Upg== X-Received: from pfbhu46.prod.google.com ([2002:a05:6a00:69ae:b0:82f:344e:386a]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:4196:b0:838:127d:a16e with SMTP id d2e1a72fcca58-84225394e9amr1230292b3a.17.1780093374509; Fri, 29 May 2026 15:22:54 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:21:53 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-11-seanjc@google.com> Subject: [PATCH v3 10/40] KVM: nSVM: Use kvm_rax_read() now that it's mode-aware From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that kvm_rax_read() truncates the output value to 32 bits if the vCPU isn't in 64-bit mode, use it instead of the more verbose (and very technically slower) kvm_register_read(). Note! VMLOAD, VMSAVE, and VMRUN emulation are still technically buggy, as they can use EAX (versus RAX) in 64-bit mode via an operand size prefix. Don't bother trying to handle that case, as it would require decoding the code stream, which would open an entirely different can of worms, and in practice no sane guest would shove garbage into RAX[63:32] and then execute VMLOAD/VMSAVE/VMRUN with just EAX. No functional change intended. Cc: Yosry Ahmed Reviewed-by: Yosry Ahmed Signed-off-by: Sean Christopherson --- arch/x86/kvm/svm/nested.c | 2 +- arch/x86/kvm/svm/svm.c | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index d817dbb350d6..1ab8b95975a4 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -1135,7 +1135,7 @@ int nested_svm_vmrun(struct kvm_vcpu *vcpu) if (WARN_ON_ONCE(!svm->nested.initialized)) return -EINVAL; =20 - vmcb12_gpa =3D kvm_register_read(vcpu, VCPU_REGS_RAX); + vmcb12_gpa =3D kvm_rax_read(vcpu); if (!page_address_valid(vcpu, vmcb12_gpa)) { kvm_inject_gp(vcpu, 0); return 1; diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 8402e94ac094..526e0fdcd16b 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -2218,7 +2218,7 @@ static int intr_interception(struct kvm_vcpu *vcpu) =20 static int vmload_vmsave_interception(struct kvm_vcpu *vcpu, bool vmload) { - u64 vmcb12_gpa =3D kvm_register_read(vcpu, VCPU_REGS_RAX); + u64 vmcb12_gpa =3D kvm_rax_read(vcpu); struct vcpu_svm *svm =3D to_svm(vcpu); struct vmcb *vmcb12; struct kvm_host_map map; @@ -2326,7 +2326,7 @@ static int gp_interception(struct kvm_vcpu *vcpu) if (nested_svm_check_permissions(vcpu)) return 1; =20 - if (!page_address_valid(vcpu, kvm_register_read(vcpu, VCPU_REGS_RAX))) + if (!page_address_valid(vcpu, kvm_rax_read(vcpu))) goto reinject; =20 /* --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 18E48436358 for ; Fri, 29 May 2026 22:22:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093381; cv=none; b=meoPRUQN3CRGNSdqx38Gg1koUeoPjUQornmeJ8Kn60sBuxKOO12ua6k0bvqxAWBMrlJwQAhjSfo1FVgJGSmKmD4DWxZFNu6ol21B8kOH9xXOl0u9xRprcFB/A3wgC6FD0yeGf2s1SFZoL7Z1YOKFC9x1vurGKLRxZ+TuO1oKteo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093381; c=relaxed/simple; bh=y7gBMyXrlhkkVrixZ3rMHYtJrOyLsBu671OBCVJh2sk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=lsdC6ZGQj3gRA69tCRjZeTIhh6Ww1JUIHhDY7p12au5xjQvyyFxoeaBwRfVo7AAwTV2fyFn8hRL1Sdw1KMRPL6XO61md20dHVDnD6FZQL2EkUsYe6xpK8+JLJ6fWGhdvKVb6pBYH1tdFywWngsh9LFiIDzkf6f6FLP+8Cf/upZY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=A5L8bPr/; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="A5L8bPr/" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2bd04e4fe3dso118360165ad.3 for ; Fri, 29 May 2026 15:22:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093376; x=1780698176; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=1d+7JTGp/HbwqWUEDQGxoEibOm7OCRTR3a6kaHjtDYs=; b=A5L8bPr/DG/tde906di8RuRDge7PPWczF3eRDYmvbnhaN7L4bgn4KwOLds716P/whg MuWH9oFlcw7utUmfTtbaQjBSGNkGYMbigTFVBuLDgdBe2dy0Byrzb5+Dyh442oLqsh0K aL52NYh/VJ9Sn2nqo9f8FcrAW2fLFvh9xyFV292CZV+l9Llybuwc1J0SaAvjzWHbskn4 VqifCa1Y9+6N29w8OW15GuLzF9P2yjQwuiIB2NNBu0aFCaJ65NGMqs3MKWBOwnTvv0lU 3Z+ZzFWSzw2vfCNK/hb/mQvk97pOdYjVQDNTqryHkOkkwiPrxb3KAkeFhamfvi/Xze4h 8iSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093376; x=1780698176; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=1d+7JTGp/HbwqWUEDQGxoEibOm7OCRTR3a6kaHjtDYs=; b=VuVLTwfr+dQ+oZFMHWYDMPbKUCOD6w2Nhg8NeKkJXEyOcF+FC0pRDFgvgE5aORFK1K PghIj3EnKXDEiYDp/mXlGSk6huaU4QA2AYqZqBOJIXzXmyd1BRgPcKUsS2fg3ao41hto J5/kweLylenA3KSendqaZFEDKFzkWOyRuOg9AzeqRTCLe7YT+J39dTPIefox2QvBKYLa LnEotA2jgWSvSFoWnsL1QHhD9kEYoKD9IN1o1Ye2Dwhs0ra8PY9JG+cCM0X+40CWby0x 7RNiRMzVrRP8ou450jYMTa0xwTx8rjxcRKpUywPxp0le4/aKL6PuwAbu9w/vO5oa7PTC HYMQ== X-Forwarded-Encrypted: i=1; AFNElJ+2F6inSUkHhMjKsg7rMwTzDcNF3s63y89POdP0pWyVn3viAEe5KOdSXpL2+143C0tDXx5tWkbxKJi3RwQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yw24mNxcb2zdxSLclCwMbUmfIvF4sngSf8VT6tsD8DCrjcj26Th b6pbjIjSfrpSxUG9DzpsCwdG8wj/nuoVrkeRGbwrdu5HmzeCxJi2c7jmdywobTV97KNrIExNeiR 93w9oVA== X-Received: from plqq10.prod.google.com ([2002:a17:902:c74a:b0:2be:bf91:ff92]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:8d0:b0:2bd:c5f8:504f with SMTP id d9443c01a7336-2bf368d9ee5mr18560845ad.40.1780093375621; Fri, 29 May 2026 15:22:55 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:21:54 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-12-seanjc@google.com> Subject: [PATCH v3 11/40] Revert "KVM: VMX: Read 32-bit GPR values for ENCLS instructions outside of 64-bit mode" From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Now that kvm__read() are mode aware, i.e. are functionally equivalent to kvm_register_read(), revert aback to the less verbose versions. No functional change intended. This reverts commit 60919eccf6764c71cef31a1afeaa1a36b8e5ab85. Acked-by: Kai Huang Reviewed-by: Binbin Wu Signed-off-by: Sean Christopherson --- arch/x86/kvm/vmx/sgx.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/vmx/sgx.c b/arch/x86/kvm/vmx/sgx.c index 2f5a1c58f3c5..876dc2814108 100644 --- a/arch/x86/kvm/vmx/sgx.c +++ b/arch/x86/kvm/vmx/sgx.c @@ -225,8 +225,8 @@ static int handle_encls_ecreate(struct kvm_vcpu *vcpu) struct x86_exception ex; int r; =20 - if (sgx_get_encls_gva(vcpu, kvm_register_read(vcpu, VCPU_REGS_RBX), 32, 3= 2, &pageinfo_gva) || - sgx_get_encls_gva(vcpu, kvm_register_read(vcpu, VCPU_REGS_RCX), 4096,= 4096, &secs_gva)) + if (sgx_get_encls_gva(vcpu, kvm_rbx_read(vcpu), 32, 32, &pageinfo_gva) || + sgx_get_encls_gva(vcpu, kvm_rcx_read(vcpu), 4096, 4096, &secs_gva)) return 1; =20 /* @@ -302,9 +302,9 @@ static int handle_encls_einit(struct kvm_vcpu *vcpu) gpa_t sig_gpa, secs_gpa, token_gpa; int ret, trapnr; =20 - if (sgx_get_encls_gva(vcpu, kvm_register_read(vcpu, VCPU_REGS_RBX), 1808,= 4096, &sig_gva) || - sgx_get_encls_gva(vcpu, kvm_register_read(vcpu, VCPU_REGS_RCX), 4096,= 4096, &secs_gva) || - sgx_get_encls_gva(vcpu, kvm_register_read(vcpu, VCPU_REGS_RDX), 304, = 512, &token_gva)) + if (sgx_get_encls_gva(vcpu, kvm_rbx_read(vcpu), 1808, 4096, &sig_gva) || + sgx_get_encls_gva(vcpu, kvm_rcx_read(vcpu), 4096, 4096, &secs_gva) || + sgx_get_encls_gva(vcpu, kvm_rdx_read(vcpu), 304, 512, &token_gva)) return 1; =20 /* --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EFF6E3E717B for ; Fri, 29 May 2026 22:22:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093379; cv=none; b=L6TrTYyYpfLuL4SvjtjIysdL/D3eUDPIA+4PACTM2lFhSvmG9tv/tVIUXQdQvrdu4BYFyqE8upHTAyxJt/zbIb95YZF9QK/oUSYddlGgw3S/bQfpvYJuqYJN/JDtDpbIbLYeTzcD6EJVt7SEiKzZNm3nPN/cy0QPYf/tQMd0Mqs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093379; c=relaxed/simple; bh=O6sCALoH9EmCcgisAiG+6wKleryQICAogURF+M73YgA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=mUgZIICjm7w3MZYyiVeXupz/+Oj5g0dpH9fwiZrrGrIPStMQ0AJJ0J7o4A7+9ZgSjULtF5JHZK9E+rXX5MIdYt0Y642VAhieoSL4Kf/Y/nUk+JKF2KKcd5j3GlnuTVfsg/M4Rgdgo+VDPd5GigEKl93CvrivKZFc5UB6VUcFdBM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=cOh/ReEf; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="cOh/ReEf" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2bd6aeb3637so318815975ad.2 for ; Fri, 29 May 2026 15:22:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093377; x=1780698177; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=YESkxGyiczq5Z97XVWhBBl+heltge9f5w5uhIKBe0e8=; b=cOh/ReEfkJwKeVn8l2OTNCWPqm43yFH4HPdtGNZkT3jiXK5SFU+fHT3vSV4tpZecz1 TDgy3s53T2VPP5kBx/UR87YcYkTXEMw6wR29GaBgHgBHe76fMkINhnTtzx86EYmvFeZi FreVb0k7TDYr7MaVfz1iChqH9SbeVjM0TTIUX5kFSXN7ULgSmj7HCKa7U/miCzJPGjkm acfnI7Q79Ijdm8f7X3yWoz+auYYeo8DqnFzWrklMtkDVuaNZEsWNIHalAMZ4HP8pC8pg rmGiHvpeMFh58QJmIogotxanjH/oxQQ0hrS1avIFp3A7sooYNK6Qrog3I+cTGrRnHYX+ kegQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093377; x=1780698177; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=YESkxGyiczq5Z97XVWhBBl+heltge9f5w5uhIKBe0e8=; b=STQuHaPwxv13fluWRlizXMHzbPZa2pAoS8ALSA+r23ZgcquB6ZbMajBgllqsYa0Amc qrx4/WBJKkgyun3TT38X3Ql0PsBDxQi55g+1GtkBaOylO5WYaH9h1gUxKafF1EY/Bcg2 ZS9OA5K/IQHMYp7I8kkuqbHq6pBsV72ONMprK4e4G3hXmSqc443vM2F+TJmpoPd0bpqe EXovOg+nT5t4rZgX5NqFOtwPhtgiHaTnHfYiF4DvR31ZPBde2ergvwM2nh48QVYQvMyd ZHrxZv5+qqGpw9WaUO1s1JM9dMOLyG5iLmorneksgcN3e1l2FfmNUat4wBOSBwDj2idq OaLA== X-Forwarded-Encrypted: i=1; AFNElJ8z542PXluKaDf9EwvgVGJo22czol/TBUvhmJCzh2WEQ3CBcBMvBfBv9Oi/pk+R3oGQRoOjrkLxMoT3YtQ=@vger.kernel.org X-Gm-Message-State: AOJu0YyG0ITra2LUxZR5Jwilv9wepEC+lRCHNynUv6KW8cewVHxLfFYs rN3QwTB9nkvHjbcp8+sz2wsbo3jCx0H5MQkB1pvUdKrEh2mOKG87mATX0AGq+j0souvV1Zk4G88 4UJOm8A== X-Received: from plbkn8.prod.google.com ([2002:a17:903:788:b0:2bd:25ca:82e9]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:d4c4:b0:2bf:e42:2a94 with SMTP id d9443c01a7336-2bf368a4096mr19964205ad.38.1780093377142; Fri, 29 May 2026 15:22:57 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:21:55 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-13-seanjc@google.com> Subject: [PATCH v3 12/40] KVM: x86: Harden is_64_bit_hypercall() against bugs on 32-bit kernels From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Unconditionally return %false for is_64_bit_hypercall() on 32-bit kernels to guard against incorrectly setting guest_state_protected, and because in a (very) hypothetical world where 32-bit KVM supports protected guests, assuming a hypercall was made in 64-bit mode is flat out wrong. Reviewed-by: Kai Huang Reviewed-by: Binbin Wu Signed-off-by: Sean Christopherson --- arch/x86/kvm/regs.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/arch/x86/kvm/regs.h b/arch/x86/kvm/regs.h index f22b3a8cd483..a57ba26279ed 100644 --- a/arch/x86/kvm/regs.h +++ b/arch/x86/kvm/regs.h @@ -39,12 +39,16 @@ static inline bool is_64_bit_mode(struct kvm_vcpu *vcpu) =20 static inline bool is_64_bit_hypercall(struct kvm_vcpu *vcpu) { +#ifdef CONFIG_X86_64 /* * If running with protected guest state, the CS register is not * accessible. The hypercall register values will have had to been * provided in 64-bit mode, so assume the guest is in 64-bit. */ return vcpu->arch.guest_state_protected || is_64_bit_mode(vcpu); +#else + return false; +#endif } =20 static __always_inline unsigned long kvm_reg_mode_mask(struct kvm_vcpu *vc= pu) --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 299A6401A29 for ; Fri, 29 May 2026 22:22:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093380; cv=none; b=ezHigjMdw3/oNgUJOt/X7juAzIj9ghYCifOMwxnZAHt0nB+v6SQwNCk0tNuKp4mXb+gReFeF7IV1roYJfgr/W688alzkIszjHOKmd6H4qgb77XDlyVhugca0m1HuhRi4rd6LL4lBymXvA06FQrujyOtwX/inV15QWiY//X1aoOQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093380; c=relaxed/simple; bh=EY1UUzjByJuYr1Hg1BkNVerLXRBIjRX+RVz1UvMuW9g=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=gkepCMfHZhC0uwWw8aRtFoKTXmHX64LQBmG9PcVfkl5IZnEHMZuw8+pE6wQ+5MDFyLIANzFRw2UNqqwMiJcGGf1GL9AEwfwL1Jdx1pvgsIbbxKMt8F5WPNpNwZrBCpxGXwna8/xa1B2gKAViceu/Ie83HkaKVMgeQP4YeVDjDo0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=XTGh4B+6; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="XTGh4B+6" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2bf08c2a24bso26327815ad.2 for ; Fri, 29 May 2026 15:22:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093378; x=1780698178; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=lYYEV5eYbnO5d9jgWyRMntwUD0jcJ4Dk7yFI3OcsKAc=; b=XTGh4B+6joX60vacpmfqCs1Qe39MK7AiCxMRJUWybOQKlSgQqIROTMZmeqcKlUtS6u PBjPkIo47J8cbNjc7YLBq7LVIeFP5PjGxBfwSm4IbNmpfQAIRJhuhRLSI1sLrC5pRUYI JpSxBC1stL8IvYA0aS9tHw+DFzzJaO82+mVqacK+UBHggAb0gI9ngRgDdmxUxRkOhtvC N2aqs4O4Q8n5eIPrsUz+fC47ahwEUrtbC8b+PbpFEFJZxk8NZI7Wr9gu1ljbi71xDjAw ybX4TWSluvMhDdXDv0PHvihG7egwXsUeqdX/GxIKvTqJ9WLrH+8FINjDXUdHPtUsqkCB PJMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093378; x=1780698178; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=lYYEV5eYbnO5d9jgWyRMntwUD0jcJ4Dk7yFI3OcsKAc=; b=HrfIbfdRZCH4/rjCA5aWSaBmH/6oQSkbsxXgacOuvinVrQ1v6bBXJdPGTyIi59WrMN ThwbilsEEB1TBolW0gTCUH8vKxEE/CeHpb16i+4Z9UC+Sl295G5oFbAvGIBaVMTAFPhz rLFt5xjQb7h5RG2Ti+pgoyLwYN3rud8h6MLwI4drEDdBkhBuCB6XtOBz/yjM/CCuZLI4 /BzUUzB0kXNbyLXS9tBDaOYncWpF1BtKw2R+pxSZRStOechpCC46O2NwfyZmL7CIBdg5 jrGcHCy92rTRdYU1atrrEGA8FA3BriRCixqISvuUXA/ectsUGvk1cHEUYEa1EJflDQ9S agzQ== X-Forwarded-Encrypted: i=1; AFNElJ8QLJV1TU8tTS7tL4+rP+rupayteySg5FN28uwGSGsil6wHXAXWjo69EVcs4Y+7kUAxJfjASNtzZmlEocM=@vger.kernel.org X-Gm-Message-State: AOJu0YylxA6xOCLTmNz+SJoL2lAIhUAgsJ2VdseDO+Cr0CxcBgaHRGRi MBUclA3fwWuu8GRMmXEXtIn0bkeqrPcuhbOTvPwuyzUPpZE4PnjHmFnDjqp0GNO55m3bwQns9l6 /Q0DoWQ== X-Received: from plem4.prod.google.com ([2002:a17:902:e404:b0:2bc:f203:12a1]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:2450:b0:2b9:86ca:5bf2 with SMTP id d9443c01a7336-2bf368463f1mr18712055ad.30.1780093378228; Fri, 29 May 2026 15:22:58 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:21:56 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-14-seanjc@google.com> Subject: [PATCH v3 13/40] KVM: x86: Move update_cr8_intercept() to lapic.c From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move update_cr8_intercept() to lapic.c so that it's globally visible in anticipation of extracting most of the register-specific code out of x86.c and into a new compilation unit. Opportunistically prefix the helper kvm_lapic_ to make its role/scope more obvious. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Reviewed-by: Yosry Ahmed --- arch/x86/kvm/lapic.c | 26 ++++++++++++++++++++++++++ arch/x86/kvm/lapic.h | 1 + arch/x86/kvm/x86.c | 34 +++------------------------------- 3 files changed, 30 insertions(+), 31 deletions(-) diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c index 15777869b292..9d2df8623f6d 100644 --- a/arch/x86/kvm/lapic.c +++ b/arch/x86/kvm/lapic.c @@ -2761,6 +2761,32 @@ u64 kvm_lapic_get_cr8(struct kvm_vcpu *vcpu) return (tpr & 0xf0) >> 4; } =20 +void kvm_lapic_update_cr8_intercept(struct kvm_vcpu *vcpu) +{ + int max_irr, tpr; + + if (!kvm_x86_ops.update_cr8_intercept) + return; + + if (!lapic_in_kernel(vcpu)) + return; + + if (vcpu->arch.apic->apicv_active) + return; + + if (!vcpu->arch.apic->vapic_addr) + max_irr =3D kvm_lapic_find_highest_irr(vcpu); + else + max_irr =3D -1; + + if (max_irr !=3D -1) + max_irr >>=3D 4; + + tpr =3D kvm_lapic_get_cr8(vcpu); + + kvm_x86_call(update_cr8_intercept)(vcpu, tpr, max_irr); +} + static void __kvm_apic_set_base(struct kvm_vcpu *vcpu, u64 value) { u64 old_value =3D vcpu->arch.apic_base; diff --git a/arch/x86/kvm/lapic.h b/arch/x86/kvm/lapic.h index f763cd29a508..71970213dc1f 100644 --- a/arch/x86/kvm/lapic.h +++ b/arch/x86/kvm/lapic.h @@ -100,6 +100,7 @@ int kvm_apic_accept_events(struct kvm_vcpu *vcpu); void kvm_lapic_reset(struct kvm_vcpu *vcpu, bool init_event); u64 kvm_lapic_get_cr8(struct kvm_vcpu *vcpu); void kvm_lapic_set_tpr(struct kvm_vcpu *vcpu, unsigned long cr8); +void kvm_lapic_update_cr8_intercept(struct kvm_vcpu *vcpu); void kvm_lapic_set_eoi(struct kvm_vcpu *vcpu); void kvm_apic_set_version(struct kvm_vcpu *vcpu); void kvm_apic_after_set_mcg_cap(struct kvm_vcpu *vcpu); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index a102269a7af9..034428b3d8e9 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -128,7 +128,6 @@ static u64 __read_mostly efer_reserved_bits =3D ~((u64)= EFER_SCE); KVM_X2APIC_ENABLE_SUPPRESS_EOI_BROADCAST | \ KVM_X2APIC_DISABLE_SUPPRESS_EOI_BROADCAST) =20 -static void update_cr8_intercept(struct kvm_vcpu *vcpu); static void process_nmi(struct kvm_vcpu *vcpu); static void __kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags); static void store_regs(struct kvm_vcpu *vcpu); @@ -5337,7 +5336,7 @@ static int kvm_vcpu_ioctl_set_lapic(struct kvm_vcpu *= vcpu, r =3D kvm_apic_set_state(vcpu, s); if (r) return r; - update_cr8_intercept(vcpu); + kvm_lapic_update_cr8_intercept(vcpu); =20 return 0; } @@ -10582,33 +10581,6 @@ static void post_kvm_run_save(struct kvm_vcpu *vcp= u) kvm_run->flags |=3D KVM_RUN_X86_GUEST_MODE; } =20 -static void update_cr8_intercept(struct kvm_vcpu *vcpu) -{ - int max_irr, tpr; - - if (!kvm_x86_ops.update_cr8_intercept) - return; - - if (!lapic_in_kernel(vcpu)) - return; - - if (vcpu->arch.apic->apicv_active) - return; - - if (!vcpu->arch.apic->vapic_addr) - max_irr =3D kvm_lapic_find_highest_irr(vcpu); - else - max_irr =3D -1; - - if (max_irr !=3D -1) - max_irr >>=3D 4; - - tpr =3D kvm_lapic_get_cr8(vcpu); - - kvm_x86_call(update_cr8_intercept)(vcpu, tpr, max_irr); -} - - int kvm_check_nested_events(struct kvm_vcpu *vcpu) { if (kvm_test_request(KVM_REQ_TRIPLE_FAULT, vcpu)) { @@ -11349,7 +11321,7 @@ static int vcpu_enter_guest(struct kvm_vcpu *vcpu) kvm_x86_call(enable_irq_window)(vcpu); =20 if (kvm_lapic_enabled(vcpu)) { - update_cr8_intercept(vcpu); + kvm_lapic_update_cr8_intercept(vcpu); kvm_lapic_sync_to_vapic(vcpu); } } @@ -12495,7 +12467,7 @@ static int __set_sregs_common(struct kvm_vcpu *vcpu= , struct kvm_sregs *sregs, kvm_set_segment(vcpu, &sregs->tr, VCPU_SREG_TR); kvm_set_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR); =20 - update_cr8_intercept(vcpu); + kvm_lapic_update_cr8_intercept(vcpu); =20 /* Older userspace won't unhalt the vcpu on reset. */ if (kvm_vcpu_is_bsp(vcpu) && kvm_rip_read(vcpu) =3D=3D 0xfff0 && --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A89D03BED35 for ; Fri, 29 May 2026 22:23:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093382; cv=none; b=PvKXoyzVcPVbPdKbq/OPhWHeVd7EuKeqNFFvAX2dZQiaOXIVldcSZSnN1lW10MYV+UHEpLkaYqTGTcLywMxINnM8U4DcRJKFtAzig6qcZhaHM3xCX4qsOAzpHv6yjXx5ijZKuMG0RrxcAM8303h3UgGH6ZFC25dXJluv24Gz4Oc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093382; c=relaxed/simple; bh=qffHEhD8jn09UL+S8DChESHEdACOyGeIIvRkxBpQBls=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=W9WQh6hB9rNn2wnmYnUEG+4poNJ9ME9Alr6SId9XwKTD0ZLvBCPmE6tGoNjl+ULD6borYNcW/Rp1LtiXMviin1DKQfnd+sLXAJuTbLVOS9XJUaSsosRxZCqhp1+ijdPjXe2jhzbpaEtePCDEMShvgRyXtQVBGlwBkvJfXLD8ITc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=bmpBtRgj; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="bmpBtRgj" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2c0532a6588so1167115ad.0 for ; Fri, 29 May 2026 15:23:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093380; x=1780698180; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=crjIC5UMAinWx92xFzz94uywqWRodLGcvs3KzXYdEwU=; b=bmpBtRgjyBR868UjCZ/DbOaNz3Ql5sdY1qTFQoBtdmLH+y9gmwE9aIb1AT9DILjlbF 7PS2695KEf/M8S1OYG6Nwkxg1kdWPXyui8yB4CFgEvGNlwPWMHJwCSAToZJln975NT2I DLBdt78LX/BFolAbta3/DbWCctrZT4jrSjcZ1cFU6yfjUE10bKYktuGM1fdja/FaIz2c 6Yz+6zKwOSv2rku5L7mnny7/yN+f7+DPFsVuL/qM7v/3vTh2B6nqP2JwLDoMggPuBtMY qT1AQrTogY4Tri587fOpzY1qCIpt2tsOx28wzwSAUeAv1w8LHRkqfsfvuepLzSAGZnFS fQIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093380; x=1780698180; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=crjIC5UMAinWx92xFzz94uywqWRodLGcvs3KzXYdEwU=; b=kFW5sjUVIgQ/bySGA/QXfqstxJNbnR+m7soDEkYfXdlVCR9y5KWL4x5t2FrEL4KO94 byglVlMkpbsd4c4yRi34qXLJWgqxr68GWsBDGWGshzOBpuechpzP7OFkcYJwW0KyCP7t u8Lvf4U5w2TIkYosoNkOYk/8etyVGqVOsCtXJppCpF3ZqA7GbzM3LNDGd4a8GzDSOEHF iSQlGlcG4sflXRoTX0hR/WfwqOk9kBecRtgPThNvx5MYYp8TC9wm2MScS29t36fWzaIg EzOk0kOzcV7rFekuOfQ2LjzMOprgwIw21X8ctA6bWLMy3cnmteow+cKA5M0bISog1kb3 +8zA== X-Forwarded-Encrypted: i=1; AFNElJ/7MopoPVKs+5LMl17pPDlnKcizJ22AP2jf6qxL7tybTvKNXPMY4qmUM9CSLyzZiJYYOp/JLPgxotADu28=@vger.kernel.org X-Gm-Message-State: AOJu0Yy7yD5Gn2fEVW8FWC9zJjdWUK4BZNRfb23nPekcaJ5Z2MRgwmf5 2+TBIkHSxFucTzN6015QDxfJnSvNpipYfDnvUu45OUGYDAbx0EgtM36nx9+EDe5USJi+iyapDHr aBvik4w== X-Received: from plmk20.prod.google.com ([2002:a17:903:1814:b0:2bd:a0e6:1a81]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:e54c:b0:2bf:30d2:e891 with SMTP id d9443c01a7336-2bf36894119mr21171075ad.36.1780093379894; Fri, 29 May 2026 15:22:59 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:21:57 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-15-seanjc@google.com> Subject: [PATCH v3 14/40] KVM: x86: Move async #PF helpers to x86.h (as inlines) From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move kvm_pv_async_pf_enabled() and kvm_async_pf_hash_reset() to x86.h in anticipation of extracting the majority of register and MSR specific code out of x86.c. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Reviewed-by: Yosry Ahmed --- arch/x86/kvm/x86.c | 19 ------------------- arch/x86/kvm/x86.h | 19 +++++++++++++++++++ 2 files changed, 19 insertions(+), 19 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 034428b3d8e9..cd68a5bad0c6 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -566,13 +566,6 @@ static struct kmem_cache *kvm_alloc_emulator_cache(voi= d) =20 static int emulator_fix_hypercall(struct x86_emulate_ctxt *ctxt); =20 -static inline void kvm_async_pf_hash_reset(struct kvm_vcpu *vcpu) -{ - int i; - for (i =3D 0; i < ASYNC_PF_PER_VCPU; i++) - vcpu->arch.apf.gfns[i] =3D ~0; -} - static void kvm_destroy_user_return_msrs(void) { int cpu; @@ -1023,18 +1016,6 @@ bool kvm_require_dr(struct kvm_vcpu *vcpu, int dr) } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_require_dr); =20 -static bool __kvm_pv_async_pf_enabled(u64 data) -{ - u64 mask =3D KVM_ASYNC_PF_ENABLED | KVM_ASYNC_PF_DELIVERY_AS_INT; - - return (data & mask) =3D=3D mask; -} - -static bool kvm_pv_async_pf_enabled(struct kvm_vcpu *vcpu) -{ - return __kvm_pv_async_pf_enabled(vcpu->arch.apf.msr_en_val); -} - static inline u64 pdptr_rsvd_bits(struct kvm_vcpu *vcpu) { return vcpu->arch.reserved_gpa_bits | rsvd_bits(5, 8) | rsvd_bits(1, 2); diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 3845b10020c9..acb22167901f 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -556,6 +556,25 @@ static inline bool kvm_pat_valid(u64 data) return (data | ((data & 0x0202020202020202ull) << 1)) =3D=3D data; } =20 +static inline bool __kvm_pv_async_pf_enabled(u64 data) +{ + u64 mask =3D KVM_ASYNC_PF_ENABLED | KVM_ASYNC_PF_DELIVERY_AS_INT; + + return (data & mask) =3D=3D mask; +} + +static inline bool kvm_pv_async_pf_enabled(struct kvm_vcpu *vcpu) +{ + return __kvm_pv_async_pf_enabled(vcpu->arch.apf.msr_en_val); +} + +static inline void kvm_async_pf_hash_reset(struct kvm_vcpu *vcpu) +{ + int i; + for (i =3D 0; i < ASYNC_PF_PER_VCPU; i++) + vcpu->arch.apf.gfns[i] =3D ~0; +} + /* * Trigger machine check on the host. We assume all the MSRs are already s= et up * by the CPU and that we still run on the same CPU as the MCE occurred on. --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pf1-f201.google.com (mail-pf1-f201.google.com [209.85.210.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1FB743AFB13 for ; Fri, 29 May 2026 22:23:01 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093387; cv=none; b=Tb2cBX/lYymO9/N9HlKfur4yGHKSuvL0xX1zZFZTiGFeYc9s5xLT/SmdLSTH5q9PpWlVxMylzHnkgyoM/BoAcGvNxh6herRFvlAWMe9BpN32+O/avrfbzIV+wHtydyY7JKX4tNHkZ5v/kiCOnqgX+a/MZUSD/FpTOb8xeqDtb/U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093387; c=relaxed/simple; bh=RZDrGiU2pyDGyMEHDy7XPWm7m2YMUa4OC6kjSo2+APs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=l1m3Q9TKigU7myftujf63lKkJdtXX2AG2oQ4MoQ/95FEilyaylVcrnf9j1Cu4Tok8upmIz2t1GTP9OnqJ6CF+BH7stsqobTOlJY3BSDj3AQVD5fxKAS1s8+KJcV4iyPub2I/a3G9ubLx0v4FBPqJAqLfZZ9FbGlgROE9vM7k128= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=gQUDlopo; arc=none smtp.client-ip=209.85.210.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="gQUDlopo" Received: by mail-pf1-f201.google.com with SMTP id d2e1a72fcca58-83ef22f8e8bso9724429b3a.0 for ; Fri, 29 May 2026 15:23:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093381; x=1780698181; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=SoQTly8/aFoEQsXn57jatIfa/btTvFNYALzpmq6Zn+Y=; b=gQUDlopoir/MoJdCojtN3m3QAHmNHFiH509G2YauRVEASVJkdH9EL6zhbxY+loRe1E b0+eWnVO6caezlLwZfR+qJ+aY6cs4Mc6O5JrDKYd2Jx+YtSPp3H9haQyTKrPXA2q2FZY 7thybzEQvPbGUDx2mN9pciYhroF1/qFTFSav/2KrXaNpydrzOX8jq3SmdeSMLq8oy4MO ykHlwFFHzDcQPDXmz2zA8BdXDRmAyssku9HYOc8bsZnITLbQcVidTsXUe+PXYo+DlAzH 5uha8yfRoCMi4YVqxR6Cs7HHWxMnCgj2Otzr5ZQuawGOELE9KRcxkBSNjE57Bvm0m9Ay DkwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093381; x=1780698181; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=SoQTly8/aFoEQsXn57jatIfa/btTvFNYALzpmq6Zn+Y=; b=SNecmUSLubQW5ChyfDG+LP2GeEThu6+B2UnDYmyxiDiMCt6pnagbo33x7H9dFX7kdd UW7yAmjUGM8OTBiRcDRtszmtfAcv3h3niuC8TYI1fQItMo2Cvy0YLHTww+s2RNoOJJws 7f9VWlrIBFM0QaIkJNGMAvc4ezZxd82P0QGrVtscEfzr3faSQ7T3Zuwshq8HhII7+96X 8RnfHapd33/+WY4xUy4OKUkYG3BgXjcJHHmRy2MwVRcFhnC3wimVMUmcuu5TjX8nJlST zno5/2IzA/euMiXtoJmzEh9zYXVr1CAOLgD+jsnbXeztgcfTVbs3AoHKzMe/VbiNG5O3 ACug== X-Forwarded-Encrypted: i=1; AFNElJ/VRXaXhXGZQ1LrchNvNv3xL39roThEC1m6Eh7pj67tbu4uipfaKNBYLujrTpNepUOfeMN+/PKRpjbRr5A=@vger.kernel.org X-Gm-Message-State: AOJu0Yw8l9tRm7TekJVF8q6HAViATUWd8r3lu53M1lC4u31M5fAVqYgn L5YMFPC9uwhaDB9l1dxM/Hib4tC3NGuyUNHTo8Emi4slmyexueKhiX754eLUPaitNhPqovsB3uB sV1MLrQ== X-Received: from pfbcz12.prod.google.com ([2002:aa7:930c:0:b0:82f:c134:e67c]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:aa7:9313:0:b0:82d:62ed:b01d with SMTP id d2e1a72fcca58-842255d2908mr1264758b3a.45.1780093380941; Fri, 29 May 2026 15:23:00 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:21:58 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-16-seanjc@google.com> Subject: [PATCH v3 15/40] KVM: x86: Move the bulk of register specific code from x86.c to regs.c From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce regs.c, and move the vast majority of register specific code out of x86.c and into regs.c. Deliberately leave behind MSR code (except for EFER, which can hardly be called an MSR), as KVM's MSR support is complex enough to warrant its own compilation unit, and doesn't have much in common with the other register code. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang --- arch/x86/include/asm/kvm_host.h | 2 - arch/x86/kvm/Makefile | 4 +- arch/x86/kvm/regs.c | 875 +++++++++++++++++++++++++++++++ arch/x86/kvm/regs.h | 31 ++ arch/x86/kvm/x86.c | 885 +------------------------------- arch/x86/kvm/x86.h | 2 + 6 files changed, 918 insertions(+), 881 deletions(-) create mode 100644 arch/x86/kvm/regs.c diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 6ae7d539af90..983bdc84f9f9 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2332,8 +2332,6 @@ static inline int __kvm_irq_line_state(unsigned long = *irq_state, void kvm_inject_nmi(struct kvm_vcpu *vcpu); int kvm_get_nr_pending_nmis(struct kvm_vcpu *vcpu); =20 -void kvm_update_dr7(struct kvm_vcpu *vcpu); - bool __kvm_mmu_unprotect_gfn_and_retry(struct kvm_vcpu *vcpu, gpa_t cr2_or= _gpa, bool always_retry); =20 diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile index 77337c37324b..f39c311fd756 100644 --- a/arch/x86/kvm/Makefile +++ b/arch/x86/kvm/Makefile @@ -5,8 +5,8 @@ ccflags-$(CONFIG_KVM_WERROR) +=3D -Werror =20 include $(srctree)/virt/kvm/Makefile.kvm =20 -kvm-y +=3D x86.o emulate.o irq.o lapic.o cpuid.o pmu.o mtrr.o \ - debugfs.o mmu/mmu.o mmu/page_track.o mmu/spte.o +kvm-y +=3D x86.o emulate.o irq.o lapic.o cpuid.o pmu.o regs.o \ + mtrr.o debugfs.o mmu/mmu.o mmu/page_track.o mmu/spte.o =20 kvm-$(CONFIG_X86_64) +=3D mmu/tdp_iter.o mmu/tdp_mmu.o kvm-$(CONFIG_KVM_IOAPIC) +=3D i8259.o i8254.o ioapic.o diff --git a/arch/x86/kvm/regs.c b/arch/x86/kvm/regs.c new file mode 100644 index 000000000000..fb4478301076 --- /dev/null +++ b/arch/x86/kvm/regs.c @@ -0,0 +1,875 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include + +#include "lapic.h" +#include "mmu.h" +#include "regs.h" +#include "x86.h" + +unsigned long kvm_get_linear_rip(struct kvm_vcpu *vcpu) +{ + /* Can't read the RIP when guest state is protected, just return 0 */ + if (vcpu->arch.guest_state_protected) + return 0; + + if (is_64_bit_mode(vcpu)) + return kvm_rip_read(vcpu); + return (u32)(kvm_get_segment_base(vcpu, VCPU_SREG_CS) + + kvm_rip_read(vcpu)); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_linear_rip); + +bool kvm_is_linear_rip(struct kvm_vcpu *vcpu, unsigned long linear_rip) +{ + return kvm_get_linear_rip(vcpu) =3D=3D linear_rip; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_is_linear_rip); + +unsigned long kvm_get_rflags(struct kvm_vcpu *vcpu) +{ + unsigned long rflags; + + rflags =3D kvm_x86_call(get_rflags)(vcpu); + if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP) + rflags &=3D ~X86_EFLAGS_TF; + return rflags; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_rflags); + +void __kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags) +{ + if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP && + kvm_is_linear_rip(vcpu, vcpu->arch.singlestep_rip)) + rflags |=3D X86_EFLAGS_TF; + kvm_x86_call(set_rflags)(vcpu, rflags); +} + +void kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags) +{ + __kvm_set_rflags(vcpu, rflags); + kvm_make_request(KVM_REQ_EVENT, vcpu); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_rflags); + +static void __get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) +{ + if (vcpu->arch.emulate_regs_need_sync_to_vcpu) { + /* + * We are here if userspace calls get_regs() in the middle of + * instruction emulation. Registers state needs to be copied + * back from emulation context to vcpu. Userspace shouldn't do + * that usually, but some bad designed PV devices (vmware + * backdoor interface) need this to work + */ + emulator_writeback_register_cache(vcpu->arch.emulate_ctxt); + vcpu->arch.emulate_regs_need_sync_to_vcpu =3D false; + } + regs->rax =3D kvm_rax_read_raw(vcpu); + regs->rbx =3D kvm_rbx_read_raw(vcpu); + regs->rcx =3D kvm_rcx_read_raw(vcpu); + regs->rdx =3D kvm_rdx_read_raw(vcpu); + regs->rsi =3D kvm_rsi_read_raw(vcpu); + regs->rdi =3D kvm_rdi_read_raw(vcpu); + regs->rsp =3D kvm_rsp_read(vcpu); + regs->rbp =3D kvm_rbp_read_raw(vcpu); +#ifdef CONFIG_X86_64 + regs->r8 =3D kvm_r8_read_raw(vcpu); + regs->r9 =3D kvm_r9_read_raw(vcpu); + regs->r10 =3D kvm_r10_read_raw(vcpu); + regs->r11 =3D kvm_r11_read_raw(vcpu); + regs->r12 =3D kvm_r12_read_raw(vcpu); + regs->r13 =3D kvm_r13_read_raw(vcpu); + regs->r14 =3D kvm_r14_read_raw(vcpu); + regs->r15 =3D kvm_r15_read_raw(vcpu); +#endif + + regs->rip =3D kvm_rip_read(vcpu); + regs->rflags =3D kvm_get_rflags(vcpu); +} + +int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *r= egs) +{ + if (vcpu->kvm->arch.has_protected_state && + vcpu->arch.guest_state_protected) + return -EINVAL; + + vcpu_load(vcpu); + __get_regs(vcpu, regs); + vcpu_put(vcpu); + return 0; +} + +static void __set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) +{ + vcpu->arch.emulate_regs_need_sync_from_vcpu =3D true; + vcpu->arch.emulate_regs_need_sync_to_vcpu =3D false; + + kvm_rax_write_raw(vcpu, regs->rax); + kvm_rbx_write_raw(vcpu, regs->rbx); + kvm_rcx_write_raw(vcpu, regs->rcx); + kvm_rdx_write_raw(vcpu, regs->rdx); + kvm_rsi_write_raw(vcpu, regs->rsi); + kvm_rdi_write_raw(vcpu, regs->rdi); + kvm_rsp_write(vcpu, regs->rsp); + kvm_rbp_write_raw(vcpu, regs->rbp); +#ifdef CONFIG_X86_64 + kvm_r8_write_raw(vcpu, regs->r8); + kvm_r9_write_raw(vcpu, regs->r9); + kvm_r10_write_raw(vcpu, regs->r10); + kvm_r11_write_raw(vcpu, regs->r11); + kvm_r12_write_raw(vcpu, regs->r12); + kvm_r13_write_raw(vcpu, regs->r13); + kvm_r14_write_raw(vcpu, regs->r14); + kvm_r15_write_raw(vcpu, regs->r15); +#endif + + kvm_rip_write(vcpu, regs->rip); + kvm_set_rflags(vcpu, regs->rflags | X86_EFLAGS_FIXED); + + vcpu->arch.exception.pending =3D false; + vcpu->arch.exception_vmexit.pending =3D false; + + kvm_make_request(KVM_REQ_EVENT, vcpu); +} + +int kvm_arch_vcpu_ioctl_set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *r= egs) +{ + if (vcpu->kvm->arch.has_protected_state && + vcpu->arch.guest_state_protected) + return -EINVAL; + + vcpu_load(vcpu); + __set_regs(vcpu, regs); + vcpu_put(vcpu); + return 0; +} + +static inline u64 pdptr_rsvd_bits(struct kvm_vcpu *vcpu) +{ + return vcpu->arch.reserved_gpa_bits | rsvd_bits(5, 8) | rsvd_bits(1, 2); +} + +/* + * Load the pae pdptrs. Return 1 if they are all valid, 0 otherwise. + */ +int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3) +{ + struct kvm_mmu *mmu =3D vcpu->arch.walk_mmu; + gfn_t pdpt_gfn =3D cr3 >> PAGE_SHIFT; + gpa_t real_gpa; + int i; + int ret; + u64 pdpte[ARRAY_SIZE(mmu->pdptrs)]; + + /* + * If the MMU is nested, CR3 holds an L2 GPA and needs to be translated + * to an L1 GPA. + */ + real_gpa =3D kvm_translate_gpa(vcpu, mmu, gfn_to_gpa(pdpt_gfn), + PFERR_USER_MASK | PFERR_WRITE_MASK | + PFERR_GUEST_PAGE_MASK, NULL, 0); + if (real_gpa =3D=3D INVALID_GPA) + return 0; + + /* Note the offset, PDPTRs are 32 byte aligned when using PAE paging. */ + ret =3D kvm_vcpu_read_guest_page(vcpu, gpa_to_gfn(real_gpa), pdpte, + cr3 & GENMASK(11, 5), sizeof(pdpte)); + if (ret < 0) + return 0; + + for (i =3D 0; i < ARRAY_SIZE(pdpte); ++i) { + if ((pdpte[i] & PT_PRESENT_MASK) && + (pdpte[i] & pdptr_rsvd_bits(vcpu))) { + return 0; + } + } + + /* + * Marking VCPU_REG_PDPTR dirty doesn't work for !tdp_enabled. + * Shadow page roots need to be reconstructed instead. + */ + if (!tdp_enabled && memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs))) + kvm_mmu_free_roots(vcpu->kvm, mmu, KVM_MMU_ROOT_CURRENT); + + memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs)); + kvm_register_mark_dirty(vcpu, VCPU_REG_PDPTR); + kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu); + vcpu->arch.pdptrs_from_userspace =3D false; + + return 1; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(load_pdptrs); + +static bool kvm_is_valid_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) +{ +#ifdef CONFIG_X86_64 + if (cr0 & 0xffffffff00000000UL) + return false; +#endif + + if ((cr0 & X86_CR0_NW) && !(cr0 & X86_CR0_CD)) + return false; + + if ((cr0 & X86_CR0_PG) && !(cr0 & X86_CR0_PE)) + return false; + + return kvm_x86_call(is_valid_cr0)(vcpu, cr0); +} + +void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsign= ed long cr0) +{ + /* + * CR0.WP is incorporated into the MMU role, but only for non-nested, + * indirect shadow MMUs. If paging is disabled, no updates are needed + * as there are no permission bits to emulate. If TDP is enabled, the + * MMU's metadata needs to be updated, e.g. so that emulating guest + * translations does the right thing, but there's no need to unload the + * root as CR0.WP doesn't affect SPTEs. + */ + if ((cr0 ^ old_cr0) =3D=3D X86_CR0_WP) { + if (!(cr0 & X86_CR0_PG)) + return; + + if (tdp_enabled) { + kvm_init_mmu(vcpu); + return; + } + } + + if ((cr0 ^ old_cr0) & X86_CR0_PG) { + /* + * Clearing CR0.PG is defined to flush the TLB from the guest's + * perspective. + */ + if (!(cr0 & X86_CR0_PG)) + kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); + /* + * Check for async #PF completion events when enabling paging, + * as the vCPU may have previously encountered async #PFs (it's + * entirely legal for the guest to toggle paging on/off without + * waiting for the async #PF queue to drain). + */ + else if (kvm_pv_async_pf_enabled(vcpu)) + kvm_make_request(KVM_REQ_APF_READY, vcpu); + } + + if ((cr0 ^ old_cr0) & KVM_MMU_CR0_ROLE_BITS) + kvm_mmu_reset_context(vcpu); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_post_set_cr0); + +int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) +{ + unsigned long old_cr0 =3D kvm_read_cr0(vcpu); + + if (!kvm_is_valid_cr0(vcpu, cr0)) + return 1; + + cr0 |=3D X86_CR0_ET; + + /* Write to CR0 reserved bits are ignored, even on Intel. */ + cr0 &=3D ~CR0_RESERVED_BITS; + +#ifdef CONFIG_X86_64 + if ((vcpu->arch.efer & EFER_LME) && !is_paging(vcpu) && + (cr0 & X86_CR0_PG)) { + int cs_db, cs_l; + + if (!is_pae(vcpu)) + return 1; + kvm_x86_call(get_cs_db_l_bits)(vcpu, &cs_db, &cs_l); + if (cs_l) + return 1; + } +#endif + if (!(vcpu->arch.efer & EFER_LME) && (cr0 & X86_CR0_PG) && + is_pae(vcpu) && ((cr0 ^ old_cr0) & X86_CR0_PDPTR_BITS) && + !load_pdptrs(vcpu, kvm_read_cr3(vcpu))) + return 1; + + if (!(cr0 & X86_CR0_PG) && + (is_64_bit_mode(vcpu) || kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE))) + return 1; + + if (!(cr0 & X86_CR0_WP) && kvm_is_cr4_bit_set(vcpu, X86_CR4_CET)) + return 1; + + kvm_x86_call(set_cr0)(vcpu, cr0); + + kvm_post_set_cr0(vcpu, old_cr0, cr0); + + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr0); + +void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw) +{ + (void)kvm_set_cr0(vcpu, kvm_read_cr0_bits(vcpu, ~0x0eul) | (msw & 0x0f)); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_lmsw); + +int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3) +{ + bool skip_tlb_flush =3D false; + unsigned long pcid =3D 0; +#ifdef CONFIG_X86_64 + if (kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE)) { + skip_tlb_flush =3D cr3 & X86_CR3_PCID_NOFLUSH; + cr3 &=3D ~X86_CR3_PCID_NOFLUSH; + pcid =3D cr3 & X86_CR3_PCID_MASK; + } +#endif + + /* PDPTRs are always reloaded for PAE paging. */ + if (cr3 =3D=3D kvm_read_cr3(vcpu) && !is_pae_paging(vcpu)) + goto handle_tlb_flush; + + /* + * Do not condition the GPA check on long mode, this helper is used to + * stuff CR3, e.g. for RSM emulation, and there is no guarantee that + * the current vCPU mode is accurate. + */ + if (!kvm_vcpu_is_legal_cr3(vcpu, cr3)) + return 1; + + if (is_pae_paging(vcpu) && !load_pdptrs(vcpu, cr3)) + return 1; + + if (cr3 !=3D kvm_read_cr3(vcpu)) + kvm_mmu_new_pgd(vcpu, cr3); + + vcpu->arch.cr3 =3D cr3; + kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); + /* Do not call post_set_cr3, we do not get here for confidential guests. = */ + +handle_tlb_flush: + /* + * A load of CR3 that flushes the TLB flushes only the current PCID, + * even if PCID is disabled, in which case PCID=3D0 is flushed. It's a + * moot point in the end because _disabling_ PCID will flush all PCIDs, + * and it's impossible to use a non-zero PCID when PCID is disabled, + * i.e. only PCID=3D0 can be relevant. + */ + if (!skip_tlb_flush) + kvm_invalidate_pcid(vcpu, pcid); + + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr3); + +static bool kvm_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) +{ + return __kvm_is_valid_cr4(vcpu, cr4) && + kvm_x86_call(is_valid_cr4)(vcpu, cr4); +} + +void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsign= ed long cr4) +{ + if ((cr4 ^ old_cr4) & KVM_MMU_CR4_ROLE_BITS) + kvm_mmu_reset_context(vcpu); + + /* + * If CR4.PCIDE is changed 0 -> 1, there is no need to flush the TLB + * according to the SDM; however, stale prev_roots could be reused + * incorrectly in the future after a MOV to CR3 with NOFLUSH=3D1, so we + * free them all. This is *not* a superset of KVM_REQ_TLB_FLUSH_GUEST + * or KVM_REQ_TLB_FLUSH_CURRENT, because the hardware TLB is not flushed, + * so fall through. + */ + if (!tdp_enabled && + (cr4 & X86_CR4_PCIDE) && !(old_cr4 & X86_CR4_PCIDE)) + kvm_mmu_unload(vcpu); + + /* + * The TLB has to be flushed for all PCIDs if any of the following + * (architecturally required) changes happen: + * - CR4.PCIDE is changed from 1 to 0 + * - CR4.PGE is toggled + * + * This is a superset of KVM_REQ_TLB_FLUSH_CURRENT. + */ + if (((cr4 ^ old_cr4) & X86_CR4_PGE) || + (!(cr4 & X86_CR4_PCIDE) && (old_cr4 & X86_CR4_PCIDE))) + kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); + + /* + * The TLB has to be flushed for the current PCID if any of the + * following (architecturally required) changes happen: + * - CR4.SMEP is changed from 0 to 1 + * - CR4.PAE is toggled + */ + else if (((cr4 ^ old_cr4) & X86_CR4_PAE) || + ((cr4 & X86_CR4_SMEP) && !(old_cr4 & X86_CR4_SMEP))) + kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu); + +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_post_set_cr4); + +int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) +{ + unsigned long old_cr4 =3D kvm_read_cr4(vcpu); + + if (!kvm_is_valid_cr4(vcpu, cr4)) + return 1; + + if (is_long_mode(vcpu)) { + if (!(cr4 & X86_CR4_PAE)) + return 1; + if ((cr4 ^ old_cr4) & X86_CR4_LA57) + return 1; + } else if (is_paging(vcpu) && (cr4 & X86_CR4_PAE) + && ((cr4 ^ old_cr4) & X86_CR4_PDPTR_BITS) + && !load_pdptrs(vcpu, kvm_read_cr3(vcpu))) + return 1; + + if ((cr4 & X86_CR4_PCIDE) && !(old_cr4 & X86_CR4_PCIDE)) { + /* PCID can not be enabled when cr3[11:0]!=3D000H or EFER.LMA=3D0 */ + if ((kvm_read_cr3(vcpu) & X86_CR3_PCID_MASK) || !is_long_mode(vcpu)) + return 1; + } + + if ((cr4 & X86_CR4_CET) && !kvm_is_cr0_bit_set(vcpu, X86_CR0_WP)) + return 1; + + kvm_x86_call(set_cr4)(vcpu, cr4); + + kvm_post_set_cr4(vcpu, old_cr4, cr4); + + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr4); + +int kvm_set_cr8(struct kvm_vcpu *vcpu, unsigned long cr8) +{ + if (cr8 & CR8_RESERVED_BITS) + return 1; + if (lapic_in_kernel(vcpu)) + kvm_lapic_set_tpr(vcpu, cr8); + else + vcpu->arch.cr8 =3D cr8; + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr8); + +unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu) +{ + if (lapic_in_kernel(vcpu)) + return kvm_lapic_get_cr8(vcpu); + else + return vcpu->arch.cr8; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_cr8); + +static void __get_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sr= egs) +{ + struct desc_ptr dt; + + if (vcpu->arch.guest_state_protected) + goto skip_protected_regs; + + kvm_handle_exception_payload_quirk(vcpu); + + kvm_get_segment(vcpu, &sregs->cs, VCPU_SREG_CS); + kvm_get_segment(vcpu, &sregs->ds, VCPU_SREG_DS); + kvm_get_segment(vcpu, &sregs->es, VCPU_SREG_ES); + kvm_get_segment(vcpu, &sregs->fs, VCPU_SREG_FS); + kvm_get_segment(vcpu, &sregs->gs, VCPU_SREG_GS); + kvm_get_segment(vcpu, &sregs->ss, VCPU_SREG_SS); + + kvm_get_segment(vcpu, &sregs->tr, VCPU_SREG_TR); + kvm_get_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR); + + kvm_x86_call(get_idt)(vcpu, &dt); + sregs->idt.limit =3D dt.size; + sregs->idt.base =3D dt.address; + kvm_x86_call(get_gdt)(vcpu, &dt); + sregs->gdt.limit =3D dt.size; + sregs->gdt.base =3D dt.address; + + sregs->cr2 =3D vcpu->arch.cr2; + sregs->cr3 =3D kvm_read_cr3(vcpu); + +skip_protected_regs: + sregs->cr0 =3D kvm_read_cr0(vcpu); + sregs->cr4 =3D kvm_read_cr4(vcpu); + sregs->cr8 =3D kvm_get_cr8(vcpu); + sregs->efer =3D vcpu->arch.efer; + sregs->apic_base =3D vcpu->arch.apic_base; +} + +static void __get_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) +{ + __get_sregs_common(vcpu, sregs); + + if (vcpu->arch.guest_state_protected) + return; + + if (vcpu->arch.interrupt.injected && !vcpu->arch.interrupt.soft) + set_bit(vcpu->arch.interrupt.nr, + (unsigned long *)sregs->interrupt_bitmap); +} + +int kvm_arch_vcpu_ioctl_get_sregs(struct kvm_vcpu *vcpu, + struct kvm_sregs *sregs) +{ + if (vcpu->kvm->arch.has_protected_state && + vcpu->arch.guest_state_protected) + return -EINVAL; + + vcpu_load(vcpu); + __get_sregs(vcpu, sregs); + vcpu_put(vcpu); + return 0; +} + +void kvm_x86_vcpu_ioctl_get_sregs2(struct kvm_vcpu *vcpu, + struct kvm_sregs2 *sregs2) +{ + int i; + + __get_sregs_common(vcpu, (struct kvm_sregs *)sregs2); + + if (vcpu->arch.guest_state_protected) + return; + + if (is_pae_paging(vcpu)) { + kvm_vcpu_srcu_read_lock(vcpu); + for (i =3D 0 ; i < 4 ; i++) + sregs2->pdptrs[i] =3D kvm_pdptr_read(vcpu, i); + sregs2->flags |=3D KVM_SREGS2_FLAGS_PDPTRS_VALID; + kvm_vcpu_srcu_read_unlock(vcpu); + } +} + +static bool kvm_is_valid_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sr= egs) +{ + if ((sregs->efer & EFER_LME) && (sregs->cr0 & X86_CR0_PG)) { + /* + * When EFER.LME and CR0.PG are set, the processor is in + * 64-bit mode (though maybe in a 32-bit code segment). + * CR4.PAE and EFER.LMA must be set. + */ + if (!(sregs->cr4 & X86_CR4_PAE) || !(sregs->efer & EFER_LMA)) + return false; + if (!kvm_vcpu_is_legal_cr3(vcpu, sregs->cr3)) + return false; + } else { + /* + * Not in 64-bit mode: EFER.LMA is clear and the code + * segment cannot be 64-bit. + */ + if (sregs->efer & EFER_LMA || sregs->cs.l) + return false; + } + + return kvm_is_valid_cr4(vcpu, sregs->cr4) && + kvm_is_valid_cr0(vcpu, sregs->cr0); +} + +static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sre= gs, + int *mmu_reset_needed, bool update_pdptrs) +{ + int idx; + struct desc_ptr dt; + + if (!kvm_is_valid_sregs(vcpu, sregs)) + return -EINVAL; + + if (kvm_apic_set_base(vcpu, sregs->apic_base, true)) + return -EINVAL; + + if (vcpu->arch.guest_state_protected) + return 0; + + dt.size =3D sregs->idt.limit; + dt.address =3D sregs->idt.base; + kvm_x86_call(set_idt)(vcpu, &dt); + dt.size =3D sregs->gdt.limit; + dt.address =3D sregs->gdt.base; + kvm_x86_call(set_gdt)(vcpu, &dt); + + vcpu->arch.cr2 =3D sregs->cr2; + *mmu_reset_needed |=3D kvm_read_cr3(vcpu) !=3D sregs->cr3; + vcpu->arch.cr3 =3D sregs->cr3; + kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); + kvm_x86_call(post_set_cr3)(vcpu, sregs->cr3); + + kvm_set_cr8(vcpu, sregs->cr8); + + *mmu_reset_needed |=3D vcpu->arch.efer !=3D sregs->efer; + kvm_x86_call(set_efer)(vcpu, sregs->efer); + + *mmu_reset_needed |=3D kvm_read_cr0(vcpu) !=3D sregs->cr0; + kvm_x86_call(set_cr0)(vcpu, sregs->cr0); + + *mmu_reset_needed |=3D kvm_read_cr4(vcpu) !=3D sregs->cr4; + kvm_x86_call(set_cr4)(vcpu, sregs->cr4); + + if (update_pdptrs) { + idx =3D srcu_read_lock(&vcpu->kvm->srcu); + if (is_pae_paging(vcpu)) { + load_pdptrs(vcpu, kvm_read_cr3(vcpu)); + *mmu_reset_needed =3D 1; + } + srcu_read_unlock(&vcpu->kvm->srcu, idx); + } + + kvm_set_segment(vcpu, &sregs->cs, VCPU_SREG_CS); + kvm_set_segment(vcpu, &sregs->ds, VCPU_SREG_DS); + kvm_set_segment(vcpu, &sregs->es, VCPU_SREG_ES); + kvm_set_segment(vcpu, &sregs->fs, VCPU_SREG_FS); + kvm_set_segment(vcpu, &sregs->gs, VCPU_SREG_GS); + kvm_set_segment(vcpu, &sregs->ss, VCPU_SREG_SS); + + kvm_set_segment(vcpu, &sregs->tr, VCPU_SREG_TR); + kvm_set_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR); + + kvm_lapic_update_cr8_intercept(vcpu); + + /* Older userspace won't unhalt the vcpu on reset. */ + if (kvm_vcpu_is_bsp(vcpu) && kvm_rip_read(vcpu) =3D=3D 0xfff0 && + sregs->cs.selector =3D=3D 0xf000 && sregs->cs.base =3D=3D 0xffff0000 = && + !is_protmode(vcpu)) + kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE); + + return 0; +} + +static int __set_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) +{ + int pending_vec, max_bits; + int mmu_reset_needed =3D 0; + int ret =3D __set_sregs_common(vcpu, sregs, &mmu_reset_needed, true); + + if (ret) + return ret; + + if (mmu_reset_needed) { + kvm_mmu_reset_context(vcpu); + kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); + } + + max_bits =3D KVM_NR_INTERRUPTS; + pending_vec =3D find_first_bit( + (const unsigned long *)sregs->interrupt_bitmap, max_bits); + + if (pending_vec < max_bits) { + kvm_queue_interrupt(vcpu, pending_vec, false); + pr_debug("Set back pending irq %d\n", pending_vec); + kvm_make_request(KVM_REQ_EVENT, vcpu); + } + return 0; +} + +int kvm_arch_vcpu_ioctl_set_sregs(struct kvm_vcpu *vcpu, + struct kvm_sregs *sregs) +{ + int ret; + + if (vcpu->kvm->arch.has_protected_state && + vcpu->arch.guest_state_protected) + return -EINVAL; + + vcpu_load(vcpu); + ret =3D __set_sregs(vcpu, sregs); + vcpu_put(vcpu); + return ret; +} + +int kvm_x86_vcpu_ioctl_set_sregs2(struct kvm_vcpu *vcpu, + struct kvm_sregs2 *sregs2) +{ + int mmu_reset_needed =3D 0; + bool valid_pdptrs =3D sregs2->flags & KVM_SREGS2_FLAGS_PDPTRS_VALID; + bool pae =3D (sregs2->cr0 & X86_CR0_PG) && (sregs2->cr4 & X86_CR4_PAE) && + !(sregs2->efer & EFER_LMA); + int i, ret; + + if (sregs2->flags & ~KVM_SREGS2_FLAGS_PDPTRS_VALID) + return -EINVAL; + + if (valid_pdptrs && (!pae || vcpu->arch.guest_state_protected)) + return -EINVAL; + + ret =3D __set_sregs_common(vcpu, (struct kvm_sregs *)sregs2, + &mmu_reset_needed, !valid_pdptrs); + if (ret) + return ret; + + if (valid_pdptrs) { + for (i =3D 0; i < 4 ; i++) + kvm_pdptr_write(vcpu, i, sregs2->pdptrs[i]); + + kvm_register_mark_dirty(vcpu, VCPU_REG_PDPTR); + mmu_reset_needed =3D 1; + vcpu->arch.pdptrs_from_userspace =3D true; + } + if (mmu_reset_needed) { + kvm_mmu_reset_context(vcpu); + kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); + } + return 0; +} + +void kvm_run_sync_regs_to_user(struct kvm_vcpu *vcpu) +{ + BUILD_BUG_ON(sizeof(struct kvm_sync_regs) > SYNC_REGS_SIZE_BYTES); + + if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_REGS) + __get_regs(vcpu, &vcpu->run->s.regs.regs); + + if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_SREGS) + __get_sregs(vcpu, &vcpu->run->s.regs.sregs); +} + +int kvm_run_sync_regs_from_user(struct kvm_vcpu *vcpu) +{ + if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_REGS) { + __set_regs(vcpu, &vcpu->run->s.regs.regs); + vcpu->run->kvm_dirty_regs &=3D ~KVM_SYNC_X86_REGS; + } + + if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_SREGS) { + struct kvm_sregs sregs =3D vcpu->run->s.regs.sregs; + + if (__set_sregs(vcpu, &sregs)) + return -EINVAL; + + vcpu->run->kvm_dirty_regs &=3D ~KVM_SYNC_X86_SREGS; + } + + return 0; +} + +void kvm_update_dr0123(struct kvm_vcpu *vcpu) +{ + int i; + + if (!(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP)) { + for (i =3D 0; i < KVM_NR_DB_REGS; i++) + vcpu->arch.eff_db[i] =3D vcpu->arch.db[i]; + } +} + +void kvm_update_dr7(struct kvm_vcpu *vcpu) +{ + unsigned long dr7; + + if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) + dr7 =3D vcpu->arch.guest_debug_dr7; + else + dr7 =3D vcpu->arch.dr7; + kvm_x86_call(set_dr7)(vcpu, dr7); + vcpu->arch.switch_db_regs &=3D ~KVM_DEBUGREG_BP_ENABLED; + if (dr7 & DR7_BP_EN_MASK) + vcpu->arch.switch_db_regs |=3D KVM_DEBUGREG_BP_ENABLED; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_update_dr7); + +static u64 kvm_dr6_fixed(struct kvm_vcpu *vcpu) +{ + u64 fixed =3D DR6_FIXED_1; + + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_RTM)) + fixed |=3D DR6_RTM; + + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_BUS_LOCK_DETECT)) + fixed |=3D DR6_BUS_LOCK; + return fixed; +} + +int kvm_set_dr(struct kvm_vcpu *vcpu, int dr, unsigned long val) +{ + size_t size =3D ARRAY_SIZE(vcpu->arch.db); + + switch (dr) { + case 0 ... 3: + vcpu->arch.db[array_index_nospec(dr, size)] =3D val; + if (!(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP)) + vcpu->arch.eff_db[dr] =3D val; + break; + case 4: + case 6: + if (!kvm_dr6_valid(val)) + return 1; /* #GP */ + vcpu->arch.dr6 =3D (val & DR6_VOLATILE) | kvm_dr6_fixed(vcpu); + break; + case 5: + default: /* 7 */ + if (!kvm_dr7_valid(val)) + return 1; /* #GP */ + vcpu->arch.dr7 =3D (val & DR7_VOLATILE) | DR7_FIXED_1; + kvm_update_dr7(vcpu); + break; + } + + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_dr); + +unsigned long kvm_get_dr(struct kvm_vcpu *vcpu, int dr) +{ + size_t size =3D ARRAY_SIZE(vcpu->arch.db); + + switch (dr) { + case 0 ... 3: + return vcpu->arch.db[array_index_nospec(dr, size)]; + case 4: + case 6: + return vcpu->arch.dr6; + case 5: + default: /* 7 */ + return vcpu->arch.dr7; + } +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_dr); + +int kvm_vcpu_ioctl_x86_get_debugregs(struct kvm_vcpu *vcpu, + struct kvm_debugregs *dbgregs) +{ + unsigned int i; + + if (vcpu->kvm->arch.has_protected_state && + vcpu->arch.guest_state_protected) + return -EINVAL; + + kvm_handle_exception_payload_quirk(vcpu); + + memset(dbgregs, 0, sizeof(*dbgregs)); + + BUILD_BUG_ON(ARRAY_SIZE(vcpu->arch.db) !=3D ARRAY_SIZE(dbgregs->db)); + for (i =3D 0; i < ARRAY_SIZE(vcpu->arch.db); i++) + dbgregs->db[i] =3D vcpu->arch.db[i]; + + dbgregs->dr6 =3D vcpu->arch.dr6; + dbgregs->dr7 =3D vcpu->arch.dr7; + return 0; +} + +int kvm_vcpu_ioctl_x86_set_debugregs(struct kvm_vcpu *vcpu, + struct kvm_debugregs *dbgregs) +{ + unsigned int i; + + if (vcpu->kvm->arch.has_protected_state && + vcpu->arch.guest_state_protected) + return -EINVAL; + + if (dbgregs->flags) + return -EINVAL; + + if (!kvm_dr6_valid(dbgregs->dr6)) + return -EINVAL; + if (!kvm_dr7_valid(dbgregs->dr7)) + return -EINVAL; + + for (i =3D 0; i < ARRAY_SIZE(vcpu->arch.db); i++) + vcpu->arch.db[i] =3D dbgregs->db[i]; + + kvm_update_dr0123(vcpu); + vcpu->arch.dr6 =3D dbgregs->dr6; + vcpu->arch.dr7 =3D dbgregs->dr7; + kvm_update_dr7(vcpu); + + return 0; +} diff --git a/arch/x86/kvm/regs.h b/arch/x86/kvm/regs.h index a57ba26279ed..c224874bbdde 100644 --- a/arch/x86/kvm/regs.h +++ b/arch/x86/kvm/regs.h @@ -389,6 +389,14 @@ static inline bool kvm_dr6_valid(u64 data) return !(data >> 32); } =20 +static inline unsigned long kvm_get_effective_dr7(struct kvm_vcpu *vcpu) +{ + if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) + return vcpu->arch.guest_debug_dr7; + + return vcpu->arch.dr7; +} + static inline void enter_guest_mode(struct kvm_vcpu *vcpu) { vcpu->arch.hflags |=3D HF_GUEST_MASK; @@ -412,4 +420,27 @@ static inline bool is_guest_mode(struct kvm_vcpu *vcpu) return vcpu->arch.hflags & HF_GUEST_MASK; } =20 +static inline unsigned long kvm_get_segment_base(struct kvm_vcpu *vcpu, in= t seg) +{ + return kvm_x86_call(get_segment_base)(vcpu, seg); +} + +void __kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags); + +void kvm_x86_vcpu_ioctl_get_sregs2(struct kvm_vcpu *vcpu, + struct kvm_sregs2 *sregs2); +int kvm_x86_vcpu_ioctl_set_sregs2(struct kvm_vcpu *vcpu, + struct kvm_sregs2 *sregs2); + +void kvm_run_sync_regs_to_user(struct kvm_vcpu *vcpu); +int kvm_run_sync_regs_from_user(struct kvm_vcpu *vcpu); + +void kvm_update_dr0123(struct kvm_vcpu *vcpu); +void kvm_update_dr7(struct kvm_vcpu *vcpu); +int kvm_vcpu_ioctl_x86_get_debugregs(struct kvm_vcpu *vcpu, + struct kvm_debugregs *dbgregs); +int kvm_vcpu_ioctl_x86_set_debugregs(struct kvm_vcpu *vcpu, + struct kvm_debugregs *dbgregs); + + #endif diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index cd68a5bad0c6..20eeff79b46d 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -129,13 +129,9 @@ static u64 __read_mostly efer_reserved_bits =3D ~((u64= )EFER_SCE); KVM_X2APIC_DISABLE_SUPPRESS_EOI_BROADCAST) =20 static void process_nmi(struct kvm_vcpu *vcpu); -static void __kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags); static void store_regs(struct kvm_vcpu *vcpu); static int sync_regs(struct kvm_vcpu *vcpu); =20 -static int __set_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2); -static void __get_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2); - static DEFINE_MUTEX(vendor_module_lock); static void kvm_load_guest_fpu(struct kvm_vcpu *vcpu); static void kvm_put_guest_fpu(struct kvm_vcpu *vcpu); @@ -1016,170 +1012,6 @@ bool kvm_require_dr(struct kvm_vcpu *vcpu, int dr) } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_require_dr); =20 -static inline u64 pdptr_rsvd_bits(struct kvm_vcpu *vcpu) -{ - return vcpu->arch.reserved_gpa_bits | rsvd_bits(5, 8) | rsvd_bits(1, 2); -} - -/* - * Load the pae pdptrs. Return 1 if they are all valid, 0 otherwise. - */ -int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3) -{ - struct kvm_mmu *mmu =3D vcpu->arch.walk_mmu; - gfn_t pdpt_gfn =3D cr3 >> PAGE_SHIFT; - gpa_t real_gpa; - int i; - int ret; - u64 pdpte[ARRAY_SIZE(mmu->pdptrs)]; - - /* - * If the MMU is nested, CR3 holds an L2 GPA and needs to be translated - * to an L1 GPA. - */ - real_gpa =3D kvm_translate_gpa(vcpu, mmu, gfn_to_gpa(pdpt_gfn), - PFERR_USER_MASK | PFERR_WRITE_MASK | - PFERR_GUEST_PAGE_MASK, NULL, 0); - if (real_gpa =3D=3D INVALID_GPA) - return 0; - - /* Note the offset, PDPTRs are 32 byte aligned when using PAE paging. */ - ret =3D kvm_vcpu_read_guest_page(vcpu, gpa_to_gfn(real_gpa), pdpte, - cr3 & GENMASK(11, 5), sizeof(pdpte)); - if (ret < 0) - return 0; - - for (i =3D 0; i < ARRAY_SIZE(pdpte); ++i) { - if ((pdpte[i] & PT_PRESENT_MASK) && - (pdpte[i] & pdptr_rsvd_bits(vcpu))) { - return 0; - } - } - - /* - * Marking VCPU_REG_PDPTR dirty doesn't work for !tdp_enabled. - * Shadow page roots need to be reconstructed instead. - */ - if (!tdp_enabled && memcmp(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs))) - kvm_mmu_free_roots(vcpu->kvm, mmu, KVM_MMU_ROOT_CURRENT); - - memcpy(mmu->pdptrs, pdpte, sizeof(mmu->pdptrs)); - kvm_register_mark_dirty(vcpu, VCPU_REG_PDPTR); - kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu); - vcpu->arch.pdptrs_from_userspace =3D false; - - return 1; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(load_pdptrs); - -static bool kvm_is_valid_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) -{ -#ifdef CONFIG_X86_64 - if (cr0 & 0xffffffff00000000UL) - return false; -#endif - - if ((cr0 & X86_CR0_NW) && !(cr0 & X86_CR0_CD)) - return false; - - if ((cr0 & X86_CR0_PG) && !(cr0 & X86_CR0_PE)) - return false; - - return kvm_x86_call(is_valid_cr0)(vcpu, cr0); -} - -void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsign= ed long cr0) -{ - /* - * CR0.WP is incorporated into the MMU role, but only for non-nested, - * indirect shadow MMUs. If paging is disabled, no updates are needed - * as there are no permission bits to emulate. If TDP is enabled, the - * MMU's metadata needs to be updated, e.g. so that emulating guest - * translations does the right thing, but there's no need to unload the - * root as CR0.WP doesn't affect SPTEs. - */ - if ((cr0 ^ old_cr0) =3D=3D X86_CR0_WP) { - if (!(cr0 & X86_CR0_PG)) - return; - - if (tdp_enabled) { - kvm_init_mmu(vcpu); - return; - } - } - - if ((cr0 ^ old_cr0) & X86_CR0_PG) { - /* - * Clearing CR0.PG is defined to flush the TLB from the guest's - * perspective. - */ - if (!(cr0 & X86_CR0_PG)) - kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); - /* - * Check for async #PF completion events when enabling paging, - * as the vCPU may have previously encountered async #PFs (it's - * entirely legal for the guest to toggle paging on/off without - * waiting for the async #PF queue to drain). - */ - else if (kvm_pv_async_pf_enabled(vcpu)) - kvm_make_request(KVM_REQ_APF_READY, vcpu); - } - - if ((cr0 ^ old_cr0) & KVM_MMU_CR0_ROLE_BITS) - kvm_mmu_reset_context(vcpu); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_post_set_cr0); - -int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0) -{ - unsigned long old_cr0 =3D kvm_read_cr0(vcpu); - - if (!kvm_is_valid_cr0(vcpu, cr0)) - return 1; - - cr0 |=3D X86_CR0_ET; - - /* Write to CR0 reserved bits are ignored, even on Intel. */ - cr0 &=3D ~CR0_RESERVED_BITS; - -#ifdef CONFIG_X86_64 - if ((vcpu->arch.efer & EFER_LME) && !is_paging(vcpu) && - (cr0 & X86_CR0_PG)) { - int cs_db, cs_l; - - if (!is_pae(vcpu)) - return 1; - kvm_x86_call(get_cs_db_l_bits)(vcpu, &cs_db, &cs_l); - if (cs_l) - return 1; - } -#endif - if (!(vcpu->arch.efer & EFER_LME) && (cr0 & X86_CR0_PG) && - is_pae(vcpu) && ((cr0 ^ old_cr0) & X86_CR0_PDPTR_BITS) && - !load_pdptrs(vcpu, kvm_read_cr3(vcpu))) - return 1; - - if (!(cr0 & X86_CR0_PG) && - (is_64_bit_mode(vcpu) || kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE))) - return 1; - - if (!(cr0 & X86_CR0_WP) && kvm_is_cr4_bit_set(vcpu, X86_CR4_CET)) - return 1; - - kvm_x86_call(set_cr0)(vcpu, cr0); - - kvm_post_set_cr0(vcpu, old_cr0, cr0); - - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr0); - -void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw) -{ - (void)kvm_set_cr0(vcpu, kvm_read_cr0_bits(vcpu, ~0x0eul) | (msw & 0x0f)); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_lmsw); - static void kvm_load_xfeatures(struct kvm_vcpu *vcpu, bool load_guest) { if (vcpu->arch.guest_state_protected) @@ -1289,89 +1121,7 @@ int kvm_emulate_xsetbv(struct kvm_vcpu *vcpu) } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_xsetbv); =20 -static bool kvm_is_valid_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) -{ - return __kvm_is_valid_cr4(vcpu, cr4) && - kvm_x86_call(is_valid_cr4)(vcpu, cr4); -} - -void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsign= ed long cr4) -{ - if ((cr4 ^ old_cr4) & KVM_MMU_CR4_ROLE_BITS) - kvm_mmu_reset_context(vcpu); - - /* - * If CR4.PCIDE is changed 0 -> 1, there is no need to flush the TLB - * according to the SDM; however, stale prev_roots could be reused - * incorrectly in the future after a MOV to CR3 with NOFLUSH=3D1, so we - * free them all. This is *not* a superset of KVM_REQ_TLB_FLUSH_GUEST - * or KVM_REQ_TLB_FLUSH_CURRENT, because the hardware TLB is not flushed, - * so fall through. - */ - if (!tdp_enabled && - (cr4 & X86_CR4_PCIDE) && !(old_cr4 & X86_CR4_PCIDE)) - kvm_mmu_unload(vcpu); - - /* - * The TLB has to be flushed for all PCIDs if any of the following - * (architecturally required) changes happen: - * - CR4.PCIDE is changed from 1 to 0 - * - CR4.PGE is toggled - * - * This is a superset of KVM_REQ_TLB_FLUSH_CURRENT. - */ - if (((cr4 ^ old_cr4) & X86_CR4_PGE) || - (!(cr4 & X86_CR4_PCIDE) && (old_cr4 & X86_CR4_PCIDE))) - kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); - - /* - * The TLB has to be flushed for the current PCID if any of the - * following (architecturally required) changes happen: - * - CR4.SMEP is changed from 0 to 1 - * - CR4.PAE is toggled - */ - else if (((cr4 ^ old_cr4) & X86_CR4_PAE) || - ((cr4 & X86_CR4_SMEP) && !(old_cr4 & X86_CR4_SMEP))) - kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu); - -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_post_set_cr4); - -int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4) -{ - unsigned long old_cr4 =3D kvm_read_cr4(vcpu); - - if (!kvm_is_valid_cr4(vcpu, cr4)) - return 1; - - if (is_long_mode(vcpu)) { - if (!(cr4 & X86_CR4_PAE)) - return 1; - if ((cr4 ^ old_cr4) & X86_CR4_LA57) - return 1; - } else if (is_paging(vcpu) && (cr4 & X86_CR4_PAE) - && ((cr4 ^ old_cr4) & X86_CR4_PDPTR_BITS) - && !load_pdptrs(vcpu, kvm_read_cr3(vcpu))) - return 1; - - if ((cr4 & X86_CR4_PCIDE) && !(old_cr4 & X86_CR4_PCIDE)) { - /* PCID can not be enabled when cr3[11:0]!=3D000H or EFER.LMA=3D0 */ - if ((kvm_read_cr3(vcpu) & X86_CR3_PCID_MASK) || !is_long_mode(vcpu)) - return 1; - } - - if ((cr4 & X86_CR4_CET) && !kvm_is_cr0_bit_set(vcpu, X86_CR0_WP)) - return 1; - - kvm_x86_call(set_cr4)(vcpu, cr4); - - kvm_post_set_cr4(vcpu, old_cr4, cr4); - - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr4); - -static void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid) +void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid) { struct kvm_mmu *mmu =3D vcpu->arch.mmu; unsigned long roots_to_free =3D 0; @@ -1414,167 +1164,6 @@ static void kvm_invalidate_pcid(struct kvm_vcpu *vc= pu, unsigned long pcid) kvm_mmu_free_roots(vcpu->kvm, mmu, roots_to_free); } =20 -int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3) -{ - bool skip_tlb_flush =3D false; - unsigned long pcid =3D 0; -#ifdef CONFIG_X86_64 - if (kvm_is_cr4_bit_set(vcpu, X86_CR4_PCIDE)) { - skip_tlb_flush =3D cr3 & X86_CR3_PCID_NOFLUSH; - cr3 &=3D ~X86_CR3_PCID_NOFLUSH; - pcid =3D cr3 & X86_CR3_PCID_MASK; - } -#endif - - /* PDPTRs are always reloaded for PAE paging. */ - if (cr3 =3D=3D kvm_read_cr3(vcpu) && !is_pae_paging(vcpu)) - goto handle_tlb_flush; - - /* - * Do not condition the GPA check on long mode, this helper is used to - * stuff CR3, e.g. for RSM emulation, and there is no guarantee that - * the current vCPU mode is accurate. - */ - if (!kvm_vcpu_is_legal_cr3(vcpu, cr3)) - return 1; - - if (is_pae_paging(vcpu) && !load_pdptrs(vcpu, cr3)) - return 1; - - if (cr3 !=3D kvm_read_cr3(vcpu)) - kvm_mmu_new_pgd(vcpu, cr3); - - vcpu->arch.cr3 =3D cr3; - kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); - /* Do not call post_set_cr3, we do not get here for confidential guests. = */ - -handle_tlb_flush: - /* - * A load of CR3 that flushes the TLB flushes only the current PCID, - * even if PCID is disabled, in which case PCID=3D0 is flushed. It's a - * moot point in the end because _disabling_ PCID will flush all PCIDs, - * and it's impossible to use a non-zero PCID when PCID is disabled, - * i.e. only PCID=3D0 can be relevant. - */ - if (!skip_tlb_flush) - kvm_invalidate_pcid(vcpu, pcid); - - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr3); - -int kvm_set_cr8(struct kvm_vcpu *vcpu, unsigned long cr8) -{ - if (cr8 & CR8_RESERVED_BITS) - return 1; - if (lapic_in_kernel(vcpu)) - kvm_lapic_set_tpr(vcpu, cr8); - else - vcpu->arch.cr8 =3D cr8; - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_cr8); - -unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu) -{ - if (lapic_in_kernel(vcpu)) - return kvm_lapic_get_cr8(vcpu); - else - return vcpu->arch.cr8; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_cr8); - -static void kvm_update_dr0123(struct kvm_vcpu *vcpu) -{ - int i; - - if (!(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP)) { - for (i =3D 0; i < KVM_NR_DB_REGS; i++) - vcpu->arch.eff_db[i] =3D vcpu->arch.db[i]; - } -} - -void kvm_update_dr7(struct kvm_vcpu *vcpu) -{ - unsigned long dr7; - - if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) - dr7 =3D vcpu->arch.guest_debug_dr7; - else - dr7 =3D vcpu->arch.dr7; - kvm_x86_call(set_dr7)(vcpu, dr7); - vcpu->arch.switch_db_regs &=3D ~KVM_DEBUGREG_BP_ENABLED; - if (dr7 & DR7_BP_EN_MASK) - vcpu->arch.switch_db_regs |=3D KVM_DEBUGREG_BP_ENABLED; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_update_dr7); - -static u64 kvm_dr6_fixed(struct kvm_vcpu *vcpu) -{ - u64 fixed =3D DR6_FIXED_1; - - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_RTM)) - fixed |=3D DR6_RTM; - - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_BUS_LOCK_DETECT)) - fixed |=3D DR6_BUS_LOCK; - return fixed; -} - -int kvm_set_dr(struct kvm_vcpu *vcpu, int dr, unsigned long val) -{ - size_t size =3D ARRAY_SIZE(vcpu->arch.db); - - switch (dr) { - case 0 ... 3: - vcpu->arch.db[array_index_nospec(dr, size)] =3D val; - if (!(vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP)) - vcpu->arch.eff_db[dr] =3D val; - break; - case 4: - case 6: - if (!kvm_dr6_valid(val)) - return 1; /* #GP */ - vcpu->arch.dr6 =3D (val & DR6_VOLATILE) | kvm_dr6_fixed(vcpu); - break; - case 5: - default: /* 7 */ - if (!kvm_dr7_valid(val)) - return 1; /* #GP */ - vcpu->arch.dr7 =3D (val & DR7_VOLATILE) | DR7_FIXED_1; - kvm_update_dr7(vcpu); - break; - } - - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_dr); - -unsigned long kvm_get_dr(struct kvm_vcpu *vcpu, int dr) -{ - size_t size =3D ARRAY_SIZE(vcpu->arch.db); - - switch (dr) { - case 0 ... 3: - return vcpu->arch.db[array_index_nospec(dr, size)]; - case 4: - case 6: - return vcpu->arch.dr6; - case 5: - default: /* 7 */ - return vcpu->arch.dr7; - } -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_dr); - -static unsigned long kvm_get_effective_dr7(struct kvm_vcpu *vcpu) -{ - if (vcpu->guest_debug & KVM_GUESTDBG_USE_HW_BP) - return vcpu->arch.guest_debug_dr7; - - return vcpu->arch.dr7; -} - int kvm_emulate_rdpmc(struct kvm_vcpu *vcpu) { u32 pmc =3D kvm_ecx_read(vcpu); @@ -5532,7 +5121,7 @@ static struct kvm_queued_exception *kvm_get_exception= _to_save(struct kvm_vcpu *v return &vcpu->arch.exception; } =20 -static void kvm_handle_exception_payload_quirk(struct kvm_vcpu *vcpu) +void kvm_handle_exception_payload_quirk(struct kvm_vcpu *vcpu) { struct kvm_queued_exception *ex =3D kvm_get_exception_to_save(vcpu); =20 @@ -5736,57 +5325,6 @@ static int kvm_vcpu_ioctl_x86_set_vcpu_events(struct= kvm_vcpu *vcpu, return 0; } =20 -static int kvm_vcpu_ioctl_x86_get_debugregs(struct kvm_vcpu *vcpu, - struct kvm_debugregs *dbgregs) -{ - unsigned int i; - - if (vcpu->kvm->arch.has_protected_state && - vcpu->arch.guest_state_protected) - return -EINVAL; - - kvm_handle_exception_payload_quirk(vcpu); - - memset(dbgregs, 0, sizeof(*dbgregs)); - - BUILD_BUG_ON(ARRAY_SIZE(vcpu->arch.db) !=3D ARRAY_SIZE(dbgregs->db)); - for (i =3D 0; i < ARRAY_SIZE(vcpu->arch.db); i++) - dbgregs->db[i] =3D vcpu->arch.db[i]; - - dbgregs->dr6 =3D vcpu->arch.dr6; - dbgregs->dr7 =3D vcpu->arch.dr7; - return 0; -} - -static int kvm_vcpu_ioctl_x86_set_debugregs(struct kvm_vcpu *vcpu, - struct kvm_debugregs *dbgregs) -{ - unsigned int i; - - if (vcpu->kvm->arch.has_protected_state && - vcpu->arch.guest_state_protected) - return -EINVAL; - - if (dbgregs->flags) - return -EINVAL; - - if (!kvm_dr6_valid(dbgregs->dr6)) - return -EINVAL; - if (!kvm_dr7_valid(dbgregs->dr7)) - return -EINVAL; - - for (i =3D 0; i < ARRAY_SIZE(vcpu->arch.db); i++) - vcpu->arch.db[i] =3D dbgregs->db[i]; - - kvm_update_dr0123(vcpu); - vcpu->arch.dr6 =3D dbgregs->dr6; - vcpu->arch.dr7 =3D dbgregs->dr7; - kvm_update_dr7(vcpu); - - return 0; -} - - static int kvm_vcpu_ioctl_x86_get_xsave2(struct kvm_vcpu *vcpu, u8 *state, unsigned int size) { @@ -6623,7 +6161,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp, r =3D -ENOMEM; if (!u.sregs2) goto out; - __get_sregs2(vcpu, u.sregs2); + kvm_x86_vcpu_ioctl_get_sregs2(vcpu, u.sregs2); r =3D -EFAULT; if (copy_to_user(argp, u.sregs2, sizeof(struct kvm_sregs2))) goto out; @@ -6642,7 +6180,7 @@ long kvm_arch_vcpu_ioctl(struct file *filp, u.sregs2 =3D NULL; goto out; } - r =3D __set_sregs2(vcpu, u.sregs2); + r =3D kvm_x86_vcpu_ioctl_set_sregs2(vcpu, u.sregs2); break; } case KVM_HAS_DEVICE_ATTR: @@ -8492,11 +8030,6 @@ static int emulator_pio_out_emulated(struct x86_emul= ate_ctxt *ctxt, return emulator_pio_out(emul_to_vcpu(ctxt), size, port, val, count); } =20 -static unsigned long get_segment_base(struct kvm_vcpu *vcpu, int seg) -{ - return kvm_x86_call(get_segment_base)(vcpu, seg); -} - static void emulator_invlpg(struct x86_emulate_ctxt *ctxt, ulong address) { kvm_mmu_invlpg(emul_to_vcpu(ctxt), address); @@ -8641,7 +8174,7 @@ static void emulator_set_idt(struct x86_emulate_ctxt = *ctxt, struct desc_ptr *dt) static unsigned long emulator_get_cached_segment_base( struct x86_emulate_ctxt *ctxt, int seg) { - return get_segment_base(emul_to_vcpu(ctxt), seg); + return kvm_get_segment_base(emul_to_vcpu(ctxt), seg); } =20 static bool emulator_get_segment(struct x86_emulate_ctxt *ctxt, u16 *selec= tor, @@ -12073,179 +11606,6 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) return r; } =20 -static void __get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) -{ - if (vcpu->arch.emulate_regs_need_sync_to_vcpu) { - /* - * We are here if userspace calls get_regs() in the middle of - * instruction emulation. Registers state needs to be copied - * back from emulation context to vcpu. Userspace shouldn't do - * that usually, but some bad designed PV devices (vmware - * backdoor interface) need this to work - */ - emulator_writeback_register_cache(vcpu->arch.emulate_ctxt); - vcpu->arch.emulate_regs_need_sync_to_vcpu =3D false; - } - regs->rax =3D kvm_rax_read_raw(vcpu); - regs->rbx =3D kvm_rbx_read_raw(vcpu); - regs->rcx =3D kvm_rcx_read_raw(vcpu); - regs->rdx =3D kvm_rdx_read_raw(vcpu); - regs->rsi =3D kvm_rsi_read_raw(vcpu); - regs->rdi =3D kvm_rdi_read_raw(vcpu); - regs->rsp =3D kvm_rsp_read(vcpu); - regs->rbp =3D kvm_rbp_read_raw(vcpu); -#ifdef CONFIG_X86_64 - regs->r8 =3D kvm_r8_read_raw(vcpu); - regs->r9 =3D kvm_r9_read_raw(vcpu); - regs->r10 =3D kvm_r10_read_raw(vcpu); - regs->r11 =3D kvm_r11_read_raw(vcpu); - regs->r12 =3D kvm_r12_read_raw(vcpu); - regs->r13 =3D kvm_r13_read_raw(vcpu); - regs->r14 =3D kvm_r14_read_raw(vcpu); - regs->r15 =3D kvm_r15_read_raw(vcpu); -#endif - - regs->rip =3D kvm_rip_read(vcpu); - regs->rflags =3D kvm_get_rflags(vcpu); -} - -int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *r= egs) -{ - if (vcpu->kvm->arch.has_protected_state && - vcpu->arch.guest_state_protected) - return -EINVAL; - - vcpu_load(vcpu); - __get_regs(vcpu, regs); - vcpu_put(vcpu); - return 0; -} - -static void __set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) -{ - vcpu->arch.emulate_regs_need_sync_from_vcpu =3D true; - vcpu->arch.emulate_regs_need_sync_to_vcpu =3D false; - - kvm_rax_write_raw(vcpu, regs->rax); - kvm_rbx_write_raw(vcpu, regs->rbx); - kvm_rcx_write_raw(vcpu, regs->rcx); - kvm_rdx_write_raw(vcpu, regs->rdx); - kvm_rsi_write_raw(vcpu, regs->rsi); - kvm_rdi_write_raw(vcpu, regs->rdi); - kvm_rsp_write(vcpu, regs->rsp); - kvm_rbp_write_raw(vcpu, regs->rbp); -#ifdef CONFIG_X86_64 - kvm_r8_write_raw(vcpu, regs->r8); - kvm_r9_write_raw(vcpu, regs->r9); - kvm_r10_write_raw(vcpu, regs->r10); - kvm_r11_write_raw(vcpu, regs->r11); - kvm_r12_write_raw(vcpu, regs->r12); - kvm_r13_write_raw(vcpu, regs->r13); - kvm_r14_write_raw(vcpu, regs->r14); - kvm_r15_write_raw(vcpu, regs->r15); -#endif - - kvm_rip_write(vcpu, regs->rip); - kvm_set_rflags(vcpu, regs->rflags | X86_EFLAGS_FIXED); - - vcpu->arch.exception.pending =3D false; - vcpu->arch.exception_vmexit.pending =3D false; - - kvm_make_request(KVM_REQ_EVENT, vcpu); -} - -int kvm_arch_vcpu_ioctl_set_regs(struct kvm_vcpu *vcpu, struct kvm_regs *r= egs) -{ - if (vcpu->kvm->arch.has_protected_state && - vcpu->arch.guest_state_protected) - return -EINVAL; - - vcpu_load(vcpu); - __set_regs(vcpu, regs); - vcpu_put(vcpu); - return 0; -} - -static void __get_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sr= egs) -{ - struct desc_ptr dt; - - if (vcpu->arch.guest_state_protected) - goto skip_protected_regs; - - kvm_handle_exception_payload_quirk(vcpu); - - kvm_get_segment(vcpu, &sregs->cs, VCPU_SREG_CS); - kvm_get_segment(vcpu, &sregs->ds, VCPU_SREG_DS); - kvm_get_segment(vcpu, &sregs->es, VCPU_SREG_ES); - kvm_get_segment(vcpu, &sregs->fs, VCPU_SREG_FS); - kvm_get_segment(vcpu, &sregs->gs, VCPU_SREG_GS); - kvm_get_segment(vcpu, &sregs->ss, VCPU_SREG_SS); - - kvm_get_segment(vcpu, &sregs->tr, VCPU_SREG_TR); - kvm_get_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR); - - kvm_x86_call(get_idt)(vcpu, &dt); - sregs->idt.limit =3D dt.size; - sregs->idt.base =3D dt.address; - kvm_x86_call(get_gdt)(vcpu, &dt); - sregs->gdt.limit =3D dt.size; - sregs->gdt.base =3D dt.address; - - sregs->cr2 =3D vcpu->arch.cr2; - sregs->cr3 =3D kvm_read_cr3(vcpu); - -skip_protected_regs: - sregs->cr0 =3D kvm_read_cr0(vcpu); - sregs->cr4 =3D kvm_read_cr4(vcpu); - sregs->cr8 =3D kvm_get_cr8(vcpu); - sregs->efer =3D vcpu->arch.efer; - sregs->apic_base =3D vcpu->arch.apic_base; -} - -static void __get_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) -{ - __get_sregs_common(vcpu, sregs); - - if (vcpu->arch.guest_state_protected) - return; - - if (vcpu->arch.interrupt.injected && !vcpu->arch.interrupt.soft) - set_bit(vcpu->arch.interrupt.nr, - (unsigned long *)sregs->interrupt_bitmap); -} - -static void __get_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2) -{ - int i; - - __get_sregs_common(vcpu, (struct kvm_sregs *)sregs2); - - if (vcpu->arch.guest_state_protected) - return; - - if (is_pae_paging(vcpu)) { - kvm_vcpu_srcu_read_lock(vcpu); - for (i =3D 0 ; i < 4 ; i++) - sregs2->pdptrs[i] =3D kvm_pdptr_read(vcpu, i); - sregs2->flags |=3D KVM_SREGS2_FLAGS_PDPTRS_VALID; - kvm_vcpu_srcu_read_unlock(vcpu); - } -} - -int kvm_arch_vcpu_ioctl_get_sregs(struct kvm_vcpu *vcpu, - struct kvm_sregs *sregs) -{ - if (vcpu->kvm->arch.has_protected_state && - vcpu->arch.guest_state_protected) - return -EINVAL; - - vcpu_load(vcpu); - __get_sregs(vcpu, sregs); - vcpu_put(vcpu); - return 0; -} - int kvm_arch_vcpu_ioctl_get_mpstate(struct kvm_vcpu *vcpu, struct kvm_mp_state *mp_state) { @@ -12365,175 +11725,6 @@ int kvm_task_switch(struct kvm_vcpu *vcpu, u16 ts= s_selector, int idt_index, } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_task_switch); =20 -static bool kvm_is_valid_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sr= egs) -{ - if ((sregs->efer & EFER_LME) && (sregs->cr0 & X86_CR0_PG)) { - /* - * When EFER.LME and CR0.PG are set, the processor is in - * 64-bit mode (though maybe in a 32-bit code segment). - * CR4.PAE and EFER.LMA must be set. - */ - if (!(sregs->cr4 & X86_CR4_PAE) || !(sregs->efer & EFER_LMA)) - return false; - if (!kvm_vcpu_is_legal_cr3(vcpu, sregs->cr3)) - return false; - } else { - /* - * Not in 64-bit mode: EFER.LMA is clear and the code - * segment cannot be 64-bit. - */ - if (sregs->efer & EFER_LMA || sregs->cs.l) - return false; - } - - return kvm_is_valid_cr4(vcpu, sregs->cr4) && - kvm_is_valid_cr0(vcpu, sregs->cr0); -} - -static int __set_sregs_common(struct kvm_vcpu *vcpu, struct kvm_sregs *sre= gs, - int *mmu_reset_needed, bool update_pdptrs) -{ - int idx; - struct desc_ptr dt; - - if (!kvm_is_valid_sregs(vcpu, sregs)) - return -EINVAL; - - if (kvm_apic_set_base(vcpu, sregs->apic_base, true)) - return -EINVAL; - - if (vcpu->arch.guest_state_protected) - return 0; - - dt.size =3D sregs->idt.limit; - dt.address =3D sregs->idt.base; - kvm_x86_call(set_idt)(vcpu, &dt); - dt.size =3D sregs->gdt.limit; - dt.address =3D sregs->gdt.base; - kvm_x86_call(set_gdt)(vcpu, &dt); - - vcpu->arch.cr2 =3D sregs->cr2; - *mmu_reset_needed |=3D kvm_read_cr3(vcpu) !=3D sregs->cr3; - vcpu->arch.cr3 =3D sregs->cr3; - kvm_register_mark_dirty(vcpu, VCPU_REG_CR3); - kvm_x86_call(post_set_cr3)(vcpu, sregs->cr3); - - kvm_set_cr8(vcpu, sregs->cr8); - - *mmu_reset_needed |=3D vcpu->arch.efer !=3D sregs->efer; - kvm_x86_call(set_efer)(vcpu, sregs->efer); - - *mmu_reset_needed |=3D kvm_read_cr0(vcpu) !=3D sregs->cr0; - kvm_x86_call(set_cr0)(vcpu, sregs->cr0); - - *mmu_reset_needed |=3D kvm_read_cr4(vcpu) !=3D sregs->cr4; - kvm_x86_call(set_cr4)(vcpu, sregs->cr4); - - if (update_pdptrs) { - idx =3D srcu_read_lock(&vcpu->kvm->srcu); - if (is_pae_paging(vcpu)) { - load_pdptrs(vcpu, kvm_read_cr3(vcpu)); - *mmu_reset_needed =3D 1; - } - srcu_read_unlock(&vcpu->kvm->srcu, idx); - } - - kvm_set_segment(vcpu, &sregs->cs, VCPU_SREG_CS); - kvm_set_segment(vcpu, &sregs->ds, VCPU_SREG_DS); - kvm_set_segment(vcpu, &sregs->es, VCPU_SREG_ES); - kvm_set_segment(vcpu, &sregs->fs, VCPU_SREG_FS); - kvm_set_segment(vcpu, &sregs->gs, VCPU_SREG_GS); - kvm_set_segment(vcpu, &sregs->ss, VCPU_SREG_SS); - - kvm_set_segment(vcpu, &sregs->tr, VCPU_SREG_TR); - kvm_set_segment(vcpu, &sregs->ldt, VCPU_SREG_LDTR); - - kvm_lapic_update_cr8_intercept(vcpu); - - /* Older userspace won't unhalt the vcpu on reset. */ - if (kvm_vcpu_is_bsp(vcpu) && kvm_rip_read(vcpu) =3D=3D 0xfff0 && - sregs->cs.selector =3D=3D 0xf000 && sregs->cs.base =3D=3D 0xffff0000 = && - !is_protmode(vcpu)) - kvm_set_mp_state(vcpu, KVM_MP_STATE_RUNNABLE); - - return 0; -} - -static int __set_sregs(struct kvm_vcpu *vcpu, struct kvm_sregs *sregs) -{ - int pending_vec, max_bits; - int mmu_reset_needed =3D 0; - int ret =3D __set_sregs_common(vcpu, sregs, &mmu_reset_needed, true); - - if (ret) - return ret; - - if (mmu_reset_needed) { - kvm_mmu_reset_context(vcpu); - kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); - } - - max_bits =3D KVM_NR_INTERRUPTS; - pending_vec =3D find_first_bit( - (const unsigned long *)sregs->interrupt_bitmap, max_bits); - - if (pending_vec < max_bits) { - kvm_queue_interrupt(vcpu, pending_vec, false); - pr_debug("Set back pending irq %d\n", pending_vec); - kvm_make_request(KVM_REQ_EVENT, vcpu); - } - return 0; -} - -static int __set_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2) -{ - int mmu_reset_needed =3D 0; - bool valid_pdptrs =3D sregs2->flags & KVM_SREGS2_FLAGS_PDPTRS_VALID; - bool pae =3D (sregs2->cr0 & X86_CR0_PG) && (sregs2->cr4 & X86_CR4_PAE) && - !(sregs2->efer & EFER_LMA); - int i, ret; - - if (sregs2->flags & ~KVM_SREGS2_FLAGS_PDPTRS_VALID) - return -EINVAL; - - if (valid_pdptrs && (!pae || vcpu->arch.guest_state_protected)) - return -EINVAL; - - ret =3D __set_sregs_common(vcpu, (struct kvm_sregs *)sregs2, - &mmu_reset_needed, !valid_pdptrs); - if (ret) - return ret; - - if (valid_pdptrs) { - for (i =3D 0; i < 4 ; i++) - kvm_pdptr_write(vcpu, i, sregs2->pdptrs[i]); - - kvm_register_mark_dirty(vcpu, VCPU_REG_PDPTR); - mmu_reset_needed =3D 1; - vcpu->arch.pdptrs_from_userspace =3D true; - } - if (mmu_reset_needed) { - kvm_mmu_reset_context(vcpu); - kvm_make_request(KVM_REQ_TLB_FLUSH_GUEST, vcpu); - } - return 0; -} - -int kvm_arch_vcpu_ioctl_set_sregs(struct kvm_vcpu *vcpu, - struct kvm_sregs *sregs) -{ - int ret; - - if (vcpu->kvm->arch.has_protected_state && - vcpu->arch.guest_state_protected) - return -EINVAL; - - vcpu_load(vcpu); - ret =3D __set_sregs(vcpu, sregs); - vcpu_put(vcpu); - return ret; -} - static void kvm_arch_vcpu_guestdbg_update_apicv_inhibit(struct kvm *kvm) { bool set =3D false; @@ -12691,11 +11882,7 @@ static void store_regs(struct kvm_vcpu *vcpu) { BUILD_BUG_ON(sizeof(struct kvm_sync_regs) > SYNC_REGS_SIZE_BYTES); =20 - if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_REGS) - __get_regs(vcpu, &vcpu->run->s.regs.regs); - - if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_SREGS) - __get_sregs(vcpu, &vcpu->run->s.regs.sregs); + kvm_run_sync_regs_to_user(vcpu); =20 if (vcpu->run->kvm_valid_regs & KVM_SYNC_X86_EVENTS) kvm_vcpu_ioctl_x86_get_vcpu_events( @@ -12704,19 +11891,8 @@ static void store_regs(struct kvm_vcpu *vcpu) =20 static int sync_regs(struct kvm_vcpu *vcpu) { - if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_REGS) { - __set_regs(vcpu, &vcpu->run->s.regs.regs); - vcpu->run->kvm_dirty_regs &=3D ~KVM_SYNC_X86_REGS; - } - - if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_SREGS) { - struct kvm_sregs sregs =3D vcpu->run->s.regs.sregs; - - if (__set_sregs(vcpu, &sregs)) - return -EINVAL; - - vcpu->run->kvm_dirty_regs &=3D ~KVM_SYNC_X86_SREGS; - } + if (kvm_run_sync_regs_from_user(vcpu)) + return -EINVAL; =20 if (vcpu->run->kvm_dirty_regs & KVM_SYNC_X86_EVENTS) { struct kvm_vcpu_events events =3D vcpu->run->s.regs.events; @@ -13818,51 +12994,6 @@ int kvm_arch_interrupt_allowed(struct kvm_vcpu *vc= pu) return kvm_x86_call(interrupt_allowed)(vcpu, false); } =20 -unsigned long kvm_get_linear_rip(struct kvm_vcpu *vcpu) -{ - /* Can't read the RIP when guest state is protected, just return 0 */ - if (vcpu->arch.guest_state_protected) - return 0; - - if (is_64_bit_mode(vcpu)) - return kvm_rip_read(vcpu); - return (u32)(get_segment_base(vcpu, VCPU_SREG_CS) + - kvm_rip_read(vcpu)); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_linear_rip); - -bool kvm_is_linear_rip(struct kvm_vcpu *vcpu, unsigned long linear_rip) -{ - return kvm_get_linear_rip(vcpu) =3D=3D linear_rip; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_is_linear_rip); - -unsigned long kvm_get_rflags(struct kvm_vcpu *vcpu) -{ - unsigned long rflags; - - rflags =3D kvm_x86_call(get_rflags)(vcpu); - if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP) - rflags &=3D ~X86_EFLAGS_TF; - return rflags; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_rflags); - -static void __kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags) -{ - if (vcpu->guest_debug & KVM_GUESTDBG_SINGLESTEP && - kvm_is_linear_rip(vcpu, vcpu->arch.singlestep_rip)) - rflags |=3D X86_EFLAGS_TF; - kvm_x86_call(set_rflags)(vcpu, rflags); -} - -void kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags) -{ - __kvm_set_rflags(vcpu, rflags); - kvm_make_request(KVM_REQ_EVENT, vcpu); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_rflags); - static inline u32 kvm_async_pf_hash_fn(gfn_t gfn) { BUILD_BUG_ON(!is_power_of_2(ASYNC_PF_PER_VCPU)); diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index acb22167901f..80ed36d5d62a 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -403,6 +403,7 @@ int handle_ud(struct kvm_vcpu *vcpu); =20 void kvm_deliver_exception_payload(struct kvm_vcpu *vcpu, struct kvm_queued_exception *ex); +void kvm_handle_exception_payload_quirk(struct kvm_vcpu *vcpu); =20 int kvm_mtrr_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data); int kvm_mtrr_get_msr(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata); @@ -597,6 +598,7 @@ static inline void kvm_machine_check(void) int kvm_spec_ctrl_test_value(u64 value); int kvm_handle_memory_failure(struct kvm_vcpu *vcpu, int r, struct x86_exception *e); +void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid); int kvm_handle_invpcid(struct kvm_vcpu *vcpu, unsigned long type, gva_t gv= a); bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type); =20 --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2632843E49D for ; Fri, 29 May 2026 22:23:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093384; cv=none; b=kcRhp1waxTpAAj/UjXTHvBUILBSYUN9EvpnWlB3ZjSENuH20Tq7qOmf6ulzI3sQSrbiDTh6BPdCmntnndmug+afS5QgXJr8+6jzuh3/rxvD7dEWHPnc0qeP4VlV/Y0//ORVVDektHv+/2bqoVcqWhxNXswdB51f9J2GZZTZvm4g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093384; c=relaxed/simple; bh=zgjEBonFpYyfZEeJOfnNulnQALKYg2PSbkfjIHq2VTI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=DwcnmZf3MlfEj4lX9YPmN6nqjWnTiEKwGtlimoncJW7UdH3Jt7NcxRZk9DkvH23RMf8s/oPzGGIy8VEO0M7JXYMYaV5ifI3bducJj4LeJBm1YdC4ZobKkFh84tknvXXFPIddf8f8tWYdxImoLRYv08oJkuwQh0pKkoOHaKalIM0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=FfXn8aWF; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="FfXn8aWF" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-3692f395339so14654355a91.1 for ; Fri, 29 May 2026 15:23:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093382; x=1780698182; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=L8jJfO93jBjZBPQnPLlI5VrnRwAv6dVrWszogTbYM3M=; b=FfXn8aWFEt/vn/a57i8E3xgo99bk6AMsmdiJluIej3yjB6Iv8kxXZj1m8S9g4lnjuy lTCiAYQsr/67q+SWKkEYxBqgpQsB6hl7NZUgam8Itau+kOC6hr2CKQndSfWkkEEHIx9k 5fM0B5ghmqDfEj+fGonnOxghSoAcGsXez3XzowRDhMImMS8n2pgyv4OBmgd5TU7cyewN hKE7bjVDf+6QaPYOxdAnrAFGo1RvQqVAXnpDBhJbpYsiZaWRxBNPno3UpCOSayRT4jOb IpIl5EPFEJWAHZvbDGqkjlGQSyyU/16WbOciDS4Rqsl/DlOFZ75cyLpYMfp57qcJknJB qbKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093382; x=1780698182; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=L8jJfO93jBjZBPQnPLlI5VrnRwAv6dVrWszogTbYM3M=; b=GiJl00/5BbBY9u4qTw5+BeSu2YgJre+di9TvrKLRctbW5LbP+cRYEJ8czZiPoDh8Nv QQrW4Lq2c0MpSdZvd4tEO2OEf77Ddrc67YISTTeVxzbC5jp8Uwgv216bwasae0IJ3EVO 4M6dcDeIN+YAjSCwZ6pU64LWy9dWmq/4POXo9MyE9V5pfDSFysjrKVkhAvJ1wcAYDCPw GIv0t68mo+dcknvr/2PDNV/I8P2DKsExTFro+gp2sr72xXWXJdasdd47DANBvi+G4eH0 b46YCod/Sf79DzD4iozaQ8bq52ihb6dGJtJJQ/vOm+743JJtsQSM95y/6WZ5AFjb0i34 o5GQ== X-Forwarded-Encrypted: i=1; AFNElJ98RA8D9eC41NsJwEnO+K7l5d6ZVr2+KBVszbhNbwmUVQGrbor3ty/Bmm76PTlLHGChvKu0joTecLNHQig=@vger.kernel.org X-Gm-Message-State: AOJu0YzfXYeMWQFLgYxVX1tOE8iIOlZ3LPmnbkdW1LriK7BBIEP0Ep+5 BSdOc7crT/VDBJxoKsxPSphhctBjXs5LUZKvTloGCzk5Z3ZLiwvfUCHKIjTS57fjRmd8KshTgmP Qb17MIw== X-Received: from plho6.prod.google.com ([2002:a17:903:23c6:b0:2bc:bdcd:c51f]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:fc84:b0:2be:22cc:e227 with SMTP id d9443c01a7336-2bf3679e713mr19921875ad.4.1780093382161; Fri, 29 May 2026 15:23:02 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:21:59 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-17-seanjc@google.com> Subject: [PATCH v3 16/40] KVM: x86: Move local APIC specific helpers out of asm/kvm_host.h From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move single-use local APIC IRQ helpers out of asm/kvm_host.h so that they are co-located with their user, and not exposed to the broader world. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Reviewed-by: Yosry Ahmed --- arch/x86/include/asm/kvm_host.h | 12 ------------ arch/x86/kvm/irq.c | 7 +++++++ arch/x86/kvm/lapic.h | 5 +++++ 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 983bdc84f9f9..ee205f8ad5af 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1772,11 +1772,6 @@ struct kvm_lapic_irq { bool msi_redir_hint; }; =20 -static inline u16 kvm_lapic_irq_dest_mode(bool dest_mode_logical) -{ - return dest_mode_logical ? APIC_DEST_LOGICAL : APIC_DEST_PHYSICAL; -} - enum kvm_x86_run_flags { KVM_RUN_FORCE_IMMEDIATE_EXIT =3D BIT(0), KVM_RUN_LOAD_GUEST_DR6 =3D BIT(1), @@ -2510,13 +2505,6 @@ void __user *__x86_set_memory_region(struct kvm *kvm= , int id, gpa_t gpa, bool kvm_vcpu_is_reset_bsp(struct kvm_vcpu *vcpu); bool kvm_vcpu_is_bsp(struct kvm_vcpu *vcpu); =20 -static inline bool kvm_irq_is_postable(struct kvm_lapic_irq *irq) -{ - /* We can only post Fixed and LowPrio IRQs */ - return (irq->delivery_mode =3D=3D APIC_DM_FIXED || - irq->delivery_mode =3D=3D APIC_DM_LOWEST); -} - static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) { kvm_x86_call(vcpu_blocking)(vcpu); diff --git a/arch/x86/kvm/irq.c b/arch/x86/kvm/irq.c index 9519fec09ee6..2b4e68e7cadb 100644 --- a/arch/x86/kvm/irq.c +++ b/arch/x86/kvm/irq.c @@ -423,6 +423,13 @@ void kvm_arch_irq_routing_update(struct kvm *kvm) kvm_make_scan_ioapic_request(kvm); } =20 +static bool kvm_irq_is_postable(struct kvm_lapic_irq *irq) +{ + /* We can only post Fixed and LowPrio IRQs */ + return (irq->delivery_mode =3D=3D APIC_DM_FIXED || + irq->delivery_mode =3D=3D APIC_DM_LOWEST); +} + static int kvm_pi_update_irte(struct kvm_kernel_irqfd *irqfd, struct kvm_kernel_irq_routing_entry *entry) { diff --git a/arch/x86/kvm/lapic.h b/arch/x86/kvm/lapic.h index 71970213dc1f..32f09b25884a 100644 --- a/arch/x86/kvm/lapic.h +++ b/arch/x86/kvm/lapic.h @@ -237,6 +237,11 @@ static inline int kvm_lapic_latched_init(struct kvm_vc= pu *vcpu) return lapic_in_kernel(vcpu) && test_bit(KVM_APIC_INIT, &vcpu->arch.apic-= >pending_events); } =20 +static inline u16 kvm_lapic_irq_dest_mode(bool dest_mode_logical) +{ + return dest_mode_logical ? APIC_DEST_LOGICAL : APIC_DEST_PHYSICAL; +} + bool kvm_apic_pending_eoi(struct kvm_vcpu *vcpu, int vector); =20 bool kvm_lapic_suppress_eoi_broadcast(struct kvm_lapic *apic); --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 23BF243E9FD for ; Fri, 29 May 2026 22:23:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093385; cv=none; b=S2jy+96C0vI8/3Nst5gbycI7YEBmbNE44NjqQYRSCJJlkrdmwtpGWg5aXSWYLkPcZwHkf16rA6P5H6CyHqeb61wZGxM0G9po5DcMtALcE1h0pGG98nnBDvEzu5DEEX8wnINiIpZrFVkRn6K4vitfxY6iH1V1KgNos8dTX0WC7Sc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093385; c=relaxed/simple; bh=+s7EmeMhycIHSHknTX3BroUkHXXTYHphV9Lyigv0Y2U=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=CciM5jXbP3iU4+mPjQim/plptgf6gCpMYDi7mf2jKIErxWhrM/TxclJEde1PjOVwB2XJFAxhkwgA5+yfv/jStXaUA9pRIxXdvZlOgqc3LfM+EnptJKQnoZJg509m3caYReeJ141SNc+lRgpa7TBttAuBtJoR6qtZxXLOYxM7L64= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=rFdytNM0; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="rFdytNM0" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-c857befe561so361717a12.0 for ; Fri, 29 May 2026 15:23:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093383; x=1780698183; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=Or3Hjg0NqITAPRhpYIql5ziLytgduUzcn5JhR8+Ygug=; b=rFdytNM0OyYny1OX1ou+OpJkgJAksM2hAXlIHxxgtLYjDiRCAluKBJmmcIrKjaPfrh BhPoppL+QlJrl1ktrdWdprokdWi0z5+2sKjd4qYxx8og9wDVO+UaykUq/n0IZqrxFjF9 R7nhAm0vVmzmA4zvsARgK4IPH1uq9ZRrlM9kFTrHVCg6j9EStNqnVsf7wL9F4X20XCHd HHyFX8v1o2i1hHYec1+vjPaZhD6Og0NhLlmUBXMEWEt/Tvqz65mH8giH8T59RXzXrEHx 0YN6ur1D0rodcHz5AMSrTbH/Y/JryiAWAyHfa7pzxuiOaGYJ/mkNWIeUqx5RUXE2O/LW AqlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093383; x=1780698183; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Or3Hjg0NqITAPRhpYIql5ziLytgduUzcn5JhR8+Ygug=; b=JynAN7k+ZWVG5N/V96ztieu2KALr2N7/xvjETFekWbftBxUkhpAQ7IMuu6ttJrYEkf 4FU0KjK/ZhRw50DLNzO0odWVDlajIqttt2UvyEe0w1Kqnu9kAdW4SSJ6YN3u+VSFTZHi wFjTuahZrfAvVK+jsrqCJ2uL7HUZbi2cIU6IgP5ej1sUdnBQnlMmbk55dsoaSCOcf+0H YPl6lOwBDIrmfwZ3EQaJ4LiLN38kmX4RQ10aWckapoT8diS/ArDd4y/HZVrNVNNCMx9n 8y+l2+VGMorkloxnngP54I1vMrx/PpnJ9FLyOkrsUpcsud0qAR/7m6fwifsoNpU9JSPV 4x8Q== X-Forwarded-Encrypted: i=1; AFNElJ+ZrD2axzUXmaR7sypiRkGDMg4gg3IE6RPyBUNH9xhZtPSEpY2CZBYyBK+oVliN3UqJiKb7NkOdSjABItw=@vger.kernel.org X-Gm-Message-State: AOJu0YzDv7NUHjlCcGJIezGWlj9FaUkwtiJfNZodelLl7pbiANTZDYFp Zafm2V7n22rjON9w0I912KhXVNDhvgSHCGnrG0w79Q6Z8AaqzMAadjCR+bD+SGVT8CASvCQ4H6j bA++A9w== X-Received: from pgbcm4.prod.google.com ([2002:a05:6a02:a04:b0:c76:6c2f:89d4]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:730e:b0:398:abe1:eaac with SMTP id adf61e73a8af0-3b427f2f9camr1282404637.27.1780093383252; Fri, 29 May 2026 15:23:03 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:00 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-18-seanjc@google.com> Subject: [PATCH v3 17/40] KVM: x86: Drop defunct vcpu_tsc_khz() declaration From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove a dead vcpu_tsc_khz() declaration. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Reviewed-by: Yosry Ahmed --- arch/x86/include/asm/kvm_host.h | 2 -- 1 file changed, 2 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index ee205f8ad5af..2e535027dd5c 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2146,8 +2146,6 @@ int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long = cr3); =20 extern bool tdp_enabled; =20 -u64 vcpu_tsc_khz(struct kvm_vcpu *vcpu); - /* * EMULTYPE_NO_DECODE - Set when re-emulating an instruction (after comple= ting * userspace I/O) to indicate that the emulation context --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2A767436358 for ; Fri, 29 May 2026 22:23:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093386; cv=none; b=sSrIcsATE+5PjTpzvVNRjAelUqak/tcvbuoJV15LCtSX+ZVr9L9643zo6HFFQvMNbjkkJLXPyRvNoRUTa65vpS+YpxyAy2cWNKymSQc58V4GZcv0MRBLxzgEdbF3e8gAJ69uXAn1LpHF1m0D3A6fQmhj2okrtP1pODN3GSGnH/g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093386; c=relaxed/simple; bh=P2zXCRLo2YdTzgguBobD3cMk4Ntf0Zw0zqR0CFK6aS4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Z3hClsfaXY+qpaoaKxAR3i7RZiEf/hLCEHBziVIOkqFUP8J7uFY+uqtHy7Kn2VB1rjY9PEnW63qWwIYJ8FO0sjbO/jzSspR38vmr1hRVtdQOgaWcaFNhXRN6GSQeLbNNC2WnLIVHbn2t53x9wtjA7GeNc7s2+FDSnBK0ILJPsGY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=PT3o+d/n; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="PT3o+d/n" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2bf1dece2ecso16221975ad.1 for ; Fri, 29 May 2026 15:23:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093384; x=1780698184; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=SUKUT1cLL25O6B1OgJD5A10iK66u2vgLhkpt4cYUGcI=; b=PT3o+d/nrzA5J4WK9HDNgdrtIpQEIxRiYGaRxWpDsbI9GtQ2Qh6o3xwWM0/q0SAekb OguV4EhMZiYKPTCBNJwX0fOP+bf64dHP475nseF+Rhv3ZhozhkbH0/EV1GE9V2OgI5zy iKvYx1X/On+FCFSaM0fM2paZH0/AGA5CuZ/1GOWCdJA6TZqmtxpH1gxgW5DZk1dbfr58 6MvD2nMkFoMfunidHmLJnCMJnoFc59jsuqHsr3YgCIDuDLn/44L4PUldxJTXuuwgMcV7 2OqxTTeD47I5ZMNND0KG+1KajR0d/ywBWgRJ6hsE0CHQBWEfE5hli1mi+YYKp4NTkORZ fX5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093384; x=1780698184; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=SUKUT1cLL25O6B1OgJD5A10iK66u2vgLhkpt4cYUGcI=; b=s7S/y36J8W4qflPRw6Z3DyI94OozuODWWlclpxdJcSaBrNwO/OAztpXLkb/H0K+Mhl DwZMgRyXm9JelPIqpGO5/8Sel0S1Dw8/cec7wdOwJJ5IvfoI/XQXFvLEuZFaZbgGkRbx xCUinwRRieL1XtWOzYXz1902F/KrZBJNEe/YNFhTWRNyBRDeQFcbUgcjQcrYo1BXNQ9o qhMTxglH7xIhxIl3AMNN5zQ2+0COiBXb8fGMhse7AOi13NSz9VXDx3+Jr1R5UGa6K1px 84LGdAlslvwUe5Qp6PNScn/6sLTX3TagO6C1LWky+osP3WDoQQLut6hVTV4e4KUybyuW QOVQ== X-Forwarded-Encrypted: i=1; AFNElJ/NP8IgWe6p9AygdsJHejRAdI482Njeldkh2Hj5f73cU5/LTbLSX91ERPygEqPPtrwy+jCFycwuuDlK0Kg=@vger.kernel.org X-Gm-Message-State: AOJu0Yw72itjdGsPKv3qlln8aGKwGDyLHwYDPdO3+rgNdtlnHE+gCx6F wGih9YpyXnTYdjwPOiLKS4oU7yHqqtF6IReN0Wt+HEjykcRtn2SGpzoHmTr/Y+tgQaZmcPKx9cx kCLMsDw== X-Received: from plblc11.prod.google.com ([2002:a17:902:fa8b:b0:2b0:6cdd:87e2]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:fc8e:b0:2ba:839e:15cb with SMTP id d9443c01a7336-2bf368463e6mr21255845ad.27.1780093384365; Fri, 29 May 2026 15:23:04 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:01 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-19-seanjc@google.com> Subject: [PATCH v3 18/40] KVM: x86: Move kvm_caps and kvm_host_values to asm/kvm_host.h From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Relocate the kvm_caps and kvm_host_values struct definitions and their associated global variable declarations to asm/kvm_host.h to allow for a variety of cleanups in x86.h and mmu.h, and to establish a (hopefully) maintainable rule that asm/kvm_host.h's role is to define common structures (and declare any associated globals), and anything needed by arch-neutral KVM. While it would be lovely to trim kvm_host.h down to the point where it *only* holds things needed by arch-neutral and/or non-KVM code, multiple attempts to do just that have failed miserably. Trying to "hide" code from arch-neutral KVM is too restrictive (and ultimately pointless), and KVM x86 itself also needs a place to define common structures and their globals, e.g. to avoid inconsistent header include chains and/or misplaced helpers. E.g. as pointed out by Kai, it's weird that x86.h, which is a kitchen sink of sorts, includes regs.h, but not mmu.h. Literally the only reason that x86.h doesn't include mmu.h is that mmu.h references struct kvm_host, which is currently defined in x86.h. As a result of odd include ordering, the very clearly MMU-specific helper mmu_is_nested() lives in x86.h, not mmu.h "Fix" the kvm_host dependency so that x86.h can be the "central" include everyone expects it to be, and set KVM x86 on the path to having somewhat sensible "rules" for what goes where: - asm/kvm_host.h holds "common" structure definitions and associated key global variables, and things that are referenced by arch-neutral KVM. - .{c,h} holds relevant declarations and definitions. - x86.{c,h} is the kitchen sink for everything else. Cc: Kai Huang Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Reviewed-by: Yosry Ahmed --- arch/x86/include/asm/kvm_host.h | 44 ++++++++++++++++++++++++++++++++ arch/x86/kvm/x86.h | 45 --------------------------------- 2 files changed, 44 insertions(+), 45 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 2e535027dd5c..f7130eb98473 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -315,6 +315,50 @@ enum x86_intercept_stage; struct kvm_kernel_irqfd; struct kvm_kernel_irq_routing_entry; =20 +struct kvm_caps { + /* control of guest tsc rate supported? */ + bool has_tsc_control; + /* maximum supported tsc_khz for guests */ + u32 max_guest_tsc_khz; + /* number of bits of the fractional part of the TSC scaling ratio */ + u8 tsc_scaling_ratio_frac_bits; + /* maximum allowed value of TSC scaling ratio */ + u64 max_tsc_scaling_ratio; + /* 1ull << kvm_caps.tsc_scaling_ratio_frac_bits */ + u64 default_tsc_scaling_ratio; + /* bus lock detection supported? */ + bool has_bus_lock_exit; + /* notify VM exit supported? */ + bool has_notify_vmexit; + /* bit mask of VM types */ + u32 supported_vm_types; + + u64 supported_mce_cap; + u64 supported_xcr0; + u64 supported_xss; + u64 supported_perf_cap; + + u64 supported_quirks; + u64 inapplicable_quirks; +}; +extern struct kvm_caps kvm_caps; + +struct kvm_host_values { + /* + * The host's raw MAXPHYADDR, i.e. the number of non-reserved physical + * address bits irrespective of features that repurpose legal bits, + * e.g. MKTME. + */ + u8 maxphyaddr; + + u64 efer; + u64 xcr0; + u64 xss; + u64 s_cet; + u64 arch_capabilities; +}; +extern struct kvm_host_values kvm_host; + /* * kvm_mmu_page_role tracks the properties of a shadow page (where shadow = page * also includes TDP pages) to determine whether or not a page can be used= in diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 80ed36d5d62a..b7d3b54cde15 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -12,48 +12,6 @@ =20 #define KVM_MAX_MCE_BANKS 32 =20 -struct kvm_caps { - /* control of guest tsc rate supported? */ - bool has_tsc_control; - /* maximum supported tsc_khz for guests */ - u32 max_guest_tsc_khz; - /* number of bits of the fractional part of the TSC scaling ratio */ - u8 tsc_scaling_ratio_frac_bits; - /* maximum allowed value of TSC scaling ratio */ - u64 max_tsc_scaling_ratio; - /* 1ull << kvm_caps.tsc_scaling_ratio_frac_bits */ - u64 default_tsc_scaling_ratio; - /* bus lock detection supported? */ - bool has_bus_lock_exit; - /* notify VM exit supported? */ - bool has_notify_vmexit; - /* bit mask of VM types */ - u32 supported_vm_types; - - u64 supported_mce_cap; - u64 supported_xcr0; - u64 supported_xss; - u64 supported_perf_cap; - - u64 supported_quirks; - u64 inapplicable_quirks; -}; - -struct kvm_host_values { - /* - * The host's raw MAXPHYADDR, i.e. the number of non-reserved physical - * address bits irrespective of features that repurpose legal bits, - * e.g. MKTME. - */ - u8 maxphyaddr; - - u64 efer; - u64 xcr0; - u64 xss; - u64 s_cet; - u64 arch_capabilities; -}; - void kvm_spurious_fault(void); =20 #define SIZE_OF_MEMSLOTS_HASHTABLE \ @@ -417,9 +375,6 @@ fastpath_t handle_fastpath_wrmsr_imm(struct kvm_vcpu *v= cpu, u32 msr, int reg); fastpath_t handle_fastpath_hlt(struct kvm_vcpu *vcpu); fastpath_t handle_fastpath_invd(struct kvm_vcpu *vcpu); =20 -extern struct kvm_caps kvm_caps; -extern struct kvm_host_values kvm_host; - void kvm_setup_xss_caps(void); =20 /* --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4754A44A71E for ; Fri, 29 May 2026 22:23:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093387; cv=none; b=Z+kowNSsKSlscdZr5Pdy4szOAgGJ3VJulPuly/nFohJv8WweTnrLh75I2hlzRx7hmPJ9rzgSbNCsYdmMiMjmvsesietQTBVK/9mUS2Q3FJWYjNVCEcl4XvcuJWH2wYIhEusIdnCB8nksyWZqWh8Sn8L6GNXlxAw1Eq33lIp4Qq0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093387; c=relaxed/simple; bh=tnvCBMKREAgQlP+QG6F27Ubo/7PoA8ieAPgptJnds2A=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=gGsXuL/sWSFERVOPDaxdsNeibn0JfBi9JnCUEj4FyFRrWg1RXEM8uO+xJc5J/+lfzSni6zTtAFgx7y9Zbe/cHEaP4efMqxC4xmS0Jr7Op/xRYU5yUrLPcfGKkFD6e2j7ricLddRcP1jmVz8sRfzHYPFEHJ8JgTSusNCdywADcp8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=TM/QxvlL; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="TM/QxvlL" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-36642d2f4deso27143576a91.3 for ; Fri, 29 May 2026 15:23:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093385; x=1780698185; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=mEqYYKIFBUgeehbwG3cS85rQVU90faoGahhTIj84wEA=; b=TM/QxvlLQImd25S4ZiLijX0gb1tFjSsk7j0Y07V0FJuKKe2vzvEusWLzKiQw9v5G3c 6TACzJ50lUEB4etQIbLrEQ7S16PZRliDTn2NhdQLs+xqkDu12EdmugPZKCjMex6y1xi0 46jwSdrglXS/wxEqFB3O3bTulQ90sOaWcv6Lmte6sKqc8PAwfYjChiR0Ap5fjGIKYAdS h2InRsGHvJ8PJCq6PgOPQ2VWo9wfX4sL9eS8MIaB5XWHVk/Nq0I39nP9Z+RZKfUDrNAg gqyiZRyfC5Ko+F4ckY9g3QaSLE2ZcYLSyujFBh88PDlyQ8tbgjOd+vEvKwxdX251yxYM TsnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093385; x=1780698185; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=mEqYYKIFBUgeehbwG3cS85rQVU90faoGahhTIj84wEA=; b=lGXP1ROJxmTLwNhycU/7oXWrLodlmSSgoyTgpW6F9k+E7JUj2kfsmTaBHOj4v4TE3K YRI6NfwMhLlR/Jk5pQb1QVbxrM8ai45ui7wWN2WLhFSyUoTFV/OToZ8mdm4pSjvPG2Ou Occk999x3NwcQq1BZSxCDpdqjL8rgoNZxhzSdD0oDrFbjidZS2NHr5bkJdkLBrIWMWsD AZ89T797nkua2tCfFfTDQWlvtt5L96IsLdDG1gadfUx/+lzlBHwOl9B1AZzc8yiGIXJe X0mHvQE3Na2SlvK3O3KNhHQMI+QEb5lH7Td8iSUYD0a4W7fH0SzfHb6oU+OIaQPJfm4v vP4w== X-Forwarded-Encrypted: i=1; AFNElJ+nT8AbNsXLok7hZ6lK0EW0i+A6/pV78UHKw37PubIz+IAvTTppsQnCdRvD/1OuGu8DTjuhD6KQXG4Nc00=@vger.kernel.org X-Gm-Message-State: AOJu0Ywed+YaXRQcz1LyIJBLZpsFi8RYjP0CKQ8aURSI4Dn8jsxyWKla RFoICyY5PeFhoWm6ULrpp/RLsFvg/f/H3B8iV79i4l0gmhbcPRi7EZ8NMuspi0xZhZ7drbjo3VW k9g5SyA== X-Received: from pgh1.prod.google.com ([2002:a05:6a02:4e01:b0:c79:8102:80bb]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:b94:b0:3b3:bf95:f7e3 with SMTP id adf61e73a8af0-3b427f81b40mr1274332637.43.1780093385457; Fri, 29 May 2026 15:23:05 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:02 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-20-seanjc@google.com> Subject: [PATCH v3 19/40] KVM: x86: Swap the include order between x86.h and mmu.h From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Invert the include ordering between x86.h and mmu.h, so that x86.h is the "top-level" include for KVM x86. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang --- arch/x86/kvm/mmu.h | 6 +++++- arch/x86/kvm/x86.h | 6 +----- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index e1bb663ebbd5..28fca48dcf64 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -4,7 +4,6 @@ =20 #include #include "regs.h" -#include "x86.h" #include "cpuid.h" =20 extern bool __read_mostly enable_mmio_caching; @@ -300,6 +299,11 @@ static inline void kvm_update_page_stats(struct kvm *k= vm, int level, int count) atomic64_add(count, &kvm->stat.pages[level - 1]); } =20 +static inline bool mmu_is_nested(struct kvm_vcpu *vcpu) +{ + return vcpu->arch.walk_mmu =3D=3D &vcpu->arch.nested_mmu; +} + static inline gpa_t kvm_translate_gpa(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, gpa_t gpa, u64 access, diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index b7d3b54cde15..a0e68eaf1f80 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -6,6 +6,7 @@ #include #include #include +#include "mmu.h" #include "regs.h" #include "kvm_emulate.h" #include "cpuid.h" @@ -210,11 +211,6 @@ static inline bool x86_exception_has_error_code(unsign= ed int vector) return (1U << vector) & exception_has_error_code; } =20 -static inline bool mmu_is_nested(struct kvm_vcpu *vcpu) -{ - return vcpu->arch.walk_mmu =3D=3D &vcpu->arch.nested_mmu; -} - static inline u8 vcpu_virt_addr_bits(struct kvm_vcpu *vcpu) { return kvm_is_cr4_bit_set(vcpu, X86_CR4_LA57) ? 57 : 48; --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 79F0644CAEC for ; Fri, 29 May 2026 22:23:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093388; cv=none; b=D8pLbLarMMzCIBZrJYfeusMaCIhsrTd2fLce9w2g9LmEGLR/g9fdYsaQczMM+0dFYG26b0aoJCkWVEYC7leIXhCKZnnLuWT/35j27gneYAW8VXAhvtDX/2tU8Dq4NG5+2IeJpUrPjdd/SjqWND3oD+8f1akhX8zzjLJ0ySpD4h0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093388; c=relaxed/simple; bh=3bWgCpS2MbNhs++wve21pEQGVhhVlqSYucSuMxg7WfU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ljVOvE2vF1HBbc/fYS9HYIKja518O/xtuLJ0CLX+DPJI0VxvPIZVs4AoIffWqObYB5DGn/xDqpZJ/ZBGxUTQikbtjX4c/cC45GpAUyOgXeZ7+xJuoqmd6Bde1jWWPAl1ILhrUDYYoGUOdbmow5PJ3WLRH2YFcdjrxsxBhK1h4x0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=DnMzeHGD; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="DnMzeHGD" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-c85868c32b6so274449a12.0 for ; Fri, 29 May 2026 15:23:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093387; x=1780698187; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=z4vKYiOMnnjQ6reYOu2qF0Gs5UxLrj65gv1yx1wc+84=; b=DnMzeHGDitw8KxuzKkhnriPV5vF7Ij/+PuXXEgP6t5sprmpIokgLBPirv9Wm237O6P TmeSHAJlI5lQzoAAWo0JCSRuJ72q1o5x7I9AMCjAk7jsMrLTSy1OBcpvo8Vo4gK5DYWo VGHS02HxOMUSHCKNPACB1iQbvjlqi2U695bt0vFSLecpyyIiWzr9CoB0RAwc0YtETlRQ KZ9H2QIlp9yV27xQC7aBanrXKZls+ZFTpRwfpaqI04uFt2jHBgTxLPIWGPpye2gq9rsE 6Hjv/Jy9hWL8ozacCfDxLZhdViaUqk98pKfnYL+zsy3RzMC4ib3G3Ifa9LnGVXiTsfqk qKJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093387; x=1780698187; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=z4vKYiOMnnjQ6reYOu2qF0Gs5UxLrj65gv1yx1wc+84=; b=jlHsy25jue2+M63cZYKOc3tqKe3FPWlosIKW6mbT3muPOE7YQFqwhw3YBTb6an4Tfj IXTc0Rfh+7rAVFvltiyhVRMXQWxyWATWXMzt34QzHSfUH1FNGuT4VQXddy8F862OqWcv TX+fcm1oSwZqvUvs0bA/mlgDgrmDjd4WXWJtKWLqKbmxDtYnV8EIS3jT+QlcO8uQ83RH uwCzVM6EN/guljCjBFSIi0VTUUllDcSqMNTFBLDa+b0Duk9RQrY9nMYC1pRSMFX8LPn8 xKOQ5YIsFmDYlumMIvjvQegd3l15nWJe8+Rm4B8Mx+Oo5a0dBYEnlRJRIRxfCW8/vytt mNcQ== X-Forwarded-Encrypted: i=1; AFNElJ8DC60pfT5hTihgh7Xt0hEkgHDJwUwV/n9rZX4hDngIYTmbco1fXApftDViLJRQ5tcRD9YaRpa6vrtyVJ4=@vger.kernel.org X-Gm-Message-State: AOJu0YwPg5hY4DaLe6U4IZ3AiIpSWlXJJmcYPnMA9dnE0D8rBY/NO7Dn EDXXMD8raoZRv7+emHm72HbaLU7H4KvcBrreGyyd3OpIkrn+0VswIQfyyG2kgr7XVlWBETLQc39 etYITeA== X-Received: from pgcq19.prod.google.com ([2002:a63:7513:0:b0:c76:3f60:53ff]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:6017:b0:39f:a42:9251 with SMTP id adf61e73a8af0-3b427f2af2cmr1232182637.22.1780093386530; Fri, 29 May 2026 15:23:06 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:03 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-21-seanjc@google.com> Subject: [PATCH v3 20/40] KVM: x86: Move tdp_enabled from kvm_host.h to mmu.h From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Relocated the declaration of tdp_enabled into mmu.h, and opportunistically hoist tdp_mmu_enabled up to the top so that the two are co-located. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Reviewed-by: Yosry Ahmed --- arch/x86/include/asm/kvm_host.h | 2 -- arch/x86/kvm/mmu.h | 12 ++++++------ 2 files changed, 6 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index f7130eb98473..19091d89d3cc 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2188,8 +2188,6 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_sta= rt, gfn_t gfn_end); =20 int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3); =20 -extern bool tdp_enabled; - /* * EMULTYPE_NO_DECODE - Set when re-emulating an instruction (after comple= ting * userspace I/O) to indicate that the emulation context diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 28fca48dcf64..0eaea2d4fac9 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -6,6 +6,12 @@ #include "regs.h" #include "cpuid.h" =20 +extern bool tdp_enabled; +#ifdef CONFIG_X86_64 +extern bool tdp_mmu_enabled; +#else +#define tdp_mmu_enabled false +#endif extern bool __read_mostly enable_mmio_caching; =20 #define PT_WRITABLE_SHIFT 1 @@ -260,12 +266,6 @@ static inline bool kvm_shadow_root_allocated(struct kv= m *kvm) return smp_load_acquire(&kvm->arch.shadow_root_allocated); } =20 -#ifdef CONFIG_X86_64 -extern bool tdp_mmu_enabled; -#else -#define tdp_mmu_enabled false -#endif - int kvm_tdp_mmu_map_private_pfn(struct kvm_vcpu *vcpu, gfn_t gfn, kvm_pfn_= t pfn); =20 static inline bool kvm_memslots_have_rmaps(struct kvm *kvm) --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7287244D685 for ; Fri, 29 May 2026 22:23:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093389; cv=none; b=lBAt3Dxr2tLFcFOOvd20ERY3Q91YmM7umlgPNY3w7r6w1Uj5lZ8LaQkiG3J42jDx9jkeitEo1TE5V5JPhPpUfIEw6B+iY6CE0EHN8Oe+9t1526joVm/zl8Ardo7YMUXbX9eBmxbGgg1KxZ10wt0aZckmVryu6TQjGCBTng6i5zE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093389; c=relaxed/simple; bh=Ti1Cn/2X567vIP0NAUi9gjSta7WxFw3xhtt5JfIZGN4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=EkYUNpV6htHE2kmApn6tc9HaphwIC52sjgXLM2DQfMlk354X5EDEXKhoepv4pwKSGOv7wOmYIhQUS7aweDGtBrFaNimMlm479cyMkLV4a92m8KegMOz0hqWcnRWY7ON2khwMQJrcavAmdKaaC0UBwPeuwTc7cWt3TGLZ84Nvs10= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=SeK0Aelb; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="SeK0Aelb" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2bf1845bddfso16186905ad.1 for ; Fri, 29 May 2026 15:23:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093388; x=1780698188; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=C+zNop4iFEAB4FpwyQmbvIE0zFzjyJkNdQov+dfVU9Y=; b=SeK0AelbGS0+kvO7zIBBx3oCqA9pIFhqI0U7jcrJy38jHGfxx7DWirJ50GbFX9KLNx +FVIMSMQt+R/jkGKC3fp3VIU7itGGXFn7jOkioMxqVLeuRFD9NUIXNZTSIexDueyVpMa TSmWCPtSRKO+oX1mqzpumj//xRSx2XacMfhAvPV2c8YWZt6IoePcAsJJIebplUfO3Mzs +/sWCgJaIyTc19A4Jdafq1tMldec66lbMEzRfGcf1axKqgss84ZWun4/7NTIfpRIFgIJ VSB0rUvT4o6uvYd8K/8LaWIvCvqlgDwSbXj4nfvsU8Qml6raY+9oGru9MifV8Zah6dWV MreA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093388; x=1780698188; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=C+zNop4iFEAB4FpwyQmbvIE0zFzjyJkNdQov+dfVU9Y=; b=CAWTJQ415HV/h1vqvTdDd8XiWZbgXn5SOxmEtIFZmqzsYoux31PM5+ux/H6nGWjTF6 C0Yn4UdRH+Sh/u6owWvZzFyh9zau0SS7xa3Ga1dnmltNdHRLpSTKnGmIcI8DRlZHy4QB d4DZAHYgvor/gEM9p7c4ABxmG4+mZPyfoUm9Eq9/ULXZabNrewHFQ4V4ZZp2b5zni26u rsKjJYALFg1nXfX7AokmIuNXUGcHoodk2oMpuKMN7xcezwg/nJiG+dgnw0KNA+aENldk 0td+Ks2UaKWo6gJUD6Hfds1/Z6HzQRfSyCh5b8OPjglTvf7XROn1MHmLicCkVvn4qUHu RycQ== X-Forwarded-Encrypted: i=1; AFNElJ+ug4IqFGFRJYSBw8MQbe++bBn/pZnPzDM5bcV2rmdb3kCPPreqcJ9hc6LXCRPQEAN0PHswDz2XbtyRStQ=@vger.kernel.org X-Gm-Message-State: AOJu0YyRQ3LBmogoJ7wHKVbxc9OnsdiWSnRvZtrNZ4TuTHnk/BPNmYBj KyN+Y7ECFjFpJ0OuWuV547KamNlccvqe0Pn/SLVzNY6LLpe+IIxam1rQGZNdzYzBbUVX+eoRFa0 bD/mOQQ== X-Received: from pgww11.prod.google.com ([2002:a05:6a02:2c8b:b0:c85:f23:dde3]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:e88c:b0:2b2:4d36:7aa with SMTP id d9443c01a7336-2bf368859d8mr18740315ad.35.1780093387619; Fri, 29 May 2026 15:23:07 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:04 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-22-seanjc@google.com> Subject: [PATCH v3 21/40] KVM: x86: Move eager_page_split to mmu.{c,h} From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move KVM's eager_page_split module param to the MMU, as it is very much an MMU knob. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Reviewed-by: Yosry Ahmed --- arch/x86/kvm/mmu.h | 1 + arch/x86/kvm/mmu/mmu.c | 3 +++ arch/x86/kvm/x86.c | 3 --- arch/x86/kvm/x86.h | 2 -- 4 files changed, 4 insertions(+), 5 deletions(-) diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index 0eaea2d4fac9..d30676935fff 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -13,6 +13,7 @@ extern bool tdp_mmu_enabled; #define tdp_mmu_enabled false #endif extern bool __read_mostly enable_mmio_caching; +extern bool eager_page_split; =20 #define PT_WRITABLE_SHIFT 1 #define PT_USER_SHIFT 2 diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index b8f2edf2cfeb..e4d971d42f0e 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -114,6 +114,9 @@ module_param_named(tdp_mmu, tdp_mmu_enabled, bool, 0444= ); EXPORT_SYMBOL_FOR_KVM_INTERNAL(tdp_mmu_enabled); #endif =20 +bool __read_mostly eager_page_split =3D true; +module_param(eager_page_split, bool, 0644); + static int max_huge_page_level __read_mostly; static int tdp_root_level __read_mostly; static int max_tdp_level __read_mostly; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 20eeff79b46d..be421f467563 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -177,9 +177,6 @@ module_param(force_emulation_prefix, int, 0644); int __read_mostly pi_inject_timer =3D -1; module_param(pi_inject_timer, bint, 0644); =20 -bool __read_mostly eager_page_split =3D true; -module_param(eager_page_split, bool, 0644); - /* Enable/disable SMT_RSB bug mitigation */ static bool __read_mostly mitigate_smt_rsb; module_param(mitigate_smt_rsb, bool, 0444); diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index a0e68eaf1f80..635a21bfa681 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -415,8 +415,6 @@ extern int pi_inject_timer; =20 extern bool report_ignored_msrs; =20 -extern bool eager_page_split; - static inline void kvm_pr_unimpl_wrmsr(struct kvm_vcpu *vcpu, u32 msr, u64= data) { if (report_ignored_msrs) --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A1F66450911 for ; Fri, 29 May 2026 22:23:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.210.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093391; cv=none; b=s5zoYVwgpMl5gWLf/P+1HrHY1HENudHqDJYk8MG6bzC+OP0fiEZdlDBou+KPaNIv6yqlmjGuqx3YsbAyOb0iFP1cpLBe0ceIHAdNRUatHtpXyS/+khbmBoNTKCcPySPzFy/tw+tmjq4OkXxmtWUaybqj+1Z8BgrKD9T7CJgwND0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093391; c=relaxed/simple; bh=2QUq4Zntx/cN1Mlgi5UbFZ2Zk6wGbdYV96fOq/ek52U=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Ip8JUDSdO6sRIsVrT/Q7z8PtPNd2DKQegCVQ2WM1awx2mYdwKXfz4fsux4xc2rPkHmD+pG/t3XjuG9If00FxIx7a0QTPTeVu779CZgJIlTJhYCXJYIuTmMJsZsOBaD4mqru1LMc5k7KMscvpc7MIPxGNNrLY0ZHlU+p4jkPke9s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=pLVI+PNc; arc=none smtp.client-ip=209.85.210.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="pLVI+PNc" Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-8422b544a4bso42714b3a.3 for ; Fri, 29 May 2026 15:23:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093389; x=1780698189; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=cOprH2r2W4OfoUuNhxoH+2jlBeBKL8+G7nnMuVk5ujk=; b=pLVI+PNcj+/oJ1AC7B9DHSlHIxRdpuLve2yQoBHKCqB31uXrbJWek7nm0dvEplNdoF DuspBxB/ecUfmVPXAua1r8e20riG0ptA7upprXGzcrxrY2GGQx6BVCKcbYKRYWp8SZfl Z0M7RkzIzg9dEEhFUut1xEEOlGX5llOASJbBCcAMEIShdaiasURvPv/7MJ4IPLm8cnJc 2pvkUf7yVlhtWp0ZCANRwTfiCCjLGSjTfNWltB+AVKgWFIwoFl3LDvEvC80NS4OtJ1T2 dBkUeRPjdTXUMeuAO1kGjs8+NTeRCsRRFeuecGNqSApfUWI5kM4neZWiM6MTveLThiMM 8AqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093389; x=1780698189; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=cOprH2r2W4OfoUuNhxoH+2jlBeBKL8+G7nnMuVk5ujk=; b=Qj8ab6MsIRQmeMcThk0KWhdFibHXU9LAw7FbzlrF5twaYqS6qVhMNF30vRiIozsqXc 7rj/4iuI7OvW/xksXKszjQ6HUsi8kCOnDcztDbUNwogzMJ5X49zMveChangF4eqv97sK 6EQWU9nwImQT0k3MVJOK8/e1IsRvPY/bHxkWXZIrN5m1aVIeCXPRQs1giYIgn9FAuEtf 4DNNWnP0DYiwHcGty2ChV+CYe0ue9f45xnzNPIODe5SrStS7mHytXNwGwlqBrLVQDDpV lbmDmyD1kmrpsnMp2CoSGVmNALfdcrSqv39XT8wNRhHIbNbo9N5EDd6wbBLzzNsetaOg H4Ig== X-Forwarded-Encrypted: i=1; AFNElJ8wjR+iTk2hc0ZCbNv/7bigtV/AqiaEmWNJomB9vnIP6GeS4Y5+i/vPvYofr3T79gCTUyWzf61DBGi/R40=@vger.kernel.org X-Gm-Message-State: AOJu0YwAMCBCBfwRQ1lAQ7sj9L+ApW6OxCN3bCJ2VxnPPZ8xKIboikj5 gq43rF+COD+4+MMrg1N/FJllHwx5bN7DYmxaf9Db3fq0XmtbM4+eJcI4vZSSyZpNEQTTMAin7dU gg2IUow== X-Received: from pfbjs21.prod.google.com ([2002:a05:6a00:9195:b0:82f:96ee:b9ad]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:4291:b0:835:4291:6975 with SMTP id d2e1a72fcca58-842255c48camr1437165b3a.39.1780093388651; Fri, 29 May 2026 15:23:08 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:05 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-23-seanjc@google.com> Subject: [PATCH v3 22/40] KVM: x86/hyperv: Eliminate an unnecessary include of x86.h in hyperv.h From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Drop an mostly unused include of x86.h from hyperv.h, and instead pull in regs.h, which is need for at least is_guest_mode(). This eliminates the last include of x86.h from a common x86 header, i.e. solidifies that x86.h is the top of the pyramid. Add a missing x86.h include in cpuid.c to avoid build breakage. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang --- arch/x86/kvm/cpuid.c | 1 + arch/x86/kvm/hyperv.h | 3 ++- 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c index fd3b02575cd0..db8be9173bd0 100644 --- a/arch/x86/kvm/cpuid.c +++ b/arch/x86/kvm/cpuid.c @@ -28,6 +28,7 @@ #include "trace.h" #include "pmu.h" #include "xen.h" +#include "x86.h" =20 /* * Unlike "struct cpuinfo_x86.x86_capability", kvm_cpu_caps doesn't need t= o be diff --git a/arch/x86/kvm/hyperv.h b/arch/x86/kvm/hyperv.h index 65e89ed65349..1c8f7aaab063 100644 --- a/arch/x86/kvm/hyperv.h +++ b/arch/x86/kvm/hyperv.h @@ -22,7 +22,8 @@ #define __ARCH_X86_KVM_HYPERV_H__ =20 #include -#include "x86.h" + +#include "regs.h" =20 #ifdef CONFIG_KVM_HYPERV =20 --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8A5A845BD6C for ; Fri, 29 May 2026 22:23:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093391; cv=none; b=ILuHr2RJN+qTJ+yrgY+Xc2RbK2Dovz1jSTSwqmwti9M36DcP58VHQyXTShkjrTO5foYafpii67GIRKdYLa4ihkhVNvFdsNxr2Ru7McJDLLKwkB9jPNwkznVXTfaiWEtFdR+TTyL7JF2g4FYQbzDV9nl+9e+++0uk86dPZwPvsn0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093391; c=relaxed/simple; bh=EGPfyjJEkMRALBldQbQlTBqALJmI6WLTaQUvqWM0994=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=msOPDYDVtuVECsCO86yapOkgHnsepcqoIUxjJ8M9zLVdmcnR+eJGPKWOacQ9+ncdYPVlHY0q14jJ0qoEQUzUENClX9ADzzEdAOeDR5sVk7L1Il+RjVMBxYwkD19U43bCeMge38sixQXgIEB4QTbTIH/a922wcaAcxS9QP7sexjM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Bqw6aNHI; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Bqw6aNHI" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2bf30576aa3so4715385ad.3 for ; Fri, 29 May 2026 15:23:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093390; x=1780698190; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=LNxa5lWov+tsCDMl+sF7sk9ofNMegwWDjQlcUaSncIw=; b=Bqw6aNHIVVFRiQFr6DSnyKl3ZHU8Xxkk5DhGuvYZsjJ+WcQHbuWpuwh+usemDrdawF z+XPC1WH4AfWyXFhPCWzr0j74mAl7lr8ePZwtLYDyiqkYF7WY0VlfA7DyxcmrKpT3lnN j31Vyzr6YouK6LciKEZIy7pjzRJBfJiB1Ss+7heSzT+ETy0/84KNuCfFi67S2nqJYi9e nYHZQe/7IA9tSW/foFFCVFmN6OQLxV9pUMRgeRzb2VZliV8wn3kWiwvZSWC/lzIy9T8E NXBkdeRVD9kvwWI2KnWO8LUDSzyYotYvYSBpCBsKWJl1HXd0K/Awi3UBUD9q33kWMPZW ItyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093390; x=1780698190; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=LNxa5lWov+tsCDMl+sF7sk9ofNMegwWDjQlcUaSncIw=; b=pPLo2YaiIOxEkWhOEN3lXFREqXG4CxK8oM3vARNB9wx1B7ap5iJWm4SCuyDfrhpya/ V9oMv5sHXca5m2nnrAW0PLs4PQVJIORLZ+lqIwa9yAkuMIcPPBdBfjyMBOUGC4blBM2w 3lo8gLCvbtT3DCGC25o7zr90+GgW+deJvFwDiwkIP4O8N5xccOwHTDKbs57Irp49i0FX TdZxe47Qq1QfSxZTWMth6+iBaIWtS/tlzfdAWi/uWEqt42LGhZmyBS38jFo54xPJLvJr JytdWkaMWkdI9X2XQ7KYM60VxpWSlNKhpOTFnH6xHZiT536h7eGZO81pOdhZLs4fNQr1 Addw== X-Forwarded-Encrypted: i=1; AFNElJ+vX71CuEdEwrUbOPVM1c8UZQ66BfhZX2CrX81aNM5jcZq34LTGjKW6BPlERl41ePcjpdCGya7ZprfZfI0=@vger.kernel.org X-Gm-Message-State: AOJu0Yw/TJ8M3MRFwac0sDhIDD7YB4nzqCiX36Od4GZFfM498zbZTCK3 9aANs6gu9CHJx91mc47omEb4JtfxslYZEqgy/EsSKXEfJxgUoPWdPWO3lKHNC8wgefU0FDjiBUO RCvl+pw== X-Received: from pllg1.prod.google.com ([2002:a17:902:7401:b0:2b4:5be2:f5a6]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:cf12:b0:2bc:cf06:6835 with SMTP id d9443c01a7336-2bf36798747mr20427575ad.8.1780093389821; Fri, 29 May 2026 15:23:09 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:06 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-24-seanjc@google.com> Subject: [PATCH v3 23/40] KVM: x86: Move kvm_{load,put}_guest_fpu() to fpu.h From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move the kvm_{load,put}_guest_fpu() helpers to fpu.h in anticipation of moving the bulk of KVM's register specific code out of x86.c. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Reviewed-by: Yosry Ahmed --- arch/x86/kvm/fpu.h | 26 ++++++++++++++++++++++++++ arch/x86/kvm/x86.c | 24 ------------------------ 2 files changed, 26 insertions(+), 24 deletions(-) diff --git a/arch/x86/kvm/fpu.h b/arch/x86/kvm/fpu.h index f898781b6a06..6b7b628f530d 100644 --- a/arch/x86/kvm/fpu.h +++ b/arch/x86/kvm/fpu.h @@ -3,8 +3,34 @@ #ifndef __KVM_FPU_H_ #define __KVM_FPU_H_ =20 +#include + +#include + #include =20 +/* Swap (qemu) user FPU context for the guest FPU context. */ +static inline void kvm_load_guest_fpu(struct kvm_vcpu *vcpu) +{ + if (KVM_BUG_ON(vcpu->arch.guest_fpu.fpstate->in_use, vcpu->kvm)) + return; + + /* Exclude PKRU, it's restored separately immediately after VM-Exit. */ + fpu_swap_kvm_fpstate(&vcpu->arch.guest_fpu, true); + trace_kvm_fpu(1); +} + +/* When vcpu_run ends, restore user space FPU context. */ +static inline void kvm_put_guest_fpu(struct kvm_vcpu *vcpu) +{ + if (KVM_BUG_ON(!vcpu->arch.guest_fpu.fpstate->in_use, vcpu->kvm)) + return; + + fpu_swap_kvm_fpstate(&vcpu->arch.guest_fpu, false); + ++vcpu->stat.fpu_reload; + trace_kvm_fpu(0); +} + typedef u32 __attribute__((vector_size(16))) sse128_t; #define __sse128_u union { sse128_t vec; u64 as_u64[2]; u32 as_u32[4]; } #define sse128_lo(x) ({ __sse128_u t; t.vec =3D x; t.as_u64[0]; }) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index be421f467563..56ccb6b77abb 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -133,8 +133,6 @@ static void store_regs(struct kvm_vcpu *vcpu); static int sync_regs(struct kvm_vcpu *vcpu); =20 static DEFINE_MUTEX(vendor_module_lock); -static void kvm_load_guest_fpu(struct kvm_vcpu *vcpu); -static void kvm_put_guest_fpu(struct kvm_vcpu *vcpu); =20 struct kvm_x86_ops kvm_x86_ops __read_mostly; =20 @@ -11432,28 +11430,6 @@ static int complete_emulated_mmio(struct kvm_vcpu = *vcpu) return 0; } =20 -/* Swap (qemu) user FPU context for the guest FPU context. */ -static void kvm_load_guest_fpu(struct kvm_vcpu *vcpu) -{ - if (KVM_BUG_ON(vcpu->arch.guest_fpu.fpstate->in_use, vcpu->kvm)) - return; - - /* Exclude PKRU, it's restored separately immediately after VM-Exit. */ - fpu_swap_kvm_fpstate(&vcpu->arch.guest_fpu, true); - trace_kvm_fpu(1); -} - -/* When vcpu_run ends, restore user space FPU context. */ -static void kvm_put_guest_fpu(struct kvm_vcpu *vcpu) -{ - if (KVM_BUG_ON(!vcpu->arch.guest_fpu.fpstate->in_use, vcpu->kvm)) - return; - - fpu_swap_kvm_fpstate(&vcpu->arch.guest_fpu, false); - ++vcpu->stat.fpu_reload; - trace_kvm_fpu(0); -} - static int kvm_x86_vcpu_pre_run(struct kvm_vcpu *vcpu) { /* --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C9C9C466B5B for ; Fri, 29 May 2026 22:23:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093393; cv=none; b=tSoNtw/kO844ulQGq0C2EgWTW8nIA8k+/wBkjjyUfR1uwRKBc0tGkLWYbbBXHKWZlGCoBROzDmSaDTvDgJzN5yGd6p03QroJl93MABHalP7mCYnYjrjMS0bTwF5MtHPqFheJJeIVCqn2+nEDZMQVpJVm8jjiTBzwOiERl09PCyg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093393; c=relaxed/simple; bh=U16q5I4nJv94rNXxPlbWDmKybiGw1J1K0KEk+xMy+jo=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=koFcqQ3icvX20Ex/vvc7n08FzqO1SrDcYl2B12X1SIb8JK2Nb2rPiVj2+vpYnyRpMuDCkC/eE1Xa0OKVD5F6lKaIF3lCeVKXkpGeXfbX2BMeYQnNrStrwATKORQmQ6PqKvqBY5i0WjuUNjV5BT4ID5k7VqFatnrQz5WydOPZxQQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=jrCH4fgx; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="jrCH4fgx" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-36bcfce8a33so622275a91.1 for ; Fri, 29 May 2026 15:23:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093391; x=1780698191; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=g41Ygn6Uf+1kQN9znuZTAZ86kg1U3hWntwdYhg6VgHo=; b=jrCH4fgxpAOldu3iKMoydnGt5gEkDhHaphzg7y1EsF8GvAQ1+OAoycVlPudH64FI1c tISjjglPLx7yVoBb+PwIBna3a7mHj2tkjev6G9bosur1mozYPaLblKox0YB2DszbI6RR shUK6T+FSJKrNgkDyRcwjpD7vswL/LkvvPU3asdMAT8R/mKG58Nm1PcpJyO4+6mSaEFk bJJnOj/OCVVImm3RqRhwARNsNxf1GOKFX7233nhaSE7snNIn4CBfBSupTMWAcnLSpC/N D8ZE6J3ZSbhjuLOChiloHKxIv+OGQx10ODWpcsVI5im9GEc98Auz38bAQrf6LZzaVO5R YUWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093391; x=1780698191; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=g41Ygn6Uf+1kQN9znuZTAZ86kg1U3hWntwdYhg6VgHo=; b=O+1GpuJ5krAqJXEIFd1ZwI2fO2LcT6WcCD7JexNWGzH49XU85vr4/GYAfkYUFTzaYJ xdD08qu28rOQhWD38LHg1P9TZpVnXIpuqwcFWSKJZX1y8Na5kt8s+/lC4MYY4/bidxon 8iJEZl8YOrwJaeEv/vblD6Exif8ZpWiqP/mMcd5/QR3pOk7FfvMtywzf4twIV5XCPhvg WUKiRzcMrPShyNtRDr7gd5FxWwTnuBzXCDQW1IoHafLk/KdU2b4ZsGnz/5PfAhSGhL0H CtQQVtNYmpRXm8odRt+EX1GHz5rZR2XYgcXgH2xfHJC/hrlBtqM4Ad0IOY8ATLpyF+fg xEAA== X-Forwarded-Encrypted: i=1; AFNElJ8T9FvDse6c+A7B7Fa5wNYqckatR9i61MzTi2oALhGKSAle51FUSwy0sC2R/xdu2v6UPpaOavamvUcnWNU=@vger.kernel.org X-Gm-Message-State: AOJu0YzGUYcNLBR+qSiKnplxJxq9aIj5nYDGnd/VPcgblvfc27KVsG5n P3RZl/K1qEU0SEWCrZjlyfCIGQGE0viUb0MJVm0BRet2PMRNCHhm0adnvA+VV3rtx0v+e7S3Tzl HSNXNAg== X-Received: from pfbfx13.prod.google.com ([2002:a05:6a00:820d:b0:82c:90c5:5e0d]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:439f:b0:3a2:c7e1:ff89 with SMTP id adf61e73a8af0-3b4267562bamr1285241637.7.1780093390874; Fri, 29 May 2026 15:23:10 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:07 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-25-seanjc@google.com> Subject: [PATCH v3 24/40] KVM: x86: Extract get/set MSR (list) ioctl logic to helpers From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Extract the code for getting/setting MSRs and MSR lists to dedicated helpers in anticipation of moving the MSR code to a new msrs.c. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Reviewed-by: Yosry Ahmed --- arch/x86/kvm/x86.c | 135 ++++++++++++++++++++++++++------------------- 1 file changed, 78 insertions(+), 57 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 56ccb6b77abb..64c3680d889b 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -4602,6 +4602,61 @@ static int kvm_x86_dev_has_attr(struct kvm_device_at= tr *attr) return __kvm_x86_dev_get_attr(attr, &val); } =20 +static int kvm_get_msr_index_list(struct kvm_msr_list __user *user_msr_lis= t) +{ + struct kvm_msr_list msr_list; + unsigned int n; + + if (copy_from_user(&msr_list, user_msr_list, sizeof(msr_list))) + return -EFAULT; + + n =3D msr_list.nmsrs; + msr_list.nmsrs =3D num_msrs_to_save + num_emulated_msrs; + if (copy_to_user(user_msr_list, &msr_list, sizeof(msr_list))) + return -EFAULT; + + if (n < msr_list.nmsrs) + return -E2BIG; + + if (copy_to_user(user_msr_list->indices, &msrs_to_save, + num_msrs_to_save * sizeof(u32))) + return -EFAULT; + + if (copy_to_user(user_msr_list->indices + num_msrs_to_save, + &emulated_msrs, num_emulated_msrs * sizeof(u32))) + return -EFAULT; + + return 0; +} + +static int kvm_get_feature_msr_index_list(struct kvm_msr_list __user *user= _msr_list) +{ + struct kvm_msr_list msr_list; + unsigned int n; + + if (copy_from_user(&msr_list, user_msr_list, sizeof(msr_list))) + return -EFAULT; + + n =3D msr_list.nmsrs; + msr_list.nmsrs =3D num_msr_based_features; + if (copy_to_user(user_msr_list, &msr_list, sizeof(msr_list))) + return -EFAULT; + + if (n < msr_list.nmsrs) + return -E2BIG; + + if (copy_to_user(user_msr_list->indices, &msr_based_features, + num_msr_based_features * sizeof(u32))) + return -EFAULT; + + return 0; +} + +static int kvm_get_feature_msrs(struct kvm_msrs __user *user_msrs) +{ + return msr_io(NULL, user_msrs, do_get_feature_msr, 1); +} + long kvm_arch_dev_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { @@ -4609,32 +4664,9 @@ long kvm_arch_dev_ioctl(struct file *filp, long r; =20 switch (ioctl) { - case KVM_GET_MSR_INDEX_LIST: { - struct kvm_msr_list __user *user_msr_list =3D argp; - struct kvm_msr_list msr_list; - unsigned n; - - r =3D -EFAULT; - if (copy_from_user(&msr_list, user_msr_list, sizeof(msr_list))) - goto out; - n =3D msr_list.nmsrs; - msr_list.nmsrs =3D num_msrs_to_save + num_emulated_msrs; - if (copy_to_user(user_msr_list, &msr_list, sizeof(msr_list))) - goto out; - r =3D -E2BIG; - if (n < msr_list.nmsrs) - goto out; - r =3D -EFAULT; - if (copy_to_user(user_msr_list->indices, &msrs_to_save, - num_msrs_to_save * sizeof(u32))) - goto out; - if (copy_to_user(user_msr_list->indices + num_msrs_to_save, - &emulated_msrs, - num_emulated_msrs * sizeof(u32))) - goto out; - r =3D 0; + case KVM_GET_MSR_INDEX_LIST: + r =3D kvm_get_msr_index_list(argp); break; - } case KVM_GET_SUPPORTED_CPUID: case KVM_GET_EMULATED_CPUID: { struct kvm_cpuid2 __user *cpuid_arg =3D argp; @@ -4662,30 +4694,11 @@ long kvm_arch_dev_ioctl(struct file *filp, goto out; r =3D 0; break; - case KVM_GET_MSR_FEATURE_INDEX_LIST: { - struct kvm_msr_list __user *user_msr_list =3D argp; - struct kvm_msr_list msr_list; - unsigned int n; - - r =3D -EFAULT; - if (copy_from_user(&msr_list, user_msr_list, sizeof(msr_list))) - goto out; - n =3D msr_list.nmsrs; - msr_list.nmsrs =3D num_msr_based_features; - if (copy_to_user(user_msr_list, &msr_list, sizeof(msr_list))) - goto out; - r =3D -E2BIG; - if (n < msr_list.nmsrs) - goto out; - r =3D -EFAULT; - if (copy_to_user(user_msr_list->indices, &msr_based_features, - num_msr_based_features * sizeof(u32))) - goto out; - r =3D 0; + case KVM_GET_MSR_FEATURE_INDEX_LIST: + r =3D kvm_get_feature_msr_index_list(argp); break; - } case KVM_GET_MSRS: - r =3D msr_io(NULL, argp, do_get_feature_msr, 1); + r =3D kvm_get_feature_msrs(argp); break; #ifdef CONFIG_KVM_HYPERV case KVM_GET_SUPPORTED_HV_CPUID: @@ -5719,6 +5732,20 @@ static int kvm_get_reg_list(struct kvm_vcpu *vcpu, return 0; } =20 +static int kvm_get_msrs(struct kvm_vcpu *vcpu, struct kvm_msrs __user *use= r_msrs) +{ + guard(srcu)(&vcpu->kvm->srcu); + + return msr_io(vcpu, user_msrs, do_get_msr, 1); +} + +static int kvm_set_msrs(struct kvm_vcpu *vcpu, struct kvm_msrs __user *use= r_msrs) +{ + guard(srcu)(&vcpu->kvm->srcu); + + return msr_io(vcpu, user_msrs, do_set_msr, 0); +} + long kvm_arch_vcpu_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { @@ -5823,18 +5850,12 @@ long kvm_arch_vcpu_ioctl(struct file *filp, r =3D 0; break; } - case KVM_GET_MSRS: { - int idx =3D srcu_read_lock(&vcpu->kvm->srcu); - r =3D msr_io(vcpu, argp, do_get_msr, 1); - srcu_read_unlock(&vcpu->kvm->srcu, idx); + case KVM_GET_MSRS: + r =3D kvm_get_msrs(vcpu, argp); break; - } - case KVM_SET_MSRS: { - int idx =3D srcu_read_lock(&vcpu->kvm->srcu); - r =3D msr_io(vcpu, argp, do_set_msr, 0); - srcu_read_unlock(&vcpu->kvm->srcu, idx); + case KVM_SET_MSRS: + r =3D kvm_set_msrs(vcpu, argp); break; - } case KVM_GET_ONE_REG: case KVM_SET_ONE_REG: r =3D kvm_get_set_one_reg(vcpu, ioctl, argp); --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D05C846AF12 for ; Fri, 29 May 2026 22:23:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093394; cv=none; b=WTB8ZjVmc4HKNJdUlMiqcKMK7huhjh4EDPXFnK9PgZpobEw4gSuhocNf8uCIs77UIe/zCL5HUPBpP18il5Mh7uqMJfdJRyf/cgloV5nMd8/H2ae29wqh2KuHePtzcoVsml/JOQwAJzjbRG6E9QsNBaJVj/pu3G+uuA7IUuPZVT0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093394; c=relaxed/simple; bh=Uir6IDB0pwquVmxdNK5xocb4DeiDBsUvz1nObKQsLvE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=safgllN6Qy0uH99rhslsKSShLN9hpS7FEurIKzx1ZuZQIE69+bEGjw0Wrck+BqIrStLmTVYpesJqOy40EGNLEvPlzKQzdAJ+EGZJ5Wkxp9BzcKW+shf4AZrcZv+jW1cpaZMwMhnu0Dqe9yYatYrvvYCKF3ci3vRvN340niHpxOs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=B+/FjJFz; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="B+/FjJFz" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2bf08c2a24bso26328925ad.2 for ; Fri, 29 May 2026 15:23:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093392; x=1780698192; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=hdC/WRkAouMWi8Y4ZlrI0Bkm4osriIlBw4YhEZJnKi4=; b=B+/FjJFz+2oFdjMJGPkG02uJxCLImDyLZuBFPK/Ft86vq7P8+IBg4pPZgiMSD7gSlV ZpXwBie7f36Q65UOUe+5Vkh00pXt1Jh3v29kfkfY0y4VmXgGEJOB4EBzsazZsMWXaMCn MPb6V4IIthMHTZ+Om/WBtnX38Q+jnTbZ66hbKgJ8Gd/ASJp4CsTM/9HiXFRfSIMqddPb foibd96nKu+erv96s41lPWTWLUna+irkfYTHdHcEniJPXCZyEDVZRAQd4VIy7WBLh3Ra DJmNpP1Cfiqpl/EDpZI/6Mpe9hyr/+c0oX5l3a5gXAIJ5DjDBZ5qR34cewZ+shfEZf/t lRYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093392; x=1780698192; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=hdC/WRkAouMWi8Y4ZlrI0Bkm4osriIlBw4YhEZJnKi4=; b=VGT1TneluB+cClwKZsJpfo2k2vg5KRjOQyM7ZjFNK6PIOy08JqSYR9gZqsda06vm90 fAT9ce7sNs4sbIfLrpsSaQJq4nG8WktX49V2HTq5NGk/w8inbObwn96mjGzYlEWAwz6c kbfzwTLR4YxaNaHy5M2u/75wrFjR8Bq/bqu49JfpB+gevm+rYR+ZlYp3K9dczqT3CEgp 1efehtan9O3taQ6kksfExlYvBf797KVWcSVyuGGjrj3vqIbpJ/CZIAxH9NTxa+O7PP7C Pns8TKb7kQUvh82Z5VGjVE6BBGPwU0zWBG6YHUPGpzl9eGwYTmL9/DqpswhGR+5tD7+h M1CA== X-Forwarded-Encrypted: i=1; AFNElJ/70prhrFaZyW7f7O2vvfq61FYXap0EPh7mhZK+saO6xWEoO1JsbNlFtCuPngt0cNcZJF6nmHNuQiI16lw=@vger.kernel.org X-Gm-Message-State: AOJu0YxLl6ceDUivMq4vn+JcaS/CQysvDhD05Ry6fpDKUFZg6e26+bBZ HIVAX+OtiPlO/jyumkv50IZMW2NZ6HObagoLJVpI9JbAS3qdYBGvePUl1dZ3gppz/Fb84tfzFxk xSuJp4w== X-Received: from pldt15.prod.google.com ([2002:a17:903:40cf:b0:2be:b2c1:9921]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:2283:b0:2bf:372a:830c with SMTP id d9443c01a7336-2bf372a8484mr16584755ad.11.1780093392030; Fri, 29 May 2026 15:23:12 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:08 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-26-seanjc@google.com> Subject: [PATCH v3 25/40] KVM: x86: Expose several TSC helpers via x86.h for use by MSR code From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Begrudgingly move adjust_tsc_offset_{guest,host}() to x86.h as inlines, and expose several other TSC helpers in anticipation of moving KVM's MSR code to a dedicated msrs.c. Unfortunately for KVM, several MSRs that KVM emulates can affect TSC state. Opportunistically drop a superfluous local "tsc_offset" variable, whose existence causes checkpatch to complain about lack of a blank line. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang --- arch/x86/kvm/x86.c | 22 +++------------------- arch/x86/kvm/x86.h | 19 +++++++++++++++++++ 2 files changed, 22 insertions(+), 19 deletions(-) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 64c3680d889b..bf15c122f837 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -2225,7 +2225,7 @@ u64 kvm_scale_tsc(u64 tsc, u64 ratio) return _tsc; } =20 -static u64 kvm_compute_l1_tsc_offset(struct kvm_vcpu *vcpu, u64 target_tsc) +u64 kvm_compute_l1_tsc_offset(struct kvm_vcpu *vcpu, u64 target_tsc) { u64 tsc; =20 @@ -2266,7 +2266,7 @@ u64 kvm_calc_nested_tsc_multiplier(u64 l1_multiplier,= u64 l2_multiplier) } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_calc_nested_tsc_multiplier); =20 -static void kvm_vcpu_write_tsc_offset(struct kvm_vcpu *vcpu, u64 l1_offset) +void kvm_vcpu_write_tsc_offset(struct kvm_vcpu *vcpu, u64 l1_offset) { if (vcpu->arch.guest_tsc_protected) return; @@ -2380,7 +2380,7 @@ static void __kvm_synchronize_tsc(struct kvm_vcpu *vc= pu, u64 offset, u64 tsc, kvm_track_tsc_matching(vcpu, !matched); } =20 -static void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value) +void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value) { u64 data =3D user_value ? *user_value : 0; struct kvm *kvm =3D vcpu->kvm; @@ -2448,22 +2448,6 @@ static void kvm_synchronize_tsc(struct kvm_vcpu *vcp= u, u64 *user_value) raw_spin_unlock_irqrestore(&kvm->arch.tsc_write_lock, flags); } =20 -static inline void adjust_tsc_offset_guest(struct kvm_vcpu *vcpu, - s64 adjustment) -{ - u64 tsc_offset =3D vcpu->arch.l1_tsc_offset; - kvm_vcpu_write_tsc_offset(vcpu, tsc_offset + adjustment); -} - -static inline void adjust_tsc_offset_host(struct kvm_vcpu *vcpu, s64 adjus= tment) -{ - if (vcpu->arch.l1_tsc_scaling_ratio !=3D kvm_caps.default_tsc_scaling_rat= io) - WARN_ON(adjustment < 0); - adjustment =3D kvm_scale_tsc((u64) adjustment, - vcpu->arch.l1_tsc_scaling_ratio); - adjust_tsc_offset_guest(vcpu, adjustment); -} - #ifdef CONFIG_X86_64 =20 static u64 read_tsc(void) diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 635a21bfa681..31e67b060148 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -345,6 +345,25 @@ uint64_t kvm_get_wall_clock_epoch(struct kvm *kvm); bool kvm_get_monotonic_and_clockread(s64 *kernel_ns, u64 *tsc_timestamp); int kvm_guest_time_update(struct kvm_vcpu *v); =20 +void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value); +u64 kvm_compute_l1_tsc_offset(struct kvm_vcpu *vcpu, u64 target_tsc); +void kvm_vcpu_write_tsc_offset(struct kvm_vcpu *vcpu, u64 l1_offset); + +static inline void adjust_tsc_offset_guest(struct kvm_vcpu *vcpu, + s64 adjustment) +{ + kvm_vcpu_write_tsc_offset(vcpu, vcpu->arch.l1_tsc_offset + adjustment); +} + +static inline void adjust_tsc_offset_host(struct kvm_vcpu *vcpu, s64 adjus= tment) +{ + if (vcpu->arch.l1_tsc_scaling_ratio !=3D kvm_caps.default_tsc_scaling_rat= io) + WARN_ON(adjustment < 0); + adjustment =3D kvm_scale_tsc((u64) adjustment, + vcpu->arch.l1_tsc_scaling_ratio); + adjust_tsc_offset_guest(vcpu, adjustment); +} + int kvm_read_guest_virt(struct kvm_vcpu *vcpu, gva_t addr, void *val, unsigned int bytes, struct x86_exception *exception); --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1CF5947277E for ; Fri, 29 May 2026 22:23:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093402; cv=none; b=lt+HXuIYxvSRv8qNk2hCClOnhlXbzCnA4HcFcucFVGPJ8WtzeKjes2EvQ/m9FYpDkdl1oBn3vSnT8pHoQwhoE1mEpoOFgHIMu3Bdcty9dzwziO3BYbLOyvt0cnBvDk+9NUnzmCdTJOkY06SX+hqFZsaqSrdD2EllFqxbh978T00= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093402; c=relaxed/simple; bh=piGJZtwTbazI51/hK3ubqimRo4+UyB5BC+coxEgNnIQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=bGmwZ13t5PCHZlMvezpoi4LefTZobLYgsLoW+1BbH8Mj5J5FBmxnCSWAFlZZupbkbWfpND7C1mT/OTtEgu7SVp4U+bvoDNJ+8oGLYdY0yJsLX4hz6ogOF+NnperioE4qXtaWObuG1vu9Mvm0hPI/3x9ghXGS4+9ytvvvrBRrjkg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=agd0aRvZ; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="agd0aRvZ" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-365fc4636bbso27725686a91.2 for ; Fri, 29 May 2026 15:23:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093393; x=1780698193; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=SDj+p1x1WXtkynVREgbfNIMZyiXUu78aCrXJO293Hcg=; b=agd0aRvZHkzfGiEKRFoTqrYck3Tn1KPT8JZ/xqr5PBfwxlLfOtet7DhNJTqJDicL7a 1VYAEcPXgbMj+zHjTyXknC0Fp4E+c/Ro9IcmVrKA98Xy1Q1HC0zd37UGsvK0xa/U/Bmq BSVaNhERNq43eZdSfnJCibKWwC2TSA07CfeIE6U9hKdpbR8w49RqbiiYEY8lJSlmbSn3 4p6Mwe6JI8JWFWoT1URaXNuxtO+nLaPlWN/YBbmqUXuHrW/CyT5J3SkOeSNFB4C0Pd+I i0FYPJq5CoXC2G/hvjjezTrMBH/drLBQYYm4zptbXVEFZpcGvYIFNiniBDdUVxTHEB59 p3UA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093393; x=1780698193; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=SDj+p1x1WXtkynVREgbfNIMZyiXUu78aCrXJO293Hcg=; b=KUkTrTvGtyylAD9v8qvuAuhf5NoCVo2y+0leDKu8whJ1EZ5kEHoUWl+JUSvyKbanv7 SpbF0Wj6mvb9AKHIM1p6MskoLdQaRU7ia3XF9d3SrnquIJCZowiEtXxPbdi5n9g56XBy qLIg5Z5k8MPaQfTSbeXsJgrIPLUnPADBVapCOaLm9cBCOfNp9NtgD8fYaI8PAdEhBTxZ sPFRbD6uQxOSivFXtCl/5lZVDIhPa117vHkd0rRsN7xH4aPnYXoO4CZV6ziNwoMgWd6z iUZcLdhwIpymKkIMO07iMqaYS1tSEI5w6NZHqpxNaC9T0D15chVOogUzxduP3/Y9MOA9 Z0ow== X-Forwarded-Encrypted: i=1; AFNElJ/dz8jg5LyT2/pjGaUBkRyIKoFQehMp62DFQr0Q50+obJE6SrmmI4oEig9Zqw3WYExUvh+C1ZFvJ9jHJCM=@vger.kernel.org X-Gm-Message-State: AOJu0YwEMJlWp6xdLG8TSrkmpVeDln3hhisOXXg8m/ztmhNAKEVlXa3/ Ku0yVxj/kjqawnkFVSM0+7T0wTIFdxsQD/1rEPLWUtg4yo6CvzgCJ4HrYDg8PZrM0QxJZlBjWTK 3uIUz3Q== X-Received: from pgbaz9.prod.google.com ([2002:a05:6a02:49:b0:c85:86ac:38d7]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:5748:b0:36b:93e6:1184 with SMTP id 98e67ed59e1d1-36c6847f81fmr947581a91.27.1780093393139; Fri, 29 May 2026 15:23:13 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:09 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-27-seanjc@google.com> Subject: [PATCH v3 26/40] KVM: x86: Move the bulk of MSR specific code from x86.c to msrs.{c,h} From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Introduce msrs.{c,h}, and move the vast majority of MSR specific code out of x86.{c,h}. Use a plural "msrs" instead of just "msr" to be consistent with regs.{c,h}, and to make it easier to differentiate KVM's code from the other 5+ msr.c files in the kernel. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang --- arch/x86/kvm/Makefile | 2 +- arch/x86/kvm/msrs.c | 2732 +++++++++++++++++++++++++++++++++++++++++ arch/x86/kvm/msrs.h | 128 ++ arch/x86/kvm/mtrr.c | 1 + arch/x86/kvm/x86.c | 2711 +--------------------------------------- arch/x86/kvm/x86.h | 87 +- 6 files changed, 2867 insertions(+), 2794 deletions(-) create mode 100644 arch/x86/kvm/msrs.c create mode 100644 arch/x86/kvm/msrs.h diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile index f39c311fd756..0474604ab8a1 100644 --- a/arch/x86/kvm/Makefile +++ b/arch/x86/kvm/Makefile @@ -5,7 +5,7 @@ ccflags-$(CONFIG_KVM_WERROR) +=3D -Werror =20 include $(srctree)/virt/kvm/Makefile.kvm =20 -kvm-y +=3D x86.o emulate.o irq.o lapic.o cpuid.o pmu.o regs.o \ +kvm-y +=3D x86.o emulate.o irq.o lapic.o cpuid.o msrs.o pmu.o regs.o \ mtrr.o debugfs.o mmu/mmu.o mmu/page_track.o mmu/spte.o =20 kvm-$(CONFIG_X86_64) +=3D mmu/tdp_iter.o mmu/tdp_mmu.o diff --git a/arch/x86/kvm/msrs.c b/arch/x86/kvm/msrs.c new file mode 100644 index 000000000000..67ed0d36ed91 --- /dev/null +++ b/arch/x86/kvm/msrs.c @@ -0,0 +1,2732 @@ +// SPDX-License-Identifier: GPL-2.0-only +#include +#include +#include + +#include "hyperv.h" +#include "lapic.h" +#include "msrs.h" +#include "pmu.h" +#include "trace.h" +#include "vmx/vmx.h" +#include "xen.h" +#include "x86.h" + +bool __read_mostly ignore_msrs =3D 0; +module_param(ignore_msrs, bool, 0644); + +bool __read_mostly report_ignored_msrs =3D true; +module_param(report_ignored_msrs, bool, 0644); +EXPORT_SYMBOL_FOR_KVM_INTERNAL(report_ignored_msrs); + +/* EFER defaults: + * - enable syscall per default because its emulated by KVM + * - enable LME and LMA per default on 64 bit KVM + */ +#ifdef CONFIG_X86_64 +static +u64 __read_mostly efer_reserved_bits =3D ~((u64)(EFER_SCE | EFER_LME | EFE= R_LMA)); +#else +static u64 __read_mostly efer_reserved_bits =3D ~((u64)EFER_SCE); +#endif + +#define MAX_IO_MSRS 256 + +/* + * Restoring the host value for MSRs that are only consumed when running in + * usermode, e.g. SYSCALL MSRs and TSC_AUX, can be deferred until the CPU + * returns to userspace, i.e. the kernel can run with the guest's value. + */ +#define KVM_MAX_NR_USER_RETURN_MSRS 16 + +struct kvm_user_return_msrs { + struct user_return_notifier urn; + bool registered; + struct kvm_user_return_msr_values { + u64 host; + u64 curr; + } values[KVM_MAX_NR_USER_RETURN_MSRS]; +}; + +u32 __read_mostly kvm_nr_uret_msrs; +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_nr_uret_msrs); +static u32 __read_mostly kvm_uret_msrs_list[KVM_MAX_NR_USER_RETURN_MSRS]; +static DEFINE_PER_CPU(struct kvm_user_return_msrs, user_return_msrs); + +void kvm_destroy_user_return_msrs(void) +{ + int cpu; + + for_each_possible_cpu(cpu) + WARN_ON_ONCE(per_cpu(user_return_msrs, cpu).registered); + + kvm_nr_uret_msrs =3D 0; +} + +static void kvm_on_user_return(struct user_return_notifier *urn) +{ + unsigned slot; + struct kvm_user_return_msrs *msrs + =3D container_of(urn, struct kvm_user_return_msrs, urn); + struct kvm_user_return_msr_values *values; + + msrs->registered =3D false; + user_return_notifier_unregister(urn); + + for (slot =3D 0; slot < kvm_nr_uret_msrs; ++slot) { + values =3D &msrs->values[slot]; + if (values->host !=3D values->curr) { + wrmsrq(kvm_uret_msrs_list[slot], values->host); + values->curr =3D values->host; + } + } +} + +static int kvm_probe_user_return_msr(u32 msr) +{ + u64 val; + int ret; + + preempt_disable(); + ret =3D rdmsrq_safe(msr, &val); + if (ret) + goto out; + ret =3D wrmsrq_safe(msr, val); +out: + preempt_enable(); + return ret; +} + +int kvm_add_user_return_msr(u32 msr) +{ + BUG_ON(kvm_nr_uret_msrs >=3D KVM_MAX_NR_USER_RETURN_MSRS); + + if (kvm_probe_user_return_msr(msr)) + return -1; + + kvm_uret_msrs_list[kvm_nr_uret_msrs] =3D msr; + return kvm_nr_uret_msrs++; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_add_user_return_msr); + +int kvm_find_user_return_msr(u32 msr) +{ + int i; + + for (i =3D 0; i < kvm_nr_uret_msrs; ++i) { + if (kvm_uret_msrs_list[i] =3D=3D msr) + return i; + } + return -1; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_find_user_return_msr); + +void kvm_user_return_msr_cpu_online(void) +{ + struct kvm_user_return_msrs *msrs =3D this_cpu_ptr(&user_return_msrs); + u64 value; + int i; + + for (i =3D 0; i < kvm_nr_uret_msrs; ++i) { + rdmsrq_safe(kvm_uret_msrs_list[i], &value); + msrs->values[i].host =3D value; + msrs->values[i].curr =3D value; + } +} + +static void kvm_user_return_register_notifier(struct kvm_user_return_msrs = *msrs) +{ + if (!msrs->registered) { + msrs->urn.on_user_return =3D kvm_on_user_return; + user_return_notifier_register(&msrs->urn); + msrs->registered =3D true; + } +} + +int kvm_set_user_return_msr(unsigned slot, u64 value, u64 mask) +{ + struct kvm_user_return_msrs *msrs =3D this_cpu_ptr(&user_return_msrs); + int err; + + value =3D (value & mask) | (msrs->values[slot].host & ~mask); + if (value =3D=3D msrs->values[slot].curr) + return 0; + err =3D wrmsrq_safe(kvm_uret_msrs_list[slot], value); + if (err) + return 1; + + msrs->values[slot].curr =3D value; + kvm_user_return_register_notifier(msrs); + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_user_return_msr); + +u64 kvm_get_user_return_msr(unsigned int slot) +{ + return this_cpu_ptr(&user_return_msrs)->values[slot].curr; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_user_return_msr); + +void drop_user_return_notifiers(void) +{ + struct kvm_user_return_msrs *msrs =3D this_cpu_ptr(&user_return_msrs); + + if (msrs->registered) + kvm_on_user_return(&msrs->urn); +} + +/* + * The three MSR lists(msrs_to_save, emulated_msrs, msr_based_features) tr= ack + * the set of MSRs that KVM exposes to userspace through KVM_GET_MSRS, + * KVM_SET_MSRS, and KVM_GET_MSR_INDEX_LIST. msrs_to_save holds MSRs that + * require host support, i.e. should be probed via RDMSR. emulated_msrs h= olds + * MSRs that KVM emulates without strictly requiring host support. + * msr_based_features holds MSRs that enumerate features, i.e. are effecti= vely + * CPUID leafs. Note, msr_based_features isn't mutually exclusive with + * msrs_to_save and emulated_msrs. + */ + +static const u32 msrs_to_save_base[] =3D { + MSR_IA32_SYSENTER_CS, MSR_IA32_SYSENTER_ESP, MSR_IA32_SYSENTER_EIP, + MSR_STAR, +#ifdef CONFIG_X86_64 + MSR_CSTAR, MSR_KERNEL_GS_BASE, MSR_SYSCALL_MASK, MSR_LSTAR, +#endif + MSR_IA32_TSC, MSR_IA32_CR_PAT, MSR_VM_HSAVE_PA, + MSR_IA32_FEAT_CTL, MSR_IA32_BNDCFGS, MSR_TSC_AUX, + MSR_IA32_SPEC_CTRL, MSR_IA32_TSX_CTRL, + MSR_IA32_RTIT_CTL, MSR_IA32_RTIT_STATUS, MSR_IA32_RTIT_CR3_MATCH, + MSR_IA32_RTIT_OUTPUT_BASE, MSR_IA32_RTIT_OUTPUT_MASK, + MSR_IA32_RTIT_ADDR0_A, MSR_IA32_RTIT_ADDR0_B, + MSR_IA32_RTIT_ADDR1_A, MSR_IA32_RTIT_ADDR1_B, + MSR_IA32_RTIT_ADDR2_A, MSR_IA32_RTIT_ADDR2_B, + MSR_IA32_RTIT_ADDR3_A, MSR_IA32_RTIT_ADDR3_B, + MSR_IA32_UMWAIT_CONTROL, + + MSR_IA32_XFD, MSR_IA32_XFD_ERR, MSR_IA32_XSS, + + MSR_IA32_U_CET, MSR_IA32_S_CET, + MSR_IA32_PL0_SSP, MSR_IA32_PL1_SSP, MSR_IA32_PL2_SSP, + MSR_IA32_PL3_SSP, MSR_IA32_INT_SSP_TAB, + MSR_IA32_DEBUGCTLMSR, + MSR_IA32_LASTBRANCHFROMIP, MSR_IA32_LASTBRANCHTOIP, + MSR_IA32_LASTINTFROMIP, MSR_IA32_LASTINTTOIP, +}; + +static const u32 msrs_to_save_pmu[] =3D { + MSR_ARCH_PERFMON_FIXED_CTR0, MSR_ARCH_PERFMON_FIXED_CTR1, + MSR_ARCH_PERFMON_FIXED_CTR0 + 2, + MSR_CORE_PERF_FIXED_CTR_CTRL, MSR_CORE_PERF_GLOBAL_STATUS, + MSR_CORE_PERF_GLOBAL_CTRL, + MSR_IA32_PEBS_ENABLE, MSR_IA32_DS_AREA, MSR_PEBS_DATA_CFG, + + /* This part of MSRs should match KVM_MAX_NR_INTEL_GP_COUNTERS. */ + MSR_ARCH_PERFMON_PERFCTR0, MSR_ARCH_PERFMON_PERFCTR1, + MSR_ARCH_PERFMON_PERFCTR0 + 2, MSR_ARCH_PERFMON_PERFCTR0 + 3, + MSR_ARCH_PERFMON_PERFCTR0 + 4, MSR_ARCH_PERFMON_PERFCTR0 + 5, + MSR_ARCH_PERFMON_PERFCTR0 + 6, MSR_ARCH_PERFMON_PERFCTR0 + 7, + MSR_ARCH_PERFMON_EVENTSEL0, MSR_ARCH_PERFMON_EVENTSEL1, + MSR_ARCH_PERFMON_EVENTSEL0 + 2, MSR_ARCH_PERFMON_EVENTSEL0 + 3, + MSR_ARCH_PERFMON_EVENTSEL0 + 4, MSR_ARCH_PERFMON_EVENTSEL0 + 5, + MSR_ARCH_PERFMON_EVENTSEL0 + 6, MSR_ARCH_PERFMON_EVENTSEL0 + 7, + + MSR_K7_EVNTSEL0, MSR_K7_EVNTSEL1, MSR_K7_EVNTSEL2, MSR_K7_EVNTSEL3, + MSR_K7_PERFCTR0, MSR_K7_PERFCTR1, MSR_K7_PERFCTR2, MSR_K7_PERFCTR3, + + /* This part of MSRs should match KVM_MAX_NR_AMD_GP_COUNTERS. */ + MSR_F15H_PERF_CTL0, MSR_F15H_PERF_CTL1, MSR_F15H_PERF_CTL2, + MSR_F15H_PERF_CTL3, MSR_F15H_PERF_CTL4, MSR_F15H_PERF_CTL5, + MSR_F15H_PERF_CTR0, MSR_F15H_PERF_CTR1, MSR_F15H_PERF_CTR2, + MSR_F15H_PERF_CTR3, MSR_F15H_PERF_CTR4, MSR_F15H_PERF_CTR5, + + MSR_AMD64_PERF_CNTR_GLOBAL_CTL, + MSR_AMD64_PERF_CNTR_GLOBAL_STATUS, + MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_CLR, + MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_SET, +}; + +static u32 msrs_to_save[ARRAY_SIZE(msrs_to_save_base) + + ARRAY_SIZE(msrs_to_save_pmu)]; +static unsigned num_msrs_to_save; + +static const u32 emulated_msrs_all[] =3D { + MSR_KVM_SYSTEM_TIME, MSR_KVM_WALL_CLOCK, + MSR_KVM_SYSTEM_TIME_NEW, MSR_KVM_WALL_CLOCK_NEW, + +#ifdef CONFIG_KVM_HYPERV + HV_X64_MSR_GUEST_OS_ID, HV_X64_MSR_HYPERCALL, + HV_X64_MSR_TIME_REF_COUNT, HV_X64_MSR_REFERENCE_TSC, + HV_X64_MSR_TSC_FREQUENCY, HV_X64_MSR_APIC_FREQUENCY, + HV_X64_MSR_CRASH_P0, HV_X64_MSR_CRASH_P1, HV_X64_MSR_CRASH_P2, + HV_X64_MSR_CRASH_P3, HV_X64_MSR_CRASH_P4, HV_X64_MSR_CRASH_CTL, + HV_X64_MSR_RESET, + HV_X64_MSR_VP_INDEX, + HV_X64_MSR_VP_RUNTIME, + HV_X64_MSR_SCONTROL, + HV_X64_MSR_STIMER0_CONFIG, + HV_X64_MSR_VP_ASSIST_PAGE, + HV_X64_MSR_REENLIGHTENMENT_CONTROL, HV_X64_MSR_TSC_EMULATION_CONTROL, + HV_X64_MSR_TSC_EMULATION_STATUS, HV_X64_MSR_TSC_INVARIANT_CONTROL, + HV_X64_MSR_SYNDBG_OPTIONS, + HV_X64_MSR_SYNDBG_CONTROL, HV_X64_MSR_SYNDBG_STATUS, + HV_X64_MSR_SYNDBG_SEND_BUFFER, HV_X64_MSR_SYNDBG_RECV_BUFFER, + HV_X64_MSR_SYNDBG_PENDING_BUFFER, +#endif + + MSR_KVM_ASYNC_PF_EN, MSR_KVM_STEAL_TIME, + MSR_KVM_PV_EOI_EN, MSR_KVM_ASYNC_PF_INT, MSR_KVM_ASYNC_PF_ACK, + + MSR_IA32_TSC_ADJUST, + MSR_IA32_TSC_DEADLINE, + MSR_IA32_ARCH_CAPABILITIES, + MSR_IA32_PERF_CAPABILITIES, + MSR_IA32_MISC_ENABLE, + MSR_IA32_MCG_STATUS, + MSR_IA32_MCG_CTL, + MSR_IA32_MCG_EXT_CTL, + MSR_IA32_SMBASE, + MSR_SMI_COUNT, + MSR_PLATFORM_INFO, + MSR_MISC_FEATURES_ENABLES, + MSR_AMD64_VIRT_SPEC_CTRL, + MSR_AMD64_TSC_RATIO, + MSR_IA32_POWER_CTL, + MSR_IA32_UCODE_REV, + + /* + * KVM always supports the "true" VMX control MSRs, even if the host + * does not. The VMX MSRs as a whole are considered "emulated" as KVM + * doesn't strictly require them to exist in the host (ignoring that + * KVM would refuse to load in the first place if the core set of MSRs + * aren't supported). + */ + MSR_IA32_VMX_BASIC, + MSR_IA32_VMX_TRUE_PINBASED_CTLS, + MSR_IA32_VMX_TRUE_PROCBASED_CTLS, + MSR_IA32_VMX_TRUE_EXIT_CTLS, + MSR_IA32_VMX_TRUE_ENTRY_CTLS, + MSR_IA32_VMX_MISC, + MSR_IA32_VMX_CR0_FIXED0, + MSR_IA32_VMX_CR4_FIXED0, + MSR_IA32_VMX_VMCS_ENUM, + MSR_IA32_VMX_PROCBASED_CTLS2, + MSR_IA32_VMX_EPT_VPID_CAP, + MSR_IA32_VMX_VMFUNC, + + MSR_K7_HWCR, + MSR_KVM_POLL_CONTROL, +}; + +static u32 emulated_msrs[ARRAY_SIZE(emulated_msrs_all)]; +static unsigned num_emulated_msrs; + +/* + * List of MSRs that control the existence of MSR-based features, i.e. MSRs + * that are effectively CPUID leafs. VMX MSRs are also included in the se= t of + * feature MSRs, but are handled separately to allow expedited lookups. + */ +static const u32 msr_based_features_all_except_vmx[] =3D { + MSR_AMD64_DE_CFG, + MSR_IA32_UCODE_REV, + MSR_IA32_ARCH_CAPABILITIES, + MSR_IA32_PERF_CAPABILITIES, + MSR_PLATFORM_INFO, +}; + +static u32 msr_based_features[ARRAY_SIZE(msr_based_features_all_except_vmx= ) + + (KVM_LAST_EMULATED_VMX_MSR - KVM_FIRST_EMULATED_VMX_MSR + 1)]; +static unsigned int num_msr_based_features; + +int kvm_get_msr_index_list(struct kvm_msr_list __user *user_msr_list) +{ + struct kvm_msr_list msr_list; + unsigned int n; + + if (copy_from_user(&msr_list, user_msr_list, sizeof(msr_list))) + return -EFAULT; + + n =3D msr_list.nmsrs; + msr_list.nmsrs =3D num_msrs_to_save + num_emulated_msrs; + if (copy_to_user(user_msr_list, &msr_list, sizeof(msr_list))) + return -EFAULT; + + if (n < msr_list.nmsrs) + return -E2BIG; + + if (copy_to_user(user_msr_list->indices, &msrs_to_save, + num_msrs_to_save * sizeof(u32))) + return -EFAULT; + + if (copy_to_user(user_msr_list->indices + num_msrs_to_save, + &emulated_msrs, num_emulated_msrs * sizeof(u32))) + return -EFAULT; + + return 0; +} + +int kvm_get_feature_msr_index_list(struct kvm_msr_list __user *user_msr_li= st) +{ + struct kvm_msr_list msr_list; + unsigned int n; + + if (copy_from_user(&msr_list, user_msr_list, sizeof(msr_list))) + return -EFAULT; + + n =3D msr_list.nmsrs; + msr_list.nmsrs =3D num_msr_based_features; + if (copy_to_user(user_msr_list, &msr_list, sizeof(msr_list))) + return -EFAULT; + + if (n < msr_list.nmsrs) + return -E2BIG; + + if (copy_to_user(user_msr_list->indices, &msr_based_features, + num_msr_based_features * sizeof(u32))) + return -EFAULT; + + return 0; +} + +/* + * All feature MSRs except uCode revID, which tracks the currently loaded = uCode + * patch, are immutable once the vCPU model is defined. + */ +static bool kvm_is_immutable_feature_msr(u32 msr) +{ + int i; + + if (msr >=3D KVM_FIRST_EMULATED_VMX_MSR && msr <=3D KVM_LAST_EMULATED_VMX= _MSR) + return true; + + for (i =3D 0; i < ARRAY_SIZE(msr_based_features_all_except_vmx); i++) { + if (msr =3D=3D msr_based_features_all_except_vmx[i]) + return msr !=3D MSR_IA32_UCODE_REV; + } + + return false; +} + +static bool kvm_is_advertised_msr(u32 msr_index) +{ + unsigned int i; + + for (i =3D 0; i < num_msrs_to_save; i++) { + if (msrs_to_save[i] =3D=3D msr_index) + return true; + } + + for (i =3D 0; i < num_emulated_msrs; i++) { + if (emulated_msrs[i] =3D=3D msr_index) + return true; + } + + return false; +} + + +/* + * Some IA32_ARCH_CAPABILITIES bits have dependencies on MSRs that KVM + * does not yet virtualize. These include: + * 10 - MISC_PACKAGE_CTRLS + * 11 - ENERGY_FILTERING_CTL + * 12 - DOITM + * 18 - FB_CLEAR_CTRL + * 21 - XAPIC_DISABLE_STATUS + * 23 - OVERCLOCKING_STATUS + */ + +#define KVM_SUPPORTED_ARCH_CAP \ + (ARCH_CAP_RDCL_NO | ARCH_CAP_IBRS_ALL | ARCH_CAP_RSBA | \ + ARCH_CAP_SKIP_VMENTRY_L1DFLUSH | ARCH_CAP_SSB_NO | ARCH_CAP_MDS_NO | \ + ARCH_CAP_PSCHANGE_MC_NO | ARCH_CAP_TSX_CTRL_MSR | ARCH_CAP_TAA_NO | \ + ARCH_CAP_SBDR_SSDP_NO | ARCH_CAP_FBSDP_NO | ARCH_CAP_PSDP_NO | \ + ARCH_CAP_FB_CLEAR | ARCH_CAP_RRSBA | ARCH_CAP_PBRSB_NO | ARCH_CAP_GDS_NO= | \ + ARCH_CAP_RFDS_NO | ARCH_CAP_RFDS_CLEAR | ARCH_CAP_BHI_NO | ARCH_CAP_ITS_= NO) + +u64 kvm_get_arch_capabilities(void) +{ + u64 data =3D kvm_host.arch_capabilities & KVM_SUPPORTED_ARCH_CAP; + + /* + * If nx_huge_pages is enabled, KVM's shadow paging will ensure that + * the nested hypervisor runs with NX huge pages. If it is not, + * L1 is anyway vulnerable to ITLB_MULTIHIT exploits from other + * L1 guests, so it need not worry about its own (L2) guests. + */ + data |=3D ARCH_CAP_PSCHANGE_MC_NO; + + /* + * If we're doing cache flushes (either "always" or "cond") + * we will do one whenever the guest does a vmlaunch/vmresume. + * If an outer hypervisor is doing the cache flush for us + * (ARCH_CAP_SKIP_VMENTRY_L1DFLUSH), we can safely pass that + * capability to the guest too, and if EPT is disabled we're not + * vulnerable. Overall, only VMENTER_L1D_FLUSH_NEVER will + * require a nested hypervisor to do a flush of its own. + */ + if (l1tf_vmx_mitigation !=3D VMENTER_L1D_FLUSH_NEVER) + data |=3D ARCH_CAP_SKIP_VMENTRY_L1DFLUSH; + + if (!boot_cpu_has_bug(X86_BUG_CPU_MELTDOWN)) + data |=3D ARCH_CAP_RDCL_NO; + if (!boot_cpu_has_bug(X86_BUG_SPEC_STORE_BYPASS)) + data |=3D ARCH_CAP_SSB_NO; + if (!boot_cpu_has_bug(X86_BUG_MDS)) + data |=3D ARCH_CAP_MDS_NO; + if (!boot_cpu_has_bug(X86_BUG_RFDS)) + data |=3D ARCH_CAP_RFDS_NO; + if (!boot_cpu_has_bug(X86_BUG_ITS)) + data |=3D ARCH_CAP_ITS_NO; + + if (!boot_cpu_has(X86_FEATURE_RTM)) { + /* + * If RTM=3D0 because the kernel has disabled TSX, the host might + * have TAA_NO or TSX_CTRL. Clear TAA_NO (the guest sees RTM=3D0 + * and therefore knows that there cannot be TAA) but keep + * TSX_CTRL: some buggy userspaces leave it set on tsx=3Don hosts, + * and we want to allow migrating those guests to tsx=3Doff hosts. + */ + data &=3D ~ARCH_CAP_TAA_NO; + } else if (!boot_cpu_has_bug(X86_BUG_TAA)) { + data |=3D ARCH_CAP_TAA_NO; + } else { + /* + * Nothing to do here; we emulate TSX_CTRL if present on the + * host so the guest can choose between disabling TSX or + * using VERW to clear CPU buffers. + */ + } + + if (!boot_cpu_has_bug(X86_BUG_GDS) || gds_ucode_mitigated()) + data |=3D ARCH_CAP_GDS_NO; + + return data; +} + +static int kvm_get_feature_msr(struct kvm_vcpu *vcpu, u32 index, u64 *data, + bool host_initiated) +{ + WARN_ON_ONCE(!host_initiated); + + switch (index) { + case MSR_IA32_ARCH_CAPABILITIES: + *data =3D kvm_get_arch_capabilities(); + break; + case MSR_IA32_PERF_CAPABILITIES: + *data =3D kvm_caps.supported_perf_cap; + break; + case MSR_PLATFORM_INFO: + *data =3D MSR_PLATFORM_INFO_CPUID_FAULT; + break; + case MSR_IA32_UCODE_REV: + rdmsrq_safe(index, data); + break; + default: + return kvm_x86_call(get_feature_msr)(index, data); + } + return 0; +} + +typedef int (*msr_access_t)(struct kvm_vcpu *vcpu, u32 index, u64 *data, + bool host_initiated); + +static __always_inline int kvm_do_msr_access(struct kvm_vcpu *vcpu, u32 ms= r, + u64 *data, bool host_initiated, + enum kvm_msr_access rw, + msr_access_t msr_access_fn) +{ + const char *op =3D rw =3D=3D MSR_TYPE_W ? "wrmsr" : "rdmsr"; + int ret; + + BUILD_BUG_ON(rw !=3D MSR_TYPE_R && rw !=3D MSR_TYPE_W); + + /* + * Zero the data on read failures to avoid leaking stack data to the + * guest and/or userspace, e.g. if the failure is ignored below. + */ + ret =3D msr_access_fn(vcpu, msr, data, host_initiated); + if (ret && rw =3D=3D MSR_TYPE_R) + *data =3D 0; + + if (ret !=3D KVM_MSR_RET_UNSUPPORTED) + return ret; + + /* + * Userspace is allowed to read MSRs, and write '0' to MSRs, that KVM + * advertises to userspace, even if an MSR isn't fully supported. + * Simply check that @data is '0', which covers both the write '0' case + * and all reads (in which case @data is zeroed on failure; see above). + */ + if (host_initiated && !*data && kvm_is_advertised_msr(msr)) + return 0; + + if (!ignore_msrs) { + kvm_debug_ratelimited("unhandled %s: 0x%x data 0x%llx\n", + op, msr, *data); + return ret; + } + + if (report_ignored_msrs) + kvm_pr_unimpl("ignored %s: 0x%x data 0x%llx\n", op, msr, *data); + + return 0; +} + +static int do_get_feature_msr(struct kvm_vcpu *vcpu, unsigned index, u64 *= data) +{ + return kvm_do_msr_access(vcpu, index, data, true, MSR_TYPE_R, + kvm_get_feature_msr); +} + +static bool __kvm_valid_efer(struct kvm_vcpu *vcpu, u64 efer) +{ + if (efer & EFER_AUTOIBRS && !guest_cpu_cap_has(vcpu, X86_FEATURE_AUTOIBRS= )) + return false; + + if (efer & EFER_FFXSR && !guest_cpu_cap_has(vcpu, X86_FEATURE_FXSR_OPT)) + return false; + + if (efer & EFER_SVME && !guest_cpu_cap_has(vcpu, X86_FEATURE_SVM)) + return false; + + if (efer & (EFER_LME | EFER_LMA) && + !guest_cpu_cap_has(vcpu, X86_FEATURE_LM)) + return false; + + if (efer & EFER_NX && !guest_cpu_cap_has(vcpu, X86_FEATURE_NX)) + return false; + + return true; + +} +bool kvm_valid_efer(struct kvm_vcpu *vcpu, u64 efer) +{ + if (efer & efer_reserved_bits) + return false; + + return __kvm_valid_efer(vcpu, efer); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_valid_efer); + +static int set_efer(struct kvm_vcpu *vcpu, struct msr_data *msr_info) +{ + u64 old_efer =3D vcpu->arch.efer; + u64 efer =3D msr_info->data; + int r; + + if (efer & efer_reserved_bits) + return 1; + + if (!msr_info->host_initiated) { + if (!__kvm_valid_efer(vcpu, efer)) + return 1; + + if (is_paging(vcpu) && + (vcpu->arch.efer & EFER_LME) !=3D (efer & EFER_LME)) + return 1; + } + + efer &=3D ~EFER_LMA; + efer |=3D vcpu->arch.efer & EFER_LMA; + + r =3D kvm_x86_call(set_efer)(vcpu, efer); + if (r) { + WARN_ON(r > 0); + return r; + } + + if ((efer ^ old_efer) & KVM_MMU_EFER_ROLE_BITS) + kvm_mmu_reset_context(vcpu); + + if (!static_cpu_has(X86_FEATURE_XSAVES) && + (efer & EFER_SVME)) + kvm_hv_xsaves_xsavec_maybe_warn(vcpu); + + return 0; +} + +void kvm_enable_efer_bits(u64 mask) +{ + efer_reserved_bits &=3D ~mask; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_enable_efer_bits); + +bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type) +{ + struct kvm_x86_msr_filter *msr_filter; + struct msr_bitmap_range *ranges; + struct kvm *kvm =3D vcpu->kvm; + bool allowed; + int idx; + u32 i; + + /* x2APIC MSRs do not support filtering. */ + if (index >=3D 0x800 && index <=3D 0x8ff) + return true; + + idx =3D srcu_read_lock(&kvm->srcu); + + msr_filter =3D srcu_dereference(kvm->arch.msr_filter, &kvm->srcu); + if (!msr_filter) { + allowed =3D true; + goto out; + } + + allowed =3D msr_filter->default_allow; + ranges =3D msr_filter->ranges; + + for (i =3D 0; i < msr_filter->count; i++) { + u32 start =3D ranges[i].base; + u32 end =3D start + ranges[i].nmsrs; + u32 flags =3D ranges[i].flags; + unsigned long *bitmap =3D ranges[i].bitmap; + + if ((index >=3D start) && (index < end) && (flags & type)) { + allowed =3D test_bit(index - start, bitmap); + break; + } + } + +out: + srcu_read_unlock(&kvm->srcu, idx); + + return allowed; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_msr_allowed); + +/* + * Write @data into the MSR specified by @index. Select MSR specific fault + * checks are bypassed if @host_initiated is %true. + * Returns 0 on success, non-0 otherwise. + * Assumes vcpu_load() was already called. + */ +static int __kvm_set_msr(struct kvm_vcpu *vcpu, u32 index, u64 data, + bool host_initiated) +{ + struct msr_data msr; + + switch (index) { + case MSR_FS_BASE: + case MSR_GS_BASE: + case MSR_KERNEL_GS_BASE: + case MSR_CSTAR: + case MSR_LSTAR: + if (is_noncanonical_msr_address(data, vcpu)) + return 1; + break; + case MSR_IA32_SYSENTER_EIP: + case MSR_IA32_SYSENTER_ESP: + /* + * IA32_SYSENTER_ESP and IA32_SYSENTER_EIP cause #GP if + * non-canonical address is written on Intel but not on + * AMD (which ignores the top 32-bits, because it does + * not implement 64-bit SYSENTER). + * + * 64-bit code should hence be able to write a non-canonical + * value on AMD. Making the address canonical ensures that + * vmentry does not fail on Intel after writing a non-canonical + * value, and that something deterministic happens if the guest + * invokes 64-bit SYSENTER. + */ + data =3D __canonical_address(data, max_host_virt_addr_bits()); + break; + case MSR_TSC_AUX: + if (!kvm_is_supported_user_return_msr(MSR_TSC_AUX)) + return 1; + + if (!host_initiated && + !guest_cpu_cap_has(vcpu, X86_FEATURE_RDTSCP) && + !guest_cpu_cap_has(vcpu, X86_FEATURE_RDPID)) + return 1; + + /* + * Per Intel's SDM, bits 63:32 are reserved, but AMD's APM has + * incomplete and conflicting architectural behavior. Current + * AMD CPUs completely ignore bits 63:32, i.e. they aren't + * reserved and always read as zeros. Enforce Intel's reserved + * bits check if the guest CPU is Intel compatible, otherwise + * clear the bits. This ensures cross-vendor migration will + * provide consistent behavior for the guest. + */ + if (guest_cpuid_is_intel_compatible(vcpu) && (data >> 32) !=3D 0) + return 1; + + data =3D (u32)data; + break; + case MSR_IA32_U_CET: + case MSR_IA32_S_CET: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) && + !guest_cpu_cap_has(vcpu, X86_FEATURE_IBT)) + return KVM_MSR_RET_UNSUPPORTED; + if (!kvm_is_valid_u_s_cet(vcpu, data)) + return 1; + break; + case MSR_KVM_INTERNAL_GUEST_SSP: + if (!host_initiated) + return 1; + fallthrough; + /* + * Note that the MSR emulation here is flawed when a vCPU + * doesn't support the Intel 64 architecture. The expected + * architectural behavior in this case is that the upper 32 + * bits do not exist and should always read '0'. However, + * because the actual hardware on which the virtual CPU is + * running does support Intel 64, XRSTORS/XSAVES in the + * guest could observe behavior that violates the + * architecture. Intercepting XRSTORS/XSAVES for this + * special case isn't deemed worthwhile. + */ + case MSR_IA32_PL0_SSP ... MSR_IA32_INT_SSP_TAB: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK)) + return KVM_MSR_RET_UNSUPPORTED; + /* + * MSR_IA32_INT_SSP_TAB is not present on processors that do + * not support Intel 64 architecture. + */ + if (index =3D=3D MSR_IA32_INT_SSP_TAB && !guest_cpu_cap_has(vcpu, X86_FE= ATURE_LM)) + return KVM_MSR_RET_UNSUPPORTED; + if (is_noncanonical_msr_address(data, vcpu)) + return 1; + /* All SSP MSRs except MSR_IA32_INT_SSP_TAB must be 4-byte aligned */ + if (index !=3D MSR_IA32_INT_SSP_TAB && !IS_ALIGNED(data, 4)) + return 1; + break; + } + + msr.data =3D data; + msr.index =3D index; + msr.host_initiated =3D host_initiated; + + return kvm_x86_call(set_msr)(vcpu, &msr); +} + +static int _kvm_set_msr(struct kvm_vcpu *vcpu, u32 index, u64 *data, + bool host_initiated) +{ + return __kvm_set_msr(vcpu, index, *data, host_initiated); +} + +static int kvm_set_msr_ignored_check(struct kvm_vcpu *vcpu, + u32 index, u64 data, bool host_initiated) +{ + return kvm_do_msr_access(vcpu, index, &data, host_initiated, MSR_TYPE_W, + _kvm_set_msr); +} + +/* + * Read the MSR specified by @index into @data. Select MSR specific fault + * checks are bypassed if @host_initiated is %true. + * Returns 0 on success, non-0 otherwise. + * Assumes vcpu_load() was already called. + */ +static int __kvm_get_msr(struct kvm_vcpu *vcpu, u32 index, u64 *data, + bool host_initiated) +{ + struct msr_data msr; + int ret; + + switch (index) { + case MSR_TSC_AUX: + if (!kvm_is_supported_user_return_msr(MSR_TSC_AUX)) + return 1; + + if (!host_initiated && + !guest_cpu_cap_has(vcpu, X86_FEATURE_RDTSCP) && + !guest_cpu_cap_has(vcpu, X86_FEATURE_RDPID)) + return 1; + break; + case MSR_IA32_U_CET: + case MSR_IA32_S_CET: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) && + !guest_cpu_cap_has(vcpu, X86_FEATURE_IBT)) + return KVM_MSR_RET_UNSUPPORTED; + break; + case MSR_KVM_INTERNAL_GUEST_SSP: + if (!host_initiated) + return 1; + fallthrough; + case MSR_IA32_PL0_SSP ... MSR_IA32_INT_SSP_TAB: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK)) + return KVM_MSR_RET_UNSUPPORTED; + break; + } + + msr.index =3D index; + msr.host_initiated =3D host_initiated; + + ret =3D kvm_x86_call(get_msr)(vcpu, &msr); + if (!ret) + *data =3D msr.data; + return ret; +} + +static int kvm_get_msr_ignored_check(struct kvm_vcpu *vcpu, + u32 index, u64 *data, bool host_initiated) +{ + return kvm_do_msr_access(vcpu, index, data, host_initiated, MSR_TYPE_R, + __kvm_get_msr); +} + +int kvm_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data) +{ + return __kvm_set_msr(vcpu, index, data, true); +} + +int kvm_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data) +{ + return __kvm_get_msr(vcpu, index, data, true); +} + +int __kvm_emulate_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data) +{ + return kvm_get_msr_ignored_check(vcpu, index, data, false); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(__kvm_emulate_msr_read); + +int __kvm_emulate_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data) +{ + return kvm_set_msr_ignored_check(vcpu, index, data, false); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(__kvm_emulate_msr_write); + +int kvm_emulate_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data) +{ + if (!kvm_msr_allowed(vcpu, index, KVM_MSR_FILTER_READ)) + return KVM_MSR_RET_FILTERED; + + return __kvm_emulate_msr_read(vcpu, index, data); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_msr_read); + +int kvm_emulate_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data) +{ + if (!kvm_msr_allowed(vcpu, index, KVM_MSR_FILTER_WRITE)) + return KVM_MSR_RET_FILTERED; + + return __kvm_emulate_msr_write(vcpu, index, data); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_msr_write); + +static fastpath_t __handle_fastpath_wrmsr(struct kvm_vcpu *vcpu, u32 msr, = u64 data) +{ + if (!kvm_pmu_is_fastpath_emulation_allowed(vcpu)) + return EXIT_FASTPATH_NONE; + + switch (msr) { + case APIC_BASE_MSR + (APIC_ICR >> 4): + if (!lapic_in_kernel(vcpu) || !apic_x2apic_mode(vcpu->arch.apic) || + kvm_x2apic_icr_write_fast(vcpu->arch.apic, data)) + return EXIT_FASTPATH_NONE; + break; + case MSR_IA32_TSC_DEADLINE: + kvm_set_lapic_tscdeadline_msr(vcpu, data); + break; + default: + return EXIT_FASTPATH_NONE; + } + + trace_kvm_msr_write(msr, data); + + if (!kvm_skip_emulated_instruction(vcpu)) + return EXIT_FASTPATH_EXIT_USERSPACE; + + return EXIT_FASTPATH_REENTER_GUEST; +} + +fastpath_t handle_fastpath_wrmsr(struct kvm_vcpu *vcpu) +{ + return __handle_fastpath_wrmsr(vcpu, kvm_ecx_read(vcpu), + kvm_read_edx_eax(vcpu)); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(handle_fastpath_wrmsr); + +fastpath_t handle_fastpath_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int r= eg) +{ + return __handle_fastpath_wrmsr(vcpu, msr, kvm_register_read(vcpu, reg)); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(handle_fastpath_wrmsr_imm); + +static void complete_userspace_rdmsr(struct kvm_vcpu *vcpu) +{ + if (!vcpu->run->msr.error) { + kvm_eax_write(vcpu, vcpu->run->msr.data); + kvm_edx_write(vcpu, vcpu->run->msr.data >> 32); + } +} + +static int complete_emulated_insn_gp(struct kvm_vcpu *vcpu, int err) +{ + if (err) { + kvm_inject_gp(vcpu, 0); + return 1; + } + + return kvm_emulate_instruction(vcpu, EMULTYPE_NO_DECODE | EMULTYPE_SKIP | + EMULTYPE_COMPLETE_USER_EXIT); +} + +static int complete_emulated_msr_access(struct kvm_vcpu *vcpu) +{ + return complete_emulated_insn_gp(vcpu, vcpu->run->msr.error); +} + +static int complete_emulated_rdmsr(struct kvm_vcpu *vcpu) +{ + complete_userspace_rdmsr(vcpu); + return complete_emulated_msr_access(vcpu); +} + +static int complete_fast_msr_access(struct kvm_vcpu *vcpu) +{ + return kvm_x86_call(complete_emulated_msr)(vcpu, vcpu->run->msr.error); +} + +static int complete_fast_rdmsr(struct kvm_vcpu *vcpu) +{ + complete_userspace_rdmsr(vcpu); + return complete_fast_msr_access(vcpu); +} + +static int complete_fast_rdmsr_imm(struct kvm_vcpu *vcpu) +{ + if (!vcpu->run->msr.error) + kvm_register_write(vcpu, vcpu->arch.cui_rdmsr_imm_reg, + vcpu->run->msr.data); + + return complete_fast_msr_access(vcpu); +} + +static u64 kvm_msr_reason(int r) +{ + switch (r) { + case KVM_MSR_RET_UNSUPPORTED: + return KVM_MSR_EXIT_REASON_UNKNOWN; + case KVM_MSR_RET_FILTERED: + return KVM_MSR_EXIT_REASON_FILTER; + default: + return KVM_MSR_EXIT_REASON_INVAL; + } +} + +static int kvm_msr_user_space(struct kvm_vcpu *vcpu, u32 index, + u32 exit_reason, u64 data, + int (*completion)(struct kvm_vcpu *vcpu), + int r) +{ + u64 msr_reason =3D kvm_msr_reason(r); + + /* Check if the user wanted to know about this MSR fault */ + if (!(vcpu->kvm->arch.user_space_msr_mask & msr_reason)) + return 0; + + vcpu->run->exit_reason =3D exit_reason; + vcpu->run->msr.error =3D 0; + memset(vcpu->run->msr.pad, 0, sizeof(vcpu->run->msr.pad)); + vcpu->run->msr.reason =3D msr_reason; + vcpu->run->msr.index =3D index; + vcpu->run->msr.data =3D data; + vcpu->arch.complete_userspace_io =3D completion; + + return 1; +} + +static int __kvm_emulate_rdmsr(struct kvm_vcpu *vcpu, u32 msr, int reg, + int (*complete_rdmsr)(struct kvm_vcpu *)) +{ + u64 data; + int r; + + r =3D kvm_emulate_msr_read(vcpu, msr, &data); + + if (!r) { + trace_kvm_msr_read(msr, data); + + if (reg < 0) { + kvm_eax_write(vcpu, data); + kvm_edx_write(vcpu, data >> 32); + } else { + kvm_register_write(vcpu, reg, data); + } + } else { + /* MSR read failed? See if we should ask user space */ + if (kvm_msr_user_space(vcpu, msr, KVM_EXIT_X86_RDMSR, 0, + complete_rdmsr, r)) + return 0; + trace_kvm_msr_read_ex(msr); + } + + return kvm_x86_call(complete_emulated_msr)(vcpu, r); +} + +int kvm_emulate_rdmsr(struct kvm_vcpu *vcpu) +{ + return __kvm_emulate_rdmsr(vcpu, kvm_ecx_read(vcpu), -1, + complete_fast_rdmsr); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_rdmsr); + +int kvm_emulate_rdmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg) +{ + vcpu->arch.cui_rdmsr_imm_reg =3D reg; + + return __kvm_emulate_rdmsr(vcpu, msr, reg, complete_fast_rdmsr_imm); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_rdmsr_imm); + +static int __kvm_emulate_wrmsr(struct kvm_vcpu *vcpu, u32 msr, u64 data) +{ + int r; + + r =3D kvm_emulate_msr_write(vcpu, msr, data); + if (!r) { + trace_kvm_msr_write(msr, data); + } else { + /* MSR write failed? See if we should ask user space */ + if (kvm_msr_user_space(vcpu, msr, KVM_EXIT_X86_WRMSR, data, + complete_fast_msr_access, r)) + return 0; + /* Signal all other negative errors to userspace */ + if (r < 0) + return r; + trace_kvm_msr_write_ex(msr, data); + } + + return kvm_x86_call(complete_emulated_msr)(vcpu, r); +} + +int kvm_emulate_wrmsr(struct kvm_vcpu *vcpu) +{ + return __kvm_emulate_wrmsr(vcpu, kvm_ecx_read(vcpu), + kvm_read_edx_eax(vcpu)); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_wrmsr); + +int kvm_emulate_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg) +{ + return __kvm_emulate_wrmsr(vcpu, msr, kvm_register_read(vcpu, reg)); +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_wrmsr_imm); + +int kvm_emulator_get_msr_with_filter(struct kvm_vcpu *vcpu, u32 msr_index, + u64 *pdata) +{ + int r; + + r =3D kvm_emulate_msr_read(vcpu, msr_index, pdata); + if (r < 0) + return X86EMUL_UNHANDLEABLE; + + if (r) { + if (kvm_msr_user_space(vcpu, msr_index, KVM_EXIT_X86_RDMSR, 0, + complete_emulated_rdmsr, r)) + return X86EMUL_IO_NEEDED; + + trace_kvm_msr_read_ex(msr_index); + return X86EMUL_PROPAGATE_FAULT; + } + + trace_kvm_msr_read(msr_index, *pdata); + return X86EMUL_CONTINUE; +} + +int kvm_emulator_set_msr_with_filter(struct kvm_vcpu *vcpu, u32 msr_index, + u64 data) +{ + int r; + + r =3D kvm_emulate_msr_write(vcpu, msr_index, data); + if (r < 0) + return X86EMUL_UNHANDLEABLE; + + if (r) { + if (kvm_msr_user_space(vcpu, msr_index, KVM_EXIT_X86_WRMSR, data, + complete_emulated_msr_access, r)) + return X86EMUL_IO_NEEDED; + + trace_kvm_msr_write_ex(msr_index, data); + return X86EMUL_PROPAGATE_FAULT; + } + + trace_kvm_msr_write(msr_index, data); + return X86EMUL_CONTINUE; +} + +int kvm_emulator_get_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 *pdata) +{ + /* + * Treat emulator accesses to the current shadow stack pointer as host- + * initiated, as they aren't true MSR accesses (SSP is a "just a reg"), + * and this API is used only for implicit accesses, i.e. not RDMSR, and + * so the index is fully KVM-controlled. + */ + if (unlikely(msr_index =3D=3D MSR_KVM_INTERNAL_GUEST_SSP)) + return kvm_msr_read(vcpu, msr_index, pdata); + + return __kvm_emulate_msr_read(vcpu, msr_index, pdata); +} + +/* + * Returns true if the MSR in question is managed via XSTATE, i.e. is cont= ext + * switched with the rest of guest FPU state. + * + * Note, S_CET is _not_ saved/restored via XSAVES/XRSTORS. + */ +static bool is_xstate_managed_msr(struct kvm_vcpu *vcpu, u32 msr) +{ + if (!vcpu) + return false; + + switch (msr) { + case MSR_IA32_U_CET: + return guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) || + guest_cpu_cap_has(vcpu, X86_FEATURE_IBT); + case MSR_IA32_PL0_SSP ... MSR_IA32_PL3_SSP: + return guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK); + default: + return false; + } +} + +/* + * Lock (and if necessary, re-load) the guest FPU, i.e. XSTATE, and access= an + * MSR that is managed via XSTATE. Note, the caller is responsible for do= ing + * the initial FPU load, this helper only ensures that guest state is resi= dent + * in hardware (the kernel can load its FPU state in IRQ context). + * + * Note, loading guest values for U_CET and PL[0-3]_SSP while executing in= the + * kernel is safe, as U_CET is specific to userspace, and PL[0-3]_SSP are = only + * consumed when transitioning to lower privilege levels, i.e. are effecti= vely + * only consumed by userspace as well. + */ +static __always_inline void kvm_access_xstate_msr(struct kvm_vcpu *vcpu, + struct msr_data *msr_info, + int access) +{ + BUILD_BUG_ON(access !=3D MSR_TYPE_R && access !=3D MSR_TYPE_W); + + KVM_BUG_ON(!is_xstate_managed_msr(vcpu, msr_info->index), vcpu->kvm); + KVM_BUG_ON(!vcpu->arch.guest_fpu.fpstate->in_use, vcpu->kvm); + + kvm_fpu_get(); + if (access =3D=3D MSR_TYPE_R) + rdmsrq(msr_info->index, msr_info->data); + else + wrmsrq(msr_info->index, msr_info->data); + kvm_fpu_put(); +} + +static void kvm_set_xstate_msr(struct kvm_vcpu *vcpu, struct msr_data *msr= _info) +{ + kvm_access_xstate_msr(vcpu, msr_info, MSR_TYPE_W); +} + +static void kvm_get_xstate_msr(struct kvm_vcpu *vcpu, struct msr_data *msr= _info) +{ + kvm_access_xstate_msr(vcpu, msr_info, MSR_TYPE_R); +} + +static void kvm_write_wall_clock(struct kvm *kvm, gpa_t wall_clock, int se= c_hi_ofs) +{ + int version; + int r; + struct pvclock_wall_clock wc; + u32 wc_sec_hi; + u64 wall_nsec; + + if (!wall_clock) + return; + + r =3D kvm_read_guest(kvm, wall_clock, &version, sizeof(version)); + if (r) + return; + + if (version & 1) + ++version; /* first time write, random junk */ + + ++version; + + if (kvm_write_guest(kvm, wall_clock, &version, sizeof(version))) + return; + + wall_nsec =3D kvm_get_wall_clock_epoch(kvm); + + wc.nsec =3D do_div(wall_nsec, NSEC_PER_SEC); + wc.sec =3D (u32)wall_nsec; /* overflow in 2106 guest time */ + wc.version =3D version; + + kvm_write_guest(kvm, wall_clock, &wc, sizeof(wc)); + + if (sec_hi_ofs) { + wc_sec_hi =3D wall_nsec >> 32; + kvm_write_guest(kvm, wall_clock + sec_hi_ofs, + &wc_sec_hi, sizeof(wc_sec_hi)); + } + + version++; + kvm_write_guest(kvm, wall_clock, &version, sizeof(version)); +} + +static void kvm_write_system_time(struct kvm_vcpu *vcpu, gpa_t system_time, + bool old_msr, bool host_initiated) +{ + struct kvm_arch *ka =3D &vcpu->kvm->arch; + + if (vcpu->vcpu_id =3D=3D 0 && !host_initiated) { + if (ka->boot_vcpu_runs_old_kvmclock !=3D old_msr) + kvm_make_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu); + + ka->boot_vcpu_runs_old_kvmclock =3D old_msr; + } + + vcpu->arch.time =3D system_time; + kvm_make_request(KVM_REQ_GLOBAL_CLOCK_UPDATE, vcpu); + + /* we verify if the enable bit is set... */ + if (system_time & 1) + kvm_gpc_activate(&vcpu->arch.pv_time, system_time & ~1ULL, + sizeof(struct pvclock_vcpu_time_info)); + else + kvm_gpc_deactivate(&vcpu->arch.pv_time); + + return; +} + +/* These helpers are safe iff @msr is known to be an MCx bank MSR. */ +static bool is_mci_control_msr(u32 msr) +{ + return (msr & 3) =3D=3D 0; +} +static bool is_mci_status_msr(u32 msr) +{ + return (msr & 3) =3D=3D 1; +} + +/* + * On AMD, HWCR[McStatusWrEn] controls whether setting MCi_STATUS results = in #GP. + */ +static bool can_set_mci_status(struct kvm_vcpu *vcpu) +{ + /* McStatusWrEn enabled? */ + if (guest_cpuid_is_amd_compatible(vcpu)) + return !!(vcpu->arch.msr_hwcr & BIT_ULL(18)); + + return false; +} + +static int set_msr_mce(struct kvm_vcpu *vcpu, struct msr_data *msr_info) +{ + u64 mcg_cap =3D vcpu->arch.mcg_cap; + unsigned bank_num =3D mcg_cap & 0xff; + u32 msr =3D msr_info->index; + u64 data =3D msr_info->data; + u32 offset, last_msr; + + switch (msr) { + case MSR_IA32_MCG_STATUS: + vcpu->arch.mcg_status =3D data; + break; + case MSR_IA32_MCG_CTL: + if (!(mcg_cap & MCG_CTL_P) && + (data || !msr_info->host_initiated)) + return 1; + if (data !=3D 0 && data !=3D ~(u64)0) + return 1; + vcpu->arch.mcg_ctl =3D data; + break; + case MSR_IA32_MC0_CTL2 ... MSR_IA32_MCx_CTL2(KVM_MAX_MCE_BANKS) - 1: + last_msr =3D MSR_IA32_MCx_CTL2(bank_num) - 1; + if (msr > last_msr) + return 1; + + if (!(mcg_cap & MCG_CMCI_P) && (data || !msr_info->host_initiated)) + return 1; + /* An attempt to write a 1 to a reserved bit raises #GP */ + if (data & ~(MCI_CTL2_CMCI_EN | MCI_CTL2_CMCI_THRESHOLD_MASK)) + return 1; + offset =3D array_index_nospec(msr - MSR_IA32_MC0_CTL2, + last_msr + 1 - MSR_IA32_MC0_CTL2); + vcpu->arch.mci_ctl2_banks[offset] =3D data; + break; + case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: + last_msr =3D MSR_IA32_MCx_CTL(bank_num) - 1; + if (msr > last_msr) + return 1; + + /* + * Only 0 or all 1s can be written to IA32_MCi_CTL, all other + * values are architecturally undefined. But, some Linux + * kernels clear bit 10 in bank 4 to workaround a BIOS/GART TLB + * issue on AMD K8s, allow bit 10 to be clear when setting all + * other bits in order to avoid an uncaught #GP in the guest. + * + * UNIXWARE clears bit 0 of MC1_CTL to ignore correctable, + * single-bit ECC data errors. + */ + if (is_mci_control_msr(msr) && + data !=3D 0 && (data | (1 << 10) | 1) !=3D ~(u64)0) + return 1; + + /* + * All CPUs allow writing 0 to MCi_STATUS MSRs to clear the MSR. + * AMD-based CPUs allow non-zero values, but if and only if + * HWCR[McStatusWrEn] is set. + */ + if (!msr_info->host_initiated && is_mci_status_msr(msr) && + data !=3D 0 && !can_set_mci_status(vcpu)) + return 1; + + offset =3D array_index_nospec(msr - MSR_IA32_MC0_CTL, + last_msr + 1 - MSR_IA32_MC0_CTL); + vcpu->arch.mce_banks[offset] =3D data; + break; + default: + return 1; + } + return 0; +} + +static int kvm_pv_enable_async_pf(struct kvm_vcpu *vcpu, u64 data) +{ + gpa_t gpa =3D data & ~0x3f; + + /* Bits 4:5 are reserved, Should be zero */ + if (data & 0x30) + return 1; + + if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_VMEXIT) && + (data & KVM_ASYNC_PF_DELIVERY_AS_PF_VMEXIT)) + return 1; + + if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT) && + (data & KVM_ASYNC_PF_DELIVERY_AS_INT)) + return 1; + + if (!lapic_in_kernel(vcpu)) + return data ? 1 : 0; + + if (__kvm_pv_async_pf_enabled(data) && + kvm_gfn_to_hva_cache_init(vcpu->kvm, &vcpu->arch.apf.data, gpa, + sizeof(u64))) + return 1; + + vcpu->arch.apf.msr_en_val =3D data; + + if (__kvm_pv_async_pf_enabled(data)) { + kvm_async_pf_wakeup_all(vcpu); + } else { + kvm_clear_async_pf_completion_queue(vcpu); + kvm_async_pf_hash_reset(vcpu); + } + return 0; +} + +static int kvm_pv_enable_async_pf_int(struct kvm_vcpu *vcpu, u64 data) +{ + /* Bits 8-63 are reserved */ + if (data >> 8) + return 1; + + if (!lapic_in_kernel(vcpu)) + return 1; + + vcpu->arch.apf.msr_int_val =3D data; + + vcpu->arch.apf.vec =3D data & KVM_ASYNC_PF_VEC_MASK; + + return 0; +} + +#ifdef CONFIG_X86_64 +static inline u64 kvm_guest_supported_xfd(struct kvm_vcpu *vcpu) +{ + return vcpu->arch.guest_supported_xcr0 & XFEATURE_MASK_USER_DYNAMIC; +} +#endif + +int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) +{ + u32 msr =3D msr_info->index; + u64 data =3D msr_info->data; + + /* + * Do not allow host-initiated writes to trigger the Xen hypercall + * page setup; it could incur locking paths which are not expected + * if userspace sets the MSR in an unusual location. + */ + if (kvm_xen_is_hypercall_page_msr(vcpu->kvm, msr) && + !msr_info->host_initiated) + return kvm_xen_write_hypercall_page(vcpu, data); + + switch (msr) { + case MSR_AMD64_NB_CFG: + case MSR_IA32_UCODE_WRITE: + case MSR_VM_HSAVE_PA: + case MSR_AMD64_PATCH_LOADER: + case MSR_AMD64_BU_CFG2: + case MSR_AMD64_DC_CFG: + case MSR_AMD64_TW_CFG: + case MSR_F15H_EX_CFG: + break; + + case MSR_IA32_UCODE_REV: + if (msr_info->host_initiated) + vcpu->arch.microcode_version =3D data; + break; + case MSR_IA32_ARCH_CAPABILITIES: + if (!msr_info->host_initiated || + !guest_cpu_cap_has(vcpu, X86_FEATURE_ARCH_CAPABILITIES)) + return KVM_MSR_RET_UNSUPPORTED; + vcpu->arch.arch_capabilities =3D data; + break; + case MSR_IA32_PERF_CAPABILITIES: + if (!msr_info->host_initiated || + !guest_cpu_cap_has(vcpu, X86_FEATURE_PDCM)) + return KVM_MSR_RET_UNSUPPORTED; + + if (data & ~kvm_caps.supported_perf_cap) + return 1; + + /* + * Note, this is not just a performance optimization! KVM + * disallows changing feature MSRs after the vCPU has run; PMU + * refresh will bug the VM if called after the vCPU has run. + */ + if (vcpu->arch.perf_capabilities =3D=3D data) + break; + + vcpu->arch.perf_capabilities =3D data; + kvm_pmu_refresh(vcpu); + kvm_make_request(KVM_REQ_RECALC_INTERCEPTS, vcpu); + break; + case MSR_IA32_PRED_CMD: { + u64 reserved_bits =3D ~(PRED_CMD_IBPB | PRED_CMD_SBPB); + + if (!msr_info->host_initiated) { + if ((!guest_has_pred_cmd_msr(vcpu))) + return 1; + + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SPEC_CTRL) && + !guest_cpu_cap_has(vcpu, X86_FEATURE_AMD_IBPB)) + reserved_bits |=3D PRED_CMD_IBPB; + + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SBPB)) + reserved_bits |=3D PRED_CMD_SBPB; + } + + if (!boot_cpu_has(X86_FEATURE_IBPB)) + reserved_bits |=3D PRED_CMD_IBPB; + + if (!boot_cpu_has(X86_FEATURE_SBPB)) + reserved_bits |=3D PRED_CMD_SBPB; + + if (data & reserved_bits) + return 1; + + if (!data) + break; + + wrmsrq(MSR_IA32_PRED_CMD, data); + break; + } + case MSR_IA32_FLUSH_CMD: + if (!msr_info->host_initiated && + !guest_cpu_cap_has(vcpu, X86_FEATURE_FLUSH_L1D)) + return 1; + + if (!boot_cpu_has(X86_FEATURE_FLUSH_L1D) || (data & ~L1D_FLUSH)) + return 1; + if (!data) + break; + + wrmsrq(MSR_IA32_FLUSH_CMD, L1D_FLUSH); + break; + case MSR_EFER: + return set_efer(vcpu, msr_info); + case MSR_K7_HWCR: { + /* + * Allow McStatusWrEn and TscFreqSel. (Linux guests from v3.2 + * through at least v6.6 whine if TscFreqSel is clear, + * depending on F/M/S. + */ + u64 valid =3D BIT_ULL(18) | BIT_ULL(24); + + data &=3D ~(u64)0x40; /* ignore flush filter disable */ + data &=3D ~(u64)0x100; /* ignore ignne emulation enable */ + data &=3D ~(u64)0x8; /* ignore TLB cache disable */ + + if (guest_cpu_cap_has(vcpu, X86_FEATURE_GP_ON_USER_CPUID)) + valid |=3D MSR_K7_HWCR_CPUID_USER_DIS; + + if (data & ~valid) { + kvm_pr_unimpl_wrmsr(vcpu, msr, data); + return 1; + } + vcpu->arch.msr_hwcr =3D data; + break; + } + case MSR_FAM10H_MMIO_CONF_BASE: + if (data !=3D 0) { + kvm_pr_unimpl_wrmsr(vcpu, msr, data); + return 1; + } + break; + case MSR_IA32_CR_PAT: + if (!kvm_pat_valid(data)) + return 1; + + vcpu->arch.pat =3D data; + break; + case MTRRphysBase_MSR(0) ... MSR_MTRRfix4K_F8000: + case MSR_MTRRdefType: + return kvm_mtrr_set_msr(vcpu, msr, data); + case MSR_IA32_APICBASE: + return kvm_apic_set_base(vcpu, data, msr_info->host_initiated); + case APIC_BASE_MSR ... APIC_BASE_MSR + 0xff: + return kvm_x2apic_msr_write(vcpu, msr, data); + case MSR_IA32_TSC_DEADLINE: + kvm_set_lapic_tscdeadline_msr(vcpu, data); + break; + case MSR_IA32_TSC_ADJUST: + if (guest_cpu_cap_has(vcpu, X86_FEATURE_TSC_ADJUST)) { + if (!msr_info->host_initiated) { + s64 adj =3D data - vcpu->arch.ia32_tsc_adjust_msr; + adjust_tsc_offset_guest(vcpu, adj); + /* Before back to guest, tsc_timestamp must be adjusted + * as well, otherwise guest's percpu pvclock time could jump. + */ + kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); + } + vcpu->arch.ia32_tsc_adjust_msr =3D data; + } + break; + case MSR_IA32_MISC_ENABLE: { + u64 old_val =3D vcpu->arch.ia32_misc_enable_msr; + + if (!msr_info->host_initiated) { + /* RO bits */ + if ((old_val ^ data) & MSR_IA32_MISC_ENABLE_PMU_RO_MASK) + return 1; + + /* R bits, i.e. writes are ignored, but don't fault. */ + data =3D data & ~MSR_IA32_MISC_ENABLE_EMON; + data |=3D old_val & MSR_IA32_MISC_ENABLE_EMON; + } + + if (!kvm_check_has_quirk(vcpu->kvm, KVM_X86_QUIRK_MISC_ENABLE_NO_MWAIT) = && + ((old_val ^ data) & MSR_IA32_MISC_ENABLE_MWAIT)) { + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_XMM3)) + return 1; + vcpu->arch.ia32_misc_enable_msr =3D data; + vcpu->arch.cpuid_dynamic_bits_dirty =3D true; + } else { + vcpu->arch.ia32_misc_enable_msr =3D data; + } + break; + } + case MSR_IA32_SMBASE: + if (!IS_ENABLED(CONFIG_KVM_SMM) || !msr_info->host_initiated) + return 1; + vcpu->arch.smbase =3D data; + break; + case MSR_IA32_POWER_CTL: + vcpu->arch.msr_ia32_power_ctl =3D data; + break; + case MSR_IA32_TSC: + if (msr_info->host_initiated) { + kvm_synchronize_tsc(vcpu, &data); + } else if (!vcpu->arch.guest_tsc_protected) { + u64 adj =3D kvm_compute_l1_tsc_offset(vcpu, data) - vcpu->arch.l1_tsc_o= ffset; + adjust_tsc_offset_guest(vcpu, adj); + vcpu->arch.ia32_tsc_adjust_msr +=3D adj; + } + break; + case MSR_IA32_XSS: + if (!guest_cpuid_has(vcpu, X86_FEATURE_XSAVES)) + return KVM_MSR_RET_UNSUPPORTED; + + if (data & ~vcpu->arch.guest_supported_xss) + return 1; + if (vcpu->arch.ia32_xss =3D=3D data) + break; + vcpu->arch.ia32_xss =3D data; + vcpu->arch.cpuid_dynamic_bits_dirty =3D true; + break; + case MSR_SMI_COUNT: + if (!msr_info->host_initiated) + return 1; + vcpu->arch.smi_count =3D data; + break; + case MSR_KVM_WALL_CLOCK_NEW: + if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE2)) + return KVM_MSR_RET_UNSUPPORTED; + + vcpu->kvm->arch.wall_clock =3D data; + kvm_write_wall_clock(vcpu->kvm, data, 0); + break; + case MSR_KVM_WALL_CLOCK: + if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE)) + return KVM_MSR_RET_UNSUPPORTED; + + vcpu->kvm->arch.wall_clock =3D data; + kvm_write_wall_clock(vcpu->kvm, data, 0); + break; + case MSR_KVM_SYSTEM_TIME_NEW: + if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE2)) + return KVM_MSR_RET_UNSUPPORTED; + + kvm_write_system_time(vcpu, data, false, msr_info->host_initiated); + break; + case MSR_KVM_SYSTEM_TIME: + if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE)) + return KVM_MSR_RET_UNSUPPORTED; + + kvm_write_system_time(vcpu, data, true, msr_info->host_initiated); + break; + case MSR_KVM_ASYNC_PF_EN: + if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF)) + return KVM_MSR_RET_UNSUPPORTED; + + if (kvm_pv_enable_async_pf(vcpu, data)) + return 1; + break; + case MSR_KVM_ASYNC_PF_INT: + if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT)) + return KVM_MSR_RET_UNSUPPORTED; + + if (kvm_pv_enable_async_pf_int(vcpu, data)) + return 1; + break; + case MSR_KVM_ASYNC_PF_ACK: + if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT)) + return KVM_MSR_RET_UNSUPPORTED; + if (data & 0x1) { + /* + * Pairs with the smp_mb__after_atomic() in + * kvm_arch_async_page_present_queued(). + */ + smp_store_mb(vcpu->arch.apf.pageready_pending, false); + + kvm_check_async_pf_completion(vcpu); + } + break; + case MSR_KVM_STEAL_TIME: + if (!guest_pv_has(vcpu, KVM_FEATURE_STEAL_TIME)) + return KVM_MSR_RET_UNSUPPORTED; + + if (unlikely(!sched_info_on())) + return 1; + + if (data & KVM_STEAL_RESERVED_MASK) + return 1; + + vcpu->arch.st.msr_val =3D data; + + if (!(data & KVM_MSR_ENABLED)) + break; + + kvm_make_request(KVM_REQ_STEAL_UPDATE, vcpu); + + break; + case MSR_KVM_PV_EOI_EN: + if (!guest_pv_has(vcpu, KVM_FEATURE_PV_EOI)) + return KVM_MSR_RET_UNSUPPORTED; + + if (kvm_lapic_set_pv_eoi(vcpu, data, sizeof(u8))) + return 1; + break; + + case MSR_KVM_POLL_CONTROL: + if (!guest_pv_has(vcpu, KVM_FEATURE_POLL_CONTROL)) + return KVM_MSR_RET_UNSUPPORTED; + + /* only enable bit supported */ + if (data & (-1ULL << 1)) + return 1; + + vcpu->arch.msr_kvm_poll_control =3D data; + break; + + case MSR_IA32_MCG_CTL: + case MSR_IA32_MCG_STATUS: + case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: + case MSR_IA32_MC0_CTL2 ... MSR_IA32_MCx_CTL2(KVM_MAX_MCE_BANKS) - 1: + return set_msr_mce(vcpu, msr_info); + + case MSR_K7_PERFCTR0 ... MSR_K7_PERFCTR3: + case MSR_P6_PERFCTR0 ... MSR_P6_PERFCTR1: + case MSR_K7_EVNTSEL0 ... MSR_K7_EVNTSEL3: + case MSR_P6_EVNTSEL0 ... MSR_P6_EVNTSEL1: + if (kvm_pmu_is_valid_msr(vcpu, msr)) + return kvm_pmu_set_msr(vcpu, msr_info); + + if (data) + kvm_pr_unimpl_wrmsr(vcpu, msr, data); + break; + case MSR_K7_CLK_CTL: + /* + * Ignore all writes to this no longer documented MSR. + * Writes are only relevant for old K7 processors, + * all pre-dating SVM, but a recommended workaround from + * AMD for these chips. It is possible to specify the + * affected processor models on the command line, hence + * the need to ignore the workaround. + */ + break; +#ifdef CONFIG_KVM_HYPERV + case HV_X64_MSR_GUEST_OS_ID ... HV_X64_MSR_SINT15: + case HV_X64_MSR_SYNDBG_CONTROL ... HV_X64_MSR_SYNDBG_PENDING_BUFFER: + case HV_X64_MSR_SYNDBG_OPTIONS: + case HV_X64_MSR_CRASH_P0 ... HV_X64_MSR_CRASH_P4: + case HV_X64_MSR_CRASH_CTL: + case HV_X64_MSR_STIMER0_CONFIG ... HV_X64_MSR_STIMER3_COUNT: + case HV_X64_MSR_REENLIGHTENMENT_CONTROL: + case HV_X64_MSR_TSC_EMULATION_CONTROL: + case HV_X64_MSR_TSC_EMULATION_STATUS: + case HV_X64_MSR_TSC_INVARIANT_CONTROL: + return kvm_hv_set_msr_common(vcpu, msr, data, + msr_info->host_initiated); +#endif + case MSR_IA32_BBL_CR_CTL3: + /* Drop writes to this legacy MSR -- see rdmsr + * counterpart for further detail. + */ + kvm_pr_unimpl_wrmsr(vcpu, msr, data); + break; + case MSR_AMD64_OSVW_ID_LENGTH: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_OSVW)) + return 1; + vcpu->arch.osvw.length =3D data; + break; + case MSR_AMD64_OSVW_STATUS: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_OSVW)) + return 1; + vcpu->arch.osvw.status =3D data; + break; + case MSR_PLATFORM_INFO: + if (!msr_info->host_initiated) + return 1; + vcpu->arch.msr_platform_info =3D data; + break; + case MSR_MISC_FEATURES_ENABLES: + if (data & ~MSR_MISC_FEATURES_ENABLES_CPUID_FAULT || + (data & MSR_MISC_FEATURES_ENABLES_CPUID_FAULT && + !(vcpu->arch.msr_platform_info & MSR_PLATFORM_INFO_CPUID_FAULT))) + return 1; + vcpu->arch.msr_misc_features_enables =3D data; + break; +#ifdef CONFIG_X86_64 + case MSR_IA32_XFD: + if (!msr_info->host_initiated && + !guest_cpu_cap_has(vcpu, X86_FEATURE_XFD)) + return 1; + + if (data & ~kvm_guest_supported_xfd(vcpu)) + return 1; + + fpu_update_guest_xfd(&vcpu->arch.guest_fpu, data); + break; + case MSR_IA32_XFD_ERR: + if (!msr_info->host_initiated && + !guest_cpu_cap_has(vcpu, X86_FEATURE_XFD)) + return 1; + + if (data & ~kvm_guest_supported_xfd(vcpu)) + return 1; + + vcpu->arch.guest_fpu.xfd_err =3D data; + break; +#endif + case MSR_IA32_U_CET: + case MSR_IA32_PL0_SSP ... MSR_IA32_PL3_SSP: + kvm_set_xstate_msr(vcpu, msr_info); + break; + default: + if (kvm_pmu_is_valid_msr(vcpu, msr)) + return kvm_pmu_set_msr(vcpu, msr_info); + + return KVM_MSR_RET_UNSUPPORTED; + } + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_msr_common); + +static int get_msr_mce(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata, bool ho= st) +{ + u64 data; + u64 mcg_cap =3D vcpu->arch.mcg_cap; + unsigned bank_num =3D mcg_cap & 0xff; + u32 offset, last_msr; + + switch (msr) { + case MSR_IA32_P5_MC_ADDR: + case MSR_IA32_P5_MC_TYPE: + data =3D 0; + break; + case MSR_IA32_MCG_CAP: + data =3D vcpu->arch.mcg_cap; + break; + case MSR_IA32_MCG_CTL: + if (!(mcg_cap & MCG_CTL_P) && !host) + return 1; + data =3D vcpu->arch.mcg_ctl; + break; + case MSR_IA32_MCG_STATUS: + data =3D vcpu->arch.mcg_status; + break; + case MSR_IA32_MC0_CTL2 ... MSR_IA32_MCx_CTL2(KVM_MAX_MCE_BANKS) - 1: + last_msr =3D MSR_IA32_MCx_CTL2(bank_num) - 1; + if (msr > last_msr) + return 1; + + if (!(mcg_cap & MCG_CMCI_P) && !host) + return 1; + offset =3D array_index_nospec(msr - MSR_IA32_MC0_CTL2, + last_msr + 1 - MSR_IA32_MC0_CTL2); + data =3D vcpu->arch.mci_ctl2_banks[offset]; + break; + case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: + last_msr =3D MSR_IA32_MCx_CTL(bank_num) - 1; + if (msr > last_msr) + return 1; + + offset =3D array_index_nospec(msr - MSR_IA32_MC0_CTL, + last_msr + 1 - MSR_IA32_MC0_CTL); + data =3D vcpu->arch.mce_banks[offset]; + break; + default: + return 1; + } + *pdata =3D data; + return 0; +} + +int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) +{ + switch (msr_info->index) { + case MSR_IA32_PLATFORM_ID: + case MSR_IA32_EBL_CR_POWERON: + case MSR_IA32_LASTBRANCHFROMIP: + case MSR_IA32_LASTBRANCHTOIP: + case MSR_IA32_LASTINTFROMIP: + case MSR_IA32_LASTINTTOIP: + case MSR_AMD64_SYSCFG: + case MSR_K8_TSEG_ADDR: + case MSR_K8_TSEG_MASK: + case MSR_VM_HSAVE_PA: + case MSR_K8_INT_PENDING_MSG: + case MSR_AMD64_NB_CFG: + case MSR_FAM10H_MMIO_CONF_BASE: + case MSR_AMD64_BU_CFG2: + case MSR_IA32_PERF_CTL: + case MSR_AMD64_DC_CFG: + case MSR_AMD64_TW_CFG: + case MSR_F15H_EX_CFG: + /* + * Intel Sandy Bridge CPUs must support the RAPL (running average power + * limit) MSRs. Just return 0, as we do not want to expose the host + * data here. Do not conditionalize this on CPUID, as KVM does not do + * so for existing CPU-specific MSRs. + */ + case MSR_RAPL_POWER_UNIT: + case MSR_PP0_ENERGY_STATUS: /* Power plane 0 (core) */ + case MSR_PP1_ENERGY_STATUS: /* Power plane 1 (graphics uncore) */ + case MSR_PKG_ENERGY_STATUS: /* Total package */ + case MSR_DRAM_ENERGY_STATUS: /* DRAM controller */ + msr_info->data =3D 0; + break; + case MSR_K7_EVNTSEL0 ... MSR_K7_EVNTSEL3: + case MSR_K7_PERFCTR0 ... MSR_K7_PERFCTR3: + case MSR_P6_PERFCTR0 ... MSR_P6_PERFCTR1: + case MSR_P6_EVNTSEL0 ... MSR_P6_EVNTSEL1: + if (kvm_pmu_is_valid_msr(vcpu, msr_info->index)) + return kvm_pmu_get_msr(vcpu, msr_info); + msr_info->data =3D 0; + break; + case MSR_IA32_UCODE_REV: + msr_info->data =3D vcpu->arch.microcode_version; + break; + case MSR_IA32_ARCH_CAPABILITIES: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_ARCH_CAPABILITIES)) + return KVM_MSR_RET_UNSUPPORTED; + msr_info->data =3D vcpu->arch.arch_capabilities; + break; + case MSR_IA32_PERF_CAPABILITIES: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_PDCM)) + return KVM_MSR_RET_UNSUPPORTED; + msr_info->data =3D vcpu->arch.perf_capabilities; + break; + case MSR_IA32_POWER_CTL: + msr_info->data =3D vcpu->arch.msr_ia32_power_ctl; + break; + case MSR_IA32_TSC: { + /* + * Intel SDM states that MSR_IA32_TSC read adds the TSC offset + * even when not intercepted. AMD manual doesn't explicitly + * state this but appears to behave the same. + * + * On userspace reads and writes, however, we unconditionally + * return L1's TSC value to ensure backwards-compatible + * behavior for migration. + */ + u64 offset, ratio; + + if (msr_info->host_initiated) { + offset =3D vcpu->arch.l1_tsc_offset; + ratio =3D vcpu->arch.l1_tsc_scaling_ratio; + } else { + offset =3D vcpu->arch.tsc_offset; + ratio =3D vcpu->arch.tsc_scaling_ratio; + } + + msr_info->data =3D kvm_scale_tsc(rdtsc(), ratio) + offset; + break; + } + case MSR_IA32_CR_PAT: + msr_info->data =3D vcpu->arch.pat; + break; + case MSR_MTRRcap: + case MTRRphysBase_MSR(0) ... MSR_MTRRfix4K_F8000: + case MSR_MTRRdefType: + return kvm_mtrr_get_msr(vcpu, msr_info->index, &msr_info->data); + case 0xcd: /* fsb frequency */ + msr_info->data =3D 3; + break; + /* + * MSR_EBC_FREQUENCY_ID + * Conservative value valid for even the basic CPU models. + * Models 0,1: 000 in bits 23:21 indicating a bus speed of + * 100MHz, model 2 000 in bits 18:16 indicating 100MHz, + * and 266MHz for model 3, or 4. Set Core Clock + * Frequency to System Bus Frequency Ratio to 1 (bits + * 31:24) even though these are only valid for CPU + * models > 2, however guests may end up dividing or + * multiplying by zero otherwise. + */ + case MSR_EBC_FREQUENCY_ID: + msr_info->data =3D 1 << 24; + break; + case MSR_IA32_APICBASE: + msr_info->data =3D vcpu->arch.apic_base; + break; + case APIC_BASE_MSR ... APIC_BASE_MSR + 0xff: + return kvm_x2apic_msr_read(vcpu, msr_info->index, &msr_info->data); + case MSR_IA32_TSC_DEADLINE: + msr_info->data =3D kvm_get_lapic_tscdeadline_msr(vcpu); + break; + case MSR_IA32_TSC_ADJUST: + msr_info->data =3D (u64)vcpu->arch.ia32_tsc_adjust_msr; + break; + case MSR_IA32_MISC_ENABLE: + msr_info->data =3D vcpu->arch.ia32_misc_enable_msr; + break; + case MSR_IA32_SMBASE: + if (!IS_ENABLED(CONFIG_KVM_SMM) || !msr_info->host_initiated) + return 1; + msr_info->data =3D vcpu->arch.smbase; + break; + case MSR_SMI_COUNT: + msr_info->data =3D vcpu->arch.smi_count; + break; + case MSR_IA32_PERF_STATUS: + /* TSC increment by tick */ + msr_info->data =3D 1000ULL; + /* CPU multiplier */ + msr_info->data |=3D (((uint64_t)4ULL) << 40); + break; + case MSR_EFER: + msr_info->data =3D vcpu->arch.efer; + break; + case MSR_KVM_WALL_CLOCK: + if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data =3D vcpu->kvm->arch.wall_clock; + break; + case MSR_KVM_WALL_CLOCK_NEW: + if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE2)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data =3D vcpu->kvm->arch.wall_clock; + break; + case MSR_KVM_SYSTEM_TIME: + if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data =3D vcpu->arch.time; + break; + case MSR_KVM_SYSTEM_TIME_NEW: + if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE2)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data =3D vcpu->arch.time; + break; + case MSR_KVM_ASYNC_PF_EN: + if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data =3D vcpu->arch.apf.msr_en_val; + break; + case MSR_KVM_ASYNC_PF_INT: + if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data =3D vcpu->arch.apf.msr_int_val; + break; + case MSR_KVM_ASYNC_PF_ACK: + if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data =3D 0; + break; + case MSR_KVM_STEAL_TIME: + if (!guest_pv_has(vcpu, KVM_FEATURE_STEAL_TIME)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data =3D vcpu->arch.st.msr_val; + break; + case MSR_KVM_PV_EOI_EN: + if (!guest_pv_has(vcpu, KVM_FEATURE_PV_EOI)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data =3D vcpu->arch.pv_eoi.msr_val; + break; + case MSR_KVM_POLL_CONTROL: + if (!guest_pv_has(vcpu, KVM_FEATURE_POLL_CONTROL)) + return KVM_MSR_RET_UNSUPPORTED; + + msr_info->data =3D vcpu->arch.msr_kvm_poll_control; + break; + case MSR_IA32_P5_MC_ADDR: + case MSR_IA32_P5_MC_TYPE: + case MSR_IA32_MCG_CAP: + case MSR_IA32_MCG_CTL: + case MSR_IA32_MCG_STATUS: + case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: + case MSR_IA32_MC0_CTL2 ... MSR_IA32_MCx_CTL2(KVM_MAX_MCE_BANKS) - 1: + return get_msr_mce(vcpu, msr_info->index, &msr_info->data, + msr_info->host_initiated); + case MSR_IA32_XSS: + if (!msr_info->host_initiated && + !guest_cpuid_has(vcpu, X86_FEATURE_XSAVES)) + return 1; + msr_info->data =3D vcpu->arch.ia32_xss; + break; + case MSR_K7_CLK_CTL: + /* + * Provide expected ramp-up count for K7. All other + * are set to zero, indicating minimum divisors for + * every field. + * + * This prevents guest kernels on AMD host with CPU + * type 6, model 8 and higher from exploding due to + * the rdmsr failing. + */ + msr_info->data =3D 0x20000000; + break; +#ifdef CONFIG_KVM_HYPERV + case HV_X64_MSR_GUEST_OS_ID ... HV_X64_MSR_SINT15: + case HV_X64_MSR_SYNDBG_CONTROL ... HV_X64_MSR_SYNDBG_PENDING_BUFFER: + case HV_X64_MSR_SYNDBG_OPTIONS: + case HV_X64_MSR_CRASH_P0 ... HV_X64_MSR_CRASH_P4: + case HV_X64_MSR_CRASH_CTL: + case HV_X64_MSR_STIMER0_CONFIG ... HV_X64_MSR_STIMER3_COUNT: + case HV_X64_MSR_REENLIGHTENMENT_CONTROL: + case HV_X64_MSR_TSC_EMULATION_CONTROL: + case HV_X64_MSR_TSC_EMULATION_STATUS: + case HV_X64_MSR_TSC_INVARIANT_CONTROL: + return kvm_hv_get_msr_common(vcpu, + msr_info->index, &msr_info->data, + msr_info->host_initiated); +#endif + case MSR_IA32_BBL_CR_CTL3: + /* This legacy MSR exists but isn't fully documented in current + * silicon. It is however accessed by winxp in very narrow + * scenarios where it sets bit #19, itself documented as + * a "reserved" bit. Best effort attempt to source coherent + * read data here should the balance of the register be + * interpreted by the guest: + * + * L2 cache control register 3: 64GB range, 256KB size, + * enabled, latency 0x1, configured + */ + msr_info->data =3D 0xbe702111; + break; + case MSR_AMD64_OSVW_ID_LENGTH: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_OSVW)) + return 1; + msr_info->data =3D vcpu->arch.osvw.length; + break; + case MSR_AMD64_OSVW_STATUS: + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_OSVW)) + return 1; + msr_info->data =3D vcpu->arch.osvw.status; + break; + case MSR_PLATFORM_INFO: + if (!msr_info->host_initiated && + !vcpu->kvm->arch.guest_can_read_msr_platform_info) + return 1; + msr_info->data =3D vcpu->arch.msr_platform_info; + break; + case MSR_MISC_FEATURES_ENABLES: + msr_info->data =3D vcpu->arch.msr_misc_features_enables; + break; + case MSR_K7_HWCR: + msr_info->data =3D vcpu->arch.msr_hwcr; + break; +#ifdef CONFIG_X86_64 + case MSR_IA32_XFD: + if (!msr_info->host_initiated && + !guest_cpu_cap_has(vcpu, X86_FEATURE_XFD)) + return 1; + + msr_info->data =3D vcpu->arch.guest_fpu.fpstate->xfd; + break; + case MSR_IA32_XFD_ERR: + if (!msr_info->host_initiated && + !guest_cpu_cap_has(vcpu, X86_FEATURE_XFD)) + return 1; + + msr_info->data =3D vcpu->arch.guest_fpu.xfd_err; + break; +#endif + case MSR_IA32_U_CET: + case MSR_IA32_PL0_SSP ... MSR_IA32_PL3_SSP: + kvm_get_xstate_msr(vcpu, msr_info); + break; + default: + if (kvm_pmu_is_valid_msr(vcpu, msr_info->index)) + return kvm_pmu_get_msr(vcpu, msr_info); + + return KVM_MSR_RET_UNSUPPORTED; + } + return 0; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_msr_common); + +static int do_get_msr(struct kvm_vcpu *vcpu, unsigned index, u64 *data) +{ + return kvm_get_msr_ignored_check(vcpu, index, data, true); +} + +static int do_set_msr(struct kvm_vcpu *vcpu, unsigned index, u64 *data) +{ + u64 val; + + /* + * Reject writes to immutable feature MSRs if the vCPU model is frozen, + * as KVM doesn't support modifying the guest vCPU model on the fly, + * e.g. changing the VMX capabilities MSRs while L2 is active is + * nonsensical. Allow writes of the same value, e.g. so that userspace + * can blindly stuff all MSRs when emulating RESET. + */ + if (!kvm_can_set_cpuid_and_feature_msrs(vcpu) && + kvm_is_immutable_feature_msr(index) && + (do_get_msr(vcpu, index, &val) || *data !=3D val)) + return -EINVAL; + + return kvm_set_msr_ignored_check(vcpu, index, *data, true); +} + +/* + * Read or write a bunch of msrs. All parameters are kernel addresses. + * + * @return number of msrs set successfully. + */ +static int __msr_io(struct kvm_vcpu *vcpu, struct kvm_msrs *msrs, + struct kvm_msr_entry *entries, + int (*do_msr)(struct kvm_vcpu *vcpu, + unsigned index, u64 *data)) +{ + bool fpu_loaded =3D false; + int i; + + for (i =3D 0; i < msrs->nmsrs; ++i) { + /* + * If userspace is accessing one or more XSTATE-managed MSRs, + * temporarily load the guest's FPU state so that the guest's + * MSR value(s) is resident in hardware and thus can be accessed + * via RDMSR/WRMSR. + */ + if (!fpu_loaded && is_xstate_managed_msr(vcpu, entries[i].index)) { + kvm_load_guest_fpu(vcpu); + fpu_loaded =3D true; + } + if (do_msr(vcpu, entries[i].index, &entries[i].data)) + break; + } + if (fpu_loaded) + kvm_put_guest_fpu(vcpu); + + return i; +} + +/* + * Read or write a bunch of msrs. Parameters are user addresses. + * + * @return number of msrs set successfully. + */ +static int msr_io(struct kvm_vcpu *vcpu, struct kvm_msrs __user *user_msrs, + int (*do_msr)(struct kvm_vcpu *vcpu, + unsigned index, u64 *data), + int writeback) +{ + struct kvm_msrs msrs; + struct kvm_msr_entry *entries; + unsigned size; + int r; + + r =3D -EFAULT; + if (copy_from_user(&msrs, user_msrs, sizeof(msrs))) + goto out; + + r =3D -E2BIG; + if (msrs.nmsrs >=3D MAX_IO_MSRS) + goto out; + + size =3D sizeof(struct kvm_msr_entry) * msrs.nmsrs; + entries =3D memdup_user(user_msrs->entries, size); + if (IS_ERR(entries)) { + r =3D PTR_ERR(entries); + goto out; + } + + r =3D __msr_io(vcpu, &msrs, entries, do_msr); + + if (writeback && copy_to_user(user_msrs->entries, entries, size)) + r =3D -EFAULT; + + kfree(entries); +out: + return r; +} + +int kvm_get_feature_msrs(struct kvm_msrs __user *user_msrs) +{ + return msr_io(NULL, user_msrs, do_get_feature_msr, 1); +} + +int kvm_get_msrs(struct kvm_vcpu *vcpu, struct kvm_msrs __user *user_msrs) +{ + guard(srcu)(&vcpu->kvm->srcu); + + return msr_io(vcpu, user_msrs, do_get_msr, 1); +} + +int kvm_set_msrs(struct kvm_vcpu *vcpu, struct kvm_msrs __user *user_msrs) +{ + guard(srcu)(&vcpu->kvm->srcu); + + return msr_io(vcpu, user_msrs, do_set_msr, 0); +} + +static int kvm_get_one_msr(struct kvm_vcpu *vcpu, u32 msr, u64 __user *use= r_val) +{ + u64 val; + + if (do_get_msr(vcpu, msr, &val)) + return -EINVAL; + + if (put_user(val, user_val)) + return -EFAULT; + + return 0; +} + +static int kvm_set_one_msr(struct kvm_vcpu *vcpu, u32 msr, u64 __user *use= r_val) +{ + u64 val; + + if (get_user(val, user_val)) + return -EFAULT; + + if (do_set_msr(vcpu, msr, &val)) + return -EINVAL; + + return 0; +} + +struct kvm_x86_reg_id { + __u32 index; + __u8 type; + __u8 rsvd1; + __u8 rsvd2:4; + __u8 size:4; + __u8 x86; +}; + +static int kvm_translate_kvm_reg(struct kvm_vcpu *vcpu, + struct kvm_x86_reg_id *reg) +{ + switch (reg->index) { + case KVM_REG_GUEST_SSP: + /* + * FIXME: If host-initiated accesses are ever exempted from + * ignore_msrs (in kvm_do_msr_access()), drop this manual check + * and rely on KVM's standard checks to reject accesses to regs + * that don't exist. + */ + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK)) + return -EINVAL; + + reg->type =3D KVM_X86_REG_TYPE_MSR; + reg->index =3D MSR_KVM_INTERNAL_GUEST_SSP; + break; + default: + return -EINVAL; + } + return 0; +} + +int kvm_get_set_one_reg(struct kvm_vcpu *vcpu, unsigned int ioctl, + void __user *argp) +{ + struct kvm_one_reg one_reg; + struct kvm_x86_reg_id *reg; + u64 __user *user_val; + bool load_fpu; + int r; + + if (copy_from_user(&one_reg, argp, sizeof(one_reg))) + return -EFAULT; + + if ((one_reg.id & KVM_REG_ARCH_MASK) !=3D KVM_REG_X86) + return -EINVAL; + + reg =3D (struct kvm_x86_reg_id *)&one_reg.id; + if (reg->rsvd1 || reg->rsvd2) + return -EINVAL; + + if (reg->type =3D=3D KVM_X86_REG_TYPE_KVM) { + r =3D kvm_translate_kvm_reg(vcpu, reg); + if (r) + return r; + } + + if (reg->type !=3D KVM_X86_REG_TYPE_MSR) + return -EINVAL; + + if ((one_reg.id & KVM_REG_SIZE_MASK) !=3D KVM_REG_SIZE_U64) + return -EINVAL; + + guard(srcu)(&vcpu->kvm->srcu); + + load_fpu =3D is_xstate_managed_msr(vcpu, reg->index); + if (load_fpu) + kvm_load_guest_fpu(vcpu); + + user_val =3D u64_to_user_ptr(one_reg.addr); + if (ioctl =3D=3D KVM_GET_ONE_REG) + r =3D kvm_get_one_msr(vcpu, reg->index, user_val); + else + r =3D kvm_set_one_msr(vcpu, reg->index, user_val); + + if (load_fpu) + kvm_put_guest_fpu(vcpu); + return r; +} + +int kvm_get_reg_list(struct kvm_vcpu *vcpu, + struct kvm_reg_list __user *user_list) +{ + u64 nr_regs =3D guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) ? 1 : 0; + u64 user_nr_regs; + + if (get_user(user_nr_regs, &user_list->n)) + return -EFAULT; + + if (put_user(nr_regs, &user_list->n)) + return -EFAULT; + + if (user_nr_regs < nr_regs) + return -E2BIG; + + if (nr_regs && + put_user(KVM_X86_REG_KVM(KVM_REG_GUEST_SSP), &user_list->reg[0])) + return -EFAULT; + + return 0; +} + +static struct kvm_x86_msr_filter *kvm_alloc_msr_filter(bool default_allow) +{ + struct kvm_x86_msr_filter *msr_filter; + + msr_filter =3D kzalloc_obj(*msr_filter, GFP_KERNEL_ACCOUNT); + if (!msr_filter) + return NULL; + + msr_filter->default_allow =3D default_allow; + return msr_filter; +} + +void kvm_free_msr_filter(struct kvm_x86_msr_filter *msr_filter) +{ + u32 i; + + if (!msr_filter) + return; + + for (i =3D 0; i < msr_filter->count; i++) + kfree(msr_filter->ranges[i].bitmap); + + kfree(msr_filter); +} + +static int kvm_add_msr_filter(struct kvm_x86_msr_filter *msr_filter, + struct kvm_msr_filter_range *user_range) +{ + unsigned long *bitmap; + size_t bitmap_size; + + if (!user_range->nmsrs) + return 0; + + if (user_range->flags & ~KVM_MSR_FILTER_RANGE_VALID_MASK) + return -EINVAL; + + if (!user_range->flags) + return -EINVAL; + + bitmap_size =3D BITS_TO_LONGS(user_range->nmsrs) * sizeof(long); + if (!bitmap_size || bitmap_size > KVM_MSR_FILTER_MAX_BITMAP_SIZE) + return -EINVAL; + + bitmap =3D memdup_user((__user u8*)user_range->bitmap, bitmap_size); + if (IS_ERR(bitmap)) + return PTR_ERR(bitmap); + + msr_filter->ranges[msr_filter->count] =3D (struct msr_bitmap_range) { + .flags =3D user_range->flags, + .base =3D user_range->base, + .nmsrs =3D user_range->nmsrs, + .bitmap =3D bitmap, + }; + + msr_filter->count++; + return 0; +} + +int kvm_vm_ioctl_set_msr_filter(struct kvm *kvm, struct kvm_msr_filter *fi= lter) +{ + struct kvm_x86_msr_filter *new_filter, *old_filter; + bool default_allow; + bool empty =3D true; + int r; + u32 i; + + if (filter->flags & ~KVM_MSR_FILTER_VALID_MASK) + return -EINVAL; + + for (i =3D 0; i < ARRAY_SIZE(filter->ranges); i++) + empty &=3D !filter->ranges[i].nmsrs; + + default_allow =3D !(filter->flags & KVM_MSR_FILTER_DEFAULT_DENY); + if (empty && !default_allow) + return -EINVAL; + + new_filter =3D kvm_alloc_msr_filter(default_allow); + if (!new_filter) + return -ENOMEM; + + for (i =3D 0; i < ARRAY_SIZE(filter->ranges); i++) { + r =3D kvm_add_msr_filter(new_filter, &filter->ranges[i]); + if (r) { + kvm_free_msr_filter(new_filter); + return r; + } + } + + mutex_lock(&kvm->lock); + old_filter =3D rcu_replace_pointer(kvm->arch.msr_filter, new_filter, + mutex_is_locked(&kvm->lock)); + mutex_unlock(&kvm->lock); + synchronize_srcu(&kvm->srcu); + + kvm_free_msr_filter(old_filter); + + /* + * Recalc MSR intercepts as userspace may want to intercept accesses to + * MSRs that KVM would otherwise pass through to the guest. + */ + kvm_make_all_cpus_request(kvm, KVM_REQ_RECALC_INTERCEPTS); + + return 0; +} + + +static void kvm_probe_feature_msr(u32 msr_index) +{ + u64 data; + + if (kvm_get_feature_msr(NULL, msr_index, &data, true)) + return; + + msr_based_features[num_msr_based_features++] =3D msr_index; +} + +static void kvm_probe_msr_to_save(u32 msr_index) +{ + u32 dummy[2]; + + if (rdmsr_safe(msr_index, &dummy[0], &dummy[1])) + return; + + /* + * Even MSRs that are valid in the host may not be exposed to guests in + * some cases. + */ + switch (msr_index) { + case MSR_IA32_BNDCFGS: + if (!kvm_mpx_supported()) + return; + break; + case MSR_TSC_AUX: + if (!kvm_cpu_cap_has(X86_FEATURE_RDTSCP) && + !kvm_cpu_cap_has(X86_FEATURE_RDPID)) + return; + break; + case MSR_IA32_UMWAIT_CONTROL: + if (!kvm_cpu_cap_has(X86_FEATURE_WAITPKG)) + return; + break; + case MSR_IA32_RTIT_CTL: + case MSR_IA32_RTIT_STATUS: + if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT)) + return; + break; + case MSR_IA32_RTIT_CR3_MATCH: + if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) || + !intel_pt_validate_hw_cap(PT_CAP_cr3_filtering)) + return; + break; + case MSR_IA32_RTIT_OUTPUT_BASE: + case MSR_IA32_RTIT_OUTPUT_MASK: + if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) || + (!intel_pt_validate_hw_cap(PT_CAP_topa_output) && + !intel_pt_validate_hw_cap(PT_CAP_single_range_output))) + return; + break; + case MSR_IA32_RTIT_ADDR0_A ... MSR_IA32_RTIT_ADDR3_B: + if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) || + (msr_index - MSR_IA32_RTIT_ADDR0_A >=3D + intel_pt_validate_hw_cap(PT_CAP_num_address_ranges) * 2)) + return; + break; + case MSR_ARCH_PERFMON_PERFCTR0 ... + MSR_ARCH_PERFMON_PERFCTR0 + KVM_MAX_NR_GP_COUNTERS - 1: + if (msr_index - MSR_ARCH_PERFMON_PERFCTR0 >=3D + kvm_pmu_cap.num_counters_gp) + return; + break; + case MSR_ARCH_PERFMON_EVENTSEL0 ... + MSR_ARCH_PERFMON_EVENTSEL0 + KVM_MAX_NR_GP_COUNTERS - 1: + if (msr_index - MSR_ARCH_PERFMON_EVENTSEL0 >=3D + kvm_pmu_cap.num_counters_gp) + return; + break; + case MSR_ARCH_PERFMON_FIXED_CTR0 ... + MSR_ARCH_PERFMON_FIXED_CTR0 + KVM_MAX_NR_FIXED_COUNTERS - 1: + if (msr_index - MSR_ARCH_PERFMON_FIXED_CTR0 >=3D + kvm_pmu_cap.num_counters_fixed) + return; + break; + case MSR_AMD64_PERF_CNTR_GLOBAL_CTL: + case MSR_AMD64_PERF_CNTR_GLOBAL_STATUS: + case MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_CLR: + case MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_SET: + if (!kvm_cpu_cap_has(X86_FEATURE_PERFMON_V2)) + return; + break; + case MSR_IA32_XFD: + case MSR_IA32_XFD_ERR: + if (!kvm_cpu_cap_has(X86_FEATURE_XFD)) + return; + break; + case MSR_IA32_TSX_CTRL: + if (!(kvm_get_arch_capabilities() & ARCH_CAP_TSX_CTRL_MSR)) + return; + break; + case MSR_IA32_XSS: + if (!kvm_caps.supported_xss) + return; + break; + case MSR_IA32_U_CET: + case MSR_IA32_S_CET: + if (!kvm_cpu_cap_has(X86_FEATURE_SHSTK) && + !kvm_cpu_cap_has(X86_FEATURE_IBT)) + return; + break; + case MSR_IA32_INT_SSP_TAB: + if (!kvm_cpu_cap_has(X86_FEATURE_LM)) + return; + fallthrough; + case MSR_IA32_PL0_SSP ... MSR_IA32_PL3_SSP: + if (!kvm_cpu_cap_has(X86_FEATURE_SHSTK)) + return; + break; + default: + break; + } + + msrs_to_save[num_msrs_to_save++] =3D msr_index; +} + +void kvm_init_msr_lists(void) +{ + unsigned i; + + BUILD_BUG_ON_MSG(KVM_MAX_NR_FIXED_COUNTERS !=3D 3, + "Please update the fixed PMCs in msrs_to_save_pmu[]"); + + num_msrs_to_save =3D 0; + num_emulated_msrs =3D 0; + num_msr_based_features =3D 0; + + for (i =3D 0; i < ARRAY_SIZE(msrs_to_save_base); i++) + kvm_probe_msr_to_save(msrs_to_save_base[i]); + + if (enable_pmu) { + for (i =3D 0; i < ARRAY_SIZE(msrs_to_save_pmu); i++) + kvm_probe_msr_to_save(msrs_to_save_pmu[i]); + } + + for (i =3D 0; i < ARRAY_SIZE(emulated_msrs_all); i++) { + if (!kvm_x86_call(has_emulated_msr)(NULL, + emulated_msrs_all[i])) + continue; + + emulated_msrs[num_emulated_msrs++] =3D emulated_msrs_all[i]; + } + + for (i =3D KVM_FIRST_EMULATED_VMX_MSR; i <=3D KVM_LAST_EMULATED_VMX_MSR; = i++) + kvm_probe_feature_msr(i); + + for (i =3D 0; i < ARRAY_SIZE(msr_based_features_all_except_vmx); i++) + kvm_probe_feature_msr(msr_based_features_all_except_vmx[i]); +} + +int kvm_spec_ctrl_test_value(u64 value) +{ + /* + * test that setting IA32_SPEC_CTRL to given value + * is allowed by the host processor + */ + + u64 saved_value; + unsigned long flags; + int ret =3D 0; + + local_irq_save(flags); + + if (rdmsrq_safe(MSR_IA32_SPEC_CTRL, &saved_value)) + ret =3D 1; + else if (wrmsrq_safe(MSR_IA32_SPEC_CTRL, value)) + ret =3D 1; + else + wrmsrq(MSR_IA32_SPEC_CTRL, saved_value); + + local_irq_restore(flags); + + return ret; +} +EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_spec_ctrl_test_value); diff --git a/arch/x86/kvm/msrs.h b/arch/x86/kvm/msrs.h new file mode 100644 index 000000000000..c34f0411ced6 --- /dev/null +++ b/arch/x86/kvm/msrs.h @@ -0,0 +1,128 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef ARCH_X86_KVM_MSR_H +#define ARCH_X86_KVM_MSR_H + +#include +#include + +#include "cpuid.h" +#include "regs.h" + +extern bool report_ignored_msrs; +extern bool ignore_msrs; + +static inline void kvm_pr_unimpl_wrmsr(struct kvm_vcpu *vcpu, u32 msr, u64= data) +{ + if (report_ignored_msrs) + vcpu_unimpl(vcpu, "Unhandled WRMSR(0x%x) =3D 0x%llx\n", msr, data); +} + +static inline void kvm_pr_unimpl_rdmsr(struct kvm_vcpu *vcpu, u32 msr) +{ + if (report_ignored_msrs) + vcpu_unimpl(vcpu, "Unhandled RDMSR(0x%x)\n", msr); +} + +/* + * The first...last VMX feature MSRs that are emulated by KVM. This may o= r may + * not cover all known VMX MSRs, as KVM doesn't emulate an MSR until there= 's an + * associated feature that KVM supports for nested virtualization. + */ +#define KVM_FIRST_EMULATED_VMX_MSR MSR_IA32_VMX_BASIC +#define KVM_LAST_EMULATED_VMX_MSR MSR_IA32_VMX_VMFUNC + +/* + * KVM's internal, non-ABI indices for synthetic MSRs. The values themselv= es + * are arbitrary and have no meaning, the only requirement is that they do= n't + * conflict with "real" MSRs that KVM supports. Use values at the upper end + * of KVM's reserved paravirtual MSR range to minimize churn, i.e. these v= alues + * will be usable until KVM exhausts its supply of paravirtual MSR indices. + */ +#define MSR_KVM_INTERNAL_GUEST_SSP 0x4b564dff + +#define MSR_IA32_CR_PAT_DEFAULT \ + PAT_VALUE(WB, WT, UC_MINUS, UC, WB, WT, UC_MINUS, UC) + +void kvm_init_msr_lists(void); +int kvm_get_msr_index_list(struct kvm_msr_list __user *user_msr_list); +int kvm_get_feature_msr_index_list(struct kvm_msr_list __user *user_msr_li= st); +int kvm_get_feature_msrs(struct kvm_msrs __user *user_msrs); + +int kvm_get_msrs(struct kvm_vcpu *vcpu, struct kvm_msrs __user *user_msrs); +int kvm_set_msrs(struct kvm_vcpu *vcpu, struct kvm_msrs __user *user_msrs); + +int kvm_get_set_one_reg(struct kvm_vcpu *vcpu, unsigned int ioctl, + void __user *argp); +int kvm_get_reg_list(struct kvm_vcpu *vcpu, + struct kvm_reg_list __user *user_list); + +void kvm_user_return_msr_cpu_online(void); +void drop_user_return_notifiers(void); +void kvm_destroy_user_return_msrs(void); + +fastpath_t handle_fastpath_wrmsr(struct kvm_vcpu *vcpu); +fastpath_t handle_fastpath_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int r= eg); + +int kvm_emulator_get_msr_with_filter(struct kvm_vcpu *vcpu, u32 msr_index, + u64 *pdata); +int kvm_emulator_set_msr_with_filter(struct kvm_vcpu *vcpu, u32 msr_index, + u64 data); +int kvm_emulator_get_msr(struct kvm_vcpu *vcpu, u32 msr_index, u64 *pdata); + +bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type); + +enum kvm_msr_access { + MSR_TYPE_R =3D BIT(0), + MSR_TYPE_W =3D BIT(1), + MSR_TYPE_RW =3D MSR_TYPE_R | MSR_TYPE_W, +}; + +/* + * Internal error codes that are used to indicate that MSR emulation encou= ntered + * an error that should result in #GP in the guest, unless userspace handl= es it. + * Note, '1', '0', and negative numbers are off limits, as they are used b= y KVM + * as part of KVM's lightly documented internal KVM_RUN return codes. + * + * UNSUPPORTED - The MSR isn't supported, either because it is completely + * unknown to KVM, or because the MSR should not exist according + * to the vCPU model. + * + * FILTERED - Access to the MSR is denied by a userspace MSR filter. + */ +#define KVM_MSR_RET_UNSUPPORTED 2 +#define KVM_MSR_RET_FILTERED 3 + +int kvm_vm_ioctl_set_msr_filter(struct kvm *kvm, struct kvm_msr_filter *fi= lter); +void kvm_free_msr_filter(struct kvm_x86_msr_filter *msr_filter); + +int kvm_mtrr_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data); +int kvm_mtrr_get_msr(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata); + +u64 kvm_get_arch_capabilities(void); +int kvm_spec_ctrl_test_value(u64 value); + +#define CET_US_RESERVED_BITS GENMASK(9, 6) +#define CET_US_SHSTK_MASK_BITS GENMASK(1, 0) +#define CET_US_IBT_MASK_BITS (GENMASK_ULL(5, 2) | GENMASK_ULL(63, 10)) +#define CET_US_LEGACY_BITMAP_BASE(data) ((data) >> 12) + +static inline bool kvm_is_valid_u_s_cet(struct kvm_vcpu *vcpu, u64 data) +{ + if (data & CET_US_RESERVED_BITS) + return false; + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) && + (data & CET_US_SHSTK_MASK_BITS)) + return false; + if (!guest_cpu_cap_has(vcpu, X86_FEATURE_IBT) && + (data & CET_US_IBT_MASK_BITS)) + return false; + if (!IS_ALIGNED(CET_US_LEGACY_BITMAP_BASE(data), 4)) + return false; + /* IBT can be suppressed iff the TRACKER isn't WAIT_ENDBR. */ + if ((data & CET_SUPPRESS) && (data & CET_WAIT_ENDBR)) + return false; + + return true; +} + +#endif \ No newline at end of file diff --git a/arch/x86/kvm/mtrr.c b/arch/x86/kvm/mtrr.c index 6f74e2b27c1e..4f3b7b0f6565 100644 --- a/arch/x86/kvm/mtrr.c +++ b/arch/x86/kvm/mtrr.c @@ -19,6 +19,7 @@ #include =20 #include "cpuid.h" +#include "msrs.h" #include "x86.h" =20 static u64 *find_mtrr(struct kvm_vcpu *vcpu, unsigned int msr) diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index bf15c122f837..c648fac802f6 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -80,7 +80,6 @@ #include #include #include -#include #include #include #include @@ -90,8 +89,6 @@ #define CREATE_TRACE_POINTS #include "trace.h" =20 -#define MAX_IO_MSRS 256 - /* * Note, kvm_caps fields should *never* have default values, all fields mu= st be * recomputed from scratch during vendor module load, e.g. to account for a @@ -108,17 +105,6 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_host); #define emul_to_vcpu(ctxt) \ ((struct kvm_vcpu *)(ctxt)->vcpu) =20 -/* EFER defaults: - * - enable syscall per default because its emulated by KVM - * - enable LME and LMA per default on 64 bit KVM - */ -#ifdef CONFIG_X86_64 -static -u64 __read_mostly efer_reserved_bits =3D ~((u64)(EFER_SCE | EFER_LME | EFE= R_LMA)); -#else -static u64 __read_mostly efer_reserved_bits =3D ~((u64)EFER_SCE); -#endif - #define KVM_EXIT_HYPERCALL_VALID_MASK (1 << KVM_HC_MAP_GPA_RANGE) =20 #define KVM_CAP_PMU_VALID_MASK KVM_PMU_CAP_DISABLE @@ -133,7 +119,6 @@ static void store_regs(struct kvm_vcpu *vcpu); static int sync_regs(struct kvm_vcpu *vcpu); =20 static DEFINE_MUTEX(vendor_module_lock); - struct kvm_x86_ops kvm_x86_ops __read_mostly; =20 #define KVM_X86_OP(func) \ @@ -146,13 +131,6 @@ EXPORT_STATIC_CALL_GPL(kvm_x86_get_cs_db_l_bits); EXPORT_STATIC_CALL_GPL(kvm_x86_cache_reg); EXPORT_STATIC_CALL_GPL(kvm_x86_get_cpl); =20 -static bool __read_mostly ignore_msrs =3D 0; -module_param(ignore_msrs, bool, 0644); - -bool __read_mostly report_ignored_msrs =3D true; -module_param(report_ignored_msrs, bool, 0644); -EXPORT_SYMBOL_FOR_KVM_INTERNAL(report_ignored_msrs); - unsigned int min_timer_period_us =3D 200; module_param(min_timer_period_us, uint, 0644); =20 @@ -179,27 +157,6 @@ module_param(pi_inject_timer, bint, 0644); static bool __read_mostly mitigate_smt_rsb; module_param(mitigate_smt_rsb, bool, 0444); =20 -/* - * Restoring the host value for MSRs that are only consumed when running in - * usermode, e.g. SYSCALL MSRs and TSC_AUX, can be deferred until the CPU - * returns to userspace, i.e. the kernel can run with the guest's value. - */ -#define KVM_MAX_NR_USER_RETURN_MSRS 16 - -struct kvm_user_return_msrs { - struct user_return_notifier urn; - bool registered; - struct kvm_user_return_msr_values { - u64 host; - u64 curr; - } values[KVM_MAX_NR_USER_RETURN_MSRS]; -}; - -u32 __read_mostly kvm_nr_uret_msrs; -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_nr_uret_msrs); -static u32 __read_mostly kvm_uret_msrs_list[KVM_MAX_NR_USER_RETURN_MSRS]; -static DEFINE_PER_CPU(struct kvm_user_return_msrs, user_return_msrs); - #define KVM_SUPPORTED_XCR0 (XFEATURE_MASK_FP | XFEATURE_MASK_SSE \ | XFEATURE_MASK_YMM | XFEATURE_MASK_BNDREGS \ | XFEATURE_MASK_BNDCSR | XFEATURE_MASK_AVX512 \ @@ -301,249 +258,6 @@ const struct kvm_stats_header kvm_vcpu_stats_header = =3D { =20 static struct kmem_cache *x86_emulator_cache; =20 -/* - * The three MSR lists(msrs_to_save, emulated_msrs, msr_based_features) tr= ack - * the set of MSRs that KVM exposes to userspace through KVM_GET_MSRS, - * KVM_SET_MSRS, and KVM_GET_MSR_INDEX_LIST. msrs_to_save holds MSRs that - * require host support, i.e. should be probed via RDMSR. emulated_msrs h= olds - * MSRs that KVM emulates without strictly requiring host support. - * msr_based_features holds MSRs that enumerate features, i.e. are effecti= vely - * CPUID leafs. Note, msr_based_features isn't mutually exclusive with - * msrs_to_save and emulated_msrs. - */ - -static const u32 msrs_to_save_base[] =3D { - MSR_IA32_SYSENTER_CS, MSR_IA32_SYSENTER_ESP, MSR_IA32_SYSENTER_EIP, - MSR_STAR, -#ifdef CONFIG_X86_64 - MSR_CSTAR, MSR_KERNEL_GS_BASE, MSR_SYSCALL_MASK, MSR_LSTAR, -#endif - MSR_IA32_TSC, MSR_IA32_CR_PAT, MSR_VM_HSAVE_PA, - MSR_IA32_FEAT_CTL, MSR_IA32_BNDCFGS, MSR_TSC_AUX, - MSR_IA32_SPEC_CTRL, MSR_IA32_TSX_CTRL, - MSR_IA32_RTIT_CTL, MSR_IA32_RTIT_STATUS, MSR_IA32_RTIT_CR3_MATCH, - MSR_IA32_RTIT_OUTPUT_BASE, MSR_IA32_RTIT_OUTPUT_MASK, - MSR_IA32_RTIT_ADDR0_A, MSR_IA32_RTIT_ADDR0_B, - MSR_IA32_RTIT_ADDR1_A, MSR_IA32_RTIT_ADDR1_B, - MSR_IA32_RTIT_ADDR2_A, MSR_IA32_RTIT_ADDR2_B, - MSR_IA32_RTIT_ADDR3_A, MSR_IA32_RTIT_ADDR3_B, - MSR_IA32_UMWAIT_CONTROL, - - MSR_IA32_XFD, MSR_IA32_XFD_ERR, MSR_IA32_XSS, - - MSR_IA32_U_CET, MSR_IA32_S_CET, - MSR_IA32_PL0_SSP, MSR_IA32_PL1_SSP, MSR_IA32_PL2_SSP, - MSR_IA32_PL3_SSP, MSR_IA32_INT_SSP_TAB, - MSR_IA32_DEBUGCTLMSR, - MSR_IA32_LASTBRANCHFROMIP, MSR_IA32_LASTBRANCHTOIP, - MSR_IA32_LASTINTFROMIP, MSR_IA32_LASTINTTOIP, -}; - -static const u32 msrs_to_save_pmu[] =3D { - MSR_ARCH_PERFMON_FIXED_CTR0, MSR_ARCH_PERFMON_FIXED_CTR1, - MSR_ARCH_PERFMON_FIXED_CTR0 + 2, - MSR_CORE_PERF_FIXED_CTR_CTRL, MSR_CORE_PERF_GLOBAL_STATUS, - MSR_CORE_PERF_GLOBAL_CTRL, - MSR_IA32_PEBS_ENABLE, MSR_IA32_DS_AREA, MSR_PEBS_DATA_CFG, - - /* This part of MSRs should match KVM_MAX_NR_INTEL_GP_COUNTERS. */ - MSR_ARCH_PERFMON_PERFCTR0, MSR_ARCH_PERFMON_PERFCTR1, - MSR_ARCH_PERFMON_PERFCTR0 + 2, MSR_ARCH_PERFMON_PERFCTR0 + 3, - MSR_ARCH_PERFMON_PERFCTR0 + 4, MSR_ARCH_PERFMON_PERFCTR0 + 5, - MSR_ARCH_PERFMON_PERFCTR0 + 6, MSR_ARCH_PERFMON_PERFCTR0 + 7, - MSR_ARCH_PERFMON_EVENTSEL0, MSR_ARCH_PERFMON_EVENTSEL1, - MSR_ARCH_PERFMON_EVENTSEL0 + 2, MSR_ARCH_PERFMON_EVENTSEL0 + 3, - MSR_ARCH_PERFMON_EVENTSEL0 + 4, MSR_ARCH_PERFMON_EVENTSEL0 + 5, - MSR_ARCH_PERFMON_EVENTSEL0 + 6, MSR_ARCH_PERFMON_EVENTSEL0 + 7, - - MSR_K7_EVNTSEL0, MSR_K7_EVNTSEL1, MSR_K7_EVNTSEL2, MSR_K7_EVNTSEL3, - MSR_K7_PERFCTR0, MSR_K7_PERFCTR1, MSR_K7_PERFCTR2, MSR_K7_PERFCTR3, - - /* This part of MSRs should match KVM_MAX_NR_AMD_GP_COUNTERS. */ - MSR_F15H_PERF_CTL0, MSR_F15H_PERF_CTL1, MSR_F15H_PERF_CTL2, - MSR_F15H_PERF_CTL3, MSR_F15H_PERF_CTL4, MSR_F15H_PERF_CTL5, - MSR_F15H_PERF_CTR0, MSR_F15H_PERF_CTR1, MSR_F15H_PERF_CTR2, - MSR_F15H_PERF_CTR3, MSR_F15H_PERF_CTR4, MSR_F15H_PERF_CTR5, - - MSR_AMD64_PERF_CNTR_GLOBAL_CTL, - MSR_AMD64_PERF_CNTR_GLOBAL_STATUS, - MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_CLR, - MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_SET, -}; - -static u32 msrs_to_save[ARRAY_SIZE(msrs_to_save_base) + - ARRAY_SIZE(msrs_to_save_pmu)]; -static unsigned num_msrs_to_save; - -static const u32 emulated_msrs_all[] =3D { - MSR_KVM_SYSTEM_TIME, MSR_KVM_WALL_CLOCK, - MSR_KVM_SYSTEM_TIME_NEW, MSR_KVM_WALL_CLOCK_NEW, - -#ifdef CONFIG_KVM_HYPERV - HV_X64_MSR_GUEST_OS_ID, HV_X64_MSR_HYPERCALL, - HV_X64_MSR_TIME_REF_COUNT, HV_X64_MSR_REFERENCE_TSC, - HV_X64_MSR_TSC_FREQUENCY, HV_X64_MSR_APIC_FREQUENCY, - HV_X64_MSR_CRASH_P0, HV_X64_MSR_CRASH_P1, HV_X64_MSR_CRASH_P2, - HV_X64_MSR_CRASH_P3, HV_X64_MSR_CRASH_P4, HV_X64_MSR_CRASH_CTL, - HV_X64_MSR_RESET, - HV_X64_MSR_VP_INDEX, - HV_X64_MSR_VP_RUNTIME, - HV_X64_MSR_SCONTROL, - HV_X64_MSR_STIMER0_CONFIG, - HV_X64_MSR_VP_ASSIST_PAGE, - HV_X64_MSR_REENLIGHTENMENT_CONTROL, HV_X64_MSR_TSC_EMULATION_CONTROL, - HV_X64_MSR_TSC_EMULATION_STATUS, HV_X64_MSR_TSC_INVARIANT_CONTROL, - HV_X64_MSR_SYNDBG_OPTIONS, - HV_X64_MSR_SYNDBG_CONTROL, HV_X64_MSR_SYNDBG_STATUS, - HV_X64_MSR_SYNDBG_SEND_BUFFER, HV_X64_MSR_SYNDBG_RECV_BUFFER, - HV_X64_MSR_SYNDBG_PENDING_BUFFER, -#endif - - MSR_KVM_ASYNC_PF_EN, MSR_KVM_STEAL_TIME, - MSR_KVM_PV_EOI_EN, MSR_KVM_ASYNC_PF_INT, MSR_KVM_ASYNC_PF_ACK, - - MSR_IA32_TSC_ADJUST, - MSR_IA32_TSC_DEADLINE, - MSR_IA32_ARCH_CAPABILITIES, - MSR_IA32_PERF_CAPABILITIES, - MSR_IA32_MISC_ENABLE, - MSR_IA32_MCG_STATUS, - MSR_IA32_MCG_CTL, - MSR_IA32_MCG_EXT_CTL, - MSR_IA32_SMBASE, - MSR_SMI_COUNT, - MSR_PLATFORM_INFO, - MSR_MISC_FEATURES_ENABLES, - MSR_AMD64_VIRT_SPEC_CTRL, - MSR_AMD64_TSC_RATIO, - MSR_IA32_POWER_CTL, - MSR_IA32_UCODE_REV, - - /* - * KVM always supports the "true" VMX control MSRs, even if the host - * does not. The VMX MSRs as a whole are considered "emulated" as KVM - * doesn't strictly require them to exist in the host (ignoring that - * KVM would refuse to load in the first place if the core set of MSRs - * aren't supported). - */ - MSR_IA32_VMX_BASIC, - MSR_IA32_VMX_TRUE_PINBASED_CTLS, - MSR_IA32_VMX_TRUE_PROCBASED_CTLS, - MSR_IA32_VMX_TRUE_EXIT_CTLS, - MSR_IA32_VMX_TRUE_ENTRY_CTLS, - MSR_IA32_VMX_MISC, - MSR_IA32_VMX_CR0_FIXED0, - MSR_IA32_VMX_CR4_FIXED0, - MSR_IA32_VMX_VMCS_ENUM, - MSR_IA32_VMX_PROCBASED_CTLS2, - MSR_IA32_VMX_EPT_VPID_CAP, - MSR_IA32_VMX_VMFUNC, - - MSR_K7_HWCR, - MSR_KVM_POLL_CONTROL, -}; - -static u32 emulated_msrs[ARRAY_SIZE(emulated_msrs_all)]; -static unsigned num_emulated_msrs; - -/* - * List of MSRs that control the existence of MSR-based features, i.e. MSRs - * that are effectively CPUID leafs. VMX MSRs are also included in the se= t of - * feature MSRs, but are handled separately to allow expedited lookups. - */ -static const u32 msr_based_features_all_except_vmx[] =3D { - MSR_AMD64_DE_CFG, - MSR_IA32_UCODE_REV, - MSR_IA32_ARCH_CAPABILITIES, - MSR_IA32_PERF_CAPABILITIES, - MSR_PLATFORM_INFO, -}; - -static u32 msr_based_features[ARRAY_SIZE(msr_based_features_all_except_vmx= ) + - (KVM_LAST_EMULATED_VMX_MSR - KVM_FIRST_EMULATED_VMX_MSR + 1)]; -static unsigned int num_msr_based_features; - -/* - * All feature MSRs except uCode revID, which tracks the currently loaded = uCode - * patch, are immutable once the vCPU model is defined. - */ -static bool kvm_is_immutable_feature_msr(u32 msr) -{ - int i; - - if (msr >=3D KVM_FIRST_EMULATED_VMX_MSR && msr <=3D KVM_LAST_EMULATED_VMX= _MSR) - return true; - - for (i =3D 0; i < ARRAY_SIZE(msr_based_features_all_except_vmx); i++) { - if (msr =3D=3D msr_based_features_all_except_vmx[i]) - return msr !=3D MSR_IA32_UCODE_REV; - } - - return false; -} - -static bool kvm_is_advertised_msr(u32 msr_index) -{ - unsigned int i; - - for (i =3D 0; i < num_msrs_to_save; i++) { - if (msrs_to_save[i] =3D=3D msr_index) - return true; - } - - for (i =3D 0; i < num_emulated_msrs; i++) { - if (emulated_msrs[i] =3D=3D msr_index) - return true; - } - - return false; -} - -typedef int (*msr_access_t)(struct kvm_vcpu *vcpu, u32 index, u64 *data, - bool host_initiated); - -static __always_inline int kvm_do_msr_access(struct kvm_vcpu *vcpu, u32 ms= r, - u64 *data, bool host_initiated, - enum kvm_msr_access rw, - msr_access_t msr_access_fn) -{ - const char *op =3D rw =3D=3D MSR_TYPE_W ? "wrmsr" : "rdmsr"; - int ret; - - BUILD_BUG_ON(rw !=3D MSR_TYPE_R && rw !=3D MSR_TYPE_W); - - /* - * Zero the data on read failures to avoid leaking stack data to the - * guest and/or userspace, e.g. if the failure is ignored below. - */ - ret =3D msr_access_fn(vcpu, msr, data, host_initiated); - if (ret && rw =3D=3D MSR_TYPE_R) - *data =3D 0; - - if (ret !=3D KVM_MSR_RET_UNSUPPORTED) - return ret; - - /* - * Userspace is allowed to read MSRs, and write '0' to MSRs, that KVM - * advertises to userspace, even if an MSR isn't fully supported. - * Simply check that @data is '0', which covers both the write '0' case - * and all reads (in which case @data is zeroed on failure; see above). - */ - if (host_initiated && !*data && kvm_is_advertised_msr(msr)) - return 0; - - if (!ignore_msrs) { - kvm_debug_ratelimited("unhandled %s: 0x%x data 0x%llx\n", - op, msr, *data); - return ret; - } - - if (report_ignored_msrs) - kvm_pr_unimpl("ignored %s: 0x%x data 0x%llx\n", op, msr, *data); - - return 0; -} - static struct kmem_cache *kvm_alloc_emulator_cache(void) { unsigned int useroffset =3D offsetof(struct x86_emulate_ctxt, src); @@ -557,128 +271,6 @@ static struct kmem_cache *kvm_alloc_emulator_cache(vo= id) =20 static int emulator_fix_hypercall(struct x86_emulate_ctxt *ctxt); =20 -static void kvm_destroy_user_return_msrs(void) -{ - int cpu; - - for_each_possible_cpu(cpu) - WARN_ON_ONCE(per_cpu(user_return_msrs, cpu).registered); - - kvm_nr_uret_msrs =3D 0; -} - -static void kvm_on_user_return(struct user_return_notifier *urn) -{ - unsigned slot; - struct kvm_user_return_msrs *msrs - =3D container_of(urn, struct kvm_user_return_msrs, urn); - struct kvm_user_return_msr_values *values; - - msrs->registered =3D false; - user_return_notifier_unregister(urn); - - for (slot =3D 0; slot < kvm_nr_uret_msrs; ++slot) { - values =3D &msrs->values[slot]; - if (values->host !=3D values->curr) { - wrmsrq(kvm_uret_msrs_list[slot], values->host); - values->curr =3D values->host; - } - } -} - -static int kvm_probe_user_return_msr(u32 msr) -{ - u64 val; - int ret; - - preempt_disable(); - ret =3D rdmsrq_safe(msr, &val); - if (ret) - goto out; - ret =3D wrmsrq_safe(msr, val); -out: - preempt_enable(); - return ret; -} - -int kvm_add_user_return_msr(u32 msr) -{ - BUG_ON(kvm_nr_uret_msrs >=3D KVM_MAX_NR_USER_RETURN_MSRS); - - if (kvm_probe_user_return_msr(msr)) - return -1; - - kvm_uret_msrs_list[kvm_nr_uret_msrs] =3D msr; - return kvm_nr_uret_msrs++; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_add_user_return_msr); - -int kvm_find_user_return_msr(u32 msr) -{ - int i; - - for (i =3D 0; i < kvm_nr_uret_msrs; ++i) { - if (kvm_uret_msrs_list[i] =3D=3D msr) - return i; - } - return -1; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_find_user_return_msr); - -static void kvm_user_return_msr_cpu_online(void) -{ - struct kvm_user_return_msrs *msrs =3D this_cpu_ptr(&user_return_msrs); - u64 value; - int i; - - for (i =3D 0; i < kvm_nr_uret_msrs; ++i) { - rdmsrq_safe(kvm_uret_msrs_list[i], &value); - msrs->values[i].host =3D value; - msrs->values[i].curr =3D value; - } -} - -static void kvm_user_return_register_notifier(struct kvm_user_return_msrs = *msrs) -{ - if (!msrs->registered) { - msrs->urn.on_user_return =3D kvm_on_user_return; - user_return_notifier_register(&msrs->urn); - msrs->registered =3D true; - } -} - -int kvm_set_user_return_msr(unsigned slot, u64 value, u64 mask) -{ - struct kvm_user_return_msrs *msrs =3D this_cpu_ptr(&user_return_msrs); - int err; - - value =3D (value & mask) | (msrs->values[slot].host & ~mask); - if (value =3D=3D msrs->values[slot].curr) - return 0; - err =3D wrmsrq_safe(kvm_uret_msrs_list[slot], value); - if (err) - return 1; - - msrs->values[slot].curr =3D value; - kvm_user_return_register_notifier(msrs); - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_user_return_msr); - -u64 kvm_get_user_return_msr(unsigned int slot) -{ - return this_cpu_ptr(&user_return_msrs)->values[slot].curr; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_user_return_msr); - -static void drop_user_return_notifiers(void) -{ - struct kvm_user_return_msrs *msrs =3D this_cpu_ptr(&user_return_msrs); - - if (msrs->registered) - kvm_on_user_return(&msrs->urn); -} - /* * Handle a fault on a hardware virtualization (VMX or SVM) instruction. * @@ -933,17 +525,6 @@ int kvm_complete_insn_gp(struct kvm_vcpu *vcpu, int er= r) } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_complete_insn_gp); =20 -static int complete_emulated_insn_gp(struct kvm_vcpu *vcpu, int err) -{ - if (err) { - kvm_inject_gp(vcpu, 0); - return 1; - } - - return kvm_emulate_instruction(vcpu, EMULTYPE_NO_DECODE | EMULTYPE_SKIP | - EMULTYPE_COMPLETE_USER_EXIT); -} - void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fa= ult, bool from_hardware) { @@ -1050,13 +631,6 @@ static void kvm_load_host_pkru(struct kvm_vcpu *vcpu) } } =20 -#ifdef CONFIG_X86_64 -static inline u64 kvm_guest_supported_xfd(struct kvm_vcpu *vcpu) -{ - return vcpu->arch.guest_supported_xcr0 & XFEATURE_MASK_USER_DYNAMIC; -} -#endif - int __kvm_set_xcr(struct kvm_vcpu *vcpu, u32 index, u64 xcr) { u64 xcr0 =3D xcr; @@ -1175,595 +749,6 @@ int kvm_emulate_rdpmc(struct kvm_vcpu *vcpu) } EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_rdpmc); =20 -/* - * Some IA32_ARCH_CAPABILITIES bits have dependencies on MSRs that KVM - * does not yet virtualize. These include: - * 10 - MISC_PACKAGE_CTRLS - * 11 - ENERGY_FILTERING_CTL - * 12 - DOITM - * 18 - FB_CLEAR_CTRL - * 21 - XAPIC_DISABLE_STATUS - * 23 - OVERCLOCKING_STATUS - */ - -#define KVM_SUPPORTED_ARCH_CAP \ - (ARCH_CAP_RDCL_NO | ARCH_CAP_IBRS_ALL | ARCH_CAP_RSBA | \ - ARCH_CAP_SKIP_VMENTRY_L1DFLUSH | ARCH_CAP_SSB_NO | ARCH_CAP_MDS_NO | \ - ARCH_CAP_PSCHANGE_MC_NO | ARCH_CAP_TSX_CTRL_MSR | ARCH_CAP_TAA_NO | \ - ARCH_CAP_SBDR_SSDP_NO | ARCH_CAP_FBSDP_NO | ARCH_CAP_PSDP_NO | \ - ARCH_CAP_FB_CLEAR | ARCH_CAP_RRSBA | ARCH_CAP_PBRSB_NO | ARCH_CAP_GDS_NO= | \ - ARCH_CAP_RFDS_NO | ARCH_CAP_RFDS_CLEAR | ARCH_CAP_BHI_NO | ARCH_CAP_ITS_= NO) - -static u64 kvm_get_arch_capabilities(void) -{ - u64 data =3D kvm_host.arch_capabilities & KVM_SUPPORTED_ARCH_CAP; - - /* - * If nx_huge_pages is enabled, KVM's shadow paging will ensure that - * the nested hypervisor runs with NX huge pages. If it is not, - * L1 is anyway vulnerable to ITLB_MULTIHIT exploits from other - * L1 guests, so it need not worry about its own (L2) guests. - */ - data |=3D ARCH_CAP_PSCHANGE_MC_NO; - - /* - * If we're doing cache flushes (either "always" or "cond") - * we will do one whenever the guest does a vmlaunch/vmresume. - * If an outer hypervisor is doing the cache flush for us - * (ARCH_CAP_SKIP_VMENTRY_L1DFLUSH), we can safely pass that - * capability to the guest too, and if EPT is disabled we're not - * vulnerable. Overall, only VMENTER_L1D_FLUSH_NEVER will - * require a nested hypervisor to do a flush of its own. - */ - if (l1tf_vmx_mitigation !=3D VMENTER_L1D_FLUSH_NEVER) - data |=3D ARCH_CAP_SKIP_VMENTRY_L1DFLUSH; - - if (!boot_cpu_has_bug(X86_BUG_CPU_MELTDOWN)) - data |=3D ARCH_CAP_RDCL_NO; - if (!boot_cpu_has_bug(X86_BUG_SPEC_STORE_BYPASS)) - data |=3D ARCH_CAP_SSB_NO; - if (!boot_cpu_has_bug(X86_BUG_MDS)) - data |=3D ARCH_CAP_MDS_NO; - if (!boot_cpu_has_bug(X86_BUG_RFDS)) - data |=3D ARCH_CAP_RFDS_NO; - if (!boot_cpu_has_bug(X86_BUG_ITS)) - data |=3D ARCH_CAP_ITS_NO; - - if (!boot_cpu_has(X86_FEATURE_RTM)) { - /* - * If RTM=3D0 because the kernel has disabled TSX, the host might - * have TAA_NO or TSX_CTRL. Clear TAA_NO (the guest sees RTM=3D0 - * and therefore knows that there cannot be TAA) but keep - * TSX_CTRL: some buggy userspaces leave it set on tsx=3Don hosts, - * and we want to allow migrating those guests to tsx=3Doff hosts. - */ - data &=3D ~ARCH_CAP_TAA_NO; - } else if (!boot_cpu_has_bug(X86_BUG_TAA)) { - data |=3D ARCH_CAP_TAA_NO; - } else { - /* - * Nothing to do here; we emulate TSX_CTRL if present on the - * host so the guest can choose between disabling TSX or - * using VERW to clear CPU buffers. - */ - } - - if (!boot_cpu_has_bug(X86_BUG_GDS) || gds_ucode_mitigated()) - data |=3D ARCH_CAP_GDS_NO; - - return data; -} - -static int kvm_get_feature_msr(struct kvm_vcpu *vcpu, u32 index, u64 *data, - bool host_initiated) -{ - WARN_ON_ONCE(!host_initiated); - - switch (index) { - case MSR_IA32_ARCH_CAPABILITIES: - *data =3D kvm_get_arch_capabilities(); - break; - case MSR_IA32_PERF_CAPABILITIES: - *data =3D kvm_caps.supported_perf_cap; - break; - case MSR_PLATFORM_INFO: - *data =3D MSR_PLATFORM_INFO_CPUID_FAULT; - break; - case MSR_IA32_UCODE_REV: - rdmsrq_safe(index, data); - break; - default: - return kvm_x86_call(get_feature_msr)(index, data); - } - return 0; -} - -static int do_get_feature_msr(struct kvm_vcpu *vcpu, unsigned index, u64 *= data) -{ - return kvm_do_msr_access(vcpu, index, data, true, MSR_TYPE_R, - kvm_get_feature_msr); -} - -static bool __kvm_valid_efer(struct kvm_vcpu *vcpu, u64 efer) -{ - if (efer & EFER_AUTOIBRS && !guest_cpu_cap_has(vcpu, X86_FEATURE_AUTOIBRS= )) - return false; - - if (efer & EFER_FFXSR && !guest_cpu_cap_has(vcpu, X86_FEATURE_FXSR_OPT)) - return false; - - if (efer & EFER_SVME && !guest_cpu_cap_has(vcpu, X86_FEATURE_SVM)) - return false; - - if (efer & (EFER_LME | EFER_LMA) && - !guest_cpu_cap_has(vcpu, X86_FEATURE_LM)) - return false; - - if (efer & EFER_NX && !guest_cpu_cap_has(vcpu, X86_FEATURE_NX)) - return false; - - return true; - -} -bool kvm_valid_efer(struct kvm_vcpu *vcpu, u64 efer) -{ - if (efer & efer_reserved_bits) - return false; - - return __kvm_valid_efer(vcpu, efer); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_valid_efer); - -static int set_efer(struct kvm_vcpu *vcpu, struct msr_data *msr_info) -{ - u64 old_efer =3D vcpu->arch.efer; - u64 efer =3D msr_info->data; - int r; - - if (efer & efer_reserved_bits) - return 1; - - if (!msr_info->host_initiated) { - if (!__kvm_valid_efer(vcpu, efer)) - return 1; - - if (is_paging(vcpu) && - (vcpu->arch.efer & EFER_LME) !=3D (efer & EFER_LME)) - return 1; - } - - efer &=3D ~EFER_LMA; - efer |=3D vcpu->arch.efer & EFER_LMA; - - r =3D kvm_x86_call(set_efer)(vcpu, efer); - if (r) { - WARN_ON(r > 0); - return r; - } - - if ((efer ^ old_efer) & KVM_MMU_EFER_ROLE_BITS) - kvm_mmu_reset_context(vcpu); - - if (!static_cpu_has(X86_FEATURE_XSAVES) && - (efer & EFER_SVME)) - kvm_hv_xsaves_xsavec_maybe_warn(vcpu); - - return 0; -} - -void kvm_enable_efer_bits(u64 mask) -{ - efer_reserved_bits &=3D ~mask; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_enable_efer_bits); - -bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type) -{ - struct kvm_x86_msr_filter *msr_filter; - struct msr_bitmap_range *ranges; - struct kvm *kvm =3D vcpu->kvm; - bool allowed; - int idx; - u32 i; - - /* x2APIC MSRs do not support filtering. */ - if (index >=3D 0x800 && index <=3D 0x8ff) - return true; - - idx =3D srcu_read_lock(&kvm->srcu); - - msr_filter =3D srcu_dereference(kvm->arch.msr_filter, &kvm->srcu); - if (!msr_filter) { - allowed =3D true; - goto out; - } - - allowed =3D msr_filter->default_allow; - ranges =3D msr_filter->ranges; - - for (i =3D 0; i < msr_filter->count; i++) { - u32 start =3D ranges[i].base; - u32 end =3D start + ranges[i].nmsrs; - u32 flags =3D ranges[i].flags; - unsigned long *bitmap =3D ranges[i].bitmap; - - if ((index >=3D start) && (index < end) && (flags & type)) { - allowed =3D test_bit(index - start, bitmap); - break; - } - } - -out: - srcu_read_unlock(&kvm->srcu, idx); - - return allowed; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_msr_allowed); - -/* - * Write @data into the MSR specified by @index. Select MSR specific fault - * checks are bypassed if @host_initiated is %true. - * Returns 0 on success, non-0 otherwise. - * Assumes vcpu_load() was already called. - */ -static int __kvm_set_msr(struct kvm_vcpu *vcpu, u32 index, u64 data, - bool host_initiated) -{ - struct msr_data msr; - - switch (index) { - case MSR_FS_BASE: - case MSR_GS_BASE: - case MSR_KERNEL_GS_BASE: - case MSR_CSTAR: - case MSR_LSTAR: - if (is_noncanonical_msr_address(data, vcpu)) - return 1; - break; - case MSR_IA32_SYSENTER_EIP: - case MSR_IA32_SYSENTER_ESP: - /* - * IA32_SYSENTER_ESP and IA32_SYSENTER_EIP cause #GP if - * non-canonical address is written on Intel but not on - * AMD (which ignores the top 32-bits, because it does - * not implement 64-bit SYSENTER). - * - * 64-bit code should hence be able to write a non-canonical - * value on AMD. Making the address canonical ensures that - * vmentry does not fail on Intel after writing a non-canonical - * value, and that something deterministic happens if the guest - * invokes 64-bit SYSENTER. - */ - data =3D __canonical_address(data, max_host_virt_addr_bits()); - break; - case MSR_TSC_AUX: - if (!kvm_is_supported_user_return_msr(MSR_TSC_AUX)) - return 1; - - if (!host_initiated && - !guest_cpu_cap_has(vcpu, X86_FEATURE_RDTSCP) && - !guest_cpu_cap_has(vcpu, X86_FEATURE_RDPID)) - return 1; - - /* - * Per Intel's SDM, bits 63:32 are reserved, but AMD's APM has - * incomplete and conflicting architectural behavior. Current - * AMD CPUs completely ignore bits 63:32, i.e. they aren't - * reserved and always read as zeros. Enforce Intel's reserved - * bits check if the guest CPU is Intel compatible, otherwise - * clear the bits. This ensures cross-vendor migration will - * provide consistent behavior for the guest. - */ - if (guest_cpuid_is_intel_compatible(vcpu) && (data >> 32) !=3D 0) - return 1; - - data =3D (u32)data; - break; - case MSR_IA32_U_CET: - case MSR_IA32_S_CET: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) && - !guest_cpu_cap_has(vcpu, X86_FEATURE_IBT)) - return KVM_MSR_RET_UNSUPPORTED; - if (!kvm_is_valid_u_s_cet(vcpu, data)) - return 1; - break; - case MSR_KVM_INTERNAL_GUEST_SSP: - if (!host_initiated) - return 1; - fallthrough; - /* - * Note that the MSR emulation here is flawed when a vCPU - * doesn't support the Intel 64 architecture. The expected - * architectural behavior in this case is that the upper 32 - * bits do not exist and should always read '0'. However, - * because the actual hardware on which the virtual CPU is - * running does support Intel 64, XRSTORS/XSAVES in the - * guest could observe behavior that violates the - * architecture. Intercepting XRSTORS/XSAVES for this - * special case isn't deemed worthwhile. - */ - case MSR_IA32_PL0_SSP ... MSR_IA32_INT_SSP_TAB: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK)) - return KVM_MSR_RET_UNSUPPORTED; - /* - * MSR_IA32_INT_SSP_TAB is not present on processors that do - * not support Intel 64 architecture. - */ - if (index =3D=3D MSR_IA32_INT_SSP_TAB && !guest_cpu_cap_has(vcpu, X86_FE= ATURE_LM)) - return KVM_MSR_RET_UNSUPPORTED; - if (is_noncanonical_msr_address(data, vcpu)) - return 1; - /* All SSP MSRs except MSR_IA32_INT_SSP_TAB must be 4-byte aligned */ - if (index !=3D MSR_IA32_INT_SSP_TAB && !IS_ALIGNED(data, 4)) - return 1; - break; - } - - msr.data =3D data; - msr.index =3D index; - msr.host_initiated =3D host_initiated; - - return kvm_x86_call(set_msr)(vcpu, &msr); -} - -static int _kvm_set_msr(struct kvm_vcpu *vcpu, u32 index, u64 *data, - bool host_initiated) -{ - return __kvm_set_msr(vcpu, index, *data, host_initiated); -} - -static int kvm_set_msr_ignored_check(struct kvm_vcpu *vcpu, - u32 index, u64 data, bool host_initiated) -{ - return kvm_do_msr_access(vcpu, index, &data, host_initiated, MSR_TYPE_W, - _kvm_set_msr); -} - -/* - * Read the MSR specified by @index into @data. Select MSR specific fault - * checks are bypassed if @host_initiated is %true. - * Returns 0 on success, non-0 otherwise. - * Assumes vcpu_load() was already called. - */ -static int __kvm_get_msr(struct kvm_vcpu *vcpu, u32 index, u64 *data, - bool host_initiated) -{ - struct msr_data msr; - int ret; - - switch (index) { - case MSR_TSC_AUX: - if (!kvm_is_supported_user_return_msr(MSR_TSC_AUX)) - return 1; - - if (!host_initiated && - !guest_cpu_cap_has(vcpu, X86_FEATURE_RDTSCP) && - !guest_cpu_cap_has(vcpu, X86_FEATURE_RDPID)) - return 1; - break; - case MSR_IA32_U_CET: - case MSR_IA32_S_CET: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) && - !guest_cpu_cap_has(vcpu, X86_FEATURE_IBT)) - return KVM_MSR_RET_UNSUPPORTED; - break; - case MSR_KVM_INTERNAL_GUEST_SSP: - if (!host_initiated) - return 1; - fallthrough; - case MSR_IA32_PL0_SSP ... MSR_IA32_INT_SSP_TAB: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK)) - return KVM_MSR_RET_UNSUPPORTED; - break; - } - - msr.index =3D index; - msr.host_initiated =3D host_initiated; - - ret =3D kvm_x86_call(get_msr)(vcpu, &msr); - if (!ret) - *data =3D msr.data; - return ret; -} - -int kvm_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data) -{ - return __kvm_set_msr(vcpu, index, data, true); -} - -int kvm_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data) -{ - return __kvm_get_msr(vcpu, index, data, true); -} - -static int kvm_get_msr_ignored_check(struct kvm_vcpu *vcpu, - u32 index, u64 *data, bool host_initiated) -{ - return kvm_do_msr_access(vcpu, index, data, host_initiated, MSR_TYPE_R, - __kvm_get_msr); -} - -int __kvm_emulate_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data) -{ - return kvm_get_msr_ignored_check(vcpu, index, data, false); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(__kvm_emulate_msr_read); - -int __kvm_emulate_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data) -{ - return kvm_set_msr_ignored_check(vcpu, index, data, false); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(__kvm_emulate_msr_write); - -int kvm_emulate_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data) -{ - if (!kvm_msr_allowed(vcpu, index, KVM_MSR_FILTER_READ)) - return KVM_MSR_RET_FILTERED; - - return __kvm_emulate_msr_read(vcpu, index, data); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_msr_read); - -int kvm_emulate_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data) -{ - if (!kvm_msr_allowed(vcpu, index, KVM_MSR_FILTER_WRITE)) - return KVM_MSR_RET_FILTERED; - - return __kvm_emulate_msr_write(vcpu, index, data); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_msr_write); - - -static void complete_userspace_rdmsr(struct kvm_vcpu *vcpu) -{ - if (!vcpu->run->msr.error) { - kvm_eax_write(vcpu, vcpu->run->msr.data); - kvm_edx_write(vcpu, vcpu->run->msr.data >> 32); - } -} - -static int complete_emulated_msr_access(struct kvm_vcpu *vcpu) -{ - return complete_emulated_insn_gp(vcpu, vcpu->run->msr.error); -} - -static int complete_emulated_rdmsr(struct kvm_vcpu *vcpu) -{ - complete_userspace_rdmsr(vcpu); - return complete_emulated_msr_access(vcpu); -} - -static int complete_fast_msr_access(struct kvm_vcpu *vcpu) -{ - return kvm_x86_call(complete_emulated_msr)(vcpu, vcpu->run->msr.error); -} - -static int complete_fast_rdmsr(struct kvm_vcpu *vcpu) -{ - complete_userspace_rdmsr(vcpu); - return complete_fast_msr_access(vcpu); -} - -static int complete_fast_rdmsr_imm(struct kvm_vcpu *vcpu) -{ - if (!vcpu->run->msr.error) - kvm_register_write(vcpu, vcpu->arch.cui_rdmsr_imm_reg, - vcpu->run->msr.data); - - return complete_fast_msr_access(vcpu); -} - -static u64 kvm_msr_reason(int r) -{ - switch (r) { - case KVM_MSR_RET_UNSUPPORTED: - return KVM_MSR_EXIT_REASON_UNKNOWN; - case KVM_MSR_RET_FILTERED: - return KVM_MSR_EXIT_REASON_FILTER; - default: - return KVM_MSR_EXIT_REASON_INVAL; - } -} - -static int kvm_msr_user_space(struct kvm_vcpu *vcpu, u32 index, - u32 exit_reason, u64 data, - int (*completion)(struct kvm_vcpu *vcpu), - int r) -{ - u64 msr_reason =3D kvm_msr_reason(r); - - /* Check if the user wanted to know about this MSR fault */ - if (!(vcpu->kvm->arch.user_space_msr_mask & msr_reason)) - return 0; - - vcpu->run->exit_reason =3D exit_reason; - vcpu->run->msr.error =3D 0; - memset(vcpu->run->msr.pad, 0, sizeof(vcpu->run->msr.pad)); - vcpu->run->msr.reason =3D msr_reason; - vcpu->run->msr.index =3D index; - vcpu->run->msr.data =3D data; - vcpu->arch.complete_userspace_io =3D completion; - - return 1; -} - -static int __kvm_emulate_rdmsr(struct kvm_vcpu *vcpu, u32 msr, int reg, - int (*complete_rdmsr)(struct kvm_vcpu *)) -{ - u64 data; - int r; - - r =3D kvm_emulate_msr_read(vcpu, msr, &data); - - if (!r) { - trace_kvm_msr_read(msr, data); - - if (reg < 0) { - kvm_eax_write(vcpu, data); - kvm_edx_write(vcpu, data >> 32); - } else { - kvm_register_write(vcpu, reg, data); - } - } else { - /* MSR read failed? See if we should ask user space */ - if (kvm_msr_user_space(vcpu, msr, KVM_EXIT_X86_RDMSR, 0, - complete_rdmsr, r)) - return 0; - trace_kvm_msr_read_ex(msr); - } - - return kvm_x86_call(complete_emulated_msr)(vcpu, r); -} - -int kvm_emulate_rdmsr(struct kvm_vcpu *vcpu) -{ - return __kvm_emulate_rdmsr(vcpu, kvm_ecx_read(vcpu), -1, - complete_fast_rdmsr); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_rdmsr); - -int kvm_emulate_rdmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg) -{ - vcpu->arch.cui_rdmsr_imm_reg =3D reg; - - return __kvm_emulate_rdmsr(vcpu, msr, reg, complete_fast_rdmsr_imm); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_rdmsr_imm); - -static int __kvm_emulate_wrmsr(struct kvm_vcpu *vcpu, u32 msr, u64 data) -{ - int r; - - r =3D kvm_emulate_msr_write(vcpu, msr, data); - if (!r) { - trace_kvm_msr_write(msr, data); - } else { - /* MSR write failed? See if we should ask user space */ - if (kvm_msr_user_space(vcpu, msr, KVM_EXIT_X86_WRMSR, data, - complete_fast_msr_access, r)) - return 0; - /* Signal all other negative errors to userspace */ - if (r < 0) - return r; - trace_kvm_msr_write_ex(msr, data); - } - - return kvm_x86_call(complete_emulated_msr)(vcpu, r); -} - -int kvm_emulate_wrmsr(struct kvm_vcpu *vcpu) -{ - return __kvm_emulate_wrmsr(vcpu, kvm_ecx_read(vcpu), - kvm_read_edx_eax(vcpu)); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_wrmsr); - -int kvm_emulate_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg) -{ - return __kvm_emulate_wrmsr(vcpu, msr, kvm_register_read(vcpu, reg)); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_emulate_wrmsr_imm); - int kvm_emulate_as_nop(struct kvm_vcpu *vcpu) { return kvm_skip_emulated_instruction(vcpu); @@ -1835,72 +820,6 @@ static inline bool kvm_vcpu_exit_request(struct kvm_v= cpu *vcpu) kvm_request_pending(vcpu) || xfer_to_guest_mode_work_pending(); } =20 -static fastpath_t __handle_fastpath_wrmsr(struct kvm_vcpu *vcpu, u32 msr, = u64 data) -{ - if (!kvm_pmu_is_fastpath_emulation_allowed(vcpu)) - return EXIT_FASTPATH_NONE; - - switch (msr) { - case APIC_BASE_MSR + (APIC_ICR >> 4): - if (!lapic_in_kernel(vcpu) || !apic_x2apic_mode(vcpu->arch.apic) || - kvm_x2apic_icr_write_fast(vcpu->arch.apic, data)) - return EXIT_FASTPATH_NONE; - break; - case MSR_IA32_TSC_DEADLINE: - kvm_set_lapic_tscdeadline_msr(vcpu, data); - break; - default: - return EXIT_FASTPATH_NONE; - } - - trace_kvm_msr_write(msr, data); - - if (!kvm_skip_emulated_instruction(vcpu)) - return EXIT_FASTPATH_EXIT_USERSPACE; - - return EXIT_FASTPATH_REENTER_GUEST; -} - -fastpath_t handle_fastpath_wrmsr(struct kvm_vcpu *vcpu) -{ - return __handle_fastpath_wrmsr(vcpu, kvm_ecx_read(vcpu), - kvm_read_edx_eax(vcpu)); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(handle_fastpath_wrmsr); - -fastpath_t handle_fastpath_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int r= eg) -{ - return __handle_fastpath_wrmsr(vcpu, msr, kvm_register_read(vcpu, reg)); -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(handle_fastpath_wrmsr_imm); - -/* - * Adapt set_msr() to msr_io()'s calling convention - */ -static int do_get_msr(struct kvm_vcpu *vcpu, unsigned index, u64 *data) -{ - return kvm_get_msr_ignored_check(vcpu, index, data, true); -} - -static int do_set_msr(struct kvm_vcpu *vcpu, unsigned index, u64 *data) -{ - u64 val; - - /* - * Reject writes to immutable feature MSRs if the vCPU model is frozen, - * as KVM doesn't support modifying the guest vCPU model on the fly, - * e.g. changing the VMX capabilities MSRs while L2 is active is - * nonsensical. Allow writes of the same value, e.g. so that userspace - * can blindly stuff all MSRs when emulating RESET. - */ - if (!kvm_can_set_cpuid_and_feature_msrs(vcpu) && - kvm_is_immutable_feature_msr(index) && - (do_get_msr(vcpu, index, &val) || *data !=3D val)) - return -EINVAL; - - return kvm_set_msr_ignored_check(vcpu, index, *data, true); -} - #ifdef CONFIG_X86_64 struct pvclock_clock { int vclock_mode; @@ -1967,72 +886,6 @@ static s64 get_kvmclock_base_ns(void) } #endif =20 -static void kvm_write_wall_clock(struct kvm *kvm, gpa_t wall_clock, int se= c_hi_ofs) -{ - int version; - int r; - struct pvclock_wall_clock wc; - u32 wc_sec_hi; - u64 wall_nsec; - - if (!wall_clock) - return; - - r =3D kvm_read_guest(kvm, wall_clock, &version, sizeof(version)); - if (r) - return; - - if (version & 1) - ++version; /* first time write, random junk */ - - ++version; - - if (kvm_write_guest(kvm, wall_clock, &version, sizeof(version))) - return; - - wall_nsec =3D kvm_get_wall_clock_epoch(kvm); - - wc.nsec =3D do_div(wall_nsec, NSEC_PER_SEC); - wc.sec =3D (u32)wall_nsec; /* overflow in 2106 guest time */ - wc.version =3D version; - - kvm_write_guest(kvm, wall_clock, &wc, sizeof(wc)); - - if (sec_hi_ofs) { - wc_sec_hi =3D wall_nsec >> 32; - kvm_write_guest(kvm, wall_clock + sec_hi_ofs, - &wc_sec_hi, sizeof(wc_sec_hi)); - } - - version++; - kvm_write_guest(kvm, wall_clock, &version, sizeof(version)); -} - -static void kvm_write_system_time(struct kvm_vcpu *vcpu, gpa_t system_time, - bool old_msr, bool host_initiated) -{ - struct kvm_arch *ka =3D &vcpu->kvm->arch; - - if (vcpu->vcpu_id =3D=3D 0 && !host_initiated) { - if (ka->boot_vcpu_runs_old_kvmclock !=3D old_msr) - kvm_make_request(KVM_REQ_MASTERCLOCK_UPDATE, vcpu); - - ka->boot_vcpu_runs_old_kvmclock =3D old_msr; - } - - vcpu->arch.time =3D system_time; - kvm_make_request(KVM_REQ_GLOBAL_CLOCK_UPDATE, vcpu); - - /* we verify if the enable bit is set... */ - if (system_time & 1) - kvm_gpc_activate(&vcpu->arch.pv_time, system_time & ~1ULL, - sizeof(struct pvclock_vcpu_time_info)); - else - kvm_gpc_deactivate(&vcpu->arch.pv_time); - - return; -} - static uint32_t div_frac(uint32_t dividend, uint32_t divisor) { do_shl32_div32(dividend, divisor); @@ -3077,151 +1930,6 @@ static void kvm_gen_kvmclock_update(struct kvm_vcpu= *v) } } =20 -/* These helpers are safe iff @msr is known to be an MCx bank MSR. */ -static bool is_mci_control_msr(u32 msr) -{ - return (msr & 3) =3D=3D 0; -} -static bool is_mci_status_msr(u32 msr) -{ - return (msr & 3) =3D=3D 1; -} - -/* - * On AMD, HWCR[McStatusWrEn] controls whether setting MCi_STATUS results = in #GP. - */ -static bool can_set_mci_status(struct kvm_vcpu *vcpu) -{ - /* McStatusWrEn enabled? */ - if (guest_cpuid_is_amd_compatible(vcpu)) - return !!(vcpu->arch.msr_hwcr & BIT_ULL(18)); - - return false; -} - -static int set_msr_mce(struct kvm_vcpu *vcpu, struct msr_data *msr_info) -{ - u64 mcg_cap =3D vcpu->arch.mcg_cap; - unsigned bank_num =3D mcg_cap & 0xff; - u32 msr =3D msr_info->index; - u64 data =3D msr_info->data; - u32 offset, last_msr; - - switch (msr) { - case MSR_IA32_MCG_STATUS: - vcpu->arch.mcg_status =3D data; - break; - case MSR_IA32_MCG_CTL: - if (!(mcg_cap & MCG_CTL_P) && - (data || !msr_info->host_initiated)) - return 1; - if (data !=3D 0 && data !=3D ~(u64)0) - return 1; - vcpu->arch.mcg_ctl =3D data; - break; - case MSR_IA32_MC0_CTL2 ... MSR_IA32_MCx_CTL2(KVM_MAX_MCE_BANKS) - 1: - last_msr =3D MSR_IA32_MCx_CTL2(bank_num) - 1; - if (msr > last_msr) - return 1; - - if (!(mcg_cap & MCG_CMCI_P) && (data || !msr_info->host_initiated)) - return 1; - /* An attempt to write a 1 to a reserved bit raises #GP */ - if (data & ~(MCI_CTL2_CMCI_EN | MCI_CTL2_CMCI_THRESHOLD_MASK)) - return 1; - offset =3D array_index_nospec(msr - MSR_IA32_MC0_CTL2, - last_msr + 1 - MSR_IA32_MC0_CTL2); - vcpu->arch.mci_ctl2_banks[offset] =3D data; - break; - case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: - last_msr =3D MSR_IA32_MCx_CTL(bank_num) - 1; - if (msr > last_msr) - return 1; - - /* - * Only 0 or all 1s can be written to IA32_MCi_CTL, all other - * values are architecturally undefined. But, some Linux - * kernels clear bit 10 in bank 4 to workaround a BIOS/GART TLB - * issue on AMD K8s, allow bit 10 to be clear when setting all - * other bits in order to avoid an uncaught #GP in the guest. - * - * UNIXWARE clears bit 0 of MC1_CTL to ignore correctable, - * single-bit ECC data errors. - */ - if (is_mci_control_msr(msr) && - data !=3D 0 && (data | (1 << 10) | 1) !=3D ~(u64)0) - return 1; - - /* - * All CPUs allow writing 0 to MCi_STATUS MSRs to clear the MSR. - * AMD-based CPUs allow non-zero values, but if and only if - * HWCR[McStatusWrEn] is set. - */ - if (!msr_info->host_initiated && is_mci_status_msr(msr) && - data !=3D 0 && !can_set_mci_status(vcpu)) - return 1; - - offset =3D array_index_nospec(msr - MSR_IA32_MC0_CTL, - last_msr + 1 - MSR_IA32_MC0_CTL); - vcpu->arch.mce_banks[offset] =3D data; - break; - default: - return 1; - } - return 0; -} - -static int kvm_pv_enable_async_pf(struct kvm_vcpu *vcpu, u64 data) -{ - gpa_t gpa =3D data & ~0x3f; - - /* Bits 4:5 are reserved, Should be zero */ - if (data & 0x30) - return 1; - - if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_VMEXIT) && - (data & KVM_ASYNC_PF_DELIVERY_AS_PF_VMEXIT)) - return 1; - - if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT) && - (data & KVM_ASYNC_PF_DELIVERY_AS_INT)) - return 1; - - if (!lapic_in_kernel(vcpu)) - return data ? 1 : 0; - - if (__kvm_pv_async_pf_enabled(data) && - kvm_gfn_to_hva_cache_init(vcpu->kvm, &vcpu->arch.apf.data, gpa, - sizeof(u64))) - return 1; - - vcpu->arch.apf.msr_en_val =3D data; - - if (__kvm_pv_async_pf_enabled(data)) { - kvm_async_pf_wakeup_all(vcpu); - } else { - kvm_clear_async_pf_completion_queue(vcpu); - kvm_async_pf_hash_reset(vcpu); - } - return 0; -} - -static int kvm_pv_enable_async_pf_int(struct kvm_vcpu *vcpu, u64 data) -{ - /* Bits 8-63 are reserved */ - if (data >> 8) - return 1; - - if (!lapic_in_kernel(vcpu)) - return 1; - - vcpu->arch.apf.msr_int_val =3D data; - - vcpu->arch.apf.vec =3D data & KVM_ASYNC_PF_VEC_MASK; - - return 0; -} - static void kvmclock_reset(struct kvm_vcpu *vcpu) { kvm_gpc_deactivate(&vcpu->arch.pv_time); @@ -3382,899 +2090,6 @@ static void record_steal_time(struct kvm_vcpu *vcpu) mark_page_dirty_in_slot(vcpu->kvm, ghc->memslot, gpa_to_gfn(ghc->gpa)); } =20 -/* - * Returns true if the MSR in question is managed via XSTATE, i.e. is cont= ext - * switched with the rest of guest FPU state. - * - * Note, S_CET is _not_ saved/restored via XSAVES/XRSTORS. - */ -static bool is_xstate_managed_msr(struct kvm_vcpu *vcpu, u32 msr) -{ - if (!vcpu) - return false; - - switch (msr) { - case MSR_IA32_U_CET: - return guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) || - guest_cpu_cap_has(vcpu, X86_FEATURE_IBT); - case MSR_IA32_PL0_SSP ... MSR_IA32_PL3_SSP: - return guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK); - default: - return false; - } -} - -/* - * Lock (and if necessary, re-load) the guest FPU, i.e. XSTATE, and access= an - * MSR that is managed via XSTATE. Note, the caller is responsible for do= ing - * the initial FPU load, this helper only ensures that guest state is resi= dent - * in hardware (the kernel can load its FPU state in IRQ context). - * - * Note, loading guest values for U_CET and PL[0-3]_SSP while executing in= the - * kernel is safe, as U_CET is specific to userspace, and PL[0-3]_SSP are = only - * consumed when transitioning to lower privilege levels, i.e. are effecti= vely - * only consumed by userspace as well. - */ -static __always_inline void kvm_access_xstate_msr(struct kvm_vcpu *vcpu, - struct msr_data *msr_info, - int access) -{ - BUILD_BUG_ON(access !=3D MSR_TYPE_R && access !=3D MSR_TYPE_W); - - KVM_BUG_ON(!is_xstate_managed_msr(vcpu, msr_info->index), vcpu->kvm); - KVM_BUG_ON(!vcpu->arch.guest_fpu.fpstate->in_use, vcpu->kvm); - - kvm_fpu_get(); - if (access =3D=3D MSR_TYPE_R) - rdmsrq(msr_info->index, msr_info->data); - else - wrmsrq(msr_info->index, msr_info->data); - kvm_fpu_put(); -} - -static void kvm_set_xstate_msr(struct kvm_vcpu *vcpu, struct msr_data *msr= _info) -{ - kvm_access_xstate_msr(vcpu, msr_info, MSR_TYPE_W); -} - -static void kvm_get_xstate_msr(struct kvm_vcpu *vcpu, struct msr_data *msr= _info) -{ - kvm_access_xstate_msr(vcpu, msr_info, MSR_TYPE_R); -} - -int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) -{ - u32 msr =3D msr_info->index; - u64 data =3D msr_info->data; - - /* - * Do not allow host-initiated writes to trigger the Xen hypercall - * page setup; it could incur locking paths which are not expected - * if userspace sets the MSR in an unusual location. - */ - if (kvm_xen_is_hypercall_page_msr(vcpu->kvm, msr) && - !msr_info->host_initiated) - return kvm_xen_write_hypercall_page(vcpu, data); - - switch (msr) { - case MSR_AMD64_NB_CFG: - case MSR_IA32_UCODE_WRITE: - case MSR_VM_HSAVE_PA: - case MSR_AMD64_PATCH_LOADER: - case MSR_AMD64_BU_CFG2: - case MSR_AMD64_DC_CFG: - case MSR_AMD64_TW_CFG: - case MSR_F15H_EX_CFG: - break; - - case MSR_IA32_UCODE_REV: - if (msr_info->host_initiated) - vcpu->arch.microcode_version =3D data; - break; - case MSR_IA32_ARCH_CAPABILITIES: - if (!msr_info->host_initiated || - !guest_cpu_cap_has(vcpu, X86_FEATURE_ARCH_CAPABILITIES)) - return KVM_MSR_RET_UNSUPPORTED; - vcpu->arch.arch_capabilities =3D data; - break; - case MSR_IA32_PERF_CAPABILITIES: - if (!msr_info->host_initiated || - !guest_cpu_cap_has(vcpu, X86_FEATURE_PDCM)) - return KVM_MSR_RET_UNSUPPORTED; - - if (data & ~kvm_caps.supported_perf_cap) - return 1; - - /* - * Note, this is not just a performance optimization! KVM - * disallows changing feature MSRs after the vCPU has run; PMU - * refresh will bug the VM if called after the vCPU has run. - */ - if (vcpu->arch.perf_capabilities =3D=3D data) - break; - - vcpu->arch.perf_capabilities =3D data; - kvm_pmu_refresh(vcpu); - kvm_make_request(KVM_REQ_RECALC_INTERCEPTS, vcpu); - break; - case MSR_IA32_PRED_CMD: { - u64 reserved_bits =3D ~(PRED_CMD_IBPB | PRED_CMD_SBPB); - - if (!msr_info->host_initiated) { - if ((!guest_has_pred_cmd_msr(vcpu))) - return 1; - - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SPEC_CTRL) && - !guest_cpu_cap_has(vcpu, X86_FEATURE_AMD_IBPB)) - reserved_bits |=3D PRED_CMD_IBPB; - - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SBPB)) - reserved_bits |=3D PRED_CMD_SBPB; - } - - if (!boot_cpu_has(X86_FEATURE_IBPB)) - reserved_bits |=3D PRED_CMD_IBPB; - - if (!boot_cpu_has(X86_FEATURE_SBPB)) - reserved_bits |=3D PRED_CMD_SBPB; - - if (data & reserved_bits) - return 1; - - if (!data) - break; - - wrmsrq(MSR_IA32_PRED_CMD, data); - break; - } - case MSR_IA32_FLUSH_CMD: - if (!msr_info->host_initiated && - !guest_cpu_cap_has(vcpu, X86_FEATURE_FLUSH_L1D)) - return 1; - - if (!boot_cpu_has(X86_FEATURE_FLUSH_L1D) || (data & ~L1D_FLUSH)) - return 1; - if (!data) - break; - - wrmsrq(MSR_IA32_FLUSH_CMD, L1D_FLUSH); - break; - case MSR_EFER: - return set_efer(vcpu, msr_info); - case MSR_K7_HWCR: { - /* - * Allow McStatusWrEn and TscFreqSel. (Linux guests from v3.2 - * through at least v6.6 whine if TscFreqSel is clear, - * depending on F/M/S. - */ - u64 valid =3D BIT_ULL(18) | BIT_ULL(24); - - data &=3D ~(u64)0x40; /* ignore flush filter disable */ - data &=3D ~(u64)0x100; /* ignore ignne emulation enable */ - data &=3D ~(u64)0x8; /* ignore TLB cache disable */ - - if (guest_cpu_cap_has(vcpu, X86_FEATURE_GP_ON_USER_CPUID)) - valid |=3D MSR_K7_HWCR_CPUID_USER_DIS; - - if (data & ~valid) { - kvm_pr_unimpl_wrmsr(vcpu, msr, data); - return 1; - } - vcpu->arch.msr_hwcr =3D data; - break; - } - case MSR_FAM10H_MMIO_CONF_BASE: - if (data !=3D 0) { - kvm_pr_unimpl_wrmsr(vcpu, msr, data); - return 1; - } - break; - case MSR_IA32_CR_PAT: - if (!kvm_pat_valid(data)) - return 1; - - vcpu->arch.pat =3D data; - break; - case MTRRphysBase_MSR(0) ... MSR_MTRRfix4K_F8000: - case MSR_MTRRdefType: - return kvm_mtrr_set_msr(vcpu, msr, data); - case MSR_IA32_APICBASE: - return kvm_apic_set_base(vcpu, data, msr_info->host_initiated); - case APIC_BASE_MSR ... APIC_BASE_MSR + 0xff: - return kvm_x2apic_msr_write(vcpu, msr, data); - case MSR_IA32_TSC_DEADLINE: - kvm_set_lapic_tscdeadline_msr(vcpu, data); - break; - case MSR_IA32_TSC_ADJUST: - if (guest_cpu_cap_has(vcpu, X86_FEATURE_TSC_ADJUST)) { - if (!msr_info->host_initiated) { - s64 adj =3D data - vcpu->arch.ia32_tsc_adjust_msr; - adjust_tsc_offset_guest(vcpu, adj); - /* Before back to guest, tsc_timestamp must be adjusted - * as well, otherwise guest's percpu pvclock time could jump. - */ - kvm_make_request(KVM_REQ_CLOCK_UPDATE, vcpu); - } - vcpu->arch.ia32_tsc_adjust_msr =3D data; - } - break; - case MSR_IA32_MISC_ENABLE: { - u64 old_val =3D vcpu->arch.ia32_misc_enable_msr; - - if (!msr_info->host_initiated) { - /* RO bits */ - if ((old_val ^ data) & MSR_IA32_MISC_ENABLE_PMU_RO_MASK) - return 1; - - /* R bits, i.e. writes are ignored, but don't fault. */ - data =3D data & ~MSR_IA32_MISC_ENABLE_EMON; - data |=3D old_val & MSR_IA32_MISC_ENABLE_EMON; - } - - if (!kvm_check_has_quirk(vcpu->kvm, KVM_X86_QUIRK_MISC_ENABLE_NO_MWAIT) = && - ((old_val ^ data) & MSR_IA32_MISC_ENABLE_MWAIT)) { - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_XMM3)) - return 1; - vcpu->arch.ia32_misc_enable_msr =3D data; - vcpu->arch.cpuid_dynamic_bits_dirty =3D true; - } else { - vcpu->arch.ia32_misc_enable_msr =3D data; - } - break; - } - case MSR_IA32_SMBASE: - if (!IS_ENABLED(CONFIG_KVM_SMM) || !msr_info->host_initiated) - return 1; - vcpu->arch.smbase =3D data; - break; - case MSR_IA32_POWER_CTL: - vcpu->arch.msr_ia32_power_ctl =3D data; - break; - case MSR_IA32_TSC: - if (msr_info->host_initiated) { - kvm_synchronize_tsc(vcpu, &data); - } else if (!vcpu->arch.guest_tsc_protected) { - u64 adj =3D kvm_compute_l1_tsc_offset(vcpu, data) - vcpu->arch.l1_tsc_o= ffset; - adjust_tsc_offset_guest(vcpu, adj); - vcpu->arch.ia32_tsc_adjust_msr +=3D adj; - } - break; - case MSR_IA32_XSS: - if (!guest_cpuid_has(vcpu, X86_FEATURE_XSAVES)) - return KVM_MSR_RET_UNSUPPORTED; - - if (data & ~vcpu->arch.guest_supported_xss) - return 1; - if (vcpu->arch.ia32_xss =3D=3D data) - break; - vcpu->arch.ia32_xss =3D data; - vcpu->arch.cpuid_dynamic_bits_dirty =3D true; - break; - case MSR_SMI_COUNT: - if (!msr_info->host_initiated) - return 1; - vcpu->arch.smi_count =3D data; - break; - case MSR_KVM_WALL_CLOCK_NEW: - if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE2)) - return KVM_MSR_RET_UNSUPPORTED; - - vcpu->kvm->arch.wall_clock =3D data; - kvm_write_wall_clock(vcpu->kvm, data, 0); - break; - case MSR_KVM_WALL_CLOCK: - if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE)) - return KVM_MSR_RET_UNSUPPORTED; - - vcpu->kvm->arch.wall_clock =3D data; - kvm_write_wall_clock(vcpu->kvm, data, 0); - break; - case MSR_KVM_SYSTEM_TIME_NEW: - if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE2)) - return KVM_MSR_RET_UNSUPPORTED; - - kvm_write_system_time(vcpu, data, false, msr_info->host_initiated); - break; - case MSR_KVM_SYSTEM_TIME: - if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE)) - return KVM_MSR_RET_UNSUPPORTED; - - kvm_write_system_time(vcpu, data, true, msr_info->host_initiated); - break; - case MSR_KVM_ASYNC_PF_EN: - if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF)) - return KVM_MSR_RET_UNSUPPORTED; - - if (kvm_pv_enable_async_pf(vcpu, data)) - return 1; - break; - case MSR_KVM_ASYNC_PF_INT: - if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT)) - return KVM_MSR_RET_UNSUPPORTED; - - if (kvm_pv_enable_async_pf_int(vcpu, data)) - return 1; - break; - case MSR_KVM_ASYNC_PF_ACK: - if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT)) - return KVM_MSR_RET_UNSUPPORTED; - if (data & 0x1) { - /* - * Pairs with the smp_mb__after_atomic() in - * kvm_arch_async_page_present_queued(). - */ - smp_store_mb(vcpu->arch.apf.pageready_pending, false); - - kvm_check_async_pf_completion(vcpu); - } - break; - case MSR_KVM_STEAL_TIME: - if (!guest_pv_has(vcpu, KVM_FEATURE_STEAL_TIME)) - return KVM_MSR_RET_UNSUPPORTED; - - if (unlikely(!sched_info_on())) - return 1; - - if (data & KVM_STEAL_RESERVED_MASK) - return 1; - - vcpu->arch.st.msr_val =3D data; - - if (!(data & KVM_MSR_ENABLED)) - break; - - kvm_make_request(KVM_REQ_STEAL_UPDATE, vcpu); - - break; - case MSR_KVM_PV_EOI_EN: - if (!guest_pv_has(vcpu, KVM_FEATURE_PV_EOI)) - return KVM_MSR_RET_UNSUPPORTED; - - if (kvm_lapic_set_pv_eoi(vcpu, data, sizeof(u8))) - return 1; - break; - - case MSR_KVM_POLL_CONTROL: - if (!guest_pv_has(vcpu, KVM_FEATURE_POLL_CONTROL)) - return KVM_MSR_RET_UNSUPPORTED; - - /* only enable bit supported */ - if (data & (-1ULL << 1)) - return 1; - - vcpu->arch.msr_kvm_poll_control =3D data; - break; - - case MSR_IA32_MCG_CTL: - case MSR_IA32_MCG_STATUS: - case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: - case MSR_IA32_MC0_CTL2 ... MSR_IA32_MCx_CTL2(KVM_MAX_MCE_BANKS) - 1: - return set_msr_mce(vcpu, msr_info); - - case MSR_K7_PERFCTR0 ... MSR_K7_PERFCTR3: - case MSR_P6_PERFCTR0 ... MSR_P6_PERFCTR1: - case MSR_K7_EVNTSEL0 ... MSR_K7_EVNTSEL3: - case MSR_P6_EVNTSEL0 ... MSR_P6_EVNTSEL1: - if (kvm_pmu_is_valid_msr(vcpu, msr)) - return kvm_pmu_set_msr(vcpu, msr_info); - - if (data) - kvm_pr_unimpl_wrmsr(vcpu, msr, data); - break; - case MSR_K7_CLK_CTL: - /* - * Ignore all writes to this no longer documented MSR. - * Writes are only relevant for old K7 processors, - * all pre-dating SVM, but a recommended workaround from - * AMD for these chips. It is possible to specify the - * affected processor models on the command line, hence - * the need to ignore the workaround. - */ - break; -#ifdef CONFIG_KVM_HYPERV - case HV_X64_MSR_GUEST_OS_ID ... HV_X64_MSR_SINT15: - case HV_X64_MSR_SYNDBG_CONTROL ... HV_X64_MSR_SYNDBG_PENDING_BUFFER: - case HV_X64_MSR_SYNDBG_OPTIONS: - case HV_X64_MSR_CRASH_P0 ... HV_X64_MSR_CRASH_P4: - case HV_X64_MSR_CRASH_CTL: - case HV_X64_MSR_STIMER0_CONFIG ... HV_X64_MSR_STIMER3_COUNT: - case HV_X64_MSR_REENLIGHTENMENT_CONTROL: - case HV_X64_MSR_TSC_EMULATION_CONTROL: - case HV_X64_MSR_TSC_EMULATION_STATUS: - case HV_X64_MSR_TSC_INVARIANT_CONTROL: - return kvm_hv_set_msr_common(vcpu, msr, data, - msr_info->host_initiated); -#endif - case MSR_IA32_BBL_CR_CTL3: - /* Drop writes to this legacy MSR -- see rdmsr - * counterpart for further detail. - */ - kvm_pr_unimpl_wrmsr(vcpu, msr, data); - break; - case MSR_AMD64_OSVW_ID_LENGTH: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_OSVW)) - return 1; - vcpu->arch.osvw.length =3D data; - break; - case MSR_AMD64_OSVW_STATUS: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_OSVW)) - return 1; - vcpu->arch.osvw.status =3D data; - break; - case MSR_PLATFORM_INFO: - if (!msr_info->host_initiated) - return 1; - vcpu->arch.msr_platform_info =3D data; - break; - case MSR_MISC_FEATURES_ENABLES: - if (data & ~MSR_MISC_FEATURES_ENABLES_CPUID_FAULT || - (data & MSR_MISC_FEATURES_ENABLES_CPUID_FAULT && - !(vcpu->arch.msr_platform_info & MSR_PLATFORM_INFO_CPUID_FAULT))) - return 1; - vcpu->arch.msr_misc_features_enables =3D data; - break; -#ifdef CONFIG_X86_64 - case MSR_IA32_XFD: - if (!msr_info->host_initiated && - !guest_cpu_cap_has(vcpu, X86_FEATURE_XFD)) - return 1; - - if (data & ~kvm_guest_supported_xfd(vcpu)) - return 1; - - fpu_update_guest_xfd(&vcpu->arch.guest_fpu, data); - break; - case MSR_IA32_XFD_ERR: - if (!msr_info->host_initiated && - !guest_cpu_cap_has(vcpu, X86_FEATURE_XFD)) - return 1; - - if (data & ~kvm_guest_supported_xfd(vcpu)) - return 1; - - vcpu->arch.guest_fpu.xfd_err =3D data; - break; -#endif - case MSR_IA32_U_CET: - case MSR_IA32_PL0_SSP ... MSR_IA32_PL3_SSP: - kvm_set_xstate_msr(vcpu, msr_info); - break; - default: - if (kvm_pmu_is_valid_msr(vcpu, msr)) - return kvm_pmu_set_msr(vcpu, msr_info); - - return KVM_MSR_RET_UNSUPPORTED; - } - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_set_msr_common); - -static int get_msr_mce(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata, bool ho= st) -{ - u64 data; - u64 mcg_cap =3D vcpu->arch.mcg_cap; - unsigned bank_num =3D mcg_cap & 0xff; - u32 offset, last_msr; - - switch (msr) { - case MSR_IA32_P5_MC_ADDR: - case MSR_IA32_P5_MC_TYPE: - data =3D 0; - break; - case MSR_IA32_MCG_CAP: - data =3D vcpu->arch.mcg_cap; - break; - case MSR_IA32_MCG_CTL: - if (!(mcg_cap & MCG_CTL_P) && !host) - return 1; - data =3D vcpu->arch.mcg_ctl; - break; - case MSR_IA32_MCG_STATUS: - data =3D vcpu->arch.mcg_status; - break; - case MSR_IA32_MC0_CTL2 ... MSR_IA32_MCx_CTL2(KVM_MAX_MCE_BANKS) - 1: - last_msr =3D MSR_IA32_MCx_CTL2(bank_num) - 1; - if (msr > last_msr) - return 1; - - if (!(mcg_cap & MCG_CMCI_P) && !host) - return 1; - offset =3D array_index_nospec(msr - MSR_IA32_MC0_CTL2, - last_msr + 1 - MSR_IA32_MC0_CTL2); - data =3D vcpu->arch.mci_ctl2_banks[offset]; - break; - case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: - last_msr =3D MSR_IA32_MCx_CTL(bank_num) - 1; - if (msr > last_msr) - return 1; - - offset =3D array_index_nospec(msr - MSR_IA32_MC0_CTL, - last_msr + 1 - MSR_IA32_MC0_CTL); - data =3D vcpu->arch.mce_banks[offset]; - break; - default: - return 1; - } - *pdata =3D data; - return 0; -} - -int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info) -{ - switch (msr_info->index) { - case MSR_IA32_PLATFORM_ID: - case MSR_IA32_EBL_CR_POWERON: - case MSR_IA32_LASTBRANCHFROMIP: - case MSR_IA32_LASTBRANCHTOIP: - case MSR_IA32_LASTINTFROMIP: - case MSR_IA32_LASTINTTOIP: - case MSR_AMD64_SYSCFG: - case MSR_K8_TSEG_ADDR: - case MSR_K8_TSEG_MASK: - case MSR_VM_HSAVE_PA: - case MSR_K8_INT_PENDING_MSG: - case MSR_AMD64_NB_CFG: - case MSR_FAM10H_MMIO_CONF_BASE: - case MSR_AMD64_BU_CFG2: - case MSR_IA32_PERF_CTL: - case MSR_AMD64_DC_CFG: - case MSR_AMD64_TW_CFG: - case MSR_F15H_EX_CFG: - /* - * Intel Sandy Bridge CPUs must support the RAPL (running average power - * limit) MSRs. Just return 0, as we do not want to expose the host - * data here. Do not conditionalize this on CPUID, as KVM does not do - * so for existing CPU-specific MSRs. - */ - case MSR_RAPL_POWER_UNIT: - case MSR_PP0_ENERGY_STATUS: /* Power plane 0 (core) */ - case MSR_PP1_ENERGY_STATUS: /* Power plane 1 (graphics uncore) */ - case MSR_PKG_ENERGY_STATUS: /* Total package */ - case MSR_DRAM_ENERGY_STATUS: /* DRAM controller */ - msr_info->data =3D 0; - break; - case MSR_K7_EVNTSEL0 ... MSR_K7_EVNTSEL3: - case MSR_K7_PERFCTR0 ... MSR_K7_PERFCTR3: - case MSR_P6_PERFCTR0 ... MSR_P6_PERFCTR1: - case MSR_P6_EVNTSEL0 ... MSR_P6_EVNTSEL1: - if (kvm_pmu_is_valid_msr(vcpu, msr_info->index)) - return kvm_pmu_get_msr(vcpu, msr_info); - msr_info->data =3D 0; - break; - case MSR_IA32_UCODE_REV: - msr_info->data =3D vcpu->arch.microcode_version; - break; - case MSR_IA32_ARCH_CAPABILITIES: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_ARCH_CAPABILITIES)) - return KVM_MSR_RET_UNSUPPORTED; - msr_info->data =3D vcpu->arch.arch_capabilities; - break; - case MSR_IA32_PERF_CAPABILITIES: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_PDCM)) - return KVM_MSR_RET_UNSUPPORTED; - msr_info->data =3D vcpu->arch.perf_capabilities; - break; - case MSR_IA32_POWER_CTL: - msr_info->data =3D vcpu->arch.msr_ia32_power_ctl; - break; - case MSR_IA32_TSC: { - /* - * Intel SDM states that MSR_IA32_TSC read adds the TSC offset - * even when not intercepted. AMD manual doesn't explicitly - * state this but appears to behave the same. - * - * On userspace reads and writes, however, we unconditionally - * return L1's TSC value to ensure backwards-compatible - * behavior for migration. - */ - u64 offset, ratio; - - if (msr_info->host_initiated) { - offset =3D vcpu->arch.l1_tsc_offset; - ratio =3D vcpu->arch.l1_tsc_scaling_ratio; - } else { - offset =3D vcpu->arch.tsc_offset; - ratio =3D vcpu->arch.tsc_scaling_ratio; - } - - msr_info->data =3D kvm_scale_tsc(rdtsc(), ratio) + offset; - break; - } - case MSR_IA32_CR_PAT: - msr_info->data =3D vcpu->arch.pat; - break; - case MSR_MTRRcap: - case MTRRphysBase_MSR(0) ... MSR_MTRRfix4K_F8000: - case MSR_MTRRdefType: - return kvm_mtrr_get_msr(vcpu, msr_info->index, &msr_info->data); - case 0xcd: /* fsb frequency */ - msr_info->data =3D 3; - break; - /* - * MSR_EBC_FREQUENCY_ID - * Conservative value valid for even the basic CPU models. - * Models 0,1: 000 in bits 23:21 indicating a bus speed of - * 100MHz, model 2 000 in bits 18:16 indicating 100MHz, - * and 266MHz for model 3, or 4. Set Core Clock - * Frequency to System Bus Frequency Ratio to 1 (bits - * 31:24) even though these are only valid for CPU - * models > 2, however guests may end up dividing or - * multiplying by zero otherwise. - */ - case MSR_EBC_FREQUENCY_ID: - msr_info->data =3D 1 << 24; - break; - case MSR_IA32_APICBASE: - msr_info->data =3D vcpu->arch.apic_base; - break; - case APIC_BASE_MSR ... APIC_BASE_MSR + 0xff: - return kvm_x2apic_msr_read(vcpu, msr_info->index, &msr_info->data); - case MSR_IA32_TSC_DEADLINE: - msr_info->data =3D kvm_get_lapic_tscdeadline_msr(vcpu); - break; - case MSR_IA32_TSC_ADJUST: - msr_info->data =3D (u64)vcpu->arch.ia32_tsc_adjust_msr; - break; - case MSR_IA32_MISC_ENABLE: - msr_info->data =3D vcpu->arch.ia32_misc_enable_msr; - break; - case MSR_IA32_SMBASE: - if (!IS_ENABLED(CONFIG_KVM_SMM) || !msr_info->host_initiated) - return 1; - msr_info->data =3D vcpu->arch.smbase; - break; - case MSR_SMI_COUNT: - msr_info->data =3D vcpu->arch.smi_count; - break; - case MSR_IA32_PERF_STATUS: - /* TSC increment by tick */ - msr_info->data =3D 1000ULL; - /* CPU multiplier */ - msr_info->data |=3D (((uint64_t)4ULL) << 40); - break; - case MSR_EFER: - msr_info->data =3D vcpu->arch.efer; - break; - case MSR_KVM_WALL_CLOCK: - if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data =3D vcpu->kvm->arch.wall_clock; - break; - case MSR_KVM_WALL_CLOCK_NEW: - if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE2)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data =3D vcpu->kvm->arch.wall_clock; - break; - case MSR_KVM_SYSTEM_TIME: - if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data =3D vcpu->arch.time; - break; - case MSR_KVM_SYSTEM_TIME_NEW: - if (!guest_pv_has(vcpu, KVM_FEATURE_CLOCKSOURCE2)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data =3D vcpu->arch.time; - break; - case MSR_KVM_ASYNC_PF_EN: - if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data =3D vcpu->arch.apf.msr_en_val; - break; - case MSR_KVM_ASYNC_PF_INT: - if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data =3D vcpu->arch.apf.msr_int_val; - break; - case MSR_KVM_ASYNC_PF_ACK: - if (!guest_pv_has(vcpu, KVM_FEATURE_ASYNC_PF_INT)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data =3D 0; - break; - case MSR_KVM_STEAL_TIME: - if (!guest_pv_has(vcpu, KVM_FEATURE_STEAL_TIME)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data =3D vcpu->arch.st.msr_val; - break; - case MSR_KVM_PV_EOI_EN: - if (!guest_pv_has(vcpu, KVM_FEATURE_PV_EOI)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data =3D vcpu->arch.pv_eoi.msr_val; - break; - case MSR_KVM_POLL_CONTROL: - if (!guest_pv_has(vcpu, KVM_FEATURE_POLL_CONTROL)) - return KVM_MSR_RET_UNSUPPORTED; - - msr_info->data =3D vcpu->arch.msr_kvm_poll_control; - break; - case MSR_IA32_P5_MC_ADDR: - case MSR_IA32_P5_MC_TYPE: - case MSR_IA32_MCG_CAP: - case MSR_IA32_MCG_CTL: - case MSR_IA32_MCG_STATUS: - case MSR_IA32_MC0_CTL ... MSR_IA32_MCx_CTL(KVM_MAX_MCE_BANKS) - 1: - case MSR_IA32_MC0_CTL2 ... MSR_IA32_MCx_CTL2(KVM_MAX_MCE_BANKS) - 1: - return get_msr_mce(vcpu, msr_info->index, &msr_info->data, - msr_info->host_initiated); - case MSR_IA32_XSS: - if (!msr_info->host_initiated && - !guest_cpuid_has(vcpu, X86_FEATURE_XSAVES)) - return 1; - msr_info->data =3D vcpu->arch.ia32_xss; - break; - case MSR_K7_CLK_CTL: - /* - * Provide expected ramp-up count for K7. All other - * are set to zero, indicating minimum divisors for - * every field. - * - * This prevents guest kernels on AMD host with CPU - * type 6, model 8 and higher from exploding due to - * the rdmsr failing. - */ - msr_info->data =3D 0x20000000; - break; -#ifdef CONFIG_KVM_HYPERV - case HV_X64_MSR_GUEST_OS_ID ... HV_X64_MSR_SINT15: - case HV_X64_MSR_SYNDBG_CONTROL ... HV_X64_MSR_SYNDBG_PENDING_BUFFER: - case HV_X64_MSR_SYNDBG_OPTIONS: - case HV_X64_MSR_CRASH_P0 ... HV_X64_MSR_CRASH_P4: - case HV_X64_MSR_CRASH_CTL: - case HV_X64_MSR_STIMER0_CONFIG ... HV_X64_MSR_STIMER3_COUNT: - case HV_X64_MSR_REENLIGHTENMENT_CONTROL: - case HV_X64_MSR_TSC_EMULATION_CONTROL: - case HV_X64_MSR_TSC_EMULATION_STATUS: - case HV_X64_MSR_TSC_INVARIANT_CONTROL: - return kvm_hv_get_msr_common(vcpu, - msr_info->index, &msr_info->data, - msr_info->host_initiated); -#endif - case MSR_IA32_BBL_CR_CTL3: - /* This legacy MSR exists but isn't fully documented in current - * silicon. It is however accessed by winxp in very narrow - * scenarios where it sets bit #19, itself documented as - * a "reserved" bit. Best effort attempt to source coherent - * read data here should the balance of the register be - * interpreted by the guest: - * - * L2 cache control register 3: 64GB range, 256KB size, - * enabled, latency 0x1, configured - */ - msr_info->data =3D 0xbe702111; - break; - case MSR_AMD64_OSVW_ID_LENGTH: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_OSVW)) - return 1; - msr_info->data =3D vcpu->arch.osvw.length; - break; - case MSR_AMD64_OSVW_STATUS: - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_OSVW)) - return 1; - msr_info->data =3D vcpu->arch.osvw.status; - break; - case MSR_PLATFORM_INFO: - if (!msr_info->host_initiated && - !vcpu->kvm->arch.guest_can_read_msr_platform_info) - return 1; - msr_info->data =3D vcpu->arch.msr_platform_info; - break; - case MSR_MISC_FEATURES_ENABLES: - msr_info->data =3D vcpu->arch.msr_misc_features_enables; - break; - case MSR_K7_HWCR: - msr_info->data =3D vcpu->arch.msr_hwcr; - break; -#ifdef CONFIG_X86_64 - case MSR_IA32_XFD: - if (!msr_info->host_initiated && - !guest_cpu_cap_has(vcpu, X86_FEATURE_XFD)) - return 1; - - msr_info->data =3D vcpu->arch.guest_fpu.fpstate->xfd; - break; - case MSR_IA32_XFD_ERR: - if (!msr_info->host_initiated && - !guest_cpu_cap_has(vcpu, X86_FEATURE_XFD)) - return 1; - - msr_info->data =3D vcpu->arch.guest_fpu.xfd_err; - break; -#endif - case MSR_IA32_U_CET: - case MSR_IA32_PL0_SSP ... MSR_IA32_PL3_SSP: - kvm_get_xstate_msr(vcpu, msr_info); - break; - default: - if (kvm_pmu_is_valid_msr(vcpu, msr_info->index)) - return kvm_pmu_get_msr(vcpu, msr_info); - - return KVM_MSR_RET_UNSUPPORTED; - } - return 0; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_get_msr_common); - -/* - * Read or write a bunch of msrs. All parameters are kernel addresses. - * - * @return number of msrs set successfully. - */ -static int __msr_io(struct kvm_vcpu *vcpu, struct kvm_msrs *msrs, - struct kvm_msr_entry *entries, - int (*do_msr)(struct kvm_vcpu *vcpu, - unsigned index, u64 *data)) -{ - bool fpu_loaded =3D false; - int i; - - for (i =3D 0; i < msrs->nmsrs; ++i) { - /* - * If userspace is accessing one or more XSTATE-managed MSRs, - * temporarily load the guest's FPU state so that the guest's - * MSR value(s) is resident in hardware and thus can be accessed - * via RDMSR/WRMSR. - */ - if (!fpu_loaded && is_xstate_managed_msr(vcpu, entries[i].index)) { - kvm_load_guest_fpu(vcpu); - fpu_loaded =3D true; - } - if (do_msr(vcpu, entries[i].index, &entries[i].data)) - break; - } - if (fpu_loaded) - kvm_put_guest_fpu(vcpu); - - return i; -} - -/* - * Read or write a bunch of msrs. Parameters are user addresses. - * - * @return number of msrs set successfully. - */ -static int msr_io(struct kvm_vcpu *vcpu, struct kvm_msrs __user *user_msrs, - int (*do_msr)(struct kvm_vcpu *vcpu, - unsigned index, u64 *data), - int writeback) -{ - struct kvm_msrs msrs; - struct kvm_msr_entry *entries; - unsigned size; - int r; - - r =3D -EFAULT; - if (copy_from_user(&msrs, user_msrs, sizeof(msrs))) - goto out; - - r =3D -E2BIG; - if (msrs.nmsrs >=3D MAX_IO_MSRS) - goto out; - - size =3D sizeof(struct kvm_msr_entry) * msrs.nmsrs; - entries =3D memdup_user(user_msrs->entries, size); - if (IS_ERR(entries)) { - r =3D PTR_ERR(entries); - goto out; - } - - r =3D __msr_io(vcpu, &msrs, entries, do_msr); - - if (writeback && copy_to_user(user_msrs->entries, entries, size)) - r =3D -EFAULT; - - kfree(entries); -out: - return r; -} - static inline bool kvm_can_mwait_in_guest(void) { return boot_cpu_has(X86_FEATURE_MWAIT) && @@ -4586,61 +2401,6 @@ static int kvm_x86_dev_has_attr(struct kvm_device_at= tr *attr) return __kvm_x86_dev_get_attr(attr, &val); } =20 -static int kvm_get_msr_index_list(struct kvm_msr_list __user *user_msr_lis= t) -{ - struct kvm_msr_list msr_list; - unsigned int n; - - if (copy_from_user(&msr_list, user_msr_list, sizeof(msr_list))) - return -EFAULT; - - n =3D msr_list.nmsrs; - msr_list.nmsrs =3D num_msrs_to_save + num_emulated_msrs; - if (copy_to_user(user_msr_list, &msr_list, sizeof(msr_list))) - return -EFAULT; - - if (n < msr_list.nmsrs) - return -E2BIG; - - if (copy_to_user(user_msr_list->indices, &msrs_to_save, - num_msrs_to_save * sizeof(u32))) - return -EFAULT; - - if (copy_to_user(user_msr_list->indices + num_msrs_to_save, - &emulated_msrs, num_emulated_msrs * sizeof(u32))) - return -EFAULT; - - return 0; -} - -static int kvm_get_feature_msr_index_list(struct kvm_msr_list __user *user= _msr_list) -{ - struct kvm_msr_list msr_list; - unsigned int n; - - if (copy_from_user(&msr_list, user_msr_list, sizeof(msr_list))) - return -EFAULT; - - n =3D msr_list.nmsrs; - msr_list.nmsrs =3D num_msr_based_features; - if (copy_to_user(user_msr_list, &msr_list, sizeof(msr_list))) - return -EFAULT; - - if (n < msr_list.nmsrs) - return -E2BIG; - - if (copy_to_user(user_msr_list->indices, &msr_based_features, - num_msr_based_features * sizeof(u32))) - return -EFAULT; - - return 0; -} - -static int kvm_get_feature_msrs(struct kvm_msrs __user *user_msrs) -{ - return msr_io(NULL, user_msrs, do_get_feature_msr, 1); -} - long kvm_arch_dev_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { @@ -5588,148 +3348,6 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcp= u *vcpu, } } =20 -struct kvm_x86_reg_id { - __u32 index; - __u8 type; - __u8 rsvd1; - __u8 rsvd2:4; - __u8 size:4; - __u8 x86; -}; - -static int kvm_translate_kvm_reg(struct kvm_vcpu *vcpu, - struct kvm_x86_reg_id *reg) -{ - switch (reg->index) { - case KVM_REG_GUEST_SSP: - /* - * FIXME: If host-initiated accesses are ever exempted from - * ignore_msrs (in kvm_do_msr_access()), drop this manual check - * and rely on KVM's standard checks to reject accesses to regs - * that don't exist. - */ - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK)) - return -EINVAL; - - reg->type =3D KVM_X86_REG_TYPE_MSR; - reg->index =3D MSR_KVM_INTERNAL_GUEST_SSP; - break; - default: - return -EINVAL; - } - return 0; -} - -static int kvm_get_one_msr(struct kvm_vcpu *vcpu, u32 msr, u64 __user *use= r_val) -{ - u64 val; - - if (do_get_msr(vcpu, msr, &val)) - return -EINVAL; - - if (put_user(val, user_val)) - return -EFAULT; - - return 0; -} - -static int kvm_set_one_msr(struct kvm_vcpu *vcpu, u32 msr, u64 __user *use= r_val) -{ - u64 val; - - if (get_user(val, user_val)) - return -EFAULT; - - if (do_set_msr(vcpu, msr, &val)) - return -EINVAL; - - return 0; -} - -static int kvm_get_set_one_reg(struct kvm_vcpu *vcpu, unsigned int ioctl, - void __user *argp) -{ - struct kvm_one_reg one_reg; - struct kvm_x86_reg_id *reg; - u64 __user *user_val; - bool load_fpu; - int r; - - if (copy_from_user(&one_reg, argp, sizeof(one_reg))) - return -EFAULT; - - if ((one_reg.id & KVM_REG_ARCH_MASK) !=3D KVM_REG_X86) - return -EINVAL; - - reg =3D (struct kvm_x86_reg_id *)&one_reg.id; - if (reg->rsvd1 || reg->rsvd2) - return -EINVAL; - - if (reg->type =3D=3D KVM_X86_REG_TYPE_KVM) { - r =3D kvm_translate_kvm_reg(vcpu, reg); - if (r) - return r; - } - - if (reg->type !=3D KVM_X86_REG_TYPE_MSR) - return -EINVAL; - - if ((one_reg.id & KVM_REG_SIZE_MASK) !=3D KVM_REG_SIZE_U64) - return -EINVAL; - - guard(srcu)(&vcpu->kvm->srcu); - - load_fpu =3D is_xstate_managed_msr(vcpu, reg->index); - if (load_fpu) - kvm_load_guest_fpu(vcpu); - - user_val =3D u64_to_user_ptr(one_reg.addr); - if (ioctl =3D=3D KVM_GET_ONE_REG) - r =3D kvm_get_one_msr(vcpu, reg->index, user_val); - else - r =3D kvm_set_one_msr(vcpu, reg->index, user_val); - - if (load_fpu) - kvm_put_guest_fpu(vcpu); - return r; -} - -static int kvm_get_reg_list(struct kvm_vcpu *vcpu, - struct kvm_reg_list __user *user_list) -{ - u64 nr_regs =3D guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) ? 1 : 0; - u64 user_nr_regs; - - if (get_user(user_nr_regs, &user_list->n)) - return -EFAULT; - - if (put_user(nr_regs, &user_list->n)) - return -EFAULT; - - if (user_nr_regs < nr_regs) - return -E2BIG; - - if (nr_regs && - put_user(KVM_X86_REG_KVM(KVM_REG_GUEST_SSP), &user_list->reg[0])) - return -EFAULT; - - return 0; -} - -static int kvm_get_msrs(struct kvm_vcpu *vcpu, struct kvm_msrs __user *use= r_msrs) -{ - guard(srcu)(&vcpu->kvm->srcu); - - return msr_io(vcpu, user_msrs, do_get_msr, 1); -} - -static int kvm_set_msrs(struct kvm_vcpu *vcpu, struct kvm_msrs __user *use= r_msrs) -{ - guard(srcu)(&vcpu->kvm->srcu); - - return msr_io(vcpu, user_msrs, do_set_msr, 0); -} - long kvm_arch_vcpu_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg) { @@ -6532,113 +4150,6 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, return r; } =20 -static struct kvm_x86_msr_filter *kvm_alloc_msr_filter(bool default_allow) -{ - struct kvm_x86_msr_filter *msr_filter; - - msr_filter =3D kzalloc_obj(*msr_filter, GFP_KERNEL_ACCOUNT); - if (!msr_filter) - return NULL; - - msr_filter->default_allow =3D default_allow; - return msr_filter; -} - -static void kvm_free_msr_filter(struct kvm_x86_msr_filter *msr_filter) -{ - u32 i; - - if (!msr_filter) - return; - - for (i =3D 0; i < msr_filter->count; i++) - kfree(msr_filter->ranges[i].bitmap); - - kfree(msr_filter); -} - -static int kvm_add_msr_filter(struct kvm_x86_msr_filter *msr_filter, - struct kvm_msr_filter_range *user_range) -{ - unsigned long *bitmap; - size_t bitmap_size; - - if (!user_range->nmsrs) - return 0; - - if (user_range->flags & ~KVM_MSR_FILTER_RANGE_VALID_MASK) - return -EINVAL; - - if (!user_range->flags) - return -EINVAL; - - bitmap_size =3D BITS_TO_LONGS(user_range->nmsrs) * sizeof(long); - if (!bitmap_size || bitmap_size > KVM_MSR_FILTER_MAX_BITMAP_SIZE) - return -EINVAL; - - bitmap =3D memdup_user((__user u8*)user_range->bitmap, bitmap_size); - if (IS_ERR(bitmap)) - return PTR_ERR(bitmap); - - msr_filter->ranges[msr_filter->count] =3D (struct msr_bitmap_range) { - .flags =3D user_range->flags, - .base =3D user_range->base, - .nmsrs =3D user_range->nmsrs, - .bitmap =3D bitmap, - }; - - msr_filter->count++; - return 0; -} - -static int kvm_vm_ioctl_set_msr_filter(struct kvm *kvm, - struct kvm_msr_filter *filter) -{ - struct kvm_x86_msr_filter *new_filter, *old_filter; - bool default_allow; - bool empty =3D true; - int r; - u32 i; - - if (filter->flags & ~KVM_MSR_FILTER_VALID_MASK) - return -EINVAL; - - for (i =3D 0; i < ARRAY_SIZE(filter->ranges); i++) - empty &=3D !filter->ranges[i].nmsrs; - - default_allow =3D !(filter->flags & KVM_MSR_FILTER_DEFAULT_DENY); - if (empty && !default_allow) - return -EINVAL; - - new_filter =3D kvm_alloc_msr_filter(default_allow); - if (!new_filter) - return -ENOMEM; - - for (i =3D 0; i < ARRAY_SIZE(filter->ranges); i++) { - r =3D kvm_add_msr_filter(new_filter, &filter->ranges[i]); - if (r) { - kvm_free_msr_filter(new_filter); - return r; - } - } - - mutex_lock(&kvm->lock); - old_filter =3D rcu_replace_pointer(kvm->arch.msr_filter, new_filter, - mutex_is_locked(&kvm->lock)); - mutex_unlock(&kvm->lock); - synchronize_srcu(&kvm->srcu); - - kvm_free_msr_filter(old_filter); - - /* - * Recalc MSR intercepts as userspace may want to intercept accesses to - * MSRs that KVM would otherwise pass through to the guest. - */ - kvm_make_all_cpus_request(kvm, KVM_REQ_RECALC_INTERCEPTS); - - return 0; -} - #ifdef CONFIG_KVM_COMPAT /* for KVM_X86_SET_MSR_FILTER */ struct kvm_msr_filter_range_compat { @@ -7159,157 +4670,6 @@ int kvm_arch_vm_ioctl(struct file *filp, unsigned i= nt ioctl, unsigned long arg) return r; } =20 -static void kvm_probe_feature_msr(u32 msr_index) -{ - u64 data; - - if (kvm_get_feature_msr(NULL, msr_index, &data, true)) - return; - - msr_based_features[num_msr_based_features++] =3D msr_index; -} - -static void kvm_probe_msr_to_save(u32 msr_index) -{ - u32 dummy[2]; - - if (rdmsr_safe(msr_index, &dummy[0], &dummy[1])) - return; - - /* - * Even MSRs that are valid in the host may not be exposed to guests in - * some cases. - */ - switch (msr_index) { - case MSR_IA32_BNDCFGS: - if (!kvm_mpx_supported()) - return; - break; - case MSR_TSC_AUX: - if (!kvm_cpu_cap_has(X86_FEATURE_RDTSCP) && - !kvm_cpu_cap_has(X86_FEATURE_RDPID)) - return; - break; - case MSR_IA32_UMWAIT_CONTROL: - if (!kvm_cpu_cap_has(X86_FEATURE_WAITPKG)) - return; - break; - case MSR_IA32_RTIT_CTL: - case MSR_IA32_RTIT_STATUS: - if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT)) - return; - break; - case MSR_IA32_RTIT_CR3_MATCH: - if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) || - !intel_pt_validate_hw_cap(PT_CAP_cr3_filtering)) - return; - break; - case MSR_IA32_RTIT_OUTPUT_BASE: - case MSR_IA32_RTIT_OUTPUT_MASK: - if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) || - (!intel_pt_validate_hw_cap(PT_CAP_topa_output) && - !intel_pt_validate_hw_cap(PT_CAP_single_range_output))) - return; - break; - case MSR_IA32_RTIT_ADDR0_A ... MSR_IA32_RTIT_ADDR3_B: - if (!kvm_cpu_cap_has(X86_FEATURE_INTEL_PT) || - (msr_index - MSR_IA32_RTIT_ADDR0_A >=3D - intel_pt_validate_hw_cap(PT_CAP_num_address_ranges) * 2)) - return; - break; - case MSR_ARCH_PERFMON_PERFCTR0 ... - MSR_ARCH_PERFMON_PERFCTR0 + KVM_MAX_NR_GP_COUNTERS - 1: - if (msr_index - MSR_ARCH_PERFMON_PERFCTR0 >=3D - kvm_pmu_cap.num_counters_gp) - return; - break; - case MSR_ARCH_PERFMON_EVENTSEL0 ... - MSR_ARCH_PERFMON_EVENTSEL0 + KVM_MAX_NR_GP_COUNTERS - 1: - if (msr_index - MSR_ARCH_PERFMON_EVENTSEL0 >=3D - kvm_pmu_cap.num_counters_gp) - return; - break; - case MSR_ARCH_PERFMON_FIXED_CTR0 ... - MSR_ARCH_PERFMON_FIXED_CTR0 + KVM_MAX_NR_FIXED_COUNTERS - 1: - if (msr_index - MSR_ARCH_PERFMON_FIXED_CTR0 >=3D - kvm_pmu_cap.num_counters_fixed) - return; - break; - case MSR_AMD64_PERF_CNTR_GLOBAL_CTL: - case MSR_AMD64_PERF_CNTR_GLOBAL_STATUS: - case MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_CLR: - case MSR_AMD64_PERF_CNTR_GLOBAL_STATUS_SET: - if (!kvm_cpu_cap_has(X86_FEATURE_PERFMON_V2)) - return; - break; - case MSR_IA32_XFD: - case MSR_IA32_XFD_ERR: - if (!kvm_cpu_cap_has(X86_FEATURE_XFD)) - return; - break; - case MSR_IA32_TSX_CTRL: - if (!(kvm_get_arch_capabilities() & ARCH_CAP_TSX_CTRL_MSR)) - return; - break; - case MSR_IA32_XSS: - if (!kvm_caps.supported_xss) - return; - break; - case MSR_IA32_U_CET: - case MSR_IA32_S_CET: - if (!kvm_cpu_cap_has(X86_FEATURE_SHSTK) && - !kvm_cpu_cap_has(X86_FEATURE_IBT)) - return; - break; - case MSR_IA32_INT_SSP_TAB: - if (!kvm_cpu_cap_has(X86_FEATURE_LM)) - return; - fallthrough; - case MSR_IA32_PL0_SSP ... MSR_IA32_PL3_SSP: - if (!kvm_cpu_cap_has(X86_FEATURE_SHSTK)) - return; - break; - default: - break; - } - - msrs_to_save[num_msrs_to_save++] =3D msr_index; -} - -static void kvm_init_msr_lists(void) -{ - unsigned i; - - BUILD_BUG_ON_MSG(KVM_MAX_NR_FIXED_COUNTERS !=3D 3, - "Please update the fixed PMCs in msrs_to_save_pmu[]"); - - num_msrs_to_save =3D 0; - num_emulated_msrs =3D 0; - num_msr_based_features =3D 0; - - for (i =3D 0; i < ARRAY_SIZE(msrs_to_save_base); i++) - kvm_probe_msr_to_save(msrs_to_save_base[i]); - - if (enable_pmu) { - for (i =3D 0; i < ARRAY_SIZE(msrs_to_save_pmu); i++) - kvm_probe_msr_to_save(msrs_to_save_pmu[i]); - } - - for (i =3D 0; i < ARRAY_SIZE(emulated_msrs_all); i++) { - if (!kvm_x86_call(has_emulated_msr)(NULL, - emulated_msrs_all[i])) - continue; - - emulated_msrs[num_emulated_msrs++] =3D emulated_msrs_all[i]; - } - - for (i =3D KVM_FIRST_EMULATED_VMX_MSR; i <=3D KVM_LAST_EMULATED_VMX_MSR; = i++) - kvm_probe_feature_msr(i); - - for (i =3D 0; i < ARRAY_SIZE(msr_based_features_all_except_vmx); i++) - kvm_probe_feature_msr(msr_based_features_all_except_vmx[i]); -} - static int vcpu_mmio_write(struct kvm_vcpu *vcpu, gpa_t addr, int len, void *__v) { @@ -8247,61 +5607,22 @@ static int emulator_get_msr_with_filter(struct x86_= emulate_ctxt *ctxt, u32 msr_index, u64 *pdata) { struct kvm_vcpu *vcpu =3D emul_to_vcpu(ctxt); - int r; =20 - r =3D kvm_emulate_msr_read(vcpu, msr_index, pdata); - if (r < 0) - return X86EMUL_UNHANDLEABLE; - - if (r) { - if (kvm_msr_user_space(vcpu, msr_index, KVM_EXIT_X86_RDMSR, 0, - complete_emulated_rdmsr, r)) - return X86EMUL_IO_NEEDED; - - trace_kvm_msr_read_ex(msr_index); - return X86EMUL_PROPAGATE_FAULT; - } - - trace_kvm_msr_read(msr_index, *pdata); - return X86EMUL_CONTINUE; + return kvm_emulator_get_msr_with_filter(vcpu, msr_index, pdata); } =20 static int emulator_set_msr_with_filter(struct x86_emulate_ctxt *ctxt, u32 msr_index, u64 data) { struct kvm_vcpu *vcpu =3D emul_to_vcpu(ctxt); - int r; =20 - r =3D kvm_emulate_msr_write(vcpu, msr_index, data); - if (r < 0) - return X86EMUL_UNHANDLEABLE; - - if (r) { - if (kvm_msr_user_space(vcpu, msr_index, KVM_EXIT_X86_WRMSR, data, - complete_emulated_msr_access, r)) - return X86EMUL_IO_NEEDED; - - trace_kvm_msr_write_ex(msr_index, data); - return X86EMUL_PROPAGATE_FAULT; - } - - trace_kvm_msr_write(msr_index, data); - return X86EMUL_CONTINUE; + return kvm_emulator_set_msr_with_filter(vcpu, msr_index, data); } =20 static int emulator_get_msr(struct x86_emulate_ctxt *ctxt, u32 msr_index, u64 *pdata) { - /* - * Treat emulator accesses to the current shadow stack pointer as host- - * initiated, as they aren't true MSR accesses (SSP is a "just a reg"), - * and this API is used only for implicit accesses, i.e. not RDMSR, and - * so the index is fully KVM-controlled. - */ - if (unlikely(msr_index =3D=3D MSR_KVM_INTERNAL_GUEST_SSP)) - return kvm_msr_read(emul_to_vcpu(ctxt), msr_index, pdata); - - return __kvm_emulate_msr_read(emul_to_vcpu(ctxt), msr_index, pdata); + return kvm_emulator_get_msr(emul_to_vcpu(ctxt), msr_index, pdata); } =20 static int emulator_check_rdpmc_early(struct x86_emulate_ctxt *ctxt, u32 p= mc) @@ -13250,32 +10571,6 @@ void kvm_arch_gmem_invalidate(kvm_pfn_t start, kvm= _pfn_t end) #endif #endif =20 -int kvm_spec_ctrl_test_value(u64 value) -{ - /* - * test that setting IA32_SPEC_CTRL to given value - * is allowed by the host processor - */ - - u64 saved_value; - unsigned long flags; - int ret =3D 0; - - local_irq_save(flags); - - if (rdmsrq_safe(MSR_IA32_SPEC_CTRL, &saved_value)) - ret =3D 1; - else if (wrmsrq_safe(MSR_IA32_SPEC_CTRL, value)) - ret =3D 1; - else - wrmsrq(MSR_IA32_SPEC_CTRL, saved_value); - - local_irq_restore(flags); - - return ret; -} -EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_spec_ctrl_test_value); - void kvm_fixup_and_inject_pf_error(struct kvm_vcpu *vcpu, gva_t gva, u16 e= rror_code) { struct kvm_mmu *mmu =3D vcpu->arch.walk_mmu; diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 31e67b060148..fd3d0a196526 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -6,6 +6,7 @@ #include #include #include +#include "msrs.h" #include "mmu.h" #include "regs.h" #include "kvm_emulate.h" @@ -45,14 +46,6 @@ do { \ failed; \ }) =20 -/* - * The first...last VMX feature MSRs that are emulated by KVM. This may o= r may - * not cover all known VMX MSRs, as KVM doesn't emulate an MSR until there= 's an - * associated feature that KVM supports for nested virtualization. - */ -#define KVM_FIRST_EMULATED_VMX_MSR MSR_IA32_VMX_BASIC -#define KVM_LAST_EMULATED_VMX_MSR MSR_IA32_VMX_VMFUNC - #define KVM_DEFAULT_PLE_GAP 128 #define KVM_VMX_DEFAULT_PLE_WINDOW 4096 #define KVM_DEFAULT_PLE_WINDOW_GROW 2 @@ -61,16 +54,6 @@ do { \ #define KVM_SVM_DEFAULT_PLE_WINDOW_MAX USHRT_MAX #define KVM_SVM_DEFAULT_PLE_WINDOW 3000 =20 -/* - * KVM's internal, non-ABI indices for synthetic MSRs. The values themselv= es - * are arbitrary and have no meaning, the only requirement is that they do= n't - * conflict with "real" MSRs that KVM supports. Use values at the upper end - * of KVM's reserved paravirtual MSR range to minimize churn, i.e. these v= alues - * will be usable until KVM exhausts its supply of paravirtual MSR indices. - */ - -#define MSR_KVM_INTERNAL_GUEST_SSP 0x4b564dff - static inline unsigned int __grow_ple_window(unsigned int val, unsigned int base, unsigned int modifier, unsigned int max) { @@ -101,9 +84,6 @@ static inline unsigned int __shrink_ple_window(unsigned = int val, return max(val, min); } =20 -#define MSR_IA32_CR_PAT_DEFAULT \ - PAT_VALUE(WB, WT, UC_MINUS, UC, WB, WT, UC_MINUS, UC) - void kvm_service_local_tlb_flush_requests(struct kvm_vcpu *vcpu); int kvm_check_nested_events(struct kvm_vcpu *vcpu); =20 @@ -378,15 +358,12 @@ void kvm_deliver_exception_payload(struct kvm_vcpu *v= cpu, struct kvm_queued_exception *ex); void kvm_handle_exception_payload_quirk(struct kvm_vcpu *vcpu); =20 -int kvm_mtrr_set_msr(struct kvm_vcpu *vcpu, u32 msr, u64 data); -int kvm_mtrr_get_msr(struct kvm_vcpu *vcpu, u32 msr, u64 *pdata); void kvm_fixup_and_inject_pf_error(struct kvm_vcpu *vcpu, gva_t gva, u16 e= rror_code); int x86_decode_emulated_instruction(struct kvm_vcpu *vcpu, int emulation_t= ype, void *insn, int insn_len); int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, int emulation_type, void *insn, int insn_len); -fastpath_t handle_fastpath_wrmsr(struct kvm_vcpu *vcpu); -fastpath_t handle_fastpath_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int r= eg); + fastpath_t handle_fastpath_hlt(struct kvm_vcpu *vcpu); fastpath_t handle_fastpath_invd(struct kvm_vcpu *vcpu); =20 @@ -432,20 +409,6 @@ extern bool enable_vmware_backdoor; =20 extern int pi_inject_timer; =20 -extern bool report_ignored_msrs; - -static inline void kvm_pr_unimpl_wrmsr(struct kvm_vcpu *vcpu, u32 msr, u64= data) -{ - if (report_ignored_msrs) - vcpu_unimpl(vcpu, "Unhandled WRMSR(0x%x) =3D 0x%llx\n", msr, data); -} - -static inline void kvm_pr_unimpl_rdmsr(struct kvm_vcpu *vcpu, u32 msr) -{ - if (report_ignored_msrs) - vcpu_unimpl(vcpu, "Unhandled RDMSR(0x%x)\n", msr); -} - static inline u64 nsec_to_cycles(struct kvm_vcpu *vcpu, u64 nsec) { return pvclock_scale_delta(nsec, vcpu->arch.virtual_tsc_mult, @@ -563,33 +526,10 @@ static inline void kvm_machine_check(void) #endif } =20 -int kvm_spec_ctrl_test_value(u64 value); int kvm_handle_memory_failure(struct kvm_vcpu *vcpu, int r, struct x86_exception *e); void kvm_invalidate_pcid(struct kvm_vcpu *vcpu, unsigned long pcid); int kvm_handle_invpcid(struct kvm_vcpu *vcpu, unsigned long type, gva_t gv= a); -bool kvm_msr_allowed(struct kvm_vcpu *vcpu, u32 index, u32 type); - -enum kvm_msr_access { - MSR_TYPE_R =3D BIT(0), - MSR_TYPE_W =3D BIT(1), - MSR_TYPE_RW =3D MSR_TYPE_R | MSR_TYPE_W, -}; - -/* - * Internal error codes that are used to indicate that MSR emulation encou= ntered - * an error that should result in #GP in the guest, unless userspace handl= es it. - * Note, '1', '0', and negative numbers are off limits, as they are used b= y KVM - * as part of KVM's lightly documented internal KVM_RUN return codes. - * - * UNSUPPORTED - The MSR isn't supported, either because it is completely - * unknown to KVM, or because the MSR should not exist according - * to the vCPU model. - * - * FILTERED - Access to the MSR is denied by a userspace MSR filter. - */ -#define KVM_MSR_RET_UNSUPPORTED 2 -#define KVM_MSR_RET_FILTERED 3 =20 int kvm_sev_es_mmio(struct kvm_vcpu *vcpu, bool is_write, gpa_t gpa, unsigned int bytes, void *data); @@ -649,27 +589,4 @@ int ____kvm_emulate_hypercall(struct kvm_vcpu *vcpu, i= nt cpl, =20 int kvm_emulate_hypercall(struct kvm_vcpu *vcpu); =20 -#define CET_US_RESERVED_BITS GENMASK(9, 6) -#define CET_US_SHSTK_MASK_BITS GENMASK(1, 0) -#define CET_US_IBT_MASK_BITS (GENMASK_ULL(5, 2) | GENMASK_ULL(63, 10)) -#define CET_US_LEGACY_BITMAP_BASE(data) ((data) >> 12) - -static inline bool kvm_is_valid_u_s_cet(struct kvm_vcpu *vcpu, u64 data) -{ - if (data & CET_US_RESERVED_BITS) - return false; - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_SHSTK) && - (data & CET_US_SHSTK_MASK_BITS)) - return false; - if (!guest_cpu_cap_has(vcpu, X86_FEATURE_IBT) && - (data & CET_US_IBT_MASK_BITS)) - return false; - if (!IS_ALIGNED(CET_US_LEGACY_BITMAP_BASE(data), 4)) - return false; - /* IBT can be suppressed iff the TRACKER isn't WAIT_ENDBR. */ - if ((data & CET_SUPPRESS) && (data & CET_WAIT_ENDBR)) - return false; - - return true; -} #endif --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7E1804779B2 for ; Fri, 29 May 2026 22:23:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093397; cv=none; b=AXCOm9Lnk3I24cZ+JzMBLSOn2Ue8jfb2SCuwbv849yXHNJwDq9obstrqytDkQQ+xK4ZG9f4JAvxqnBhcQksYaYgjg9YKoSJVcPMlvkg5sj2h8DVk6S6OK/kEHtq0C/8ob6A8mkZXvZMt9pvR6o102qa5mqC06AFhJZxJMaHCpjA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093397; c=relaxed/simple; bh=67zW30Fghsqm2bbgoRDmbDTWcpgL4geOGG0pkoZa+DQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ZDOIPuk23kdfRjrsl0B9pkTPeSqe6bTTiAAg7GF3odYDX1/e0+76qtUSUeAscrS86PTUAoKR86BQ4ORwjuTywY6VEM7glWRycBKjTjphzm5ICvPjvQxYkCwKDLorfuloO8KqIA3lQ59f8+64F/ZD/CId0j+ieCVGrESlld2y8dY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=UDRA/qcI; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="UDRA/qcI" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-c85807671b1so307889a12.2 for ; Fri, 29 May 2026 15:23:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093396; x=1780698196; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=4YZDFZBCarOdxlxdCzemgJxyqnyDXT5M/adWibMWDkM=; b=UDRA/qcICcOiNfGR0SUbiJJsc437nlYOUgUzjpxZulD9/NkhKgFNeABpfvVvLDvTy7 JhvWDp/mxgGPnXgQ5BlvOyFnf241skP06E4Je2VZr1hY9bHOtA2i+QCLP5BUWMuOozSK 1wakd7SKzc50yWuXSm7vByAjfFOeL7Bv5z31KU0H0xfyWVKyHEjtpCrxCDYAwdxTHpX5 2nIrvqggvoRoDAwL7z/ORr1nmuTglOXjqD3K9Q0WdEskCaOG+RMfaS+t5ygtrzon0EC4 IZmjy91NZDoc8IJPSVQSczDHuXEY8bmHjBr4MBTC1G7oisvXUH1PiT9N8dTqCDy2oYsO q34w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093396; x=1780698196; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=4YZDFZBCarOdxlxdCzemgJxyqnyDXT5M/adWibMWDkM=; b=N3MbduBUaXyOkTdRXGI12bO3rZYlB4DnOAsu6JZiMTwJ+UMYhpBUGILnH3HmpFeZ3U Pr4oEkC85h/S1uh8S3xDGefr2+gtNMQOOHfmOxUGrrVYOCb6+UoZJc0z4b23xHdwpLSG FL1c7bmCYyQB6tHkVAQEbqmmfUgMK1B/DL7h9rlf5bUCZc7MHiqlgdCOrCwoTSKpbaRw jpcO254Yox1g5fOBx2f/68s0yqU0RRcBS2e1Uu+P4ysIQaGFOWiEt9jBGWi+9Kb2lyd5 DpU8bVx3gGLAGy/kRSGuVjxVY7f9ENR+QuDg+WomdARAC7OcHa5eFO+S8HnPVO0bRjmZ pTlQ== X-Forwarded-Encrypted: i=1; AFNElJ8fLIhi31Vednew9p8PVYUAn2I3pbDUainXwg4bvJZiHGOraviTpC89KQmLaKEcR6HABSbqvBTUP79KJbI=@vger.kernel.org X-Gm-Message-State: AOJu0YxpEKZKhWEgAovevGG06FxwXl8Yqy30tdYXYgBX8VXmkFhbiQgJ +7H7uWbrw6vJNyOAnVsL7fSvdHaC/x7SHtGCTkzPhhHrznsFN3f+3R5+V14zNNkXK638ujQXGhe sNiC+Vg== X-Received: from pgig5.prod.google.com ([2002:a63:f405:0:b0:c80:2399:151]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6300:189:b0:39b:89e0:2e37 with SMTP id adf61e73a8af0-3b427c23375mr1271417637.14.1780093395678; Fri, 29 May 2026 15:23:15 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:10 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-28-seanjc@google.com> Subject: [PATCH v3 27/40] KVM: x86: Move register helper declarations from kvm_host.h => regs.h From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Relocate declarations of Control/Debug Register, EFLAGS and RIP helpers from x86's kvm_host.h to regs.h, to continue trimming down kvm_host.h. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang --- arch/x86/include/asm/kvm_host.h | 17 ----------------- arch/x86/kvm/regs.h | 17 +++++++++++++++++ 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 19091d89d3cc..0f4b16b26a27 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2186,8 +2186,6 @@ void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u= 64 gen); void kvm_mmu_change_mmu_pages(struct kvm *kvm, unsigned long kvm_nr_mmu_pa= ges); void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end); =20 -int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3); - /* * EMULTYPE_NO_DECODE - Set when re-emulating an instruction (after comple= ting * userspace I/O) to indicate that the emulation context @@ -2313,24 +2311,12 @@ void kvm_vcpu_deliver_sipi_vector(struct kvm_vcpu *= vcpu, u8 vector); int kvm_task_switch(struct kvm_vcpu *vcpu, u16 tss_selector, int idt_index, int reason, bool has_error_code, u32 error_code); =20 -void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsign= ed long cr0); -void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsign= ed long cr4); -int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0); -int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3); -int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4); -int kvm_set_cr8(struct kvm_vcpu *vcpu, unsigned long cr8); -int kvm_set_dr(struct kvm_vcpu *vcpu, int dr, unsigned long val); -unsigned long kvm_get_dr(struct kvm_vcpu *vcpu, int dr); -unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu); -void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw); int __kvm_set_xcr(struct kvm_vcpu *vcpu, u32 index, u64 xcr); int kvm_emulate_xsetbv(struct kvm_vcpu *vcpu); =20 int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr); int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr); =20 -unsigned long kvm_get_rflags(struct kvm_vcpu *vcpu); -void kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags); int kvm_emulate_rdpmc(struct kvm_vcpu *vcpu); =20 void kvm_queue_exception(struct kvm_vcpu *vcpu, unsigned nr); @@ -2520,9 +2506,6 @@ u64 kvm_read_l1_tsc(struct kvm_vcpu *vcpu, u64 host_t= sc); u64 kvm_calc_nested_tsc_offset(u64 l1_offset, u64 l2_offset, u64 l2_multip= lier); u64 kvm_calc_nested_tsc_multiplier(u64 l1_multiplier, u64 l2_multiplier); =20 -unsigned long kvm_get_linear_rip(struct kvm_vcpu *vcpu); -bool kvm_is_linear_rip(struct kvm_vcpu *vcpu, unsigned long linear_rip); - void kvm_make_scan_ioapic_request(struct kvm *kvm); void kvm_make_scan_ioapic_request_mask(struct kvm *kvm, unsigned long *vcpu_bitmap); diff --git a/arch/x86/kvm/regs.h b/arch/x86/kvm/regs.h index c224874bbdde..30ef08d60a74 100644 --- a/arch/x86/kvm/regs.h +++ b/arch/x86/kvm/regs.h @@ -16,6 +16,18 @@ =20 static_assert(!(KVM_POSSIBLE_CR0_GUEST_BITS & X86_CR0_PDPTR_BITS)); =20 +void kvm_post_set_cr0(struct kvm_vcpu *vcpu, unsigned long old_cr0, unsign= ed long cr0); +void kvm_post_set_cr4(struct kvm_vcpu *vcpu, unsigned long old_cr4, unsign= ed long cr4); +int kvm_set_cr0(struct kvm_vcpu *vcpu, unsigned long cr0); +int kvm_set_cr3(struct kvm_vcpu *vcpu, unsigned long cr3); +int kvm_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4); +int kvm_set_cr8(struct kvm_vcpu *vcpu, unsigned long cr8); +int kvm_set_dr(struct kvm_vcpu *vcpu, int dr, unsigned long val); +unsigned long kvm_get_dr(struct kvm_vcpu *vcpu, int dr); +unsigned long kvm_get_cr8(struct kvm_vcpu *vcpu); +void kvm_lmsw(struct kvm_vcpu *vcpu, unsigned long msw); +int load_pdptrs(struct kvm_vcpu *vcpu, unsigned long cr3); + static inline bool is_long_mode(struct kvm_vcpu *vcpu) { #ifdef CONFIG_X86_64 @@ -425,7 +437,12 @@ static inline unsigned long kvm_get_segment_base(struc= t kvm_vcpu *vcpu, int seg) return kvm_x86_call(get_segment_base)(vcpu, seg); } =20 +unsigned long kvm_get_linear_rip(struct kvm_vcpu *vcpu); +bool kvm_is_linear_rip(struct kvm_vcpu *vcpu, unsigned long linear_rip); + +unsigned long kvm_get_rflags(struct kvm_vcpu *vcpu); void __kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags); +void kvm_set_rflags(struct kvm_vcpu *vcpu, unsigned long rflags); =20 void kvm_x86_vcpu_ioctl_get_sregs2(struct kvm_vcpu *vcpu, struct kvm_sregs2 *sregs2); --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B5E6D477E32 for ; Fri, 29 May 2026 22:23:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093398; cv=none; b=EjwBZF97LlqvsG69tGoNERT4lAw2bkGUXmXSdyScYeJn4MD7+ku2ciI+QN7fcXeAXMV3NqdaIiUYueUugpWfYa4RxaL6wQDlD/S+Ye2LLZuSr0ZJAjYwiMABKbsIpgKMM4D/yrErCoxYBNzcIDgc0tXTgdMJY0UVYo++Vedo844= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093398; c=relaxed/simple; bh=hQlztW9k5qUi+RS2tHkICEwQcFdk//PT0p9sdq++9zk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=fxMil8NEPAZsbcXtIxYNGxhBmb/9UrgIl/eR8kks64Y7XubSUpXwMnFLkCvemkPymHeWiCYpDpwqV93Pi6E+0LiWGTAucT5l+golMiblEgdm1aaxsYsWm10zzLzNjv0mZd5th/F8PC1mKuSmpPMP38Wy+jpJJPE7bVzP9IiSj6g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Fc/L91gx; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Fc/L91gx" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-36b9d265308so1634388a91.2 for ; Fri, 29 May 2026 15:23:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093397; x=1780698197; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=VRGvG+QPcfiu5+a02Ie+c018EHuB9fhsX5SkwkrCswc=; b=Fc/L91gxUWn5UKxT3PW1FkIwKECTuXImIW3IjRDs3Bx8qmyyeCYjCq3pThFNfOxBPC p8tgMK/LtcFa7IWROAFDKdGbC52Qik9vwNZOSkZRjqzPusp2q/iHZcSwFKHW7/jmmpZp Xb1eKxe3EH8fqtZrKpxXEdnIoCN/60h1H59o+a9eMM8TRebkVxgQCYyKLqmTkX/WAOp5 YdqL34NDPzI8IVabYCysnLfphle0zeuvZbOgdzCpElnBQ9JO9UuRTvY60kUiqEz+yKau +ipmIa8K81gNYF8KxL5k+45EiKiUSd0K9BedHIDJ7bPE9HurK0dLRp1uaxxjDEQy7+c6 PR3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093397; x=1780698197; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=VRGvG+QPcfiu5+a02Ie+c018EHuB9fhsX5SkwkrCswc=; b=YWoD5iwgqUUEIlpleWulQGBiSz6XUxl/T+puk2aegh+wSuLuIl3tcs9wMLHHQt7fMf 0TXhlk1OTCzbY4X6thpVIBwX4p7pDi0OnWaZWIIcKY/GsQcsjVv74N/vD9nxvqIGww40 OULXvZiVkopx+PzibNeGfucaqIQeOglUa3NnjYkkbe6dzST3BUINKeGrtn2gELpsHZqC R7zW3cTSDQgLjObXNydeTm2C6xUmzX8JdY0lyxeTKyvEYDqtrZuGo+FZlTqCn0wrTMmS HAwbzXbM1lTcn1v9OYH7R3IIYun7v6UDA6H681dE5BeqipBMDF66yj8yAj6IyYWd7Odv lqgQ== X-Forwarded-Encrypted: i=1; AFNElJ+C6StJjuya4oX2IOzgRuOY37RkTQAUcbWvMDmIJBZJA6YDvpoqJnyeLyNzbPkuXEWXxV6AmDtX7EBa9Fo=@vger.kernel.org X-Gm-Message-State: AOJu0Yw6EP0i8D1US6vS0ZwvaB/9cAq4PIvJr4kuwBNVaqLAiM/cjIzY XTEROPsmOix2KmLyrsBtzVKsYoP90rbTjrBY0EQmsQJN1i6lj1PDBnmzJHKtPAJTYIvFxrpLjmf cRvIkEw== X-Received: from pgww22.prod.google.com ([2002:a05:6a02:2c96:b0:c63:55bd:18f0]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:5286:b0:36a:8240:2477 with SMTP id 98e67ed59e1d1-36c59abd658mr1076084a91.19.1780093396783; Fri, 29 May 2026 15:23:16 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:11 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-29-seanjc@google.com> Subject: [PATCH v3 28/40] KVM: x86: Move kvm_{g,s}et_segment() to inline helpers in regs.h From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Define kvm_{g,s}et_segment() as inline functions in regs.h, as they are literally one-line wrappers to invoke vendor code. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Reviewed-by: Yosry Ahmed --- arch/x86/include/asm/kvm_host.h | 2 -- arch/x86/kvm/regs.h | 12 ++++++++++++ arch/x86/kvm/x86.c | 12 ------------ 3 files changed, 12 insertions(+), 14 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 0f4b16b26a27..ca2e69c80a8d 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2303,8 +2303,6 @@ int kvm_emulate_halt_noskip(struct kvm_vcpu *vcpu); int kvm_emulate_ap_reset_hold(struct kvm_vcpu *vcpu); int kvm_emulate_wbinvd(struct kvm_vcpu *vcpu); =20 -void kvm_get_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int s= eg); -void kvm_set_segment(struct kvm_vcpu *vcpu, struct kvm_segment *var, int s= eg); int kvm_load_segment_descriptor(struct kvm_vcpu *vcpu, u16 selector, int s= eg); void kvm_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector); =20 diff --git a/arch/x86/kvm/regs.h b/arch/x86/kvm/regs.h index 30ef08d60a74..7a823422d78e 100644 --- a/arch/x86/kvm/regs.h +++ b/arch/x86/kvm/regs.h @@ -437,6 +437,18 @@ static inline unsigned long kvm_get_segment_base(struc= t kvm_vcpu *vcpu, int seg) return kvm_x86_call(get_segment_base)(vcpu, seg); } =20 +static inline void kvm_set_segment(struct kvm_vcpu *vcpu, + struct kvm_segment *var, int seg) +{ + kvm_x86_call(set_segment)(vcpu, var, seg); +} + +static inline void kvm_get_segment(struct kvm_vcpu *vcpu, + struct kvm_segment *var, int seg) +{ + kvm_x86_call(get_segment)(vcpu, var, seg); +} + unsigned long kvm_get_linear_rip(struct kvm_vcpu *vcpu); bool kvm_is_linear_rip(struct kvm_vcpu *vcpu, unsigned long linear_rip); =20 diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index c648fac802f6..660ba27c76ec 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -4719,18 +4719,6 @@ static int vcpu_mmio_read(struct kvm_vcpu *vcpu, gpa= _t addr, int len, void *v) return handled; } =20 -void kvm_set_segment(struct kvm_vcpu *vcpu, - struct kvm_segment *var, int seg) -{ - kvm_x86_call(set_segment)(vcpu, var, seg); -} - -void kvm_get_segment(struct kvm_vcpu *vcpu, - struct kvm_segment *var, int seg) -{ - kvm_x86_call(get_segment)(vcpu, var, seg); -} - gpa_t kvm_mmu_gva_to_gpa_read(struct kvm_vcpu *vcpu, gva_t gva, struct x86_exception *exception) { --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3FF9547885C for ; Fri, 29 May 2026 22:23:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093400; cv=none; b=nHOe1+hUJ+6M4yMgiCz5YMY2U3d2uUPyd5/pXx17E9GEFqS5/+FPovSmxAsDhEFgMWcSJAqqhZzIeOJt+49WOk9mpc5xIcR8QEeDtvCWsWUx3Y2lqDiy6R0qB8Xz8k96tcVxrPvoWEZRTp0rgVFCdkCf/DFKpdHYPtKl4fNr7a8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093400; c=relaxed/simple; bh=C0TPGUq872+vTHaWb23bDYJo5eC/8eQ3y1jYm66lwjA=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=RrSfftKPCvdIKSQQ2Z6av4I9DtTnIJ0r9nMMO8/+AUnUz1s7jSiAOSP88gqkkbRDJVRkXEjEH209e/NkXDsvL3yvCJNtBkLK1C6Wr5K1Q8yF48JtZTAwpulyNigvH5pAZ8VVTCOYe1gLYXK/0zE6rQ8rS5L9doXPDxPXG7YbiwA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=rtoLwhD8; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="rtoLwhD8" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2baf7378ad0so156258885ad.0 for ; Fri, 29 May 2026 15:23:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093398; x=1780698198; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=Pi5uBfbWTd8F8248YP0/y/lIMSNaZw4vTpSUB9flOag=; b=rtoLwhD8H3Z+Q7T+FR8+tfrNuABCeTsU2BSLZ0KYo0Zd+o3+FOS30mUvr21L/DOyFe pz/kzUc50/w6MOSrh+VpdU3j5tFJZ8U0xGAUTJkOQGn9kN8gGm/VSa6tUiHHRS4lb4V0 BbTVsDALEpiJMg5vTbzMd5zecxfOveUHOAmM8VBtlfuYcUoPt7bmj8WpKdNF6B5jepdy +K8wfXQo118jboAaR3QIDals5tCZbOw/X/j9/hTjxvlCcAvqPbD8awCfiZIta0xyQyKm nzOPcEUfC1n+brsmoGnoRaW/+rseZ1fE5Pa126IePxyD964xre7UNYBXTtXV0dJmakxF v+zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093398; x=1780698198; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Pi5uBfbWTd8F8248YP0/y/lIMSNaZw4vTpSUB9flOag=; b=oFJkD0qnP386ZwyvcNMyoG8kQ7c6yW6hlQ3UUlW3mJskXyGwfkJeV5/zx+IYc3aXbE E62cxxGdLseGFDNVVWx3QIqW0jqjmm5Lj9plz0p3PrEjLpxjHHVqvFC96vPi5zZs305Q 98pMjUapFnHK+WKm8YfAsnhaE1wtS87jzO94uoss+c/m2Ho/lYoVP4ef3powqAFUEsqk lcIawNn/UYJWit9SiFdk7pQtnhq/KBNP2QETFlHPBVcgousuEfgiAGVHCOeNiW8XMdg7 dpEUWxSdw3GrmqX/zfbGVqJkc7YoxztbpCaN/C0gQD8b1rRHoSIMUFPrQz9OoMlz17Nk Xetw== X-Forwarded-Encrypted: i=1; AFNElJ99iPaFb+2cZB/nwcK7eENipOrz11IUDWpeb8IjODrDvlspktY9y+HK1jLYU7Pu45w5ZbjkBEwAlVufxhQ=@vger.kernel.org X-Gm-Message-State: AOJu0Yy+q/dSAM38xJi6GAcqs0YG9TqepOfxzutNb+6MjvENH9KJsd2x +poU6t0TlLXHfcpniu1p78Ujdgkzlvr6wL02YZmXBr2Pz9lDVT7wZlShjbT1X9XvOiKT1j0blkF dfYwolg== X-Received: from pgut6.prod.google.com ([2002:a65:6086:0:b0:c85:87db:a5c6]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:e552:b0:2be:fcca:31fe with SMTP id d9443c01a7336-2bf3682bafdmr19810605ad.23.1780093398378; Fri, 29 May 2026 15:23:18 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:12 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-30-seanjc@google.com> Subject: [PATCH v3 29/40] KVM: x86: Remove defunct kvm_load_segment_descriptor() declaration. From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Remove a dead kvm_load_segment_descriptor() declaration, no functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Reviewed-by: Yosry Ahmed --- arch/x86/include/asm/kvm_host.h | 1 - 1 file changed, 1 deletion(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index ca2e69c80a8d..a861c0d70be0 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2303,7 +2303,6 @@ int kvm_emulate_halt_noskip(struct kvm_vcpu *vcpu); int kvm_emulate_ap_reset_hold(struct kvm_vcpu *vcpu); int kvm_emulate_wbinvd(struct kvm_vcpu *vcpu); =20 -int kvm_load_segment_descriptor(struct kvm_vcpu *vcpu, u16 selector, int s= eg); void kvm_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector); =20 int kvm_task_switch(struct kvm_vcpu *vcpu, u16 tss_selector, int idt_index, --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3706D47A0A4 for ; Fri, 29 May 2026 22:23:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093402; cv=none; b=J/0ps2YLiUca1yYLwsny0qa3w7t8eHsrsO7lQty+SE7ljRtEXxyhu0NxI4c9dUDNHLz2+PXdFuWyUdPzo7UKm5kjDYm6M1dnOIsQXrj1sBTU9JTA0oVulv9ZUJ6+dhHnO8tdHxXzSZc65AMQ9PNw2st8pcNHNstZbxSia3VwEGE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093402; c=relaxed/simple; bh=otrVM3e4Lurh14D9F8StjWN8v6dvvHUUGpICOjvEYWw=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Ytj8rVQwQzdAdvV9o3U43wv95CWnU/jidiT/zGzwbiwDuNeyF0K0UMyRzrojZu/SOfYGbDA/ZS6sIC7fx/Cb2IsT/JsTt+FW99RmE1GEEYB4S1b0hMnrazZidgPAHw6yuoCcCzNwGJWcG60K3O9TT+sPEF48DHjTBVYHBQUGI+k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=EvcAoGIo; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="EvcAoGIo" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2b99eb06178so72290785ad.2 for ; Fri, 29 May 2026 15:23:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093399; x=1780698199; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=U06ajxKWCrZV2qZ8Tb0oiDb/K09cySTCLuYRf+Qkq+A=; b=EvcAoGIoFlHuJMS3ugwl5wqfpaGIVpaVdmVOFsCorgSTxlv8tLKBE5Ar84UKhmSsTE dj7QyPq4jT5TNDz8cOJKND05awSpLxVMWAfC3iJGAdplGfhxRg9MZnwjpO3uhxE14Xy8 lZT82aS10F212BcnKev/dTDFVw2sUlRbjxGu1CYCerWF7nhvYLaFYRB+sFO2Xzh0xp+N i7iHfXg8eAH+PQVSR7xJ1YnK1DmWkgS+tkrD1qAiPnQpzJ9wU3lzXiAJGEjvO9J2ZWoq gdbgWVFY9m2GpUw+qj2Db0+Ry/0y5OcXTuvVrNFcUur0fbdLPzEeB+2QHOu1embOCt5K 1+RA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093399; x=1780698199; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=U06ajxKWCrZV2qZ8Tb0oiDb/K09cySTCLuYRf+Qkq+A=; b=lDa9whpyiFcOycDrMp99rwZ0i8GT9pzr88t7I2MzEfbyvmkfifOHoVZk7g+O2WU8Cf jDzJZtbp0FxrSoX0+QQ+bc9QwBdmEblcoaY15z6cSYOgX/6tzEkPmblSnR0Km2KDyfPv SCDcDA1eM6TGAk3VYMaSlzCWYSU30rI0FJ9AaL/X8968sFRqnoVDiGipqhvI+1TKJtOO 6mAIiPwXWQX/LUoPmtMNof4q3Hd7oDwin+QoqrrhXdLgZ0iwSuq1Hhm1Ix0TSAh4+65M 17+YY+0VGlh6jo4C7tOeN0xQa+X83KjxxzjkHYpPC58BAqZchdtMxx1L8/j3UVgBNg8D STmA== X-Forwarded-Encrypted: i=1; AFNElJ/OKvpwDiW6TpRy8fiWEcj9R6SwHg18zoiqzG/ZsSyeb9x15OLgxppdFUY3geAY28l9JoEkyvKGQTx7NAI=@vger.kernel.org X-Gm-Message-State: AOJu0YyVnyvKmj5BAcplIu2FnMm+xYF7+gGuZDFOOYBDWcwb+4nSKwOz Ea2WH2nc9nv/1ZKGdp93mBBhwPC8FrEvBDasdxHarMTtEDK/VXZr9484oFU1crp52nAKxq7UtkP zuRTF0A== X-Received: from plav13.prod.google.com ([2002:a17:902:f0cd:b0:2bf:25c4:45ec]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:138a:b0:2b4:5f69:715d with SMTP id d9443c01a7336-2bf36809d00mr21727735ad.25.1780093399437; Fri, 29 May 2026 15:23:19 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:13 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-31-seanjc@google.com> Subject: [PATCH v3 30/40] KVM: x86: Move MSR helper declarations from kvm_host.h => msrs.h From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Relocate declarations of MSR helpers (and kvm_nr_uret_msrs) from x86's x86's kvm_host.h to msrs, to continue trimming down kvm_host.h. Deliberately leave the funky read_msr() where it is, as it will hopefully be removed entirely as part of a broader kernel-API cleanup. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang --- arch/x86/include/asm/kvm_host.h | 26 ------------------------- arch/x86/kvm/msrs.h | 34 ++++++++++++++++++++++++++++++--- 2 files changed, 31 insertions(+), 29 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index a861c0d70be0..1143140592df 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2094,7 +2094,6 @@ struct kvm_arch_async_pf { u64 error_code; }; =20 -extern u32 __read_mostly kvm_nr_uret_msrs; extern bool __read_mostly allow_smaller_maxphyaddr; extern bool __read_mostly enable_apicv; extern bool __read_mostly enable_ipiv; @@ -2278,18 +2277,6 @@ void kvm_prepare_emulation_failure_exit(struct kvm_v= cpu *vcpu); void kvm_prepare_event_vectoring_exit(struct kvm_vcpu *vcpu, gpa_t gpa); void kvm_prepare_unexpected_reason_exit(struct kvm_vcpu *vcpu, u64 exit_re= ason); =20 -void kvm_enable_efer_bits(u64); -bool kvm_valid_efer(struct kvm_vcpu *vcpu, u64 efer); -int kvm_emulate_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data); -int kvm_emulate_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data); -int __kvm_emulate_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data); -int __kvm_emulate_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data); -int kvm_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data); -int kvm_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data); -int kvm_emulate_rdmsr(struct kvm_vcpu *vcpu); -int kvm_emulate_rdmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg); -int kvm_emulate_wrmsr(struct kvm_vcpu *vcpu); -int kvm_emulate_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg); int kvm_emulate_as_nop(struct kvm_vcpu *vcpu); int kvm_emulate_invd(struct kvm_vcpu *vcpu); int kvm_emulate_mwait(struct kvm_vcpu *vcpu); @@ -2311,9 +2298,6 @@ int kvm_task_switch(struct kvm_vcpu *vcpu, u16 tss_se= lector, int idt_index, int __kvm_set_xcr(struct kvm_vcpu *vcpu, u32 index, u64 xcr); int kvm_emulate_xsetbv(struct kvm_vcpu *vcpu); =20 -int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr); -int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr); - int kvm_emulate_rdpmc(struct kvm_vcpu *vcpu); =20 void kvm_queue_exception(struct kvm_vcpu *vcpu, unsigned nr); @@ -2488,16 +2472,6 @@ int kvm_pv_send_ipi(struct kvm *kvm, unsigned long i= pi_bitmap_low, unsigned long ipi_bitmap_high, u32 min, unsigned long icr, int op_64_bit); =20 -int kvm_add_user_return_msr(u32 msr); -int kvm_find_user_return_msr(u32 msr); -int kvm_set_user_return_msr(unsigned index, u64 val, u64 mask); -u64 kvm_get_user_return_msr(unsigned int slot); - -static inline bool kvm_is_supported_user_return_msr(u32 msr) -{ - return kvm_find_user_return_msr(msr) >=3D 0; -} - u64 kvm_scale_tsc(u64 tsc, u64 ratio); u64 kvm_read_l1_tsc(struct kvm_vcpu *vcpu, u64 host_tsc); u64 kvm_calc_nested_tsc_offset(u64 l1_offset, u64 l2_offset, u64 l2_multip= lier); diff --git a/arch/x86/kvm/msrs.h b/arch/x86/kvm/msrs.h index c34f0411ced6..5c8362a8fd97 100644 --- a/arch/x86/kvm/msrs.h +++ b/arch/x86/kvm/msrs.h @@ -11,6 +11,8 @@ extern bool report_ignored_msrs; extern bool ignore_msrs; =20 +extern u32 __read_mostly kvm_nr_uret_msrs; + static inline void kvm_pr_unimpl_wrmsr(struct kvm_vcpu *vcpu, u32 msr, u64= data) { if (report_ignored_msrs) @@ -56,13 +58,39 @@ int kvm_get_set_one_reg(struct kvm_vcpu *vcpu, unsigned= int ioctl, int kvm_get_reg_list(struct kvm_vcpu *vcpu, struct kvm_reg_list __user *user_list); =20 +void kvm_enable_efer_bits(u64); +bool kvm_valid_efer(struct kvm_vcpu *vcpu, u64 efer); +int kvm_emulate_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data); +int kvm_emulate_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data); +int __kvm_emulate_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data); +int __kvm_emulate_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data); +int kvm_msr_read(struct kvm_vcpu *vcpu, u32 index, u64 *data); +int kvm_msr_write(struct kvm_vcpu *vcpu, u32 index, u64 data); +int kvm_emulate_rdmsr(struct kvm_vcpu *vcpu); +int kvm_emulate_rdmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg); +int kvm_emulate_wrmsr(struct kvm_vcpu *vcpu); +int kvm_emulate_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int reg); + +fastpath_t handle_fastpath_wrmsr(struct kvm_vcpu *vcpu); +fastpath_t handle_fastpath_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int r= eg); + +int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr); +int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr); + +int kvm_add_user_return_msr(u32 msr); +int kvm_find_user_return_msr(u32 msr); +int kvm_set_user_return_msr(unsigned index, u64 val, u64 mask); +u64 kvm_get_user_return_msr(unsigned int slot); + +static inline bool kvm_is_supported_user_return_msr(u32 msr) +{ + return kvm_find_user_return_msr(msr) >=3D 0; +} + void kvm_user_return_msr_cpu_online(void); void drop_user_return_notifiers(void); void kvm_destroy_user_return_msrs(void); =20 -fastpath_t handle_fastpath_wrmsr(struct kvm_vcpu *vcpu); -fastpath_t handle_fastpath_wrmsr_imm(struct kvm_vcpu *vcpu, u32 msr, int r= eg); - int kvm_emulator_get_msr_with_filter(struct kvm_vcpu *vcpu, u32 msr_index, u64 *pdata); int kvm_emulator_set_msr_with_filter(struct kvm_vcpu *vcpu, u32 msr_index, --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 864E547AF6A for ; Fri, 29 May 2026 22:23:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093403; cv=none; b=YT66GnEYz5n9H2fuHJGsDAWdy9AmUPOWXiO/jTWdRlk/SESoKT1ZjNbpw4Wdd14IIc/E8OJYj84xRKp5L/AVDem6jsWAKoNDhw279VPA5YuPUIofG7gXawybnA31licp6met3O68SHNM56DEaybKEte5MhQ1JHYl5NTqAJfSjkY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093403; c=relaxed/simple; bh=VRgMRhtjVBgYv8JI3E7pVGBY+7qN4gb9G0f1N7S5p40=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=mfKzY2Z1/UoNdYCvf/MqMq81xA/n26lG0dEDcGNutR/cOZtXjvd+WGDc2juduGdj6c5L/Q9aP+C99QR/O2Ndu4rnm0xgfpdqISlEUrP2aY4ybGPncLAu1h7eTIvGafM10yV2qo4S5o5t9wkHrkyy1UpwXx1icUH4JGm5vJvrsNc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=GgJSYYSO; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="GgJSYYSO" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-c827adbf00cso6643460a12.2 for ; Fri, 29 May 2026 15:23:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093401; x=1780698201; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=I88SBvpclb3TmLRZLMUOgTRAM1nkrimTOG6pLdh+qCU=; b=GgJSYYSOubVg4zFTvJ1NBy+h3cHgCqWM9FGd1jS1RogF/qE6TB2AqKFjEtu5y7vPrd vRgaMw/MmhiILgx+lOESGCw07u/O8EhpvTfDMzMOb7hpWTvaHVRuV2y42l1jUm9VJI5s L7I8XBqNRsyQPHa9m4StBv5fX17VGhZXac0T/Y7xTx2q6dfz23ot690xWm5UTwbV+Yu8 ZQRlFRBoS2m8iFevguzTt9b0JDqwhTLBzmLSGouoynufiFi9UoIO/73rA6Sg10NwFcpE VUaMLGtAz9BwV4ognyKWRLpi5en35N3HKen+zef0GB6FYe85idYUEIfI35TR1YlEFLCy kuxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093401; x=1780698201; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=I88SBvpclb3TmLRZLMUOgTRAM1nkrimTOG6pLdh+qCU=; b=Gzjbps6Of3fnMCUeIG2C+84usLg0WvOgrTGEqg6t0/eCV/qUUN4tTg4hFlCfEa6FGq URYR9Y5OQmBYKjBraNL3yT+FcM6d38bJ79F2DtnpMJQKRgyTKVYiRIWU71jN0d0LSJ48 b/BNdUz5UlNyhz4egLENgOxcMv1FHc8nXNJeJzxBhV+2rFykywu3hvcXncBZz05/Yqs4 W+91IplZsYzNWKv7aAyrci4sUmHewLtE0yrPfXkuQSmRgR3vVoIByJVDFOZt5tNpHzEX tMWuh4frQgHmzq7xDPDY1ILtFu1aQrulrqdp0iTFxSoKdfkksky8wdGLgw/YxV9fgQK+ 2oDw== X-Forwarded-Encrypted: i=1; AFNElJ+UvFSHObkp70v1dB0ygax8wosYQcic2rujoa08ojfaqFVyw+iTJOz4uy+gWaA0MsMWNextaxsB5qG6i8U=@vger.kernel.org X-Gm-Message-State: AOJu0YzmgW/8NcZWcje6jLijmEXP2hQ8mdg9b970bK0NU1IP2ZXthN9H 07cy9ICwZWDLLsJEoTWPR1MmhVYTsQXgKztDrrsEBvRzyODLaMzSCsfrYKtyPUVYeowKXQhZV4i z3qWfBw== X-Received: from pgbds10.prod.google.com ([2002:a05:6a02:430a:b0:c81:5509:27fc]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:a107:b0:398:c0ba:9ceb with SMTP id adf61e73a8af0-3b427c3f5b8mr1270452637.12.1780093400506; Fri, 29 May 2026 15:23:20 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:14 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-32-seanjc@google.com> Subject: [PATCH v3 31/40] KVM: x86: Move MMU helper declarations from kvm_host.h => mmu.h From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move a pile of MMU helper declarations into mmu.h, as they are very much KVM x86 internal APIs and not intended to be exposed to arch-neutral KVM, and certainly not to the broader kernel. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Reviewed-by: Yosry Ahmed --- arch/x86/include/asm/kvm_host.h | 70 -------------------------------- arch/x86/kvm/mmu.h | 71 +++++++++++++++++++++++++++++++++ 2 files changed, 71 insertions(+), 70 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 1143140592df..f217403e18fc 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -161,12 +161,6 @@ #define KVM_HPAGE_MASK(x) (~(KVM_HPAGE_SIZE(x) - 1)) #define KVM_PAGES_PER_HPAGE(x) (KVM_HPAGE_SIZE(x) / PAGE_SIZE) =20 -#define KVM_MEMSLOT_PAGES_TO_MMU_PAGES_RATIO 50 -#define KVM_MIN_ALLOC_MMU_PAGES 64UL -#define KVM_MMU_HASH_SHIFT 12 -#define KVM_NUM_MMU_PAGES (1 << KVM_MMU_HASH_SHIFT) -#define KVM_MIN_FREE_MMU_PAGES 5 -#define KVM_REFILL_PAGES 25 #define KVM_MAX_CPUID_ENTRIES 256 #define KVM_NR_VAR_MTRR 8 =20 @@ -2153,38 +2147,6 @@ enum kvm_intr_type { ((vcpu) && (vcpu)->arch.handling_intr_from_guest && \ (!!in_nmi() =3D=3D ((vcpu)->arch.handling_intr_from_guest =3D=3D KVM_HAN= DLING_NMI))) =20 -void __init kvm_mmu_x86_module_init(void); -int kvm_mmu_vendor_module_init(void); -void kvm_mmu_vendor_module_exit(void); - -void kvm_mmu_destroy(struct kvm_vcpu *vcpu); -int kvm_mmu_create(struct kvm_vcpu *vcpu); -int kvm_mmu_init_vm(struct kvm *kvm); -void kvm_mmu_uninit_vm(struct kvm *kvm); - -void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm, - struct kvm_memory_slot *slot); - -void kvm_mmu_after_set_cpuid(struct kvm_vcpu *vcpu); -void kvm_mmu_reset_context(struct kvm_vcpu *vcpu); -void kvm_mmu_slot_remove_write_access(struct kvm *kvm, - const struct kvm_memory_slot *memslot, - int start_level); -void kvm_mmu_slot_try_split_huge_pages(struct kvm *kvm, - const struct kvm_memory_slot *memslot, - int target_level); -void kvm_mmu_try_split_huge_pages(struct kvm *kvm, - const struct kvm_memory_slot *memslot, - u64 start, u64 end, - int target_level); -void kvm_mmu_recover_huge_pages(struct kvm *kvm, - const struct kvm_memory_slot *memslot); -void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, - const struct kvm_memory_slot *memslot); -void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen); -void kvm_mmu_change_mmu_pages(struct kvm *kvm, unsigned long kvm_nr_mmu_pa= ges); -void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end); - /* * EMULTYPE_NO_DECODE - Set when re-emulating an instruction (after comple= ting * userspace I/O) to indicate that the emulation context @@ -2334,25 +2296,6 @@ static inline int __kvm_irq_line_state(unsigned long= *irq_state, void kvm_inject_nmi(struct kvm_vcpu *vcpu); int kvm_get_nr_pending_nmis(struct kvm_vcpu *vcpu); =20 -bool __kvm_mmu_unprotect_gfn_and_retry(struct kvm_vcpu *vcpu, gpa_t cr2_or= _gpa, - bool always_retry); - -static inline bool kvm_mmu_unprotect_gfn_and_retry(struct kvm_vcpu *vcpu, - gpa_t cr2_or_gpa) -{ - return __kvm_mmu_unprotect_gfn_and_retry(vcpu, cr2_or_gpa, false); -} - -void kvm_mmu_free_roots(struct kvm *kvm, struct kvm_mmu *mmu, - ulong roots_to_free); -void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu); -gpa_t kvm_mmu_gva_to_gpa_read(struct kvm_vcpu *vcpu, gva_t gva, - struct x86_exception *exception); -gpa_t kvm_mmu_gva_to_gpa_write(struct kvm_vcpu *vcpu, gva_t gva, - struct x86_exception *exception); -gpa_t kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu, gva_t gva, - struct x86_exception *exception); - bool kvm_apicv_activated(struct kvm *kvm); bool kvm_vcpu_apicv_activated(struct kvm_vcpu *vcpu); void __kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu); @@ -2385,19 +2328,6 @@ static inline void kvm_dec_apicv_irq_window_req(stru= ct kvm *kvm) kvm_inc_or_dec_irq_window_inhibit(kvm, false); } =20 -int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_= code, - void *insn, int insn_len); -void kvm_mmu_print_sptes(struct kvm_vcpu *vcpu, gpa_t gpa, const char *msg= ); -void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva); -void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, - u64 addr, unsigned long roots); -void kvm_mmu_invpcid_gva(struct kvm_vcpu *vcpu, gva_t gva, unsigned long p= cid); -void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd); - -void kvm_configure_mmu(bool enable_tdp, int tdp_forced_root_level, - int tdp_max_root_level, int tdp_huge_page_level); - - #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES #define kvm_arch_has_private_mem(kvm) ((kvm)->arch.has_private_mem) #endif diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h index d30676935fff..a6b871253bd7 100644 --- a/arch/x86/kvm/mmu.h +++ b/arch/x86/kvm/mmu.h @@ -15,6 +15,13 @@ extern bool tdp_mmu_enabled; extern bool __read_mostly enable_mmio_caching; extern bool eager_page_split; =20 +#define KVM_MEMSLOT_PAGES_TO_MMU_PAGES_RATIO 50 +#define KVM_MIN_ALLOC_MMU_PAGES 64UL +#define KVM_MMU_HASH_SHIFT 12 +#define KVM_NUM_MMU_PAGES (1 << KVM_MMU_HASH_SHIFT) +#define KVM_MIN_FREE_MMU_PAGES 5 +#define KVM_REFILL_PAGES 25 + #define PT_WRITABLE_SHIFT 1 #define PT_USER_SHIFT 2 =20 @@ -96,6 +103,38 @@ static inline bool mmu_has_mbec(struct kvm_mmu *mmu) =20 u8 kvm_mmu_get_max_tdp_level(void); =20 +void __init kvm_mmu_x86_module_init(void); +int kvm_mmu_vendor_module_init(void); +void kvm_mmu_vendor_module_exit(void); + +void kvm_mmu_destroy(struct kvm_vcpu *vcpu); +int kvm_mmu_create(struct kvm_vcpu *vcpu); +int kvm_mmu_init_vm(struct kvm *kvm); +void kvm_mmu_uninit_vm(struct kvm *kvm); + +void kvm_mmu_init_memslot_memory_attributes(struct kvm *kvm, + struct kvm_memory_slot *slot); + +void kvm_mmu_after_set_cpuid(struct kvm_vcpu *vcpu); +void kvm_mmu_reset_context(struct kvm_vcpu *vcpu); +void kvm_mmu_slot_remove_write_access(struct kvm *kvm, + const struct kvm_memory_slot *memslot, + int start_level); +void kvm_mmu_slot_try_split_huge_pages(struct kvm *kvm, + const struct kvm_memory_slot *memslot, + int target_level); +void kvm_mmu_try_split_huge_pages(struct kvm *kvm, + const struct kvm_memory_slot *memslot, + u64 start, u64 end, + int target_level); +void kvm_mmu_recover_huge_pages(struct kvm *kvm, + const struct kvm_memory_slot *memslot); +void kvm_mmu_slot_leaf_clear_dirty(struct kvm *kvm, + const struct kvm_memory_slot *memslot); +void kvm_mmu_invalidate_mmio_sptes(struct kvm *kvm, u64 gen); +void kvm_mmu_change_mmu_pages(struct kvm *kvm, unsigned long kvm_nr_mmu_pa= ges); +void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end); + void kvm_mmu_set_mmio_spte_mask(u64 mmio_value, u64 mmio_mask, u64 access_= mask); void kvm_mmu_set_mmio_spte_value(struct kvm *kvm, u64 mmio_value); void kvm_mmu_set_me_spte_mask(u64 me_value, u64 me_mask); @@ -107,6 +146,19 @@ void kvm_init_shadow_npt_mmu(struct kvm_vcpu *vcpu, un= signed long cr4, void kvm_init_shadow_ept_mmu(struct kvm_vcpu *vcpu, bool execonly, int huge_page_level, bool accessed_dirty, bool mbec, gpa_t new_eptp); + +int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_= code, + void *insn, int insn_len); +void kvm_mmu_print_sptes(struct kvm_vcpu *vcpu, gpa_t gpa, const char *msg= ); +void kvm_mmu_invlpg(struct kvm_vcpu *vcpu, gva_t gva); +void kvm_mmu_invalidate_addr(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu, + u64 addr, unsigned long roots); +void kvm_mmu_invpcid_gva(struct kvm_vcpu *vcpu, gva_t gva, unsigned long p= cid); +void kvm_mmu_new_pgd(struct kvm_vcpu *vcpu, gpa_t new_pgd); + +void kvm_configure_mmu(bool enable_tdp, int tdp_forced_root_level, + int tdp_max_root_level, int tdp_huge_page_level); + bool kvm_can_do_async_pf(struct kvm_vcpu *vcpu); int kvm_handle_page_fault(struct kvm_vcpu *vcpu, u64 error_code, u64 fault_address, char *insn, int insn_len); @@ -121,6 +173,25 @@ void kvm_mmu_sync_prev_roots(struct kvm_vcpu *vcpu); void kvm_mmu_track_write(struct kvm_vcpu *vcpu, gpa_t gpa, const u8 *new, int bytes); =20 +bool __kvm_mmu_unprotect_gfn_and_retry(struct kvm_vcpu *vcpu, gpa_t cr2_or= _gpa, + bool always_retry); + +static inline bool kvm_mmu_unprotect_gfn_and_retry(struct kvm_vcpu *vcpu, + gpa_t cr2_or_gpa) +{ + return __kvm_mmu_unprotect_gfn_and_retry(vcpu, cr2_or_gpa, false); +} + +void kvm_mmu_free_roots(struct kvm *kvm, struct kvm_mmu *mmu, + ulong roots_to_free); +void kvm_mmu_free_guest_mode_roots(struct kvm *kvm, struct kvm_mmu *mmu); +gpa_t kvm_mmu_gva_to_gpa_read(struct kvm_vcpu *vcpu, gva_t gva, + struct x86_exception *exception); +gpa_t kvm_mmu_gva_to_gpa_write(struct kvm_vcpu *vcpu, gva_t gva, + struct x86_exception *exception); +gpa_t kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu, gva_t gva, + struct x86_exception *exception); + static inline int kvm_mmu_reload(struct kvm_vcpu *vcpu) { if (kvm_check_request(KVM_REQ_MMU_FREE_OBSOLETE_ROOTS, vcpu)) --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C170647B43D for ; Fri, 29 May 2026 22:23:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093404; cv=none; b=iidHfPbllbXI+b85W0CCB8NbLvYEJwD4v58VqZjqDq4QJ1yLzSFngtof9kWEqSVmNwhz8prjOiKq1mQOtp1zzibGG8x9qMjdcAZ/CILNNU658Obi53URPbqJ2VgrfF5OD/VqPQD1RryuS9MCY8Ws5z4gJ4Gv25AZv3oKKzLsyV8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093404; c=relaxed/simple; bh=L4ZZHusyWBmw7VMjY6K30cfDZe1TEcBzTRn7gHmkge4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=PHdII20Z/PDaecqmjINCOPjSKlIUZMfN48AIxwwGmVflvgJSyCYqc3+3iu00UDK6HVI52G+liHhVPwxJN6JTrncqu4PvCiSGMqlnMvJ996IrcVFVhENANo9pRyTymEcqYei13Y+e82PuZDbRQSAf9rvm/Ln9/z6q02wjlFicfGc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=lwCeJQvv; arc=none smtp.client-ip=209.85.216.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="lwCeJQvv" Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-36ba98cc003so1282842a91.1 for ; Fri, 29 May 2026 15:23:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093402; x=1780698202; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=GWLlWBc07Hx2by9RWraHiTLfxGizbsphDkdU8UthhCA=; b=lwCeJQvvk7Kzv+Llr+nCf2aWR3+21bPa84GDmpzXDkXbhImD+YFxJuInBX6uO+fGEn pHWWPthrzCiKDYZ/5EVbgsfn4cHgLKQEEnw5EA3+DfMSUxdMZgKS7D5NwWJRRiMEoP6p fdHkKyjuQ3c05JPkjjnztRCoyDft9altCjAXsXG2RZCDhPwlUtJp4AFfSDyYiTxUn0+K Pi66BeDTVp5rbf+F3ppymi4/HBDINYbuk1Nr2bphOBLDAxcER9Wae66wPVzcQR3b4iAz hPZ8NqfRaV+u7gymA/aXmrO/DKdIslycaKvvvZkz5aazGtA5B1FuVoD6hxfwVMh/qHSB l8NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093402; x=1780698202; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=GWLlWBc07Hx2by9RWraHiTLfxGizbsphDkdU8UthhCA=; b=Qve7iIpGWmnd11ByS9MzMAyIY6CvbgjxN2SFOSCTWaBeJBEqJtvZHJGm0DVqEdUhv3 NS2yS48k1jCZeZkyw67+/QtEeXUaUTxo8wMyCDn2uQXec7mf2C3lZAboGAo7/hXVJhKu SlCy65GkZtHXsCfUkSKijjsAT+ZeodBnMnGWVe2jOiOh1WSl9HnLO4SP6vjclBXgDnc+ 84EnkI5s6npK3lRoIj3ZA1JEyBsWA3jyQNV9cgCKss+c678qpJ97i2PJovObWbq6bAcm m5MvhQNvHIYOoCcWK6rH1xeoKWwQCbBrIa1yCtlVzqQA1UMe+/0hnpsFAuBejLWtt1ur TMbg== X-Forwarded-Encrypted: i=1; AFNElJ/WTYjekSxoHXXGUpA2a78pWfDB0WuB1NlQEu8tJrSIy2DhveN850ALYR97W8PBI/9VvtX/ldlSn9RlRqE=@vger.kernel.org X-Gm-Message-State: AOJu0YyB0WvekSA+4MFlMYbZKZQ0xS0MsVj6NigAZMTWg8P0Wxd9Vb4n uhO7GEWjLQXLbmB5xhihWlYbg/45A3zd+iTLyGbF05NA7wDKkjz3rzPk/hEawSqr8So0BasX6Ty K9iHF6g== X-Received: from pgvt11.prod.google.com ([2002:a65:64cb:0:b0:c82:743c:ea8f]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2f0f:b0:36a:95c:7613 with SMTP id 98e67ed59e1d1-36c4ff61422mr1094560a91.10.1780093401711; Fri, 29 May 2026 15:23:21 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:15 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-33-seanjc@google.com> Subject: [PATCH v3 32/40] KVM: x86: Move LLDT assembly wrappers into VMX From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move kvm_{load,read}_ldt() into vmx.c, as they are exclusively used by VMX to save/restore host state, and have no business being globally visible. No functional change intended. Signed-off-by: Sean Christopherson --- arch/x86/include/asm/kvm_host.h | 12 ------------ arch/x86/kvm/vmx/vmx.c | 12 ++++++++++++ 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index f217403e18fc..3e0884a862d7 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2334,18 +2334,6 @@ static inline void kvm_dec_apicv_irq_window_req(stru= ct kvm *kvm) =20 #define kvm_arch_has_readonly_mem(kvm) (!(kvm)->arch.has_protected_state) =20 -static inline u16 kvm_read_ldt(void) -{ - u16 ldt; - asm("sldt %0" : "=3Dg"(ldt)); - return ldt; -} - -static inline void kvm_load_ldt(u16 sel) -{ - asm("lldt %0" : : "rm"(sel)); -} - #ifdef CONFIG_X86_64 static inline unsigned long read_msr(unsigned long msr) { diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index bb19f6df921b..c8ff787d330d 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -1186,6 +1186,18 @@ static void vmx_remove_autostore_msr(struct vcpu_vmx= *vmx, u32 msr) vmx_remove_auto_msr(&vmx->msr_autostore, msr, VM_EXIT_MSR_STORE_COUNT); } =20 +static u16 kvm_read_ldt(void) +{ + u16 ldt; + asm("sldt %0" : "=3Dg"(ldt)); + return ldt; +} + +static void kvm_load_ldt(u16 sel) +{ + asm("lldt %0" : : "rm"(sel)); +} + #ifdef CONFIG_X86_32 /* * On 32-bit kernels, VM exits still load the FS and GS bases from the --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6296E47D928 for ; Fri, 29 May 2026 22:23:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093405; cv=none; b=DsRdpmMuS7HzIAIS1HQdmHKjv2QlxTBUSC3VRUnqpyhGMRqUHw3u6pQPZGaBVwsRL4iS96ImszaI6JR0wzTV9VHLpfUyeZtMmVk+xqx6drBvGH8LnY9DtcSEhif61iPbrwgMDeJHJFLICymlF6qxKYb2drcHkfcCVh6uGJZQRM8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093405; c=relaxed/simple; bh=sM71/V/NEGLDRFoPFaHWGc/4+j5RVrXjbHZgs3jZqHk=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=khOJpSAxStGfI54MngOZM0RZ7zPlnKAT8N0SuIdXa90hiExb6p+28djPTxrSyeMwsP+Bc/WTfC0gqXNmbGZCostEBsPmSVHKWjpHCGsy04jq2mxGf9gsMLzWPheccqSmqsXyzi4/w2zBg9qebnGYJ0tScGr4HSRWMK+A2K3YYGc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=eDP2fdxo; arc=none smtp.client-ip=209.85.215.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="eDP2fdxo" Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-bce224720d8so7367450a12.1 for ; Fri, 29 May 2026 15:23:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093403; x=1780698203; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=OYRO9/6QUt8JjFWwLPnwN9Nibat8wZa6YMCn9yp2By4=; b=eDP2fdxoM4xwN1SlmEZp+kV4Pgb0NKOqEFMMg0aOCVvJhKvgtvoQ7T0gBCna5DDiS3 zIan811BECT5C1gUUnH3PtDEOQOXwF9yWAyE+DBFLhO6iKtKXBym/ymkrEJ4N+f94XWi Qrbkq1oyn5kTn3tiGLQ0sNidumlZbgGmS1n8CW76YLymFsHnlcNkydsLPO01RVOwnHj8 aKARKSlN8ekNSkVtNGF8fRMP/wGjMjoTcz0jyMnLU2oIcIfxNG8DJSjQ0ULCNvS2u8oU cO5iBm3ELWxwUFQbcPpFVbF1KgUXFpdOc6JCCNhoaaI4/J6Aqq2x/NfuZq6ZqGXpuDdh lN2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093403; x=1780698203; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=OYRO9/6QUt8JjFWwLPnwN9Nibat8wZa6YMCn9yp2By4=; b=NsfAMjd/Pg2LP6e+rv8hJONQrtvcfBy+oVeEYd+uGxmCOm7L8E7kzlf6Hg4XjHe9Al otW1actv6YoR9tHHmxjp41DVaOP/AtAmwq+eUI2aYTTo+YpiHhshI/hUc/BGnbf6oSt+ DfygJJiIXmRu5yVjKlsgLNdGPN6l1lZYyHOeBw80IiN3S7fnfuL4szgXDBPfFGR6cBXj bYV6pDo3HMsNsgryGMdWAz8K66dL8BZUlckAww4bNOU+buiqWJmCkzVBPv2L0X+gPtfH cVrLBzkLO29oIxQL0sAKu7rzLhRqeV3dzE1HQBxLyU2KbJYSFC7N2LIrF3658QoQn12V lyaA== X-Forwarded-Encrypted: i=1; AFNElJ8xaaok6/fxAK8BwbJClwEdVPFHzUaYP8pqSt+IcKDNhMn3l+tq5bl7A3F4Ye8DIZ4ch+RFSSx6aad9C9c=@vger.kernel.org X-Gm-Message-State: AOJu0Yy5whV8dS2k+2/kM026EJlvFNDgnGYoxCd4BH3nvbvSsgY6qEYk 5sNLZRACdY8imX29Hl7YLFrKkkUjaMDQRjwMdV+x7SYP9PBZOgmcZlf3LvDO0aM+THNbdDJAoqI D2Kgb/g== X-Received: from pgbs189.prod.google.com ([2002:a63:5ec6:0:b0:c79:89d2:43e8]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:9f17:b0:399:12d:41f with SMTP id adf61e73a8af0-3b427be7b60mr1372617637.7.1780093403446; Fri, 29 May 2026 15:23:23 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:16 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-34-seanjc@google.com> Subject: [PATCH v3 33/40] KVM: x86: Move kvm_cpu_get_apicid() from kvm_host.h => avic.c From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Opportunistically drop the CONFIG_X86_LOCAL_APIC=3Dn stub, as KVM hard depends on CONFIG_X86_LOCAL_APIC=3Dy (the stub was there purely to deal with kvm_host.h being included by non-KVM code). No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Yosry Ahmed --- arch/x86/include/asm/kvm_host.h | 10 ---------- arch/x86/kvm/svm/avic.c | 5 +++++ 2 files changed, 5 insertions(+), 10 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 3e0884a862d7..989294c7501b 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2427,16 +2427,6 @@ static inline void kvm_arch_vcpu_unblocking(struct k= vm_vcpu *vcpu) kvm_x86_call(vcpu_unblocking)(vcpu); } =20 -static inline int kvm_cpu_get_apicid(int mps_cpu) -{ -#ifdef CONFIG_X86_LOCAL_APIC - return default_cpu_present_to_apicid(mps_cpu); -#else - WARN_ON_ONCE(1); - return BAD_APICID; -#endif -} - int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages); =20 #define KVM_CLOCK_VALID_FLAGS \ diff --git a/arch/x86/kvm/svm/avic.c b/arch/x86/kvm/svm/avic.c index b7083cd692ad..9264c8ef1fa1 100644 --- a/arch/x86/kvm/svm/avic.c +++ b/arch/x86/kvm/svm/avic.c @@ -146,6 +146,11 @@ static void avic_set_x2apic_msr_interception(struct vc= pu_svm *svm, svm->x2avic_msrs_intercepted =3D intercept; } =20 +static int kvm_cpu_get_apicid(int mps_cpu) +{ + return default_cpu_present_to_apicid(mps_cpu); +} + static u32 __avic_get_max_physical_id(struct kvm *kvm, struct kvm_vcpu *vc= pu) { u32 arch_max; --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3E1CB47D945 for ; Fri, 29 May 2026 22:23:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093406; cv=none; b=Fptj8KZDg34VnltFQzS7YcGgeerCR/QwdQmV5BXQ7ng5z+aMM2l19ff3jjIPQMyR3mRVsAoKF7cNck16mJiZVucjTm4EZ85nw6f8sbpyLVu3AuX8rSMfXoiHBtH8Yb1wET27u9yABVovbtDK9zxzJ+m4kHIPWlq7CjNk1UIZDyc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093406; c=relaxed/simple; bh=+s6xBOQ8qu6SI6nRP+HhTyt6JUEzgxZY18AyIiC69OM=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=MZhbauWb86ZOJbtwILXxWjAY/+HL8nN0eqLEXgtzIq7EZ7kxJvJiaJ23kEtaCflzPv3eKTQFEPlapG9fvBVzK5YM59AFBLJwTPe+5LVNgV26aYRQv21whCmd5MUiSy9kHG/2isUDIw/jNl+F3XbagYKrFpPo8cU8FB4Dtja6yKc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Z1nnn3Yw; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Z1nnn3Yw" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2ba15e384c7so103781605ad.3 for ; Fri, 29 May 2026 15:23:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093405; x=1780698205; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=B45QF1eRSPwRPsQdGnQfeoTY9Ry5cop0GiLPIJpG3/U=; b=Z1nnn3YwIxqdjHlSlrtrxVE0RUKvn9jiT2rrrXaNdGvCub/+GNPb9jAxnH4sumCO1W wwYD1xcMECImyoTLkCpJmLYAM8BqPiNHKewUH74OWJtF+NAB8sQsrAIya/Z2KLPBX698 vsPVi14g3qVQMMVcgzOCsn1nkVnNO1zi7VtWwgAbxcLFs1p7ktorAkM+ZS2PacL1lJez r9ngpvhXfij7zAxvxvfiJUnCZcY1RSQeZWf7WsqIgEBurOdzltb+Dc2hjg7NbhbuT81P 1CVqjMUq7PA/RdiLHdHGS80ZEAwdCmgAAX1Y+9ouS9uKASJXLnj2yzoojUq1sxcfJ5Po 2NkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093405; x=1780698205; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=B45QF1eRSPwRPsQdGnQfeoTY9Ry5cop0GiLPIJpG3/U=; b=WMC3hWsaRpucYrlRUjVuUQYbJfnMaoFV3ta+OqDw/+jhMGQXLsfjBtfsrh38/kbhyG R4fnE6fsSqbFTH+h7lMRzEGnfyK94vKkLKZ+Nl6oniXGWI8NEFfHOAogI8Cit0hgfnmy KMFEmOQrYqZWAWuA9QTrIyxYoTrR/PqD/2OUtzSblUDQ6Ykm1OaVlim5fzlJLJuJHFst VAc+u3AdF2Wg6t2vRECDiXyb+CWclLrbvHNNkwNGoFMLT0nbd+SoZEVcCezd7BTDODcs mXAf5hO1GOJcuq29ygXoSo+DCFUBA7fJzf9YRkpHriVAxm/n7DBUY79gVwxkX9Tu1w7r CsiQ== X-Forwarded-Encrypted: i=1; AFNElJ+nBuI4q+mW3yKq7Ydiakx3aJhYM5sotlMbHFhsLSbVE1nig+N7F7e+pVYPLkIB1ITRL/uPNbzrpYUWv8M=@vger.kernel.org X-Gm-Message-State: AOJu0YzL55EmcDveuNreSbsEWPXI3WyL8Sec2OgE32nUvIQZmqAUkX3e 1MuVoaRuByX8ig+PGxNrwYRu6MZ1TvdX7X0hRlHrJnkq+pd67E66fypCtr2v0LsuFTpw6w3OjJ2 n7ZYGZQ== X-Received: from plgv8.prod.google.com ([2002:a17:902:e8c8:b0:2bf:2f05:e721]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:3884:b0:2bf:33a9:bf4a with SMTP id d9443c01a7336-2bf368a3d92mr19960435ad.38.1780093404494; Fri, 29 May 2026 15:23:24 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:17 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-35-seanjc@google.com> Subject: [PATCH v3 34/40] KVM: x86: Move misc "VALID MASK" defines from kvm_host.h => x86.c From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move a variety of "VALID MASK" defines, e.g. that capture which flags in a given ioctl are supported by KVM, from kvm_host.h to x86.c. The set of valid flags/bits is very much a KVM-internal detail, as the hardcoded defines are often massaged at runtime, i.e. *directly* using the macros outside of KVM x86 would be actively dangerous. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Reviewed-by: Yosry Ahmed --- arch/x86/include/asm/kvm_host.h | 34 --------------------------------- arch/x86/kvm/x86.c | 33 ++++++++++++++++++++++++++++++++ 2 files changed, 33 insertions(+), 34 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 989294c7501b..53994b8292fc 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -78,12 +78,6 @@ #define KVM_DIRTY_LOG_MANUAL_CAPS (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE |= \ KVM_DIRTY_LOG_INITIALLY_SET) =20 -#define KVM_BUS_LOCK_DETECTION_VALID_MODE (KVM_BUS_LOCK_DETECTION_OFF | \ - KVM_BUS_LOCK_DETECTION_EXIT) - -#define KVM_X86_NOTIFY_VMEXIT_VALID_BITS (KVM_X86_NOTIFY_VMEXIT_ENABLED | \ - KVM_X86_NOTIFY_VMEXIT_USER) - /* x86-specific vcpu->requests bit members */ #define KVM_REQ_MIGRATE_TIMER KVM_ARCH_REQ(0) #define KVM_REQ_REPORT_TPR_ACCESS KVM_ARCH_REQ(1) @@ -2429,34 +2423,6 @@ static inline void kvm_arch_vcpu_unblocking(struct k= vm_vcpu *vcpu) =20 int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages); =20 -#define KVM_CLOCK_VALID_FLAGS \ - (KVM_CLOCK_TSC_STABLE | KVM_CLOCK_REALTIME | KVM_CLOCK_HOST_TSC) - -#define KVM_X86_VALID_QUIRKS \ - (KVM_X86_QUIRK_LINT0_REENABLED | \ - KVM_X86_QUIRK_CD_NW_CLEARED | \ - KVM_X86_QUIRK_LAPIC_MMIO_HOLE | \ - KVM_X86_QUIRK_OUT_7E_INC_RIP | \ - KVM_X86_QUIRK_MISC_ENABLE_NO_MWAIT | \ - KVM_X86_QUIRK_FIX_HYPERCALL_INSN | \ - KVM_X86_QUIRK_MWAIT_NEVER_UD_FAULTS | \ - KVM_X86_QUIRK_SLOT_ZAP_ALL | \ - KVM_X86_QUIRK_STUFF_FEATURE_MSRS | \ - KVM_X86_QUIRK_IGNORE_GUEST_PAT | \ - KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM | \ - KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT) - -#define KVM_X86_CONDITIONAL_QUIRKS \ - (KVM_X86_QUIRK_CD_NW_CLEARED | \ - KVM_X86_QUIRK_IGNORE_GUEST_PAT) - -/* - * KVM previously used a u32 field in kvm_run to indicate the hypercall was - * initiated from long mode. KVM now sets bit 0 to indicate long mode, but= the - * remaining 31 lower bits must be 0 to preserve ABI. - */ -#define KVM_EXIT_HYPERCALL_MBZ GENMASK_ULL(31, 1) - static inline bool kvm_arch_has_irq_bypass(void) { return enable_device_posted_irqs; diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 660ba27c76ec..a854b83e9881 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -105,6 +105,12 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_host); #define emul_to_vcpu(ctxt) \ ((struct kvm_vcpu *)(ctxt)->vcpu) =20 +/* + * KVM previously used a u32 field in kvm_run to indicate the hypercall was + * initiated from long mode. KVM now sets bit 0 to indicate long mode, but= the + * remaining 31 lower bits must be 0 to preserve ABI. + */ +#define KVM_EXIT_HYPERCALL_MBZ GENMASK_ULL(31, 1) #define KVM_EXIT_HYPERCALL_VALID_MASK (1 << KVM_HC_MAP_GPA_RANGE) =20 #define KVM_CAP_PMU_VALID_MASK KVM_PMU_CAP_DISABLE @@ -114,6 +120,33 @@ EXPORT_SYMBOL_FOR_KVM_INTERNAL(kvm_host); KVM_X2APIC_ENABLE_SUPPRESS_EOI_BROADCAST | \ KVM_X2APIC_DISABLE_SUPPRESS_EOI_BROADCAST) =20 +#define KVM_CLOCK_VALID_FLAGS \ + (KVM_CLOCK_TSC_STABLE | KVM_CLOCK_REALTIME | KVM_CLOCK_HOST_TSC) + +#define KVM_X86_VALID_QUIRKS \ + (KVM_X86_QUIRK_LINT0_REENABLED | \ + KVM_X86_QUIRK_CD_NW_CLEARED | \ + KVM_X86_QUIRK_LAPIC_MMIO_HOLE | \ + KVM_X86_QUIRK_OUT_7E_INC_RIP | \ + KVM_X86_QUIRK_MISC_ENABLE_NO_MWAIT | \ + KVM_X86_QUIRK_FIX_HYPERCALL_INSN | \ + KVM_X86_QUIRK_MWAIT_NEVER_UD_FAULTS | \ + KVM_X86_QUIRK_SLOT_ZAP_ALL | \ + KVM_X86_QUIRK_STUFF_FEATURE_MSRS | \ + KVM_X86_QUIRK_IGNORE_GUEST_PAT | \ + KVM_X86_QUIRK_VMCS12_ALLOW_FREEZE_IN_SMM | \ + KVM_X86_QUIRK_NESTED_SVM_SHARED_PAT) + +#define KVM_X86_CONDITIONAL_QUIRKS \ + (KVM_X86_QUIRK_CD_NW_CLEARED | \ + KVM_X86_QUIRK_IGNORE_GUEST_PAT) + +#define KVM_BUS_LOCK_DETECTION_VALID_MODE (KVM_BUS_LOCK_DETECTION_OFF | \ + KVM_BUS_LOCK_DETECTION_EXIT) + +#define KVM_X86_NOTIFY_VMEXIT_VALID_BITS (KVM_X86_NOTIFY_VMEXIT_ENABLED | \ + KVM_X86_NOTIFY_VMEXIT_USER) + static void process_nmi(struct kvm_vcpu *vcpu); static void store_regs(struct kvm_vcpu *vcpu); static int sync_regs(struct kvm_vcpu *vcpu); --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 70A1347DD57 for ; Fri, 29 May 2026 22:23:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093407; cv=none; b=Vv8BqL3sEp/0SqZV5NAi0Oq+QQfbogeKYexemcfnkOt6pY1I24z/NELCfmy1zWQILIgS8pzcoxvSeFB4xr/ohCJI6hznkSM9ArrSUfDn6sntTYi712X6eSCKOFp+mxsgjy74EV6d4NB1G229cw+gBVYDyWmPCUFlH9thP7NLNuo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093407; c=relaxed/simple; bh=cAVTA4X118KjTxfXzG4OtoKCmSjIFhw3tbSFgguW9S4=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=dP1rojhXCupa+g2XUzNI+0iAfFFWaaVYVp8IlRe+5xLXOkVkPPD6OSf7wpu9opVsa1uUZ5Tu/l4fOJmiBwxNKTNfShNwrjyg7Ae6WrLV/WDf24F8HO/ilvLLgRVgXgJInvuXQ5NIxcojlUpwovg7RnjbaZ0FZLok4dmjmLZfu2Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ryR36NoH; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ryR36NoH" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2bf0d79d41eso15849745ad.1 for ; Fri, 29 May 2026 15:23:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093406; x=1780698206; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=bZ/xu/SiWZyqJK5vauNXTcwVMA30rU7kWatOu0HtYnk=; b=ryR36NoHtsbmncAApR7IgfQlDl6RwVUsZlBGjToHDhyHWdaH6j0/uWD6W0Etsgj7ry wDdoJpd5Dx6i4yhV4IEZjBXguE3DJpYSikjTbiC2ANs2lTShPVHtkfen7dtw3kEFB3fE +h4+KFxqfy7QUr1+T0YPcG469YfR4HW0ycfyItReo/jE3tv4oh0bXAGtD8mpWQWASGbw lam8Vy9rgrA27P83oG/QfNPCbFkl8IrV/p8TBMOxDR/q65qmvz1wW747cxRgM+vpstPc 9psQ5xteP4bTXv25p6hSoYwzsIrJVzygnsk9XUDJkWOKBmDyV0WEeXW+dsoZL4NofkOo e1zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093406; x=1780698206; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=bZ/xu/SiWZyqJK5vauNXTcwVMA30rU7kWatOu0HtYnk=; b=LtVjG2evWOrjeM6m9slRxotud6yKImBxdzQz0RuG8nugPBEpQgAQ30UK41Ne6Ec6yt Nt6BCZVIuFBO2WPRpUFNxrIDotKzd/Ajn+5T/n+lI8OHiuDamaxHjU3IUnVhMsgbKFfF qkASuedX76x/HRyfGjWV0qtKAgogkLF22VniVQdFUY3qaCQZ3LeW+CwMV6BzSCNRaDR1 rI2Kxmt+uZYEaLPuSlLqaGR4tF8XHLQRoIqUM4i/7vYP9vWIUt5VsPBV+pHAwoeKkaLv GLYpCl93WOcNiXS0hlYofWDtXGVRQ0W0XVIj8iJhZlI7xG7nt3ySxBlmc30O97dydTZ+ EqCQ== X-Forwarded-Encrypted: i=1; AFNElJ8t6lzvvQjo7cJ8JsASYrYTftY+bsdb47UeSf+0+78s51trhjkneKpv4KrddDQTihL/kompHqgmP6LdWoE=@vger.kernel.org X-Gm-Message-State: AOJu0YxBE1K4oJNs3TtWqthQwM0FUWrpeY3rzwAb0L2OVrE72xFbtKJE CW6fswKuTSNvw1txsGs8uuYB7Hl8tvv+OFoP6hogVzkA5VNMLino/DTH21JHRY4t6IpHxypRs0m 84Xnf/A== X-Received: from plbmo16.prod.google.com ([2002:a17:903:a90:b0:2b2:5117:a3f]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:c949:b0:2be:1c3c:72bb with SMTP id d9443c01a7336-2bf36860a25mr21141965ad.33.1780093405558; Fri, 29 May 2026 15:23:25 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:18 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-36-seanjc@google.com> Subject: [PATCH v3 35/40] KVM: x86: Move __kvm_irq_line_state() from kvm_host.h => ioapic.h From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Bury __kvm_irq_line_state() in CONFIG_KVM_IOAPIC=3Dy code, as it's only used by PIC and I/O APIC code. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Reviewed-by: Yosry Ahmed --- arch/x86/include/asm/kvm_host.h | 12 ------------ arch/x86/kvm/ioapic.h | 12 ++++++++++++ 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 53994b8292fc..866d33abaee0 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2275,18 +2275,6 @@ static inline void kvm_inject_emulated_page_fault(st= ruct kvm_vcpu *vcpu, =20 bool kvm_require_dr(struct kvm_vcpu *vcpu, int dr); =20 -static inline int __kvm_irq_line_state(unsigned long *irq_state, - int irq_source_id, int level) -{ - /* Logical OR for level trig interrupt */ - if (level) - __set_bit(irq_source_id, irq_state); - else - __clear_bit(irq_source_id, irq_state); - - return !!(*irq_state); -} - void kvm_inject_nmi(struct kvm_vcpu *vcpu); int kvm_get_nr_pending_nmis(struct kvm_vcpu *vcpu); =20 diff --git a/arch/x86/kvm/ioapic.h b/arch/x86/kvm/ioapic.h index 3dadae093690..81b576513116 100644 --- a/arch/x86/kvm/ioapic.h +++ b/arch/x86/kvm/ioapic.h @@ -113,6 +113,18 @@ void kvm_get_ioapic(struct kvm *kvm, struct kvm_ioapic= _state *state); void kvm_set_ioapic(struct kvm *kvm, struct kvm_ioapic_state *state); void kvm_ioapic_scan_entry(struct kvm_vcpu *vcpu, ulong *ioapic_handled_vectors); + +static inline int __kvm_irq_line_state(unsigned long *irq_state, + int irq_source_id, int level) +{ + /* Logical OR for level trig interrupt */ + if (level) + __set_bit(irq_source_id, irq_state); + else + __clear_bit(irq_source_id, irq_state); + + return !!(*irq_state); +} #endif /* CONFIG_KVM_IOAPIC */ =20 static inline int ioapic_in_kernel(struct kvm *kvm) --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 52B8347B43D for ; Fri, 29 May 2026 22:23:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093408; cv=none; b=B7Ep2f2WzdxhbqDbPb+cJHPY3hHFQMaBTPM0dW4AiJatZQfXYO9d0QaRQK3aer49TGAWagt+vOEy9kc9oItbdRGRxVnWasG0CBLi+HI4oZ58smnkhGewVOUd45E1fhnzc9w3WW16ZWYtSnXVP+N2OrO1YwpajCNKciap2xKyspw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093408; c=relaxed/simple; bh=JpBvH7r8fr5j6fRXa8XHX2REUYUMDv45z0vOFjhJqfQ=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=dfa+y6AIdlytOcxFaMtRgQtzA9zsTycQMeHPWYzlM4RfJHgfIqFv6bkIuQoqroG715rP6CfXDCLS6ua7Qtl/Xh82f3ajAXzTZzJUIHxcBrzFbqMcGoQkYRkin81dgSED7jXcM77+HQBMyRsD3yP6WzZOlZ5ylOpb1Wdu8Rphhk0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ZCi6fiuF; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ZCi6fiuF" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2bf004bf8beso32262525ad.3 for ; Fri, 29 May 2026 15:23:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093407; x=1780698207; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=WdjQYB+bJBo3ge+pQ5WkXdfoYUD+JPPPdYiP6D03Qng=; b=ZCi6fiuF72mQJjmn9iNlGiVuy1WzhrXeV8bOIkyI+P4+vVyQLJnidjBh4LY7rh5fDg qAbVTCQRiR9SlxmiNIaqgYnTcVYkRebT+QPkxfTGOZusk/GePO0NcslCC/Gub/yPnCTU /oqfrTZRZERnVkBaGOfU09m8pkZvVGaiCwDTNGLotnBJAVMYSKzOB70AqpvisPRYs1bt qh532bwWJAk1GkVmJsMY86NK9hlTXg1VC8DgueHvwE3Bc3oQSgjBDSHlvsMWgOGev75t qFGdAeotr3uqP8UpXtyzgLuQtGp764nosbM4EFumzrO5K5Z7Vy1aPZ9iEkhmC6ekjsif 4N8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093407; x=1780698207; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=WdjQYB+bJBo3ge+pQ5WkXdfoYUD+JPPPdYiP6D03Qng=; b=fMyBIYF34KsE6dnG+4fdkfMcRAZLv90gJdBEmw4p9GIwVJQC4NqoAuelvG1gKEAv8M zf7R4J+VacXKoRLGT1g43ZmF/EI6glaJN1+tGaHxnURsrt4KvcQsTd6c5HpL5gY64sfT nBbtRqmVYFtsuS7TBFJiEW2JWdRERxr4uZkYlp73g6c+4Kq41W856uBILBEBNHORdt5N crtZljdXR66kHV0jkXsE2U3bUwU9622eCOYDtlTTlLnyazqXaKI6qN9XETbSrfpKdkVo 7qry3JlFeyE4f7GfQpNcvIoJGdEbPX7f3HwVxmvQtSIAczvG1o8hLewgG2i3cPM8Rfj5 /L5w== X-Forwarded-Encrypted: i=1; AFNElJ/O1tN4EI8GTpm2tFD5KBLdg9wWRzEeCVNJKeRckNU6Nj5qR15EroCHhZWFf8+P49W6AO9rw35eJRESCAY=@vger.kernel.org X-Gm-Message-State: AOJu0YyY4I5N6GPZ+Cz3nFHQ+dNCztaWwP3Q+N/mOYXlj+IbUPTLm2AK cYGFG66j6HBfMuWggoXShDJAr2lFmQz6jvUDvPicspJYNsSP2knOLDSPZJFXrut0NOKfQSd/DYn yh38unQ== X-Received: from plbkg4.prod.google.com ([2002:a17:903:604:b0:2b0:a8dc:303f]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:d4ce:b0:2ba:7749:f89a with SMTP id d9443c01a7336-2bf367a00fcmr20935015ad.11.1780093406587; Fri, 29 May 2026 15:23:26 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:19 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-37-seanjc@google.com> Subject: [PATCH v3 36/40] KVM: x86: Move IRQ-related helper declarations from kvm_host.h => irq.h From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move the function declaration for APIs to get/query pending IRQs from kvm_host.h to irq.h, as the APIs are only used by KVM x86 code. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Yosry Ahmed --- arch/x86/include/asm/kvm_host.h | 5 ----- arch/x86/kvm/irq.h | 6 ++++++ arch/x86/kvm/svm/nested.c | 1 + arch/x86/kvm/vmx/nested.c | 1 + 4 files changed, 8 insertions(+), 5 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 866d33abaee0..38de6c0dc743 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2360,12 +2360,7 @@ enum { # define kvm_memslots_for_spte_role(kvm, role) __kvm_memslots(kvm, 0) #endif =20 -int kvm_cpu_has_injectable_intr(struct kvm_vcpu *v); -int kvm_cpu_has_interrupt(struct kvm_vcpu *vcpu); -int kvm_cpu_has_extint(struct kvm_vcpu *v); int kvm_arch_interrupt_allowed(struct kvm_vcpu *vcpu); -int kvm_cpu_get_extint(struct kvm_vcpu *v); -int kvm_cpu_get_interrupt(struct kvm_vcpu *v); void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event); =20 int kvm_pv_send_ipi(struct kvm *kvm, unsigned long ipi_bitmap_low, diff --git a/arch/x86/kvm/irq.h b/arch/x86/kvm/irq.h index 34f4a78a7a01..1a84ea31e7fd 100644 --- a/arch/x86/kvm/irq.h +++ b/arch/x86/kvm/irq.h @@ -112,6 +112,12 @@ static inline int irqchip_in_kernel(struct kvm *kvm) return mode !=3D KVM_IRQCHIP_NONE; } =20 +int kvm_cpu_has_injectable_intr(struct kvm_vcpu *v); +int kvm_cpu_has_interrupt(struct kvm_vcpu *vcpu); +int kvm_cpu_has_extint(struct kvm_vcpu *v); +int kvm_cpu_get_extint(struct kvm_vcpu *v); +int kvm_cpu_get_interrupt(struct kvm_vcpu *v); + void kvm_inject_pending_timer_irqs(struct kvm_vcpu *vcpu); void kvm_inject_apic_timer_irqs(struct kvm_vcpu *vcpu); void kvm_apic_nmi_wd_deliver(struct kvm_vcpu *vcpu); diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 1ab8b95975a4..ef10298cb320 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -23,6 +23,7 @@ =20 #include "kvm_emulate.h" #include "trace.h" +#include "irq.h" #include "mmu.h" #include "x86.h" #include "smm.h" diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index b2c851cc7d5c..adbb8358ade7 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -11,6 +11,7 @@ #include "x86.h" #include "cpuid.h" #include "hyperv.h" +#include "irq.h" #include "mmu.h" #include "nested.h" #include "pmu.h" --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 868E347DFA7 for ; Fri, 29 May 2026 22:23:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093409; cv=none; b=ILte03VntbakTMnUnfIjTgsQxIYT+87whqLAQ1vkm0AUbpFOJe09uDS2XkmMeoVw9tSl0iJXqhsYCEGlB92Fd0CMLNjfZd3c6oFyzVyp2euFJ+/Gtd27dTDKvZ/Xjrr0ibv49qJ/nUOeO81h2DP5NBtYR48CA8hwrypoG7MnMHw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093409; c=relaxed/simple; bh=KgmvYQ4aIIIwB/uuTF147khWhwDE3Ci/RNN+FkKz8y8=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=gyO8Dpo1A7fa79exEjeWL+TExf4t4O6wERLKVO0LZ8KYUChoFVF5kE1WjkO2zkkO9QOAWViEunPDQHLhEG0Z4sklK1ejUpBZAQZE8OaIQULkjpRObSiAtG0GVEfkUJO9KEzQ5PoQ4Uh7+tk4T2xXM9sF6zWksl0tXwRY20UdjfM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=snUDUwwX; arc=none smtp.client-ip=209.85.214.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="snUDUwwX" Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2bd6aeb3637so318822735ad.2 for ; Fri, 29 May 2026 15:23:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093408; x=1780698208; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=EwR+3vho0VQrVi1Nz9jliESbfPqmEObd/3RZBDK6xpg=; b=snUDUwwXznHNttAIbiAeWb+yf6cniqSjcvHOAVyj6x/7yDXzoOPiSPIM2AX+4A8/Jo iml2L7+8VB9qwZlkhGGio+gTGkCYZSOTxut4FF+kcakCCkC9QSTwbTEeQZ/xhu66y2vz mY/5VXUfZNMEn7Gpk3ikTns3BoxI4gjqTkLg61r/ezA9pgGs6875jHYfGcvf9s+1jkpz syRONXE4ZvHNX3qaxnrS3/0TYA30PLcF7taOoOnodgCbSEw/7Vwbw5dt4M1z0QnKbxrF pslffISHGmflhP9fu0/Eawbt5WRayiv4WjPrO988cgK/loEYZiWRtm6sEoOEmGL141gz UiEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093408; x=1780698208; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=EwR+3vho0VQrVi1Nz9jliESbfPqmEObd/3RZBDK6xpg=; b=NNGPlvTFGnuELgl3ATwKA8dWZIDgOZGcMSjvvGdyfhcbuYrPV3JCYxAEjCY0HdaUld 8F8iAS4sFKaqeJSRtL5ckOPXaYhR1H+n8fTxAiFMK0H45O0+omG5iiouRFRYPJ1Ux2jK Jod/bcu1nwgL/w4I/6PGt2BRdqlU4v/owVPAQ+uJz7BYRfAhps4srwCeaSC4Bs93dLJ2 hL03mGmJjzB/jM80DpZEkmpO+fWGzh3Ye6Ey3qsdYFMfb+PZ7v7ufBpiV1ucpUQIrxhl 2uqNonI+8BJkoUZHnc1ye+BClOxkNF51JKVhOXGILs+G3B4EXyoI6nmRKdgw2v9adAFR Ew9A== X-Forwarded-Encrypted: i=1; AFNElJ8wZETN/zclLrl27OG7G3urnqUDAJkc0T4Mgi+ZAw8WuH7QW7FXom4maxwaA/05yKkY11UjJQQkjZ1NTiY=@vger.kernel.org X-Gm-Message-State: AOJu0YyfqVnAT/grh35eiSunNTaz8JOGub7TezqUqn16o7AWNKNkOLhW 6V6f1dVrt1dNzz1hGr9GS2DesfhVrLE5mVAG7Hm/iyO1trKELdrMev81qqi/H1C45dE+uHub9c2 qSP/vKQ== X-Received: from plxj8.prod.google.com ([2002:a17:902:da88:b0:2bf:10b7:41d7]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:902:ccc8:b0:2b4:5aff:de60 with SMTP id d9443c01a7336-2bf36833fefmr19350705ad.22.1780093407781; Fri, 29 May 2026 15:23:27 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:20 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-38-seanjc@google.com> Subject: [PATCH v3 37/40] KVM: x86: Move kvm_pv_send_ipi() declaration from kvm_host.h => lapic.h From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move the declaration of kvm_pv_send_ipi() into lapic.h, as its implementation is provided by lapic.c (sending PV IPIs relies on the optimized APIC map provided by the in-kernel local APIC), and it's only used by KVM x86 code. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Reviewed-by: Yosry Ahmed --- arch/x86/include/asm/kvm_host.h | 4 ---- arch/x86/kvm/lapic.h | 3 +++ 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index 38de6c0dc743..a3c1ff784e5c 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2363,10 +2363,6 @@ enum { int kvm_arch_interrupt_allowed(struct kvm_vcpu *vcpu); void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event); =20 -int kvm_pv_send_ipi(struct kvm *kvm, unsigned long ipi_bitmap_low, - unsigned long ipi_bitmap_high, u32 min, - unsigned long icr, int op_64_bit); - u64 kvm_scale_tsc(u64 tsc, u64 ratio); u64 kvm_read_l1_tsc(struct kvm_vcpu *vcpu, u64 host_tsc); u64 kvm_calc_nested_tsc_offset(u64 l1_offset, u64 l2_offset, u64 l2_multip= lier); diff --git a/arch/x86/kvm/lapic.h b/arch/x86/kvm/lapic.h index 32f09b25884a..58dbb94f980d 100644 --- a/arch/x86/kvm/lapic.h +++ b/arch/x86/kvm/lapic.h @@ -131,6 +131,9 @@ static inline int kvm_irq_delivery_to_apic(struct kvm *= kvm, } =20 void kvm_apic_send_ipi(struct kvm_lapic *apic, u32 icr_low, u32 icr_high); +int kvm_pv_send_ipi(struct kvm *kvm, unsigned long ipi_bitmap_low, + unsigned long ipi_bitmap_high, u32 min, + unsigned long icr, int op_64_bit); =20 int kvm_apic_set_base(struct kvm_vcpu *vcpu, u64 value, bool host_initiate= d); int kvm_apic_get_state(struct kvm_vcpu *vcpu, struct kvm_lapic_state *s); --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CA77047ECD1 for ; Fri, 29 May 2026 22:23:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.215.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093412; cv=none; b=a7vq2xmEbVBSvmQDuszO76zvd/3F7nanv48aDjyecPTu6CTBhe4ELtT+xb6TGRPfbJiHkDu8Ac0jlRSw/+FyR5N4nZ+s5qlWL1fCtOS3kEIaMa9qjhNGsRBGqp8Vd+aW3dd8GLxRr0wiY5ZmrdAJ2WxGHPfl59yVc8/liB49k7s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093412; c=relaxed/simple; bh=6wkYDhTrv2mLr2QDxNjAG/d6sB8lrG23VG9S3UDHwwU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=I2q/eiMXK7lqQnDH5Y5hHtNheLv2M2NwKdav4gPVwO9dkeoCeQ1DpbSKVjzk6Ctnc7nASNpiK1OHdL7+scLMIYqW/1zaBVJnGV9Ng35AkNhgEntTk8u2VP5FkyY5yRYwkpVcCApuw/8fxm7N9aiHnuXN8ndkSknmb8TFIPzKfTM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ReEWxTh/; arc=none smtp.client-ip=209.85.215.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ReEWxTh/" Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-c82ba4715b6so15295746a12.2 for ; Fri, 29 May 2026 15:23:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093409; x=1780698209; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=C5xIBF4dxThLK3TpNNRniIv9BT/2RVg6EsLkvI68oaI=; b=ReEWxTh/V2OZt66YIMn9VfbsVJOjO5Z6ON4u/k2KVfd81QdJ0Zx2QfyacdQGr6TfDY Z9yZPBFWOeMBBAXJ/hyFNC4rmQ1CSEt5kOoTnXgCDiehO47k1mKWEAHmHLDydnqt7gdU F/aEjeRyYfujegKoUppMpHIK1KkYbBxLHRUlwZ2AhJ1MAL6M/xOvHdPmw7ILZxy4HL+5 CQ9fKY7pX9GIOVqezIUIlYFrwZoedNakGwLXoUowXYw96A7zQV5a6kqYYRxL3KLJR9K5 EquF+hZnl+qm1h05YsfOwrGzzYD1qVROJmEdLsZ4gCidMK041A2K56e8g4nAS6+cV5Hw KbsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093409; x=1780698209; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=C5xIBF4dxThLK3TpNNRniIv9BT/2RVg6EsLkvI68oaI=; b=gXZbFfRffVtj9XLNXbUCMCnctIKYot8RClIQhpnN0aqbcXM7qh+NboRls8OeX8PS6k NNznT27h7xQ3boEcudSgCozoGMTzEfqq/Z4q7/UnYYx8q4eSg4WmVxOaJ9pDr2Th/adU YJpi4EeifpIyHzytTz0+VhAlHJ1swq77o0KGJxlNX1CVA/WLXiZVSOxFaDS42I5gTnrB XkVwQLKrJieYWXrZxj/Yyzuf32rMFE1zduyF+w/xuwfN3e+Rg6JwNCzOdOq3tnbm1CS0 ZWXYU8UwruL3oR+74GuhOgUewCiwsGk45Fx7RVeU21GxhdnD/N7R0kgKbfc43c8AL8OY lN3w== X-Forwarded-Encrypted: i=1; AFNElJ8GC1FbuQFcTvkddtzILjfbxxrzaMQtyI3479/WPZGi5QUNE5zaUq3v05AExnkwTP42Vjzut04OOKEbiEU=@vger.kernel.org X-Gm-Message-State: AOJu0YwsBFVvKOSQth3yRIg5Ute2ys63dcdN0nDwjgzTgyraqwSZ5hIx n6N0+jUyJhLAKzrdeNj178j+/nBJwJBuDGwvLHB5onCs3UlpG14c7/3+pK+4OIAE4CqWsJV3bI5 q4uCTxQ== X-Received: from pgum19.prod.google.com ([2002:a65:6a13:0:b0:c79:61dd:3232]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:7105:b0:3a8:9dd:75d5 with SMTP id adf61e73a8af0-3b4281d6c58mr1272726637.24.1780093408888; Fri, 29 May 2026 15:23:28 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:21 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-39-seanjc@google.com> Subject: [PATCH v3 38/40] KVM: x86/mmu: Move kvm_arch_async_page_ready() below kvm_tdp_page_fault() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move the implementation of kvm_arch_async_page_ready() "down" in mmu.c so that it lives below kvm_tdp_page_fault(). This will allow moving kvm_mmu_do_page_fault() into mmu.c without needing a forward declaration. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Reviewed-by: Yosry Ahmed --- arch/x86/kvm/mmu/mmu.c | 62 +++++++++++++++++++++--------------------- 1 file changed, 31 insertions(+), 31 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index e4d971d42f0e..f217e71c3af0 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4571,37 +4571,6 @@ static bool kvm_arch_setup_async_pf(struct kvm_vcpu = *vcpu, kvm_vcpu_gfn_to_hva(vcpu, fault->gfn), &arch); } =20 -void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf = *work) -{ - int r; - - if (WARN_ON_ONCE(work->arch.error_code & PFERR_PRIVATE_ACCESS)) - return; - - if ((vcpu->arch.mmu->root_role.direct !=3D work->arch.direct_map) || - work->wakeup_all) - return; - - r =3D kvm_mmu_reload(vcpu); - if (unlikely(r)) - return; - - if (!vcpu->arch.mmu->root_role.direct && - work->arch.cr3 !=3D kvm_mmu_get_guest_pgd(vcpu, vcpu->arch.mmu)) - return; - - r =3D kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, work->arch.error_code, - true, NULL, NULL); - - /* - * Account fixed page faults, otherwise they'll never be counted, but - * ignore stats for all other return times. Page-ready "faults" aren't - * truly spurious and never trigger emulation - */ - if (r =3D=3D RET_PF_FIXED) - vcpu->stat.pf_fixed++; -} - static void kvm_mmu_finish_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault, int r) { @@ -5058,6 +5027,37 @@ long kvm_arch_vcpu_pre_fault_memory(struct kvm_vcpu = *vcpu, return min(range->size, end - range->gpa); } =20 +void kvm_arch_async_page_ready(struct kvm_vcpu *vcpu, struct kvm_async_pf = *work) +{ + int r; + + if (WARN_ON_ONCE(work->arch.error_code & PFERR_PRIVATE_ACCESS)) + return; + + if ((vcpu->arch.mmu->root_role.direct !=3D work->arch.direct_map) || + work->wakeup_all) + return; + + r =3D kvm_mmu_reload(vcpu); + if (unlikely(r)) + return; + + if (!vcpu->arch.mmu->root_role.direct && + work->arch.cr3 !=3D kvm_mmu_get_guest_pgd(vcpu, vcpu->arch.mmu)) + return; + + r =3D kvm_mmu_do_page_fault(vcpu, work->cr2_or_gpa, work->arch.error_code, + true, NULL, NULL); + + /* + * Account fixed page faults, otherwise they'll never be counted, but + * ignore stats for all other return times. Page-ready "faults" aren't + * truly spurious and never trigger emulation + */ + if (r =3D=3D RET_PF_FIXED) + vcpu->stat.pf_fixed++; +} + #ifdef CONFIG_KVM_GUEST_MEMFD static void kvm_assert_gmem_invalidate_lock_held(struct kvm_memory_slot *s= lot) { --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E950D47DFB7 for ; Fri, 29 May 2026 22:23:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093412; cv=none; b=TFoBoPry9xNuQp38y0Y/tPyxuBy6daeTxiIHZnm6AU1Gkq6wSW9pcwuX7TFROHF5EkvUlhsUbxX+Dl99b75J9XyB9R2+7jUVaxW3B0e0EKJKCB0d5cg1mHPlvTvMupZMdn2mArYtPWgRHAoNO/Z6XMFyw9PmI03/qW55aj2zjPg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093412; c=relaxed/simple; bh=3ZYHUkj+oeXdrJoTlTtZXI4J3QdrnBWue6cHmIfCYIs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=X2PggCa3USS8kf7W3WlKeSPCYc3eq8Cq2QXd+7M+8slguPyw/vlTnKZTQR8VDK8FUjwrMhyzNwuqHyBXOZqukrJlpTl/CY3MF+AOQt80nlMEOJ8QKLVKs38FZ8s3BEEGtaApekT3F0/A3IrWgB1yIEDcl6jDMnkLGqKWw7cBvck= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=S7aiZImd; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="S7aiZImd" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-36642d2f4deso27144035a91.3 for ; Fri, 29 May 2026 15:23:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093410; x=1780698210; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=MH2p3W+wH0SgE7RS7gaDUYjHN6A3fmfpRXwInongdJ8=; b=S7aiZImdreHym8ZOuL/bM5HVkDWygCQzKsYCBAkhTXjqIkn/ddepRqcBjNZrJkEdnR yVnoXLmmRZw4Pq4lJkh/4dj1Jt+SHA/A1TyuyChNvgzit5GmBR98h7xosIBYIL7k8qnH NDWJf3UFuiHz8NlvwvlSIubPJuDVpXVk3gEHasTskpEN430nf8gT0AVosFnyh8LHZnP9 BMOnJ94R26dJ+gie/O2WSkxRmo0LD+gRYZuHKthN5sAoTrZO9ZxWR11OOl5OeJSUU8As 4suOMrLQpNIO0j6KDDpFn525wAzhxY+rsiwoqunXcQYxt5E3CeebYrXWgvW0mH52Zxbf 3/mA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093410; x=1780698210; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=MH2p3W+wH0SgE7RS7gaDUYjHN6A3fmfpRXwInongdJ8=; b=mzSl5ITJBCWutwSS18v0ynJcOvu0Otnh1lVbuSCLUHDf80x8vgEOZq/EUFOrxCvSjA UyxjbRIXSUUZY98wwYELPZqFMh95wbobV2/UlcB2jitkm/JS9ApFXSMMu+2q3zU7hNPD gm32HuRmo/Qa2fsDa86/ty6Jg65tko1OC8+/i4ZW/gEZnQ+bcMJhrx+jig6tT5FiXdnC A8+GTxAQyRBplWGL/U7K9eqcSFFdZPzMmzGlFvMOlMU/AzBEKaH9+t9TcaQkmvxnEj8+ +6njmSZU3pmv2ZgCK+7hcfSV1vttqXleLUO0PfAMrrVATsssb+CveobXIwZnIWEq3uRd wHPg== X-Forwarded-Encrypted: i=1; AFNElJ/38Lyu8H1bGh37vkMfx2EhKEePyS6cczP2Q7zLfjYm88X/TrjPD89Fut5eNWUMPuAsIuL+VpMwhE8ip7Q=@vger.kernel.org X-Gm-Message-State: AOJu0YxpLQZn5UuJ9IMLq6zSF3RkeRCPbYMA5tHuGmRCqi2BWF0BQlTI c4KZ1DftVaoQnLTaKdknvwI++c9YQYtNR15rtp6YuX62LmDjfovz5nRzg1mcvu1UFKx6g3+D110 5A8ODYQ== X-Received: from pgnp18.prod.google.com ([2002:a63:7f52:0:b0:c85:a9c:435f]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2d4d:b0:36b:a2cc:485b with SMTP id 98e67ed59e1d1-36c501f4cd3mr977167a91.21.1780093409985; Fri, 29 May 2026 15:23:29 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:22 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-40-seanjc@google.com> Subject: [PATCH v3 39/40] KVM: x86/mmu: Move kvm_mmu_do_page_fault() from mmu_internal.h => mmu.c From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move kvm_mmu_do_page_fault() into mmu.c, as there are no users outside of mmu.c, and the function typically isn't inlined by the compiler anyways. This will allow moving the EMULTYPE_xxx definitions into x86.h without having to include x86.h in mmu_internal.h, i.e. will help preserve the goal of making x86.h KVM x86's "top-level" include. No functional change intended. Signed-off-by: Sean Christopherson Reviewed-by: Kai Huang Reviewed-by: Yosry Ahmed --- arch/x86/kvm/mmu/mmu.c | 67 ++++++++++++++++++++++++++++++++- arch/x86/kvm/mmu/mmu_internal.h | 66 -------------------------------- 2 files changed, 66 insertions(+), 67 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index f217e71c3af0..2796230ec398 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -4927,7 +4927,7 @@ static int kvm_tdp_mmu_page_fault(struct kvm_vcpu *vc= pu, } #endif =20 -int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) +static int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault= *fault) { #ifdef CONFIG_X86_64 if (tdp_mmu_enabled) @@ -4937,6 +4937,71 @@ int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct= kvm_page_fault *fault) return direct_page_fault(vcpu, fault); } =20 +static int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, + u64 err, bool prefetch, int *emulation_type, + u8 *level) +{ + struct kvm_page_fault fault =3D { + .addr =3D cr2_or_gpa, + .error_code =3D err, + .exec =3D err & PFERR_FETCH_MASK, + .write =3D err & PFERR_WRITE_MASK, + .present =3D err & PFERR_PRESENT_MASK, + .rsvd =3D err & PFERR_RSVD_MASK, + .user =3D err & PFERR_USER_MASK, + .prefetch =3D prefetch, + .is_tdp =3D likely(vcpu->arch.mmu->page_fault =3D=3D kvm_tdp_page_fault), + .nx_huge_page_workaround_enabled =3D + is_nx_huge_page_enabled(vcpu->kvm), + + .max_level =3D KVM_MAX_HUGEPAGE_LEVEL, + .req_level =3D PG_LEVEL_4K, + .goal_level =3D PG_LEVEL_4K, + .is_private =3D err & PFERR_PRIVATE_ACCESS, + + .pfn =3D KVM_PFN_ERR_FAULT, + }; + int r; + + if (vcpu->arch.mmu->root_role.direct) { + /* + * Things like memslots don't understand the concept of a shared + * bit. Strip it so that the GFN can be used like normal, and the + * fault.addr can be used when the shared bit is needed. + */ + fault.gfn =3D gpa_to_gfn(fault.addr) & ~kvm_gfn_direct_bits(vcpu->kvm); + fault.slot =3D kvm_vcpu_gfn_to_memslot(vcpu, fault.gfn); + } + + /* + * With retpoline being active an indirect call is rather expensive, + * so do a direct call in the most common case. + */ + if (IS_ENABLED(CONFIG_MITIGATION_RETPOLINE) && fault.is_tdp) + r =3D kvm_tdp_page_fault(vcpu, &fault); + else + r =3D vcpu->arch.mmu->page_fault(vcpu, &fault); + + /* + * Not sure what's happening, but punt to userspace and hope that + * they can fix it by changing memory to shared, or they can + * provide a better error. + */ + if (r =3D=3D RET_PF_EMULATE && fault.is_private) { + pr_warn_ratelimited("kvm: unexpected emulation request on private memory= \n"); + kvm_mmu_prepare_memory_fault_exit(vcpu, &fault); + return -EFAULT; + } + + if (fault.write_fault_to_shadow_pgtable && emulation_type) + *emulation_type |=3D EMULTYPE_WRITE_PF_TO_SP; + if (level) + *level =3D fault.goal_level; + + return r; +} + + static int kvm_tdp_page_prefault(struct kvm_vcpu *vcpu, gpa_t gpa, u64 error_code, u8 *level) { diff --git a/arch/x86/kvm/mmu/mmu_internal.h b/arch/x86/kvm/mmu/mmu_interna= l.h index 73cdcbccc89e..c29002c60126 100644 --- a/arch/x86/kvm/mmu/mmu_internal.h +++ b/arch/x86/kvm/mmu/mmu_internal.h @@ -290,8 +290,6 @@ struct kvm_page_fault { bool write_fault_to_shadow_pgtable; }; =20 -int kvm_tdp_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault= ); - /* * Return values of handle_mmio_page_fault(), mmu.page_fault(), fast_page_= fault(), * and of course kvm_mmu_do_page_fault(). @@ -337,70 +335,6 @@ static inline void kvm_mmu_prepare_memory_fault_exit(s= truct kvm_vcpu *vcpu, fault->is_private); } =20 -static inline int kvm_mmu_do_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_o= r_gpa, - u64 err, bool prefetch, - int *emulation_type, u8 *level) -{ - struct kvm_page_fault fault =3D { - .addr =3D cr2_or_gpa, - .error_code =3D err, - .exec =3D err & PFERR_FETCH_MASK, - .write =3D err & PFERR_WRITE_MASK, - .present =3D err & PFERR_PRESENT_MASK, - .rsvd =3D err & PFERR_RSVD_MASK, - .user =3D err & PFERR_USER_MASK, - .prefetch =3D prefetch, - .is_tdp =3D likely(vcpu->arch.mmu->page_fault =3D=3D kvm_tdp_page_fault), - .nx_huge_page_workaround_enabled =3D - is_nx_huge_page_enabled(vcpu->kvm), - - .max_level =3D KVM_MAX_HUGEPAGE_LEVEL, - .req_level =3D PG_LEVEL_4K, - .goal_level =3D PG_LEVEL_4K, - .is_private =3D err & PFERR_PRIVATE_ACCESS, - - .pfn =3D KVM_PFN_ERR_FAULT, - }; - int r; - - if (vcpu->arch.mmu->root_role.direct) { - /* - * Things like memslots don't understand the concept of a shared - * bit. Strip it so that the GFN can be used like normal, and the - * fault.addr can be used when the shared bit is needed. - */ - fault.gfn =3D gpa_to_gfn(fault.addr) & ~kvm_gfn_direct_bits(vcpu->kvm); - fault.slot =3D kvm_vcpu_gfn_to_memslot(vcpu, fault.gfn); - } - - /* - * With retpoline being active an indirect call is rather expensive, - * so do a direct call in the most common case. - */ - if (IS_ENABLED(CONFIG_MITIGATION_RETPOLINE) && fault.is_tdp) - r =3D kvm_tdp_page_fault(vcpu, &fault); - else - r =3D vcpu->arch.mmu->page_fault(vcpu, &fault); - - /* - * Not sure what's happening, but punt to userspace and hope that - * they can fix it by changing memory to shared, or they can - * provide a better error. - */ - if (r =3D=3D RET_PF_EMULATE && fault.is_private) { - pr_warn_ratelimited("kvm: unexpected emulation request on private memory= \n"); - kvm_mmu_prepare_memory_fault_exit(vcpu, &fault); - return -EFAULT; - } - - if (fault.write_fault_to_shadow_pgtable && emulation_type) - *emulation_type |=3D EMULTYPE_WRITE_PF_TO_SP; - if (level) - *level =3D fault.goal_level; - - return r; -} - int kvm_mmu_max_mapping_level(struct kvm *kvm, struct kvm_page_fault *faul= t, const struct kvm_memory_slot *slot, gfn_t gfn); void kvm_mmu_hugepage_adjust(struct kvm_vcpu *vcpu, struct kvm_page_fault = *fault); --=20 2.54.0.823.g6e5bcc1fc9-goog From nobody Mon Jun 8 09:48:36 2026 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EA7D947F2D4 for ; Fri, 29 May 2026 22:23:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093414; cv=none; b=AbpvzotYEnkHW8DIW+HMKGA904khzR1Hp8jtNp9GEZajPfIx7guTdzB+0N1NeHTPgqkEpl3oSwyvUGyaYfjbKhPNqgYHVgpPBwQ2EPHhDxQMGaJw/WqrSz80aQzycA8WSBgT73tCM8wB9C/jAFmkjar3ZWo3DOVU/6DnRYP+k3U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780093414; c=relaxed/simple; bh=fdJmNJUoat0SYuIr/75lqwrLzad999r/nA5VcUzRK+E=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=sf+3IMlkgpuMXz3+wqusyaT75Tz3AlppjxeuawS2y0fD3ua8zkp3D43vVZJQdSGoqQ7jrOpp8HFvWatPGhwZW4WqsRCitmMytNfdyx4cGg3ko63quupoytqpTRZX0gY01k8n8md0fiON6aSi5zy5lQOWN/G3yjpHDpzuFbDOa2I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=a8gU17A6; arc=none smtp.client-ip=209.85.216.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--seanjc.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="a8gU17A6" Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-365d4d2fa04so13668508a91.3 for ; Fri, 29 May 2026 15:23:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1780093411; x=1780698211; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=dfsOABP1OessODfFIlBqL2zCE7gI1KoCdzMzK1bY8DI=; b=a8gU17A69QtRR9S9nlbeSkIGjoI4RkpQHBKqKPJCPfsgc938tSK4C1MeWARLiGifh4 kjB+Hqj+Rkq5GZsU8hv9tww3r/KS6bzA/vcDpb637+hADEpecXhIgxI7IPEmv9Jq4Xb3 LnetQKtKhI1diP6YKkXozsBp6XRbj/EX28eBSqcAIBGZiQpvWeXTWmAF3890+BavVOxL owI1fbo89wzXJ6LqznnPM+JTKmLflkp+TelFO2zWA/WRVhW98RiEnyhfk2n9H9QbaX/o g9eFn22hfa2Ulpueg5gdaULH1rdxqfKItq6ZNhO8gtoAzB6PFsgpausemslMz6nAns+8 8lIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1780093411; x=1780698211; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=dfsOABP1OessODfFIlBqL2zCE7gI1KoCdzMzK1bY8DI=; b=Q6JT1++nYGfHVX6qaE6oJyNNKbAUHmGtFZqw1v5J16Su0bAFY+nf4wgeOo6ULPCb5W Ny7dTLe7egD5VN9EY8GMugMAXZoBcP1eT7K7TGsw5YJPP5kLTWdkV60M8ykOLbqFjptt fosV6i97nYowPtDlYILMjnoTkUGguTA0Ak1i+UreSPwvFUV6zvAYPM7T5EYRRMZ/6gwX GEgDN82++uvKAkO3kZL2G8OGQHxUsmnIvijRwq2G7DcCle2Xx+uh6c8LYD0H6/VXrwqd iGPbMAKXp39OeV9VI6j4MKnKs+pomyvO1RtnURlM0e+iR3H3yU7w/nHHsAxzU8YKkpq3 kfNg== X-Forwarded-Encrypted: i=1; AFNElJ8+dC43LLsz3JObQrzB2jbFwT35JzfCerYtl6WHLV73eDo7nEgFOSfgy0N8yYtlba9TM33x6lA8IdvZars=@vger.kernel.org X-Gm-Message-State: AOJu0Ywp2Tb96J4++qFOjgSke8tvDUt33vFjLzchlIKtuCIj4W9B2vLd aafwl6yfv6FGdlb07l/uSZkqF4c3MKqlBq5ixoR4A5z5gjUfIbuaRIJHk0jQEHAwoXH5Lmx06on Ytx285A== X-Received: from pgww25.prod.google.com ([2002:a05:6a02:2c99:b0:c85:6d7c:2c15]) (user=seanjc job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:d492:b0:398:7d4a:c2c4 with SMTP id adf61e73a8af0-3b428303a0emr1259457637.48.1780093411109; Fri, 29 May 2026 15:23:31 -0700 (PDT) Reply-To: Sean Christopherson Date: Fri, 29 May 2026 15:22:23 -0700 In-Reply-To: <20260529222223.870923-1-seanjc@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260529222223.870923-1-seanjc@google.com> X-Mailer: git-send-email 2.54.0.823.g6e5bcc1fc9-goog Message-ID: <20260529222223.870923-41-seanjc@google.com> Subject: [PATCH v3 40/40] KVM: x86: Move a pile of stuff from kvm_host.h => x86.h From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini , Vitaly Kuznetsov , David Woodhouse , Paul Durrant Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Binbin Wu , David Woodhouse , Kai Huang Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move the majority of remaining KVM-internal declarations and defines in kvm_host.h to x86.h, so that kvm_host.h only holds structure and function definitions that need to be visible to arch-neutral KVM. Land the emulator interfaces in x86.h, even though kvm_emulate.h *seems* like a good home, as the interfaces and defines being moved are provided by x86.c. I.e. keep kvm_emulate.h as an interface to the emulator proper. Note, any "misses" are likely unintentional. Signed-off-by: Sean Christopherson --- arch/x86/include/asm/kvm_host.h | 197 -------------------------------- arch/x86/kvm/ioapic.c | 1 + arch/x86/kvm/x86.h | 195 +++++++++++++++++++++++++++++++ 3 files changed, 196 insertions(+), 197 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_hos= t.h index a3c1ff784e5c..4efad65e53e2 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2096,9 +2096,6 @@ extern struct kvm_x86_ops kvm_x86_ops; #define KVM_X86_OP_OPTIONAL_RET0 KVM_X86_OP #include =20 -int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops); -void kvm_x86_vendor_exit(void); - #define __KVM_HAVE_ARCH_VM_ALLOC static inline struct kvm *kvm_arch_alloc_vm(void) { @@ -2141,175 +2138,6 @@ enum kvm_intr_type { ((vcpu) && (vcpu)->arch.handling_intr_from_guest && \ (!!in_nmi() =3D=3D ((vcpu)->arch.handling_intr_from_guest =3D=3D KVM_HAN= DLING_NMI))) =20 -/* - * EMULTYPE_NO_DECODE - Set when re-emulating an instruction (after comple= ting - * userspace I/O) to indicate that the emulation context - * should be reused as is, i.e. skip initialization of - * emulation context, instruction fetch and decode. - * - * EMULTYPE_TRAP_UD - Set when emulating an intercepted #UD from hardware. - * Indicates that only select instructions (tagged with - * EmulateOnUD) should be emulated (to minimize the emulator - * attack surface). See also EMULTYPE_TRAP_UD_FORCED. - * - * EMULTYPE_SKIP - Set when emulating solely to skip an instruction, i.e. = to - * decode the instruction length. For use *only* by - * kvm_x86_ops.skip_emulated_instruction() implementations if - * EMULTYPE_COMPLETE_USER_EXIT is not set. - * - * EMULTYPE_ALLOW_RETRY_PF - Set when the emulator should resume the guest= to - * retry native execution under certain conditions, - * Can only be set in conjunction with EMULTYPE_PF. - * - * EMULTYPE_TRAP_UD_FORCED - Set when emulating an intercepted #UD that was - * triggered by KVM's magic "force emulation" prefix, - * which is opt in via module param (off by default). - * Bypasses EmulateOnUD restriction despite emulating - * due to an intercepted #UD (see EMULTYPE_TRAP_UD). - * Used to test the full emulator from userspace. - * - * EMULTYPE_VMWARE_GP - Set when emulating an intercepted #GP for VMware - * backdoor emulation, which is opt in via module param. - * VMware backdoor emulation handles select instructions - * and reinjects the #GP for all other cases. - * - * EMULTYPE_PF - Set when an intercepted #PF triggers the emulation, in wh= ich case - * the CR2/GPA value pass on the stack is valid. - * - * EMULTYPE_COMPLETE_USER_EXIT - Set when the emulator should update inter= ruptibility - * state and inject single-step #DBs after skipping - * an instruction (after completing userspace I/O). - * - * EMULTYPE_WRITE_PF_TO_SP - Set when emulating an intercepted page fault = that - * is attempting to write a gfn that contains one or - * more of the PTEs used to translate the write itself, - * and the owning page table is being shadowed by KVM. - * If emulation of the faulting instruction fails and - * this flag is set, KVM will exit to userspace instead - * of retrying emulation as KVM cannot make forward - * progress. - * - * If emulation fails for a write to guest page tables, - * KVM unprotects (zaps) the shadow page for the target - * gfn and resumes the guest to retry the non-emulatable - * instruction (on hardware). Unprotecting the gfn - * doesn't allow forward progress for a self-changing - * access because doing so also zaps the translation for - * the gfn, i.e. retrying the instruction will hit a - * !PRESENT fault, which results in a new shadow page - * and sends KVM back to square one. - * - * EMULTYPE_SKIP_SOFT_INT - Set in combination with EMULTYPE_SKIP to only = skip - * an instruction if it could generate a given so= ftware - * interrupt, which must be encoded via - * EMULTYPE_SET_SOFT_INT_VECTOR(). - */ -#define EMULTYPE_NO_DECODE (1 << 0) -#define EMULTYPE_TRAP_UD (1 << 1) -#define EMULTYPE_SKIP (1 << 2) -#define EMULTYPE_ALLOW_RETRY_PF (1 << 3) -#define EMULTYPE_TRAP_UD_FORCED (1 << 4) -#define EMULTYPE_VMWARE_GP (1 << 5) -#define EMULTYPE_PF (1 << 6) -#define EMULTYPE_COMPLETE_USER_EXIT (1 << 7) -#define EMULTYPE_WRITE_PF_TO_SP (1 << 8) -#define EMULTYPE_SKIP_SOFT_INT (1 << 9) - -#define EMULTYPE_SET_SOFT_INT_VECTOR(v) ((u32)((v) & 0xff) << 16) -#define EMULTYPE_GET_SOFT_INT_VECTOR(e) (((e) >> 16) & 0xff) - -static inline bool kvm_can_emulate_event_vectoring(int emul_type) -{ - return !(emul_type & EMULTYPE_PF); -} - -int kvm_emulate_instruction(struct kvm_vcpu *vcpu, int emulation_type); -int kvm_emulate_instruction_from_buffer(struct kvm_vcpu *vcpu, - void *insn, int insn_len); -void __kvm_prepare_emulation_failure_exit(struct kvm_vcpu *vcpu, - u64 *data, u8 ndata); -void kvm_prepare_emulation_failure_exit(struct kvm_vcpu *vcpu); - -void kvm_prepare_event_vectoring_exit(struct kvm_vcpu *vcpu, gpa_t gpa); -void kvm_prepare_unexpected_reason_exit(struct kvm_vcpu *vcpu, u64 exit_re= ason); - -int kvm_emulate_as_nop(struct kvm_vcpu *vcpu); -int kvm_emulate_invd(struct kvm_vcpu *vcpu); -int kvm_emulate_mwait(struct kvm_vcpu *vcpu); -int kvm_handle_invalid_op(struct kvm_vcpu *vcpu); -int kvm_emulate_monitor(struct kvm_vcpu *vcpu); - -int kvm_fast_pio(struct kvm_vcpu *vcpu, int size, unsigned short port, int= in); -int kvm_emulate_cpuid(struct kvm_vcpu *vcpu); -int kvm_emulate_halt(struct kvm_vcpu *vcpu); -int kvm_emulate_halt_noskip(struct kvm_vcpu *vcpu); -int kvm_emulate_ap_reset_hold(struct kvm_vcpu *vcpu); -int kvm_emulate_wbinvd(struct kvm_vcpu *vcpu); - -void kvm_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector); - -int kvm_task_switch(struct kvm_vcpu *vcpu, u16 tss_selector, int idt_index, - int reason, bool has_error_code, u32 error_code); - -int __kvm_set_xcr(struct kvm_vcpu *vcpu, u32 index, u64 xcr); -int kvm_emulate_xsetbv(struct kvm_vcpu *vcpu); - -int kvm_emulate_rdpmc(struct kvm_vcpu *vcpu); - -void kvm_queue_exception(struct kvm_vcpu *vcpu, unsigned nr); -void kvm_queue_exception_e(struct kvm_vcpu *vcpu, unsigned nr, u32 error_c= ode); -void kvm_queue_exception_p(struct kvm_vcpu *vcpu, unsigned nr, unsigned lo= ng payload); -void kvm_requeue_exception(struct kvm_vcpu *vcpu, unsigned int nr, - bool has_error_code, u32 error_code); -void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fa= ult, - bool from_hardware); -void __kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, - struct x86_exception *fault, - bool from_hardware); - -static inline void kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, - struct x86_exception *fault) -{ - __kvm_inject_emulated_page_fault(vcpu, fault, false); -} - -bool kvm_require_dr(struct kvm_vcpu *vcpu, int dr); - -void kvm_inject_nmi(struct kvm_vcpu *vcpu); -int kvm_get_nr_pending_nmis(struct kvm_vcpu *vcpu); - -bool kvm_apicv_activated(struct kvm *kvm); -bool kvm_vcpu_apicv_activated(struct kvm_vcpu *vcpu); -void __kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu); -void __kvm_set_or_clear_apicv_inhibit(struct kvm *kvm, - enum kvm_apicv_inhibit reason, bool set); -void kvm_set_or_clear_apicv_inhibit(struct kvm *kvm, - enum kvm_apicv_inhibit reason, bool set); - -static inline void kvm_set_apicv_inhibit(struct kvm *kvm, - enum kvm_apicv_inhibit reason) -{ - kvm_set_or_clear_apicv_inhibit(kvm, reason, true); -} - -static inline void kvm_clear_apicv_inhibit(struct kvm *kvm, - enum kvm_apicv_inhibit reason) -{ - kvm_set_or_clear_apicv_inhibit(kvm, reason, false); -} - -void kvm_inc_or_dec_irq_window_inhibit(struct kvm *kvm, bool inc); - -static inline void kvm_inc_apicv_irq_window_req(struct kvm *kvm) -{ - kvm_inc_or_dec_irq_window_inhibit(kvm, true); -} - -static inline void kvm_dec_apicv_irq_window_req(struct kvm *kvm) -{ - kvm_inc_or_dec_irq_window_inhibit(kvm, false); -} - #ifdef CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES #define kvm_arch_has_private_mem(kvm) ((kvm)->arch.has_private_mem) #endif @@ -2326,11 +2154,6 @@ static inline unsigned long read_msr(unsigned long m= sr) } #endif =20 -static inline void kvm_inject_gp(struct kvm_vcpu *vcpu, u32 error_code) -{ - kvm_queue_exception_e(vcpu, GP_VECTOR, error_code); -} - #define TSS_IOPB_BASE_OFFSET 0x66 #define TSS_BASE_SIZE 0x68 #define TSS_IOPB_SIZE (65536 / 8) @@ -2361,16 +2184,6 @@ enum { #endif =20 int kvm_arch_interrupt_allowed(struct kvm_vcpu *vcpu); -void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event); - -u64 kvm_scale_tsc(u64 tsc, u64 ratio); -u64 kvm_read_l1_tsc(struct kvm_vcpu *vcpu, u64 host_tsc); -u64 kvm_calc_nested_tsc_offset(u64 l1_offset, u64 l2_offset, u64 l2_multip= lier); -u64 kvm_calc_nested_tsc_multiplier(u64 l1_multiplier, u64 l2_multiplier); - -void kvm_make_scan_ioapic_request(struct kvm *kvm); -void kvm_make_scan_ioapic_request_mask(struct kvm *kvm, - unsigned long *vcpu_bitmap); =20 bool kvm_arch_async_page_not_present(struct kvm_vcpu *vcpu, struct kvm_async_pf *work); @@ -2382,14 +2195,6 @@ void kvm_arch_async_page_present_queued(struct kvm_v= cpu *vcpu); bool kvm_arch_can_dequeue_async_page_present(struct kvm_vcpu *vcpu); extern bool kvm_find_async_pf_gfn(struct kvm_vcpu *vcpu, gfn_t gfn); =20 -int kvm_skip_emulated_instruction(struct kvm_vcpu *vcpu); -int kvm_complete_insn_gp(struct kvm_vcpu *vcpu, int err); - -void __user *__x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa, - u32 size); -bool kvm_vcpu_is_reset_bsp(struct kvm_vcpu *vcpu); -bool kvm_vcpu_is_bsp(struct kvm_vcpu *vcpu); - static inline void kvm_arch_vcpu_blocking(struct kvm_vcpu *vcpu) { kvm_x86_call(vcpu_blocking)(vcpu); @@ -2400,8 +2205,6 @@ static inline void kvm_arch_vcpu_unblocking(struct kv= m_vcpu *vcpu) kvm_x86_call(vcpu_unblocking)(vcpu); } =20 -int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages); - static inline bool kvm_arch_has_irq_bypass(void) { return enable_device_posted_irqs; diff --git a/arch/x86/kvm/ioapic.c b/arch/x86/kvm/ioapic.c index eed96ff6e722..b114f646d798 100644 --- a/arch/x86/kvm/ioapic.c +++ b/arch/x86/kvm/ioapic.c @@ -33,6 +33,7 @@ #include "lapic.h" #include "irq.h" #include "trace.h" +#include "x86.h" =20 static int ioapic_service(struct kvm_ioapic *vioapic, int irq, bool line_status); diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index fd3d0a196526..9db1f663e5b1 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -14,6 +14,9 @@ =20 #define KVM_MAX_MCE_BANKS 32 =20 +int kvm_x86_vendor_init(struct kvm_x86_init_ops *ops); +void kvm_x86_vendor_exit(void); + void kvm_spurious_fault(void); =20 #define SIZE_OF_MEMSLOTS_HASHTABLE \ @@ -318,6 +321,8 @@ static __always_inline void kvm_request_l1tf_flush_l1d(= void) #endif } =20 +void kvm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event); + void kvm_inject_realmode_interrupt(struct kvm_vcpu *vcpu, int irq, int inc= _eip); =20 u64 get_kvmclock_ns(struct kvm *kvm); @@ -326,6 +331,10 @@ bool kvm_get_monotonic_and_clockread(s64 *kernel_ns, u= 64 *tsc_timestamp); int kvm_guest_time_update(struct kvm_vcpu *v); =20 void kvm_synchronize_tsc(struct kvm_vcpu *vcpu, u64 *user_value); +u64 kvm_scale_tsc(u64 tsc, u64 ratio); +u64 kvm_read_l1_tsc(struct kvm_vcpu *vcpu, u64 host_tsc); +u64 kvm_calc_nested_tsc_offset(u64 l1_offset, u64 l2_offset, u64 l2_multip= lier); +u64 kvm_calc_nested_tsc_multiplier(u64 l1_multiplier, u64 l2_multiplier); u64 kvm_compute_l1_tsc_offset(struct kvm_vcpu *vcpu, u64 target_tsc); void kvm_vcpu_write_tsc_offset(struct kvm_vcpu *vcpu, u64 l1_offset); =20 @@ -363,10 +372,196 @@ int x86_decode_emulated_instruction(struct kvm_vcpu = *vcpu, int emulation_type, void *insn, int insn_len); int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, int emulation_type, void *insn, int insn_len); +/* + * EMULTYPE_NO_DECODE - Set when re-emulating an instruction (after comple= ting + * userspace I/O) to indicate that the emulation context + * should be reused as is, i.e. skip initialization of + * emulation context, instruction fetch and decode. + * + * EMULTYPE_TRAP_UD - Set when emulating an intercepted #UD from hardware. + * Indicates that only select instructions (tagged with + * EmulateOnUD) should be emulated (to minimize the emulator + * attack surface). See also EMULTYPE_TRAP_UD_FORCED. + * + * EMULTYPE_SKIP - Set when emulating solely to skip an instruction, i.e. = to + * decode the instruction length. For use *only* by + * kvm_x86_ops.skip_emulated_instruction() implementations if + * EMULTYPE_COMPLETE_USER_EXIT is not set. + * + * EMULTYPE_ALLOW_RETRY_PF - Set when the emulator should resume the guest= to + * retry native execution under certain conditions, + * Can only be set in conjunction with EMULTYPE_PF. + * + * EMULTYPE_TRAP_UD_FORCED - Set when emulating an intercepted #UD that was + * triggered by KVM's magic "force emulation" prefix, + * which is opt in via module param (off by default). + * Bypasses EmulateOnUD restriction despite emulating + * due to an intercepted #UD (see EMULTYPE_TRAP_UD). + * Used to test the full emulator from userspace. + * + * EMULTYPE_VMWARE_GP - Set when emulating an intercepted #GP for VMware + * backdoor emulation, which is opt in via module param. + * VMware backdoor emulation handles select instructions + * and reinjects the #GP for all other cases. + * + * EMULTYPE_PF - Set when an intercepted #PF triggers the emulation, in wh= ich case + * the CR2/GPA value pass on the stack is valid. + * + * EMULTYPE_COMPLETE_USER_EXIT - Set when the emulator should update inter= ruptibility + * state and inject single-step #DBs after skipping + * an instruction (after completing userspace I/O). + * + * EMULTYPE_WRITE_PF_TO_SP - Set when emulating an intercepted page fault = that + * is attempting to write a gfn that contains one or + * more of the PTEs used to translate the write itself, + * and the owning page table is being shadowed by KVM. + * If emulation of the faulting instruction fails and + * this flag is set, KVM will exit to userspace instead + * of retrying emulation as KVM cannot make forward + * progress. + * + * If emulation fails for a write to guest page tables, + * KVM unprotects (zaps) the shadow page for the target + * gfn and resumes the guest to retry the non-emulatable + * instruction (on hardware). Unprotecting the gfn + * doesn't allow forward progress for a self-changing + * access because doing so also zaps the translation for + * the gfn, i.e. retrying the instruction will hit a + * !PRESENT fault, which results in a new shadow page + * and sends KVM back to square one. + * + * EMULTYPE_SKIP_SOFT_INT - Set in combination with EMULTYPE_SKIP to only = skip + * an instruction if it could generate a given so= ftware + * interrupt, which must be encoded via + * EMULTYPE_SET_SOFT_INT_VECTOR(). + */ +#define EMULTYPE_NO_DECODE (1 << 0) +#define EMULTYPE_TRAP_UD (1 << 1) +#define EMULTYPE_SKIP (1 << 2) +#define EMULTYPE_ALLOW_RETRY_PF (1 << 3) +#define EMULTYPE_TRAP_UD_FORCED (1 << 4) +#define EMULTYPE_VMWARE_GP (1 << 5) +#define EMULTYPE_PF (1 << 6) +#define EMULTYPE_COMPLETE_USER_EXIT (1 << 7) +#define EMULTYPE_WRITE_PF_TO_SP (1 << 8) +#define EMULTYPE_SKIP_SOFT_INT (1 << 9) + +#define EMULTYPE_SET_SOFT_INT_VECTOR(v) ((u32)((v) & 0xff) << 16) +#define EMULTYPE_GET_SOFT_INT_VECTOR(e) (((e) >> 16) & 0xff) + +static inline bool kvm_can_emulate_event_vectoring(int emul_type) +{ + return !(emul_type & EMULTYPE_PF); +} + +int kvm_emulate_instruction(struct kvm_vcpu *vcpu, int emulation_type); +int kvm_emulate_instruction_from_buffer(struct kvm_vcpu *vcpu, + void *insn, int insn_len); +void __kvm_prepare_emulation_failure_exit(struct kvm_vcpu *vcpu, + u64 *data, u8 ndata); +void kvm_prepare_emulation_failure_exit(struct kvm_vcpu *vcpu); + +void kvm_prepare_event_vectoring_exit(struct kvm_vcpu *vcpu, gpa_t gpa); +void kvm_prepare_unexpected_reason_exit(struct kvm_vcpu *vcpu, u64 exit_re= ason); =20 fastpath_t handle_fastpath_hlt(struct kvm_vcpu *vcpu); fastpath_t handle_fastpath_invd(struct kvm_vcpu *vcpu); =20 +int kvm_emulate_as_nop(struct kvm_vcpu *vcpu); +int kvm_emulate_invd(struct kvm_vcpu *vcpu); +int kvm_emulate_mwait(struct kvm_vcpu *vcpu); +int kvm_handle_invalid_op(struct kvm_vcpu *vcpu); +int kvm_emulate_monitor(struct kvm_vcpu *vcpu); + +int kvm_fast_pio(struct kvm_vcpu *vcpu, int size, unsigned short port, int= in); +int kvm_emulate_cpuid(struct kvm_vcpu *vcpu); +int kvm_emulate_halt(struct kvm_vcpu *vcpu); +int kvm_emulate_halt_noskip(struct kvm_vcpu *vcpu); +int kvm_emulate_ap_reset_hold(struct kvm_vcpu *vcpu); +int kvm_emulate_wbinvd(struct kvm_vcpu *vcpu); + +void kvm_vcpu_deliver_sipi_vector(struct kvm_vcpu *vcpu, u8 vector); + +int kvm_task_switch(struct kvm_vcpu *vcpu, u16 tss_selector, int idt_index, + int reason, bool has_error_code, u32 error_code); + +int __kvm_set_xcr(struct kvm_vcpu *vcpu, u32 index, u64 xcr); +int kvm_emulate_xsetbv(struct kvm_vcpu *vcpu); +int kvm_emulate_rdpmc(struct kvm_vcpu *vcpu); + +int kvm_skip_emulated_instruction(struct kvm_vcpu *vcpu); +int kvm_complete_insn_gp(struct kvm_vcpu *vcpu, int err); + +void kvm_queue_exception(struct kvm_vcpu *vcpu, unsigned nr); +void kvm_queue_exception_e(struct kvm_vcpu *vcpu, unsigned nr, u32 error_c= ode); +void kvm_queue_exception_p(struct kvm_vcpu *vcpu, unsigned nr, unsigned lo= ng payload); +void kvm_requeue_exception(struct kvm_vcpu *vcpu, unsigned int nr, + bool has_error_code, u32 error_code); +void kvm_inject_page_fault(struct kvm_vcpu *vcpu, struct x86_exception *fa= ult, + bool from_hardware); +void __kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, + struct x86_exception *fault, + bool from_hardware); + +static inline void kvm_inject_emulated_page_fault(struct kvm_vcpu *vcpu, + struct x86_exception *fault) +{ + __kvm_inject_emulated_page_fault(vcpu, fault, false); +} + +bool kvm_require_dr(struct kvm_vcpu *vcpu, int dr); + +static inline void kvm_inject_gp(struct kvm_vcpu *vcpu, u32 error_code) +{ + kvm_queue_exception_e(vcpu, GP_VECTOR, error_code); +} + +void kvm_inject_nmi(struct kvm_vcpu *vcpu); +int kvm_get_nr_pending_nmis(struct kvm_vcpu *vcpu); + +void __user *__x86_set_memory_region(struct kvm *kvm, int id, gpa_t gpa, + u32 size); +int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages); + +bool kvm_vcpu_is_reset_bsp(struct kvm_vcpu *vcpu); +bool kvm_vcpu_is_bsp(struct kvm_vcpu *vcpu); + +bool kvm_apicv_activated(struct kvm *kvm); +bool kvm_vcpu_apicv_activated(struct kvm_vcpu *vcpu); +void __kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu); +void __kvm_set_or_clear_apicv_inhibit(struct kvm *kvm, + enum kvm_apicv_inhibit reason, bool set); +void kvm_set_or_clear_apicv_inhibit(struct kvm *kvm, + enum kvm_apicv_inhibit reason, bool set); + +static inline void kvm_set_apicv_inhibit(struct kvm *kvm, + enum kvm_apicv_inhibit reason) +{ + kvm_set_or_clear_apicv_inhibit(kvm, reason, true); +} + +static inline void kvm_clear_apicv_inhibit(struct kvm *kvm, + enum kvm_apicv_inhibit reason) +{ + kvm_set_or_clear_apicv_inhibit(kvm, reason, false); +} + +void kvm_inc_or_dec_irq_window_inhibit(struct kvm *kvm, bool inc); + +static inline void kvm_inc_apicv_irq_window_req(struct kvm *kvm) +{ + kvm_inc_or_dec_irq_window_inhibit(kvm, true); +} + +static inline void kvm_dec_apicv_irq_window_req(struct kvm *kvm) +{ + kvm_inc_or_dec_irq_window_inhibit(kvm, false); +} + +void kvm_make_scan_ioapic_request(struct kvm *kvm); +void kvm_make_scan_ioapic_request_mask(struct kvm *kvm, + unsigned long *vcpu_bitmap); + void kvm_setup_xss_caps(void); =20 /* --=20 2.54.0.823.g6e5bcc1fc9-goog