From nobody Thu Sep 18 01:11:31 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE537C4167B for ; Tue, 13 Dec 2022 06:09:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234413AbiLMGJ3 (ORCPT ); Tue, 13 Dec 2022 01:09:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46194 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234095AbiLMGJU (ORCPT ); Tue, 13 Dec 2022 01:09:20 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D8C91A832 for ; Mon, 12 Dec 2022 22:09:19 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-3b0af5bcbd3so157025607b3.0 for ; Mon, 12 Dec 2022 22:09:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=+xrIAhvGbtHzogIJ3aNRRLQUShQPCNp9dtUJo01jRd4=; b=TGHdJLGEzqgYr9cAqGjdS3I4g33rP9ZZMIsLVLRpNw5QthQ5hNOCOIbR2FnxmU3emT BG0VoYoi8EQYjr9Ps/mEbzLbYgXxhMaaGFakXoyRnMg7exBdOpMv/P2grFeziK1ZZ+t8 krjqRIC9W5rOMjaeNVgM7X/8G/PrWxPeTiBjtTYuyrTCIYBLEHrpZ1RHKvivzqBnwNRl Pyc9obee24XNzNAwLDVLPjAq7t+STbiLwQvNSJJf/a4VdYdDZ+nos6y2MLX9w775lzYX ScnLoAgHVKPv4imTnqHRhUte42MDWnoAkRUPvRfADnr8YMZIWKMm2gQb4U4AXh2rJuyo chfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=+xrIAhvGbtHzogIJ3aNRRLQUShQPCNp9dtUJo01jRd4=; b=KKNnPrKtg5byJ93e8fSbKk7cisiy/SjlYCKp+hVJJfhO15PdOgz54DrdeRgsj4syn5 cV4haJU0SsNc8/J9f8FoK+0HruNRSxpGgaKJRkmUvSyrT6KhO3PpOrsQVIjM3gz7fAxG mQGiIViuqXWgqz9YL4KCH4xH2ZH7jjniCDdGUhJEYL78PJNvVOKjfug/4Z6xQBpuedRI xyMfNUQwuAy+TbhVHPa/kOj+tn4IAaiXYWbFxEJpbiPXKyDb4JHWAmrdDXzPrgdBrD68 4tZhhkpSkH8gc+hEIEloUyY1dPYhzlwKqMB34CutIZBut9CkBL5dlFIbczyncfcuOKni GFbg== X-Gm-Message-State: ANoB5pm4XqW11CTGscsPHfWBTxzmT84FVpYtnpTZ+9vL5/QUuxkjrt3M nRYvq8iGaaCyUICFBU8Eqo51gfD1eZI= X-Google-Smtp-Source: AA0mqf7eC+eRpSt2nLS+ccpNXnX4+Q2nZkSb/BXNkLxOr7fVQmesMF3CSY69wsR2dSK2ImsFWm8EkCm7V9c= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a25:1683:0:b0:6f5:6a39:978e with SMTP id 125-20020a251683000000b006f56a39978emr52138384ybw.6.1670911758316; Mon, 12 Dec 2022 22:09:18 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 13 Dec 2022 06:09:06 +0000 In-Reply-To: <20221213060912.654668-1-seanjc@google.com> Mime-Version: 1.0 References: <20221213060912.654668-1-seanjc@google.com> X-Mailer: git-send-email 2.39.0.rc1.256.g54fd8350bd-goog Message-ID: <20221213060912.654668-2-seanjc@google.com> Subject: [PATCH 1/7] KVM: x86: Make vmx_get_exit_qual() and vmx_get_intr_info() noinstr-friendly From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Zijlstra , Andy Lutomirski , Thomas Gleixner Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Add an extra special noinstr-friendly helper to test+mark a "register" available and use it when caching vmcs.EXIT_QUALIFICATION and vmcs.VM_EXIT_INTR_INFO. Make the caching helpers __always_inline too so that they can be used in noinstr functions. A future fix will move VMX's handling of NMI exits into the noinstr vmx_vcpu_enter_exit() so that the NMI is processed before any kind of instrumentation can trigger a fault and thus IRET, i.e. so that KVM doesn't invoke the NMI handler with NMIs enabled. Cc: Peter Zijlstra Signed-off-by: Sean Christopherson Acked-by: Peter Zijlstra (Intel) --- arch/x86/kvm/kvm_cache_regs.h | 12 ++++++++++++ arch/x86/kvm/vmx/vmx.h | 14 ++++++-------- 2 files changed, 18 insertions(+), 8 deletions(-) diff --git a/arch/x86/kvm/kvm_cache_regs.h b/arch/x86/kvm/kvm_cache_regs.h index c09174f73a34..4c91f626c058 100644 --- a/arch/x86/kvm/kvm_cache_regs.h +++ b/arch/x86/kvm/kvm_cache_regs.h @@ -75,6 +75,18 @@ static inline void kvm_register_mark_dirty(struct kvm_vc= pu *vcpu, __set_bit(reg, (unsigned long *)&vcpu->arch.regs_dirty); } =20 +/* + * kvm_register_test_and_mark_available() is a special snowflake that uses= an + * arch bitop directly to avoid the explicit instrumentation that comes wi= th + * the generic bitops. This allows code that cannot be instrumented (noin= str + * functions), e.g. the low level VM-Enter/VM-Exit paths, to cache registe= rs. + */ +static __always_inline bool kvm_register_test_and_mark_available(struct kv= m_vcpu *vcpu, + enum kvm_reg reg) +{ + return arch___test_and_set_bit(reg, (unsigned long *)&vcpu->arch.regs_ava= il); +} + /* * The "raw" register helpers are only for cases where the full 64 bits of= a * register are read/written irrespective of current vCPU mode. In other = words, diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index a3da84f4ea45..bb720a2f11ab 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -669,25 +669,23 @@ void intel_pmu_cross_mapped_check(struct kvm_pmu *pmu= ); int intel_pmu_create_guest_lbr_event(struct kvm_vcpu *vcpu); void vmx_passthrough_lbr_msrs(struct kvm_vcpu *vcpu); =20 -static inline unsigned long vmx_get_exit_qual(struct kvm_vcpu *vcpu) +static __always_inline unsigned long vmx_get_exit_qual(struct kvm_vcpu *vc= pu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); =20 - if (!kvm_register_is_available(vcpu, VCPU_EXREG_EXIT_INFO_1)) { - kvm_register_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_1); + if (!kvm_register_test_and_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_1)) vmx->exit_qualification =3D vmcs_readl(EXIT_QUALIFICATION); - } + return vmx->exit_qualification; } =20 -static inline u32 vmx_get_intr_info(struct kvm_vcpu *vcpu) +static __always_inline u32 vmx_get_intr_info(struct kvm_vcpu *vcpu) { struct vcpu_vmx *vmx =3D to_vmx(vcpu); =20 - if (!kvm_register_is_available(vcpu, VCPU_EXREG_EXIT_INFO_2)) { - kvm_register_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_2); + if (!kvm_register_test_and_mark_available(vcpu, VCPU_EXREG_EXIT_INFO_2)) vmx->exit_intr_info =3D vmcs_read32(VM_EXIT_INTR_INFO); - } + return vmx->exit_intr_info; } =20 --=20 2.39.0.rc1.256.g54fd8350bd-goog From nobody Thu Sep 18 01:11:31 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5721CC4332F for ; Tue, 13 Dec 2022 06:09:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234429AbiLMGJd (ORCPT ); Tue, 13 Dec 2022 01:09:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46204 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232252AbiLMGJV (ORCPT ); Tue, 13 Dec 2022 01:09:21 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 332D619286 for ; Mon, 12 Dec 2022 22:09:20 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id ep17-20020a17090ae65100b00219702c495cso1339189pjb.2 for ; Mon, 12 Dec 2022 22:09:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=3rYplRUH90lferx/1QrjPQLbygHUDtj5qeJE1KQCTqc=; b=EI40B1d/Mbv8m0VRGcWzxsd7VuKCZkwAPkWVNlQv9uYdsZdudkMdcUKtlj3sz3lzDa LUEvap9m5gVbRVLArHGjcPy1pJ1T0m1vx5tGFB7nzLVu6AbFptybI8XxoFizChEiw11b Xluwh7PxOVZthsV34BGOkV0j7R35fZic+oWBLpklUUJMaeKQh0ePUgTdiBi1QwYG9uQ/ Y8VI8cc/xhlGHau44SRNm3GSt4rYlXVCNL1nR2HMxr47eALh9q6z3n65fdaXu06LCVWV HngCvPGCyXTRA5NAp54D7J80f9dtcUdssrAsbngPyX1B17IdPjhXDZyFtP0PwjqGOjP/ 5dlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=3rYplRUH90lferx/1QrjPQLbygHUDtj5qeJE1KQCTqc=; b=hvGRlDqTI6QGpjl01FPxoAWz0h4xA2OiI0eVCFDVWadu/uyajAcBmbiB6eWe8qFX4X bDwmG9tPYOzxs+KgOx76uY8VqIhITGwyySGVAik96KU3ZP6Zm+7NxUG/1MUvcSENxm9D 1y5bFdbrAXWSXAnjNaUC6hz1pfW4ZfjpS5e7v9GMZLVMy0RmWv6idAOhfGnOSzL/UzMB 0ueUedHNSAWsI7rU+LsM1HUi9PQl4g8QKe4crjmYtTahRlnLPZDHU3bSCokISRYtOl3X 7d1LqBX5D7SFw0zCTLq0GFnwqkMc0W/LJ6AFWTCg0rvJtDDSy475QVxNkvIp4+KCgd69 0Sqw== X-Gm-Message-State: ANoB5pngMIQVTDV/KItmC8iADC4yumxhV6RxKVZP3i1NPgBiZ+Lti8Oz sFKwNUDA8bUrKgk1qc8FI+33aFV9zQ0= X-Google-Smtp-Source: AA0mqf59poWFoBEYN/ZqdGI/4uRVPx0JCk1L8iDLHBT4JiNXVR0fo+3paILjT1aP4fCgo/AxBSkuWshdn50= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a62:1991:0:b0:578:5559:d5a5 with SMTP id 139-20020a621991000000b005785559d5a5mr501316pfz.74.1670911759951; Mon, 12 Dec 2022 22:09:19 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 13 Dec 2022 06:09:07 +0000 In-Reply-To: <20221213060912.654668-1-seanjc@google.com> Mime-Version: 1.0 References: <20221213060912.654668-1-seanjc@google.com> X-Mailer: git-send-email 2.39.0.rc1.256.g54fd8350bd-goog Message-ID: <20221213060912.654668-3-seanjc@google.com> Subject: [PATCH 2/7] KVM: VMX: Allow VM-Fail path of VMREAD helper to be instrumented From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Zijlstra , Andy Lutomirski , Thomas Gleixner Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Allow instrumentation in the VM-Fail path of __vmcs_readl() so that the helper can be used in noinstr functions, e.g. to get the exit reason in vmx_vcpu_enter_exit() in order to handle NMI VM-Exits in the noinstr section. While allowing instrumentation isn't technically safe, KVM has much bigger problems if VMREAD fails in a noinstr section. Note, all other VMX instructions also allow instrumentation in their VM-Fail paths for similar reasons, VMREAD was simply omitted by commit 3ebccdf373c2 ("x86/kvm/vmx: Move guest enter/exit into .noinstr.text") because VMREAD wasn't used in a noinstr section at the time. Signed-off-by: Sean Christopherson Acked-by: Peter Zijlstra (Intel) --- arch/x86/kvm/vmx/vmx_ops.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/x86/kvm/vmx/vmx_ops.h b/arch/x86/kvm/vmx/vmx_ops.h index 842dc898c972..5838489e719b 100644 --- a/arch/x86/kvm/vmx/vmx_ops.h +++ b/arch/x86/kvm/vmx/vmx_ops.h @@ -100,8 +100,10 @@ static __always_inline unsigned long __vmcs_readl(unsi= gned long field) return value; =20 do_fail: + instrumentation_begin(); WARN_ONCE(1, "kvm: vmread failed: field=3D%lx\n", field); pr_warn_ratelimited("kvm: vmread failed: field=3D%lx\n", field); + instrumentation_end(); return 0; =20 do_exception: --=20 2.39.0.rc1.256.g54fd8350bd-goog From nobody Thu Sep 18 01:11:31 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23714C4332F for ; Tue, 13 Dec 2022 06:09:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234435AbiLMGJh (ORCPT ); Tue, 13 Dec 2022 01:09:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46224 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234371AbiLMGJX (ORCPT ); Tue, 13 Dec 2022 01:09:23 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 171BC1AD80 for ; Mon, 12 Dec 2022 22:09:22 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id md9-20020a17090b23c900b00218fa3308a9so1317925pjb.8 for ; Mon, 12 Dec 2022 22:09:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=vIlOHydaCh0ao3eFY0wKJPSpCdXFKH/dETpychmunOk=; b=VPMI9+ZU1BsAAUUeJUhhb/vSZ0K6O7FxmmGujRXdbsYFdCthw2opRpUDUo05tgMl7/ UAXsgfaJIDtdQjv0gQWhKHu7t1Kb3ZKBGBmDp8R+/uaRObMKKI/qIHywslWfBcWugzQn X74V27+dpXTJKwaD3we+FcosDS3oA++Kx3Y5XTdoNLdjVoW+r5wWOm1m3KrNXUi79VnS pQXShGI/HjzdNdcN2abo9e0E6uGUbGF6MR8/oAW1BUnD3OnSR4SgSNWgY3ti+j7/SNLo ffA6rdN9rF8+xRcUjWLRIXuiBPJyLExE6DrezUnktN5GHRdKPbbIcG+J/VJARYXR7x58 lk0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=vIlOHydaCh0ao3eFY0wKJPSpCdXFKH/dETpychmunOk=; b=YhVqOJYzqYYWlxBDH5SPtQ7L2gM88fwa7xpBySIQWv6CAvydWDYv6sHkjPMh4epIRs I4cXnOCD1TNgp+Vo08y7ncby5fcosK3FCv77bJM6lwE4t/HWuyy6ja/LUUxltPxOP16V vivV48+1GCjncFbgXVqdPtFOt8c8eJIbxhGDUKsl5vnFvc3h6aGhR0kNFQThRckqqmak hiUEaY9FPEhtD4aKcuti4+dDwfZx9RM3ut7ai9DXBGF+bFjauynlIZvblOVWl0nqUz/0 AXyGa9YrikIA8c9TxgbV8nI8CTMrGiTWSfoXRLIamVir+DPztaD8Kgg0o7WslS4Iws0w J/nw== X-Gm-Message-State: ANoB5plrYGcBWFS7DcMIqWQmmBNCTOvx5pTqqTkucbzA5wAiPHke1BWp iP8xA3lXDLbUFNKRr+1m5uSz17CxQ+4= X-Google-Smtp-Source: AA0mqf4OZ8bwKNFcvO4cMdVPaYXvqgT+KB2K18aKoB1pDt/JKnnFLVYK8XvS1wDohZTUf7y+iIpDvVlrKBE= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:903:181:b0:189:8f11:f2f with SMTP id z1-20020a170903018100b001898f110f2fmr53434946plg.133.1670911761544; Mon, 12 Dec 2022 22:09:21 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 13 Dec 2022 06:09:08 +0000 In-Reply-To: <20221213060912.654668-1-seanjc@google.com> Mime-Version: 1.0 References: <20221213060912.654668-1-seanjc@google.com> X-Mailer: git-send-email 2.39.0.rc1.256.g54fd8350bd-goog Message-ID: <20221213060912.654668-4-seanjc@google.com> Subject: [PATCH 3/7] KVM: VMX: Always inline eVMCS read/write helpers From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Zijlstra , Andy Lutomirski , Thomas Gleixner Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Tag all evmcs_{read,write}() helpers __always_inline so that they can be freely used in noinstr sections, e.g. to get the VM-Exit reason in vcpu_vmx_enter_exit() (in a future patch). For consistency and to avoid more spot fixes in the future, e.g. see commit 010050a86393 ("x86/kvm: Always inline evmcs_write64()"), tag all accessors even though evmcs_read32() is the only anticipated use case in the near future. In practice, non-KASAN builds are all but guaranteed to inline the helpers anyways. vmlinux.o: warning: objtool: vmx_vcpu_enter_exit+0x107: call to evmcs_rea= d32() leaves .noinstr.text section Reported-by: kernel test robot Signed-off-by: Sean Christopherson Acked-by: Peter Zijlstra (Intel) --- arch/x86/kvm/vmx/hyperv.h | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/arch/x86/kvm/vmx/hyperv.h b/arch/x86/kvm/vmx/hyperv.h index 571e7929d14e..3f4049e4f35d 100644 --- a/arch/x86/kvm/vmx/hyperv.h +++ b/arch/x86/kvm/vmx/hyperv.h @@ -136,7 +136,7 @@ static __always_inline void evmcs_write64(unsigned long= field, u64 value) current_evmcs->hv_clean_fields &=3D ~clean_field; } =20 -static inline void evmcs_write32(unsigned long field, u32 value) +static __always_inline void evmcs_write32(unsigned long field, u32 value) { u16 clean_field; int offset =3D get_evmcs_offset(field, &clean_field); @@ -148,7 +148,7 @@ static inline void evmcs_write32(unsigned long field, u= 32 value) current_evmcs->hv_clean_fields &=3D ~clean_field; } =20 -static inline void evmcs_write16(unsigned long field, u16 value) +static __always_inline void evmcs_write16(unsigned long field, u16 value) { u16 clean_field; int offset =3D get_evmcs_offset(field, &clean_field); @@ -160,7 +160,7 @@ static inline void evmcs_write16(unsigned long field, u= 16 value) current_evmcs->hv_clean_fields &=3D ~clean_field; } =20 -static inline u64 evmcs_read64(unsigned long field) +static __always_inline u64 evmcs_read64(unsigned long field) { int offset =3D get_evmcs_offset(field, NULL); =20 @@ -170,7 +170,7 @@ static inline u64 evmcs_read64(unsigned long field) return *(u64 *)((char *)current_evmcs + offset); } =20 -static inline u32 evmcs_read32(unsigned long field) +static __always_inline u32 evmcs_read32(unsigned long field) { int offset =3D get_evmcs_offset(field, NULL); =20 @@ -180,7 +180,7 @@ static inline u32 evmcs_read32(unsigned long field) return *(u32 *)((char *)current_evmcs + offset); } =20 -static inline u16 evmcs_read16(unsigned long field) +static __always_inline u16 evmcs_read16(unsigned long field) { int offset =3D get_evmcs_offset(field, NULL); =20 @@ -213,11 +213,11 @@ static inline void evmcs_load(u64 phys_addr) =20 #else /* !IS_ENABLED(CONFIG_HYPERV) */ static __always_inline void evmcs_write64(unsigned long field, u64 value) = {} -static inline void evmcs_write32(unsigned long field, u32 value) {} -static inline void evmcs_write16(unsigned long field, u16 value) {} -static inline u64 evmcs_read64(unsigned long field) { return 0; } -static inline u32 evmcs_read32(unsigned long field) { return 0; } -static inline u16 evmcs_read16(unsigned long field) { return 0; } +static __always_inline void evmcs_write32(unsigned long field, u32 value) = {} +static __always_inline void evmcs_write16(unsigned long field, u16 value) = {} +static __always_inline u64 evmcs_read64(unsigned long field) { return 0; } +static __always_inline u32 evmcs_read32(unsigned long field) { return 0; } +static __always_inline u16 evmcs_read16(unsigned long field) { return 0; } static inline void evmcs_load(u64 phys_addr) {} static inline void evmcs_touch_msr_bitmap(void) {} #endif /* IS_ENABLED(CONFIG_HYPERV) */ --=20 2.39.0.rc1.256.g54fd8350bd-goog From nobody Thu Sep 18 01:11:31 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8FF70C4332F for ; Tue, 13 Dec 2022 06:09:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229889AbiLMGJm (ORCPT ); Tue, 13 Dec 2022 01:09:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46250 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234391AbiLMGJY (ORCPT ); Tue, 13 Dec 2022 01:09:24 -0500 Received: from mail-pf1-x44a.google.com (mail-pf1-x44a.google.com [IPv6:2607:f8b0:4864:20::44a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9EC781A809 for ; Mon, 12 Dec 2022 22:09:23 -0800 (PST) Received: by mail-pf1-x44a.google.com with SMTP id cp23-20020a056a00349700b005775c52dbceso1406303pfb.21 for ; Mon, 12 Dec 2022 22:09:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=NFEhFRkg0+nIHx94jx/VqF2L4sTiUmledWciOk7WPsw=; b=Mb6tzs3YFjNk5QMuKsNyQaWTpQK6Q7lsJPdTl4vbogpOKDC+ntZV7Jpct4rB9qejgS +TtDI14OD1/tbbyL0L1wge29Emn1TKCNDyUVIVW/9MyHaXI0nXK8IvBdIC6yg8guyoAF gfuWOj8X/Vo1tvZWFwPCHpGbpQmSokz4ohTqy8O5ywgy8xNUQdq25HZyfOn06xVPbeHz f1jN5vSQgRJo2tLg2dmUbtNrbiKUpHc6IEDkC93EvSlNUWgEFgWzh466hT2saqDJ4VPo 6z92FRVBWysyeOs5/s4MlzrzvuBOUfUzSekVTf7PEWy4nB84hbdH6GhBOxMWrJyxaqJ0 D0sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=NFEhFRkg0+nIHx94jx/VqF2L4sTiUmledWciOk7WPsw=; b=OMdqhkjHrmc7VDOj4Rfj1Tyx1Z2pDX6f02PGC2NqgwtneMlyjHIgvUJ0QIC4dOgRVN 1P6WnAlCZlRr+zvNgdGz30/M74iQow8I7OMRDNvtz9MKjfTuhoswx7YM4jSqAcrMDWFB efF2nKkLwrCeZ7ot24HV/2SB2nw8jcrkQWOlZEBqxBsOPnASt+knCRRSDIFLVxcsf/Kl faT87OtPnuIeLUuy8SSRsslP+FKZmdznafTU/iHCiKmpCXIkeCZHeuBUjJnGbZFNBjv6 ojXX7BHsOvltZf6Eu/dVjrbzCE6iEc7U5jvJpVV25IxSf2lWUmzPlUr/N/llZghQ41Yp T8sw== X-Gm-Message-State: ANoB5plsP/e8HyOPdWrw6xyT/dUbLDhrmMEf3SZycRQZ27d3mphz2Jrr 3msOjD5YFAR3S99FhynPYWUh3pBSjeM= X-Google-Smtp-Source: AA0mqf5jt5c245aUwJezKZHvkmsfBuO04Z+Vnt9FiIfW7ooVnZwK9K2l+TpVRWvrG4KVi9WhnTLJKZwG7M8= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:90b:2d90:b0:219:7904:6de with SMTP id sj16-20020a17090b2d9000b00219790406demr166821pjb.136.1670911763239; Mon, 12 Dec 2022 22:09:23 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 13 Dec 2022 06:09:09 +0000 In-Reply-To: <20221213060912.654668-1-seanjc@google.com> Mime-Version: 1.0 References: <20221213060912.654668-1-seanjc@google.com> X-Mailer: git-send-email 2.39.0.rc1.256.g54fd8350bd-goog Message-ID: <20221213060912.654668-5-seanjc@google.com> Subject: [PATCH 4/7] KVM: VMX: Always inline to_vmx() and to_kvm_vmx() From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Zijlstra , Andy Lutomirski , Thomas Gleixner Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Tag to_vmx() and to_kvm_vmx() __always_inline as they both just reflect the passed in pointer (the embedded struct is the first field in the container), and drop the @vmx param from vmx_vcpu_enter_exit(), which likely existed purely to make noinstr validation happy. Amusingly, when the compiler decides to not inline the helpers, e.g. for KASAN builds, to_vmx() and to_kvm_vmx() may end up pointing at the same symbol, which generates very confusing objtool warnings. E.g. the use of to_vmx() in a future patch led to objtool complaining about to_kvm_vmx(), and only once all use of to_kvm_vmx() was commented out did to_vmx() pop up in the obj tool report. vmlinux.o: warning: objtool: vmx_vcpu_enter_exit+0x160: call to to_kvm_vm= x() leaves .noinstr.text section Signed-off-by: Sean Christopherson Acked-by: Peter Zijlstra (Intel) --- arch/x86/kvm/vmx/vmx.c | 5 +++-- arch/x86/kvm/vmx/vmx.h | 4 ++-- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index fe5615fd8295..e2c96f204b82 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -7096,9 +7096,10 @@ static fastpath_t vmx_exit_handlers_fastpath(struct = kvm_vcpu *vcpu) } =20 static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, - struct vcpu_vmx *vmx, unsigned long flags) { + struct vcpu_vmx *vmx =3D to_vmx(vcpu); + guest_state_enter_irqoff(); =20 /* L1D Flush includes CPU buffer clear to mitigate MDS */ @@ -7216,7 +7217,7 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu) kvm_wait_lapic_expire(vcpu); =20 /* The actual VMENTER/EXIT is in the .noinstr.text section. */ - vmx_vcpu_enter_exit(vcpu, vmx, __vmx_vcpu_run_flags(vmx)); + vmx_vcpu_enter_exit(vcpu, __vmx_vcpu_run_flags(vmx)); =20 /* All fields are clean at this point */ if (static_branch_unlikely(&enable_evmcs)) { diff --git a/arch/x86/kvm/vmx/vmx.h b/arch/x86/kvm/vmx/vmx.h index bb720a2f11ab..2acdc54bc34b 100644 --- a/arch/x86/kvm/vmx/vmx.h +++ b/arch/x86/kvm/vmx/vmx.h @@ -640,12 +640,12 @@ BUILD_CONTROLS_SHADOW(tertiary_exec, TERTIARY_VM_EXEC= _CONTROL, 64) (1 << VCPU_EXREG_EXIT_INFO_1) | \ (1 << VCPU_EXREG_EXIT_INFO_2)) =20 -static inline struct kvm_vmx *to_kvm_vmx(struct kvm *kvm) +static __always_inline struct kvm_vmx *to_kvm_vmx(struct kvm *kvm) { return container_of(kvm, struct kvm_vmx, kvm); } =20 -static inline struct vcpu_vmx *to_vmx(struct kvm_vcpu *vcpu) +static __always_inline struct vcpu_vmx *to_vmx(struct kvm_vcpu *vcpu) { return container_of(vcpu, struct vcpu_vmx, vcpu); } --=20 2.39.0.rc1.256.g54fd8350bd-goog From nobody Thu Sep 18 01:11:31 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3B5CC10F1B for ; Tue, 13 Dec 2022 06:09:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234398AbiLMGJ4 (ORCPT ); Tue, 13 Dec 2022 01:09:56 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46696 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234443AbiLMGJj (ORCPT ); Tue, 13 Dec 2022 01:09:39 -0500 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3975B1ADA3 for ; Mon, 12 Dec 2022 22:09:25 -0800 (PST) Received: by mail-pl1-x649.google.com with SMTP id d7-20020a170902b70700b0018f4bf00569so5160600pls.4 for ; Mon, 12 Dec 2022 22:09:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=3ygtdZJHpe2jAbhrmEODv8rggvGUoguv5qPo3aQqhPM=; b=pcmVTu4JD8o5BzVbE5RqsAHQF4iTjRKXTPoe+Wr1AmHVQrlVOOLTREcGLzBdX+CoDo X5MOObPom/RmvwSbNsO6KyVsAm0XRd0voNAY0/brtLjD+YSWb/xEvQB+xHexbwoeoPei 9jdk3IzS747RceXkfVCGzDvhLBGFC74oZOJdMmE48mLLaRx/wH091RT6FwBPkS/zbRf9 Oyb4HYwaW1FIGfBTJJj30F7YXbKC/w0okF46x5P04R3+nRZpvXvjuTWHfIssZ5QAoX7I 6ex6l/Gs54L1I9eeJHSnhDUtkWvSF4Whla4N2HvTnwSewDlW82hqZ8zQ+QeeiOZJ0UkE 4cag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=3ygtdZJHpe2jAbhrmEODv8rggvGUoguv5qPo3aQqhPM=; b=XsGojyzMPaUTbuBvbDTvEMfB4C3LvEwuZMEjkK9cfQkJsp791KbxeTMo+uJ3He2vTc 1m0oahd+r+03yrbGcDm1Bp4CRRzj/iiZW4RsL5XLsP+z59YE7OrKuco7Cr5TLQFzQt/z NzdjCQ33N3xTO6NLYDVoLhDzsO9ZExiKm6WxUgmwOO7AErVHtc+fU7+9w59o4nULwt8p TBPH9f3nwkMzuZN7XS+N60m9WtuFBFBOcuarI/md/zwSafZqIdqg+xxpNsE37YXQy9IR 8me75pWcDNNr23ig6ttZ+k+77Dl66S9nOdlcJ0mCp7QI57LfewYQn2Z82i5pFd3AYOHC xQSw== X-Gm-Message-State: ANoB5pl1BWrSsFcSrYn1AlXCMUUAov9qXuWd4fp+ro/poLZeGZjr+rRH WshcVkLOEA4N47O+mOMNQA1iPrwaVe0= X-Google-Smtp-Source: AA0mqf5qUYpkTbaxWjOo5zAYsTVPsJWj4HukpB0jjDg+TpR/x3X6VGRcXqTYdfccQRGJk7CJ8Xrm8cfONws= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:f14b:b0:18f:5a3:6069 with SMTP id d11-20020a170902f14b00b0018f05a36069mr706521plb.46.1670911764799; Mon, 12 Dec 2022 22:09:24 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 13 Dec 2022 06:09:10 +0000 In-Reply-To: <20221213060912.654668-1-seanjc@google.com> Mime-Version: 1.0 References: <20221213060912.654668-1-seanjc@google.com> X-Mailer: git-send-email 2.39.0.rc1.256.g54fd8350bd-goog Message-ID: <20221213060912.654668-6-seanjc@google.com> Subject: [PATCH 5/7] x86/entry: KVM: Use dedicated VMX NMI entry for 32-bit kernels too From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Zijlstra , Andy Lutomirski , Thomas Gleixner Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Use a dedicated entry for invoking the NMI handler from KVM VMX's VM-Exit path for 32-bit even though using a dedicated entry for 32-bit isn't strictly necessary. Exposing a single symbol will allow KVM to reference the entry point in assembly code without having to resort to more #ifdefs (or #defines). identry.h is intended to be included from asm files only once, and so simply including idtentry.h in KVM assembly isn't an option. Bypassing the ESP fixup and CR3 switching in the standard NMI entry code is safe as KVM always handles NMIs that occur in the guest on a kernel stack, with a kernel CR3. Cc: Andy Lutomirski Cc: Thomas Gleixner Signed-off-by: Sean Christopherson Acked-by: Peter Zijlstra (Intel) Reviewed-by: Lai Jiangshan --- arch/x86/include/asm/idtentry.h | 16 ++++++---------- arch/x86/kernel/nmi.c | 8 ++++---- arch/x86/kvm/vmx/vmx.c | 4 ++-- 3 files changed, 12 insertions(+), 16 deletions(-) diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentr= y.h index 72184b0b2219..b241af4ce9b4 100644 --- a/arch/x86/include/asm/idtentry.h +++ b/arch/x86/include/asm/idtentry.h @@ -582,18 +582,14 @@ DECLARE_IDTENTRY_RAW(X86_TRAP_MC, xenpv_exc_machine_c= heck); =20 /* NMI */ =20 -#if defined(CONFIG_X86_64) && IS_ENABLED(CONFIG_KVM_INTEL) +#if IS_ENABLED(CONFIG_KVM_INTEL) /* - * Special NOIST entry point for VMX which invokes this on the kernel - * stack. asm_exc_nmi() requires an IST to work correctly vs. the NMI - * 'executing' marker. - * - * On 32bit this just uses the regular NMI entry point because 32-bit does - * not have ISTs. + * Special entry point for VMX which invokes this on the kernel stack, eve= n for + * 64-bit, i.e. without using an IST. asm_exc_nmi() requires an IST to wo= rk + * correctly vs. the NMI 'executing' marker. Used for 32-bit kernels as w= ell + * to avoid more ifdeffery. */ -DECLARE_IDTENTRY(X86_TRAP_NMI, exc_nmi_noist); -#else -#define asm_exc_nmi_noist asm_exc_nmi +DECLARE_IDTENTRY(X86_TRAP_NMI, exc_nmi_kvm_vmx); #endif =20 DECLARE_IDTENTRY_NMI(X86_TRAP_NMI, exc_nmi); diff --git a/arch/x86/kernel/nmi.c b/arch/x86/kernel/nmi.c index cec0bfa3bc04..e37faba95bb5 100644 --- a/arch/x86/kernel/nmi.c +++ b/arch/x86/kernel/nmi.c @@ -527,14 +527,14 @@ DEFINE_IDTENTRY_RAW(exc_nmi) mds_user_clear_cpu_buffers(); } =20 -#if defined(CONFIG_X86_64) && IS_ENABLED(CONFIG_KVM_INTEL) -DEFINE_IDTENTRY_RAW(exc_nmi_noist) +#if IS_ENABLED(CONFIG_KVM_INTEL) +DEFINE_IDTENTRY_RAW(exc_nmi_kvm_vmx) { exc_nmi(regs); } -#endif #if IS_MODULE(CONFIG_KVM_INTEL) -EXPORT_SYMBOL_GPL(asm_exc_nmi_noist); +EXPORT_SYMBOL_GPL(asm_exc_nmi_kvm_vmx); +#endif #endif =20 void stop_nmi(void) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index e2c96f204b82..7ace22ee240d 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6791,7 +6791,7 @@ void vmx_do_interrupt_nmi_irqoff(unsigned long entry); static void handle_interrupt_nmi_irqoff(struct kvm_vcpu *vcpu, unsigned long entry) { - bool is_nmi =3D entry =3D=3D (unsigned long)asm_exc_nmi_noist; + bool is_nmi =3D entry =3D=3D (unsigned long)asm_exc_nmi_kvm_vmx; =20 kvm_before_interrupt(vcpu, is_nmi ? KVM_HANDLING_NMI : KVM_HANDLING_IRQ); vmx_do_interrupt_nmi_irqoff(entry); @@ -6820,7 +6820,7 @@ static void handle_nm_fault_irqoff(struct kvm_vcpu *v= cpu) =20 static void handle_exception_nmi_irqoff(struct vcpu_vmx *vmx) { - const unsigned long nmi_entry =3D (unsigned long)asm_exc_nmi_noist; + const unsigned long nmi_entry =3D (unsigned long)asm_exc_nmi_kvm_vmx; u32 intr_info =3D vmx_get_intr_info(&vmx->vcpu); =20 /* if exit due to PF check for async PF */ --=20 2.39.0.rc1.256.g54fd8350bd-goog From nobody Thu Sep 18 01:11:31 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E03EFC4332F for ; Tue, 13 Dec 2022 06:10:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234446AbiLMGKE (ORCPT ); Tue, 13 Dec 2022 01:10:04 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46282 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229645AbiLMGJp (ORCPT ); Tue, 13 Dec 2022 01:09:45 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 125B61ADAA for ; Mon, 12 Dec 2022 22:09:27 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id o6-20020a17090ab88600b0021edf53917eso1336719pjr.3 for ; Mon, 12 Dec 2022 22:09:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=XJmCztab5I+GOPa0DQl6Gc8jULACLfl5WBPqqp47Nbo=; b=eOYAUekkYFKW32/sardr9LbkqefyMBx11vxUVkrH0ldjTntI2ZqZ12Z1YKA8CIJJll 5gJKypePk09DROtfI5aCEo66BA5V+NzqriBgU1mn2W2AI3R8kOtCMoF3y5solFD0HK9p p320W+0By14HNqdn0+hlsW1X/3EZmtWRJH2n3mdjsIYLGDt3ZRy7pdDm1JUce8nvs+U+ agGnzIMGnjy9CKnRAg/9Z18OiIYhxe/3PQZzB9sfpM59C3Qi4oSGs7XPTYOW3TOVjoiO e+gPF63js8NwFX6PxL4drafuNhkepvmEbblZQV/V0zrMJiMYs2C9kZeO/uUexIci8mmg Pl7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=XJmCztab5I+GOPa0DQl6Gc8jULACLfl5WBPqqp47Nbo=; b=2dgXaa7LvRPq2vmKxbmVq3w10jepr2t9Uz4ZkcfuikbLuqnrYt48FqsuujmARyK2lh xJJSKbRtb3dl2BWBXNXZTsuxaykaFu4Pn8JTCtdKwRX/1gD8o5t/ChyMjB8mypla9M5q j2FPTXbuOW70Ol3Vw7H3ATRsNKSQGwCHCEK9Fd66JTUi537H39NUNnkEqfH1Bb+g/nJd In9m3OW1f6/pjFFfJkiAe9VNYKKpxtT99xoiFEMIIUqCmHyU9pjv+hDAK81QfiLpiGs0 hS8X4j+dNsco8d/Qe2NyQqbi2cwf1/yrSBnvAqTpek+yFuaCF6of2BB6ThDLeoohpWNY K+yg== X-Gm-Message-State: ANoB5plCFbRJUCjqoAvJ/yAgdVsbx/p6nATDMKALdwt6z8Q0vbBwpkaa dXgArlVmwoW67mnD5fjz4fy02t4cow4= X-Google-Smtp-Source: AA0mqf4IGPRfK+e8J3A9Bf74E8Zn6ZE1wP/oefeh6jkbHly8QijandEcr34pV6l+ae2KwuKns91h9X4Yb7M= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:90a:2ecb:b0:219:5b3b:2b9f with SMTP id h11-20020a17090a2ecb00b002195b3b2b9fmr22689pjs.2.1670911766390; Mon, 12 Dec 2022 22:09:26 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 13 Dec 2022 06:09:11 +0000 In-Reply-To: <20221213060912.654668-1-seanjc@google.com> Mime-Version: 1.0 References: <20221213060912.654668-1-seanjc@google.com> X-Mailer: git-send-email 2.39.0.rc1.256.g54fd8350bd-goog Message-ID: <20221213060912.654668-7-seanjc@google.com> Subject: [PATCH 6/7] KVM: VMX: Provide separate subroutines for invoking NMI vs. IRQ handlers From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Zijlstra , Andy Lutomirski , Thomas Gleixner Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Split the asm subroutines for handling NMIs versus IRQs that occur in the guest so that the NMI handler can be called from a noinstr section. As a bonus, the NMI path doesn't need an indirect branch. Signed-off-by: Sean Christopherson Acked-by: Peter Zijlstra (Intel) --- arch/x86/kvm/vmx/vmenter.S | 70 +++++++++++++++++++++----------------- arch/x86/kvm/vmx/vmx.c | 26 ++++++-------- 2 files changed, 50 insertions(+), 46 deletions(-) diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index 766c6b3ef5ed..9d987e7e48c4 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -31,6 +31,39 @@ #define VCPU_R15 __VCPU_REGS_R15 * WORD_SIZE #endif =20 +.macro VMX_DO_EVENT_IRQOFF call_insn call_target + /* + * Unconditionally create a stack frame, getting the correct RSP on the + * stack (for x86-64) would take two instructions anyways, and RBP can + * be used to restore RSP to make objtool happy (see below). + */ + push %_ASM_BP + mov %_ASM_SP, %_ASM_BP + +#ifdef CONFIG_X86_64 + /* + * Align RSP to a 16-byte boundary (to emulate CPU behavior) before + * creating the synthetic interrupt stack frame for the IRQ/NMI. + */ + and $-16, %rsp + push $__KERNEL_DS + push %rbp +#endif + pushf + push $__KERNEL_CS + \call_insn \call_target + + /* + * "Restore" RSP from RBP, even though IRET has already unwound RSP to + * the correct value. objtool doesn't know the callee will IRET and, + * without the explicit restore, thinks the stack is getting walloped. + * Using an unwind hint is problematic due to x86-64's dynamic alignment. + */ + mov %_ASM_BP, %_ASM_SP + pop %_ASM_BP + RET +.endm + .section .noinstr.text, "ax" =20 /** @@ -320,35 +353,10 @@ SYM_FUNC_START(vmread_error_trampoline) SYM_FUNC_END(vmread_error_trampoline) #endif =20 -SYM_FUNC_START(vmx_do_interrupt_nmi_irqoff) - /* - * Unconditionally create a stack frame, getting the correct RSP on the - * stack (for x86-64) would take two instructions anyways, and RBP can - * be used to restore RSP to make objtool happy (see below). - */ - push %_ASM_BP - mov %_ASM_SP, %_ASM_BP +SYM_FUNC_START(vmx_do_nmi_irqoff) + VMX_DO_EVENT_IRQOFF call asm_exc_nmi_kvm_vmx +SYM_FUNC_END(vmx_do_nmi_irqoff) =20 -#ifdef CONFIG_X86_64 - /* - * Align RSP to a 16-byte boundary (to emulate CPU behavior) before - * creating the synthetic interrupt stack frame for the IRQ/NMI. - */ - and $-16, %rsp - push $__KERNEL_DS - push %rbp -#endif - pushf - push $__KERNEL_CS - CALL_NOSPEC _ASM_ARG1 - - /* - * "Restore" RSP from RBP, even though IRET has already unwound RSP to - * the correct value. objtool doesn't know the callee will IRET and, - * without the explicit restore, thinks the stack is getting walloped. - * Using an unwind hint is problematic due to x86-64's dynamic alignment. - */ - mov %_ASM_BP, %_ASM_SP - pop %_ASM_BP - RET -SYM_FUNC_END(vmx_do_interrupt_nmi_irqoff) +SYM_FUNC_START(vmx_do_interrupt_irqoff) + VMX_DO_EVENT_IRQOFF CALL_NOSPEC _ASM_ARG1 +SYM_FUNC_END(vmx_do_interrupt_irqoff) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 7ace22ee240d..c242e2591896 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6786,17 +6786,8 @@ static void vmx_apicv_post_state_restore(struct kvm_= vcpu *vcpu) memset(vmx->pi_desc.pir, 0, sizeof(vmx->pi_desc.pir)); } =20 -void vmx_do_interrupt_nmi_irqoff(unsigned long entry); - -static void handle_interrupt_nmi_irqoff(struct kvm_vcpu *vcpu, - unsigned long entry) -{ - bool is_nmi =3D entry =3D=3D (unsigned long)asm_exc_nmi_kvm_vmx; - - kvm_before_interrupt(vcpu, is_nmi ? KVM_HANDLING_NMI : KVM_HANDLING_IRQ); - vmx_do_interrupt_nmi_irqoff(entry); - kvm_after_interrupt(vcpu); -} +void vmx_do_interrupt_irqoff(unsigned long entry); +void vmx_do_nmi_irqoff(void); =20 static void handle_nm_fault_irqoff(struct kvm_vcpu *vcpu) { @@ -6820,7 +6811,6 @@ static void handle_nm_fault_irqoff(struct kvm_vcpu *v= cpu) =20 static void handle_exception_nmi_irqoff(struct vcpu_vmx *vmx) { - const unsigned long nmi_entry =3D (unsigned long)asm_exc_nmi_kvm_vmx; u32 intr_info =3D vmx_get_intr_info(&vmx->vcpu); =20 /* if exit due to PF check for async PF */ @@ -6833,8 +6823,11 @@ static void handle_exception_nmi_irqoff(struct vcpu_= vmx *vmx) else if (is_machine_check(intr_info)) kvm_machine_check(); /* We need to handle NMIs before interrupts are enabled */ - else if (is_nmi(intr_info)) - handle_interrupt_nmi_irqoff(&vmx->vcpu, nmi_entry); + else if (is_nmi(intr_info)) { + kvm_before_interrupt(&vmx->vcpu, KVM_HANDLING_NMI); + vmx_do_nmi_irqoff(); + kvm_after_interrupt(&vmx->vcpu); + } } =20 static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu) @@ -6847,7 +6840,10 @@ static void handle_external_interrupt_irqoff(struct = kvm_vcpu *vcpu) "KVM: unexpected VM-Exit interrupt info: 0x%x", intr_info)) return; =20 - handle_interrupt_nmi_irqoff(vcpu, gate_offset(desc)); + kvm_before_interrupt(vcpu, KVM_HANDLING_IRQ); + vmx_do_interrupt_irqoff(gate_offset(desc)); + kvm_after_interrupt(vcpu); + vcpu->arch.at_instruction_boundary =3D true; } =20 --=20 2.39.0.rc1.256.g54fd8350bd-goog From nobody Thu Sep 18 01:11:31 2025 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44DECC4332F for ; Tue, 13 Dec 2022 06:10:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234536AbiLMGKJ (ORCPT ); Tue, 13 Dec 2022 01:10:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:47006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234461AbiLMGJu (ORCPT ); Tue, 13 Dec 2022 01:09:50 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8BE721ADA8 for ; Mon, 12 Dec 2022 22:09:28 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id k7-20020a17090a39c700b002192c16f19aso1347144pjf.1 for ; Mon, 12 Dec 2022 22:09:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=FqdJjiwJ+3KBa9yJbuoyosmHEFIdRQ7rtSq2Gl3y3sA=; b=L0uRFcqMg4zxrXLXzS5agUTJtSHNHZaZB/Umstjib0KDLbzEMDkaIE5IYILiNh4ihO Ks61vZYMKGfYWtht5ScwwNerE+C+y6uWagJ0CcBgoMLA2MHnwrngn38QzloYlobOUjZ2 LyWEcAhE8EP0Ftp6XZJM/ey9/GhxkepiJvO7Zdo3G+MDxVA6RcDRZc9XqkSiJ0pa5KsN GjAVmtse3/tNYkngK2mewx7lLyeujrbP9aRghooZ2FUrMr++mTifKKsdKXJjGPNfl4VC EVTOIpaCOflVepNg538v55OTMIVlEOvHsRVJ0ZKCh92IxSJ3cgUee0q54Mfe68Bz0vwf ej5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=FqdJjiwJ+3KBa9yJbuoyosmHEFIdRQ7rtSq2Gl3y3sA=; b=yA+JhqbwSL3/J4aIEaFlAPY+4aLnwsuaVBZj0qJrFcvEORq+HbiPRnCYrp16OY3WRk GNDRQeIkx4/NPqdEg5hETYZ7RjNUsyj2a1NhNR7nVB8DNJrp2VExduWPC5gGiELCNj3d j4+r3/1ju3cv+BxLn7o0eqc937Gm7U2mzCXpiu3xqXjzcinssccAlbgjcsc7xscpF3Rx ASQABG2fZBhPqTfPw1X6rwCpXKDE356H6il6JEorV5yUSYs8PoU0yV2s6jpFs9TZfUYd UcGvjb4pVzP/wFXv7EVICjK7KeGLiiyKthqr2o10VCTVlN4ZPwqyE9Nn8ODIQzRt6jdO KvUw== X-Gm-Message-State: ANoB5pkHxviwmmyW9y+wI53RfLVfwdn5IqIYEWRubppvQTT9+h3cCMrP e3zxQ11Mdv468qqp1sXPJ/6xqr1TsoQ= X-Google-Smtp-Source: AA0mqf68WNGPX3LjDP06mA0PiEMAKBsyVZ5/dIfSh6Ot3lsLIv8000phHCdV7ZPecbA8WXpdCc4Ktcn+8wY= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:aa7:85d9:0:b0:577:53a9:836c with SMTP id z25-20020aa785d9000000b0057753a9836cmr10672135pfn.5.1670911768146; Mon, 12 Dec 2022 22:09:28 -0800 (PST) Reply-To: Sean Christopherson Date: Tue, 13 Dec 2022 06:09:12 +0000 In-Reply-To: <20221213060912.654668-1-seanjc@google.com> Mime-Version: 1.0 References: <20221213060912.654668-1-seanjc@google.com> X-Mailer: git-send-email 2.39.0.rc1.256.g54fd8350bd-goog Message-ID: <20221213060912.654668-8-seanjc@google.com> Subject: [PATCH 7/7] KVM: VMX: Handle NMI VM-Exits in noinstr region From: Sean Christopherson To: Sean Christopherson , Paolo Bonzini Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org, Peter Zijlstra , Andy Lutomirski , Thomas Gleixner Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Move VMX's handling of NMI VM-Exits into vmx_vcpu_enter_exit() so that the NMI is handled prior to leaving the safety of noinstr. Handling the NMI after leaving noinstr exposes the kernel to potential ordering problems as an instrumentation-induced fault, e.g. #DB, #BP, #PF, etc. will unblock NMIs when IRETing back to the faulting instruction. Reported-by: Peter Zijlstra Signed-off-by: Sean Christopherson Acked-by: Peter Zijlstra (Intel) Reported-by: Like Xu --- arch/x86/kvm/vmx/vmcs.h | 4 ++-- arch/x86/kvm/vmx/vmenter.S | 8 ++++---- arch/x86/kvm/vmx/vmx.c | 34 +++++++++++++++++++++------------- arch/x86/kvm/x86.h | 6 +++--- 4 files changed, 30 insertions(+), 22 deletions(-) diff --git a/arch/x86/kvm/vmx/vmcs.h b/arch/x86/kvm/vmx/vmcs.h index ac290a44a693..7c1996b433e2 100644 --- a/arch/x86/kvm/vmx/vmcs.h +++ b/arch/x86/kvm/vmx/vmcs.h @@ -75,7 +75,7 @@ struct loaded_vmcs { struct vmcs_controls_shadow controls_shadow; }; =20 -static inline bool is_intr_type(u32 intr_info, u32 type) +static __always_inline bool is_intr_type(u32 intr_info, u32 type) { const u32 mask =3D INTR_INFO_VALID_MASK | INTR_INFO_INTR_TYPE_MASK; =20 @@ -146,7 +146,7 @@ static inline bool is_icebp(u32 intr_info) return is_intr_type(intr_info, INTR_TYPE_PRIV_SW_EXCEPTION); } =20 -static inline bool is_nmi(u32 intr_info) +static __always_inline bool is_nmi(u32 intr_info) { return is_intr_type(intr_info, INTR_TYPE_NMI_INTR); } diff --git a/arch/x86/kvm/vmx/vmenter.S b/arch/x86/kvm/vmx/vmenter.S index 9d987e7e48c4..059243085211 100644 --- a/arch/x86/kvm/vmx/vmenter.S +++ b/arch/x86/kvm/vmx/vmenter.S @@ -299,6 +299,10 @@ SYM_INNER_LABEL(vmx_vmexit, SYM_L_GLOBAL) =20 SYM_FUNC_END(__vmx_vcpu_run) =20 +SYM_FUNC_START(vmx_do_nmi_irqoff) + VMX_DO_EVENT_IRQOFF call asm_exc_nmi_kvm_vmx +SYM_FUNC_END(vmx_do_nmi_irqoff) + =20 .section .text, "ax" =20 @@ -353,10 +357,6 @@ SYM_FUNC_START(vmread_error_trampoline) SYM_FUNC_END(vmread_error_trampoline) #endif =20 -SYM_FUNC_START(vmx_do_nmi_irqoff) - VMX_DO_EVENT_IRQOFF call asm_exc_nmi_kvm_vmx -SYM_FUNC_END(vmx_do_nmi_irqoff) - SYM_FUNC_START(vmx_do_interrupt_irqoff) VMX_DO_EVENT_IRQOFF CALL_NOSPEC _ASM_ARG1 SYM_FUNC_END(vmx_do_interrupt_irqoff) diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index c242e2591896..b03020ca1840 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -5095,8 +5095,13 @@ static int handle_exception_nmi(struct kvm_vcpu *vcp= u) vect_info =3D vmx->idt_vectoring_info; intr_info =3D vmx_get_intr_info(vcpu); =20 + /* + * Machine checks are handled by handle_exception_irqoff(), or by + * vmx_vcpu_run() if a #MC occurs on VM-Entry. NMIs are handled by + * vmx_vcpu_enter_exit(). + */ if (is_machine_check(intr_info) || is_nmi(intr_info)) - return 1; /* handled by handle_exception_nmi_irqoff() */ + return 1; =20 /* * Queue the exception here instead of in handle_nm_fault_irqoff(). @@ -6809,7 +6814,7 @@ static void handle_nm_fault_irqoff(struct kvm_vcpu *v= cpu) rdmsrl(MSR_IA32_XFD_ERR, vcpu->arch.guest_fpu.xfd_err); } =20 -static void handle_exception_nmi_irqoff(struct vcpu_vmx *vmx) +static void handle_exception_irqoff(struct vcpu_vmx *vmx) { u32 intr_info =3D vmx_get_intr_info(&vmx->vcpu); =20 @@ -6822,12 +6827,6 @@ static void handle_exception_nmi_irqoff(struct vcpu_= vmx *vmx) /* Handle machine checks before interrupts are enabled */ else if (is_machine_check(intr_info)) kvm_machine_check(); - /* We need to handle NMIs before interrupts are enabled */ - else if (is_nmi(intr_info)) { - kvm_before_interrupt(&vmx->vcpu, KVM_HANDLING_NMI); - vmx_do_nmi_irqoff(); - kvm_after_interrupt(&vmx->vcpu); - } } =20 static void handle_external_interrupt_irqoff(struct kvm_vcpu *vcpu) @@ -6857,7 +6856,7 @@ static void vmx_handle_exit_irqoff(struct kvm_vcpu *v= cpu) if (vmx->exit_reason.basic =3D=3D EXIT_REASON_EXTERNAL_INTERRUPT) handle_external_interrupt_irqoff(vcpu); else if (vmx->exit_reason.basic =3D=3D EXIT_REASON_EXCEPTION_NMI) - handle_exception_nmi_irqoff(vmx); + handle_exception_irqoff(vmx); } =20 /* @@ -7119,6 +7118,18 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_v= cpu *vcpu, =20 vmx_enable_fb_clear(vmx); =20 + if (unlikely(vmx->fail)) + vmx->exit_reason.full =3D 0xdead; + else + vmx->exit_reason.full =3D vmcs_read32(VM_EXIT_REASON); + + if ((u16)vmx->exit_reason.basic =3D=3D EXIT_REASON_EXCEPTION_NMI && + is_nmi(vmx_get_intr_info(vcpu))) { + kvm_before_interrupt(vcpu, KVM_HANDLING_NMI); + vmx_do_nmi_irqoff(); + kvm_after_interrupt(vcpu); + } + guest_state_exit_irqoff(); } =20 @@ -7260,12 +7271,9 @@ static fastpath_t vmx_vcpu_run(struct kvm_vcpu *vcpu) =20 vmx->idt_vectoring_info =3D 0; =20 - if (unlikely(vmx->fail)) { - vmx->exit_reason.full =3D 0xdead; + if (unlikely(vmx->fail)) return EXIT_FASTPATH_NONE; - } =20 - vmx->exit_reason.full =3D vmcs_read32(VM_EXIT_REASON); if (unlikely((u16)vmx->exit_reason.basic =3D=3D EXIT_REASON_MCE_DURING_VM= ENTRY)) kvm_machine_check(); =20 diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 9de72586f406..44d1827f0a30 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -382,13 +382,13 @@ enum kvm_intr_type { KVM_HANDLING_NMI, }; =20 -static inline void kvm_before_interrupt(struct kvm_vcpu *vcpu, - enum kvm_intr_type intr) +static __always_inline void kvm_before_interrupt(struct kvm_vcpu *vcpu, + enum kvm_intr_type intr) { WRITE_ONCE(vcpu->arch.handling_intr_from_guest, (u8)intr); } =20 -static inline void kvm_after_interrupt(struct kvm_vcpu *vcpu) +static __always_inline void kvm_after_interrupt(struct kvm_vcpu *vcpu) { WRITE_ONCE(vcpu->arch.handling_intr_from_guest, 0); } --=20 2.39.0.rc1.256.g54fd8350bd-goog